Comment by abronan

One detail is that it relies on Strong Eventual Consistency, which is a slightly stronger model than Eventual Consistency. Replicas that have observed the same updates are guaranteed to converge to the same end state, regardless of the order in which those updates have been received.

That said, convergence alone doesn’t guarantee scheduling correctness. This is why it separates convergence from scheduling coordination. The CRDTs and MSTs provide the replicated state, and the scheduling protocol handles placement decisions and conflicting operations (with atomic slot reservation that is node local). There is a reconciliation loop (owned by a single node that is picked for the service/workload to deploy) that is then responsible for bringing the cluster back to its desired state if it potentially made a wrong call.

And it comes with trade-offs as mentioned in my post. You accept temporary divergences rather than requiring global agreement before every operation. This may or may not be acceptable for certain workloads. This removes the need for a quorum-based and centralized control plane, but it also means that certain invariants and guarantees have to be enforced at the scheduling and workload layer rather than by the datastore itself.

Strong consistency is also not a silver bullet to handle distributed systems. It is relatively hard to get it right and not serve stale values because of the presence of bugs in the protocol or state corruption. It could also bring the availability down and prevent from scheduling any more tasks due to a loss of quorum (which is by design and acceptable in most cases).

Mantissa is just making a different trade-off and accepting a different set of problems to solve: favoring availability and accepting that the desired state could shift temporarily on partition and network blips. If your cluster has good connectivity, then it should be a rare event, for which you should have prepared with some replicas that could handle being moved around. Again, it is not for everybody, and will probably remain a niche tool for users who want a simplified cluster deployment/updates to schedule mostly stateless tasks (even though stateful tasks are supported).

One remaining comment: replicated volumes do use a Raft quorum per volume. There are things that couldn't accept anything weaker than strong consistency, and volumes are one of them. So we are using strong eventual consistency for most of the state replication, but fallback to Raft for things that truly require strong consistency.