The impact of technological failures can take their toll on a company, such as Uber. So, to limit the impacts of node failures, maintenance, and cluster scaling, Uber has re-designed shard placement in M3DB by introducing fixed-size sub-clusters. Note, though, that this won’t remove the failure, but at the very least, it can limit their effects. Here’s the scoop.
Shard placement in M3DB
Uber has successfully re-designed shard placement in M3DB through introducing fixed-size sub-clusters in order to limit the ill-effects of node failures, maintenance, and cluster scaling.
Through this, the limitation in the previous placement model can be addressed, wherein a node failure may affect up to (n-1)/n of the cluster as shard dependencies grew and developed.
There are two things to discuss here: shard placement and M3DB. Let’s start with shard placement.
In industries, shard placement pertains to the process of deciding which physical server, node, or container in a cluster stores a specific database shard. There are four sub-processes involved here:
- Distribution
- Allocation
- Balancing
- Isolation
On the other hand, M3DB is the distributed time-series database of Uber, wherein data is divided into shards and replicated across several nodes. Its placement algorithms determine the ownershi of the shards while enforcing isolation in-between replicas, such as organizing these.
According to Uber’s official engineers, “the original sharded placement model worked well for small-to-medium clusters but became more difficult to operate as clusters grew.”
Further notations
With the original model, all nodes could own a shard, provided that the replicas were not placed in the similar isolation group. Specifically, in a permissive configuration, this can create a dependency graph in which a topology change affects O(N) nodes. A node can share data to increase recovery activity.
This also partitions the nodes into fixed-sized sub-clusters. Each sub-cluster owns a distinct, non-overlapping portion of the shard space. Interesting!
Scaling is also implemented here through Uber’s greedy algorithm that can evaluate the effect of removing each candidate shard from the donor sub-cluster, selecting shards that leave the rest of the nodes evenly loaded as possible.
However, engineers admit that the sub-cluster approach has deficiencies. It requires equal instance weights, scaling in multiples of the sub-cluster size + a sub-cluster size that is a multiple of the replication factor.
So, the mend this, the ridesharing app retained M3DB’s existing instance-level placement operations rather than introducing atomic subcluster operations. This is effective to preserve the compatibility with existing tooling and avoids triggering a large bootstrap operation in which several shards migrate simultaneously. This time, M3DB’s placement implementation constitutes fields for sub-clustered placement and the number of instances per sub-cluster.
Got questions about these strategies? Open your account on RSF to share your thoughts! You may also share this story around online with your family and friends. Uber is a ridesharing app that makes it way easier for passengers to find drivers to bring them to their destinations. Stay with RSF, the newsroom for ridesharing, coupons, food deliveries, business, lifestyle, and so much more. ‘Til then!