Shard Manager: A Generic Shard Management Framework for Geo-distributed Applications

Hello new and old subscribers!

This week’s paper, Shard Manager: A Generic Shard Management Framework for Geo-distributed Applications, describes a system for running sharded applications at scale within Facebook - link to the paper review here.

Application sharding assigns subsets of requests to instances of an application, allowing tasks to specialize - a group of tasks specialized for a subset of requests is called a shard.

This approach is particularly useful when tasks fetch state or other metadata. As an example, a speech recognition application may load machine learning models to process languages. Assigning requests for different languages to different shards means that an application’s tasks don’t need to download every model (which would be time and bandwidth-intensive).

The paper review is best read on my blog.

As always, feel free to reach out with comments or suggestions by responding to this email.

Until next time!

Micah