Systems Papers - Meta’s Next-generation Realtime Monitoring and Analytics Platform

Hello new and old subscribers!

This is one of my first posts on Substack - if you have any feedback, I would love to hear it! Feel free to respond to this email with what is on your mind or if you have found any interesting papers recently.

This week’s paper is Meta’s Next-generation Realtime Monitoring and Analytics Platform.

Many companies perform real time analytics in order to customize user experiences or alert on anomalous behavior, and Meta performed this function using a system called Scuba. While the existing system scaled to meet initial demand, over time operational and usability issues led to an internal push to evolve.

Beyond discussing the tradeoffs and challenges to the project, the paper provides extensive technical references to the underlying primitives that Kraken is built on (one of which, Shard Manager, was a previous paper review), making this a great reference for learning about distributed systems.

The paper review is best enjoyed on my blog.

Until next time,

Micah