There is a new open social ecosystem emerging around atproto. Never heard of it? You should check it out, it's cool tech with an ethos.
The open system is a dream for builders: from day one you can tap into the social graph of more than 40 million Bluesky users. Building on this userbase, you can create everyones new favorite social network on atproto. It's no surprise that there is already a number of alternative communities emerging: Northsky, Eurosky, Blacksky, to name a few.
This is a blog post though, so you already know there is a problem.
The Bluesky dataplane
Bluesky runs almost entirely on open sourced components which enables alternatives to get started quickly.
There is one notable exception: the part of the stack that processes the event stream of the atproto firehose, the AppView, needs one component to digest and store data. This part is called dataplane, and it does the most heavy lifting in processing the event stream. However, the open source dataplane implementation is not very performant. Bluesky uses a closed source dataplane implementation in production for this reason.
The open source dataplane functions on fan-in principle:
- events are streamed into the system (e.g. "new post")
- the dataplane stores and indexes the events in ordinary Postgres tables
- on user request, data is queried and presented to the user (fan-in)
This approach leads to the classic Twitter timeline problem, and in fact limits the user numbers that can safely be served to 100k users, often less.
Fan-out to ETS dataplane
The trick to solve this problem is to change the approach. Instead of querying the data on user request, we prepare data in a dedicated place for each user on write (fan-out). When the user makes a request, the data is already sitting there, just waiting to be served to the user.
This is the also the approach followed by the Bluesky closed source dataplane which builds on ScyllaDB. From all publicly available information, it seems to be pretty optimized to their hardware, and pretty expensive to run.
As you know, we are big fans of Elixir at bitcrowd, so we've built a Proof of Concept of a dataplane with ETS taking on the role of ScyllaDB. The idea here is that we can build something that runs on commodity hardware but scales way better than Bluesky's open source dataplane implementation.
A peek into performance
To evaluate the PoC, we measured reading and writing directly to the dataplane. We compared a Postgres backed version (representing Bluesky's open source implementation) with an ETS backed version.
We simulated write traffic and let a number of users make requests to load their timeline (get_timeline).
In the dashboards below, you can see the number of simultaneously active users ("Active Sessions"), the write traffic ("Posts created"), the latencies of get_timeline requests, and the throughput of get_timeline requests.
Under heavy load, the Postgres backed fan-in implementation showed significantly larger latencies and lower throughput in comparison to the ETS backed fan-out implementation.
Advanced simulations
We extended this PoC into a performance toolkit you can use to simulate traffic via atproto events and user requests.
As a basic principle, we want to be able to repeatedly simulate scenarios. Therefore, you provide the configuration for the scenario as file that can be repeatedly loaded into the system.
We provide three components:
Base data
You can quickly create base data, such as users, to prepare your application instance for the simulated scenario. It makes a difference for your app's performance whether you already have millions of users in your database or none.
Traffic simulation
Based on the simulation plan, we perform requests or emit events to create read and write traffic.
Measuring performance
Based on the simulated traffic, we can measure the performance users would experience. For instance, how long would it take for their timeline to load?