A few weeks ago, we open sourced our proof of concept of a DataPlane for Bluesky. It indexes posts and follows and serves timelines.

Getting from there to a complete DataPlane is a lot of work. In this post, we first outline what that work is, with Postgres as the first database. Then we look at a compromise that makes the proof of concept useful long before that: running it next to Bluesky's open source DataPlane and letting it serve only the timelines.

A complete DataPlane in Elixir

Our goal, as outlined in our first post, is an open source DataPlane that is low effort to operate and scales way better than the reference implementation.

In our companion post, we split the DataPlane's work into two paths.

The hot path serves timelines from memory. The cold path covers records, threads and profiles, and lives in a database.

For the cold path, we want a storage interface, an Elixir behaviour, so the database can be swapped. Postgres will be the first adapter. Everyone who runs an AppView today already has their data in it, and the reference schema works fine for the cold path as long as you don't build timelines from it. If we start from that schema, an existing AppView can switch to our DataPlane without migrating its data.

Our proof of concept covers a small part of the hot path.

Here is what's missing:

• A gRPC endpoint for Bluesky's bsky.proto, which currently declares 124 methods. Many of them are lookups by URI or DID. Some, like search and notifications, are more work.
• An indexer for every record type. The reference implementation has 21 indexer plugins: posts, likes, reposts, follows, blocks, lists, starter packs, verifications and more. Deletes need some care, because the firehose only tells you which record is gone, not what was in it.
• A write path that batches inserts into Postgres and stores the firehose cursor in the same transaction as the records. That makes replaying events after a crash safe. The in-memory structures only get updated after the commit.
• A subscription to bsync. Mutes and bookmarks don't come through the firehose.
• A backfill. The firehose only tells you what happens from now on, so a new instance has to fetch the existing repositories first.
• The follower graph in Roaring Bitmaps, with a stable mapping from DIDs to the integers the bitmaps need. Stable, because we want to snapshot the graph to disk to make restarts faster.
• A startup routine that rebuilds the hot path from Postgres: load the follower graph and the recent posts, run the fan-out, and only then take traffic.

Most of it isn't hard, and most of the endpoints are straightforward, as we noted back in March. It's just a lot of work, and nobody can run the DataPlane in production before all of it is done.

The next step: a timeline proxy

The timeline is where Bluesky's open source DataPlane gives in first. As we described in our first post on the topic, the user experience often degrades with fewer than 100_000 users. Many of the other requests are lookups by URI or DID, which Postgres handles well for much longer.

So instead of waiting for the complete version, we can run our DataPlane next to the open source one and let it serve only the timelines. That's the part our proof of concept already covers. In the rest of this post, we look at how such a setup would work and what it takes to keep the two DataPlanes consistent.

Splitting the DataPlane API

The setup we have in mind builds on how the AppView talks to its DataPlane. It uses gRPC, and every call is a POST to a path that names the method, for example /bsky.Service/GetTimeline. That makes it easy to send different methods to different services.

The AppView does accept a list of DataPlanes in BSKY_DATAPLANE_URLS. But it picks one of them at random for every call, and only tries another one when a call fails with Unavailable or Aborted. That's meant for balancing load between identical DataPlanes and won't help us here. Instead, we put a proxy that understands gRPC in between.

With Envoy, most of the work is done by two routes:

route_config:
  virtual_hosts:
    - name: dataplane
      domains: ["*"]
      routes:
        - match: { path: "/bsky.Service/GetTimeline" }
          route: { cluster: dataplane_elixir }
        - match: { prefix: "/" }
          route: { cluster: dataplane_node }

GetTimeline goes to our Elixir DataPlane, every other call goes to Bluesky's. The Node DataPlane keeps doing what it did before: it consumes the firehose, writes to Postgres and answers all the other methods.

The Node DataPlane only speaks HTTP/1.1, so the status of each gRPC call comes back in an HTTP/1.1 trailer. Envoy drops those by default, so the Node cluster needs enable_trailers.

Our DataPlane gets its data from Postgres rather than from the firehose.

Our proof of concept can't take over that route just yet. It answers JSON on /bsky.Service/GetTimeline, which is all our simulator needs, while the AppView sends protobuf messages over gRPC. So the gRPC endpoint from the list above is the first thing the timeline proxy needs, too.

Where the IDs come from

The DataPlane only returns a skeleton of a timeline: a list of post URIs, a repost URI where an item is a repost, and a cursor. The AppView then hydrates that skeleton into full views. It calls GetPostRecords, GetRepostRecords, GetActors, GetInteractionCounts and friends, and all of those end up at the Node DataPlane and Postgres.

So whatever our DataPlane returns has to mean the same thing to Postgres. Luckily, atproto already gives us IDs that mean the same everywhere. A post is identified by its AT-URI, like at://did:plc:…/app.bsky.feed.post/3k…, and a user by their DID. Both come from the network, not from either database.

Our proof of concept doesn't use them yet. It numbers posts with a counter as they arrive. That's good enough for the simulator, but Postgres has never heard of post number 4711. The timeline entries have to carry URIs instead.

The integer IDs that Roaring Bitmaps need for the follower graph can stay private. We can keep a mapping from DID to integer in one ETS table and the reverse in another, and translate right before we build the response. Nobody outside our DataPlane ever sees them, so they don't have to match anything in Postgres.

Agreeing on time

With URIs in place, both DataPlanes talk about the same posts. They can still disagree about when a post exists. If both consume the firehose on their own, our DataPlane may put a post into a timeline before the Node DataPlane has written it to Postgres. The AppView can't hydrate that post and drops it from the page. Reload a second later and it's there. It's not a disaster, but users notice that kind of flakiness.

The way out is to not read the firehose at all, and read Postgres instead. More precisely, our DataPlane follows the changes to two tables with logical replication:

• follow holds the follower graph.
• feed_item holds a row for every post and every repost, with its author (originatorDid), its CID and the timestamp timelines are sorted by (sortAt).

Those are exactly the two tables the reference implementation joins for every single timeline request. The difference is that we don't join them on read. We fan out on write instead.

This gives us a useful guarantee: every URI we hand out is already committed in Postgres, so hydration will find it. The opposite case is harmless. A deleted post can linger in our memory for a moment, hydration comes back empty and the AppView drops it.

On the Postgres side, we need a publication for the two tables:

-- requires wal_level = logical, and the Elixir side connects
-- with a role that has the REPLICATION attribute
CREATE PUBLICATION timeline FOR TABLE bsky.follow, bsky.feed_item;

ALTER TABLE bsky.follow REPLICA IDENTITY FULL;
ALTER TABLE bsky.feed_item REPLICA IDENTITY FULL;

By default, a delete only carries the primary key, which is the record's URI for both tables. With REPLICA IDENTITY FULL, an unfollow also tells us who unfollowed whom, and a deleted post tells us its author. We pay for that with a bit more WAL, but we don't need an extra lookup table just to process deletes.

On the Elixir side, Postgrex comes with Postgrex.ReplicationConnection. A sketch of our replication process:

defmodule DataplaneEx.Replication do
  use Postgrex.ReplicationConnection

  alias DataplaneEx.{Bootstrap, Indexer, PgOutput}

  @impl true
  def handle_connect(state) do
    query = "CREATE_REPLICATION_SLOT timeline TEMPORARY LOGICAL pgoutput EXPORT_SNAPSHOT"
    {:query, query, state}
  end

  @impl true
  def handle_result([%Postgrex.Result{rows: [[_slot, _lsn, snapshot, _plugin]]}], state) do
    # drop what we had, then load follows and recent feed items as of the snapshot
    :ok = Bootstrap.load(snapshot)

    query =
      "START_REPLICATION SLOT timeline LOGICAL 0/0 " <>
        "(proto_version '1', publication_names 'timeline')"

    {:stream, query, [], state}
  end

  @impl true
  def handle_data(<<?w, _wal_start::64, _wal_end::64, _clock::64, message::binary>>, state) do
    message
    |> PgOutput.decode()
    |> Indexer.apply_change()

    {:noreply, state}
  end

  # plus start_link, init and the keepalive handling
  # straight from the Postgrex docs
end

Postgrex hands us the raw pgoutput messages, decoding them is up to us. It's a small, well documented binary protocol, and binary pattern matching is one of the things Elixir is really good at.

Persistence for free

This also takes care of a problem from our last post: our data doesn't survive a restart. With Postgres as the source of truth, it doesn't have to.

On every start, we create a temporary replication slot and let Postgres export a snapshot along with it. A second connection opens a transaction on exactly that snapshot (SET TRANSACTION SNAPSHOT '…'), loads the follows and the recent feed items, and runs the fan-out. Then we start streaming from the slot. Changes that happen during the load wait in the slot, so nothing gets lost and nothing arrives twice. And since the slot is temporary, Postgres doesn't hold on to WAL for a node that is down.

Cursors and falling back

The AppView pages through a timeline with a cursor it gets from the DataPlane. It doesn't care which DataPlane created it. The reference implementation builds the cursor from the last item on the page: its sortAt in milliseconds and its CID, joined by two underscores, like 1792497600000__bafyrei….

If our DataPlane produces cursors in exactly that format, the Node DataPlane can continue any timeline we started. That's our way out whenever we can't answer a request ourselves:

• The user's timeline isn't there yet, for example because we are still loading after a restart.
• The cursor points to posts older than what we keep in memory. Timelines are bounded, so somebody who scrolls far enough will get there.

In both cases, our DataPlane calls GetTimeline on the Node DataPlane with the same cursor and passes the answer on. For that to work, our timeline entries carry sortAt and CID next to the URIs. Sorting by them also gives us the same order as the reference implementation, ties included.

The reference implementation does two more things we have to copy. It mixes the user's own posts into their timeline, up to ten per page even when followed accounts fill it. And it supports a since parameter that bounds the range and comes back as the cursor once that range is used up. If our DataPlane behaves differently, users would see their timeline change whenever we hand over to the Node DataPlane.

What the timeline proxy doesn't solve

This setup takes the most expensive query off Postgres. It doesn't change anything else:

• The Node DataPlane still indexes the firehose and writes everything to Postgres. This is no improvement, but indexing is not a bottleneck in normal operations.
• Threads, profiles, notifications, search and all the other requests are still served by the Node DataPlane from Postgres.
• Postgres stays a single database that you can't scale horizontally.

So it lets you grow further with the open source DataPlane, but it won't get you anywhere near the size of Bluesky. But that's not our goal here: we are aiming to close the gap between a small community and over 100k users, until the user count ensures a serious engineering budget.

Our simulator works with both DataPlanes, so that's the tool we'll use to find out how far your setup can go.

From the timeline proxy to the complete DataPlane

The timeline proxy also gives us a way to get to the complete version step by step.

Every method we implement in Elixir becomes another route in Envoy. We can move them over one at a time, starting with the ones that hurt most. At first, they read the same Postgres tables the Node DataPlane writes to, which is why starting from the reference schema matters so much.

Before we switch a route, we want to know that our answers match. Envoy can mirror requests to our DataPlane, but it throws the mirrored responses away. So the comparison has to happen on our side: for a sample of requests, our DataPlane also asks the Node DataPlane and logs any difference.

The last piece to move is the indexer. Once our DataPlane consumes the firehose itself and writes to Postgres, it doesn't need the replication stream anymore. At that point, the Node DataPlane can be switched off, and we have the complete DataPlane from the beginning of this post.

This is still a side project for us. If you run an AppView and your Postgres is already sweating over timelines, we'd like to hear from you. Give us a shout if you want to help us take it further, faster.