Posting: saved once, delivered after
The post service writes the post and, in the same write, an outbox row saying “deliver this post”, as in the transactional outbox (opens in a new tab) pattern. It adds the id to the author’s own list, so they see the post at once, and returns 201. A relay moves outbox rows onto a durable log partitioned by author, and workers page through the followers and add the id to their lists in batches. Each list holds an id once, so a retried batch adds nothing (Redis sorted sets (opens in a new tab) are collections of unique members).
- The app posts, with the same
client_post_idon every retry. - One write stores the post and its outbox row.
- The author’s own list gets the id; the reply is 201.
- The relay publishes the row to the delivery log.
- Workers add the id to followers’ lists in batches.
Diagram as text
Components: Posts and outbox (NoSQL DB); Outbox relay (Worker); Delivery log (Queue); Phone (Client); Post service (Service); Delivery workers (Worker); Follow graph (SQL DB); Author’s own list (Cache); Followers’ lists (Cache). Connections: Phone → Post service (POST /v1/posts); Post service → Posts and outbox (post + outbox row); Post service → Author’s own list (add id, then reply); Outbox relay → Posts and outbox (read outbox); Outbox relay → Delivery log (publish), asynchronous; Delivery workers → Delivery log (consume); Delivery workers → Follow graph (page followers); Delivery workers → Followers’ lists (add ids in batches).
Read:Learn: Queues, delivery semantics and backpressureLearn: Apache Kafka
Keep reading+1 min
A change stream instead of an outbox
Driving delivery from the posts store’s change stream works too, with a sweep that re-delivers anything the stream expired before it was read: DynamoDB Streams (opens in a new tab), for example, keeps its log for up to 24 hours. Either way the delivery work is recorded with the post, so a crash can’t lose it.
Check yourself
Why write the outbox row in the same write as the post?