03
Two sources of posts
Now it gathers candidates from two places, at the same time. Posts from accounts you follow (“in-network”, served by Thunder), and posts from people you don’t follow but might enjoy (“out-of-network”, found by Phoenix).
Two scouts go out: one combs through your inner circle’s posts, the other searches the entire world for things that smell like you.
Under the hood: Thunder is a Rust in-memory store fed by Kafka, with sub-millisecond lookups. Phoenix retrieval uses a two-tower model: one tower turns you into a vector, another turns posts into vectors, and the closest matches win. thunder/ · phoenix/recsys_retrieval_model.py
A little deeper
Thunder ingests post create and delete events from Kafka and keeps recent posts in memory, organized per user. It can answer “what did the people I follow just post?” in under a millisecond.
Phoenix retrieval is the magic of the “For You” feed: it can surface posts from people you’ve never heard of. The user tower encodes your taste; the candidate tower encodes every post; a similarity search picks the best matches.
View on GitHub →Made by Henrique Martins