Content recommendation

The instinct to "just train one model to score everything" dies on latency arithmetic, and walking through it is the core signal.

A rich ranking model costs, say, tens of microseconds to score one (user, item) pair with all its features. Scoring a 100-million-item catalogue per request would take minutes — utterly impossible inside a ~50 ms budget. So you split the problem by what each stage optimises:

• Candidate generation optimises recall, cheaply. ANN over embeddings, CF lookups, and trending lists each return their best few hundred items in milliseconds using sub-linear or precomputed structures. The goal is "don't miss anything good" while keeping the set small (~1000). Errors here are omissions.
• Ranking optimises precision, expensively. Now that the set is bounded, you can afford a heavy model with rich user×item cross features to order those ~1000 precisely and pick the top K. Errors here are mis-orderings.

This is the same cheap-recall-then-expensive-rank shape as search and geospatial dispatch, and naming that parallel signals pattern fluency. A subtle point: the two stages are trained on different objectives (retrieval vs ranking) and can disagree — the ranker may down-rank a candidate the retriever loved — which is healthy; the funnel is designed to be generous early and selective late.

A recommender that retrieves candidates from one source inherits that source's blind spots, and the fix is to blend complementary channels.

Each channel contributes a different kind of signal:

• Embedding ANN — personalised similarity ("things like what you engage with").
• Collaborative filtering — taste-neighbourhood signal ("people like you liked").
• Trending / popular-in-segment — social proof and timeliness.
• Fresh / exploration — new items with no history yet, so the catalogue doesn't ossify.
• Social ("from people you follow") — relationship signal where it applies.

You pull the top-N from each channel, merge and dedupe into a single ~1000-candidate pool (an item retrieved by several channels is a strong signal), and hand it to the ranker. The blend is what keeps the slate diverse: relying on embeddings alone amplifies whatever the embeddings over-favour; adding trending and fresh channels guarantees timely and new content gets a shot. The channel weights themselves become a tuning knob — and a place to enforce product goals (e.g. guarantee a fresh slot, or cap any one creator). The phrase to land: candidate generation is recall; ranking is precision — and recall is a portfolio, not a single bet.

The most common way a recommender that "works in the notebook" fails in production isn't a bad model — it's train/serve feature skew.

The trap: training features are computed in a batch job (SQL joins over a warehouse, "30-day watch time" defined one way, timestamps rounded one way), while serving features are computed in the online path (cached values, a slightly different window, different rounding). The model learns the relationship between the training feature distribution and the label, then at serving time sees subtly different numbers — so its predictions degrade in ways offline metrics never reveal. Offline AUC goes up; online CTR stays flat.

The fix is a feature store with a single source of truth for feature definitions:

• One pipeline computes each feature, materialising it to an offline store (for training) and an online store (for low-latency serving), guaranteeing the same definition in both.
• Streaming features (last 5 clicks, current session) flow through Kafka into the online layer so the model can react in-session.
• Feature definitions are versioned, and the exact feature values used at serving time are logged, so an offline debugging run can reproduce the real decision.

"Train/serve skew kills recommender quality" and "one codepath generates features for both train and serve" are the sentences that show you've operated one of these systems rather than just read about the model.

A recommender is a closed loop, and left greedy it eats itself.

Every slate you serve generates the impressions and clicks that become training data for the next model. But the model can only get feedback on items it showed. A purely greedy ranker shows what it already believes users like, so it only ever learns about those items, reinforces its existing beliefs, and progressively buries everything else — new content, niche tastes, the long tail. Coverage shrinks into a filter bubble, and the selection bias in the logs (you only see labels for shown items) means the model is learning "what the old system showed," not "what users actually want."

Two mechanisms counter this:

• Exploration. Reserve ~5–10% of slate slots for non-greedy picks — fresh items, high-uncertainty items, or bandit-selected variants. Epsilon-greedy is the simple form; Thompson sampling / contextual bandits concentrate exploration where uncertainty is highest and decay it as confidence grows. You trade a little immediate CTR for the signal that keeps the model learning and the catalogue alive.
• Right metrics. Optimising pure short-term CTR accelerates the collapse (clickbait, repetition). Track long-horizon retention, catalogue coverage, and diversity alongside CTR so the loop's health is visible.

The interview line: no impressions, no labels — and a greedy system stops generating impressions for anything new, so exploration isn't a nicety, it's what keeps the training data honest.

Collaborative signal needs history, and two situations have none — a new user and a new item — which require different fallbacks. Conflating them is a common miss.

New user (no interaction history). CF and personalised embeddings have nothing to work with, so fall back to popularity-in-segment / context: best items for the user's coarse attributes (region, device, referral, declared interests at signup). Optionally collect a few lightweight preference signals up front. Then progressively personalise as the first interactions arrive — the system should visibly adapt within the first session, not pretend it knows the user.

New item (no clicks yet). It has no CF column and no learned embedding from behaviour, but it does have content: title, description, category, creator, media. Use a content-based embedding (from item metadata) to place it near similar items in the ANN space, so it can be retrieved for users who like that neighbourhood — and give it exploration impressions so it can earn real engagement signal. Once it accumulates interactions, the behavioural embedding takes over.

The general principle: cold-start is solved by falling back to a different signal that doesn't require history (content, popularity) and then graduating to behavioural signal as data accrues. Naming both the user-side and item-side fallbacks distinctly is the senior tell.

A rec call has a hard latency budget — often ~50 ms p99, because it sits on a page load — and the whole architecture is shaped by it.

The budget is split across the funnel: candidate generation (ANN lookup + channel merges), feature fetching, and ranking. The two places it blows up:

• Candidate count × feature lookups. Ranking N candidates means fetching features for N items and N user×item pairs. If you naively fetch features one item at a time, latency explodes. The fixes: cap the candidate count (more candidates rarely improve the final slate proportionally), batch the feature fetch into one multi-get against the online store, and precompute/cache user features per request so they're fetched once, not per candidate.
• Model cost. A heavy neural ranker over 1000 candidates can exceed the budget. Mitigations: keep the ranker lean (GBDT is fast), cap candidates, warm model and feature caches, and consider a lightweight pre-ranker to trim 1000→200 before the expensive model — a mini two-stage within ranking.

Caching helps broadly: per-user candidate sets and feature vectors can be cached for the session, and popular contexts (logged-out home, trending rows) can be cached outright. The discipline is to treat the budget as a design constraint, allocate it across stages explicitly, and make every per-candidate operation batched — the same way you'd budget a search query.

YouTube's recommendation system is the textbook reference for the two-stage architecture, described in their 2016 paper "Deep Neural Networks for YouTube Recommendations." Faced with a corpus of hundreds of millions of videos and a hard serving latency budget, they explicitly split the problem into candidate generation and ranking.

The candidate-generation network treats recommendation as extreme multiclass classification — it learns user and video embeddings and, at serving time, performs an approximate nearest-neighbour lookup of the user vector against video vectors to retrieve a few hundred candidates from the millions, in milliseconds. This is the two-tower/ANN retrieval pattern at planet scale. The ranking network then scores those few hundred candidates with a much richer feature set — including features that only make sense on a small candidate set, like the user's history with that specific channel, time since last watch, and impression-frequency capping — and orders them.

Two details the paper emphasises that map directly onto this pattern: they rank on expected watch time, not click probability, because clicks reward clickbait while watch time rewards genuine satisfaction (the "right metric" lesson); and they take great care with train/serve consistency and feature freshness, including using the age of a training example as a feature so the model doesn't over-favour stale popular videos. The architecture is the blueprint: cheap deep-retrieval candidate gen → rich deep ranking → careful objective and feature engineering.

Spotify's Discover Weekly is a celebrated example of multi-channel candidate generation producing a personalised, diverse slate. It blends three complementary signals rather than betting on one: collaborative filtering over listening behaviour and the billions of user-curated playlists (tracks that co-occur in similar playlists are similar — exactly the taste-neighbourhood signal CF captures); NLP over text scraped from the web, blogs, and playlist titles to build a semantic profile of artists and tracks; and raw audio models (CNNs over spectrograms) that let brand-new tracks with no listening history be placed near sonically similar songs — a direct content-based cold-start solution for items.

That combination is why Discover Weekly can surface obscure and new music a pure CF system would never reach: CF supplies the collaborative backbone, while the content (text + audio) channels rescue the long tail and cold items. The weekly cadence is itself a form of structured exploration — a fresh slate every Monday keeps users discovering rather than looping the same favourites, and Spotify watches engagement with these recommendations to feed the next cycle.

For an interview, Discover Weekly is the perfect illustration of two pattern principles at once: never a single candidate source (CF + NLP + audio), and content-based embeddings solve item cold-start (audio models for tracks with no plays yet).

Netflix treats the home screen as a personalised ranking problem at two levels: which rows (genres/themes) to show, and which titles to rank within each row — both personalised per member. Their long-running work (rooted in the Netflix Prize era and far beyond it) established several lessons this pattern encodes.

First, the business metric matters more than offline accuracy: Netflix optimises for long-term member retention, not raw predicted-rating error, and has publicly noted that the rating-prediction accuracy that won the Prize was less valuable than improvements to ranking, page construction, and diversity. This is the "optimise the right metric" lesson taken to its conclusion — they evaluate via online A/B tests on retention, not offline AUC. Second, diversity and page construction are first-class: a row of ten near-identical thrillers underperforms a varied row, so re-ranking enforces diversity rather than pure relevance. Third, even the artwork/thumbnail is personalised and bandit-optimised — a contextual-bandit problem that surfaces different imagery to different members, a concrete instance of the exploration layer.

Netflix is the case study for the parts of recommendation beyond the model: optimise the real objective (retention), construct and diversify the whole page, run everything through online experiments, and use bandits for exploration even at the presentation layer.