Apache Flink
A stream-processing engine for stateful real-time computation: windows, joins, enrichment and live materialised views.
Also worth naming: Apache Flink · Kafka Streams (lighter, library-based) · Spark Structured Streaming · Amazon Kinesis Data Analytics
Flink prepares you for prompts where moving events is not enough and the system must compute on them as they arrive. Use it when correctness needs event time, managed state and checkpoint recovery.
What it is
Apache Flink is a distributed stream-processing engine: it runs continuous computations over unbounded streams of events with low latency and stateful operators. Where Kafka is the durable pipe that moves events, Flink is the engine that computes on them as they flow — counting clicks per ad per minute, joining a stream of orders with a stream of payments, detecting fraud patterns, maintaining a live materialized view — and emits results continuously.
Two things make Flink more than a simple consumer loop. First, managed state: operators remember things across events (running counts, window contents, join buffers), and Flink stores that state durably and checkpoints it, so a failed job restarts from the last checkpoint with its state intact — giving exactly-once processing semantics within the pipeline. Second, event-time processing with watermarks: because events arrive late and out of order in the real world, Flink computes over the time an event actually happened, not when it was received, so a 1-minute window is correct even if some events show up seconds late.
In an interview, Flink is the answer when the requirement is real-time computation over a stream — windowed aggregations, stream joins, sessionization, anomaly detection, live dashboards, or a continuously refreshed SQL/materialized-table pipeline — as opposed to just transporting events (Kafka) or batch-crunching them later (a warehouse/Spark batch). The canonical shape is Kafka → Flink → store/topic: Kafka is the source and the checkpoint-friendly substrate; Flink does the stateful work; results land in a database, lakehouse table, cache, or another topic. Reach for it when you need answers as the data arrives, and not when a periodic batch job or a simple consumer would do.
When to reach for it
Reach for this when…
- Real-time windowed aggregations — counts, sums, percentiles per interval (ad clicks, metrics)
- Stream joins or enrichment — combining two live streams, or a stream with reference data
- Sessionization, pattern detection, or anomaly/fraud detection on live events
- Maintaining a continuously-updated materialized view or live dashboard
Not really this pattern when…
- You only need to move events, not compute on them (that is Kafka alone)
- Periodic batch analytics is fine — a warehouse / Spark batch is simpler and cheaper
- The aggregation is trivial and low-volume — a simple consumer writing to a DB may suffice
- Light, Kafka-only processing — Kafka Streams (a library) avoids running a Flink cluster
How it works
Four ideas explain what Flink does and why:
1. Continuous computation over an unbounded stream. A Flink job is a dataflow graph — source → operators → sink — that runs forever, processing each event as it arrives rather than in scheduled batches. Operators transform, key, window, aggregate, and join; the source is typically a Kafka topic and the sink a database, cache, or another topic.
Flink reads an unbounded stream from a source (usually Kafka), transforms it through operators that keep state — windowed counts, joins, aggregations — and writes results to a sink, forever. State is checkpointed so the job recovers exactly where it left off.
2. Windows turn infinity into something computable. You can't "sum a stream" — it never ends. So aggregations operate over windows: tumbling (fixed, non-overlapping — every 1 minute), sliding (overlapping), or session (bounded by inactivity gaps). "Clicks per ad per minute" is a tumbling window keyed by ad id.
Because a stream never ends, aggregations operate over windows — fixed tumbling intervals (e.g. every 1 minute), sliding windows, or activity-based sessions. Event-time windows with watermarks handle events that arrive late or out of order.
3. Event time + watermarks handle the messy real world. Events arrive late and out of order (a mobile client was offline; a network hiccup). Flink computes over event time (when it happened) not processing time (when it arrived), and uses watermarks — a moving assertion that "no events older than T will arrive" — to decide when a window is complete and can emit, while still handling stragglers. This is what makes streaming results correct, not just fast.
4. Checkpointed state is a source/Flink/sink contract. Operators hold state (window contents, running aggregates, join buffers). Flink checkpoints state plus source positions to durable storage; on failure it restores the state and rewinds the source. End-to-end exactly-once only holds when the sink is idempotent or transactional.
| Piece | Contract you need |
|---|---|
| Source | Can replay from checkpointed offsets or positions |
| Flink state | Durable checkpoints/savepoints restore operator state |
| Sink | Upsert/transactional commit prevents duplicate side effects |
This durable, recoverable state is the core thing a hand-rolled consumer loop lacks.
Performance envelope
Flink characteristics — order-of-magnitude anchors; actual numbers depend on hardware, state size, event shape, sinks, checkpoints, and configuration.
| Dimension | Reality | Why it matters |
|---|---|---|
| Latency | Milliseconds to seconds (true streaming; workload-dependent) | Real-time results, not minutes-late batch |
| Throughput | Can reach millions of events/sec with enough parallelism | Parallelism, state I/O, serialization, and sink speed set the ceiling |
| State | Large managed state; RocksDB common, ForSt/disaggregated state is the 2.x cloud direction | Windows/joins/aggregates survive failure without pinning all state to local disk |
| Semantics | Exactly-once within the pipeline | Checkpoints + source rewind; sink must cooperate |
| Time model | Event-time + watermarks | Correct windows despite late/out-of-order events |
| Ops burden | A cluster + state backend + checkpoints/savepoints to operate | Heavier than a simple consumer or Kafka Streams |
Capabilities in interviews
Windowed aggregations
Compute real-time counts, sums, and percentiles per time interval over a stream.
The canonical streaming job — aggregate an unbounded stream into per-window results:
clicks.keyBy(ad_id)
.window(Tumbling(1 minute)) // event time
.aggregate(count) // → click-counts topic / DBThis is exactly the ad-click aggregator: a firehose of click events in, per-ad per-minute counts out, with event-time windows and watermarks so late clicks still land in the right minute. A batch job would give you the numbers minutes or hours late; Flink gives them continuously.
Choose this variant when
- Real-time metrics / counters per interval
- Live dashboards (engagement, traffic, revenue)
- Ad-click / event aggregation
Stream joins & enrichment
Join two live streams, or enrich a stream with reference data, in real time.
Flink can join streams within a time window or enrich events with slowly-changing reference data held in state:
orders.join(payments).where(order_id).window(5 min) // correlate within a window
events.enrichWith(userProfileState) // add reference dataCorrelating "order placed" with "payment received" within a window, or attaching user/product attributes to raw events, is a core streaming pattern — and one that needs Flink's managed state to buffer one side while waiting for the other.
Choose this variant when
- Correlating two event streams in real time
- Enriching events with reference/profile data
- Building denormalized real-time views
Pattern detection & anomaly alerting
Detect sequences and anomalies across events as they happen, for fraud and monitoring.
Complex-event-processing detects patterns over a stream — a sequence of actions, a threshold breach, an unusual rate — and emits an alert in real time:
detect: login → large transfer → from new device, within 2 min → flag fraudBecause Flink keeps state and reasons over event time, it can recognize multi-event patterns and rate anomalies as they occur, rather than discovering them in a nightly batch. This powers real-time fraud detection, security monitoring, and operational alerting.
Choose this variant when
- Real-time fraud / abuse detection
- Security and operational anomaly alerting
- Multi-event sequence / threshold detection
Live materialized views
Continuously maintain a denormalized, query-ready view from a stream of changes.
Consume a change stream (often Kafka via CDC) and keep a derived, denormalized view always up to date in a serving store:
order events → Flink (aggregate per customer) → customer-summary table (always current)Instead of recomputing expensive aggregates on read, Flink updates the view incrementally as each event arrives, so reads are cheap and fresh. In Flink 2.x, materialized tables make part of this story more declarative in Flink SQL: define the target table/query and let Flink manage refresh jobs, lakehouse integration, and stream/batch execution details. This is streaming's answer to the CQRS read model — the view is maintained by the stream processor, not rebuilt on demand.
Choose this variant when
- Always-fresh denormalized read models
- Incremental aggregates instead of read-time recompute
- CQRS read side fed by an event stream
Operating knobs
Flink vs Kafka Streams vs Spark
Flink for demanding stateful, low-latency streaming at scale, either as DataStream jobs or Flink SQL/materialized tables. Kafka Streams is a library embedded in your app — lighter to operate, great when data is already in Kafka and the state/SQL needs are modest. Spark Structured Streaming suits teams already on Spark and tolerant of micro-batch latency. Match the tool to latency, state, SQL/lakehouse needs, and operational appetite.
Window type
Tumbling (fixed, non-overlapping) for "per minute/hour" metrics; sliding (overlapping) for moving averages; session (gap-based) for user activity bursts. The window choice encodes the question you are answering — pick it from the metric, not by default.
Event time vs processing time + watermarks
Use event time when correctness under late/out-of-order data matters (almost always for analytics), with a watermark strategy that trades latency for completeness — a longer allowed-lateness catches more stragglers but delays emitting the window. Processing time is simpler but wrong when events are delayed.
State backend & checkpointing
RocksDB remains the common, well-understood state backend for large local state, but Flink 2.x makes disaggregated/cloud-native state interview-relevant: ForSt stores state primarily on remote/DFS or object storage and uses async state access to avoid blocking operators. Either way, tune checkpoint interval, state TTL, and recovery targets carefully — unbounded keyed state is the streaming equivalent of an unbounded partition.
Versus the alternatives
Flink vs the alternatives.
| Dimension | Flink | Kafka (alone) | Batch (Spark / warehouse) |
|---|---|---|---|
| Role | Compute continuously on streams (DataStream, SQL, materialized tables) | Move/store the stream | Crunch data after the fact |
| Latency | Milliseconds–seconds | Transport only | Minutes–hours |
| State | Large, checkpointed, exactly-once | Offsets only | N/A (recomputed each run) |
| Best at | Windows, joins, real-time analytics | Durable event backbone | Heavy historical analytics |
| Pairs with | Kafka as source + sink | Flink for processing | Object storage / lakes |
Failure modes & gotchas
If you only need to move events or do a trivial write-through, a Flink cluster is over-engineering — Kafka alone, or a simple consumer, is simpler and cheaper. Flink earns its operational weight specifically for stateful, low-latency computation (windows, joins, patterns); name that requirement before introducing it.
Windowing on when events arrive rather than when they happened produces wrong aggregates whenever events are late or out of order — a click at 11:59 that arrives at 12:01 lands in the wrong minute. Use event time with watermarks for any analytics where correctness matters.
State keyed by something that grows without bound (every session id forever, every user with no expiry) bloats the state backend and slows checkpoints until the job degrades. Use TTLs on state, bounded windows, and session timeouts so state is reclaimed — the streaming analog of bounding a partition.
Flink gives exactly-once within the pipeline via checkpoints + source rewind, but a write to an external database or API is only exactly-once if the sink is idempotent or transactional. Otherwise a recovery replays the last segment and double-writes. Make sinks idempotent (upsert by key) for true end-to-end correctness.
Too aggressive a watermark drops legitimate late events; too lax delays every window emit and grows state. Tune allowed lateness to the real distribution of event delay, and decide explicitly what happens to events later than that (drop, side-output, or late-update).
In production
Alibaba Cloud
Double 11 2020 — four billion records per second
Alibaba Cloud reported that during Double 11 2020, Realtime Compute for Apache Flink reached a peak processing rate of four billion records per second and seven TB per second of data. The same post describes Flink supporting the GMV dashboard plus search/recommendation, ad anti-fraud, logistics tracking, infrastructure monitoring, and attack detection.
The interview lesson is not the exact number; it is the architecture at that scale: a durable event/log layer feeds Flink jobs, Flink performs stateful real-time aggregation and enrichment, and results power live dashboards and operational systems. For a candidate, this is the public proof that Flink is used when the job is not merely moving events, but continuously computing over them.
Netflix
Delta / Keystone — Flink-based CDC enrichment for search
Netflix's 2019 Delta write-up describes CDC events sent to a Keystone Kafka topic; a Delta application built on a Flink-based stream-processing framework consumes those events, enriches them by calling other microservices, and writes updated search documents to Elasticsearch nearly in real time.
The lesson maps cleanly to this page: Flink is useful when a change stream needs managed state, enrichment, and a reliable sink, not just a consumer loop. The case is smaller in public numbers than the common conference folklore, so teach the architecture and avoid memorizing unverifiable scale claims.
Good vs bad answer
Interviewer probe
“An ad platform ingests 500K click events/sec and needs near-real-time counts per ad per minute for billing and dashboards. How do you compute them?”
Weak answer
"Write every click into a database, and run a scheduled query every few minutes that does GROUP BY ad_id over the last interval to get the counts."
Strong answer
"Stream processing with Flink on top of Kafka. Clicks land in a topic partitioned by ad_id; Flink keys by ad_id, runs a tumbling 1-minute event-time window, and writes per-ad-per-minute counts to a serving store and a downstream topic. Watermarks let late clicks land in the right minute, which matters for billing. Flink checkpoints state plus source offsets, then rewinds on failure; I make the sink an idempotent upsert by ad_id + minute so recovery does not double-write. It scales by partition/parallelism to handle 500K/sec. I could implement this as a DataStream job or, if the team is SQL-heavy, a Flink SQL/materialized-table pipeline. A scheduled GROUP BY gives counts late, re-scans huge tables, and mis-buckets delayed events."
Why it wins: Chooses Kafka→Flink with a keyed tumbling window, uses event-time + watermarks for correctness under late events, names the source/checkpoint/sink contract for exactly-once, scales by partition, and contrasts against the batch GROUP BY's real shortcomings.
Interview playbook
When it comes up
- Real-time aggregation / metrics — ad clicks, engagement counts, live dashboards
- Stream joins, enrichment, sessionization, or fraud/anomaly detection
- Maintaining a live materialized view from an event stream
- The interviewer wants answers "as the data arrives," not minutes-late batch
Order of reveal
- 11. Compute on the stream, not after. This needs results as events arrive, so I process the stream with Flink (Kafka as source and sink), not a periodic batch job.
- 22. Window it. I key by the entity and apply a tumbling/sliding/session window — "per ad per minute" is a tumbling window keyed by ad_id.
- 33. Event time + watermarks. I compute over event time with watermarks so late and out-of-order events still land in the correct window.
- 44. Checkpointed source/state/sink contract. Flink checkpoints state and source offsets; replay is safe only when the sink is idempotent or transactional.
- 55. Right-size the tool. Flink for heavy stateful streaming; Kafka Streams if it is light and already in Kafka; batch if minutes-late is fine.
Signature phrases
- “Kafka moves the stream; Flink computes on it.” — The crisp division of labour interviewers want to hear.
- “Window the unbounded stream — tumbling, sliding, or session.” — Shows you know how to make a stream computable.
- “Event time with watermarks, so late events still land correctly.” — The detail that makes streaming results correct.
- “Checkpointed state gives exactly-once within the pipeline.” — Names Flink's core differentiator over a consumer loop.
Likely follow-ups
?“Why event time instead of processing time?”Reveal
Because events arrive late and out of order in the real world — a mobile client buffers events offline, a network hiccup delays a batch — so a click that happened at 11:59 might arrive at 12:01. Processing-time windows would put it in the wrong minute, corrupting the aggregate; this matters enormously when the numbers feed billing. Event-time windows bucket by when the event actually occurred, and watermarks — a moving "no events older than T will arrive" assertion — tell Flink when a window is complete enough to emit while still accommodating stragglers up to an allowed-lateness bound. The trade is latency: a longer lateness allowance catches more late events but delays emitting the result.
?“How does Flink achieve exactly-once?”Reveal
Through checkpointing plus source rewind. Periodically Flink takes a consistent snapshot of all operator state (window contents, aggregates) and the corresponding source offsets, stored durably. On failure, the job restarts from the last checkpoint and the source (Kafka) rewinds to the matching offset, so every event affects the state exactly once — no loss, no double-count — within the pipeline. The caveat is the sink: writing to an external store is only exactly-once if that write is idempotent (upsert by key) or transactional; otherwise recovery replays the last segment and double-writes. So end-to-end exactly-once = Flink checkpoints + an idempotent/transactional sink.
?“When would Kafka Streams be the better choice than Flink?”Reveal
When the processing is lighter and the data is already in Kafka, and you would rather not operate a separate cluster. Kafka Streams is a library you embed in your own service — it scales with your app instances and uses Kafka itself for state and coordination, so there is no extra cluster to run. It is ideal for per-record transforms, simpler aggregations, and joins where the throughput and state are moderate. Flink wins when you need very large state, complex event-time windowing and CEP, the highest throughput, or sophisticated recovery — at the cost of running and tuning a Flink cluster. Rule of thumb: light + Kafka-native → Kafka Streams; heavy stateful streaming at scale → Flink.
Worked example
Setup. An ad platform ingests 500K click events/sec and needs near-real-time counts per ad per minute for billing and dashboards — correct even when events arrive late or out of order, and without losing or double-counting on failure.
The move. Stream processing with Flink on top of Kafka. Clicks land in a Kafka topic partitioned by ad_id; a Flink job reads it, keys by ad_id, and applies a tumbling 1-minute window, aggregating counts and writing per-ad-per-minute results to a serving store and a downstream topic. Kafka moves the stream; Flink computes on it.
Event time + watermarks. I window on event time (when the click happened), not processing time — so a click that occurred at 11:59 but arrives at 12:01 (mobile was offline) still counts in the 11:59 minute. Watermarks decide when a window is complete enough to emit while allowing a bounded lateness for stragglers. This is what makes the numbers correct, which matters because they feed billing.
Exactly-once. Flink checkpoints its window state durably and rewinds the Kafka source to the matching offset on failure, so each event affects the result exactly once within the pipeline. For end-to-end correctness I make the sink idempotent (upsert by ad_id + minute), so a recovery replay can't double-write.
Scale + state. It scales horizontally by partition to absorb 500K/sec. Window state lives in managed state — often RocksDB today, with Flink 2.x's ForSt/disaggregated state as the cloud-native direction for very large state — and is checkpointed on an interval. I bound keyed state (windows close, TTLs on anything long-lived) so state doesn't grow unbounded — the streaming equivalent of a wide partition.
What breaks. A scheduled GROUP BY batch job would give counts minutes late, re-scan huge tables, and mis-bucket late events — wrong for billing. If the processing were trivial I'd consider Kafka Streams to avoid a Flink cluster; windowed aggregation with exactly-once at this scale is squarely Flink's job.
The result. Per-ad-per-minute counts within seconds, correct under late/out-of-order events, exactly-once through checkpoints + idempotent sinks, scaling to 500K events/sec — real-time analytics a batch job can't match.
Cheat sheet
- •Flink = distributed engine for stateful, low-latency computation over unbounded streams.
- •Kafka moves the stream; Flink computes on it. Canonical shape: Kafka → Flink → store/topic.
- •Window the stream to make it computable: tumbling (fixed), sliding (overlapping), session (gap).
- •Event time + watermarks → correct windows despite late/out-of-order events (vs wrong processing-time).
- •Checkpointed state + source rewind → exactly-once within the pipeline (sink must be idempotent for end-to-end).
- •Scales by partition/parallelism; RocksDB is common, while ForSt/disaggregated state is the Flink 2.x cloud-native direction.
- •Bound keyed state with TTLs/sessions — unbounded state is the streaming wide-partition mistake.
- •Kafka Streams (library, lighter) for simple Kafka-native jobs; batch/Spark if minutes-late is fine.
Drills
Why can't you just "sum a stream" — why are windows necessary?Reveal
Because a stream is unbounded — it never ends — so a plain aggregate like a sum or count has no point at which it is "done" and can be emitted. Windows impose finite, computable chunks on the infinite stream: a tumbling window ("every 1 minute") produces one result per interval; sliding windows give moving aggregates; session windows group bursts of activity separated by gaps. The window encodes the actual question — "clicks per ad per minute" is a tumbling 1-minute window keyed by ad id — and gives Flink a boundary at which to compute and emit a result while the stream keeps flowing.
Interviewer: "your per-minute counts are missing clicks that happened near the minute boundary. Why?"Reveal
Almost certainly you are windowing on processing time instead of event time, or your watermark allows too little lateness. Events arrive late and out of order — a click that occurred at 11:59:58 might reach Flink at 12:00:02 — and a processing-time window would file it under 12:00 (or miss the 11:59 window that already closed). Fix it by windowing on event time (the timestamp in the event) with a watermark strategy that allows enough lateness to capture stragglers, deciding explicitly what to do with events later than that (drop, side-output, or late-update). This trades a bit of emit latency for correctness, which is the right call when the counts feed billing.
Does Flink's exactly-once mean your database will never see a duplicate write?Reveal
Not by itself. Flink guarantees exactly-once within the pipeline: checkpoints snapshot operator state and source offsets together, so on recovery the source rewinds and each event affects Flink's state exactly once. But the write to an external sink is only exactly-once if the sink is idempotent or transactional — on recovery, Flink replays the events since the last checkpoint, which would double-write to a naive sink. So for true end-to-end exactly-once you make the sink idempotent (upsert by a key like ad_id+minute) or use a transactional sink that commits atomically with the checkpoint. Flink does its half; the sink has to do the other half.
What it is