Analytics: OLAP, stream and batch

Late data is where many streaming answers become vague. A watermark is a practical statement that event time has progressed, not a promise that no older event can ever appear. The policy should match the business value of the metric. A live operations dashboard may update a recent window for a short allowed-lateness interval and then send very late records to an audit stream. A billing report may keep corrections forever and restate closed periods through batch reconciliation. A fraud signal may ignore very late events for online action, but still persist them for model training and investigation.

Say the policy out loud: "We allow an illustrative two minutes of lateness for live dashboards, emit correction updates while the window is retained, and write later events to a side-output topic for daily reconciliation." That one sentence covers latency, correctness and operational repair. It also prevents the common mistake of treating dropped late events as if they never existed.

Pre-aggregation is a serving decision. If a product surface asks the same expensive question all day, store the answer at the grain the product needs. Query-time aggregation is a flexibility decision. If analysts are exploring new slices, keep raw or lightly modelled facts and pay the scan cost when they ask. Most mature systems use both: raw facts for truth and backfills, modelled tables for common analysis, and small aggregate tables for hot product views.

The trade-off is change. If you pre-aggregate by country and device, a new request for campaign and browser needs a new table or a backfill. If you aggregate at query time, the same question is easy to ask but may be too slow or too expensive for every page view. In interviews, pick the known hot metrics for pre-aggregation and leave exploratory analysis in the warehouse or lakehouse.

A stream job is a long-running state machine. The late data policy decides what happens when event time is messy; checkpoints decide whether the job can survive failure without losing its remembered state. This agrees with the page's exactly-once section: Flink can recover its own state and source positions, but the sink still has to be idempotent or transactional.

Keyed state and windows

Flink's state docs say keyed state is scoped to the key of the current input element and requires a keyed stream. Jobs partition by key so all events for the same user, ad or auction reach the same logical state, while different keys run in parallel. Session windows are gap-based: Flink's window docs describe session assigners as creating windows separated by inactivity gaps, and note that session windows merge when events bridge the gap.

Checkpoints and replay

Flink's checkpoint docs say checkpoints recover state and corresponding stream positions, giving the application the same semantics as failure-free execution. The mechanism comes from asynchronous barrier snapshotting: Carbone, Fóra, Ewen, Haridi and Tzoumas's 2015 ABS paper describes checkpoint barriers moving through a distributed dataflow so operators capture a consistent snapshot without stopping the whole job. During aligned checkpoints, Flink's large-state tuning docs say channels that already received a barrier are blocked until the remaining channels catch up. Flink's checkpointing-under-backpressure docs say unaligned checkpoints include in-flight buffered data so barriers can overtake buffers; that can reduce checkpoint time under backpressure, but adds state-storage I/O.

After failure, the job restores from the last completed checkpoint and replays input from the recorded source position. That is why the source must be replayable: Kafka's consumer-position design says a consumer position is the offset of the next message in each partition and can be checkpointed; it also says consumers can rewind to an old offset and re-consume data.

Savepoints and state backends

Checkpoints are the automatic failure-recovery log. Flink's checkpoints-versus-savepoints docs say checkpoints are created, owned and released by Flink for unexpected failures and stored in backend-specific native format. Savepoints are the operator-controlled handoff: the same docs say savepoints are created, owned and deleted by the user, use a backend-independent canonical format by default, and support planned upgrades and rescaling. Flink's savepoint docs also describe savepoints as consistent images used to stop and resume, fork or update jobs, with a native format available when you choose that trade-off.

State backend choice sets the cost shape. Flink's state backend docs say HashMap state keeps objects on the Java heap, while EmbeddedRocksDB stores serialized bytes on local RocksDB and, among those two, supports incremental checkpoints. Shorter checkpoint intervals reduce replay after failure, but add more checkpoint I/O. Longer intervals reduce steady overhead, but increase recovery time because more input must replay.