What Is Stream Processing?

Stream processing is a way of handling data continuously, computing results on each event as it arrives rather than collecting events and processing them later in a scheduled run. A stream is an unbounded sequence - it has a beginning but no defined end - so a stream processor is designed to produce answers over a moving slice of that sequence instead of over a complete dataset.

The practical consequence is latency. A nightly job that totals yesterday's payments gives an answer the next morning; a stream processor totalling the same payments emits an updated figure within milliseconds of each transaction. That difference is what makes fraud detection, live dashboards, real-time personalization, and operational alerting possible at all.

How Stream Processing Works

Nearly every stream processing system is assembled from the same four parts:

Source: Events enter from an append-only log, a message broker, a change data capture feed, or a set of sensors. The source is durable and replayable in most serious designs, so that reprocessing is possible after a bug or an outage.
Operators: The processing logic itself - filtering, mapping, enriching, joining two streams, and aggregating. Operators are chained into a topology, and each one consumes a stream and produces another.
State: Any operator that aggregates, joins, or deduplicates has to remember something between events. That working memory is the state store, and it is what separates real stream processing from simple message consumption.
Sink: Results are written out to a database, a cache, a dashboard, an alerting system, or another stream for a downstream stage to consume.

Because the input never ends, the interesting question is not what to compute but over which events to compute it. That question is answered by windowing, and it is the central concept in the field.

Stream Processing vs. Batch Processing

Batch processing collects data over a period and processes it in scheduled chunks - hourly, nightly, or on demand. Stream processing handles each event on arrival. The distinction is often presented as a rivalry, but the two models answer different questions and most mature platforms run both.

Data boundaries: A batch job operates on a bounded, complete dataset. A stream processor operates on an unbounded one and must decide for itself where to draw boundaries.
Latency: Batch latency is measured in minutes to hours and is governed by the schedule. Stream latency is measured in milliseconds to seconds and is governed by the pipeline itself. See latency vs. throughput for why optimizing one often costs the other.
Correctness: A batch job sees all of its input before it produces an answer, so its result is final. A stream processor produces an answer from what has arrived so far, which may need revising when late data shows up.
Cost and simplicity: Batch is cheaper per record and far easier to reason about, test, and re-run. Streaming carries permanent operational overhead because the pipeline is always live.
Reprocessing: Re-running a batch job is routine. Reprocessing a stream requires a replayable log and careful handling of already-emitted results.

The reasonable default is batch. Streaming earns its complexity only where the value of an answer decays quickly - a fraudulent transaction that has already settled, a stockout that has already lost the sale, an outage that has already reached customers. Where both are needed, the Lambda and Kappa architectures described in data pipeline are the two established ways to combine them.

Windowing in Stream Processing

A window is a bounded slice of an unbounded stream, and windowing is the mechanism that makes aggregation possible at all. "Count the orders" is meaningless against a stream that never ends; "count the orders in each one-minute interval" is a computable question. Windows are what turn a continuous feed into a series of finite datasets that operators can actually aggregate.

Three window types cover the overwhelming majority of real workloads.

Tumbling Windows

A tumbling window has a fixed size and does not overlap. The stream is cut into adjacent, equal intervals - 09:00:00 to 09:01:00, then 09:01:00 to 09:02:00 - and every event belongs to exactly one window. Because the intervals are disjoint, each event is counted once, which makes tumbling windows the natural choice for periodic reporting: requests per minute, revenue per hour, error counts per five-minute bucket.

Sliding and Hopping Windows

A sliding window also has a fixed size but advances by a smaller step, so consecutive windows overlap and a single event can fall into several of them. A five-minute window advancing every thirty seconds reports on the last five minutes, refreshed twice a minute. Some systems reserve "hopping" for a fixed advance interval and "sliding" for a window that is recomputed on every event; the terms are frequently used interchangeably.

Overlap is the point. Alerting thresholds - more than 100 failed logins in any five-minute period - need overlapping windows, because a tumbling window would miss a burst that straddles two interval boundaries. The cost is that each event is processed multiple times and more state is retained.

Session Windows

A session window has no fixed size. It groups events separated by less than a defined gap, and closes once the stream goes quiet for longer than that gap. A thirty-minute inactivity timeout will treat a burst of user clicks as one session and a burst two hours later as another, regardless of how long either lasted.

Session windows are the right model for anything driven by human activity - clickstream analysis, in-app behavior, support conversations - because the meaningful unit is the period of engagement rather than the clock. They are also the most demanding to implement, since the system must hold each session open indefinitely until the gap elapses.

Event Time, Processing Time, and Late Data

Windowing raises an awkward question: which clock defines the boundary?

Event time is when the event actually occurred, recorded at the source. It gives correct, reproducible results - replaying the same data yields the same windows - but requires waiting for stragglers.
Processing time is when the event reached the processor. It is trivial to implement and has the lowest latency, but results depend on network conditions and are not reproducible.

Events routinely arrive out of order: a mobile client buffers while offline, a partition lags, a retry succeeds minutes later. A stream processor working in event time therefore uses a watermark - a moving assertion that no events older than a given timestamp are still expected. When the watermark passes the end of a window, that window is closed and its result emitted.

Anything arriving afterwards is late data, and there are three options: drop it, hold windows open longer and accept the added latency, or emit a corrected result and require downstream consumers to handle updates. This trade-off between completeness and latency has no universal answer, and choosing badly is one of the more common causes of quietly wrong numbers in streaming systems.

Stateful Stream Processing and Delivery Guarantees

Operators that merely filter or reshape events are stateless - each event is handled independently, and a crashed worker can restart with no memory of what came before. Windowed aggregations, stream joins, and deduplication are stateful: they accumulate a running result that must survive restarts, scale with the workload, and be recoverable.

State is what makes delivery semantics matter. Three levels are commonly distinguished:

At-most-once: Events may be lost but never processed twice. Acceptable for sampled metrics, unacceptable for money.
At-least-once: No event is lost, but retries may deliver duplicates. This is the common default, and it works well when operations are idempotent.
Exactly-once: Each event affects the result once, even across failures. Achieved by coordinating state updates with offset commits, and always at some cost in throughput and complexity.

The distinction is sharper than it first appears. Duplicate delivery is harmless when the sink is a key overwrite, and corrupting when the sink is a counter increment inside a window.

Stream Processing Use Cases

Fraud and anomaly detection: Scoring transactions against sliding windows of recent account behavior, where a decision after settlement is worthless.
Real-time analytics and dashboards: Continuously updated counts, rates, and percentiles over tumbling windows.
Monitoring and alerting: Detecting error-rate spikes or threshold breaches within a rolling interval.
Personalization and recommendations: Adjusting what a user sees based on their behavior within the current session window.
IoT and telemetry: Aggregating high-frequency sensor readings into windowed summaries before storing them in a time series database.
Change data capture: Propagating database changes to caches, search indexes, and read models as they happen.

Most of these sit inside a broader event-driven architecture, where services communicate by producing and reacting to events rather than calling one another directly.

Stream Processing With Valkey, Redis, and Redisson

Redis Streams provides the durable, ordered, replayable log that a streaming pipeline is built on. Entries are appended with automatically generated IDs, consumer groups let multiple workers share a stream while tracking their own progress, and unacknowledged messages stay pending so a crashed worker's work can be claimed and retried. Redisson exposes this through the RStream interface:

RStream<String, String> stream = redisson.getStream("clicks:events");

Map<String, String> event = new HashMap<>();
event.put("userId", "8842");
event.put("page", "/pricing");
event.put("eventTime", String.valueOf(System.currentTimeMillis()));

stream.add(StreamAddArgs.entries(event));

// A consumer group reads new events and acknowledges each one
stream.createGroup(StreamCreateGroupArgs.name("windowing").makeStream());

Map<StreamMessageId, Map<String, String>> messages =
        stream.readGroup("windowing", "worker-1", StreamReadGroupArgs.neverDelivered());

for (Map.Entry<StreamMessageId, Map<String, String>> entry : messages.entrySet()) {
    accumulate(entry.getValue());
    stream.ack("windowing", entry.getKey());
}

Implementing Windows With Sorted Sets

Redis Streams transports events; it does not window them. The windowing itself is built on top, and a sorted set scored by event-time timestamp is the standard technique. Range queries by score then become window queries, and Redisson exposes them through RScoredSortedSet.

A tumbling window closes an interval, reads its contents, and discards them:

RScoredSortedSet<String> events = redisson.getScoredSortedSet("clicks:/pricing");

// Score every event by its event-time timestamp, not arrival time
events.add(eventTimeMillis, eventId);

// Close the one-minute window [windowStart, windowStart + 60s)
long windowStart = 1753862400000L;          // 09:00:00, aligned to the minute
long windowEnd   = windowStart + 60_000;

int count = events.count(windowStart, true, windowEnd, false);
emit(windowStart, count);

// Discard only after the grace period for late data has elapsed;
// deleting at emit time makes any correction impossible
events.removeRangeByScore(windowStart, true, windowEnd, false);

A sliding window keeps the same structure but never discards data that a later, overlapping window still needs. It queries a fixed interval ending at the present moment and trims only what has fallen out of the widest window in use:

RScoredSortedSet<String> failures = redisson.getScoredSortedSet("logins:failed:8842");
failures.add(eventTimeMillis, attemptId);

long now = System.currentTimeMillis();
long fiveMinutesAgo = now - 300_000;

// Recompute the trailing five minutes on every evaluation
int recent = failures.count(fiveMinutesAgo, true, now, true);
if (recent > 100) {
    raiseAlert(recent);
}

// Retain only what an open window could still need
failures.removeRangeByScore(0, true, fiveMinutesAgo, false);

Session windows follow from the same primitive: store the last-seen timestamp per user, and treat a gap larger than the timeout as a session boundary. For fixed-capacity windows measured in events rather than time, a ring buffer (RRingBuffer) bounds retention automatically, while RTimeSeries offers timestamp-keyed storage with per-entry expiration and its own range() query. Approximate aggregations that would otherwise dominate memory - unique visitors per window, heavy hitters, latency percentiles - are served by HyperLogLog, RTopK, and RTDigest.

For pipelines that need stronger guarantees than raw streams provide, Redisson PRO adds Reliable PubSub and Reliable Queue, which supply dead letter queues, visibility timeouts, negative acknowledgment, and seek-and-replay without hand-written plumbing.

When a Dedicated Stream Processor Is the Better Fit

It is worth being precise about the boundary. Valkey and Redis, with Redisson, give you a durable event log, consumer groups, and fast windowed state - which is genuinely sufficient for counters, thresholds, rate limits, leaderboards, and rolling aggregates at low latency. What they do not give you is a stream processing engine: there are no built-in windowing operators, no watermarks, no event-time semantics, and no framework-managed exactly-once state. Those are your responsibility to implement.

For multi-stage topologies, stream-to-stream joins, event-time correctness with automatic late-data handling, or checkpointed exactly-once processing across large state, a purpose-built processor such as Apache Flink or Kafka Streams is the right tool. For a comparison of the underlying logs, see Redis Streams vs Kafka and Redis vs. Kafka: When to Use Each.

Stream Processing: Frequently Asked Questions

What is the difference between stream processing and batch processing?

Batch processing operates on a bounded dataset collected over a period and produces a final answer on a schedule. Stream processing operates on an unbounded sequence and produces continuously updated answers as events arrive. Batch is cheaper and easier to reason about; streaming is chosen when the value of an answer decays within seconds or minutes.

What is windowing in stream processing?

Windowing is the practice of dividing an unbounded stream into bounded slices so that aggregation becomes possible. The three common types are tumbling windows (fixed size, no overlap), sliding or hopping windows (fixed size, overlapping), and session windows (variable size, bounded by a period of inactivity).

What is the difference between a tumbling window and a sliding window?

Tumbling windows do not overlap, so every event belongs to exactly one window and is counted once - suitable for periodic reporting. Sliding windows advance by less than their own width, so they overlap and an event can appear in several windows - suitable for alerting, where a burst spanning a boundary must not be missed.

What is the difference between event time and processing time?

Event time is when the event happened at its source; processing time is when it reached the processor. Event time yields correct, reproducible windows but requires watermarks and a tolerance for late arrivals. Processing time is simpler and faster but produces results that vary with network conditions and cannot be reproduced by replaying the data.

Can you do stream processing with Redis?

Yes, for a substantial class of workloads. Redis Streams supplies the durable, replayable log and consumer groups, and sorted sets, time series, and ring buffers supply the windowed state. What Redis does not provide is a full stream processing engine with windowing operators, watermarks, and framework-managed exactly-once state - those must be implemented in the application, or delegated to a system such as Flink or Kafka Streams.

Is stream processing the same as event-driven architecture?

No. Event-driven architecture is a communication style in which services react to events instead of calling one another. Stream processing is a computation model for deriving results from a continuous flow of events. Stream processing usually runs inside an event-driven system, but a system can be event-driven while doing no windowed computation at all.