What Is a Data Pipeline?

A data pipeline is a set of automated steps that move data from one or more sources to a destination, processing and transforming it along the way. It is the backbone of analytics, machine learning, and real-time applications, ensuring that raw data arrives where it is needed in a clean, usable, and timely form.

In modern systems, data rarely lives in a single place. It is generated by user actions, sensors, payment systems, application logs, and third-party services, and it must be collected, reshaped, and delivered to databases, warehouses, dashboards, and downstream services. A data pipeline is what connects those endpoints reliably and repeatably, so that engineers do not have to move data by hand every time it is needed.

How a Data Pipeline Works

Although architectures vary widely, almost every data pipeline is built from the same logical stages:

• Source (ingestion): Data is captured from origin systems such as application databases, message queues, APIs, log files, or IoT devices. Capture can be a one-off pull, a scheduled extract, or a continuous feed.
• Transformation: Raw data is cleaned, validated, enriched, joined, aggregated, or reformatted. This is where business rules are applied and where inconsistent data is normalized into a usable shape.
• Destination (load): The processed data lands in a target system - a data warehouse, a distributed database, an analytics store, a cache, or another service that consumes it.
• Orchestration and monitoring: A scheduler or event trigger coordinates the steps, handles retries and failures, and tracks data quality and throughput so problems surface before they corrupt downstream results.

The defining characteristic of a pipeline is automation. Once defined, it runs on its own - on a schedule, on a trigger, or continuously - without manual intervention for each batch of data.

Data Pipeline Architecture

Data pipeline architecture describes how these stages are arranged into a working system. Most designs separate into four layers: an ingestion layer that captures data from sources, a processing layer that transforms it, a storage layer that holds the results, and an orchestration and monitoring layer that schedules work, retries failures, and tracks health.

Two reference architectures dominate in practice:

• Lambda architecture runs a batch layer and a streaming ("speed") layer in parallel. The batch layer reprocesses complete datasets for accuracy, the speed layer serves fresh results with low latency, and a serving layer merges the two. It is powerful but means maintaining two code paths.
• Kappa architecture drops the separate batch layer and treats everything as a stream, replaying the event log whenever historical reprocessing is needed. It is simpler to operate when a durable, replayable log - such as Redis Streams - sits at the core.

The right architecture depends on latency requirements, data volume, and how much operational complexity the team can absorb. Most organizations start simple and add a streaming path only when freshness genuinely demands it.

Batch vs. Streaming Data Pipelines

The single biggest design decision in any pipeline is when data moves through it. This splits data pipelines into two main types.

Batch Data Pipelines

A batch pipeline collects data over a window of time and processes it in scheduled chunks - for example, every hour or once nightly. Batch processing is ideal when latency is not critical and when throughput and cost efficiency matter more than freshness, such as for daily reporting, billing runs, or training data preparation.

Batch workloads are frequently distributed across many machines so that large datasets can be processed in parallel. The classic model for this is MapReduce, which splits a job into independent "map" tasks and then combines their output in a "reduce" step. Even today, most batch frameworks borrow this divide-and-aggregate pattern.

Streaming Data Pipelines

A streaming pipeline processes each event continuously, the moment it arrives, rather than waiting for a batch to accumulate. This delivers very low latency and powers use cases such as fraud detection, live dashboards, real-time personalization, and alerting.

Streaming pipelines are typically built on an append-only log or message broker that buffers events between producers and consumers. Redis Streams is a well-known example: it stores an ordered, append-only sequence of events and lets multiple consumer groups read from it independently. Lighter-weight messaging such as Redis Pub/Sub can also feed a streaming pipeline when at-most-once delivery is acceptable.

Choosing Between Them

Many real systems use both. A common hybrid keeps a fast streaming path for fresh results and a slower batch path for accurate, complete reprocessing. The right choice comes down to a simple trade-off: how fresh the data must be, versus how much throughput, cost efficiency, and simplicity you are willing to give up to get it.

ETL vs. ELT Pipelines

ETL - Extract, Transform, Load - is the traditional pattern. Data is extracted from sources, transformed in a dedicated processing layer, and only then loaded into the destination in its final shape. ETL is a strong fit when the target store is expensive to compute in, when data must be cleansed before it lands, or when strict schema and compliance rules apply.

ELT - Extract, Load, Transform - reverses the last two steps. Raw data is loaded into the destination first, and transformation happens inside the destination itself. This pattern became dominant with cloud data warehouses and lakehouses, whose elastic compute makes it cheap to transform large volumes of raw data on demand and to re-transform it later as requirements change.

It is worth being precise about terminology: ETL is a type of data pipeline, not a synonym for one. Every ETL or ELT job is a data pipeline, but plenty of pipelines - event routing between microservices, real-time feature delivery for machine learning, log shipping - do far more than the classic extract-transform-load cycle.

Data Pipeline vs. ETL

Because the two terms are often used interchangeably, here is the distinction at a glance:

In short, every ETL pipeline is a data pipeline, but not every data pipeline is ETL.

Common Data Pipeline Use Cases

Some of the most common examples of data pipelines in production include:

• Real-time analytics and dashboards: Streaming pipelines feed metrics into live views so teams act on current data, not yesterday's.
• Change Data Capture: Change Data Capture (CDC) watches a source database's transaction log and emits each insert, update, and delete as an event. It is one of the most reliable ways to keep caches, search indexes, and warehouses in sync with a system of record.
• Event-driven microservices: Pipelines move events between loosely coupled services, letting each one react to changes without tight, synchronous dependencies.
• Machine learning feature pipelines: Raw data is transformed into model-ready features, both in batch for training and in real time for inference.
• IoT and observability: High-volume sensor readings and application logs are ingested, aggregated, and routed into a time series database for monitoring and analysis.

Building Data Pipelines With Redis Streams and Redisson

For low-latency, high-throughput streaming pipelines on the JVM, Redis is a popular choice because it can act as the in-memory log that connects producers and consumers. Redisson is a Valkey and Redis Java client that exposes these capabilities through familiar Java types instead of raw commands.

Redis Streams With RStream

Redis Streams fits the pipeline model directly. Producers append events to an append-only stream through the RStream API, and one or more consumer groups read those events in parallel. Each message is assigned to exactly one consumer within a group and must be acknowledged once processed, which gives at-least-once delivery and lets you scale out workers without processing the same event twice.

A minimal producer that appends events to a stream:

Config config = new Config();
config.useSingleServer().setAddress("redis://127.0.0.1:6379");
RedissonClient redisson = Redisson.create(config);

RStream<String, String> stream = redisson.getStream("orders:events");

Map<String, String> event = new HashMap<>();
event.put("orderId", "10324");
event.put("status", "PLACED");

StreamMessageId id = stream.add(StreamAddArgs.entries(event));

A consumer group reads new events, processes them, and acknowledges each one:

RStream<String, String> stream = redisson.getStream("orders:events");

// Create the consumer group once (makeStream() creates the stream if absent)
stream.createGroup(StreamCreateGroupArgs.name("billing").makeStream());

// Read messages never yet delivered to this group, then acknowledge them
Map<StreamMessageId, Map<String, String>> messages =
        stream.readGroup("billing", "worker-1", StreamReadGroupArgs.neverDelivered());

for (Map.Entry<StreamMessageId, Map<String, String>> entry : messages.entrySet()) {
    process(entry.getValue());          // transform / load step of the pipeline
    stream.ack("billing", entry.getKey());
}

Because the stream is durable and consumer groups track what each worker has acknowledged, a crashed worker's unfinished messages stay pending and can be claimed and retried by another worker. As volume grows, the data can be partitioned across nodes (see Sharding) so the pipeline scales horizontally.

Stronger Delivery Guarantees With Reliable Queue and Reliable PubSub

Raw streams are a solid foundation, but production pipelines usually need more: automatic redelivery of failed work, somewhere to park messages that can never be processed, and durability that survives a node going down. Redisson PRO adds two higher-level primitives built for exactly this, both operating directly on Valkey or Redis with no separate message broker to run.

The Reliable Queue (RReliableQueue) is a FIFO work queue with built-in acknowledgments, a configurable visibility timeout that re-queues a message if a worker crashes before acknowledging it, a delivery limit, and a Dead Letter Queue (DLQ) for messages that exhaust their retries. It also supports priorities, delayed delivery, deduplication, and synchronous replication so an acknowledged message is not lost during failover. That makes it a natural fit for the processing stage of a pipeline, where each unit of work must be handled once and failures must be isolated rather than dropped:

RReliableQueue<OrderEvent> queue = redisson.getReliableQueue("orders:processing");

// Messages that exhaust their retries are routed to a dead-letter queue
queue.setConfig(QueueConfig.defaults()
        .deliveryLimit(5)
        .visibilityTimeout(Duration.ofSeconds(60))
        .deadLetterQueueName("orders:dlq"));

// Producer
queue.add(QueueAddArgs.messages(MessageArgs.payload(event)));

// Consumer: poll, process, then acknowledge - or negatively acknowledge to retry
Message<OrderEvent> msg = queue.poll(QueuePollArgs.defaults()
        .acknowledgeMode(AcknowledgeMode.MANUAL));
try {
    process(msg.getPayload());
    queue.acknowledge(QueueAckArgs.ids(msg.getId()));
} catch (Exception e) {
    queue.negativeAcknowledge(QueueNegativeAckArgs.failed(msg.getId())
            .delay(Duration.ofSeconds(30)));
}

When the same data must feed several independent pipelines at once, Reliable PubSub (RReliablePubSubTopic) is the better fit. It follows a topic → subscription → consumer model in which each subscription keeps its own offset, so one published stream can fan out to multiple consumers that each receive every message and progress at their own pace. It adds FIFO ordering, message grouping (events sharing a groupId are processed in order by the same consumer), and seek-and-replay for reprocessing historical data - all valuable when rebuilding a downstream store or running a new pipeline over past events:

RReliablePubSubTopic<OrderEvent> topic = redisson.getReliablePubSubTopic("orders:events");

// Each pipeline gets its own subscription, and therefore its own offset
Subscription<OrderEvent> analytics = topic.createSubscription(
        SubscriptionConfig.name("analytics").position(Position.earliest()));

// Publish once; every subscription receives it
topic.publish(PublishArgs.messages(MessageArgs.payload(event).groupId("customer-123")));

// A consumer on this subscription pulls messages, then acknowledges them
PullConsumer<OrderEvent> consumer = analytics.createPullConsumer();
Message<OrderEvent> msg = consumer.pull();
process(msg.getPayload());
consumer.acknowledge(MessageAckArgs.ids(msg.getId()));

// Later, replay from the beginning to rebuild a downstream view
analytics.seek(Position.earliest());

Together these keep Change Data Capture feeds, event streams, and work queues durable, acknowledged, and replayable on a single platform - with the Reliable Queue providing exactly-once processing once a message is acknowledged.