Amazon Kinesis vs Valkey & Redis Based Reliable PubSub and Queue

Amazon Kinesis Data Streams is a managed, sharded log. Producers write records with a partition key. Each shard keeps its records in order for 24 hours by default, or up to 365 days if you pay for longer retention, and every consumer application reads at its own position and saves its progress as a checkpoint. That design suits high-volume ingestion and long replay, and it plugs straight into Amazon Data Firehose, Amazon Managed Service for Apache Flink, AWS Lambda and Amazon Redshift.

Redisson PRO's Reliable PubSub and Reliable Queue work at a different level. They are messaging objects stored in the Valkey or Redis you may already run on Amazon ElastiCache or MemoryDB, and they keep state for every message. A subscription knows which messages each consumer holds, retries a failed one after a delay you choose, and moves one that keeps failing to a dead-letter topic. A Kinesis stream keeps no delivery state per record, which is why Kinesis consumers that cannot lose records carry their own retry and dead-letter code.

So the decision depends mostly on the job your stream does. If Kinesis is the front door of an analytics pipeline, stay on it. If it carries business events between Java services at moderate volume, and your team keeps writing glue to retry failed records, this comparison is for you.

Redisson is our product. The Kinesis figures below come from the AWS documentation, API reference and pricing pages as they stood on 5 November 2026, and the Kinesis Client Library (KCL) behaviour was checked against KCL 3.5.4, released on 30 October 2026. Every Java sample compiles against Redisson 4.8.0, KCL 3.5.4 and the AWS SDK for Java 2.x.

A Stream and a Message Bus Do Different Jobs

The word "streaming" covers three jobs, and our guide to Kafka alternatives separates them in a way that applies to Kinesis unchanged:

• A durable, replayable event log for clickstreams, application logs, IoT telemetry or change data capture, read by many consumers over a long retention window. Kinesis was built for this.
• Work distribution, where each record is a job that exactly one worker should finish, with retries when it fails. Kinesis divides it by shard across a pool of KCL workers, or by partition key within a shard when Lambda's ParallelizationFactor is raised.
• Broadcasting events between services. An order is placed, and billing, shipping and notifications each need their own copy with their own retries. With Kinesis, each service runs its own consumer application.

Reliable PubSub handles the broadcast job and Reliable Queue the work-distribution job (our glossary covers pub/sub and message queues if you want the background). Neither replaces Kinesis as a high-volume log, because their data lives in memory. The rest of this page compares them on the other two jobs, where what happens to one failed message matters more than raw throughput.

Why Java Teams on Kinesis Are Taking Another Look

Three changes in the past year make this a reasonable moment to re-check the choice.

The first is that the Java client stack was retired under existing consumers. KCL 1.x and the Kinesis Producer Library (KPL) 0.x reached end of support on 30 January 2026, a month after the AWS SDK for Java 1.x (31 December 2025). Leaving KCL 1.x means rewriting code: AWS's migration guide says "You must migrate the record processor, record processor factory, and worker classes", because IRecordProcessor became ShardRecordProcessor, a Scheduler replaced the Worker, and the library no longer creates AWS clients for you. KCL 3.x also assigns shards to workers by CPU utilization and throughput by default, instead of by lease count. KCL 3.0 to 3.4 keep three DynamoDB tables per application (the lease table, a worker metrics table and a coordinator state table); KCL 3.5, which AWS recommends for teams leaving 1.x, keeps everything in the lease table by default for those migrations. A team that has to rewrite its consumers anyway is well placed to ask whether the transport should change too.

The second is pricing. On-demand Advantage, launched on 4 November 2025, removes the per-stream hourly fee and cuts per-GB rates by about 60 percent, in exchange for an account-level minimum of 25 MB/s ingested and 25 MB/s retrieved across all on-demand streams in a Region. AWS's own example puts that floor at $3,037.50 a month in US East. That suits large accounts. A team whose service events add up to a few hundred kilobytes per second pays per stream instead: one provisioned shard at $0.015 an hour ($10.80 for a 30-day month) plus PUT units, or On-demand Standard's $0.04 per stream-hour ($28.80) plus per-GB charges.

The third is that ElastiCache became durable. Since 2 June 2026, ElastiCache for Valkey can store writes in a Multi-AZ transactional log, the same kind of log MemoryDB uses. In synchronous mode, data "is persisted across at least two Availability Zones (AZs) in a Multi-AZ transactional log before responding to the client". Since 27 October 2026 it can also be switched on for a running cluster. That answers the old objection that a failover can drop messages kept in a cache, with a mechanism AWS runs for you. Its requirements and price are in the durability section.

Amazon Kinesis vs Reliable PubSub and Reliable Queue at a Glance

How the Concepts Map

Streams and Partition Keys Become Topics

A Kinesis stream becomes a Reliable PubSub topic. The partition key has no direct counterpart: in Kinesis it chooses the shard, and the shard is both the unit of ordering and the unit of throughput. A topic is ordered as a whole and has no shards to plan, so the producer gets simpler. Here is the Kinesis side with the AWS SDK for Java 2.x:

KinesisClient kinesis = KinesisClient.create();

// the partition key picks the shard, and the shard is the ordering scope
kinesis.putRecord(PutRecordRequest.builder()
        .streamName("orders")
        .partitionKey(event.customerId())
        .data(SdkBytes.fromUtf8String(json))
        .build());

And the same event published to a topic:

RReliablePubSubTopic<OrderEvent> topic =
        redisson.getReliablePubSubTopic("orders");

// a typed payload: the codec serializes it, so there is no SdkBytes step
topic.publish(PublishArgs.messages(MessageArgs.payload(event)));

The payload is a Java object written by whichever Redisson codec you configure, and headers can carry metadata of any serializable type.

Consumer Applications Become Subscriptions

Each KCL application, identified by an application name that by default is also the name of its lease table, reads the stream independently, and so does each Lambda event source mapping. In Reliable PubSub, each consuming service gets a subscription with its own offset, delivery limit, visibility timeout and dead-letter topic. The Kinesis starting positions map onto subscription positions:

Rewinding works differently. KCL uses its configured initial position "only during initial lease creation", so once a shard has a checkpoint the setting does nothing. AWS's guidance for making an existing KCL consumer reread data (written for the DynamoDB Streams adapter, which runs on KCL) is to stop the workers and delete the lease table, or to start the application under a new name. Any subscription can be moved with seek at any time, and one created with retainAfterAck() can also be rewound over messages it has already processed, as the replay section shows; the option is set when the subscription is created.

Workers and Shard Leases Become Consumers

KCL spreads shard leases over the workers of an application and runs one record processor per shard, so parallelism stops at the shard count: a ninth worker on an eight-shard stream sits idle. Lambda can run up to 10 concurrent batches per shard with ParallelizationFactor and still keeps order per partition key. A subscription hands its messages to however many consumers you start, pull or push, and redelivers anything a consumer holds past its visibility timeout. That is the competing-consumers model, and its cost to ordering is covered below.

A Stream Used as a Job Queue Becomes a Reliable Queue

Some streams are work queues in disguise: each record is a job, such as resizing an image or sending an email, that one worker should complete. Kinesis divides that work by shard, or by partition key within a shard under Lambda, and a slow job holds up the records queued behind it. Reliable Queue is built for this case. Any worker takes any message, a crashed worker's job reappears after the visibility timeout, and each message can carry a priority and a delay. Amazon SQS is the managed answer for the same shape, and our SQS comparison covers that choice.

RReliableQueue<ResizeJob> jobs = redisson.getReliableQueue("image-resize");

jobs.setConfig(QueueConfig.defaults()
        .deliveryLimit(3)
        .visibility(Duration.ofMinutes(5))
        .deadLetterQueueName("image-resize-dlq"));

// any worker takes any job: no shard decides who gets what
List<Message<ResizeJob>> batch = jobs.pollMany(QueuePollArgs.defaults()
        .count(10)
        .timeout(Duration.ofSeconds(20)));

for (Message<ResizeJob> job : batch) {
    try {
        resize(job.getPayload());
        jobs.acknowledge(QueueAckArgs.ids(job.getId()));
    } catch (RuntimeException e) {
        // only this job is retried; the rest of the batch carries on
        jobs.negativeAcknowledge(QueueNegativeAckArgs
                .failed(job.getId())
                .delay(Duration.ofSeconds(30)));
    }
}

The visibility timeout has to cover the whole polled batch, because each message's timeout starts when it is polled. When one event has to reach several queues, each with its own consumers and retry settings, Reliable Fanout publishes it to a set of Reliable Queues with an optional filter per queue. Reliable Fanout use cases shows the patterns.

Poison Records, Retries and Dead Letters

Failure handling is where a log and a message bus differ most. A Kinesis consumer's progress is one number per shard, the sequence number of its last checkpoint, and everything before that number counts as done. Nothing in the stream can say that one record failed and should be tried again later while the rest move on. What happens instead depends on the consumer.

KCL Skips the Batch

If your processRecords method throws, KCL does not retry. AWS's troubleshooting guide says any exception "is absorbed by the KCL" and that "the KCL does not resend the batch", which "effectively results in consumer applications observing skipped records". KCL 3.5.4 still behaves this way: it logs "Skipping over the following data records" and, new in that release, adds the records to a RecordsFailedProcessing metric. Because checkpoint() records progress "at the last data record that was delivered to the record processor", the next successful batch checkpoints straight past the failed one. A KCL consumer that must not lose records therefore ends up with code like this:

@Override
public void processRecords(ProcessRecordsInput input) {
    if (stopped) {
        // a failure earlier on this lease: never checkpoint past it
        return;
    }
    try {
        for (KinesisClientRecord record : input.records()) {
            try {
                handle(record);
            } catch (RuntimeException e) {
                // Kinesis has nowhere to keep a failed record, so park it
                // in a queue we run before the checkpoint moves past it
                retries.send(record, e);
            }
        }
        input.checkpointer().checkpoint();
    } catch (Exception e) {
        // throwing would make KCL skip the batch and the next checkpoint
        // pass it, so stop; a restart resumes from the last checkpoint
        stopped = true;
        log.error("record processor stopped", e);
    }
}

The retry queue in that sample is usually SQS, so the team now runs a second messaging system to hold the failures of the first. The processor also has to stop rather than throw when parking or checkpointing fails, because a throw makes KCL skip the batch anyway, and then it needs a restart to resume from its last checkpoint. And the checkpoint covers the whole batch, so a crash between processing and checkpointing replays records that already succeeded. KCL delivers at least once, and for those duplicates AWS's advice is: "If the destination of the final data can handle duplicates well, we recommend relying on the final destination to achieve idempotent processing."

Lambda Blocks the Shard

Lambda's event source mapping takes the opposite approach. By default it "reprocesses the entire batch until the function succeeds, or the items in the batch expire", and in the meantime it "pauses processing for the affected shard". One malformed record can hold up everything behind it in that shard until the record ages out of the stream. Several settings soften this:

• BisectBatchOnFunctionError splits a failing batch in two, repeatedly, to isolate the bad record.
• MaximumRetryAttempts and MaximumRecordAgeInSeconds cap the retries. Both default to -1, which means no limit.
• ReportBatchItemFailures lets the function return the sequence numbers that failed. Lambda then retries from the lowest of them onward, including any later records that had succeeded.
• An on-failure destination receives what Lambda discards. For SQS and SNS destinations, "The actual records aren't included", only the shard and the sequence number range, so something has to fetch the records from the stream before they expire. S3 destinations also receive the payload.

Reliable PubSub Retries One Message

In Reliable PubSub, failure belongs to one message in one subscription. A consumer acknowledges what succeeded, negatively acknowledges what failed, and the rest of the subscription keeps moving. The policy lives on the subscription:

RReliablePubSubTopic<OrderEvent> topic =
        redisson.getReliablePubSubTopic("orders");

// a day of history, like a stream left at its default retention
topic.setConfig(TopicConfig.defaults()
        .timeToLive(Duration.ofHours(24)));

// one subscription per consuming service, created once at deploy time
Subscription<OrderEvent> billing = topic.createSubscription(
        SubscriptionConfig.name("billing")
                .deliveryLimit(5)
                .visibility(Duration.ofMinutes(2))
                .deadLetterTopicName("orders-billing-dlt"));

A push consumer then decides, message by message, what a failure means:

PushConsumer<OrderEvent> consumer = billing.createPushConsumer(
        ConsumerConfig.name("billing-1"));

consumer.registerListener(MessageListenerArgs.listener((msg, ack) -> {
    try {
        charge(msg.getPayload());
        ack.acknowledge(MessageAckArgs.ids(msg.getId()));
    } catch (GatewayTimeoutException e) {
        // transient: try again in 30 s (each delivery counts toward the 5)
        ack.negativeAcknowledge(MessageNegativeAckArgs
                .failed(msg.getId())
                .delay(Duration.ofSeconds(30)));
    } catch (InvalidOrderException e) {
        // permanent: straight to orders-billing-dlt, no attempts wasted
        ack.negativeAcknowledge(
                MessageNegativeAckArgs.rejected(msg.getId()));
    }
}));

Four rules apply. A message that is neither acknowledged nor negatively acknowledged becomes visible again when its visibility timeout expires, which is what makes a crashed consumer safe. A failed negative acknowledgment redelivers the message after the delay you give, and every delivery counts toward the limit. A rejected message goes to the dead-letter topic at once. And with no dead-letter topic configured, a message that reaches its delivery limit is deleted; the Javadoc says it "may be moved to a dead letter topic if it's configured, otherwise it will be deleted". Configure one on every subscription.

The dead-letter topic is an ordinary Reliable PubSub topic, so you can inspect it, subscribe to it, or republish from it once the bug is fixed. Subscription statistics count acknowledged, negatively acknowledged and dead-lettered messages, along with redelivery attempts. For the general pattern, see dead letter queues, and for retry delays that grow with each attempt, exponential backoff and jitter in Java.

Ordering and Parallelism

Kinesis orders records within a shard, and an MD5 hash of the partition key decides the shard, so, in the words of the API reference, "all data records with the same partition key map to the same shard within the stream". The result is order per key with parallelism across keys, and it is the property most worth checking before a move. Two caveats from the same reference: PutRecords "doesn't guarantee the ordering of records" within a request, and strictly increasing order from one producer needs serial writes with SequenceNumberForOrdering. On-demand streams can also opt into service-managed partition keys (September 2026), which ignore the key and are meant for workloads that do not need ordering.

Reliable PubSub delivers a topic in first-in, first-out order. Once a subscription has more than one consumer, though, a fast consumer can finish message 4 before a slow one finishes message 3, the same trade-off Redis Streams consumer groups make.

Redisson's answer is message grouping, in Redisson PRO since 4.0.0. The group ID is set per message, so the value you used as the partition key can carry over unchanged, as in MessageArgs.payload(event).groupId(event.customerId()). The documentation says that messages with the same group ID "are guaranteed to be processed by the same consumer", and ConsumerConfig.groupIdClaimTimeout moves a group to another consumer when its owner stalls. That is the closest counterpart of a partition key, with weaker ordering: the same page warns that the "Order of redelivered messages with the same groupId isn't guaranteed", and priority and delay can also reorder messages within a group.

A work queue that must be processed strictly in order can use Reliable Queue's ProcessingMode.SEQUENTIAL, which consumes messages "strictly in the order they arrive in the queue", one at a time.

Scaling follows a different rule. A Kinesis stream scales by adding shards. Redisson documents partitioning for its other topic types but not for Reliable PubSub, so plan on each topic being served by a single Valkey or Redis primary. If one topic outgrows a node, split the traffic across several topics, for example one per tenant or per region, and plan that split where you would have planned partition keys.

Delay, Priority and Deduplication

Kinesis has none of the three. Its FAQ points readers who need "Individual message delay", message-level acknowledgment or a visibility timeout toward Amazon SQS, and for duplicates caused by producer retries the developer guide's advice is to "embed a primary key within the record to remove duplicates later when processing". Reliable PubSub and Reliable Queue take all three as per-message arguments:

topic.publish(PublishArgs.messages(
        MessageArgs.payload(event)
                .priority(9)
                .delay(Duration.ofMinutes(15))
                // the ID names this event, not the order it belongs to
                .deduplicationById(event.eventId(), Duration.ofHours(24))));

Priority runs from 0 to 9. A delay holds the message back for the duration you give, and Redisson documents no upper bound (SQS caps delays at 15 minutes). Deduplication drops a second message with the same ID inside the window (or, with deduplicationByHash, the same payload). Use an ID that names the event, since one order produces several events. A window longer than your producers' retry horizon removes the duplicates that appear when a timed-out publish is retried. Consumers should still be idempotent, because a redelivery after a visibility timeout is the same message delivered again and no deduplication window removes it.

Retention and Replay

Retention is Kinesis's strongest ground. A stream keeps every record for 24 hours by default and up to 8,760 hours (365 days) on AWS-managed storage, and any number of consumer applications can read any part of that window. Past seven days, long-term retention costs $0.023 per GB-month in US East. If you replay months of history, run stream processing over it, or start new consumers from last quarter, Kinesis (or Kafka) is the right home and Reliable PubSub is not.

Reliable PubSub keeps messages in Valkey or Redis memory, and three settings decide how long:

• The topic's retention mode. The default, SUBSCRIPTION_OPTIONAL_RETAIN_ALL, stores messages "regardless of subscription state or processing status". SUBSCRIPTION_REQUIRED_DELETE_PROCESSED discards a message once every subscription that existed when it was published has acknowledged it, failed it or exhausted its delivery limit, and it stores nothing published while the topic has no subscription.
• The topic's timeToLive, 0 by default, which means messages never expire. Set it, because in the default mode nothing else removes a message. maxSize caps the topic and removes nothing: once a topic is full, publish waits up to the timeout set in PublishArgs, if any, and then returns null without storing the message.
• Each subscription's retainAfterAck(). The Javadoc says that "By default, messages are removed after acknowledgment", and the documentation adds that an acknowledged message "is deleted from the subscription", so create a subscription with this option if you will want to rewind it over messages it has already processed.

// a new subscription that starts from the oldest message still kept
Subscription<OrderEvent> audit = topic.createSubscription(
        SubscriptionConfig.name("audit")
                .position(Position.earliest())
                .retainAfterAck());

// after a bad release: rewind to the minute before it went out
audit.seek(Position.timestamp(Instant.parse("2026-11-04T09:00:00Z")));

seek moves the subscription's offset for all of its consumers at once. The Kinesis equivalent for an existing KCL application is the lease-table reset described earlier.

Memory sets the limit. A stream averaging 100 KB/s produces about 8.6 GB of payload a day before any overhead, which is a lot of RAM to hold for replay alone. Keep replay windows on a topic short, in hours or a few days, and if you also need long history, archive the events to S3 separately.

Fan-Out and Read Limits

Without enhanced fan-out, every consumer of a Kinesis stream competes for the same per-shard read budget: 2 MB/s and five GetRecords calls per second, shared. AWS's latency figures show the cost of sharing, "around 200 ms if you have one consumer", going "up to around 1000 ms if you have five consumers". Enhanced fan-out gives each registered consumer its own 2 MB/s per shard over an HTTP/2 push connection, "Typically an average of 70 ms whether you have one consumer or five", for up to 20 consumers per stream, or 50 with On-demand Advantage. KCL 2.x and later use enhanced fan-out by default, so a KCL application gets the faster path, and its charges, unless you configure it to poll. In US East it costs $0.015 per consumer-shard-hour plus $0.013 per GB on provisioned streams and $0.05 per GB read on On-demand Standard, while On-demand Advantage bills those reads at the same $0.016 per GB as any other read.

Reliable PubSub has no per-shard read budget. Each subscription keeps its own offset and per-message state, so adding a subscription for a new service costs the memory for that state and the reads it makes against your Valkey or Redis node. Node-based clusters charge nothing per consumer, while ElastiCache Serverless bills those reads as ECPUs like any other command. The node's CPU and network set the limit.

Throughput and Message Size

Kinesis handles volumes that messaging on Valkey or Redis should not try to match. According to its quotas, each shard accepts 1 MB/s or 1,000 records per second of writes. An on-demand stream starts at 4 MB/s and scales up to 10 GB/s of writes in US East (N. Virginia), US West (Oregon) and Europe (Ireland), and to 200 MB/s by default in other Regions. Records are limited to 1 MiB by default, and since October 2025 a stream can be configured to accept records of up to 10 MiB, which AWS says should stay under 2 percent of overall traffic.

A Reliable PubSub topic is bounded by the primary that serves it, and every message sits in memory until retention removes it. With ElastiCache durability enabled, AWS also documents a ceiling of 100 MiB/s of writes per primary node. Keep messages small and put large payloads in S3 with a reference in the message, the same advice that applies to SQS. Before moving a stream, benchmark one node at your peak publish rate; if that peak runs to hundreds of megabytes per second, keep Kinesis.

What It Costs

Kinesis bills by capacity mode. In US East (N. Virginia), on 5 November 2026:

KCL adds a line of its own. Each consumer application keeps its lease table in DynamoDB, plus a worker metrics table and a coordinator state table unless it uses the KCL 3.5 single-table format, and AWS notes that "Your account is charged for the costs associated with the DynamoDB table". KCL 3.x creates these tables in on-demand capacity mode by default. Lambda consumers add their invocation charges.

Reliable PubSub and Reliable Queue add no per-stream fee. On node-based ElastiCache you pay for nodes, which you may already be paying for. MemoryDB also charges for data written ($0.20 per GB for Redis OSS; for Valkey, nothing up to 10 TB a month and $0.04 per GB beyond), and ElastiCache Serverless bills every command in ECPUs, so each publish and acknowledgment shows up there. Then add the Redisson PRO licence. On ElastiCache, Valkey is priced 20 percent below Redis OSS on nodes and 33 percent below it on Serverless, and synchronous durability adds 18 percent to the node-hour price.

The two bills grow differently. Kinesis charges scale with streams, shards, bytes and consumers, while the Redisson cost is fixed. Moving messages into a node-based cluster you already run saves the most when you have many small streams, each paying an hourly fee and read by a few consumers. That holds only if the cluster meets the durability you need. If it doesn't, add 18 percent per node-hour for synchronous durability, which a qualifying cluster can now switch on in place, or the price of a cluster that qualifies. A few large, busy streams usually cost less per gigabyte on Kinesis than they would in RAM. We do not publish a break-even table, because AWS prices change and your node sizes are your own.

Durability and Availability

Kinesis "synchronously replicates data across three Availability Zones", and its SLA promises at least 99.9 percent monthly uptime in each Region. There is nothing to configure and no way to weaken it.

Reliable PubSub is as durable as the Valkey or Redis deployment under it:

• ElastiCache for Valkey 9.0 or later with durability in synchronous mode persists each write in a Multi-AZ transactional log in at least two Availability Zones before responding, for 18 percent on top of the node-hour price. Asynchronous mode costs nothing extra and can lose up to 10 seconds of writes in a failure. Durability needs a node-based cluster with cluster mode enabled, Multi-AZ with at least one replica per shard and Graviton M6g, R6g or C7gn nodes or newer (T4g is not supported), and it turns on encryption at rest. Since 27 October 2026 you can also switch it on for an existing cluster that uses encryption in transit; the change replaces nodes, with a brief failover in each shard. In US East the smallest such cluster, two cache.m6g.large nodes with synchronous durability, costs about $200 for a 30-day month before the Redisson licence. Without the feature, ElastiCache replicates asynchronously and does not support the appendonly or appendfsync parameters, so there is no append-only file to fall back on.
• MemoryDB stores writes in a Multi-AZ transactional log before acknowledging them, and it offers smaller node types than a durable ElastiCache cluster.
• Self-managed Valkey or Redis gives you replication and, if you enable it, the append-only file (see Redis persistence).

On top of that, Redisson's syncMode sets what each operation waits for. The default, AUTO, waits for the append-only file on the primary and replicas where it is enabled, falls back to waiting for replicas to acknowledge, and "If neither durability mechanism is available, proceeds without synchronization guarantees". ACK waits for replica acknowledgment, and ACK_AOF waits for the append-only file, which needs Redis 7.2 or later or any Valkey version. On ElastiCache, which has no append-only file, choose ACK rather than ACK_AOF. The setting to change is syncFailureMode: its default, LOG_WARNING, logs a write that missed its replicas and lets the call succeed anyway.

// wait until the replicas acknowledge the write; throw after 2 s if not
topic.publish(PublishArgs.messages(MessageArgs.payload(event))
        .syncMode(SyncMode.ACK)
        .syncFailureMode(SyncFailureMode.THROW_EXCEPTION)
        .syncTimeout(Duration.ofSeconds(2)));

ElastiCache Serverless does not offer the WAIT command, so test what syncMode confirms there, and on any service test it against the engine version you deploy before relying on it.

Two Valkey settings matter as much as replication. Set maxmemory-policy to noeviction on any node that holds messages, because an eviction policy deletes keys under memory pressure and a deleted message is lost (see eviction policies). And alarm on memory well before the limit: with noeviction a full node rejects writes, which is the right kind of backpressure but a bad thing to discover in production.

How to Choose

If you are comparing Kinesis with other AWS services rather than with Redis, SQS is the queue for distributing work, with per-message acknowledgment and a dead-letter queue but no replay (see Amazon SQS vs Reliable Queue). Amazon MSK runs Kafka, which shares Kinesis's log model and gives you more configuration to own (see Apache Kafka vs Reliable PubSub and Redis Streams vs Kafka).

Moving From Kinesis to Reliable PubSub, Step by Step

Inventory the Readers

List every reader of the stream: each KCL application (its lease table in DynamoDB carries the application name unless that was overridden), each Lambda event source mapping and each enhanced fan-out consumer. For each one, note its starting position, what it does when a record fails, and where the failures go. In code, search for ShardRecordProcessor, IRecordProcessor, PutRecord, PutRecords and the KPL's KinesisProducer. Then decide stream by stream: streams that feed Firehose, Flink or other analytics stay, and streams whose only readers are your own Java services are the candidates.

Map the Settings

Prepare Valkey for Messages

Use ElastiCache for Valkey 9.0 or later with durability enabled (synchronous mode for events you cannot lose), or MemoryDB. Set noeviction, size memory for the replay window you chose plus the backlog of a long consumer outage, and alarm on memory.

Bridge, Then Cut Over

Create every service's subscription first, since a new subscription starts at the latest message by default. Then run a small KCL application that republishes every record to the topic. Its deduplication ID combines the shard, the sequence number and the sub-sequence number, so a replay after a restart publishes nothing twice:

@Override
public void processRecords(ProcessRecordsInput input) {
    if (stopped) {
        // a publish already failed on this lease: never checkpoint past it
        return;
    }
    try {
        for (KinesisClientRecord r : input.records()) {
            // unique across shards and KPL sub-records, so a replay of
            // this batch after a restart publishes nothing twice
            String id = shardId + "/" + r.sequenceNumber()
                    + "/" + r.subSequenceNumber();
            topic.publish(PublishArgs.messages(
                    MessageArgs.payload(decode(r.data()))
                            // longer than the stream's retention period
                            .deduplicationById(id, Duration.ofHours(25)))
                    // a write that misses its replicas must stop the bridge
                    .syncFailureMode(SyncFailureMode.THROW_EXCEPTION));
        }
        input.checkpointer().checkpoint();
    } catch (Exception e) {
        // KCL would skip this batch and the next checkpoint would pass it,
        // so stop and let a restart resume from the last good checkpoint
        stopped = true;
        log.error("bridge stopped on " + shardId, e);
    }
}

Set shardId in initialize(), apply the same stopped check in shardEnded() and shutdownRequested(), which also checkpoint, and alarm on the log line. After a restart the worker resumes from its last checkpoint, and the deduplication window, which must be longer than the stream's retention period (24 hours here), drops whatever was already published. Give the bridged topic a timeToLive but no maxSize: publish returns null both for a duplicate and for a full topic, and the bridge cannot tell the two apart. The sample stops on any failure, including transient ones such as a throttled checkpoint, and publishes one record at a time; if restarts become frequent or the shard is busy, retry transient errors with backoff and publish each batch with publishMany. With the bridge running, move consumers one service at a time. Stop the service's KCL application or event source mapping and, before its Redisson consumers start, seek its subscription with Position.timestamp to a few minutes before the oldest record it had not yet processed: its stop time minus its largest MillisBehindLatest, or minus its IteratorAge on Lambda. Without the seek, the service reprocesses every message the bridge has kept since the subscription was created; with it, idempotent consumers absorb a few minutes of overlap. When every service has moved, switch the producers to publish to the topic directly, wait until the bridge's MillisBehindLatest is zero on every shard, stop the bridge, and delete the stream once its retention period has passed.

Test the Failure Paths

Before production, check each of these on a staging cluster:

• A poison message reaches the dead-letter topic after deliveryLimit deliveries.
• A consumer killed mid-message has the message redelivered after its visibility timeout.
• A primary failover with syncFailureMode set to THROW_EXCEPTION surfaces errors to producers instead of a log line.
• A full node pushes back on publishers instead of evicting messages.

Frequently Asked Questions

Can Redis Replace Amazon Kinesis?

For one use, yes: Java services exchanging business events at moderate volume, where per-message retries, dead-letter topics, delays and priorities matter more than long replay. Redisson PRO's Reliable PubSub provides those on Valkey or Redis, including ElastiCache and MemoryDB. It does not replace Kinesis as an analytics pipeline, because its retention is bounded by Valkey or Redis memory, each topic has to fit one node, and it has no Firehose, Flink or Lambda integrations.

Is Amazon Kinesis a Message Queue?

No. Kinesis Data Streams is a sharded log. Consumers read records in order and save their position as a checkpoint, and records stay until the retention period ends whether or not anyone processed them. A message queue tracks each message, hides it while a consumer works on it, and removes it on acknowledgment. The Kinesis FAQ itself points users who need message-level acknowledgment, visibility timeouts or per-message delays to Amazon SQS.

Does Kinesis Data Streams Have a Dead-Letter Queue?

Not in the stream. KCL consumers have to catch failures and send them somewhere themselves. Lambda event source mappings can send discarded batches to an on-failure destination: SQS and SNS destinations receive metadata about the batch, such as the shard and sequence numbers, rather than the records, and S3 destinations receive the records too. The managed delivery to S3 and S3 Tables added in August 2026 has its own dead-letter location, but only for records it fails to deliver. Redisson's Reliable PubSub has a dead-letter topic per subscription.

What Happens When a KCL Record Processor Throws an Exception?

KCL logs the exception and moves on without retrying the batch, which AWS describes as consumer applications "observing skipped records". KCL 3.5.4 also counts those records in a RecordsFailedProcessing metric. Because the next checkpoint covers everything delivered so far, the skipped records are lost unless processRecords catches the failure and stores the records somewhere.

How Long Do Kinesis and Reliable PubSub Keep Messages?

Kinesis keeps records for 24 hours by default, configurable up to 365 days. A Reliable PubSub topic keeps messages until their time-to-live expires or its retention mode discards them, and by default nothing expires, so set a time-to-live that fits in memory. A size limit removes nothing; once the topic is full, it stops new messages.

How Do Redis Streams Compare with Kinesis?

Redis and Valkey Streams are logs too, with consumer groups, explicit acknowledgment through XACK, and a pending entries list for reclaiming messages from a stalled consumer, so they sit closer to Kinesis with some per-message state added. They have no delivery limits, dead-lettering, delays or priorities, which leaves those to your code. Reliable PubSub adds them on top of Valkey or Redis.

Is ElastiCache Durable Enough for Messaging?

Since June 2026, ElastiCache for Valkey 9.0 and later can store writes in a Multi-AZ transactional log. In synchronous mode a write is persisted in at least two Availability Zones before ElastiCache responds; asynchronous mode can lose up to 10 seconds of writes. The feature requires a node-based cluster with cluster mode enabled, Multi-AZ with at least one replica per shard and Graviton M6g, R6g or C7gn nodes or newer, and it is not available on ElastiCache Serverless. Since October 2026 it can be turned on for an existing cluster as well as a new one. MemoryDB uses the same kind of log.

Is Reliable PubSub Available in the Open-Source Edition?

No. Reliable PubSub, Reliable Queue and Reliable Fanout are Redisson PRO features. The PubSub and Queue interfaces ship in the open-source jar, so code using them compiles against community Redisson and then throws UnsupportedOperationException at runtime; getReliableFanout() is not in the open-source client at all.

Does Reliable PubSub Work with Valkey?

Yes. It runs on Valkey and Redis, self-hosted or managed, including Amazon ElastiCache and MemoryDB. The ACK_AOF synchronization mode needs Redis 7.2 or later, or any Valkey version.

Next Steps

All about Reliable PubSub walks through topics, subscriptions and acknowledgments, and All about Reliable Queue does the same for queues. Reliable PubSub vs Valkey and Redis Streams covers what the plain Streams commands leave to you. To connect Redisson to AWS, see Redisson on Amazon ElastiCache and Redisson on Amazon MemoryDB. The full configuration reference is in the Reliable PubSub documentation and the Reliable Queue documentation.

Reliable PubSub and Reliable Queue are part of Redisson PRO. You can try it for free.

Similar articles

• Amazon SQS vs. Reliable Queue for Valkey and Redis
• Apache Kafka vs Valkey & Redis Based Reliable PubSub
• Google Pub/Sub vs Redis for Messaging
• Azure Service Bus vs Valkey & Redis Based Reliable Queue and PubSub
• Reliable PubSub vs. Valkey & Redis Streams
• Apache Kafka Alternatives: 7 Options Compared