All about Reliable PubSub for Valkey and Redis
In today's distributed Java applications, the Publish-Subscribe (PubSub) messaging pattern provides the foundation for real-time communication. PubSub allows Java developers to easily decouple services and apply logic to events as they occur.
For years, the high-performance data store has been the go-to engine for PubSub, thanks to its enterprise-grade speed and lightweight footprint. With the recent rise of Valkey — the lightning-fast, open-source fork of Redis — Java developers now have even more options for their data layer. However, as robust as Valkey and Redis are, their native PubSub implementations have one glaring flaw: they're designed for speed, not reliability.
For mission-critical applications, where every message matters, fast performance isn't enough. You need guaranteed delivery. Fortunately, Java developers can add reliability to Valkey and Redis with Redisson PRO and its Reliable PubSub implementation. With Reliable PubSub, Redisson PRO adds the durability, ordering, and consistency your distributed Java applications require. Here's everything you need to know about Reliable PubSub for Valkey and Redis.
The Problem With Native PubSub in Valkey and Redis
To fully appreciate the value of Reliable PubSub, you first need to understand where the native PubSub implementations in Valkey and Redis fall short.
When a publisher sends a message to a channel in Valkey or Redis, the server immediately broadcasts it to all currently connected subscribers. Once sent, the message is gone from the server's memory. This PubSub model is well-suited to use cases such as live chat rooms or volatile metrics where missing a single packet is acceptable.
However, for modern microservices and event-driven architectures, this approach introduces three critical problems:
- A lack of persistence: If a subscriber is offline when a message is published (for example, during a deployment or after a crash), the message is lost permanently.
- No acknowledgments: The publisher has no way of knowing if the subscriber successfully processed the message. If a consumer crashes while processing an event, the event is effectively dropped.
- Subscriber overload: In a broadcast model, all subscribers receive all messages. If one consumer is slow, it cannot keep up; it either blocks the connection or drops messages, resulting in inconsistencies.
This type of delivery is known as "at-most once." For common workflows such as financial transaction processing, order fulfillment, or system-wide notifications, you have no tolerance for dropped messages or delivery inconsistencies. Instead, you need a system that guarantees "at least once" (and ideally "exactly once") processing.
How Reliable PubSub Works
Unlike the native Valkey and Redis broadcast model, Redisson PRO's Reliable PubSub offers a three-tiered, structured hierarchy:
- Topic: The central channel where messages are published. It acts as the durable storage mechanism.
- Subscription: A logical entity that tracks the consumption progress (offsets) for a group of consumers.
- Consumer: The actual Java process or thread that retrieves and processes messages from a subscription.
How Reliable PubSub is Different
Redisson PRO's Reliable PubSub model differs from the stock Valkey and Redis implementation in several key ways, including:
Multiple retention modes: Reliable PubSub provides flexible message storage options that you can tailor to your application's needs. For example, you can immediately remove messages once all consumers have acknowledged them to optimize memory usage. Alternatively, you can retain a fixed-size history, such as keeping the last 10,000 events.
Pre-subscription offsets: Each subscription maintains its own pointer in the message stream. One consumer group can be processing real-time data while another "replays" historical data from hours ago without affecting the other.
Active consumer tracking: If a consumer goes silent, its workload can be rebalanced to maintain high availability.
Reliable PubSub's Core Messaging Patterns
Reliable PubSub is flexible enough to support various architectural patterns, allowing you to tailor the data flow to your specific business logic.
Many-to-One
In the Many-to-One pattern, multiple publishers send messages to a single topic, which are then processed by a single subscription. This is ideal for log aggregation or audit trails, where many services generate events (such as user logins or errors) that must be serialized and stored by a central processor.
Many-to-Many (Fan-Out)
The Many-to-Many or Fan-Out pattern is like the classic PubSub scenario, but with durability. Multiple publishers send messages to a topic, and multiple subscriptions listen to it.
For example, an "Order Placed" event might be published once but consumed by three distinct downstream services:
- Inventory service: To decrement stock.
- Shipping service: To generate a label.
- Notification service: To email the customer.
Each service has its own subscription, so they process the same message independently and at their own pace.
One-to-Many (Load Balancing)
With the One-to-Many pattern, a single subscription is shared by multiple consumers. Redisson PRO automatically distributes messages among available consumers, effectively load-balancing the workload. If you have a high-volume message topic, you can simply spin up more Java consumer instances to increase throughput.
How Redisson PRO Maintains Reliability
Experienced Java developers know that programming is easy when everything works as it should, but the real trick is in the error handling. Standard PubSub implementations don't offer the reliability required for mission-critical apps. Redisson PRO's Reliable PubSub maintains reliability with these features:
Message Acknowledgment and Visibility Timeouts
In a reliable system, message processing isn't considered "done" until the consumer says so. Redisson PRO ensures this happens with explicit message acknowledgements.
When a consumer pulls a message, it becomes invisible to other consumers for a set period — this is Reliable PubSub's visibility timeout. If the consumer processes it successfully, it sends an ack (acknowledgment), ending the timeout and providing confirmation to the rest of the system.
Automatic Redelivery and Delivery Limits
Take a look at this Java code:
// Accessing the Reliable Topic
RReliablePubSubTopic topic = redisson.getReliablePubSubTopic("orders");
// Get a subscription
Subscription subscription = topic.getSubscription("order-processing-sub");
// Create a Pull consumer with manual acknowledgement
PullConsumer consumer = subscription.createPullConsumer(ConsumerConfig.name("worker-1"));
// Pull and process
List<Message> messages = consumer.pull(PullArgs.defaults()
.timeout(Duration.ofMinutes(5)); // Visibility timeout
for (Message msg : messages) {
try {
processOrder(msg.getValue());
// Success: Acknowledge the message
consumer.ack(msg.getId());
} catch (Exception e) {
// Failure handling...
}
}So, what happens if the processOrder method above fails?
If the consumer crashes or fails to acknowledge the message within the timeout, Redisson PRO automatically makes the message visible again. It will be redelivered to the next available consumer.
Reliable PubSub also offers a Delivery Limit feature. You might use this for instances when a message is redelivered five times and still fails. The limit prevents a failed message from permanently blocking your queue.
Dead Letter Topics (DLT) for Failed Messages
When a message exceeds its delivery limit, Redisson PRO automatically moves it to a Dead Letter Topic (DLT).
The DLT is a special queue for failed messages. It allows a developer or a sysadmin team to investigate the problematic data later without stopping the system from processing other messages.
Deduplication and Message Grouping
Reliable PubSub also automatically handles message grouping. By assigning a Group ID to messages, you ensure that related messages — perhaps all updates for a specific order in an eCommerce app — are always processed by the same consumer instance.
Durability Beyond In-Memory
Part of what makes Valkey and Redis so fast is that they store data in memory. However, a common concern is what happens to that data if the power goes out, effectively clearing the server's memory.
Redisson PRO mitigates this by leveraging the database engine's underlying persistence options to deliver these features:
Synchronous replication: You can configure Redisson to wait for a message to be replicated to a specified number of replicas (slaves) before marking the publish operation as successful. This ensures that even if the master node fails immediately after a write, the data exists on a replica.
AOF integration: By configuring your Valkey or Redis instance with append-only file (AOF) persistence (fsync every second or always), your topics are written to disk. Since Reliable PubSub stores messages as data structures rather than ephemeral buffers, they persist just like any other key-value pair.
Failure handling: If a node goes down, Redisson PRO's client-side logic automatically reconnects to the new master node. Consumers resume processing from their last known offset, ensuring no data is lost.
When to Use Reliable PubSub
Reliable PubSub is ideally suited for event-driven microservices, such as:
- Order processing in an eCommerce app.
- Ingesting data streams from IoT devices.
- Financial ledger systems where regulations require transaction logs to be immutable and complete.
However, there are some potential trade-offs to consider. Depending on your Java app, Reliable PubSub may incur a slight performance overhead compared to the "fire-and-forget" PubSub approach native to Valkey and Redis, as it adds the workload of storing data and managing acknowledgments.
On the other hand, deploying Redisson PRO with your existing Valkey or Redis infrastructure is often easier to implement and manage than separate message-broker clusters such as Kafka or RabbitMQ.