Handling Backpressure in Java with Valkey and Redis Reliable Queue

Move a producer-consumer pipeline from one JVM to twenty and you lose something you probably never noticed you had. A local ArrayBlockingQueue applies backpressure for free: when the queue is full, put() stops the producer thread until space appears. Nobody designed that behaviour into the application — it came with the data structure.

Now the queue lives in Valkey or Redis and the producers run on other machines. The queue has no thread of theirs to suspend, so the producers keep writing and the queue keeps growing. Neither server bounds queue depth. What happens as memory fills depends on configuration — unbounded growth until the operating system intervenes, server-wide write failures, eviction of unrelated keys, or eviction of the queue itself — and none of those outcomes reaches a producer as a signal about this queue. That breakdown belongs to backpressure; this article is about getting the lost signal back.

Why Distributed Queues Lose Backpressure

Backpressure needs two things: a buffer with a limit, and a way to tell the producer the limit has been reached. In one process both come free — a fixed-size array, and the scheduler suspending a thread. Across a network neither implementation survives, and the consequence is sharper than it first looks: the queue can no longer suspend a producer on its own initiative. A producer may still choose to wait, but it has to be told there is something to wait for, so the signal has to become an explicit part of the API — something the enqueue call returns and the producer is written to handle.

This is why the obvious distributed equivalents do not solve the problem. Redisson's RBlockingQueue is unbounded on the producer side: put() never waits for space and offer() never returns false, however far behind the consumers are. It blocks only consumers waiting for work to arrive. RBoundedBlockingQueue does provide a capacity via trySetCapacity(), and it is in the open-source edition, but Redisson 3.46 deprecated it alongside RDelayedQueue when Reliable Queue arrived. Deprecated objects still work and are still shipped; they stop receiving fixes and eventually go. Building new producers on one is borrowing against a removal date.

The result is a queue that absorbs an indefinite backlog in server memory. Queue depth is climbing the whole time, but unless something is watching it, the first visible event is the ceiling itself. Of the outcomes listed at the top, only the write failure announces itself promptly. The others stay quiet until the ceiling arrives, and an out-of-memory kill is not quiet so much as terminal. On a shared instance it may not even surface as a problem with this queue.

Bounding the Queue with maxSize

Reliable Queue is a Redisson PRO feature; the interfaces are in the open-source jar, so the code below compiles either way and fails at runtime on the community edition. It is configured through QueueConfig, and two of its settings turn it into a bounded buffer — the third names a dead-letter queue that matters later:

RReliableQueue<Order> queue = redisson.getReliableQueue("orders");

queue.setConfig(QueueConfig.defaults()
    .maxSize(10_000)
    .maxMessageSize(64 * 1024)
    .deadLetterQueueName("orders-dlq"));

maxSize is the bound backpressure requires — without it there is nothing for a producer to run into. maxMessageSize rejects individual payloads above a byte limit. Both default to 0, which means unlimited, so a queue you never configured has no backpressure at all.

The two limits guard against different failures. A message count protects against many small messages; a byte limit protects against a small number of very large ones. Ten thousand messages sounds bounded until someone starts enqueueing 5 MB payloads, at which point the "bounded" queue is holding 50 GB. Set both.

Sizing maxSize starts with arithmetic rather than instinct. Take the burst you intend to absorb: if producers peak at 2,000 messages a second, consumers drain 1,500, and bursts run about twenty seconds, the backlog to hold is (2,000 − 1,500) × 20 = 10,000 messages, which is the figure used above. Then check the payload range against memory: at a typical 8 KB payload that backlog is roughly 80 MB, but against the 64 KB maxMessageSize also configured above, the worst case is nearer 640 MB. Size the queue so the worst case still fits, or lower maxMessageSize until it does.

Raise the count a little beyond that figure, because Redisson documents maxSize as the number of messages "stored in the queue" without saying whether in-flight and delayed messages count toward it — assume they do until you have measured it, and recheck the memory figure against whatever number you land on. A limit set from rough arithmetic still converts an invisible accumulation into a visible error, which no limit at all can do.

One deployment constraint sits underneath all of this. A maxSize you have carefully chosen is worth nothing if the queue shares an instance with a cache running an allkeys-* policy, because the queue key is then evictable like any other. Give the queue an instance set to noeviction, with maxmemory headroom above maxSize × maxMessageSize.

The Signal Is What add() Returns

A bound on its own already gives you a signal: with no timeout set, a full queue turns the add away immediately. What the timeout adds is patience — a window in which a brief burst can drain before the producer gives up on it:

Message<Order> msg = queue.add(QueueAddArgs
    .messages(MessageArgs.payload(order))
    .timeout(Duration.ofSeconds(5)));

if (msg == null) {
    // The queue stayed full for five seconds.
    // Consumers are behind. Decide what this producer does
    // about it — but do not immediately retry.
}

When the queue is at maxSize, the add operation waits for space. If space appears within the timeout the message is enqueued normally. If it does not, add() returns null. That null is the backpressure signal — the moment the producer learns that the consumer fleet cannot keep up. A non-null return means accepted, which is not the same as durable: with asynchronous replication an acknowledged enqueue can still be lost in a failover, and that is a persistence question, not a backpressure one.

The batch equivalent needs a different check. addMany() returns the messages it did add — an empty list when nothing fit, and a list shorter than the batch when only some did. Deduplication can shorten it too, independently of space. Compare sizes rather than testing for empty:

List<Message<Order>> added = queue.addMany(QueueAddArgs
    .messages(MessageArgs.payload(first), MessageArgs.payload(second))
    .timeout(Duration.ofSeconds(5)));

if (added.size() < 2) {
    // At least one message was not enqueued.
}

The name of one value is a trap. Duration.ZERO does not mean "do not wait" — it means wait indefinitely, the opposite of the no-timeout default above. That gives the closest equivalent to a local put(): the calling thread blocks until space exists.

One thing the null does not cover: these calls go over a network, and a failover or a connection-level timeout throws rather than returning null. A producer that only handles null will bypass its own overflow handling whenever the connection, not the queue, is the problem. Treat a thrown exception as a third outcome — not enqueued, and possibly not even attempted. An add that fails after the server accepted the message is the harder case: retrying it duplicates the work, which is what deduplicationById exists to absorb.

Whatever the timeout, the one response to avoid is an immediate retry. A producer that loops on a full queue converts backpressure into a hot spin, sending more traffic to a system that has just reported it has too much. Use exponential backoff with jitter, so that a fleet of producers does not converge on one retry schedule and arrive together — and bound whatever holds the work while it waits, because a retry buffer is a buffer like any other.

Four Producer Strategies When the Queue Is Full

Receiving the signal is the easy half. Deciding what to do with it is the design work. The obvious answer is to add consumers — a bounded queue with a depth metric is a good autoscaling trigger, and if the mismatch is sustained it is the only answer that fixes anything. Everything below is what the producer does in the meantime, and there are four defensible options.

Block. Set timeout(Duration.ZERO) and let the producer wait for space. Correct for batch importers, ETL jobs and background workers — anything where the work must eventually happen and arriving late is acceptable. Wrong anywhere a caller is waiting on a response.

Fail fast. Use a short timeout, handle the null, and propagate the failure to whatever called you. In an HTTP path this becomes a 503 or a 429, which pushes the backpressure signal all the way out to the client where it belongs. This is usually the right default for synchronous request handling.

Shed. Drop work deliberately, choosing what to lose instead of losing arbitrarily. The important detail is that the admission check comes before the add — otherwise every low-priority message still burns the full timeout before you discard it, which is the worst of both worlds:

private static final Duration SHED_WINDOW = Duration.ofSeconds(10);
private final AtomicLong shedUntil = new AtomicLong(System.nanoTime());

// Admission check first: shed before spending a slot or a timeout.
if (isLowPriority(order) && System.nanoTime() - shedUntil.get() < 0) {
    metrics.counter("orders.shed").increment();
    return;
}

Message<Order> msg = queue.add(QueueAddArgs
    .messages(MessageArgs.payload(order))
    .timeout(Duration.ofSeconds(2)));

if (msg == null) {
    shedUntil.set(System.nanoTime() + SHED_WINDOW.toNanos());
    metrics.counter("orders.enqueue.timeout").increment();
    throw new QueueFullException(order.id());
}

What makes this work is the shape of the control flow. Every path either enqueues, counts a shed, or throws — silently returning on a null is how a shed strategy turns into undiagnosed data loss. And the shed window is a deadline rather than a flag, so it expires on its own: a boolean latch set on failure and cleared on success looks equivalent, but a producer whose traffic is entirely low priority never reaches the clearing line and sheds for the life of the process.

Three details matter here. null strictly means "not enqueued" rather than "queue full", because add() also returns it when a message is dropped by deduplication — harmless here, but a producer that sets deduplicationById must distinguish them, or a duplicate submission will open a shed window and report congestion that never happened. The window is per-process, so in a fleet each producer decides independently and shedding ramps up as instances hit the wall instead of all at once. And MessageArgs.priority() governs the order in which messages are dequeued, not whether they are admitted — admission is the application's decision, as above.

Be careful what "low priority" is allowed to mean. Shedding is only safe where the work is regenerable or superseded by the next update — an analytics event, a recommendation refresh, a re-index request. Transactional work does not qualify, however low its priority, and belongs in divert rather than shed.

Divert. Handle the null and write the work somewhere cheaper — object storage, a secondary queue — then replay it when the system recovers. This preserves the work at the cost of ordering and immediacy, and suits pipelines where completeness matters more than latency. Note that this is application code, not a queue setting: the deadLetterQueueName configured earlier receives messages that failed processing — after their delivery limit, or on an explicit rejected acknowledgment — not messages that failed to be enqueued. The two paths are unrelated, and conflating them is an easy way to lose overflow silently.

One producer may use more than one. Shedding low-priority work while failing fast on the rest is a sensible combination.

Bounding the Consumer Side

Capping the queue protects the queue. It does not stop a consumer from pulling more work than it can process and running out of memory on its own, which reintroduces the same failure one hop downstream.

List<Message<Order>> batch = queue.pollMany(QueuePollArgs.defaults()
    .count(10)
    .timeout(Duration.ofSeconds(20))
    .visibility(Duration.ofSeconds(120))
    .acknowledgeMode(AcknowledgeMode.MANUAL));

for (Message<Order> m : batch) {
    try {
        process(m.getPayload());
    } catch (Exception e) {
        queue.negativeAcknowledge(QueueNegativeAckArgs
            .failed(m.getId())
            .delay(Duration.ofSeconds(30)));
        continue;
    }
    queue.acknowledge(QueueAckArgs.ids(m.getId()));
}

Four of the poll arguments carry weight here. count bounds how much a single consumer takes at once — the consumer-side equivalent of maxSize. timeout makes this a long poll: without it the call returns immediately, and a consumer loop around it becomes a hot spin against an empty queue. visibility hides in-flight messages from other consumers for a fixed period, so a slow worker does not have its work duplicated, and a dead one has its work returned to the queue when the window expires. AcknowledgeMode.MANUAL means a message is only removed once processing has actually finished, so a consumer that dies mid-batch does not take the work with it.

Handling the failure explicitly is the other half of it. Without the negativeAcknowledge branch, a message whose processing throws is neither acknowledged nor negatively acknowledged, so nothing happens until the visibility window expires and it comes back on its own.

Size the visibility window against the whole batch, not one message. The window starts when the batch is polled, so ten messages at seven seconds each need more than seventy seconds. The 120-second window configured above covers that; a sixty-second one would not, and the tail would be redelivered to another consumer while the first was still working on it. The interaction between count and visibility is easy to miss precisely because it only bites under load, when batches are full and processing is slowest.

Consumer concurrency is set queue-wide with QueueConfig.processingMode(...) rather than per poll. That setting arrived in Redisson 3.47, along with the queue listeners used below. PARALLEL, the default, allows concurrent processing across consumers, while SEQUENTIAL enforces one message at a time, which is what you want when the downstream system tolerates no concurrency at all. For the more common case — a downstream that tolerates some concurrency but not unlimited — neither setting helps, because the two modes offer one or unbounded-by-the-setting, with nothing in between. A distributed semaphore is the right tool there, bounding work in flight across the whole fleet at whatever number the dependency can take.

Monitoring Queue Depth

Backpressure that nobody observes is just a source of mysterious errors. The metric that matters is queue depth. Read it with size(), which counts messages ready for polling — note that it excludes delayed and unacknowledged messages, so a queue whose consumers are all mid-batch can report a smaller number than the work outstanding. countUnacknowledgedMessages() and countDelayedMessages() supply the rest. Sum all three for a true picture of outstanding work.

Throughput is a lagging indicator. A saturated pipeline shows healthy throughput right up to the moment it fails, since every consumer is busy by definition. Depth is a leading indicator: a queue that gains a few hundred messages an hour is telling you, well before anything breaks, that producers are outpacing consumers.

Alert on sustained growth rather than absolute depth. A queue briefly at eighty percent during a traffic spike is a buffer doing its job. A queue that has climbed steadily for six hours is a capacity problem regardless of how full it currently is. Two producer-side counters make the picture complete — both appear in the shed example above:

• orders.enqueue.timeout — how often backpressure actually engaged.
• orders.shed — what engaging it cost you.

Read them together: timeouts rising while sheds stay flat means the window is opening but no low-priority traffic is arriving inside it, so callers are taking the exceptions; both rising means shedding has engaged and is buying the consumers time. Neither counter moving while depth climbs steadily — not the brief spike above, but hours of it — means the bound is set too high to be doing anything.

Queue-side instrumentation is available through listeners. FullEventListener fires when the queue reaches its limit, which reports the backpressure condition directly instead of inferring it from producer timeouts; AddedEventListener and PolledEventListener give enqueue and poll rates. Note the cast: the addListener parameter type, QueueEventListener, is an empty marker interface rather than a functional one, so the lambda has to name the subinterface it implements. Keep the returned id if you ever need to remove the listener:

String listenerId = queue.addListener((FullEventListener) name ->
    metrics.counter("orders.queue.full").increment());

Client-level instrumentation is covered in Redis client metrics in Java.

How This Compares to Kafka and SQS

Neither of the obvious alternatives bounds a backlog. Kafka refuses writes for unrelated reasons — oversized batches, quotas, storage failures — but never because the backlog is large; SQS refuses sends only for oversized messages or throughput quotas.

Kafka gives producers a local buffer governed by buffer.memory, and max.block.ms controls how long send() waits when that buffer is full before failing the send with BufferExhaustedException — delivered through the returned Future and the callback rather than thrown from send() itself. That is real backpressure, but it is producer-local: it reflects the state of one client's send buffer, not the state of the topic. Consumer lag can grow for hours without any signal reaching the producer, because the broker accepts writes regardless of how far behind the consumer group has fallen. Consumer parallelism is no longer pinned to partition count either — share groups, generally available in Kafka 4.2, decouple the two, bounded instead by group.share.max.size.

Retention discards old segments by age or size, read or not. Note also that retention.bytes applies per partition rather than per topic, so a 10 GB setting on a 50-partition topic is 500 GB. If lag exceeds retention the consumer's offset falls off the log entirely, which is silent data loss for the consumer and still invisible to the producer.

Amazon SQS holds an unlimited backlog: there is no depth limit to reach and no full-queue signal for a producer to receive, so producers never learn that consumers are behind. The only bound on a backlog is the retention period — four days by default, fourteen at most — after which unread messages are deleted outright. SQS does bound concurrency at roughly 120,000 in-flight messages, surfaced as an OverLimit error on short polling, but that limits work being processed, not work waiting, and it is returned to the consumer rather than the producer.

The distinction is not that Reliable Queue is uniformly better — Kafka's retention model is the right design for event streaming, where replay matters more than admission control. It is that a bounded queue is the only one of the three that tells a producer to stop because the backlog is too large.

Frequently Asked Questions

Does RBlockingQueue Support Backpressure?

No. Its blocking is on the consumer side only — workers wait for messages to arrive, but nothing a producer calls will ever wait or refuse. RBoundedBlockingQueue does support a capacity, but it was deprecated in Redisson 3.46 in favour of the Reliable Queue size limit.

What Happens If I Set maxSize but No Timeout?

The add returns immediately instead of waiting: with no timeout there is no window in which space can appear, so a full queue produces null from add() straight away. That is fail-fast behaviour, which is a reasonable default for request handling — but it means a brief burst that a two-second wait would have absorbed is rejected instead. Set a timeout whenever the producer can afford to wait.

What Happens When a Reliable Queue Reaches maxSize?

Add operations wait for space up to the configured timeout. If space does not appear, add() returns null and addMany() returns only the messages it managed to add — an empty list if none fit, so compare sizes rather than testing for empty. A timeout of Duration.ZERO waits indefinitely. A maxSize of 0, which is the default, means no limit is applied and no backpressure occurs.

Should the Producer Block or Fail Fast When the Queue Is Full?

Block when the work must eventually happen and no caller is waiting — batch imports, scheduled jobs, background processing. Fail fast when a user or upstream service is waiting on a response, and turn the failure into a 503 or 429 so the signal reaches the client. Blocking inside a request handler exhausts the thread pool and turns a queue problem into an outage.

How Do I Monitor Whether Backpressure Is Being Applied?

Track queue depth over time and alert on sustained growth rather than a fixed threshold. Add a counter for enqueue timeouts, which records how often backpressure engaged, and one for shed messages, which records what it cost. Depth is a leading indicator; throughput looks healthy until the moment of collapse.

Is Reliable Queue Available in the Open-Source Edition?

No, it is a Redisson PRO feature. The interfaces ship in the open-source jar, so this code compiles against community Redisson and then fails when it runs: getReliableQueue() throws UnsupportedOperationException. Compiling successfully is not evidence that you have the feature.

Does This Work with Valkey?

Yes. Valkey is wire-compatible with Redis and every example here runs unchanged against either.

Next Steps

For the concept behind all this, see what backpressure is, which covers how it differs from rate limiting, throttling and load shedding. If you are choosing a queue type rather than tuning one, Distributed priority queues in Java takes on priorities and starvation, and All about Reliable Queue goes through acknowledgments, delivery limits and dead-letter routing. Full configuration is in the Redisson queues documentation.

The Reliable Queue, along with local caching, data partitioning and advanced eviction, is part of Redisson PRO. You can try it for free.