Is Kafka a Message Queue?

For most of its life the answer was no. Apache Kafka is a distributed commit log: producers append records, consumers track their own position in the log, and nothing is removed because somebody read it. You could bend that into a work queue, and plenty of teams did, but you were working against the design rather than with it.

That changed in 2026. Share groups, added by KIP-932 and generally available in Apache Kafka 4.2, put queue semantics into the broker itself — per-record acknowledgment, delivery counting, and consumer parallelism that is no longer capped by partition count. So the honest current answer is: Kafka is a log that can now also behave like a queue, and whether that is good enough depends entirely on which queue features you need.

This post covers what actually shipped, what is still missing, and how to decide.

Why Kafka Wasn't a Queue

The gap was never about durability. Kafka has always been better than most brokers at not losing data. The gap was about the unit of progress.

In a queue, progress is per message: you take a job, you finish it, you acknowledge it, and it is gone. In Kafka's consumer groups, progress is an offset — a single position per partition marking how far a consumer has read. Commit offset 500 and you have implicitly declared everything before it done. There is no way to say "497 and 499 succeeded, 498 is still running," which is exactly what a work queue needs to express.

That model brought a second constraint. Consumer groups map each partition to exactly one consumer, so partition count is a hard ceiling on parallelism. A topic with six partitions cannot usefully run more than six consumers, no matter how deep the backlog. Teams responded by over-provisioning partitions for peak load and carrying that cost permanently. And because a stalled record blocks the partition behind it, one slow job could hold up everything queued after it.

None of this makes Kafka bad. It makes Kafka a log — exactly right when you need to replay history, rebuild state from scratch, or feed a stream processor, and awkward when you just need ten thousand password-reset emails sent once each.

What Share Groups Changed

KIP-932 introduces a second consumption model alongside consumer groups. A share group lets many consumers read the same partitions cooperatively, with no sticky partition-to-consumer assignment. You can run more consumers than partitions, and scale them up and down without repartitioning the topic.

The mechanism is an acquisition lock. When a share consumer fetches a record, the broker locks it for that consumer for a limited time — 30 seconds by default. The consumer then does one of four things with it:

• Accept. Processing succeeded; the record is marked done.
• Release. Processing failed transiently; the lock drops immediately and another consumer can pick it up.
• Reject. The record is unprocessable and should not be retried.
• Renew. Work is still in progress; extend the lock rather than let it lapse.

If the consumer does nothing at all, the lock expires and the record becomes available again. The broker counts delivery attempts per record, with a configurable maximum that defaults to five. Anyone who has used SQS or RabbitMQ will recognise all of this: it is a visibility timeout, a negative acknowledgment, and a delivery limit, implemented at the broker.

The API is deliberately familiar. KafkaShareConsumer mirrors KafkaConsumer closely enough that simple applications can swap one for the other, and share consumers can read topics that traditional consumer groups are already consuming — adoption is additive rather than a migration.

Properties props = new Properties();
props.setProperty("bootstrap.servers", "localhost:9092");
props.setProperty("group.id", "order-workers");
props.setProperty("share.acknowledgement.mode", "explicit");

KafkaShareConsumer<String, String> consumer =
    new KafkaShareConsumer<>(props, new StringDeserializer(), new StringDeserializer());
consumer.subscribe(Arrays.asList("orders"));

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(1000));
    for (ConsumerRecord<String, String> record : records) {
        try {
            processOrder(record);
            consumer.acknowledge(record, AcknowledgeType.ACCEPT);   // done
        } catch (TransientFailure e) {
            consumer.acknowledge(record, AcknowledgeType.RELEASE);  // retry
        }
    }
    consumer.commitSync();
}

Two things in there are easy to get wrong. The first is RELEASE versus REJECT: RELEASE drops the lock so another consumer retries the record and the delivery count goes up, while REJECT declares the record permanently unprocessable and sends it straight to the archived state. In Kafka 4.2 an archived record is not written anywhere, so reaching for REJECT on an ordinary transient failure destroys the payload on the first attempt. Reserve it for records you have positively determined can never succeed.

The second is the try/catch. In explicit acknowledgement mode every record from a poll must be acknowledged before the next poll, or the client throws IllegalStateException. An exception escaping your processing logic leaves a record unacknowledged and takes the consumer down on the following loop — so the handler is not optional defensiveness, it is a requirement of the mode.

This is a real improvement and worth knowing about. If you already run Kafka and have been building queue behaviour by hand on top of consumer groups, you can now stop.

What Kafka Queues Still Don't Do

Share groups closed the acknowledgment gap. They did not close every gap, and the remaining ones are the features most job-processing systems assume they have.

There is no dead letter queue yet

This is the one to understand before putting anything important through a share group. When a record exhausts its delivery attempts or is explicitly rejected, Kafka marks it archived and moves on. It is not routed anywhere. There is no dead letter topic, no artifact to inspect, no record of the payload beyond metrics telling you a count went up.

A poison message — a malformed payload, a schema change nobody coordinated, a null where the consumer expected a value — is therefore silently discarded after five attempts. For an analytics event that is survivable. For a payment instruction or an order webhook, it is data loss you find out about from a customer.

The maintainers regard this as a gap too. KIP-1191 was voted through in January 2026 to close it, turning on behind a share.version=2 feature flag and slated to arrive with Kafka 4.4. Until that lands, capturing poison messages from a share group is code you write yourself — or something you go without. Our guide to the dead letter queue pattern covers how other brokers handle it.

Ordering is traded away, not preserved

Share groups deliberately give up ordering in exchange for elastic scaling. Multiple consumers process the same partition concurrently, so records do not complete in the order they were produced. This is the correct trade for a work queue and a real problem if you assumed partition ordering still applied. Key-based ordering — partial ordering per record key alongside cooperative consumption — is on the roadmap, not in the product.

No priority, no delayed delivery

Neither feature exists in share groups, and neither appears on the published roadmap. If some jobs must jump the line, or a retry should fire in ten minutes rather than immediately, that logic lives in your application — usually as a second topic and a scheduler, which is the kind of accidental infrastructure a broker is supposed to save you from. Exponential backoff between redelivery attempts is also roadmap rather than shipped.

The rest of the fine print

Exactly-once semantics for share groups, through delivery acknowledgment in Kafka transactions, is planned but not available. On Confluent Cloud, Queues for Kafka runs on Enterprise and Dedicated clusters only. And at GA only Apache Kafka 4.2+ Java clients are supported, with other languages targeted for the second half of 2026 — so a polyglot estate cannot adopt this uniformly yet.

Kafka Topics vs. Queues

A point of terminology that causes real confusion: Kafka still has no queue object. You do not create a queue, and there is no queue in the admin API. A share group is a way of consuming a topic, not a different kind of destination. The same topic can be read by a consumer group and a share group simultaneously, with each maintaining its own independent state.

This matters operationally. Retention is still a topic-level property governed by time or size, not by whether the work was done. A record that every share group member has accepted stays on disk until retention expires — useful if you want to replay it, surprising if you expected acknowledgment to free storage the way it does in a conventional message queue.

So Should You Use Kafka as a Queue?

The decision is less about Kafka's capabilities than about what is already running in your estate.

Use share groups if Kafka is already in production and well operated; the work is ordinary task distribution without priority or scheduling requirements; you can tolerate the DLQ gap until 4.4, or you are willing to write the routing yourself; you are on the JVM; and replay is genuinely valuable to you, because retaining processed jobs is something conventional queues cannot do at all. Consolidating a queue onto infrastructure you already run is a legitimate win, and this is the case Confluent makes fairly.

Look elsewhere if you need priority, delayed delivery, or dead-lettering today; you are considering adopting Kafka specifically to get a queue, which means taking on a partitioned distributed log and its operational weight to solve a problem that queues solved decades ago; your consumers are not all Java; or your throughput does not remotely justify the cluster. For a wider survey of the options, see our comparison of Apache Kafka alternatives.

Running Queues on Valkey or Redis Instead

If Valkey or Redis is already in your stack, the same consolidation argument applies with a much lighter footprint — but the native primitives deserve the same scrutiny we just applied to Kafka.

Lists give you a queue in two commands, LPUSH and BRPOP, and no acknowledgment whatsoever: a worker that dies mid-job takes the message with it. Streams are the serious option, adding consumer groups, explicit acknowledgment, and a pending entries list that records a delivery count per message. That is genuine at-least-once delivery.

And Streams stop at almost exactly the same line share groups do. The pending entries list counts deliveries but nothing acts on the count — routing an exhausted message to a dead letter destination is code you write and maintain. Priority, delayed delivery, and deduplication are likewise yours to build. The primitive is solid; the reliability layer above it is missing.

Reliable Queue: The Missing Layer

Redisson PRO's Reliable Queue supplies that layer for Java teams, turning Valkey or Redis into a queue with the features share groups are still working toward. The mapping is close to one-for-one with the gaps above.

RReliableQueue<Order> queue = redisson.getReliableQueue("orders");
queue.setConfigIfAbsent(QueueConfig.defaults()
    .deliveryLimit(5)
    .visibility(Duration.ofSeconds(60))
    .deadLetterQueueName("orders-dlq"));

Message<Order> msg = queue.poll(QueuePollArgs.defaults()
    .acknowledgeMode(AcknowledgeMode.MANUAL));

if (msg != null) {
    if (processOrder(msg.getPayload())) {
        queue.acknowledge(QueueAckArgs.ids(msg.getId()));
    } else {
        queue.negativeAcknowledge(QueueNegativeAckArgs
            .failed(msg.getId())
            .delay(Duration.ofSeconds(10)));
    }
}

The line worth drawing is not "Kafka bad, Redis good." If you run Kafka for streaming and want one fewer system, share groups are a reasonable answer and will get better. If you are contemplating a Kafka cluster to obtain a work queue, that is a large amount of distributed log to operate for a problem a queue solves directly — and if Valkey or Redis is already running, the queue can go there without adding anything new to monitor.

Frequently Asked Questions

Is Kafka a message queue?

Both, now. Kafka remains a commit log at heart, but Apache Kafka 4.2 shipped share groups under KIP-932, which add genuine queue behaviour: the broker locks each record for one consumer at a time, counts how often it has been delivered, and lets consumers accept or reject individual records instead of committing a single offset. The features still absent are the ones job-processing systems tend to assume — dead-lettering, priority, and scheduled delivery — which is why whether Kafka can do queues and whether it can replace your queue are two different questions.

What is a share group in Kafka?

A share group is a consumption model in which several consumers cooperatively read the same partitions with no fixed partition-to-consumer assignment. The broker places a time-limited acquisition lock on each record, defaulting to 30 seconds, and the consumer can accept, release, reject, or renew it. This removes the partition count as a ceiling on consumer parallelism.

Does Kafka have a dead letter queue?

Not for share groups, not in 4.2. A record that runs out of delivery attempts, or that a consumer rejects outright, is flagged archived and then dropped — nothing copies it elsewhere first. The fix is KIP-1191, voted through in January 2026 and slated for 4.4. Note that the dead-letter support in Kafka Connect and Kafka Streams is a separate, client-side mechanism and does nothing for share groups.

What is the difference between a Kafka topic and a queue?

A topic is where records are stored; a share group is one way of consuming them. Kafka has no queue object, and acknowledging a record does not delete it — retention remains a topic-level setting based on time or size. In a conventional queue an acknowledged message is removed immediately.

Does Kafka support message priority?

No. Share groups have no notion of priority, and it does not appear on the published roadmap. Prioritising work in Kafka means separate topics per priority level and application logic to decide which to drain first.

Can Kafka replace RabbitMQ?

For straightforward task distribution, share groups now cover much of what teams use RabbitMQ for. RabbitMQ still leads on routing flexibility through exchanges and on failure handling: dead-lettering is mature, and version 4.3 added strict priority to quorum queues along with native delayed retries. If you already operate Kafka and your routing needs are simple, consolidating is reasonable; if you depend on RabbitMQ routing, priority, or scheduling, share groups are not a replacement yet.

For the wider picture, see our head-to-head on Redis vs Kafka and the full walkthrough of Reliable Queue for Valkey and Redis. Definitions of the underlying concepts live in our glossary entries on the message queue and the message broker.