What Is Latency vs. Throughput?

System architects, engineers, and developers alike use a wide range of metrics to measure the performance of enterprise applications. While everyone's perception of an app's responsiveness differs, the numbers don't lie. But as important as these figures are, some performance metrics are often conflated. For example, "latency" and "throughput" are often used interchangeably.

It's easy to understand the confusion about latency and throughput, because if you have an issue with either, the end user often experiences the same thing: an app that seems slow and unresponsive. This is especially true in distributed enterprise applications, which generally need low latency and high throughput to maintain strong performance and responsiveness across nodes.

You can look at a slow-loading app and say, "The latency is really slowing things down," or "Throughput is severely affected right now." But which statement is correct, and how do you know? To accurately pinpoint the problem and fix it, you need to know the difference between the two. This guide to latency and throughput provides definitions, guidance on the differences, and steps to ensure they don't slow down your apps.

Latency vs. Throughput: The Highway Analogy

First, let's define these two terms:

Latency is the time it takes for a system to respond to a user request. It is typically measured in milliseconds or microseconds, in the context of a single operation (rather than a batch of operations).
Throughput is the number of operations a system completes per unit of time. It is commonly expressed as requests per second (RPS), operations per second (OPS), or queries per second (QPS).

Latency and throughput are like two sides of the same coin because they both measure how many operations a system can complete in a given amount of time. However, if you have a problem with either one, the cause will be quite different. A good way to understand the difference is to think of cars on a highway. Latency measures how long it takes a single car to drive the length of the highway, while throughput measures how many cars complete the drive in a given amount of time.

What About Bandwidth?

If you're thinking that throughput sounds a lot like bandwidth, that's because these two terms are often used interchangeably, which can create even more confusion. To complete the highway analogy, think of bandwidth as the highway's maximum capacity (how many cars it can hold), while throughput is the measure of how many cars actually complete a drive on that highway in a given amount of time.

Whether for a highway or a computer system, the theoretical maximum capacity is a useful number, but actual throughput is more important because it measures real-world performance.

The Latency vs. Throughput Trade-Off

We previously established that latency and throughput, while separate, are two sides of the same coin. In other words, they're different, but related. Little's Law is a mathematical formula that defines the relationship.

Developed by MIT professor John Little in 1954, Little's Law states that concurrency equals throughput multiplied by latency (L = λW). Computer systems were the farthest thing from Little's mind, as he formulated the law based on the physical queues he saw all around him, from grocery store lines to factory assembly lines. Yet, Little's Law has proven remarkably relevant in the digital age, as it describes a fundamental trade-off that system architects routinely make.

According to Little's Law, reducing per-request latency often increases achievable throughput, while pushing throughput past the saturation point causes latency to spike as requests queue up. So, in systems, these two things are true:

Batching and higher concurrency raise throughput but can add latency for a request that is currently waiting to be processed.
Optimizing a system for low single-request latency means sacrificing some throughput.

In other words, a service can have excellent throughput and still feel slow, or it can have a low average latency while a significant number of users wait far too long. This is the trade-off you have to make, and the right balance depends on your needs. In many cases, a service-level objective (SLO) features a defined latency target, which in turn sets expectations for throughput.

Tail Latency, or Why Averages Lie

You might counter that your metrics routinely show both low latency and high throughput. More than likely, that's because you're using averages for your latency figures.

Average (mean) latency is one of the most misleading metrics in systems engineering. Averages obscure slow requests and favor fast ones, effectively hiding the unresponsiveness that end users notice.

Consider an endpoint where 99 of 100 requests return in 5 ms and 1 returns in 2,000 ms. The mean is about 25 ms, which looks excellent. However, 1% of your users wait two full seconds. At scale, especially in today's globally distributed systems, this can mean very poor performance for the 1%. This phenomenon is known as tail latency, or unacceptably high response times experienced for a small fraction of requests in a system.

In the real world, tail latency determines whether an app feels responsive, because users remember their worst experiences, not the average ones. Percentiles provide a much more useful metric, as they report the latency below which a given fraction of requests complete within that latency. The standard set of percentiles includes:

P50 (median): Half of the requests finish faster, and the other half finish slower. This is a good representation of the typical experience.
P95: 95% finish faster, and the slowest 5% take longer. This is where tail latency really becomes a problem.
P99: 99% finish faster, 1% take longer. P99 is the standard SLO target for user-facing systems.
P99.9 (aka "three nines"): At normal request volumes, the 0.1% experience is rare, but at higher volumes, it happens constantly.

Fan-Out and Tail Latency Amplification

In 2013, two Google employees published a paper titled "The Tail at Scale", which explains why P99 and P99.9 matter so much in distributed systems: a single user request often fans out to many backend services in parallel, and the response cannot return until the slowest branch completes.

In a real service, individual servers returned a P99 of just 10 ms, yet once a request fanned out across multiple leaf servers, the end-to-end P99 rose to roughly 140 ms — a 14× increase — with the slowest 5% of sub-requests accounting for half of the total. So, while you might be tempted to dismiss a per-service tail, it can quickly become the typical user experience once requests fan out across distributed nodes. This is why system architects strive for P99.9 across their entire infrastructure.

Reducing Tail Latency in Java with Redisson

For Java applications backed by Valkey or Redis, the biggest contributor to latency is the network round trip to the data store. Meanwhile, the dominant contributor to the tail is contention, whether for connections, data, or an overloaded shard. Redisson, the Valkey and Redis client for Java developers, solves all three contention issues, enabling you to deliver a user-facing API with a 50 ms P99 SLO. Here's how:

Near Cache

Redisson's Near Cache, delivered via the RLocalCachedMap object, keeps frequently accessed entries in the application's JVM heap and synchronizes invalidations across nodes over pub/sub. Reads that hit the local cache skip the network entirely, which Redisson reports can boost Valkey or Redis read operations by up to 45×. Because the network hop is removed, near-caching collapses the tail of read-heavy workloads. Here's how to use it:

RLocalCachedMap catalog = redisson.getLocalCachedMap(
    "catalog", LocalCachedMapOptions.defaults());

// Hot reads are served from the JVM — no network hop, no shard contention.
Product p = catalog.get("sku-12345");

Pipelining

When a single API call requires several Valkey or Redis operations, issuing them serially incurs the round-trip time (RTT) per command, and those RTTs compound into a long, jittery tail. Pipelining with Redisson's RBatch aggregates commands into a single network call, so the group incurs roughly one RTT instead of N, lowering latency and raising throughput at once.

RBatch batch = redisson.createBatch(BatchOptions.defaults());
batch.getMap("orders").fastPutAsync(orderId, order);
batch.getMap("inventory").fastPutAsync(sku, qty);
batch.getAtomicLong("orderCount").incrementAndGetAsync();
batch.execute(); // one round-trip for all three commands

Connection Pooling

A request that opens fresh TCP and TLS connections incurs overhead, which, under load, causes threads to stall as they wait for scarce connections to open — a perfect recipe for a tail spike. Redisson maintains connection pools, sized to your concurrency, to keep the tail flat.

Solving Latency and Throughput Problems With Redisson

Using all three of these methods, Redisson targets all sources of tail latency, shrinking the distribution from the slow end inward. With Redisson, you can add distributed caching to a Valkey or Redis Cluster to tune your stack for both tail latency and throughput. To get started, learn more about the features of Redisson and Redisson PRO.

Latency vs. Throughput: Frequently Asked Questions

Here are quick answers to common questions about latency and throughput:

What is the difference between latency and throughput?

Latency is how long a single request takes, a time delay often measured in milliseconds. Meanwhile, throughput is the number of requests a system completes in a given time; this rate is typically expressed as requests completed per second.

What is tail latency?

Tail latency is the slow end of the response-time distribution. Put another way, it's what a small percentage of requests actually experience. It matters because users tend to remember their worst experiences, and the fan-out inherent in distributed systems can turn a rare per-service tail into a common end-to-end problem.

What does P99 latency mean?

P99 latency is the 99th percentile of response times, meaning 99% of requests complete faster than this value, and the slowest 1% take longer. P99 is a common SLO target because it's a more honest representation of the slowest response time than what a latency average can present.

Is bandwidth the same as throughput?

No, bandwidth and throughput are not the same thing. Bandwidth is the maximum capacity a channel can carry, while throughput is the rate actually achieved in that channel. Throughput is always lower than bandwidth once you account for latency, network congestion, and protocol overhead.