What Is Latency vs. Throughput?

System architects, engineers, and developers alike use a wide range of metrics to measure the performance of enterprise applications. While everyone's perception of an app's responsiveness differs, the numbers don't lie. But as important as these figures are, some performance metrics are often conflated. For example, "latency" and "throughput" are often used interchangeably.

It's easy to understand the confusion about latency and throughput, because if you have an issue with either, the end user often experiences the same thing: an app that seems slow and unresponsive. This is especially true in distributed enterprise applications, which generally need low latency and high throughput to maintain strong performance and responsiveness across nodes.

You can look at a slow-loading app and say, "The latency is really slowing things down," or "Throughput is severely affected right now." But which statement is correct, and how do you know? To accurately pinpoint the problem and fix it, you need to know the difference between the two. This guide to latency and throughput provides definitions, guidance on the differences, and steps to ensure they don't slow down your apps.

Latency vs. Throughput: The Highway Analogy

First, let's define these two terms:

• Latency is the time it takes for a system to respond to a user request. It is typically measured in milliseconds or microseconds, in the context of a single operation (rather than a batch of operations).
• Throughput is the number of operations a system completes per unit of time. It is commonly expressed as requests per second (RPS), operations per second (OPS), or queries per second (QPS).

Latency and throughput are like two sides of the same coin because they both measure how many operations a system can complete in a given amount of time. However, if you have a problem with either one, the cause will be quite different. A good way to understand the difference is to think of cars on a highway. Latency measures how long it takes a single car to drive the length of the highway, while throughput measures how many cars complete the drive in a given amount of time.

What About Bandwidth?

If you're thinking that throughput sounds a lot like bandwidth, that's because these two terms are often used interchangeably, which can create even more confusion. To complete the highway analogy, think of bandwidth as the highway's maximum capacity (how many cars it can hold), while throughput is the measure of how many cars actually complete a drive on that highway in a given amount of time.

Whether for a highway or a computer system, the theoretical maximum capacity is a useful number, but actual throughput is more important because it measures real-world performance.

The Latency vs. Throughput Trade-Off

We previously established that latency and throughput, while separate, are two sides of the same coin. In other words, they're different, but related. Little's Law is a mathematical formula that defines the relationship.

Developed by MIT professor John Little in 1954, Little's Law states that concurrency equals throughput multiplied by latency (L = λW). Computer systems were the farthest thing from Little's mind, as he formulated the law based on the physical queues he saw all around him, from grocery store lines to factory assembly lines. Yet, Little's Law has proven remarkably relevant in the digital age, as it describes a fundamental trade-off that system architects routinely make.

According to Little's Law, reducing per-request latency often increases achievable throughput, while pushing throughput past the saturation point causes latency to spike as requests queue up. So, in systems, these two things are true:

• Batching and higher concurrency raise throughput but can add latency for a request that is currently waiting to be processed.
• Optimizing a system for low single-request latency means sacrificing some throughput.

In other words, a service can have excellent throughput and still feel slow, or it can have a low average latency while a significant number of users wait far too long. This is the trade-off you have to make, and the right balance depends on your needs. In many cases, a service-level objective (SLO) features a defined latency target, which in turn sets expectations for throughput.

Tail Latency, or Why Averages Lie

You might counter that your metrics routinely show both low latency and high throughput. More than likely, that's because you're using averages for your latency figures.

Average (mean) latency is one of the most misleading metrics in systems engineering. Averages obscure slow requests and favor fast ones, effectively hiding the unresponsiveness that end users notice.

Consider an endpoint where 99 of 100 requests return in 5 ms and 1 returns in 2,000 ms. The mean is about 25 ms, which looks excellent. However, 1% of your users wait two full seconds. At scale, especially in today's globally distributed systems, this can mean very poor performance for the 1%. This phenomenon is known as tail latency, or unacceptably high response times experienced for a small fraction of requests in a system.

In the real world, tail latency determines whether an app feels responsive, because users remember their worst experiences, not the average ones. Percentiles provide a much more useful metric, as they report the latency below which a given fraction of requests complete. Read as a distribution rather than as a single number, a typical service looks like this:

The mean of that distribution is roughly 20 ms, which would pass almost any dashboard review. Yet 10,000 requests a day take nearly a fifth of a second, and 1,000 of them take longer than most users will wait before reloading the page. The average is not wrong, exactly — it is answering a question nobody asked.

Fan-Out and Tail Latency Amplification

In 2013, two Google employees published a paper titled "The Tail at Scale", which explains why P99 and P99.9 matter so much in distributed systems: a single user request often fans out to many backend services in parallel, and the response cannot return until the slowest branch completes.

In a real service, individual servers returned a P99 of just 10 ms, yet once a request fanned out across multiple leaf servers, the end-to-end P99 rose to roughly 140 ms — a 14× increase — with the slowest 5% of sub-requests accounting for half of the total. So, while you might be tempted to dismiss a per-service tail, it can quickly become the typical user experience once requests fan out across distributed nodes. This is why system architects strive for P99.9 across their entire infrastructure.

P95 vs. P99: Which Should You Target?

P95 and P99 answer different questions, and teams that track only one often pick the wrong one. P95 describes the system's general health. It moves predictably with load, which makes it a good regression and capacity signal: if P95 climbs after a deploy, something in the common path got slower. P99 describes the system's failure modes. It is dominated by discrete events rather than by average load — a garbage collection pause, an exhausted connection pool, a hot shard, a TLS handshake on a cold connection, or a cache miss that falls through to the database.

A practical consequence is that P99 needs far more samples before it means anything. In a one-minute window carrying 100 requests, exactly one request sits above the 99th percentile, so that "P99" is a single measurement and mostly noise. Either widen the aggregation window, or watch P95 on low-traffic endpoints and reserve P99 for the ones that carry real volume.

The gap between the two is itself diagnostic. When P95 is healthy and P99 is many times larger, the common path is fine and something intermittent is stalling a small number of requests — look at contention, locks, and queueing. When P95 and P99 rise together and stay close, the whole distribution has shifted, and the system is simply undersized for the load it is being given.

How Percentiles Are Actually Computed

Computing an exact percentile means keeping every sample, sorting them, and indexing into the result — accurate, and impractical to do continuously at production volume. Real systems approximate. Fixed-bucket histograms such as HdrHistogram or Prometheus histograms assign each observation to a pre-defined bucket and interpolate within it, so accuracy depends entirely on the bucket layout: a P99 that lands inside your widest bucket is an estimate with a very wide error bar. A T-Digest takes the opposite approach and deliberately keeps its finest resolution at the extremes of the distribution — which is exactly where P99 and P99.9 live.

The rule that catches most teams: you cannot average percentiles. The mean of each shard's P99 is not the fleet's P99, and neither is the maximum. Averaging per-minute P99 values into an hourly figure is wrong for the same reason — a percentile is a property of a distribution, not a quantity that adds. The only correct way to combine them is to merge the underlying distributions and recompute, which is why mergeable sketches matter operationally rather than academically: they let you roll per-shard or per-minute digests up into a fleet-wide or per-day number that still means something. If a dashboard shows a "P99" aggregated across instances, it is worth checking which of those two things it is doing. Redisson exposes per-command latency through client-side metrics, so the client's own contribution to the tail can be measured rather than inferred.

Reducing Tail Latency in Java with Redisson

For Java applications backed by Valkey or Redis, the biggest contributor to latency is the network round trip to the data store. Meanwhile, the dominant contributor to the tail is contention, whether for connections, data, or an overloaded shard. Redisson, the Valkey and Redis client for Java developers, solves all three contention issues, enabling you to deliver a user-facing API with a 50 ms P99 SLO. Here's how:

Near Cache

Redisson's Near Cache, delivered via the RLocalCachedMap object, keeps frequently accessed entries in the application's JVM heap and synchronizes invalidations across nodes over pub/sub. Reads that hit the local cache skip the network entirely, which Redisson reports can boost Valkey or Redis read operations by up to 45×. Because the network hop is removed, near-caching collapses the tail of read-heavy workloads. Here's how to use it:

RLocalCachedMap catalog = redisson.getLocalCachedMap(
    "catalog", LocalCachedMapOptions.defaults());

// Hot reads are served from the JVM — no network hop, no shard contention.
Product p = catalog.get("sku-12345");

Pipelining

When a single API call requires several Valkey or Redis operations, issuing them serially incurs the round-trip time (RTT) per command, and those RTTs compound into a long, jittery tail. Pipelining with Redisson's RBatch aggregates commands into a single network call, so the group incurs roughly one RTT instead of N, lowering latency and raising throughput at once.

RBatch batch = redisson.createBatch(BatchOptions.defaults());
batch.getMap("orders").fastPutAsync(orderId, order);
batch.getMap("inventory").fastPutAsync(sku, qty);
batch.getAtomicLong("orderCount").incrementAndGetAsync();
batch.execute(); // one round-trip for all three commands

Connection Pooling

A request that opens fresh TCP and TLS connections incurs overhead, which, under load, causes threads to stall as they wait for scarce connections to open — a perfect recipe for a tail spike. Redisson maintains connection pools, sized to your concurrency, to keep the tail flat.

Solving Latency and Throughput Problems With Redisson

Using all three of these methods, Redisson targets all sources of tail latency, shrinking the distribution from the slow end inward. With Redisson, you can add distributed caching to a Valkey or Redis Cluster to tune your stack for both tail latency and throughput. To get started, learn more about the features of Redisson and Redisson PRO.

Latency vs. Throughput: Frequently Asked Questions

Here are quick answers to common questions about latency and throughput:

What is the difference between latency and throughput?

Latency is how long a single request takes, a time delay often measured in milliseconds. Meanwhile, throughput is the number of requests a system completes in a given time; this rate is typically expressed as requests completed per second.

What is tail latency?

Tail latency is the slow end of the response-time distribution. Put another way, it's what a small percentage of requests actually experience. It matters because users tend to remember their worst experiences, and the fan-out inherent in distributed systems can turn a rare per-service tail into a common end-to-end problem.

What does P99 latency mean?

P99 latency is the 99th percentile of response times, meaning 99% of requests complete faster than this value, and the slowest 1% take longer. P99 is a common SLO target because it's a more honest representation of the slowest response time than what a latency average can present.

What is the difference between P95 and P99 latency?

P95 is the latency below which 95% of requests complete; P99 is the same figure for 99%. In practice, P95 tracks a system's general health and moves with load, while P99 is dominated by intermittent events such as garbage collection pauses, connection pool exhaustion, or a hot shard. A large gap between the two usually points to contention rather than to insufficient capacity.

What is a good P99 latency?

There is no universal number. A good P99 is whatever your service-level objective says it is, and that target should come from the user experience you are protecting rather than from a benchmark. Internal services that will be fanned out are usually held far tighter than user-facing ones, because every branch's tail is inherited by the request waiting on it. The more useful questions are whether your P99 is stable over time and how far it sits from your P50.

Why is P99 latency higher than average latency?

Because response-time distributions have a long right tail. The average is pulled toward the many fast requests, while P99 reports what the slowest 1% actually experience. A service can show a 20 ms mean while its P99 sits near 200 ms, and reporting only the mean hides precisely the requests users notice.

Can you average percentiles across shards or time windows?

No. The mean of each shard's P99 is not the fleet's P99, and neither is the maximum; averaging per-minute P99 values into an hourly figure is wrong for the same reason. A percentile is a property of a distribution, not a quantity that adds. The only correct way to combine them is to merge the underlying distributions — for example with a mergeable sketch such as a T-Digest — and recompute.

Is bandwidth the same as throughput?

No, bandwidth and throughput are not the same thing. Bandwidth is the maximum capacity a channel can carry, while throughput is the rate actually achieved in that channel. Throughput is always lower than bandwidth once you account for latency, network congestion, and protocol overhead.

Similar articles

• What Is the CAP Theorem?
• What Is Consistent Hashing?