What Is Cache Memory? CPU Cache and L1/L2/L3

Computers deal with massive quantities of data that no single human brain could ever process - and yet, just like humans, computers need to remember things. Also like humans, computers distinguish between different types of memory: "short-term" cache memory, and "long-term" disk storage.

But what is cache memory exactly, how do the L1, L2 and L3 levels of a CPU cache differ, and what does any of it have to do with the caches you build in application code? In this article, we'll cover the definition of cache memory, how CPU cache works, the difference between cache memory and RAM, and how the same principle scales all the way out to a distributed cache.

What is cache memory?

Cache memory is a type of computer memory for temporary storage of important, frequently accessed information. Reading from and writing to cache memory is much faster than other forms of data storage. By storing data in cache memory, you can dramatically speed up time-consuming or high-volume transactions.

When the term refers specifically to the memory built into a processor, it is usually called CPU cache. The two terms describe the same idea at different scopes: cache memory is the general concept, and CPU cache is its implementation in silicon.

CPU cache: L1, L2 and L3

Cache memory sits between the CPU and main memory, which is what allows it to be accessed so quickly. But a modern processor does not have one cache - it has a hierarchy of them, each level larger and slower than the one above it. The pattern is always the same trade-off: the closer memory sits to the execution units, the faster it can be read and the less of it you can afford.

These figures are orders of magnitude rather than exact specifications - they vary by vendor, generation and product line, and the nanosecond values assume a clock speed of roughly 3 GHz - but the ratios between the levels hold. Reading from L3 costs roughly fifteen times what reading from L1 costs, and reading from main memory costs several times more again.

Because L3 is shared across every core on the chip, it also serves a second purpose: it is where cores exchange data with one another. When two threads on different cores work on the same memory, the hardware has to keep their private L1 and L2 copies consistent, which is why writes to memory shared between threads are so much more expensive than writes to memory that only one thread touches.

How does cache memory work?

Before searching through slower forms of memory such as RAM or disk storage, an application first checks the cache to see if the information is already present there. If it is - a cache hit - the application simply reads the data from the cache, saving a significant amount of time. Otherwise, on a cache miss, the application retrieves the data from its original source, and then stores a copy of this data in the cache for possible future access.

Only a small amount of data can be stored in cache memory, due to the cache's limited size. In order to make room for new information, caches need a replacement policy that defines how to remove the least "useful" information in cache memory. For example, when making room for new records, the cache may remove data that is least frequently used (LFU) or least recently used (LRU).

Another challenge when implementing caching is the issue of "stale" data that remains in the cache, while the underlying source data has changed. One possible solution for stale data is a definite expiration deadline that removes or refreshes the entry after a set period - the idea behind Redis TTL. The broader problem of removing entries that are no longer correct is known as cache invalidation.

Cache memory vs. RAM

Cache memory is often confused or conflated with RAM (random access memory), but there's an important distinction between the two concepts.

First, we note that modern computer systems have multiple cascading levels of memory:

• CPU registers, which can be accessed extremely quickly but which store an extremely limited amount of information.
• Cache memory, which we've already discussed.
• Random access memory (RAM), which makes up the bulk of a computer's memory resources. The term "random access" comes from the fact that data anywhere within the RAM can be accessed in roughly the same time, regardless of the data's physical location in memory.
• Disk storage such as hard drives, which are the slowest form of memory. Accessing data stored on disk is orders of magnitude slower than accessing the same data stored in a cache.

In strict terms, cache memory is RAM that has been specially designated to serve as a cache. In practice, however, the term "cache memory" is often distinguished and treated as separate from RAM, due to their differing use cases and performance.

More technically, cache memory is usually implemented with static RAM (SRAM), while system RAM is typically dynamic RAM (DRAM). SRAM is faster and costlier than DRAM, which makes it an ideal choice for cache memory.

From CPU cache to distributed cache

The hierarchy does not stop at the edge of the processor. Extend it outward and the same trade-off keeps repeating: every step away from the CPU buys more capacity at the cost of latency. What changes is the unit - nanoseconds while the data stays on the chip, milliseconds once it leaves the machine.

Two things are worth drawing out of that table. The first is where the widest gap falls: everything inside the machine sits within two orders of magnitude of everything else, and then leaving the machine costs three to four orders of magnitude on its own. The boundary that matters is not L1 against L3, it is memory against network. The second point follows from the first: when a Valkey or Redis lookup takes half a millisecond, almost none of that time is spent in the data store itself. It is network round trip. That is why batching commands into a single round trip matters far more to throughput than shaving work off any individual operation, and why the number of network hops usually dominates the design of a fast system.

One distinction is worth drawing here, because the two are easily confused: a cache and an in-memory database occupy the same row of that table but answer different questions. A cache holds a disposable copy of data that something else owns, so losing it costs latency and nothing more. An in-memory database is itself the system of record, which is why durability and persistence matter for one and not the other.

The same reasoning explains why a client-side or near cache exists at all. A near cache is to a distributed cache what L1 is to RAM: a small, fast copy held as close to the consumer as possible, holding only the hottest entries, and buying its speed at the cost of having to be kept consistent with the authoritative copy. Both solve the same problem, and both inherit the same difficulty - the moment you keep a copy somewhere closer, you own the job of invalidating it.

For how this plays out across multiple application servers rather than a single machine, see distributed caching and Redis caching.

Cache memory use cases

The increased speed and efficiency of cache memory has obvious benefits. Both users and servers can take advantage of caching for websites and web applications.

For example, the user's machine may cache web content such as HTML documents, CSS stylesheets, and JavaScript code. This helps the browser load websites more quickly when revisiting the site or refreshing the page.

Meanwhile, servers make use of cache memory in order to provide content more quickly to users. Caching is especially important for database-heavy websites and applications. If users are repeatedly accessing certain data in the database, the website can improve performance by storing this information in cache memory instead.

Databases that can be queried (e.g. for business intelligence and analytics) also make frequent use of caching. The intermediate results of certain highly complex query operations, which can take a long time to execute, can be stored in cache memory. The performance boost provided by caching enables real-time dashboards and reporting.

Frequently asked questions

What is CPU cache?
CPU cache is a small amount of very fast memory built directly into the processor, holding copies of data and instructions the CPU is likely to need next. It exists because main memory is roughly a hundred times slower to read than the processor's own cache, so without it the CPU would spend most of its time waiting.

What is the difference between L1, L2 and L3 cache?
They are levels of the same hierarchy, ordered by distance from the execution units. L1 is the smallest and fastest and belongs to a single core. L2 is larger and slower, and is usually also per-core. L3 is the largest and slowest of the three and is shared by every core on the chip, which is what makes it the meeting point for data shared between threads.

Is cache memory the same as RAM?
Not in practice. Cache memory is built from SRAM, which is faster and more expensive per byte, while system RAM is DRAM. They also differ in how they are used: cache holds a small working copy of data that lives somewhere else, whereas RAM holds the data itself.

Does more CPU cache make an application run faster?
Only if the application's working set is close to the cache size - once the data a program actively touches fits in cache, adding more brings very little. For most server-side applications the far larger win comes from cutting network round trips and database queries, because those are measured in milliseconds while cache misses are measured in nanoseconds.

How does CPU cache relate to a distributed cache like Valkey or Redis?
They sit at opposite ends of the same hierarchy and follow the same principle: keep frequently used data closer to whatever consumes it. CPU cache does this in nanoseconds within a single chip, managed entirely by hardware. A distributed cache does it in milliseconds across a network, managed by your application, and adds something the CPU cache cannot - one shared copy that many servers can read.

Similar terms

• In-Memory Database
• Cache Invalidation
• Cache Miss
• Distributed Caching
• Distributed Database
• Hibernate second-level cache
• Java LRU cache
• Latency vs. Throughput
• Read-through cache
• Redis Caching
• Redis Client-Side Caching
• Redis TTL
• Spring Cache
• Spring Session
• Write-through and write-behind caching