Posted On: 2019-10-07
Caching is a programming concept that routinely impacts the lives of every software user. Many people (of all levels of technical literacy) are familiar with clearing their web browser cache as a first step in troubleshooting browser issues, yet the use of a cache is by no means limited to the web browsers. Caching is a fundamental programming concept that is used to solve a wide variety of problems at every level of software engineering. As such, I thought I would write a bit about what caching is, and what kinds of problems it can solve, in the hopes that it will help those who are only familiar with its everyday usage.
Like most programming patterns, caching solves a certain category of problem using a particular, well established approach. Specifically, caching is aimed at the problem that occurs when getting a resource is expensive* in some way (time cost, CPU cost, network cost, etc.) but that resource needs to be accessed multiple times. To avoid incurring that cost multiple times, the resource is copied to another, much lower cost location (such as a local copy of a file from the internet) which can then be freely accessed.
As a real-world example of this - imagine seeing a recipe book in the reference section of the library. It contains a recipe that you want to try, but since it is in the reference section you can't borrow the book and take it with you. If you simply go straight home and begin cooking, you may find that you don't remember the exact steps, so you have to go back to the library just to read the next step in the recipe. This is quite inefficient (it takes time to go to the library) and it may outright ruin some recipes (for example, if you are in the library when you need to take something out of the oven.) If you instead make a copy of the recipe and take the copy home with you, then you can easily read the next step right in the kitchen, saving lots of time.
For software developers, it is frequently the case that some form of caching is occurring without their awareness of it. Web browsers will cache the contents of websites (to avoid the network cost of requesting them again,) modern operating systems may cache frequently accessed files on the hard-drive (to avoid the slowness of reading the hard-drive,) and even the CPU itself relies heavily on caching in order to avoid waiting for new instructions or data. Fortunately, most of those caches are implemented in ways that are transparent to the developers: without doing anything, the developers get the performance benefits. Unfortunately, having things "just work" like this can mean that some developers may rely on these caches, perhaps without even knowing they exist.
In addition to all the existing caching systems, it is often the case that programs can benefit from their own caches. If a small number of records are frequently accessed, caching them could lead to many small time savings. Conversely, records with particularly high access costs could benefit from being cached - to make the costs manageable whenever they are used. Caching is often used as a part of other programming patterns as well, such as storing lazy-loaded records to avoid having to load them again*.
Like any pattern, caching has weaknesses and pitfalls. The primary weakness of the cache is the space it takes up. For records that are cached on the hard-drive or in memory, this means that caching more things leads to less space available for other purposes. If implemented incorrectly, this can lead to significant issues (such as a browser cache that never evicts records filling up the hard-drive.) For records in dedicated hardware caches (such as the L1 CPU cache) the size of the cache is quite small, which often means that only a portion of the data is actually in the cache at any time - anything else has to be found in another, slower location (this is commonly referred to as a cache miss.)
Another weakness of caches is accuracy - since the cache contains a copy of data, the contents may become out-of-date if the original data is modified (this is typically called a "stale" cache.) This weakness can be avoided in closed systems by making sure to update (or invalidate) the cache when the underlying data source changes. Unfortunately, it cannot be avoided when the data source can be modified by outside systems (such as a web site.) In such situations it often involves coordinating multiple systems together, some of which may be outside the developer's control. Web browsers are a good example of this: most browsers have a variety of workarounds available (such as requiring re-validation or expiring the cache after a certain time-period) but each workaround has its own weaknesses, so each website is responsible for telling the browser which one to use.
A third weakness of caches is inconsistency. Whether or not a record is in a cache can depend on a variety of outside factors (contention for space, recently executed instructions, etc.) As such, the performance gains from the cache may not always materialize, so systems that rely upon them may suddenly misbehave. Unfortunately, I personally haven't found any good solutions to this issue (troubleshooting one of these issues often boils down to the user trying again and it mysteriously working the second time.) Although I try to avoid these issues by making sure performance is acceptable even on a cache miss/fresh install, the reality of the cache is that there will always be some discrepancy in performance (otherwise the cache wouldn't be worth using.)
In summary, caching copies data from a slow/expensive location into a much faster/cheaper one. In doing so, it trades resource space (memory, disk, etc.) for improved performance. Since it is working with a copy, it runs the risk of becoming "stale", at which point it no longer accurately reflects the original source (typically due to changes in the source.) Caching is present throughout computer systems' architecture, and as programmers we are often benefiting from it, even when we aren't aware of it.
Hopefully this post has helped your understanding of caches and why they are so often used. If you ever find yourself considering using one, hopefully the explanation of their strengths and weaknesses will help you with making that decision. As always, if you have any thoughts or feedback, please let me know.