Introduction

There are a ton of blog posts discussing software development paradigms and best practices and there are a few on hardware. At the same time, I can probably count on one hand the number of articles discussing what every developer should know from a computer architecture perspective. The goal of this article is not to deep dive into CPU Architecture, Operating Systems, etc. I’ll give an overview of some principals that we can loose touch with during our busy days as a developer.

Threads vs Processes

Security and reliability

Before Docker Containers and before VM’s, CPU hardware has one of the oldest tricks in sandboxing: processes and virtual memory. As a developer, you should understand that a process will run within its own isolated memory space and can only talk to other processes through specific mechanisms such as Inter-Process Communication (IPC). Thus, if you have process A and process B and they both have a logical address 0x0, they actually point to two separate regions in physical memory. Modern CPU hardware has safe guards through paging that ensures the process can only read and write from its own memory space. The all so common Segmentation Fault is actually originally triggered by CPU hardware via a #PF (Page Fault) due to an illegal access. (Note: I’ll leave out x86 segmentation as segmentation registers are not really “used” anymore in 64-bit mode).

On the other hand, threads within the same process share the same address space. Access to the same address space can be a blessing and a curse. If a thread crashes, the state of the application may be unknown. There are no security protections in hardware preventing rogue threads from accessing other thread’s data in an unintended way. In fact, Chrome moved to running every browser tab in its own process instead of just using multiple threads. If you ever see the “Oh Snap…” in a Chrome tab, that is a process that executed something unintentional. The failed tab crashed and no longer running, but the remaining tabs can continue with business as usual.

Doesn’t this mean processes have to replicate code and data in memory?

Yes and no. Modern operating systems and hardware can do some clever things with something called copy-on-write (COW). When you call fork() in Linux, not everything is copied to a new section of memory space initially. The same paging hardware that ensures processes don’t write to out-of-process memory space can also throw a #PF if a write happens to a page that is marked as read only. This flag allows the operating system to take care of the fault which means copying over the old page to a new page in memory, if needed.

Doesn’t using processes require slow context switches?

Yes, processes require context switches, but so do threads. A thread also has execution context attached to it which consists of it’s various CPU register values (such as EAX, EBX… in x86), among other things which needs to be stored before the next thread can start executing. In fact, modern SIMD code such as many of your video encoding and compression algorithms to watch your favorite Netflix shows in HD shows use some pretty large registers which would need to write to memory. In fact, the latest incarnation of AVX consists of 32 64 byte wide registers! When referring to processes being slower than threads, usually the reference is not just the context switch itself but flushing entries in the Translation Lookaside Buffers (TLBs). TLBs holds cached translations of the paging mechanism we were referring to earlier. Because a new process will execute in its own memory space, it cannot use the old processes’s translations and will start fresh. This means the TLB will be cold for the new process. If a translation is not in the TLB, before the memory access can complete, a Page Miss Handler (PMH) needs to walk the page tables level by level. Page table walks are heavy pointer chasing algorithms and can slow load latency. There are shortcuts to minimize the number of levels required to walk, but the end issue is that a cold TLB can result in load latencies far greater than a warm TLB. This occurs even if the accessed variable is already in a CPU Cache somewhere (which we talk about later)

So the end conclusion is that the context switch for a process may not be a whole lot longer, but there can be lingering effects that slow down even post context.

Too many threads.

Processes are not the only thing that can under go this cold TLB. Threads, if not scheduled on the same logical CPU (CPU Affinity), can also undergo this. Which brings up the next point: There are only a fixed number of logical CPUs that your application can run on. While a modern operating system has many processes and threads that can appear running simultaneously in various blocked and waits states, a CPU can only run a fixed number of threads at a time in reality. This is true regardless if the threads are in the same process or not. As you launch more threads then you can actively run, the operating system has to preemptively context switch. If you just spawn thread after thread for each task thinking it will run all in parallel, you may be surprised that you may be hurting performance more than running a small number of threads in a thread pool.

In general, a large number of threads can cause something called thrashing. Thrashing is a generic term used when the CPU starts swapping or moving resources around more than performing actual execution. A CPU has limited resource sizes, the TLBs, the caches, even the page tables allocated in memory are all limited. As you starting switching between more threads, you can put pressure on these subsystems causing evictions of still hot data, which can be detrimental to performance. Many asynchronous web frameworks are by default configured to match their worker thread pool to the real number of logical cores, or they may add a few extra for blocking, but still within an order of magnitude. Think of Uber drivers. Sometimes it’s easier and quicker to pick up passengers for longer trips and just drive for awhile then to be constantly picking up new passengers and dropping old ones off. The number of passengers is your fixed resources such as logical processors, cache size, and TLBs. If you pick up new passengers, you have to get rid of the old ones first to make room for the new ones. This in and out behavior is thrashing if you start finding yourself waiting for unloading and loading rather than just driving (doing work or execution).