Context Switches, Page Tables and the TLB

A hot thread is pinned to an isolated core and busy-polls, never making a system call, yet its p999 occasionally gains 1–5 µs. perf shows no hot spot, but the TLB row of /proc/interrupts keeps rising for that core: a logging thread in the same process freed a large block, glibc munmaped it back to the kernel, the kernel changed the page tables, and that meant interrupting the hot core. To understand this you need to know what actually changes when the CPU switches threads or processes, and whether page tables and the TLB belong to a process or to a core. This post starts with a single switch, moves on to page tables and the TLB, and ends with why changing a page table interrupts other cores and which operations on the hot path set it off.

1. Context Switches: Where Threads and Processes Differ

“Threads are lighter than processes” is often explained as: a thread switch doesn’t change page tables or flush the TLB, it only swaps a few registers, so it is an order of magnitude faster. The first half is right; the second half is not.

1.1 What a Switch Does: Save, Swap Stacks, Restore

1. Before the switch
   • Already in the kernel: when a thread enters the kernel (a system call, an interrupt, preemption), its user-mode registers have already been saved on its own kernel stack (user and kernel mode, in Chinese).
   • Only inside the kernel: the scheduler is kernel code, so a thread can only be switched out after entering the kernel.
2. context_switch() does two things
   • Switch address space (switch_mm): only if the next task belongs to another process is CR3 changed to that process’s page table (2.1). Threads of one process share one mm, so this step is skipped.
   • Switch execution state (switch_to): save the current task’s few kernel registers and stack pointer, move to the next task’s kernel stack, restore its registers. On the way back to user mode, its user registers are taken back off its kernel stack. Floating-point and vector registers are swapped too; with AVX-512 enabled that alone is over 2 KiB.
3. Where the time goes
   • Direct cost: about 1–5 µs including wakeup and scheduling, and thread and process switches cost about the same.
   • Not in swapping registers: all of this is done on the CPU, with no disk and no waiting on anyone. Most of the time goes into entering and leaving the kernel, waking up, and the scheduler picking the next task.
   • Conclusion: at this step a thread switch and a process switch differ in one place only: whether CR3 is rewritten.

1.2 Warming Up Again After the Switch, and After Moving Cores

1. After switching processes

   • Cold caches: the new process’s data is mostly gone from L1 and L2 (each core has only tens of KiB and a MiB or two), so every access goes to L3 or memory.
   • Cold TLB: it has few TLB entries left, so every new page it touches needs a page-table walk (2.2).

2. Between threads of one process

   • Still warm: the code, shared heap data, and the translations in the TLB.
   • Not necessarily warm: each thread’s own stack and private data.

3. A thread moving to another core

   • Page tables untouched: the new core’s CR3 points at the same page table, the translation rules are the same, nothing needs flushing.
   • But everything is cold: L1, L2 and the TLB are private to each core, so the hot data, instructions and translations the thread built up stay behind on the old core. On the new core, data is fetched again from L3 (from remote memory across sockets), and the TLB is refilled page by page. Pinning (#1) removes exactly this cost.
   • Conclusion: most of a switch’s cost is spread over the execution that follows it, so timing only the switch underestimates it. Threads are lighter because the TLB and caches stay usable after the switch.

4. How to see it

   grep ctxt_switches /proc/<PID>/task/<TID>/status          # voluntary: went to sleep; nonvoluntary: preempted
   perf stat -e context-switches,cpu-migrations -t <TID> -- sleep 10
   

   • ctxt_switches: one per thread in /proc. voluntary rising means a call on the hot path sleeps; nonvoluntary rising means another task is competing for the core.
   • cpu-migrations: how many times the thread was moved to another core; if it rises, the pinning isn’t holding.
   • Hot threads: all of these should stay near 0.

2. Page Tables, CR3 and the TLB: Which Layer Translates Addresses

2.1 One Page Table per Process, One CR3 per Core

1. Page tables
   • What they are: programs use virtual addresses, which must be translated to physical addresses before memory is accessed, and the rules are the process’s page table. On x86-64 it is a 4-level tree with 512 entries per level.
   • One per process: the kernel’s mm_struct, shared by all threads of the process.
2. CR3
   • What it is: a register, one per core, holding the physical address of the root page of the page table that core is using right now.
   • At any moment: several cores’ CR3 can point at the same page table (several threads of one process on different cores) or at different ones (different processes).
   • On a switch: switching to another thread of the same process, CR3 is already right and stays; switching to another process writes that process’s page-table root. From then on, the same virtual address on this core translates to a different physical page, or to nothing at all.

Figure: CR3 and the TLB belong to the core, the page table belongs to the process. Threads of one process walk the same page table, so switching between them leaves CR3 alone; only a process switch rewrites CR3.

2.2 The TLB: a Translation Cache, One per Core

1. Why it exists
   • Walking the table is slow: walking the 4-level page table on every access would mean four extra memory reads before each one.
   • So translations are cached: each core keeps recently used translations in its own TLB (translation lookaside buffer).
2. How big, and how much it covers
   • Two levels: the L1 dTLB has 64–96 entries, the second-level STLB 1500–3000, each entry covering one page.
   • TLB reach: with 4 KiB pages the STLB covers about 6–12 MiB; with 2 MiB huge pages the same number of entries covers 3–6 GiB.
   • More detail: working-set steps and how huge pages help are in section 3.2 of Memory Latency and Bandwidth.
3. What a miss costs
   • Page walk: the hardware starts from CR3 and reads page-table entries level by level. Those entries are ordinary data in memory, the frequently used ones sit in L1, L2 and L3, and the CPU also has a page-walk cache for the upper levels.
   • So only a little more: reading a value from memory takes about 70–90 ns with a TLB hit and about 85–120 ns with a miss, 15–30 ns more (Reference 2.1). Having to read all four levels from memory is rare; when it happens, one access takes about 300–400 ns.
   • Large tables on the hot path: order books, id maps and other randomly accessed tables larger than a few MiB lose this cost when placed on huge pages.

2.3 PCID: a Process Switch Need Not Flush the TLB

1. Without PCID
   • Writing CR3 flushes: every TLB entry was computed from one particular page table; after switching tables the old translations are wrong for the new process, so writing CR3 drops every non-global entry.
   • Global entries stay: kernel mappings are the same in every process, are marked global, and are not flushed.
2. With PCID
   • Tagging: PCID (process-context identifier) tags each entry with its address space. Writing CR3 with the new tag needs no flush: the old process’s entries remain, the new process just can’t see them, and they are usable again when the old one returns.
   • Limited help: Linux has used PCID since 4.14, but each core rotates through only a few tags, and the TLB holds only a few thousand entries in total, so a process that was away for a while usually finds its entries evicted.
3. KPTI depends on it too
   • Why switch at all: on CPUs affected by Meltdown, every kernel entry and exit writes CR3 to switch between the user page table and the full one (user and kernel mode, in Chinese).
   • What PCID saves: without it, every system call would flush the TLB; with it, the entries of the two page tables carry different tags, and switching back and forth needs no flush.

2.4 The Three Cases Side by Side

The differences all come after the switch. Creation works the same way: a thread created with clone shares the mm and copies no page table; fork must copy a page table for the child (processes and threads section 2.2, in Chinese).

3. TLB Shootdown: Changing a Page Table Interrupts Other Cores

3.1 The TLB Has No MESI

1. Data caches and the TLB differ
   • Data caches: they have MESI between them; when one core writes a line, the hardware invalidates the copies in other cores.
   • The TLB: has no such mechanism. When a page-table entry in memory changes, stale translations cached in other cores’ TLBs do not invalidate themselves.
2. How the kernel makes up for it
   • Local core: after changing the entries (munmap, mprotect, page migration, transparent huge page collapsing), it first invalidates its own entries with invlpg.
   • Other cores: it sends an inter-processor interrupt (IPI) to the cores that need to know; each enters its interrupt handler, invalidates the entries, and replies.
   • Wait for every reply: the initiating core cannot continue until all replies are in. Before that, another core could still follow a stale translation and write to a physical page already given back to the system.
   • The name: this round trip is a TLB shootdown.

Figure: data caches invalidate each other in hardware; the TLB relies on the kernel sending interrupts. Every core still holding this page table stops to handle one, and the initiator waits for all their replies; cores that have switched away clean up when they come back.

3. Who pays
   • The initiating core: waits synchronously, for the slowest core to reply.
   • Cores receiving the IPI: interrupted once to enter the kernel, invalidate and reply, about like an interrupt, 1–5 µs, and their caches have been disturbed when they return.
   • Conclusion: neither pinning nor isolcpus can stop it. They control where the scheduler places tasks, while the IPI is a hardware interrupt that never goes through the scheduler (#3 section 2.2).

3.2 Who Is Notified: Cores Whose CR3 Still Points at This Page Table

The IPI goes neither to “the cores running this process” nor to “every core that ever ran it”, but to the cores whose CR3 still points at this page table right now.

1. Why the line is drawn at CR3
   • TLB entries have one source: page walks. A CPU only ever walks the table its own CR3 points at, and speculative execution walks ahead of time too.
   • Cores whose CR3 points at this table: may hold stale entries and may fill new ones from it at any moment, so they must be told immediately.
   • Cores whose CR3 already points elsewhere: can no longer walk this table and hold at most a few stale entries. With PCID those entries carry this process’s tag and survive the process switch. But they only become usable when the core switches back to this process, and switching back always goes through the kernel’s switch_mm.
   • So: the first kind must be interrupted now; the second kind can deal with it when it switches back.
2. What the kernel does
   • The list: each address space keeps a set of cores, mm_cpumask, and a core adds itself when it switches to the process. Before sending IPIs the kernel also checks which address space each core has loaded right now (the per-core loaded_mm) and sends only to the cores still holding this one.
   • The version number: each address space keeps a page-table generation tlb_gen, incremented on every change; each core records which generation it has flushed up to for that address space. When a core switches back it compares the two, and if it is behind it flushes before running, so a core busy with something else never has to be interrupted.
   • Conclusion: a change to user page tables interrupts only the cores still holding that page table; cores that ran it and switched away postpone the cleanup to the moment they return, and do it themselves.
3. Lazy TLB: CR3 still points at it, but no interrupt yet
   • What it is: kernel threads have no user address space of their own, so switching to one keeps CR3 and borrows the previous process’s page table. This is called lazy TLB.
   • Ordinary page-table changes: no IPI. Kernel threads don’t touch user addresses, so stale entries are harmless for now; when the core leaves lazy mode it flushes according to the version number.
   • Freeing page-table pages needs one: CR3 still points at these tables, and speculative execution could follow it into page-table pages already given back, so lazy cores are told immediately in this case.
4. The kernel’s own mappings
   • Which: vmalloc regions and loaded modules, present in every address space.
   • Who is told: every core. The kernel batches these and flushes them together.
5. Hardware broadcast
   • Which CPUs: the ARM64 TLBI … IS instructions, and INVLPGB on newer AMD.
   • How it works: the hardware broadcasts the invalidation to all cores, which don’t take an interrupt.
   • No list: cores are not picked per process, so the list above isn’t needed.

3.3 What Triggers It on the Hot Path

In one process, whenever any thread changes the user page tables, every other running thread is interrupted along with it. Even if the hot thread changes nothing, other threads in the process and kernel background tasks do. Adding a mapping needs no notification: an unmapped address never enters the TLB, so only removing mappings, tightening permissions and replacing physical pages require invalidation.

• Threads in the process: the first three rows come from your own code; find them and move them to startup.
• Kernel background tasks: the last three rows are the kernel’s own work and have to be turned off with boot parameters and sysctl.
• Conclusion: once a hot-path process is running, its address space is read-only: map, touch and lock the memory it needs at startup; leave freeing and tidying until after the close.

3.4 How to See It

watch -d -n 1 'grep TLB /proc/interrupts'                  # one column per core: is the hot core's column rising?
strace -f -e trace=munmap,mprotect,mremap,madvise -p <PID>  # which thread in the process changes mappings

• The TLB row of /proc/interrupts: how many TLB shootdowns each core has received. If the hot core’s column rises, something is changing this process’s page tables.
• strace catches calls: a thread of the process itself is doing it; -f prints the thread id, so you can tell which.
• strace catches nothing: check whether the kernel background tasks (transparent huge pages, NUMA balancing, KSM) are on.
• strace slows the traced process: every system call stops to be recorded, so use it only while investigating, never on a production hot-path process.