Performance Bottlenecks: Signals, Causes, Fixes

Once perf has found the hot spot, the next step is recognizing which bottleneck it is and how to fix it. This page writes each common bottleneck as a card: the signal is what you see in perf or in the source, the cause is why it is slow, the fix is how to change it, verify is how to confirm the fix, and the mechanism links to the post on this site that explains it.

This material belongs to Intel’s open-source performance-patterns (intel/intel-performance-skills, commit 3e33aec, MIT license). The thresholds and speedups on the cards are the original’s numbers and were not re-measured on my own machines; the serial accumulator is the exception, measured in the perf special, 4.6.

1. From Signal to Bottleneck: A Lookup Table

When perf shows you something, find its card in this table first. The left column is what you see in perf’s output or the source; the right column links to the card.

2. Bottleneck Cards: Signal, Cause, Fix, Verify

Each card’s title is its number and the bottleneck’s name, and its anchor is its English id. Numbers follow the order cards were added and never change once given, and a new card takes the next number and goes at the end; a post that explains a bottleneck’s mechanism carries the same card at its end.

B1 Serial Accumulator

• Signal: one vaddss/vfmadd dominates annotate (samples often land on the instruction after it, so look one back); IPC ≪ 1 while cache misses stay low. That combination is the key.
• Cause: every += waits for the previous result, so the speed is set by the latency of a floating-point add (4–5 cycles) rather than its throughput (about 0.5 cycles), a gap of 8–10 times.
• Fix: split into several independent accumulators. Four in plain C is about 4 times faster; SSE2 vector accumulation 4–8 times; AVX2 vectors with 4 accumulators 8–15 times.
• How many accumulators: $\text{latency} \div \text{throughput} = 4 \div 0.5 = 8$. Four is usually enough; tight loops use 8.
• The tail: the leftover elements that don’t fill a group need multiple accumulators too, or small inputs fall back to serial.
• Precondition: the operation must be associative (add, multiply, min, max). The result shifts by a rounding error; if it must match bit for bit, this fix is out.
• Verify: the output agrees within rounding error, and IPC at least doubles.
• Mechanism: #9 Loop Optimization 3.2; the accumulator-count formula is in #2 3.5; whether the compiler splits it for you is in Memory Latency and Bandwidth 1.5.1; measured in the perf special, 4.6.

B2 Vectors Too Narrow

• Signal: the hot loop is scalar floating point; or it uses xmm on a machine with AVX2; or ymm on a machine with AVX-512 (checks 1 and 2 of the annotate scan).
• Cause: the vector unit issues one instruction per cycle whatever the register width. Each doubling of width roughly doubles throughput; scalar to zmm is at most 8 times faster.
• Rule out three cases first:
  • A loop already limited by memory bandwidth gains far less than 2 times from widening. It only counts as compute-bound if IPC is near or above 1 with few misses.
  • With a serial accumulator, switch to multiple accumulators first (B1 Serial Accumulator); widening alone does nothing.
  • The ymm-to-zmm step: many Intel CPUs lower their clock when running zmm, cancelling part or all of the gain. Check the sustained clock with turbostat before and after; if it drops a lot and the workload is not compute-heavy, compare against the AVX2 version.
• Fix (when intrinsics or assembly already exist): a ten-step “zipper” rewrite:
  • copy the original loop as the narrow fallback;
  • unroll the original loop once and give the second half another set of registers;
  • move the second half’s instructions one by one directly under their counterparts in the first half;
  • merge adjacent pairs of integer adds into one, pairs of narrow vector operations into one wide operation, and adjacent multiplies and adds into FMA;
  • handle loop invariants and accumulators, and finally update the clobber list and the CPUID comment;
  • if any step cannot be done (an illegal swap, registers that cannot be merged), abandon the whole rewrite.
• The tail: AVX-512 can use a mask register for one masked operation instead of a scalar tail.
• No hand-written intrinsics, the compiler just emitted xmm: use target_clones to have the compiler build several versions (see picking the instruction set at run time).
• Verify:
  • floating-point results may differ by 1 ULP (unit in the last place; reordering and FMA both change rounding), and bit-exact reference values must be regenerated with the new code;
  • check signed and unsigned semantics in integer loops;
  • the CPUID guard covers every newly used instruction set;
  • wide registers appear in annotate.
• Mechanism: how wide the registers are, and why only 128 bits without an instruction-set flag, is in #15 Part 1; what blocks vectorization is in #15 2.2.

B3 Missing vzeroupper

• Signal: other_assists.avx_to_sse is non-zero; the first SSE instruction after an AVX function takes a huge number of cycles; a cold start, before any AVX code has run, does not pay it.
• Cause: once ymm or zmm0–15 have been written, the upper halves of the registers are “dirty”. The next legacy SSE instruction has to save and restore the whole register state, which costs hundreds of cycles on some microarchitectures. vzeroupper clears it in about 1 cycle.
• Fix:
  • in intrinsics code, add _mm256_zeroupper() before every exit that may reach SSE: before return, before calling an SSE function, before a callback;
  • at the end of inline assembly, add vzeroupper and put all of ymm0–15 in the clobber list, because it clears the upper half of every register.
• When it is not needed: only zmm16–31 are used, or only xmm. Functions using target("avx2") or target_clones get it inserted by the compiler.
• Verify: other_assists.avx_to_sse drops to zero. If it does not, search objdump -d output for ymm/zmm to find other missed spots.
• Mechanism: #15 3.3 Alignment, Instruction-Set Detection, vzeroupper.

B4 Missing restrict

• Applies to: C only.
• Signal: an overlap check of pointer subtractions, compares and jumps before the loop; the same loop compiled into a vector and a scalar version; or a simple loop with no dependencies and no branches that was not vectorized at all.
• Cause: the compiler cannot assume the output pointer does not overlap the inputs, so it either adds a run-time check (10–20 instructions plus two copies of the loop) or gives up on vectorizing.
• Fix: write read-only inputs as const float * restrict and the output as float * restrict. C++ has no standard restrict, only GCC/Clang’s __restrict__.
• Trap: restrict is a promise to the compiler. If a caller really passes overlapping pointers, that is undefined behavior, not just slower code. Parameters meant to be processed in place must not get it.
• Verify: compare gcc -O2 -S output before and after; the check is gone. If tests fail after adding it, a caller is passing overlapping pointers: remove restrict and fix the caller.
• Mechanism: #10 Part 1, __restrict: why two pointers may point at the same memory is in 1.1; the compiler checking at run time itself is in 1.2.

B5 Test-and-Set Spinlock

• Signal: lock cmpxchg piles up in a tight loop in annotate; spin_lock, cas and similar functions stand out in perf report; throughput falls as threads are added, not merely stops growing.
• Cause: a TAS (test-and-set) lock does a read-modify-write on every attempt. Even a failed cmpxchg must own the cache line exclusively, so every waiter and the holder fight over one line, and under contention the bus traffic is $O(N^2)$. Data sharing the lock’s line gets dragged along.
• Fix: read normally first and CAS only once the lock looks free (TTAS, test-and-test-and-set). While waiting, the line stays in the shared state and the traffic drops to $O(N)$.
• Production form: try one CAS first and enter the read loop only if it fails, so the uncontended case pays no extra read.
• Lock and data on one line: pad the lock onto a line of its own.
• Verify: the sample density on lock cmpxchg drops; the throughput curve over 1/2/4/8 threads no longer falls; HITM on the lock’s line drops in c2c.
• Mechanism: every attempt by a waiting core takes the line, see #16 1.4; reading before grabbing is #16 1.5 TTAS; the simplest CAS loop is #16 1.3.

B6 False Sharing

• Signal: a line above 5% Tot Hitm in c2c, with different offsets in it written by different threads; inconspicuous on 1 core, it shows up only with several; mostly HITMs within one socket.
• Cause: the cache line (usually 64 bytes) is the smallest unit of coherence. When thread A writes field X, thread B’s copy of the whole line is invalidated and has to be read again from L3 or memory.
• Fix in three steps:
  • first pad generously to push the fighting fields onto different lines. This step only proves the diagnosis; don’t worry about space yet;
  • re-measure with c2c: it only counts if that line’s HITM drops below 1% and throughput comes up. If it doesn’t drop, the diagnosis was wrong: either it is true sharing, or other fields are involved;
  • then reorder by access pattern: fields each thread writes often get a line each, fields many threads only read can share a line, rarely touched fields fill the gaps.
• The thorough fix: move thread-private fields into a per-thread struct.
• Verify: compare c2c before and after; guard the struct size with _Static_assert.
• Mechanism: #2 Part 2, False Sharing; how the line’s state goes back and forth when two cores each write their own variable is in the MESI special, Part 6; alignas(64), which gives an object whole lines of its own, is in #6 1.3.

B7 Shared Statistics Counter

• Signal: lock add/inc/xadd on the hot path, with no cmpxchg retry loop; c2c shows the same offset written by many CPUs (true sharing); inconspicuous on 1 core, it shows up only with several.
• Cause: every atomic add must own the cache line exclusively, so the line travels between cores, each time at close to memory latency (about 100–300 ns). Relaxed memory order does not help; the hardware still needs exclusive ownership.
• Fix, one of three in order of preference:
  • increments inside one function’s loop, at no more than 2 hot spots: accumulate in a local variable and add it once with a relaxed atomic per batch or at function exit. A pure compute loop adds once at the end.
  • increments spread over many places: bucket by CPU number. Use an array of 64-byte-aligned stats structs indexed by sched_getcpu() & 63; put all counters in the same struct so sched_getcpu is called once; sum the buckets when reading.
  • a shared library that cannot easily own a global array: use __thread thread-local variables, flushed to the global on thread exit or periodically.
• Still contended after bucketing: check whether flushes are too frequent, for example once per request, and lengthen the interval.
• Verify: the total is only approximate at any moment, so confirm the business can live with that; lock instructions disappear from the hot path; HITM on that line drops in c2c.
• Mechanism: where the cost of a write comes from is in the MESI special, Part 5; sharding a counter per thread and merging afterwards is in the MESI special, Part 6; the counter both cores write is in #4 1.2.

B8 Unmarked Cold Path

• Signal: a hot function calls functions only on rare branches, such as error and logging functions, asserts, one-time initialization, or calls inside if (ret < 0) or a switch’s default.
• What counts as cold: the branch at the call site is taken less than about 0.1% of the time.
• Cause: cold code sits between hot code in source order and takes up instruction cache; the compiler also cannot lay out the branch as “almost never taken”.
• Fix: put [[gnu::cold]] on the called function’s definition (__attribute__((cold)) for older standards and C); on the declaration alone it does nothing. Error functions called from a cold function must be marked too; it does not propagate.
• The caller is what benefits: the cold function itself is no faster; the hot function’s instructions get more compact.
• Verify:
  • first confirm the branch really is cold: Intel’s branchprob.py measures each branch’s actual taken rate with PMU events; with no workload to run, gccbranchprob.py parses GCC’s compile-time static estimate (-fdump-tree-profile_estimate-lineno);
  • then check with -O2 -S that the cold function has moved after the hot function’s ret.
• Mechanism: #7 3.2 Hot/Cold Splitting.

B9 Condition Variable Thundering Herd

• Signal: futex_wake and try_to_wake_up are hot; the context-switch rate grows with the number of threads each broadcast wakes; the CPU is busy but IPC collapses; perf trace -e futex shows lots of FUTEX_WAKE val=INT_MAX.
• How it differs from lock contention: wait times in perf lock stat are normal; the hot symbols are on the wake-up path, not native_queued_spin_lock_slowpath.
• Cause: notify_all wakes N threads, only one of which gets the mutex; the rest make a wasted futex round trip (wake, get scheduled, fail to lock, sleep again). With 160 threads one round is about $N \times 5\ \mu\text{s} \approx 800\ \mu\text{s}$; with 160 cores and only 40 jobs, about 75% of the wake-ups are wasted.
• Fix, chosen by the wake-up structure:
  • the dispatching thread knows how many jobs there are: wake only $\min(\text{threads}, \text{jobs})$ and stop once the jobs are taken;
  • everyone must be woken: tree wake-up, first wake $K \approx \sqrt{N}$ helpers who wake the rest in parallel, cutting the time from $O(N)$ to about $O(\sqrt{N})$;
  • only one can work at a time: relay with notify_one, notifying after releasing the mutex;
  • the condition is just “how many units are left”: switch to a semaphore and skip the lock fight after each wake-up.
• Not for: barrier synchronization where every thread must proceed; use std::barrier or std::latch.
• Verify: stress tests confirm no lost wake-ups and no thread asleep forever; futex WAKE counts and context switches drop. Cases cited in the documentation: precise wake-up 49–84% faster (OpenCV), tree wake-up 3 times faster (RocksDB).
• Mechanism: what one sleep and wake-up of a thread costs is in #16 1.1 Futex.

B10 Mutex Where a Reader-Writer Lock Belongs

• Signal: a mutex guards lookups and state reads, writes happen only on rare branches, and reads are above about 75%; the kernel mutex’s osq_lock is hot; the kernel-time share of a pthread mutex grows with the core count; in perf lock stat the wait time far exceeds the hold time.
• Cause: a mutex makes even a read-only acquisition own the cache line exclusively, so N readers are fully serialized.
• Fix: take a read lock on read paths and a write lock on write paths. A reader that finds it must insert releases the read lock, takes the write lock, and must look again once it has it, because another thread may have inserted in between.
• In the kernel: rwlock_t is a spinlock and must not sleep while held; use rw_semaphore for critical sections that may sleep.
• Not for: writes > 50%; critical sections of only a few instructions; time-sensitive writes (some implementations let readers starve writers forever; writer preference can be set).
• Measurements included in the original (one socket, 32 cores, pure reads):
  • the mutex falls from 1.64 Mops/s with 1 thread to 1.06 with 32;
  • the reader-writer lock reaches 18.2 Mops/s with 16 threads (16.8 times) and falls back to 15.47 with 32, because the reader count itself fights over one cache line;
  • with 32 threads it is still 2.7 times faster with 1% writes, and 1.7 times faster with 5%;
  • the mutex makes about 1.04 million context switches per second; the reader-writer lock made 225 in 3 seconds in total;
  • perf record’s own overhead: 1.1% with 1 thread, 5.5% with 32.
• Mechanism: taking a lock is a read-modify-write that must own the cache line, see the MESI special, Part 5; a thread that cannot get it sleeps in the kernel, see #16 1.1 Futex.

B11 Outdated System Library

• Signal: in perf report the hot symbol’s DSO (dynamic shared object) is a system .so, and the symbol is listed in Intel’s version table (references/library-versions.md).
• Fix: check the installed version; if it is below the table’s minimum, report the gap. No code changes; how to upgrade is the user’s call.
• Checking the version: for most libraries, readlink -f on the file gives the real file name with the full version; OpenSSL’s file name only carries the major version, so use openssl version.
• What the table holds: so far only a few OpenSSL AES-GCM and AES-XTS symbols, minimum 3.3, because the AVX-512 implementation got much faster from 3.3 on. A hot non-AVX-512 aesni_xts_encrypt on a new CPU means even the AVX-512 version is missing.
• New enough and still hot: the bottleneck is inside the function; go back to perf (the annotate scan).

B12 Slow CRC32C

• Signal: crc32b/l/q dominates annotate; there is a single _mm_crc32_u64 accumulator; or a byte-by-byte table implementation. A function name containing crc32c is worth a look.
• Cause: _mm_crc32_u64 has a latency of about 3 cycles and handles 8 bytes at a time, so a single accumulator tops out at about 2.5 GB/s per GHz.
• Fix: use corsix fast-crc32’s fusion algorithm. The vector path (VPCLMULQDQ) and the scalar crc32 instruction use different execution ports, so two streams run at once; the vector path also uses several independent accumulators.
• Three ready-made versions, picked by CPU at run time:
  • AVX-512 + VPCLMULQDQ (Ice Lake, AMD Zen 4 and later), 64–97 GB/s;
  • SSE4.2 + PCLMULQDQ, 15–25 GB/s;
  • plain C, 1–3 GB/s.
• Single-core comparison (the table in the documentation): on Sapphire Rapids, a naive scalar version does about 23 GB/s, Chromium’s AVX-512 version 65 GB/s, fusion 97 GB/s.
• New code: if you need CRC32C, use a ready-made one; don’t write your own.
• Verify: an empty buffer with seed 0 gives 0; crc32c(0, "123456789", 9) == 0xE3069283.
• Mechanism: a single accumulator’s ceiling is set by the instruction’s latency, and only several independent chains keep the execution units busy, see #2 3.5 Turning “Break the Dependency Chain” Into a Formula.

B13 std::sort on Numeric Arrays

• Applies to: std::sort, nth_element, partial_sort, qsort with a numeric comparison, and hand-written quicksort on arrays of float, double, int32/uint32/int64/uint64. Not strings, pointers, or objects without a numeric key.
• Cause: introsort compares one element at a time, every comparison is a branch, and there is no data parallelism.
• Fix: switch to x86-simd-sort (used by NumPy, PyTorch, OpenJDK). The quicksort partition is vectorized with AVX-512 compress instructions, and small subarrays use SIMD sorting networks. 3–8 times faster with AVX-512, 2–4 times with AVX2, picked at run time.
• Traps:
  • it is not stable, and there is no replacement for stable_sort;
  • float arrays that may contain NaN must pass hasnan=true, otherwise the behavior is undefined;
  • qsort turns every NaN into a quiet NaN; argsort silently falls back to std::sort on NaN, with no speedup;
  • it needs C++17.
• Verify: sort a copy with std::sort and compare element by element.

B14 Known Algorithms

• Signal: a function named cosine, hamming, jaccard or similar triggers it, with a vectorized recipe per instruction set.
• Cosine similarity: compute $a \cdot b$, $a \cdot a$ and $b \cdot b$ in one pass, each with 4 independent accumulators (12 ymm in total with AVX2). With normalized inputs it reduces to a dot product; a denominator below $\varepsilon$ returns 0.
• Hamming distance: XOR, then count the ones. The POPCNT baseline takes 8-byte chunks with 4 accumulators, about 4–6 GB/s; AVX2 uses Harley-Seal bit-sliced counting, about 20–30 GB/s; AVX-512 VPOPCNTDQ about 40–60 GB/s. Don’t hand-write popcount.
• Jaccard distance: compute the popcounts of AND and OR in the same pass. When both sides’ popcounts are known, $|A \cup B| = |A| + |B| - |A \cap B|$ removes the OR and halves the work. Two all-zero vectors return similarity 1 by convention.
• In common: all three must handle the tail that doesn’t fill a group; the final division and square root are negligible and need no tuning.
• Mechanism: how to write a dot product with vector instructions and accumulators is in #15 3.2 An AVX2 + FMA Dot Product.

3. Writing New Code: Avoid These Traps from the Start

When writing new code from scratch, follow this list and the bottlenecks of Part 2 are avoided from the start. Each item links to its card.

• Pointers: give non-overlapping pointer parameters restrict (C), and read-only inputs const (B4 Missing restrict).
• Floating-point reductions such as sums: no single accumulator in a hot loop; state in the function’s documentation that a changed summation order shifts the result by a rounding error (B1 Serial Accumulator).
• SIMD:
  • use the widest registers the target machine supports (B2 Vectors Too Narrow);
  • functions that used ymm or zmm0–15 add vzeroupper at their exits (B3 Missing vzeroupper);
  • handle the tail when the length is not a multiple of the vector width;
  • annotate inline assembly with /* CPUID: <feature> */, and add "cc" to the clobbers when flags change.
• Several SIMD versions: when one function has several versions, pick one by CPU at run time (Part 4) rather than hard-coding it with a compile-time #ifdef.
• Multithreading:
  • pad per-thread data apart at 64-byte alignment (B6 False Sharing);
  • one counter per CPU or per thread, no global atomics (B7 Shared Statistics Counter);
  • spinlocks read before they CAS (B5 Test-and-Set Spinlock).
• Rare paths: mark error-handling and rare-path functions [[gnu::cold]] (B8 Unmarked Cold Path).
• Don’t rewrite what exists: use a ready-made CRC32C (B12 Slow CRC32C); write known algorithms by the table (B14 Known Algorithms); sort numbers with x86-simd-sort (B13 std::sort on Numeric Arrays).

4. Letting a Program Pick Its Instruction Set on Each CPU

Prepare several versions of one function, say one each for SSE2, AVX2 and AVX-512, and have the program check at run time which instructions the CPU supports and pick the fastest. The same binary then runs on old machines and still uses wide vectors on new ones. There are two ways:

• Let the compiler generate them (target_clones): suits ordinary loops the compiler can vectorize by itself. Put the attribute on the function and list the tiers to generate (say AVX-512 and AVX2); the compiler builds several versions and one is picked by CPU when the program loads.
  • the last tier is "default" as the fallback; name each tier by feature, since GCC does not accept level names such as x86-64-v3 in this attribute;
  • the function must not be static and must be noinline: if it is inlined everywhere, no dispatch code is generated and the automatic choice silently stops working;
  • it does not upgrade hand-written _mm256_* to _mm512_*.
• Write the dispatch yourself (__builtin_cpu_supports): suits hand-written intrinsics and assembly. When unsure, use this one: the logic is out in the open and can be reviewed.
  • on the first call, check x86-64-v4, v3, v2 from high to low and store the chosen version in a static function pointer; every later call costs only one extra indirect jump. Several threads making the first call at once is harmless, because they all write the same value;
  • put an attribute such as __attribute__((target("avx2,fma"))) on each version’s definition: when the whole file is compiled for a lower -march, this is what lets the compiler emit the higher tier’s instructions for that function.
• The check must cover every instruction set used: miss one and the program dies with an illegal instruction on a CPU that passes the check but lacks that feature. Re-check after every widening or rewrite.
• Other platforms:
  • Clang supports both ways equally;
  • MSVC checks features with IsProcessorFeaturePresent();
  • ARM/AArch64 checks features with getauxval(AT_HWCAP) and has no counterpart to target_clones;
  • shared libraries should prefer writing the dispatch themselves, since the compiler-generated kind has initialization-order problems with some linkers.

An example on this site is #15 3.3: __attribute__((target("avx2,fma"))) turns on AVX2 for one function only, and the CPU is checked once at start-up before that path is taken.