malloc and free are not slow. Call them a thousand times in a tight loop and the average looks perfectly fine. The problem HFT code has with them isn’t the mean — it’s the tail, and the tail depends on state you don’t control. glibc hands out per-thread arenas to avoid lock contention, but once there are more threads than arenas, two threads fight over one arena’s lock. A request the current arena can’t satisfy may mean a fresh brk or mmap call into the kernel. Freed memory goes into size-class bins that have to be walked and possibly coalesced, and how long that takes depends on the fragmentation history of everything the heap has seen. None of these costs is fixed; all of them depend on “what has this process been doing so far” — exactly the question a hot path can’t afford to ask.
Everything in this post is a variation on one fix: find the operation whose cost depends on first touch, contention or fragmentation, and move it to a point in the program’s life where you don’t care how long it takes — usually startup. Section 1 moves the allocation itself. Section 2 moves the kernel’s decision about whether your pages may leave RAM, and — less obviously — whether they are backed by physical memory at all. Sections 3 and 4 generalize the move to arbitrary STL containers and disagree about where the resulting flexibility should be paid for: at compile time or at run time. Section 5 folds the useful parts into two reference implementations.
1. The Object Pool: Trading a Search for a Pointer
The fix for “I don’t want to call malloc on the hot path” is almost embarrassingly direct: allocate everything you will ever need once, at startup, and hand pieces of that pre-allocated block out and back on demand. That’s an object pool — a fixed number of instances of one type T, held in memory the program already owns.
1.1 The Naive Version Has the Same Disease in Miniature
struct ObjectBlock {
T object_;
bool is_free_ = true;
};
std::vector<ObjectBlock> store_;
size_t next_free_index_ = 0;
allocate() placement-constructs into store_[next_free_index_], marks it taken, and then calls updateNextFreeIndex() to find the next free slot for next time — by scanning forward from the current position, wrapping around at the end. In an access pattern close to FIFO, the very next slot is usually free and the scan stops almost immediately. But nothing guarantees that pattern. Hold objects out of order, release them out of order, and the scan can walk the entire pool before finding a hole — worst case O(n), and the worst case is exactly as unpredictable as the fragmentation-dependent cost this pool was supposed to replace. The kernel round-trip got eliminated; a smaller, same-shaped problem grew back inside the pool.
1.2 The Fix: Don’t Search, Maintain
T* objects_;
std::size_t* free_list_;
std::size_t next_free_index_; // head of the free list
static constexpr std::size_t kInvalidIndex = -1; // list terminator
free_list_ is a linked list of free slots — but implemented as array indices instead of pointers. free_list_[i] doesn’t record whether slot i is free; it records which slot comes next in the free list, if i is currently the head. next_free_index_ is that head. allocate() takes whatever slot next_free_index_ currently points to and moves the head to free_list_[that slot] — one read, one write, no scan. deallocate() pushes the freed slot back onto the head the same way. Both operations are exactly the same handful of instructions every single time, because the free list is maintained incrementally on every call — by the time you call allocate(), “what’s the next free slot” is already sitting in a variable, not something you go looking for.
That’s worth being precise about, because it’s a common mislabel: this is not an intrusive linked list. An intrusive list stores its next pointer inside the payload type itself — the object doubles as its own node. Here, free_list_ is a completely separate array running in parallel with objects_; a T has no idea it’s linked into anything. The genuinely intrusive version of this pool would union the “next free index” into the same memory a not-yet-constructed T would otherwise occupy, saving the second array at the cost of a less obvious ownership story — a real memory-vs-simplicity tradeoff the reference implementation opted out of.
One layout detail worth keeping: ObjectBlock packs T and is_free_ into a single struct rather than two parallel arrays. A single allocate() call touches both fields together, so keeping them adjacent means one cache-line fetch does the job instead of two independent ones landing on unrelated lines.
2. Locked Isn’t Enough: The mmap Trap and the TLB Shootdown Tax
The pool solves “don’t ask the allocator for memory on the hot path.” It does nothing about a second, easy-to-miss failure mode: the memory you already have can still be taken away from you, or can still cost you a surprise the first time you touch it.
2.1 mlockall Alone Has a Hole in It
mlockall(MCL_CURRENT | MCL_FUTURE) tells the kernel: every page this process currently holds, and every page it will ever hold, stays resident — never swapped out. That sounds complete. It isn’t, because of one detail: munmap doesn’t unlock a page, it tears down the entire mapping the lock was attached to. glibc routes any allocation above M_MMAP_THRESHOLD (128 KB by default) through mmap instead of the heap, and free()-ing that memory calls munmap on it immediately — silently discarding whatever mlockall had guaranteed, the moment that allocation is released.
Closing that hole takes three separate mallopt calls, each blocking a different path back to the kernel:
M_MMAP_MAX = 0— forbid the mmap path entirely. Every allocation, regardless of size, goes through the heap;free()becomes pure bookkeeping and never callsmunmap.M_TRIM_THRESHOLD = -1— the same idea applied to the heap itself. By default, glibc shrinks the heap back toward the OS viasbrkonce enough contiguous free space accumulates at the top; this disables that, so a large chunk of freed space at the top of the heap is never handed back.M_ARENA_MAX = 1— a different axis: multiple worker arenas are themselves backed by additionalmmap-obtained regions. Forcing a single arena (allocation only happens at startup anyway, so the concurrency benefit of multiple arenas is moot here) removes those extra mmap sources up front.
2.2 The Tax isolcpus Can’t Hide You From: TLB Shootdowns
Any page-table change — munmap, mprotect, or the kernel’s own transparent-huge-page background compaction — forces a TLB shootdown: the core making the change doesn’t know which other cores have cached the now-stale translation, so it broadcasts an inter-processor interrupt (IPI) to every core currently running the same process’s address space. Each one has to stop, trap into the kernel, invalidate the affected TLB entry, and resume — microsecond-scale, and it scales with how many cores share that address space.
The sharp, easy-to-get-wrong point: isolcpus does not protect against this. isolcpus operates at the scheduler layer — it controls whether the scheduler will place tasks on a given core. A TLB shootdown is a hardware interrupt, delivered based on which cores are recorded in the kernel’s mm_cpumask for that address space — a completely different mechanism that the scheduler has no say over. A perfectly isolated, pinned hot-core thread can still get interrupted mid-flight by a cold thread in the same process calling munmap or mprotect, because both threads share the same mm_struct, and the shootdown targets the address space, not the scheduling class. This is the deeper reason THP and automatic NUMA balancing get disabled on tuned systems, beyond “unpredictable latency from background compaction” — both routinely rewrite page tables, and every rewrite is a shootdown that reaches cores the scheduler was told never to touch.
3. An Arena Behind the STL: A Pointer That Only Moves Forward
The object pool only ever hands out one type, T. Real hot-path code wants to use std::vector, std::string, ordinary STL containers — without those containers ever touching the global heap. That means writing something that satisfies the C++ allocator interface, backed by memory the program already owns.
3.1 A Pointer That Only Ever Moves Forward
struct MemoryBlock {
size_t used, capacity;
bool is_active;
// ...
};
An arena allocates from a pre-reserved block by tracking one number: how many bytes of this block are already spoken for. allocate(size) rounds the current write position up to the requested alignment, checks whether used + size still fits in capacity, and if so simply advances used and returns the old position — no search, because there’s nothing to search: the “next free position” is always exactly wherever the pointer currently sits. When a block fills up, the arena activates the next pre-reserved block the same way. This is strictly cheaper than section 1’s free list — not even a linked-list pointer swap, just an add — and the price is that there is no way to free a single allocation out of the middle. Nothing records where any individual allocation started or ended, so there’s nothing to look up even if you wanted to release one. The only reset operation is global: rewind every block’s used back to zero and start over. That’s the right tradeoff for a batch of objects with identical lifetimes — everything allocated while processing one tick, thrown away together once the tick is done — and the wrong one for objects with staggered lifetimes, which still belong in section 1’s pool.
3.2 Pre-Faulting: Closing the Gap mlock Leaves at First Touch
The arena’s backing memory is obtained once, at construction, with posix_memalign and mlock() — this time locking just the arena’s own region, a more surgical complement to section 2’s process-wide mlockall. But a freshly obtained virtual address range isn’t backed by physical memory yet: Linux maps pages lazily, and the actual physical page only gets assigned on the first write, via a page fault that traps into the kernel. mlock guarantees a page won’t be evicted once it’s backed — it says nothing about whether the mapping has been established yet. The constructor closes that gap with one line: memset(raw_memory, 0, total_size), touching every page once, at startup, so every fault that was ever going to happen already has. By the time the hot path runs, every page in the arena is not just locked, but already mapped — nothing left to discover for the first time.
The allocator that adapts this arena to the STL interface (allocate/deallocate/construct/destroy/rebind) is mostly boilerplate worth knowing exists rather than dwelling on — with one exception: deallocate() is a deliberate no-op, for the same reason reset() is the only way to reclaim memory. Construction and destruction of individual objects still happen normally through construct/destroy; only the memory side of the contract is neutered.
4. std::pmr: The Same Answer, Paid for at Runtime
4.1 Moving the Allocator Out of the Type
Section 3’s custom allocator has a sharp edge nothing in the type system warns you about: std::vector<int> and std::vector<int, LowLatencyAllocator<int>> are unrelated types. The allocator is a template parameter — part of the container’s type, not a runtime setting — so a function written to take std::vector<int>& will reject the arena-backed version outright, and every function that needs to accept either has to be templated on the allocator too.
std::pmr (C++17) solves the same “let the STL use my allocation strategy” problem the opposite way: move the choice of strategy out of the type and into a runtime pointer. std::pmr::memory_resource is an abstract base class — do_allocate, do_deallocate, do_is_equal — and any concrete allocation strategy is expressed by subclassing it. Every std::pmr container uses the single, uniform std::pmr::polymorphic_allocator<T>, which holds nothing but a memory_resource*; because that pointer’s concrete target isn’t part of the type, std::pmr::vector<int> is always the same type regardless of which resource backs a given instance — free to pass around, assign, return, no template gymnastics required. The price is paid on every allocation: a virtual call through that pointer, dispatched at runtime, instead of section 3’s version, where the concrete allocator type is known at compile time and can be inlined away entirely. It’s the same static-vs-dynamic-polymorphism tradeoff C++ programmers already navigate with virtual functions vs. templates, applied here to memory itself.
4.2 The Standard Resources, Mapped to Sections 1 and 3
The standard library ships a few resources that map directly onto what’s already been covered:
monotonic_buffer_resourceis section 3’s arena, standardized: allocate forward only, release everything together, no individual reclamation. It even accepts an initial caller-supplied buffer (a stackstd::array, say) with a fallback resource for overflow — the standard-library version of theallow_fallback_parameter.unsynchronized_pool_resourceis closer to section 1’s pool, generalized to many object sizes instead of one fixed type, managed as size-class “slabs.”synchronized_pool_resourceis the thread-safe version of the above — and the name says exactly what that costs: a lock, the same class of contention this entire post exists to avoid.
4.3 The Lifetime Trap
std::pmr::vector looks exactly like an ordinary value type — return one by value and it seems as safe as returning any other std::vector. It isn’t, and the reason is specific: a polymorphic_allocator only points at its memory_resource; it never owns it. If that resource is a local variable, returning the vector does nothing to extend the resource’s lifetime:
auto create_vec() -> std::pmr::vector<int> {
auto resource = PrintingResource{}; // local, destroyed at scope exit
auto vec = std::pmr::vector<int>{&resource}; // vec stores only &resource
return vec; // resource is destroyed here
}
auto vec = create_vec();
vec.emplace_back(1); // undefined behavior
vec itself moves out cleanly — no deep copy, nothing about the return looks wrong. The danger is invisible at the call site precisely because std::pmr::vector<int> advertises the same value semantics as std::vector<int> without actually owning the thing its correctness depends on. The rule this forces is unglamorous but absolute: a memory_resource’s lifetime must outlive every container built on it — which usually means it has to live in a strictly outer scope, never inside the function that merely constructs and returns the container.
5. Putting It Together: Two Reference Implementations
Two shapes from the sections above are worth keeping as complete, reusable references: a fixed-type pool for objects with independent lifetimes, and an arena for a batch of objects that all get thrown away together. std::pmr isn’t a third — it solves a type-compatibility problem, not a performance one, and paying a virtual call on every allocation makes it a worse default than either of these for something that’s actually meant to sit on a hot path. Reach for it only when the flexibility is worth that specific cost.
Zooming out, every section above is really the same question asked again at a different layer: where does this storage actually come from, and who’s managing it?
5.1 A Reference Object Pool
Sections 1 and 2 compose cleanly into one self-contained thing: a fixed-type pool with O(1) alloc/dealloc, backed by memory that’s pre-allocated, pre-faulted, and locked before the hot path ever runs. One refinement beyond either version in section 1: instead of a separate free_list_ array, the “next free slot” index lives inside the same slot as T, so a single allocation touches one cache line instead of two unrelated arrays.
template <typename T>
class ObjectPool {
struct Slot {
alignas(T) unsigned char storage[sizeof(T)]; // T lives here once constructed
std::size_t next_free; // free-list link, co-located with T
};
static constexpr std::size_t kInvalid = static_cast<std::size_t>(-1);
Slot* slots_;
std::size_t capacity_;
std::size_t free_head_ = 0;
public:
explicit ObjectPool(std::size_t capacity) : capacity_(capacity) {
std::size_t bytes = sizeof(Slot) * capacity_;
slots_ = static_cast<Slot*>(std::aligned_alloc(alignof(Slot), bytes));
if (!slots_) throw std::bad_alloc{};
mlock(slots_, bytes); // never swapped out (check the return value in production —
std::memset(slots_, 0, bytes); // needs CAP_IPC_LOCK / RLIMIT_MEMLOCK); memset pre-faults every page now
for (std::size_t i = 0; i < capacity_; ++i)
slots_[i].next_free = (i + 1 < capacity_) ? i + 1 : kInvalid;
}
~ObjectPool() {
munlock(slots_, sizeof(Slot) * capacity_);
std::free(slots_);
}
ObjectPool(const ObjectPool&) = delete;
ObjectPool& operator=(const ObjectPool&) = delete;
template <typename... Args>
T* allocate(Args&&... args) {
if (free_head_ == kInvalid) return nullptr; // pool exhausted — no exception on the hot path
Slot& slot = slots_[free_head_];
free_head_ = slot.next_free; // O(1): the next slot was already known
return ::new (slot.storage) T(std::forward<Args>(args)...);
}
void deallocate(T* obj) {
obj->~T();
auto* slot = reinterpret_cast<Slot*>(
reinterpret_cast<unsigned char*>(obj) - offsetof(Slot, storage));
std::size_t index = slot - slots_;
slot->next_free = free_head_; // push back onto the free-list head
free_head_ = index;
}
};
What each piece contributed, concretely: the next_free-inside-Slot layout and the head-pointer swap are section 1’s free list; aligned_alloc + mlock + the eager memset are section 2’s “never let the kernel take it back, and never fault on first touch.”
5.2 A Reference Arena
The pool above hands out one type, individually, with independent lifetimes. An arena is for the opposite shape: a batch of allocations — possibly different sizes, possibly different types — that all get released together. Same startup treatment as the pool (one allocation, mlock, memset to pre-fault), but the allocation logic itself is simpler still: no free list to maintain at all, just a running offset.
class Arena {
unsigned char* base_;
std::size_t capacity_;
std::size_t used_ = 0;
bool allow_malloc_fallback_;
public:
explicit Arena(std::size_t capacity, bool allow_malloc_fallback = false)
: capacity_(capacity), allow_malloc_fallback_(allow_malloc_fallback) {
base_ = static_cast<unsigned char*>(
std::aligned_alloc(alignof(std::max_align_t), capacity_));
if (!base_) throw std::bad_alloc{};
mlock(base_, capacity_); // same locking + pre-faulting as the pool
std::memset(base_, 0, capacity_); // (check mlock's return value in production)
}
~Arena() {
munlock(base_, capacity_);
std::free(base_);
}
Arena(const Arena&) = delete;
Arena& operator=(const Arena&) = delete;
void* allocate(std::size_t size, std::size_t alignment = alignof(std::max_align_t)) {
std::size_t aligned_used = (used_ + alignment - 1) & ~(alignment - 1);
if (aligned_used + size > capacity_) {
return allow_malloc_fallback_ ? std::malloc(size) : nullptr; // arena exhausted
}
void* ptr = base_ + aligned_used;
used_ = aligned_used + size;
return ptr;
}
void reset() noexcept { used_ = 0; } // the only way to reclaim — bulk, not per-allocation
};
allocate() is the whole bump-allocator idea in four lines: round the current offset up to the requested alignment, check it still fits, advance the offset, return the old one. No search, no list to maintain — the “next free position” is always just used_. reset() is the only reclaim path, matching the arena’s whole premise: it doesn’t track individual allocations, so it can’t release one. One deliberate simplification versus section 3’s multi-block design: a single fixed-capacity buffer rather than a chain of pre-reserved blocks — simpler to get right, at the cost of a hard capacity ceiling instead of the ability to keep growing. allow_malloc_fallback_ is the same escape hatch section 3’s arena had for exactly that ceiling.
Pick the pool when objects have independent lifetimes and get freed one at a time. Pick the arena when a whole batch — everything touched while handling one tick, one order, one request — shares a single lifetime and gets thrown away as a unit.