Architecture Map

Click a box to see what it does and why it is built that way. Pick a path to step through how a market data message becomes a fill. Blue labels open the posts.

Trading system architecture map: how a market data message becomes a fill Blue text = where it is covered (#N = Trading System Notes #N, Ref = a section of the reference page). Times are orders of magnitude. Black = hot path (market data → order); red = reports coming back (fill → position → strategy); grey dashed = side paths (recording, logging, monitoring, push). Green = hot thread, each pinned to an isolated core; blue = a structure that hands data between threads; yellow = a join point; dashed = side-path threads or processes. Outside Trading server: each hot thread pinned to an isolated core #1 pinning / isolation Decision thread: plain calls, no queues Side paths: never block the hot path Exchange Matching engine Market data: multicast by channel Orders in, reports out Broker system Broker risk checks, then routing Fast tier: tens of µs Standard: up to ms NIC Receives market data Kernel bypass: ~1 µs Kernel stack: 5–10 µs #1 keep IRQs off hot cores Feed handler thread Decode to internal format Check per-channel sequence, fill gaps Ring first, then by stock #5 broadcast ring 1.7 Shard queues SPSC, one per shard Stock → shard by table One stock, one shard Handoff: tens of ns #4 lock-free queue Ref 2.6 cross-core Shard threads ×N Book rebuild → incremental factors Touches only its own stocks Hundreds of ns to µs a message Cache case study: 64 options a tick #14 warm up on the working core Join: bar complete Wait for every shard Cross-sectional rank, z-score Cadence: ms to s Broadcast ring (shm) Feed thread writes once Recorder, monitor read Producer waits for no one #5 SPMC broadcast ring Two cadences Per bar (multi-factor): shards → join → strategy, ms to s. Per tick (MM, arbitrage): signal to order inside the shard thread, in µs. Account state Positions, cash, open orders One writer; others read snapshots #14 double-buffered snapshots Strategy Combine scores, rank Check positions, cash, orders Decide buy, sell, cancel Pre-trade risk Size and position limits, price bands, rate, kill switch Order manager (OMS) Order state machine: sent → partial → filled/cxl Slice big orders over time (TWAP and the like) #3 no hot-path allocation Order gateway Encode the broker/exchange protocol Session, sequence numbers, throttle Orders leave through the NIC Replay and research Recorded data goes back into the same factor code Offline must match live, value for value Recorder Writes all data to files in batches A ring behind = lapped: detect it, alert #5 2.3 deployment traps Monitoring Gaps, backlog, latency tails Rejects, cancels, positions Reads counters, off hot path Async logger Hot threads enqueue one record and move on; a background thread batches the writes #4 queue and batching Trader commands Start/stop, cancel, set params via a service and a queue into the decision thread Rare, but must act fast Client push Orders, positions, algo progress to traders' screens For people: tens of ms is fine Reads snapshots only data write once ← orders orders reports reports: acks, fills, cancels → each hot thread enqueues and moves on commands via a queue order/position changes Times are orders of magnitude to show which stage is worth optimising, not measurements of any one system; see Part 2 of the reference page. The map grows with the series: new components are added as later posts cover them.