Low-Level System Design For High-Throughput C++ Services

What's being tested

Interviewers probe your ability to design and reason about low-latency, high-throughput C++ services: efficient use of CPU, memory, and networking; concurrency models that avoid contention; and pragmatic tradeoffs between latency, throughput, and correctness. They want to see that you can turn SLOs (throughput, p99/p999 latency) into concrete architecture, pick appropriate data structures and synchronization, and identify hot-path optimizations and observability hooks.

Core knowledge

Latency vs throughput tradeoff: know that batching increases throughput but raises latency; quantify with Little's Law $L = \lambda W$ and relate queue depth to tail latency.
CPU cost budgeting: a 3GHz core ~3e9 cycles/s; if hot-path needs 300 cycles, max throughput/core ≈ 10M ops/s; use this to dimension cores and sharding.
Cache and false sharing: align hot objects to cache-line boundaries (typically 64B); pad per-thread counters to avoid cross-core invalidations.
Lock-free / wait-free primitives: use std::atomic with appropriate memory_order_* semantics; prefer SPSC/SPSC ring buffers (Disruptor-style) for handoff to avoid blocking syscall overhead.
Work partitioning: prefer sharding by key (affinity) to make processors own state exclusively, eliminating cross-shard locks and enabling linear scalability.
Network I/O patterns: for high rate use recvmmsg/sendmmsg batching or kernel-bypass DPDK/PF_RING; fallback is io_uring for async I/O on modern Linux, epoll for scalable event demux.
Memory allocation: avoid per-message new/delete; use object pools, per-thread arenas, or jemalloc/tcmalloc with per-thread caches to avoid contention and fragmentation.
Polling vs interrupt: busy-polling reduces latency jitter; use adaptive busy-polling only where SLOs demand it, because it consumes CPU.
I/O zero-copy: use scatter/gather syscalls and pinned mmap/huge pages where copying dominates CPU; for NIC-level zero-copy consider DPDK/RDMA when required.
NUMA and thread affinity: bind threads and memory to CPU sockets to avoid remote memory access; measure with numactl and pin via sched_setaffinity.
Profiling toolchain: use perf, bpftools, VTune, gdb, heaptrack, and flamegraphs; measure p50/p95/p99 across end-to-end path, not only function-level.
Failure and correctness: design for graceful degradation (backpressure, drop-in favor of tail control), idempotency for retries, and audit logs for post-mortem. Persist critical events asynchronously with strict ordering if required.

Worked example — "Design a low-latency order ingestion and matching service in C++"

First 30s clarifying questions: target throughput (orders/sec, messages/sec), latency SLOs (p50/p99), loss tolerance, protocol (UDP/TCP), persistence durability (sync/async), and deployment topology (single machine or cluster). Skeleton of a strong answer:

Ingress: high-performance packet reader (recvmmsg/DPDK) with minimal parsing, deserialize in-place.
Demux & validation: per-instrument shard mapping to a single core/thread to avoid locking.
Matching core: in-memory order book per shard, implemented with cache-friendly data structures (arrays + index maps), and risk checks inline.
Acks/persistence: ack to client immediately; persist trades/events asynchronously via batching to disk or append-only log.
Observability and backpressure: track queue depths, latencies, and apply backpressure (reject or throttle) when queues exceed thresholds.

Key tradeoff to flag: synchronous persistence increases durability but multiplies latency—use batched fsync with bounded queueing to keep p99 within SLO. Close by saying: "If more time, I'd prototype the hot-path with microbenchmarks, run perf to find mem-copy hotspots, and present measured CPU/latency envelopes."

A second angle — "Scale a C++ market-data consumer to 1M messages/sec with p99 < 200µs"

Here the workload is read-only and multicast-heavy. Design pivots:

Use recvmsg/recvmmsg with large batch sizes or DPDK to avoid kernel overhead.
Decode messages in-place; employ a preallocated object pool and per-thread caches to avoid allocations.
Maintain per-subscription state on the consuming thread; push updates to consumers via lock-free ring buffers.
Optimize for memory layout (struct-of-arrays vs array-of-structs) depending on access pattern to improve SIMD/prefetch.

Constraints change tradeoffs: since no persistence and fewer consistency needs, accept looser ordering for performance, but implement sequence-number checks and small reassembly buffers to detect and heal gaps.

Common pitfalls

Pitfall: Confusing average latency with tail latency. Averaging hides jitter; designing for p50 but not p99/p999 will fail real workloads. Always instrument and budget for tails.

Pitfall: Reaching for locks first. Saying "we'll lock a shared map" in a high-throughput hot path is tempting but unacceptable; instead show partitioning, lock-striping, or lock-free alternatives and measure their costs.

Pitfall: Omitting observability and test harness. A design that lacks microbenchmarks, synthetic-load tests, and perf-driven measurement is speculation. Propose specific metrics and how you'd load-test the service.

Connections

Interviewers may pivot to distributed consistency (e.g., ordering across shards) or to hardware acceleration (FPGA/NIC offload) once low-level design is solid. They might also ask about fault tolerance patterns (replication, consensus) if the role intersects reliability.