Performance Profiling And Capacity Planning For C++ Services

What's being tested

Interviewers are checking your ability to discover and quantify performance bottlenecks in a production-quality C++ service, and to translate measurements into capacity/scale decisions. They want to see systematic profiling skills (sampling vs instrumentation), familiarity with CPU/memory/I/O metrics, and the math to convert latency/service-time observations into instance or core counts. At eBay, this maps to reducing tail latency, avoiding perf regressions, and sizing services so business SLAs are met with predictable headroom.

Core knowledge

• Profiling modalities — Know the difference between sampling profilers (low overhead, e.g., perf, statistical; good for hotspots) and instrumentation profilers (precise call counts, higher overhead, e.g., gprof, function wrappers). Use sampling first for production-like behavior.

• Flame graphs & call stacks — Produce call-stack samples (perf record -g) and render flame graphs (Brendan Gregg) to find hot call paths; they show CPU proportion per stack, not exact timings.

• CPU counters — Use perf stat or perf record to get instructions, cycles, cache-misses, branch-misses; high cache-misses or stalled-cycles point to memory or branch issues, not just algorithmic slowness.

• Memory profiling — Use Massif, heaptrack, or tcmalloc/jeprof for heap allocation hotspots and ASan/LSan for leaks and use-after-free; watch for allocation churn causing pauses or increased GC-like behavior.

• Concurrency issues — Detect lock contention and thread sleep/waiting via perf top, lockdep-style traces, or thread flame graphs; watch for false sharing (cache-line ping-pong) in tight loops on multi-core systems.

• I/O and syscall analysis — Use strace/sysdig to spot blocking syscalls; correlate with disk or network metrics (iostat, netstat) to separate CPU vs I/O bottlenecks.

• Microbenchmarks vs end-to-end — Microbenchmarks (e.g., google/benchmark) help validate algorithms, but real workloads expose serialization, batching, data-shape effects, and startup/warmup variance; always run end-to-end load tests.

• Little’s Law & utilization — Use Little’s Law: $L = \lambda W$ (concurrency = throughput × latency) and utilization per server/core: $\rho = \lambda \times S$ where $S$ is average service time. Keep $\rho$ < 0.7–0.8 to avoid steep latency degradation.

• Capacity arithmetic — To estimate servers: compute total required processing time per second = $\lambda \times S$; divide by per-server core capacity (cores × core_util_target × 1 second) and round up. Include headroom for bursts.

• Benchmark hygiene — Warm up CPUs, pin threads to cores if testing NUMA, disable turbo/CPU freq scaling for reproducible results, and measure long enough to capture tails (p95/p99).

• Build/toolchain effects — Compile with -g and appropriate -O for realistic behavior; note that disabling optimizations for ease of debugging can drastically change hot paths and inlining behavior.

• Regression detection — Add perf/unit tests and CI checks that capture representative P50/P95/P99 and allocations; track metric deltas and set meaningful thresholds rather than single-number asserts.

Worked example (diagnose high p99 CPU latency for "CheckoutService")

Start by framing: confirm SLA (target p99 latency), workload (requests/sec, payload size), and whether problem is new or after a change. Pillars: (1) reproduce under controlled load to capture CPU/memory counters; (2) collect sampled CPU stacks (perf record -g) and produce flame graphs to spot hot functions; (3) inspect perf stat for cycles, instructions, cache-misses and branches; (4) check thread-state and lock contention (thread flame graphs or pthread mutex stats). A specific tradeoff: if flame graph implicates a library call, decide between optimizing the call path (complex, high payoff) or adding caching/batching (simpler, may shift load). Close by proposing follow-ups: add microbenchmarks for the hot function, CI perf tests, and a canary deployment to validate fixes under production traffic.

A second angle (capacity estimate for "SearchIndexService")

This framing focuses on capacity math: given observed mean service time $S=12ms$ and sustained throughput $\lambda=2,000$ RPS, compute concurrency $L = \lambda \times S = 2000 \times 0.012 = 24$ concurrent requests. If each core can sustainably serve ~1000 RPS at target utilization 0.7, then per-core service capacity $S_{core} = 0.7/0.012 \approx 58$ RPS; required cores ≈ ceil(2000 / 58) = 35 cores. Discuss queuing tail effects: as utilization approaches 1.0, p99 balloons; therefore include safety margin (20–30%) and validate with a production-like load test. This shows the same measurement→math→validation loop but under capacity-planning constraints.

Common pitfalls

Pitfall: Assuming microbenchmark results generalize. Microbenchmarks often optimize away memory layout, inlining, and branch behavior found in production; root conclusions in end-to-end tests.

Pitfall: Ignoring tail metrics. Focusing only on p50 or average latency will miss issues that appear at p95/p99 under high utilization; capacity math must target tail SLOs, not means.

Pitfall: Over-attributing to code. Blaming application code without checking OS-level or hardware signals (CPU-steal, IRQs, NUMA) leads to wasted refactors; always correlate app traces with host metrics.

Connections

Profiling and capacity planning commonly pivot to distributed tracing (e.g., Jaeger) to follow latency across services, and to CI/perf regression tooling to block PRs that regress p95/p99. Interviewers may also ask about memory safety (ASan) or lock-free data structures when concurrency shows up in profiles.

Further reading

• Flame Graphs — Brendan Gregg — practical guide to call-stack sampling and visualization.

• Systems Performance — Brendan Gregg (book) — deep reference on OS and application performance interactions.