Debug a slow concurrent network server under load

Scenario

You are on-call for a high-throughput network service (e.g., a TCP/HTTP server). Under load, users report:

• Throughput suddenly drops
• p99 latency increases significantly
• CPU may be high or low depending on the incident

The server uses a concurrency model based on an event loop and OS I/O multiplexing (e.g., epoll), and may use a thread pool for some work.

Task

Walk through how you would debug and improve performance. Your interviewer may ask about:

• Connection handling and backlog
• Event loop behavior
• Blocking vs non-blocking I/O
• I/O multiplexing ( select/poll/epoll , edge vs level triggered)
• Where io_uring might help vs where it would not
• Typical bottlenecks: syscalls, context switching, lock contention, head-of-line blocking, buffer management

Output expected

Explain a structured approach:

1. What you would measure/collect first
2. How you would narrow down the bottleneck
3. Concrete fixes and trade-offs
4. How you would validate improvements and avoid regressions