Troubleshoot Processes That Randomly Crash While Being Created on Linux Cluster Hosts

Workloads are scheduled onto a cluster of Linux servers, and some of their processes crash at random while they are being created. The interviewer has planted one specific hidden cause and plays the role of the system: you say what you would check or which command you would run, and they tell you what you would see. The round tests Linux fundamentals, so be concrete about commands, the files under `/proc` and `/sys` they read, and the error messages you would expect. ### Clarifying Questions - What exactly does a crash look like: an exit code, a signal, an error reported by the scheduler, or a line in the application's own log? - Does it happen on every host or only on some, and at particular times of day? - How are workloads launched: as plain processes, as systemd units, or as containers under a container runtime? - Did anything change on the hosts recently, such as the kernel, system configuration, or newly added jobs? - Is there shell access to affected hosts, and are logs and metrics from the times of the crashes still available? ### Part 1 — Investigate the crashes Describe your investigation step by step. For each area (processes, memory, network, and disk I/O), give the commands you would run and what output would count as a finding. Explain how each answer decides what you look at next. ```hint Correlate before you dig Whether the failures cluster by host, by time of day, or by what else was running at that moment tells you which area to look at first. ``` #### What This Part Should Cover - Characterizing the failure precisely (exit status, signal, error text, kernel log) before collecting data - A concrete set of commands per area and the specific signal to look for in each - Correlating failure times with other activity on the host to narrow the search ### Part 2 — Connect the clue to the crashes As the questioning continues, the interviewer mentions that some cron jobs on these hosts can create tens of thousands of threads. Explain how that could make other workloads fail while they are being created, which limits are involved, what error the failing processes would report, how you would confirm the diagnosis with commands, and how you would fix the problem. ```hint Count what each limit counts Check whether each relevant limit counts processes or threads, and whether it applies to the whole system, to one user, or to one control group. ``` #### What This Part Should Cover - The system-wide, per-user and per-control-group limits on the number of tasks, and how threads count against them - The error returned at creation time, and where it shows up for different runtimes - Commands that confirm the diagnosis by comparing task counts with limits and matching crash times to the cron schedule - A fix for the offending jobs, plus isolation and monitoring that prevent a recurrence ### What a Strong Answer Covers - A hypothesis-driven investigation rather than a recital of every command known - Fluency with standard Linux tools and with the `/proc` and `/sys` interfaces behind them - A correct model of how the kernel accounts for threads and processes - Distinguishing a resource-exhaustion failure at startup from a bug in the workload's own code - A remediation that addresses the root cause, not only a raised limit ### Follow-up Questions - Why can `ps aux | wc -l` report only a few hundred processes while new threads still cannot be created? - The workloads run in containers. Which limit would you expect to be hit first, and how does that change the diagnosis? - How would you alert on this condition before any workload fails? - A Java service on the same hosts reports an out-of-memory error even though plenty of memory is free. How could the same root cause produce that message?