Diagnose failures via SSH and large logs evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
A production service is failing. You have only SSH access to a host and the log file(s) are very large. How would you efficiently locate relevant errors or time ranges, and what specific commands, filters, or strategies would you use? If this issue recurs, how would you design a centralized logging and search solution, and what trade-offs would you consider?
Overview: Diagnose failures via SSH and large logs evaluates requirements, scale assumptions, API/data design, architecture, trade-offs, failure modes, and rollout in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Troubleshooting Large Logs Over SSH and Designing Centralized Logging
Context
You are on-call for a production service that is failing. You have SSH access to a Linux host, but the application log files are very large (and may be rotated/compressed). You need to quickly locate relevant errors and determine the problematic time window. If this problem recurs, you should outline a centralized logging/search solution and discuss trade-offs.
Assume:
You can use common CLI tools available on most Linux hosts (e.g., journalctl, grep/awk/sed, less, zgrep, lsof).
Logs may be written to systemd-journald or to files in /var/log or an app directory, with rotation (e.g., .gz files).
Network bandwidth is limited; avoid transferring large files.
Tasks
On-host triage: Describe how you would efficiently find relevant errors/time ranges in very large logs. Specify concrete commands, filters, and strategies (including for rotated/compressed logs, multiline stack traces, and time filtering).
If this issue recurs: Propose a centralized logging and search architecture. Include ingestion, processing, storage, and query/visualization. Discuss trade-offs among common choices (e.g., Elasticsearch/OpenSearch, Loki, ClickHouse, object storage + query engines, managed services), including cost, scale, performance, and operability.
Clarifying Questions to Ask Guidance
Clarify users, core use cases, read/write patterns, scale, latency, availability, and data retention.
State explicit assumptions before making sizing or architecture decisions.
Prioritize the functional path first, then address reliability, security, observability, and rollout.
What a Strong Answer Covers Guidance
A scoped requirements summary with concrete non-goals and success metrics.
API, data model, architecture, consistency, capacity, and operations.
Reasoned trade-offs among simple and scalable designs, including bottlenecks and failure modes.
A validation, monitoring, migration, and launch plan appropriate for the risk level.
Follow-up Questions Guidance
What breaks first at 10x traffic or data volume?
How would you degrade gracefully during dependency failures?
What metrics and alerts would prove the design is healthy after launch?