Design a Log Processing Service

Context

Build a service that ingests application logs from many services and provides low-latency search and analytics.

Functional Requirements

1. Filtering logs by attributes:
   • Filter by service, level (INFO/WARN/ERROR), time, and substring/pattern in the message.
2. Error counts:
   • Return counts of error-level logs over a specified time window.
3. Histograms:
   • Build an hourly histogram for a specific log pattern or pattern ID.

What to Specify

• Ingestion flow (agents, transport, parsing, enrichment, deduplication).
• Storage and indexing (e.g., time-series partitioning, inverted indexes, hot/warm/cold tiers).
• Query API surface:
  • filter(query)
  • countErrors(window)
  • histogramByHour(query, window)
• Handling late and duplicated events.
• Scalability and partitioning strategy.
• Retention policy.
• Correctness vs. latency trade-offs.
• Schema design with example fields.
• Complexity analysis for common queries.