Implement a Threshold Alert Engine with Notify, Resolve and a Repeat Interval
Company: Chronosphere
Role: Software Engineer
Category: Software Engineering Fundamentals
Difficulty: medium
Interview Round: Technical Screen
In a phone screen you implement an alert engine and test it with a fake alert client provided for the exercise. Each alert has a query and a threshold. The engine runs each alert's query periodically, compares the metric value the query returns with the threshold, and, depending on the result, calls the client to notify or to resolve the alert. Part 2 adds a repeat interval that controls how often a firing alert is notified. Both parts are expected in one session and time is tight, so get Part 1 working and tested before extending it.
The report does not give the exact interfaces. Assume the following, or use whatever the interviewer provides:
```python
from dataclasses import dataclass
class AlertClient:
def query(self, query: str) -> float: ... # current metric value
def notify(self, alert_name: str, value: float) -> None: ...
def resolve(self, alert_name: str) -> None: ...
@dataclass
class Alert:
name: str
query: str
threshold: float
interval_secs: float # how often to evaluate the alert
```
### Clarifying Questions
- Does an alert fire when the value is above the threshold, at or above it, or is the direction configurable per alert?
- While an alert stays above its threshold, should notify be called at every evaluation or only when it first crosses?
- Should resolve be called only when a firing alert recovers, and never for an alert that was not firing?
- If a query fails or times out, should the alert keep its state, resolve, or report an error?
- Does each alert have its own evaluation interval, and may the engine take a clock so that tests can control time?
### Part 1 — Evaluate alerts and notify or resolve
Implement the engine: given a client and a list of alerts, evaluate every alert at its interval by running its query, comparing the value with the threshold, and calling notify or resolve. Write test cases that drive the engine with the fake client.
```hint Remember between evaluations
Ask what the engine must remember about each alert from one evaluation to the next in order to know whether a resolve is needed.
```
```hint Testing time without waiting
Think about how a test can check behavior that happens every few seconds without actually sleeping.
```
#### What This Part Should Cover
- Per-alert state, and exactly which transitions call notify and which call resolve
- Scheduling each alert at its own interval without drift, in a form tests can drive deterministically
- Tests against the fake client for staying below, crossing, staying above, sitting exactly on the threshold, recovering, and a failing query
### Part 2 — Add a notify repeat interval
Each alert now also has a repeat interval for notifications. While an alert stays firing, the engine must not notify at every evaluation: after a notification it notifies again only once the repeat interval has passed. Extend the engine and its tests.
```hint What else to remember
Decide which extra fact per alert the new rule needs, and which timestamp you compare it with when an evaluation runs a little late.
```
#### Clarifying Questions for this Part
- If an alert resolves and fires again before the repeat interval has passed, should the new firing notify immediately?
- When the repeat interval is not a multiple of the evaluation interval, is notifying at the first evaluation after the interval has passed acceptable?
- Should a markedly worse metric value bypass the repeat interval?
#### What This Part Should Cover
- The extra per-alert state and the rule that decides whether to notify again
- Edge cases: resolving and then firing again, a repeat interval that is not a multiple of the evaluation interval, a missing or zero repeat interval
- Tests that advance a fake clock across the repeat interval, including an evaluation that runs late
### What a Strong Answer Covers
- Evaluation logic, scheduling and the client kept separate, so each can be tested on its own
- Deterministic tests built on the fake client and an injected clock
- Correct transitions: the first notification never skipped, no repeated resolves, no false resolve after an error
- One failing or slow alert never stalls or stops the others
- Explicit choices wherever the prompt is silent, stated before coding
### Follow-up Questions
- With thousands of alerts and slow queries, how would you evaluate concurrently while never running two evaluations of the same alert at once?
- The value hovers around the threshold and the alert flaps between firing and resolved. How would you damp it?
- The engine restarts. How do you avoid re-notifying every firing alert, or losing a resolve that was due?
- How would you support several thresholds with different severities on one alert?
Overview: Implement an alert engine that periodically runs each alert's metric query, compares the value with a threshold, and calls a client to notify or resolve, then extend it with a notify repeat interval. Tests per-alert state handling, drift-free scheduling and deterministic testing against a fake client.
Read the full Chronosphere Software Engineer interview experience this question came from