Implement a Threshold Alert Engine with Notify, Resolve and a Repeat Interval

Quick Overview

Implement an alert engine that periodically runs each alert's metric query, compares the value with a threshold, and calls a client to notify or resolve, then extend it with a notify repeat interval. Tests per-alert state handling, drift-free scheduling and deterministic testing against a fake client.

Implement a Threshold Alert Engine with Notify, Resolve and a Repeat Interval

Company: Chronosphere

Role: Software Engineer

Category: Software Engineering Fundamentals

Difficulty: medium

Interview Round: Technical Screen

In a phone screen you implement an alert engine and test it with a fake alert client provided for the exercise. Each alert has a query and a threshold. The engine runs each alert's query periodically, compares the metric value the query returns with the threshold, and, depending on the result, calls the client to notify or to resolve the alert. Part 2 adds a repeat interval that controls how often a firing alert is notified. Both parts are expected in one session and time is tight, so get Part 1 working and tested before extending it. The report does not give the exact interfaces. Assume the following, or use whatever the interviewer provides: ```python from dataclasses import dataclass class AlertClient: def query(self, query: str) -> float: ... # current metric value def notify(self, alert_name: str, value: float) -> None: ... def resolve(self, alert_name: str) -> None: ... @dataclass class Alert: name: str query: str threshold: float interval_secs: float # how often to evaluate the alert ``` ### Clarifying Questions - Does an alert fire when the value is above the threshold, at or above it, or is the direction configurable per alert? - While an alert stays above its threshold, should notify be called at every evaluation or only when it first crosses? - Should resolve be called only when a firing alert recovers, and never for an alert that was not firing? - If a query fails or times out, should the alert keep its state, resolve, or report an error? - Does each alert have its own evaluation interval, and may the engine take a clock so that tests can control time? ### Part 1 — Evaluate alerts and notify or resolve Implement the engine: given a client and a list of alerts, evaluate every alert at its interval by running its query, comparing the value with the threshold, and calling notify or resolve. Write test cases that drive the engine with the fake client. ```hint Remember between evaluations Ask what the engine must remember about each alert from one evaluation to the next in order to know whether a resolve is needed. ``` ```hint Testing time without waiting Think about how a test can check behavior that happens every few seconds without actually sleeping. ``` #### What This Part Should Cover - Per-alert state, and exactly which transitions call notify and which call resolve - Scheduling each alert at its own interval without drift, in a form tests can drive deterministically - Tests against the fake client for staying below, crossing, staying above, sitting exactly on the threshold, recovering, and a failing query ### Part 2 — Add a notify repeat interval Each alert now also has a repeat interval for notifications. While an alert stays firing, the engine must not notify at every evaluation: after a notification it notifies again only once the repeat interval has passed. Extend the engine and its tests. ```hint What else to remember Decide which extra fact per alert the new rule needs, and which timestamp you compare it with when an evaluation runs a little late. ``` #### Clarifying Questions for this Part - If an alert resolves and fires again before the repeat interval has passed, should the new firing notify immediately? - When the repeat interval is not a multiple of the evaluation interval, is notifying at the first evaluation after the interval has passed acceptable? - Should a markedly worse metric value bypass the repeat interval? #### What This Part Should Cover - The extra per-alert state and the rule that decides whether to notify again - Edge cases: resolving and then firing again, a repeat interval that is not a multiple of the evaluation interval, a missing or zero repeat interval - Tests that advance a fake clock across the repeat interval, including an evaluation that runs late ### What a Strong Answer Covers - Evaluation logic, scheduling and the client kept separate, so each can be tested on its own - Deterministic tests built on the fake client and an injected clock - Correct transitions: the first notification never skipped, no repeated resolves, no false resolve after an error - One failing or slow alert never stalls or stops the others - Explicit choices wherever the prompt is silent, stated before coding ### Follow-up Questions - With thousands of alerts and slow queries, how would you evaluate concurrently while never running two evaluations of the same alert at once? - The value hovers around the threshold and the alert flaps between firing and resolved. How would you damp it? - The engine restarts. How do you avoid re-notifying every firing alert, or losing a resolve that was due? - How would you support several thresholds with different severities on one alert?

Overview: Implement an alert engine that periodically runs each alert's metric query, compares the value with a threshold, and calls a client to notify or resolve, then extend it with a notify repeat interval. Tests per-alert state handling, drift-free scheduling and deterministic testing against a fake client.

Read the full Chronosphere Software Engineer interview experience this question came from

Chronosphere logo
Chronosphere
Nov 13, 2026
mediumSoftware EngineerTechnical ScreenSoftware Engineering Fundamentals
0
0

In a phone screen you implement an alert engine and test it with a fake alert client provided for the exercise. Each alert has a query and a threshold. The engine runs each alert's query periodically, compares the metric value the query returns with the threshold, and, depending on the result, calls the client to notify or to resolve the alert. Part 2 adds a repeat interval that controls how often a firing alert is notified. Both parts are expected in one session and time is tight, so get Part 1 working and tested before extending it.

The report does not give the exact interfaces. Assume the following, or use whatever the interviewer provides:

from dataclasses import dataclass


class AlertClient:
    def query(self, query: str) -> float: ...            # current metric value
    def notify(self, alert_name: str, value: float) -> None: ...
    def resolve(self, alert_name: str) -> None: ...


@dataclass
class Alert:
    name: str
    query: str
    threshold: float
    interval_secs: float                                 # how often to evaluate the alert

Clarifying Questions Guidance

  • Does an alert fire when the value is above the threshold, at or above it, or is the direction configurable per alert?
  • While an alert stays above its threshold, should notify be called at every evaluation or only when it first crosses?
  • Should resolve be called only when a firing alert recovers, and never for an alert that was not firing?
  • If a query fails or times out, should the alert keep its state, resolve, or report an error?
  • Does each alert have its own evaluation interval, and may the engine take a clock so that tests can control time?

Part 1 — Evaluate alerts and notify or resolve

Implement the engine: given a client and a list of alerts, evaluate every alert at its interval by running its query, comparing the value with the threshold, and calling notify or resolve. Write test cases that drive the engine with the fake client.

What This Part Should Cover Guidance

  • Per-alert state, and exactly which transitions call notify and which call resolve
  • Scheduling each alert at its own interval without drift, in a form tests can drive deterministically
  • Tests against the fake client for staying below, crossing, staying above, sitting exactly on the threshold, recovering, and a failing query

Part 2 — Add a notify repeat interval

Each alert now also has a repeat interval for notifications. While an alert stays firing, the engine must not notify at every evaluation: after a notification it notifies again only once the repeat interval has passed. Extend the engine and its tests.

Clarifying Questions for this Part Guidance

  • If an alert resolves and fires again before the repeat interval has passed, should the new firing notify immediately?
  • When the repeat interval is not a multiple of the evaluation interval, is notifying at the first evaluation after the interval has passed acceptable?
  • Should a markedly worse metric value bypass the repeat interval?

What This Part Should Cover Guidance

  • The extra per-alert state and the rule that decides whether to notify again
  • Edge cases: resolving and then firing again, a repeat interval that is not a multiple of the evaluation interval, a missing or zero repeat interval
  • Tests that advance a fake clock across the repeat interval, including an evaluation that runs late

What a Strong Answer Covers Guidance

  • Evaluation logic, scheduling and the client kept separate, so each can be tested on its own
  • Deterministic tests built on the fake client and an injected clock
  • Correct transitions: the first notification never skipped, no repeated resolves, no false resolve after an error
  • One failing or slow alert never stalls or stops the others
  • Explicit choices wherever the prompt is silent, stated before coding

Follow-up Questions Guidance

  • With thousands of alerts and slow queries, how would you evaluate concurrently while never running two evaluations of the same alert at once?
  • The value hovers around the threshold and the alert flaps between firing and resolved. How would you damp it?
  • The engine restarts. How do you avoid re-notifying every firing alert, or losing a resolve that was due?
  • How would you support several thresholds with different severities on one alert?