Ranking Metrics And Experimentation

What's being tested

These prompts test whether you can turn an ambiguous “trending” requirement into a reliable, near-real-time ranking system. You should be able to define event contracts, select and combine time-sensitive signals, design the serving architecture, and explain how ranking quality and system health are measured. Figma cares because ranking features must be fast, explainable, resilient to noisy interaction data, and safe to evolve without degrading the user experience.

Core knowledge

• Trending definition should be explicit: recent popularity, velocity, and breadth of engagement relative to comparable content—not simply total clicks. Specify the audience, freshness window, eligible article states, and what “trending” means operationally.

• Event modeling needs immutable identifiers such as article_id, user_id or anonymous session ID, event type, event timestamp, ingestion timestamp, surface, and experiment assignment. Deduplicate with an event ID and avoid counting retries twice.

• Use time decay so old interactions lose influence: $w(\Delta t)=e^{-\lambda\Delta t}$, where $\Delta t$ is event age. A half-life is often easier to reason about: $\lambda=\ln(2)/h$. Short half-lives create responsiveness; long ones provide stability.

• A practical ranking score can combine normalized signals:
  $S(a)=\alpha V(a)+\beta U(a)+\gamma B(a)-\delta R(a)$
  where $V$ is recent velocity, $U$ unique engaged users, $B$ breadth across segments, and $R$ reports, spam, or quality penalties.

• Normalization prevents raw counts from dominating. Log transforms such as $\log(1+x)$ reduce the impact of viral outliers; z-scores or percentile ranks make clicks, saves, comments, and shares comparable.

• Velocity should compare adjacent windows, not just count events: $V=C_{\text{recent}}/(\epsilon+C_{\text{previous}})$. Guard against division by zero, tiny denominators, clock skew, and a single burst from one actor.

• For stream processing, partition events by article_id or a stable hash, aggregate tumbling or hopping windows, and materialize top candidates. Kafka plus a stream processor can support high volume; a simpler scheduled job may be preferable at modest scale.

• Separate candidate generation from final ranking. Maintain a broad set of recently active articles, then apply eligibility, deduplication, penalties, and ranking. This keeps expensive scoring away from the entire corpus and makes latency predictable.

• Serving architecture should use a precomputed top-N result in a low-latency store such as Redis, with periodic refreshes and stale-but-safe fallback data. Do not make every feed request scan events or synchronously query the stream processor.

• Consistency and freshness are explicit tradeoffs. Eventual consistency is usually acceptable for a trending list, while the system should define an upper bound such as “results refresh within 30 seconds” and expose the timestamp of the last successful computation.

• Abuse resistance belongs in the design: cap contribution per user or device, discount repeated actions, detect coordinated bursts, and exclude blocked or reported content. Keep the raw signal and penalty reason available for debugging rather than silently changing scores.

• Evaluation metrics should cover ranking quality and system behavior. Track click-through or open rate, saves, meaningful engagement, hide/report rate, duplicate rate, freshness lag, ranking computation duration, cache hit rate, and p99 serving latency.

Worked example: Define and measure article trending

For “Define and measure article trending,” start by clarifying whether “trending” means fastest-growing, most popular, or most relevant to a particular audience, and ask for freshness and latency targets. Declare assumptions: articles are eligible only after publication, events may arrive late or be retried, and the list should update within roughly one minute. Organize the answer around event definitions, a time-decayed scoring function, candidate-generation and serving architecture, and evaluation plus abuse controls. Define a canonical event schema with an idempotency key, then compute per-article unique engagement and velocity over hopping windows. Explicitly choose between a simple weighted score and a more complex model; a transparent weighted score is easier to debug and launch, while a learned ranker may improve quality but requires stronger data and operational tooling. Explain that raw clicks alone reward old or sensational articles, so use decay, normalization, diversity, and negative signals such as reports. Close by saying that with more time you would validate the score against manually reviewed rankings, add dashboards for freshness and p99 latency, and run a controlled rollout with guardrails.

A second angle

For “Design a trending-articles platform,” the same ranking concept becomes a distributed-systems problem with explicit throughput, latency, fault tolerance, and recovery requirements. Frame the design as ingestion, aggregation, candidate storage, ranking, serving, and observability rather than spending the entire answer on the formula. Discuss partitioning, replaying events, checkpointing, idempotent aggregation, cache invalidation, and behavior when the stream processor or ranking job is unavailable. The key tradeoff is freshness versus cost and stability: frequent recomputation improves responsiveness but increases load and can make the list oscillate. A strong design serves the last known-good result while continuously repairing the pipeline.

Common pitfalls

Pitfall: Treating total clicks as “trending” is analytically weak because older articles accumulate exposure. Define recency, velocity, unique engagement, and negative feedback, then explain how each enters the score.

Pitfall: Jumping directly into Kafka, caches, and databases without clarifying the product contract makes the design feel unfocused. First state the ranking semantics, freshness target, scale assumptions, and acceptable staleness.

Pitfall: Saying “use machine learning” without explaining data, fallback behavior, or debugging depth will not land well for a Software Engineer. Start with an interpretable baseline, identify its failure modes, and describe how the system can later support a more sophisticated ranker.

Connections

The interviewer may pivot to feature flags and experiment instrumentation, idempotent event processing, cache invalidation and stale reads, or backpressure and replay in stream processing. They may also ask how you would debug a sudden drop in engagement by separating ranking regressions from serving latency, missing events, or changes in exposure.