Click And Engagement Prediction

What's being tested

These problems test whether you can build a reliable probabilistic prediction system end to end: define an impression-level label, choose a model and loss, evaluate discrimination and calibration, and deploy it with consistent features. The interviewer is probing judgment around biased feedback loops, delayed or missing labels, feature freshness, online/offline parity, and latency—not merely whether you can train a classifier. Reddit cares because click, comment, and watch predictions directly influence ranking, and a small serving or calibration error can affect millions of feed decisions.

Core knowledge

Label definition must match the serving decision: for click prediction, label $y=1$ when an eligible impression receives a click within a declared window; distinguish “not clicked” from unobserved, filtered, or never-rendered content.
Position bias makes observed clicks non-random: higher-ranked items receive more exposure. Use randomized traffic, propensity weighting, position features cautiously, or counterfactual estimators; otherwise the model learns “shown high” rather than genuine relevance.
A strong baseline is logistic regression: $p(y=1\mid x)=\sigma(w^\top x)$ , trained with binary cross-entropy. Sparse crossed features can work well; LightGBM or XGBoost captures nonlinear interactions, while deep models help with large-scale embeddings and sequences.
Log loss evaluates probabilistic accuracy: $-\frac{1}{N}\sum_i[y_i\log p_i+(1-y_i)\log(1-p_i)]$ . AUC-ROC measures ranking across thresholds, but can look strong while probabilities are poorly calibrated; use PR-AUC for rare positive events.
Calibration matters when scores feed expected-value ranking or downstream budgets. Inspect reliability diagrams and expected calibration error, then apply Platt scaling or isotonic regression on a temporally separate validation set; never calibrate on training predictions.
Use temporal validation rather than random cross-validation when behavior, inventory, or policies change. Train on earlier impressions and validate later; split by user or author when leakage through repeated entities would inflate offline performance.
Negative sampling changes the effective class prior. If training negatives are sampled, correct probabilities using the sampling ratio or calibrate against production traffic; otherwise ranking may remain useful while predicted CTR is systematically wrong.
Feature parity requires the same definitions, transformations, defaults, and time cutoffs offline and online. A point-in-time feature such as recent clicks must exclude events after the impression; online feature stores need freshness, TTLs, and explicit missing-value behavior.
A serving path commonly includes candidate retrieval, feature lookup, model inference, post-processing, and fallback. Set a latency budget such as p99 under 50–100 ms, parallelize independent lookups, cache stable features, version models and features together, and fall back to a simpler model when dependencies fail.
Drift monitoring should cover input distributions, missingness, feature freshness, score distributions, calibration, and delayed label metrics. Compare production slices by device, subreddit, locale, new versus returning users, and content age; alert on both data drift and performance degradation.
For comment or click likelihood, prevent label leakage from post-impression actions, moderation outcomes, future engagement, or aggregate counters computed using the target window. Log the feature timestamp and model version with each prediction so failures are reproducible.
Evaluate beyond aggregate metrics: measure calibration by segment, ranking quality such as NDCG or MRR where appropriate, and operational metrics including timeout rate, CPU/memory, feature-store errors, and fallback frequency. Guard against optimizing clicks at the expense of long-term engagement or content diversity.

Worked example: Build and evaluate click prediction models

A strong candidate first clarifies whether the target is click-through rate for every rendered impression, the prediction horizon, traffic volume, latency budget, and whether the score is used for ranking or expected-value estimation. They would explicitly state that impressions—not page views or sampled clicks—are the unit of training, and that exposure and delayed labels must be handled. The answer can then be organized around label and data construction, baseline/model selection, offline evaluation and calibration, and production deployment and monitoring. They might begin with regularized logistic regression, compare it with LightGBM, and use a time-based holdout with log loss, AUC, PR-AUC, and calibration error. A key tradeoff is that a richer model may improve offline ranking while increasing inference latency and worsening calibration, so the choice depends on the downstream use of the score. They should call out position bias and explain how randomized exposure or propensity weighting would make evaluation more trustworthy. If they had more time, they would add slice-based analysis, shadow deployment, online A/B validation, and drift thresholds tied to rollback procedures.

A second angle

Design a Comment-Likelihood Prediction System with a Focus on Feature and Model Serving

Here the same modeling principles are constrained more heavily by feature serving and model serving than by model comparison. The candidate should frame the online path around user, post, community, and recent-interaction features, with explicit freshness, TTL, fallback, and point-in-time semantics. A useful distinction is between precomputed features that tolerate minutes of staleness and request-time features that require low-latency retrieval. The design should preserve offline/online parity through shared transformations, feature versioning, and replayable prediction logs. Compared with click prediction, comments are typically sparser and more delayed, so cold-start defaults, calibration, and delayed-label monitoring deserve extra emphasis.

Common pitfalls

Pitfall: Treating every non-click as a clean negative and reporting only AUC is tempting but wrong. A stronger answer separates unobserved impressions, exposure bias, class-prior changes from negative sampling, log loss, calibration, and time-based validation.

Pitfall: Jumping directly into Kafka, feature stores, or a neural architecture before clarifying the prediction target makes the response feel unfocused. Start with the unit of prediction, label window, decision latency, and downstream use; then organize the design around those constraints.

Pitfall: Saying “monitor drift” without naming what is monitored is too shallow. Specify feature missingness and freshness, score distribution, delayed log loss/calibration, slice-level metrics, serving p99, timeout rate, and rollback or fallback behavior.

Connections

An interviewer may pivot to learning to rank, counterfactual evaluation, feature-store consistency, or online experimentation. They may also ask about cold-start handling, delayed feedback, multi-objective ranking, and how model predictions interact with exploration and recommendation diversity.

Visa Tools

Visa Sponsors

H-1B Odds Calculator

Visa Bulletin