Design a hashtag recommender for News Feed

Context: You’re adding hashtag recommendations alongside posts in a large social app’s News Feed to increase useful engagement without harming core feed health. Design the system end‑to‑end. Be precise and justify trade‑offs. Address the following: - Problem framing: What is the exact prediction target Y and unit of observation (e.g., user–post–hashtag impression within 24h)? Define your positive/negative labels from logs and how you will construct negatives (e.g., downsampled unclicked exposures) while avoiding selection bias. - Candidate generation: List at least three complementary candidate sources (e.g., personalized affinity, content-based from post text/media, trending/recency), and how you’ll cap or diversify candidates to avoid popularity bias. - Features: Specify at least 10 concrete features you would feed into a logistic regression ranker, covering user–hashtag affinity, post–hashtag semantic relevance, temporal/popularity signals, and platform/locale. For each, explain expected sign/shape and how you’ll bucket/normalize it. - Model and training: Explain why a calibrated logistic regression is a good starting point versus deeper models. Detail regularization (L1/L2), handling class imbalance, negative sampling ratio, time-based splits, and leakage prevention (e.g., exclude post-publication features, use pre-impression snapshots). - Calibration and thresholds: How will you check and correct calibration (e.g., Platt/Isotonic) and choose display thresholds per user cohort or surface? How will you set exploration rate for new hashtags? - Offline evaluation: Define primary metrics (e.g., log loss, AUC‑PR), calibration plots, and counterfactual estimation for top‑k ranking (e.g., IPS/propensity weighting) to mitigate position bias from historical data. - Online experimentation: Specify randomization unit, guardrails (feed time, session exits, complaint rate), primary outcomes (hashtag CTR, downstream dwell, creator engagement), novelty‑effect detection, ramp plan, and stopping criteria. - Cold start and freshness: Propose strategies for unseen hashtags/users and concept drift detection; include decay factors and automated retires for stale tags. - Safety and policy: Identify risks (e.g., sensitive or crisis‑related tags, misinformation) and propose real‑time blocks/filters and fairness checks across languages/regions. - Interpretability: Describe how you’d translate key logistic regression coefficients into actionable product insights (e.g., diminishing returns of showing >2 tags, language mismatch penalties).