Choose and compute recommender evaluation metrics

You’re building a restaurant recommender with a logistic regression scorer p(y=1|x) for whether a user engages with a recommendation. A/B tests and user surveys are unavailable for now. 1) Offline evaluation design: Propose a defensible offline protocol to compare two models M0 and M1 without A/B (e.g., counterfactual evaluation with inverse propensity scores from historical logging policy). Specify the exact metric(s) you’ll compute (e.g., PR-AUC, Precision@K, calibrated Brier score), how you’ll avoid leakage, and how you’ll choose K. Name one pitfall of IPS when propensities are small and a mitigation. 2) Thresholded metrics: Suppose on a holdout set of 1,000 recommendations at threshold 0.7 you observe: TP=120, FP=30, TN=820, FN=30. Compute precision, recall, specificity, F1, accuracy. Explain why accuracy can be misleading here and which metric aligns best if the goal is “every shown item should be relevant.” 3) Calibration: Describe how you would check and fix probability calibration (e.g., reliability diagrams, Platt scaling vs. isotonic). Why does good calibration matter when setting business rules like “only show if score ≥ τ”? 4) Model choice: Justify logistic regression over more complex models for this case; name two failure modes (e.g., feature multicollinearity, class imbalance) and concrete fixes. 5) Network effects: If friend activity features introduce interference, what offline split strategy reduces leakage (e.g., time-based, user-disjoint, or graph-clustered splits)? State the trade-offs.