# Daily Temperatures by Town: Linear Models to Predict NYC Temperature You are given a dataset of historical daily temperature readings for New York City (NYC) and `P` nearby towns. Each row corresponds to one calendar day, and the columns are: | Column | Type | Description | |---|---|---| | `NYC` | float | Daily temperature in NYC, in degrees Fahrenheit | | `Town1` ... `TownP` | float | Daily temperature in each of the `P` towns, in degrees Fahrenheit, measured on the same day | All series are aligned by day and there are no missing values. You will write code (any mainstream language/stack; Python with NumPy or pandas is typical) to answer a series of questions about this data. ### Constraints & Assumptions - The dataset covers several years of daily readings — on the order of a few thousand rows. - `P` is moderate — on the order of 10 to 30 towns. - "Linear model" always means ordinary least squares (OLS) **with an intercept**. - Model quality is judged by **in-sample mean squared error (MSE)** on the given data — no train/test split is requested. - When a question asks for a rounded answer, round only the final value, to the nearest integer. ### Clarifying Questions to Ask - For the standard deviation in Q1, should I use the sample (n-1) or population (n) convention — and can the choice change the answer? - How should ties be broken if two locations/models are exactly equally good? - For the median in Q2, if the filtered set has an even number of days, is the median the average of the two middle values? - How large is `P` exactly — is exhaustive search over subsets of towns computationally acceptable? - Are library least-squares solvers (e.g., `numpy.linalg.lstsq`, scikit-learn) allowed, or should the regression be implemented from scratch? ### Part 1 Write a function that answers the following five questions about the data: - **Q1.** Find the name of the location (a town or NYC) whose daily temperature varies the most. Use standard deviation as the measure of variation. - **Q2.** What is the median daily temperature of NYC on the days when `Town2`'s daily temperature is between 90 and 100 degrees inclusive? Round the answer to the nearest integer. - **Q3.** Fit `P` simple linear models (each with an intercept), one per town, using least squares to predict NYC's daily temperature from that town's daily temperature. Compute the sum of the absolute values of the regression slope coefficients (do **not** include the intercepts). Round the sum to the nearest integer. - **Q4.** For the given data, find the town that best predicts NYC's daily temperature. "Best" means: when a linear model with an intercept is fitted on that town's temperatures, it achieves the lowest MSE on the given data. - **Q5.** For the given data, find the **two** towns that jointly best predict NYC's daily temperature — i.e., the pair whose two-predictor linear model with an intercept achieves the lowest MSE on the given data. ```hint Q1 and Q2 in one pass Both are vectorized one-liners: Q1 is the argmax of the per-column standard deviation (with equal row counts, the n vs. n-1 convention rescales every column by the same factor, so the argmax is unchanged). Q2 is a boolean mask on the `Town2` column — `(t >= 90) & (t <= 100)` — followed by the median of `NYC` over the masked rows. ``` ```hint Least-squares mechanics A simple regression $y = a + b x$ has the closed form $b = \operatorname{cov}(x, y) / \operatorname{var}(x)$ and $a = \bar{y} - b\bar{x}$; equivalently, stack a column of ones next to the predictor(s) and call a least-squares solver. Q3 sums only the $|b|$ values — a classic trap is accidentally including the intercept. ``` ```hint Q4 and Q5 are searches over models Q4 is a loop over the `P` towns comparing MSEs; for a single predictor with an intercept, minimizing MSE is equivalent to maximizing the squared correlation with NYC. Q5 is the same idea over all $\binom{P}{2}$ pairs, fitting a two-column (plus intercept) regression for each — at `P ≈ 30` that is only ~435 cheap fits. ``` #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### Part 2 Write a function that answers the sixth question: - **Q6.** For the given data, return a set of **five** towns that jointly predict NYC's daily temperature as well as possible, again using a linear model with an intercept judged by in-sample MSE. ```hint Size the search space first Exhaustive best-subset search costs $\binom{P}{5}$ regressions — about 142,506 at `P = 30`, which is entirely feasible for a dataset of a few thousand rows. Decide between exact search and a heuristic based on `P`, and say so out loud. ``` ```hint The wording is a signal "As well as possible" (rather than "the best") hints that a good heuristic is acceptable: greedy forward selection adds one town at a time — the town that most reduces MSE given those already chosen — costing only about $5P$ fits. Know its failure mode: correlated predictors can make greedy miss the true best subset. ``` #### What This Part Should Cover ```premium-lock What This Part Should Cover ``` ### What a Strong Answer Covers ```premium-lock What a Strong Answer Covers ``` ### Follow-up Questions - In-sample MSE never gets worse as you add predictors. How would you change Q6's objective to guard against overfitting — adjusted R-squared, AIC/BIC, or cross-validated MSE — and what would change in your code? - If two towns' temperatures are nearly collinear, what happens to the fitted coefficients in Q5/Q6, and does that matter for prediction accuracy versus interpretation? - How would you scale Part 2 to `P = 1,000` candidate towns? (Consider correlation screening, greedy selection with incremental QR updates, or lasso.) - Prove or explain why the town with the highest absolute correlation with NYC is exactly the Q4 answer when models include an intercept.