Evaluating Alpha Signals on 3,000 Stocks: A t-Stat of 2 and Comparing Two Signals
Company: Quantbot Technologies
Role: Quantitative Researcher
Category: Statistics & Math
Difficulty: easy
Interview Round: Technical Screen
You have a daily signal for each of 3,000 stocks and want to know whether it predicts their returns. Both parts of this question are about evaluating signals of this kind.
### Clarifying Questions
- How many years of history are available, and is each day's signal known before the return it is meant to predict starts?
- Are returns raw, or already adjusted for the market and sectors?
- Is the regression in Part 1 pooled across all stock-days, or run separately for each day?
- Will a signal be traded on its own, or combined with others in a portfolio with costs and constraints?
### Part 1 — What is wrong with a t-statistic of 2?
To evaluate a signal meant to predict each stock's return over the following week, you regress that weekly return on the signal, using every stock on every day. The coefficient on the signal has a t-statistic of 2. What are the problems with this evaluation?
```hint Count independent observations
Ask whether the rows of this regression are as independent as its standard errors assume, both from one day to the next and across stocks on the same day.
```
#### What This Part Should Cover
- The effect of overlapping weekly returns on the standard errors
- Correlation between stocks on the same day, and what it does to a pooled regression
- Whether a t-statistic of 2 is convincing, given outliers, the number of signals tried, and economic size
- A better evaluation design
### Part 2 — Which of two signals is better?
Two alpha signals each predict the next day's return for the same 3,000 stocks. How do you compare them and decide which one is better?
```hint Define better first
Decide what "better" means for the portfolio you would actually trade, then find a way to measure it day by day so that you can test a difference.
```
#### What This Part Should Cover
- Day-by-day measures of predictive power and a statistical test of the difference
- Performance of the portfolio each signal would trade, gross and net of costs
- Turnover, decay, capacity, and the correlation and incremental value of the two signals
### What a Strong Answer Covers
- Identifies the dependence structures that make naive standard errors too small, and corrects for them
- Explains why a t-statistic of 2 is weak evidence in this setting
- Compares signals with time series of daily metrics and a paired test rather than one pooled number
- Accounts for costs, turnover and capacity, not only predictive correlation
- Guards against look-ahead bias, survivorship and overfitting in the evaluation itself
### Follow-up Questions
- How would you compute a standard error that is robust to both the overlap and the correlation across stocks?
- One signal has a higher average information coefficient but four times the turnover. How do you decide?
- The two signals are highly correlated. How would you test whether the weaker one adds anything?
- How would you check that a signal's predictive power is not concentrated in small, illiquid stocks?
Overview: Critique evaluating a daily stock signal by regressing weekly returns on it across 3,000 stocks when the coefficient's t-statistic is 2, then explain how to decide which of two next-day alpha signals is better. Tests overlapping observations, correlation across stocks, information coefficients, turnover and costs.