Survey Short of Its Response Target: Sample Size, Nonresponse Bias and Recovery Tactics
Company: Invoice Simple
Role: Product Manager
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Other
You are running a survey of your product's users, and far fewer people are responding than you planned for. How do you handle it?
```hint Ask "enough for what?"
Work out which decision the survey feeds and how much precision that decision really needs, then find the stage between invitation and completed survey where people drop out before choosing fixes.
```
### Clarifying Questions
- What decision will the survey results drive, and how costly would it be to get that decision wrong?
- Where did the target number come from: a sample-size calculation, a comparison between user segments, or a round number someone picked?
- Is the shortfall caused by few people opening the invitation, or by people starting the survey and abandoning it?
- Is the deadline fixed, or can the survey stay open longer and reach a wider group of users?
### What a Strong Answer Covers
- Defining "enough" from the decision and the precision it needs, including whether the current sample already suffices
- Diagnosing the shortfall stage by stage (delivery, open, start, completion) and matching each fix to its stage
- Recruitment levers such as channel, timing, length, reminders and incentives, with the bias each can introduce
- Checking respondents against the user base for nonresponse bias, then adjusting or caveating the results
- Fallbacks when the target is still missed: wider error bars, other evidence sources and transparent reporting
### Follow-up Questions
- Suppose the people who did respond skew heavily toward your most active users. How does that change what you can conclude, and what would you do about it?
- An incentive doubles the response rate. How would you tell whether it also changed who answers and how carefully they answer?
- When would you drop the survey altogether and rely on product analytics or customer interviews instead?
Overview: A product manager interview question about a user survey that is falling well short of its planned number of responses. It tests whether a candidate can judge how many responses a decision really needs, diagnose why people are not responding, raise participation without biasing the sample, and fall back on other evidence when the target is missed.