I've been interviewing with a few different companies lately and kept running into some pretty classic Product Case questions. I don't have a ton of interview experience, so I'm honestly not sure how good my answers actually are. Wanted to share them here and see what everyone else thinks.
First, on success metrics — the standard playbook is obviously to start by asking the interviewer what the business goal of the product or project is, and then define your success metrics based on that goal. I ran into a similar setup at two different companies (I signed an NDA so I can't really say which ones): the company has a subscription-based product, and we want to add some benefit or feature to the subscription — something that only subscribers get, non-subscribers don't have access to it. In that situation, what would the business goal be? I noticed some interviewers won't just tell you the goal — they'll flip it back on the candidate and ask what you think the goal is. Honestly this is kind of backwards from how it actually works at a real job — at least on every project I've worked on, we start with a goal we want to hit, and then talk about what feature to build for it. Nobody decides to build a feature first and then goes looking for a goal to attach to it afterward. That said, for this specific subscription scenario, the most likely goals I could come up with were:
- Retain existing subscribers — boost their engagement and reduce churn
- Attract new subscribers — improve the non-subscriber-to-subscriber conversion rate
- Increase revenue? Not just subscription fee revenue, since these benefits/features might tie directly into revenue too
After I laid out these possible business goals, the interviewer would often turn around and ask which one I thought was actually the most reasonable one. At that point I'm genuinely just staring blankly, and I end up picking one based on gut feeling and then watching the interviewer's reaction. So what I'm stuck on is: in this kind of situation, is there a better answer or a better way to approach it?
Second, there's a question that almost every AB testing interview asks: if we set alpha at 0.05 but the test's p-value comes back at 0.10, should we still launch the feature? In my actual job I've seen cases where we didn't launch and cases where we launched anyway — there are a lot of factors that go into that decision. But in an interview setting, if the interviewer wants a definitive yes-or-no answer, is "don't launch" the better answer here? First, statistically the result didn't hit the bar for launching. Second, generally speaking, the impact of a false positive on the product is bigger than a false negative (this is just my own understanding, might not be right) — meaning if we launch a new feature and it turns out the actual effect doesn't meet expectations, or even backfires, then maintenance cost, rollback cost, reputation cost, all of that comes from the false positive. On the flip side, not launching a feature that might have had a positive effect does mean an opportunity loss, but a company usually has a lot of features waiting to be delivered — even if we don't launch A, B, C, D, E are right behind it ready to fill the gap. And not launching A doesn't mean giving up on it right away either — we can still do more analysis, see if there's room to improve it, and then run another round of testing.
Curious if anyone has other thoughts on this — hoping we can discuss it together and all get a bit better at this.
Discussion
Loading comments…