Writing, Evaluating and Iterating on LLM Prompts for a Production AI Feature
Company: Sierra
Role: Product Manager
Category: Machine Learning
Difficulty: medium
Interview Round: Other
How do you go about prompt engineering? Walk through how you write the prompts behind a language-model feature and how you decide one is ready to ship.
```hint Know when a prompt is better
Before any wording technique, think about how you would tell that one version of a prompt beats another, and at what point in your process you would set that up.
```
### Clarifying Questions
- Is the prompt behind a single, bounded model call, such as classifying or summarizing a support conversation, or a multi-turn agent that follows business policies and calls tools?
- Besides the prompt itself, which levers are available: retrieval over the business's own documents, fine-tuning, or switching to a different model?
- Are there latency or cost-per-request limits that cap how long the prompt and the response can be?
- Who maintains the prompt after launch: engineers, the PM, or the operations team of the business that deploys it?
### What a Strong Answer Covers
- Defining the task, the inputs the model will see, and measurable success criteria before the first draft.
- How a well-built prompt is organized: context and role, explicit instructions with the reasons behind them, examples, input data kept apart from instructions, and an output format the product can rely on.
- An iteration loop driven by a fixed test set of typical, edge and adversarial cases, with a grading method whose reliability is checked.
- Recognizing when prompting has hit its limit and the task needs decomposition, retrieval, tools, fine-tuning or a different model, including the cost and latency trade-offs.
- Running prompts in production: versioning, regression testing, monitoring live outputs, and not relying on the prompt as a security boundary.
### Follow-up Questions
- A prompt change fixes a failure a customer reported but breaks two cases that used to pass. How do you decide whether to ship it?
- Your evaluation uses one model to grade another model's answers. How do you know the grader can be trusted?
- A new model version becomes available. What do you do with your existing prompts before switching?
Overview: A product manager interview question on how you approach prompt engineering for a language-model feature, from the first draft to a version ready for production. It tests how you define quality, evaluate and iterate on prompts, recognize when prompting is not enough, and weigh cost, latency and security risks.