Consumer AI Agent Design: Autonomy Limits, Consent Rules and User Overrides

You are interviewing for a product manager role at Amazon. The interviewer asks you to design an AI agent for consumers that does more than answer questions: it carries out actions for the people who use it. You then need to define its permission and control model, meaning what the agent may do by itself, what needs the user's approval first, and how the user stays in charge of it. ### Clarifying Questions - Should the agent act only inside Amazon's own services, such as orders, returns and subscriptions, or also on third-party websites and apps through the user's other accounts? - May the first version spend the user's money, or should it start with actions that cost nothing, such as tracking deliveries or managing lists? - Which surface matters most: a voice assistant, the shopping app, a chat interface, or all of them? - Is the agent used by one person, or by a household sharing one account where several people could give it instructions? ### Part 1 — Choose the agent's users and first jobs Pick the users the agent serves first and the jobs it handles for them, list the actions it needs to finish those jobs, and explain why this is the right place to start. ```hint Start where mistakes are recoverable An early version teaches people whether to trust the agent at all. Look for jobs that come up often enough to matter but where a wrong action can still be undone. ``` #### What This Part Should Cover - A specific user group and the recurring jobs the agent takes off their hands - The concrete actions those jobs require, and the actions deliberately left out of the first version - Why the scope balances user value against the cost of an agent mistake ### Part 2 — Define the permission and control model Explain how the agent receives permission to act, how that permission is limited, when the agent must stop and ask, and how the user can see, undo or revoke what it has done. ```hint Not every action is equal Sort the agent's actions by what a mistake would cost and how easily it could be reversed, then decide how much user involvement each group needs. ``` #### What This Part Should Cover - How permission is granted and bounded: which actions, how much money, for how long, through which accounts - When the agent must ask for confirmation, including uncertain or unusual requests - How users monitor, undo, pause or revoke the agent's actions - Where the rules are enforced, and how they hold up against agent errors or manipulated instructions ### What a Strong Answer Covers - A focused first use case tied to a real user need rather than a do-anything assistant - Permissions proportional to risk, with conservative defaults and a path to more autonomy as trust is earned - Accountability when the agent gets something wrong: refunds, customer support, and who bears the loss - Success metrics paired with guardrails on unintended actions, approval fatigue and trust - A phased launch and an explicit stance on convenience versus control ### Follow-up Questions - A product page the agent reads contains hidden text telling it to buy a different item. How does your design stop the agent from acting on it? - Users approve almost every confirmation request within a second, without reading it. What would you change? - How would your permission model change if the agent had to complete purchases on third-party sites using the user's accounts there?