Stream and Cancel Agent Responses Without Unsafe Retries

Design streaming, cancellation, and failure recovery for a multi-turn Agent/RAG conversation system. Users must be able to stop generation, while the call chain can include an orchestrator, a model, vector retrieval, and external tools with side effects. The source requests immediate interruption of the model call and billing when a user clicks Stop. Explain which parts can be controlled locally and which require explicit provider support. ### Constraints & Assumptions - Streamed text may already have reached the client before cancellation or failure. - A closed browser connection does not by itself prove that remote generation stopped. - Tool operations may write data or charge money, so arbitrary retries can duplicate effects. - The source supplies no universal provider cancellation, refund, request-lookup, or idempotency guarantee. ### Clarifying Questions to Ask - Is server-to-client token delivery sufficient, or is bidirectional interaction on one connection required? - Does the provider expose cancellation and authoritative request status, and what happens to in-flight billing? - Which tools are read-only, idempotent, or irreversible? - Should a stopped turn remain as partial history, and can a later retry create a new attempt? ### Part 1 — Stream and Stop Compare SSE and WebSocket for this application. Trace a stop request from the UI through stream delivery, orchestration, model execution, and pending tools. #### What This Part Should Cover - Protocol choice based on traffic direction and operational behavior. - An authorized cancellation operation identified by session, turn, and attempt. - Immediate local suppression versus confirmed upstream cancellation and billing uncertainty. ### Part 2 — Recover the Agent Chain Describe deadlines, retries, durable operation identity, and partial outcomes when a component times out or crashes. #### What This Part Should Cover - Different retry treatment for retrieval and side-effecting tools. - Attempt fencing and a recorded canceled, failed, complete, or uncertain outcome. - No duplicate tool effects caused by blindly replaying an entire agent turn. ```hint Stop delivery and stop computation are separate The server can stop sending bytes even if the provider has already accepted a long-running request. Decide how that uncertainty should appear in durable state. ``` ### What a Strong Answer Covers - An appropriate streaming protocol and end-to-end cancellation flow. - Honest limits on immediate remote interruption and billing. - Recovery that respects the side effects and completion state of each tool operation. ### Follow-up Questions - What happens when completion and cancellation arrive at the same time? - How should a reconnecting client distinguish a stopped attempt from a fresh retry? - Why is replaying the same prompt not sufficient to deduplicate a tool charge?