Stream and Cancel Agent Responses Without Unsafe Retries

Quick Overview

Design SSE or WebSocket streaming, end-to-end stop handling, and safe Agent retries while distinguishing local cancellation from provider billing and tool effects.

Stream and Cancel Agent Responses Without Unsafe Retries

Company: Qualified Health

Role: Software Engineer

Category: System Design

Difficulty: hard

Interview Round: Onsite

Design streaming, cancellation, and failure recovery for a multi-turn Agent/RAG conversation system. Users must be able to stop generation, while the call chain can include an orchestrator, a model, vector retrieval, and external tools with side effects. The source requests immediate interruption of the model call and billing when a user clicks Stop. Explain which parts can be controlled locally and which require explicit provider support. ### Constraints & Assumptions - Streamed text may already have reached the client before cancellation or failure. - A closed browser connection does not by itself prove that remote generation stopped. - Tool operations may write data or charge money, so arbitrary retries can duplicate effects. - The source supplies no universal provider cancellation, refund, request-lookup, or idempotency guarantee. ### Clarifying Questions to Ask - Is server-to-client token delivery sufficient, or is bidirectional interaction on one connection required? - Does the provider expose cancellation and authoritative request status, and what happens to in-flight billing? - Which tools are read-only, idempotent, or irreversible? - Should a stopped turn remain as partial history, and can a later retry create a new attempt? ### Part 1 — Stream and Stop Compare SSE and WebSocket for this application. Trace a stop request from the UI through stream delivery, orchestration, model execution, and pending tools. #### What This Part Should Cover - Protocol choice based on traffic direction and operational behavior. - An authorized cancellation operation identified by session, turn, and attempt. - Immediate local suppression versus confirmed upstream cancellation and billing uncertainty. ### Part 2 — Recover the Agent Chain Describe deadlines, retries, durable operation identity, and partial outcomes when a component times out or crashes. #### What This Part Should Cover - Different retry treatment for retrieval and side-effecting tools. - Attempt fencing and a recorded canceled, failed, complete, or uncertain outcome. - No duplicate tool effects caused by blindly replaying an entire agent turn. ```hint Stop delivery and stop computation are separate The server can stop sending bytes even if the provider has already accepted a long-running request. Decide how that uncertainty should appear in durable state. ``` ### What a Strong Answer Covers - An appropriate streaming protocol and end-to-end cancellation flow. - Honest limits on immediate remote interruption and billing. - Recovery that respects the side effects and completion state of each tool operation. ### Follow-up Questions - What happens when completion and cancellation arrive at the same time? - How should a reconnecting client distinguish a stopped attempt from a fresh retry? - Why is replaying the same prompt not sufficient to deduplicate a tool charge?

Overview: Design SSE or WebSocket streaming, end-to-end stop handling, and safe Agent retries while distinguishing local cancellation from provider billing and tool effects.

Read the full Qualified Health Software Engineer interview experience this question came from

Qualified Health logo
Qualified Health
Sep 8, 2026
hardSoftware EngineerOnsiteSystem Design
2
0

Design streaming, cancellation, and failure recovery for a multi-turn Agent/RAG conversation system. Users must be able to stop generation, while the call chain can include an orchestrator, a model, vector retrieval, and external tools with side effects.

The source requests immediate interruption of the model call and billing when a user clicks Stop. Explain which parts can be controlled locally and which require explicit provider support.

Constraints & Assumptions

  • Streamed text may already have reached the client before cancellation or failure.
  • A closed browser connection does not by itself prove that remote generation stopped.
  • Tool operations may write data or charge money, so arbitrary retries can duplicate effects.
  • The source supplies no universal provider cancellation, refund, request-lookup, or idempotency guarantee.

Clarifying Questions to Ask Guidance

  • Is server-to-client token delivery sufficient, or is bidirectional interaction on one connection required?
  • Does the provider expose cancellation and authoritative request status, and what happens to in-flight billing?
  • Which tools are read-only, idempotent, or irreversible?
  • Should a stopped turn remain as partial history, and can a later retry create a new attempt?

Part 1 — Stream and Stop

Compare SSE and WebSocket for this application. Trace a stop request from the UI through stream delivery, orchestration, model execution, and pending tools.

What This Part Should Cover Guidance

  • Protocol choice based on traffic direction and operational behavior.
  • An authorized cancellation operation identified by session, turn, and attempt.
  • Immediate local suppression versus confirmed upstream cancellation and billing uncertainty.

Part 2 — Recover the Agent Chain

Describe deadlines, retries, durable operation identity, and partial outcomes when a component times out or crashes.

What This Part Should Cover Guidance

  • Different retry treatment for retrieval and side-effecting tools.
  • Attempt fencing and a recorded canceled, failed, complete, or uncertain outcome.
  • No duplicate tool effects caused by blindly replaying an entire agent turn.

What a Strong Answer Covers Guidance

  • An appropriate streaming protocol and end-to-end cancellation flow.
  • Honest limits on immediate remote interruption and billing.
  • Recovery that respects the side effects and completion state of each tool operation.

Follow-up Questions Guidance

  • What happens when completion and cancellation arrive at the same time?
  • How should a reconnecting client distinguish a stopped attempt from a fresh retry?
  • Why is replaying the same prompt not sufficient to deduplicate a tool charge?