Design real-time delivery tracking with async updates

Scenario

Design a scalable, fault-tolerant backend system to track deliveries in real time.

Context

• Your company provides a tracking platform.
• Third-party delivery providers (3P) perform the actual delivery and emit events (e.g., accepted, picked up, in transit, delivered, failed).
• Users/clients need to:
  1. Query the latest delivery status (low latency).
  2. Optionally subscribe to real-time updates (e.g., WebSocket/SSE/push).

Core requirement (most important)

Handle asynchronous task execution and ensure correct state updates + queries even with:

• Out-of-order events
• Duplicate events (at-least-once delivery)
• Provider downtime and retries
• Partial failures and regional outages

Deliverables

Propose an architecture and explain:

• APIs (ingest events, query status, subscriptions)
• Data model (deliveries, events, status timeline)
• How you process events asynchronously and update the “current status”
• Ordering/idempotency strategy
• Fault tolerance, scaling, and observability
• Tradeoffs (consistency vs availability, storage choices)