Design resilient auth with flaky third-party tokens
Overview
This question evaluates a candidate's ability to design resilient, secure authentication and authorization flows that tolerate unreliable third-party token services, testing competencies in distributed systems architecture, security engineering, reliability, API design, and operational engineering.
System Design: Resilient, Secure Request Flow with Unreliable Third‑Party Authorization
Context
You are designing a multi‑region, HTTP/JSON API where clients must obtain an access token from a third‑party authorization service before calling your main API. The third‑party service is intermittently unreliable: it may return malformed responses, time out, or fail.
Assume:
- Tokens can be JWTs (signed with rotating keys) or opaque tokens (requiring introspection).
- Your system must operate in multiple regions with strict isolation to contain blast radius.
- Clients perform both read and write operations against your API.
Task
Design an end‑to‑end request flow that is resilient and secure. Explicitly cover:
- Token acquisition patterns, validation, caching and proactive refresh.
- Retries with exponential backoff and jitter; bounded attempts.
- Circuit breakers and request hedging to the third‑party.
- Idempotency keys for client‑visible mutations.
- Fallback paths (e.g., degraded mode, cached permissions) under dependency failures.
- Quarantining/poison‑pill detection for malformed or dangerous responses.
- Observability: metrics, tracing, logs, and alerts.
- Rate limiting and abuse‑prevention controls.
- SLA/SLO objectives and error‑budget‑driven degradation.
- Blast‑radius containment across regions and tenants.
- Describe external and internal APIs and key data models.
- Provide failure‑mode playbooks for common outage scenarios.
Be explicit about security controls (key management, token handling, storage, and transport security), and call out trade‑offs and pitfalls.
Solution
Login required