You are asked to design several large-scale backend systems in a single onsite session. The bar is breadth and judgment under time pressure: for each system, drive a crisp design rather than an exhaustive one. For every system, be ready to cover: - Core requirements and assumptions (and the back-of-the-envelope numbers that justify your choices) - High-level architecture and main components - Data model and storage choices - How you meet scale, reliability, and latency targets - How you handle failures, retries, and consistency The five systems are independent; treat each as its own design but reuse patterns where they genuinely apply. ### Constraints & Assumptions - This is a time-boxed onsite round. Each sub-design gets only a few minutes, so prioritize the architecture, the data model, and the one or two hardest trade-offs over exhaustive coverage. - You may make reasonable simplifying assumptions as long as you state them. State each system's read/write ratio and the dominant bottleneck before diving in. - Per-system scale targets are given inline in each Part. ### Clarifying Questions to Ask These are scoping questions a candidate should raise up front, before designing any of the five systems: - How much depth does the interviewer want per system given the time box — a full design of one, or a breadth pass across all five? Where should I go deep if forced to choose? - For each system, what is the dominant axis to optimize: throughput, tail latency, durability, or cost? - Are there hard correctness/compliance requirements (e.g. no lost events, no double charges) that override availability, or is the system allowed to degrade gracefully? - What is the expected read/write ratio, and is eventual consistency acceptable for reads? - Are there geographic/multi-region requirements, or is a single-region design acceptable for the first pass? --- ### Part 1 — Webhook delivery service Design a webhook service that lets clients register callbacks for specific events and reliably delivers an HTTP request to the registered URL whenever a matching event fires. **Functional requirements** - Clients register a subscription with: an `eventId` (string or numeric) and a callback URL (HTTP endpoint). The `eventId → callback URL` mapping is unique: each `eventId` triggers exactly one callback URL. - When an event with a given `eventId` is triggered, the system sends an HTTP request to the registered callback URL carrying a payload with event data. - The system exposes per-event delivery status: success, failure, and retry-attempt history. **Non-functional requirements** - Scale: up to 1 billion events per day (about $1{,}000{,}000{,}000 / 86{,}400 \approx 11.6\text{k}$ events/second average; assume peaks of 3–5x). - High availability and durability of events — an accepted event must never be silently dropped. - Delivery guarantee: at-least-once is acceptable; receivers achieve effective exactly-once via idempotency. - Latency: near-real-time but not strict — delivery within a few seconds is fine. - Multi-tenant and secure: only authorized clients may create or manage their subscriptions. Design the full system: APIs, storage, event ingestion, dispatch workers, retry logic, and monitoring, and explain how it scales to 1B events/day. ```hint Decompose the path Separate "accept the event durably" from "deliver it." Ingestion should commit the event to a durable, replayable log and return fast; a separate worker fleet does the actual outbound HTTP and the slow, failure-prone retries. ``` ```hint Handling failing endpoints The hard part is misbehaving receivers (timeouts, 5xx, slow responses) without head-of-line blocking healthy ones. Think exponential backoff with jitter, a bounded retry budget, a dead-letter queue, and per-tenant concurrency limits so one bad endpoint can't starve others. ``` ```hint Partitioning Partition the durable log by `hash(eventId)` (or per-tenant) so ordering and load distribute evenly and a hot subscriber stays on its own partition. ``` #### What This Part Should Cover - Durable ingestion (accept-then-dispatch) backed by a partitioned, replayable log; sizing it against ~11.6k eps average with peak headroom. - Retry strategy: exponential backoff with jitter, a max-attempts / TTL cap, and a dead-letter path with an admin requeue interface. - Security: per-tenant authz, HTTPS-only callbacks, and HMAC-signed payloads so receivers can verify authenticity; idempotency identifier per delivery. - Observability: a per-delivery status store (attempt count, last error, terminal state) plus dashboards and alerting. --- ### Part 2 — Places-of-interest search service (Yelp / Foursquare-like) Design a service that manages and searches places of interest worldwide. **Data model (given)** — each place has at least: `id`, `name`, `location` (latitude, longitude), `place_type` (restaurant, hotel, park, …), and optional metadata (rating, address, opening hours). **Functional requirements** - Register, update, and remove places. - Search: given a user location (lat, lon) and a place type, return the N nearest places; optionally bound by a max radius. Sort primarily by distance, with secondary ranking signals such as rating. **Non-functional requirements** - Data scale: hundreds of millions of places globally. - High read throughput with low-latency search (e.g. p95 < 100 ms). - Writes need not be visible in real time; eventual consistency is acceptable. Address: how you store place data, how you index geospatial locations for efficient nearby search, how you filter by type and other attributes, how you shard and replicate globally, and how you handle hotspots (city centers) and caching. ```hint Geospatial indexing A B-tree on lat/lon can't answer "nearest N" efficiently. Map 2-D coordinates onto a 1-D, locality-preserving key — geohash or S2/H3 cells — so nearby points share a prefix and a query only scans the target cell plus its neighbors. ``` ```hint Read vs write split Reads dominate massively and consistency is relaxed. That argues for a source-of-truth store feeding an async pipeline (CDC / event stream) into a read-optimized geo search index (e.g. Elasticsearch/OpenSearch geo_point), rather than serving search directly off the primary. ``` #### What This Part Should Cover - A two-store split: a source-of-truth DB plus a read-optimized geo index, kept in sync asynchronously (eventual consistency justified by the relaxed write SLA). - Concrete geospatial indexing scheme (geohash / S2 / H3) and how a nearest-N query scans the target cell plus adjacent cells, then re-ranks by exact (haversine) distance. - Type/attribute filtering combined with the geo filter, and global sharding/replication (by region) with reads routed to the nearest region. - Hotspot mitigation: cell/tile-level caching for dense areas and an in-memory cache for repeated (location, type) queries. --- ### Part 3 — Slack-like chat system Design a messaging system supporting one-to-one and group conversations. **Functional requirements** - Send messages to another individual (1:1) or to a group with multiple members. - Create group chats; add/remove members (permissions simplified). - Deliver notifications to recipients when new messages arrive. - Support rich media (e.g. images); store large media efficiently (out of band). - A user can delete a message they sent — clarify the semantics: delete-for-everyone vs hide-for-self. **Non-functional requirements** - A large number of concurrent users and active conversations. - Low-latency, near-real-time delivery for online users. - Durable, queryable message history (e.g. loading history when opening a conversation). - High availability and graceful handling of clients that disconnect and reconnect. Cover: overall architecture and services, how clients maintain real-time connections, the data model (users, conversations, membership, messages), message vs media storage, reliable in-order delivery per conversation, and how message deletion works. ```hint Connection layer Online delivery needs a persistent transport (WebSocket) terminated by a stateless gateway fleet. Keep a presence/routing layer that maps user → the gateway holding their connection, so a message event can be steered to the right box. ``` ```hint Ordering and history Persist messages in a write-heavy, per-conversation-partitioned store (wide-column / log-structured) so all messages for a conversation share a partition and a natural sort key — that gives you both per-conversation ordering and efficient history reads. ``` ```hint Deletion Model deletion as a tombstone (`deleted_at`), not a physical delete: it preserves history/audit, propagates as a real-time event, and a retention job can purge content later. ``` #### What This Part Should Cover - Separation of a stateless real-time gateway (WebSocket + presence/routing) from the chat service and the message store. - Send/fan-out path: persist first, then publish a message event (keyed by conversation) so online recipients get a push and offline ones get unread counts + push notifications. - Data model + storage split: messages partitioned by `conversation_id` for ordering and history; media in object storage via pre-signed upload, referenced by id from the message. - Deletion semantics (tombstone, delete-for-everyone) and how the deletion propagates to connected clients. --- ### Part 4 — Distributed CI/CD workflow system (GitHub Actions-like) Design a CI/CD workflow engine that triggers and manages workflows in response to repository events. **Scenario** - Integrated with a source-code hosting service. Repo events (git push, PR opened, tag created, scheduled) should automatically trigger predefined workflows. - Workflow definitions live as config files (e.g. YAML) in the repository itself. **Functional requirements** - Detect repo events and map them to the workflows that should run. - For each trigger, create a workflow run containing multiple jobs and steps. - Schedule jobs onto a distributed worker pool (container-based runners or VMs). - Manage each run's lifecycle: queueing, execution, retries, timeouts, cancellation; collect and store logs and artifacts. - Report status back to the source-control system (success / failure / in-progress). **Non-functional requirements** - Multi-tenant: many organizations and repositories. - Scalable to tens or hundreds of thousands of concurrent workflow runs. - Fair scheduling across projects/orgs with rate limiting. - Strong isolation between workflows for security. Cover: event ingestion from the git service; how you find and parse the workflow config from the triggering commit; the orchestration layer managing run state and inter-job dependencies; the runner/worker infrastructure; storage for runs, logs, and artifacts; and how you scale, ensure reliability, and handle failures. ```hint Two control loops Split a stateful orchestrator (resolves which workflows fire for a commit, parses the config at that commit SHA, builds the job DAG, tracks run state) from a stateless runner fleet that pulls ready jobs from a queue. The orchestrator only enqueues a job once its dependencies are satisfied. ``` ```hint Fairness and isolation With many tenants, naive FIFO lets one org starve others. Think per-org/per-repo concurrency quotas and weighted-fair queue partitioning. For security, jobs from different tenants must run in isolated, ephemeral environments (fresh container/VM, network policy), never a shared mutable host. ``` #### What This Part Should Cover - Event ingestion (queue between the git service and the orchestrator) and resolving + parsing the workflow config from the exact triggering commit SHA. - An orchestration layer that builds the job DAG, enqueues jobs as dependencies clear, and tracks run/job/step state in a strongly-consistent store. - A runner fleet pulling jobs, executing steps in isolated ephemeral environments, and streaming logs/artifacts to object storage. - Multi-tenant fairness (per-org quotas, rate limiting), failure handling (runner heartbeats, timeouts, reschedule-if-safe), and status reporting back to the git service. --- ### Part 5 — Payment processor service Design a simplified but realistic payment-processing system. **Functional requirements** - Merchants integrate via an API to process payments. For a payment request the system: receives the initiation request (e.g. charge a card or wallet), routes it to an appropriate downstream gateway/processor (card network, bank, third party), receives the response, and returns a clear result (authorized / declined / error). - Related operations such as refund and capture should be addressable. **Non-functional requirements** - Strong security and compliance, including safe handling of sensitive payment data. - High availability and low latency for the synchronous authorization path. - At-least-once / effectively-exactly-once behavior that never double-charges a customer, even under retries. - Auditability: every payment event is logged and traceable; a ledger / transaction history is maintained. Cover: the merchant-facing API (auth + idempotency); the services that route requests to different processors; the data model for payments, merchants, and transaction history; how you integrate with external gateways and handle their failures; how you maintain a reliable ledger and support reconciliation; and how you address security/compliance. ```hint Don't lose money on a retry The central problem is exactly-once over an unreliable downstream. Require a client idempotency key, enforce uniqueness on `(merchant_id, idempotency_key)`, and persist the processor's transaction id so an ambiguous timeout never re-submits a charge. ``` ```hint Ledger correctness Use a double-entry, append-only ledger (every credit has a matching debit) instead of mutating balances in place. Tie the DB write and the "tell the merchant success" step together with a transactional outbox so you never ack a payment you didn't durably record. ``` ```hint Keep card data out of scope Don't store raw PANs in your main DB — tokenize via a PCI-scoped vault and reference the token, shrinking PCI scope and audit surface. ``` #### What This Part Should Cover - Idempotency and exactly-once: client idempotency key, uniqueness on `(merchant_id, idempotency_key)`, and replaying the stored result on duplicate requests. - Synchronous authorization path: API gateway → payment service → routing/adapters → external processor, with the declined/authorized/error state machine and ledger writes. - Reliable ledger: double-entry, append-only, corrections via compensating entries; transactional outbox so the DB record and merchant ack stay consistent; reconciliation for ambiguous (timeout/unknown) outcomes. - Security/compliance: tokenization + PCI-scoped vault, encryption at rest, TLS, strict merchant auth, and full audit logging. - Asynchronous flows (3-D Secure, bank transfers) via webhooks/polling that later finalize the payment and ledger. --- ### What a Strong Answer Covers Across all five parts, beyond each part's specific rubric, a strong candidate demonstrates the following cross-cutting judgment: - **Time management and prioritization** — given the round is time-boxed, leading with requirements + the single hardest trade-off per system rather than uniform shallow coverage, and saying out loud where they'd go deeper. - **Back-of-the-envelope grounding** — translating each system's scale numbers (events/sec, place count, concurrent connections/runs, payment QPS) into concrete storage/partition/worker sizing, not hand-waving "we'll scale it." - **Reusing patterns deliberately** — recognizing the shared primitives (durable queue/log + worker fleet, async indexing, outbox/idempotency, tenant isolation and fair scheduling) across webhooks, chat, CI/CD, and payments, and reusing them without forcing a bad fit. - **Consistency vs availability stance** — stating per system whether eventual consistency is acceptable (POI search, chat history) or strong correctness dominates (payments ledger, webhook durability), and designing accordingly. - **Multi-tenancy and security as first-class** — per-tenant authz, isolation, rate limiting, and quotas treated as design inputs in webhooks, CI/CD, and payments, not bolted on. ### Follow-up Questions - **Webhooks:** how do you handle a subscriber whose endpoint is down for hours — buffer indefinitely, drop after TTL, or expose a pull/replay API? How do you stop one slow tenant from consuming the whole worker fleet? - **POI search:** how do you keep results correct near cell boundaries (the true nearest place sits in an adjacent cell), and how do you handle dense city centers where one cell holds millions of candidates? - **Chat:** how do you guarantee per-conversation ordering when the same user is connected on two devices and sends concurrently, and how do you reconcile history after a long disconnect? - **CI/CD:** how do you fairly schedule when one org submits 10,000 jobs while another submits 5, and how do you guarantee strong isolation so a malicious workflow can't read another tenant's secrets? - **Payments:** walk through the exact recovery when your call to the downstream processor times out with no response — how do you avoid both double-charging and silently dropping a successful charge?