Reliable API Integration And Distributed Design

What's being tested

These problems test whether you can turn an unreliable, partially specified API workflow into a correct distributed system with explicit contracts, state transitions, and recovery behavior. The interviewer is probing idempotency, consistency, concurrency control, failure isolation, and the ability to distinguish your service’s guarantees from those of an external dependency. Stripe cares because payments, ledgers, feature controls, counters, and third-party integrations must remain correct when requests are retried, duplicated, delayed, reordered, or only partially completed.

Core knowledge

• Contract-first API design starts by defining request and response schemas, validation rules, coordinate or currency units, authentication, timeouts, pagination, and error semantics before choosing storage or infrastructure. State assumptions explicitly when the prompt leaves behavior unspecified.

• Idempotency keys let a client safely retry a mutation: persist (account_id, idempotency_key, request_hash, status, response) and return the original result for an identical request. Reject reuse with a different payload rather than silently applying a second operation.

• Retry policy must distinguish transient failures (429, connection resets, many 5xx) from permanent failures (400, authorization errors, malformed input). Use exponential backoff with jitter, bounded attempts, and a deadline; retries without an idempotency key can duplicate side effects.

• Timeouts and circuit breakers prevent a slow dependency from consuming every worker. Set connect, read, and overall deadlines; trip a circuit after measured failures, serve a safe fallback where possible, and use half-open probes to recover gradually.

• Exactly-once effects are usually implemented as at-least-once delivery plus deduplication, not promised by the network. A unique database constraint, request log, or inbox table converts duplicate deliveries into one durable state transition.

• Linearizability means each operation appears to occur atomically at one point between invocation and response. For a single-winner dispatch acceptance, use a conditional update such as UPDATE ... WHERE status = 'open', then verify affected-row count.

• Consistency choices should follow the invariant. A financial ledger and winner selection need strong consistency; a feature-flag read path may tolerate bounded staleness if rollout safety is preserved. State whether the system offers read-after-write, eventual, or monotonic-read guarantees.

• Transactional boundaries matter when combining local writes with external APIs. A local database transaction cannot atomically commit with a routing provider; use an outbox, durable workflow state, reconciliation, and compensating actions rather than claiming distributed atomicity.

• Outbox/inbox patterns persist an event alongside the business mutation, then publish asynchronously and deduplicate consumption. Include a stable event ID, versioned schema, retry state, dead-letter handling, and replay-safe handlers.

• Hot-key mitigation is essential for counters and popular feature flags. Shard a frequently updated key across $k$ buckets, write to one bucket per event, and read by summing buckets; this trades write contention for read amplification and requires a defined time-window strategy.

• Geospatial matching commonly uses an index such as geohash, S2, or H3 to query nearby cells before calculating exact distance with the Haversine formula. Expand neighboring cells near boundaries and define freshness, radius, and tie-breaking rules.

• Observability and security should expose request IDs, idempotency keys, dependency latency, retry counts, circuit state, and invariant violations without logging secrets or payment data. Track p50, p95, and p99 separately; averages hide tail failures that trigger cascading retries.

Worked example: Design ledger and bikemap integration

In the first 30 seconds, I would clarify the ledger’s currency and precision, whether debits must be strictly ordered, the routing provider’s latency and quota limits, acceptable stale route data, and whether a successful response means “ledger committed” or “route obtained.” I would declare that ledger mutations require strong consistency, while route lookup is an external, fallible read that must not corrupt balances. I would organize the answer around the ledger data model and invariants, the idempotent mutation API, the routing-adapter boundary, and asynchronous recovery and reconciliation. For the ledger, I would use integer minor units, immutable double-entry records, balance checks, and a unique idempotency key enforced transactionally in Postgres. For the bikemap call, I would apply a short timeout, bounded jittered retries for transient errors, caching keyed by normalized coordinates, and a circuit breaker. The central tradeoff is whether to block the user-visible operation until routing succeeds or commit the financial operation first and calculate the route asynchronously; I would choose based on whether the route is essential to the business invariant. I would explicitly explain that a local transaction cannot provide atomic commit with the provider, so an outbox and reconciliation job are safer than a two-phase commit assumption. I would close by saying that, with more time, I would specify provider failover, quota handling, audit retention, and property-based tests for duplicate and reordered requests.

A second angle

Design a Feature Flag Service with Percentage Rollouts and Allow/Block Lists uses the same reliability principles but shifts the primary constraint from transactional money movement to low-latency, highly available reads. Evaluation should be deterministic: hash a stable subject and flag key, map the result to a bucket, then apply precedence such as block list, allow list, explicit targeting, percentage rollout, and default. A stale cached configuration may be safer than failing closed or open, but the choice must be explicit for each flag’s risk level. Configuration writes need versioning, audit history, propagation monitoring, and idempotent update requests.

Common pitfalls

Pitfall: Saying “use retries for reliability” without specifying idempotency, retryable status codes, deadlines, backoff, or duplicate suppression leaves the most dangerous failure mode unanswered. Tie every retry policy to a particular side effect and invariant.

Pitfall: Jumping directly to Kafka, sharding, or microservices before clarifying correctness and scale makes the design feel like infrastructure shopping. Start with API semantics, state transitions, throughput, latency, and failure guarantees; then choose components that support them.

Pitfall: Claiming a distributed transaction makes a ledger update and third-party API call atomic is incorrect. Explain the unavoidable partial-failure window and provide an outbox, workflow state machine, reconciliation, and compensating behavior.

Connections

An interviewer may pivot to sagas, outbox/inbox delivery, leader election, multi-region consistency, or property-based testing of invariants. They may also ask how to design rate limits, webhook verification, dead-letter recovery, or safe schema evolution without breaking old clients.

Further reading

• Stripe: Idempotent requests — concrete treatment of safely retrying API mutations.

• Designing Data-Intensive Applications by Martin Kleppmann — strong coverage of replication, consistency, transactions, and distributed failure modes.

• Pat Helland, “Life Beyond Distributed Transactions” — explains practical alternatives to global atomic transactions.