Design a Shared Proxy Layer That Cuts Microservices' API Call Costs

An organization runs many microservices, and a growing share of its operating cost comes from the API endpoints those services call. Design a proxy layer that sits between the services and those endpoints and reduces that cost without changing what callers get back. For every cost-saving mechanism you propose, be ready to explain where the saving comes from, when it is safe to apply, and what it risks. For practice, assume the endpoints charge per request (third-party providers, or internal endpoints billed back per call), that traffic mixes read-style lookups with writes that must not be duplicated, and that the proxy can be introduced gradually. No scale or prices are given; state your own assumptions where you need numbers. ### Constraints and Clarifications - Correctness comes before savings: a caller must never receive a response meant for someone else, a stale answer where freshness matters, or a duplicated write. - The proxy adds a hop or a processing step, so its own latency and availability are part of the design. - The proxy costs money to run; its savings have to exceed that. ### Clarifying Questions - What makes up the bill: per-call fees from external providers, internal per-request chargeback, data transfer, or a mix? - Which endpoints are read-only, how fresh must their answers be, and do the providers' terms allow caching? - Do any upstreams offer bulk endpoints, cheaper pricing tiers or alternative providers? - Is it known which services and endpoints account for most of the spend today? - Can callers change their code to adopt the proxy, or must it be transparent to them? ### Part 1 — Requirements and placement Define the functional and non-functional requirements, decide where the proxy runs (a central gateway, a sidecar next to each service, or a client library), and sketch the components and the path of one request. ```hint Global or local Before picking a topology, sort the cost-saving decisions into those that need a view of all traffic and those that can be made next to a single caller. ``` #### What This Part Should Cover - Functional and non-functional requirements, including the proxy's latency and availability budget - A placement decision with its trade-offs - The main components and the request path - How existing services adopt the proxy ### Part 2 — Mechanisms that reduce cost Propose concrete techniques the proxy uses to cut the number of paid calls or the price per call, and for each one state when it is safe. ```hint Look for repeated work Look for calls whose answer you already have, identical calls that are in flight at the same moment, and separate calls that could share one upstream request. ``` #### What This Part Should Cover - Response caching: what is cacheable, cache keys and freshness rules - Collapsing duplicate in-flight requests and batching separate ones - Per-service quotas, rate limits and spending budgets - Routing to cheaper providers, tiers or request shapes ### Part 3 — Correctness, failure handling and cost visibility Explain how you prevent stale or leaked data and double charges on retries, what happens when the proxy, its cache or an upstream fails, and how you measure spend and savings per service. ```hint The ambiguous timeout Ask what should happen to a write whose upstream call times out, and who pays if it is retried. ``` #### What This Part Should Cover - Safe retries and idempotency for writes - Fail-open versus fail-closed behavior for each failure - Metering and attribution of cost to the calling service - Metrics that show savings and catch regressions ### What a Strong Answer Covers - Mechanisms tied to where the money actually goes, starting from measurement - A placement decision that weighs a global view of traffic against extra hops and blast radius - Safety rules: allow-listed cacheable endpoints, caller-scoped cache keys, idempotent retries - Explicit failure policies, including what callers experience when the proxy itself is down - Per-service cost attribution that proves savings and creates accountability - An incremental rollout plan ### Follow-up Questions - How would you roll this out to existing services without a big-bang migration? - One upstream raises its price tenfold overnight. How does your design respond? - How would you choose a cache lifetime per endpoint, and detect when caching starts serving wrong answers? - How do you keep the proxy from becoming a single point of failure, or a large cost of its own?