Store, Deliver and Rotate an API Server's Database Credentials

The interviewer asks how your team updated the database credentials used by your API server: where the credential is stored, how the running service obtains it, and how it is changed. Describe a sound approach. As a concrete setting, assume the API runs as AWS Lambda functions that connect to a relational database with a username and password. ### Clarifying Questions - How many services and environments use this database user, and do they share it? - Is rotation scheduled, triggered by a suspected leak, or both, and how much disruption is acceptable during a change? - Can the database accept short-lived credentials derived from the function's cloud identity, or only static passwords? ### Part 1 — Store and deliver the credential Compare these ways of giving the function its database credential, and recommend one: hardcoding it in the source code; a plaintext environment variable; an environment variable holding the credential encrypted with a managed key and decrypted by the function's code at startup; and fetching it at runtime from a secrets manager. ```hint Count the copies For each option, list every place the plaintext credential ends up (repository history, build artifacts, deployment configuration and state, consoles, logs, memory) and who can read each of them. ``` #### What This Part Should Cover - The exposure of each option, including what the encrypted environment variable does and does not protect against - A recommended design with least-privilege access to the secret and its key - How the function caches the credential and keeps it out of logs ### Part 2 — Rotate the credential without downtime Describe how to change the credential while the API keeps serving traffic, and what changes when the rotation is a response to a leak. ```hint Old and new at the same time A fleet of running functions cannot all switch at the same instant. Decide how the old and new credentials overlap, and when it is safe to revoke the old one. ``` #### What This Part Should Cover - A step-by-step rotation with an overlap period and a verification step - How running functions and their open connections pick up the new credential - The extra steps when the credential has leaked ### What a Strong Answer Covers - Never in source code, with a clear explanation of why - An accurate comparison of environment variables, encrypted environment variables and a secrets manager - Least-privilege access for the function's identity and the encryption key, auditing, and no secrets in logs - A zero-downtime rotation procedure with overlap, verification and revocation - Leak response, and the option of short-lived credentials that remove the long-lived password ### Follow-up Questions - The password was committed to the repository long ago. What do you do, and why is deleting it in a new commit not enough? - Encrypting a credential before putting it in the function's environment, then decrypting it in code, can look redundant. What does it protect against, and what does it not? - After a rotation, how do you confirm that nothing still uses the old credential? - How would you remove the long-lived password altogether?