Terraform IaC, State, Modules, And Drift Management

What's being tested

Interviewers probe a candidate’s ability to design, reason about, and operate Infrastructure as Code at application scale: how you structure Terraform code, manage its state, compose reusable modules, and detect/resolve configuration drift safely. They want to see clear tradeoffs (simplicity vs. reuse, remote state vs. local), an understanding of idempotence and resource lifecycle, and concrete patterns you would implement in code or CI to avoid outages. Expect to justify decisions under constraints like multiple environments, team ownership, and automated pipelines.

Core knowledge

Terraform lifecycle: terraform init, terraform plan, terraform apply, terraform destroy — plan shows a diff, apply reconciles to reach the desired state; never skip plan in automation.
State file semantics: the state is the source of truth for resource IDs, attributes, and dependencies; treat it as authoritative and versioned — losing or corrupting it breaks reconciliation.
Remote state backends: using S3/GCS/Azure Blob plus DynamoDB/GCS locking prevents concurrent writes; remote backends scale to large teams and enable CI usage.
State locking & consistency: locking prevents concurrent applys; explain how you would implement locks in CI to avoid race conditions and partial applies.
Modules design: prefer small, opinionated modules with clear inputs/outputs and semantic versioning (use vX.Y.Z tags); reuse via registries or shared repos, avoid over‑generalization.
Workspaces vs. separate state: terraform workspace is fine for small testing; for production, prefer separate state files per environment to reduce blast radius and accidental cross-env changes.
Drift detection: run terraform plan against remote state in CI regularly or on PRs; integrate drift checks and alerting; automated remediation requires human-approved plans for nontrivial changes.
Importing existing resources: use terraform import to adopt resources safely; afterwards update configuration to match imported attributes to avoid immediate diffs.
Resource lifecycle meta-arguments: lifecycle { create_before_destroy = true }, prevent_destroy, and ignore_changes are powerful but can mask problems; use sparingly and document rationale.
Idempotence and immutability: favor immutable patterns (replace resources) for safe upgrades, but account for resources that must be in-place (databases, stateful disks) and plan for migration steps.
Secrets & state security: never store secrets in plaintext in state; use encryption at rest (backend provider), restrict IAM roles, and consider partial state scrubbing or external secret sources.
Testing and validation: use terraform validate, terraform fmt, static linters (tflint), and unit/integration tests (e.g., Terratest) in CI to catch module regressions before apply.

Tip: tag resources with ownership and environment metadata to make drift triage and auditing straightforward.

Worked example — "Design Terraform modules and state layout for multi-environment microservices"

First 30s framing: clarify number of environments (dev/stage/prod), deployment cadence per team, whether environments share cloud accounts, and constraints on resource isolation or cross-account access. Skeleton answer pillars: (1) module topology — a small base module per service (networking, compute, IAM) plus environment overlays; (2) state layout — separate remote state per environment and per service to minimize blast radius; (3) CI integration — plan on PRs, apply on merge with approval gates for prod. Call out a tradeoff: many small state files improve safety but add operational complexity for cross-resource references (use remote_state data sources or outputs with careful access controls). Close by noting incremental improvements: "if more time, I'd add automated drift scans, finer-grained module tests with Terratest, and documented migration playbooks for state moves."

A second angle — "Detect and remediate Terraform drift in CI/CD pipeline"

Here the constraints change: the focus is on automation and safety under continuous delivery. You’d propose a pipeline where every PR runs terraform plan against the current remote state and fails if drift is detected unexpectedly. For remediation, prefer an explicit human-reviewed apply for nontrivial changes; for trivial config fixes (tagging, metadata), a safe auto-apply policy can exist with strict IAM/approval controls. Discuss performance: live plan for large state can be slow — mitigate with targeted plans using -target sparingly and scheduled full-plan jobs off-peak. Mention one design decision: choose to alert and block merges when drift originates outside CI (manual console changes), forcing teams to import or reconcile rather than silently overwrite.

Common pitfalls

Pitfall: Treating the state as a backup rather than the authoritative source.
Many engineers export resources manually and then overwrite state with new configs, causing resource duplication or orphaned resources; always import resources into state and reconcile config to imported attributes first.

Pitfall: Overusing ignore_changes or prevent_destroy.
These meta-arguments can hide real problems and become technical debt; interviewers will prefer small, explicit exceptions with clear justification and tickets than broad, permanent ignores.

Pitfall: Relying on terraform workspace for environment isolation at scale.
Workspaces can leak configuration and make accidental cross-env updates easier; for multi-team production systems, separate state files/accounts per environment are safer despite added indirection.

Connections

Interviewers may pivot to adjacent topics like CI/CD pipeline design (how terraform runs in Jenkins/GitHub Actions), configuration drift detection across other IaC tools (e.g., kubectl vs. terraform), or testing strategy (unit tests with terratest and policy-as-code checks). Be ready to map your Terraform choices to app deployment, monitoring, and incident recovery workflows.