CI/CD, Jenkins, Blue-Green, Canary, And Rollbacks

What's being tested

Candidates must show practical mastery of CI/CD pipeline design and deployment strategies that enable safe, repeatable releases: you should reason about build artifacts, automated tests, deployment orchestration, traffic shifting, health gating, and rollback mechanics. Interviewers probe for clear tradeoffs between Blue-Green and Canary approaches, how to automate rollback triggers, and what an engineer owns (pipeline code, deployment manifests, health checks) versus what infra teams own (cluster provisioning, load balancers). Expect to justify choices with failure modes, metrics, and recovery SLAs.

Core knowledge

CI/CD fundamentals: separate build, test, and deploy stages; produce an immutable artifact (e.g., Docker image tagged by SHA) that moves through environments to ensure repeatability and traceability.
Jenkins pipeline patterns: use declarative pipelines, stage-level post steps for cleanup, credentials binding, and store artifacts in a registry (ECR, DockerHub) and metadata in the pipeline (build number, git SHA).
Blue-Green deployment: maintain two identical environments (blue, green), switch traffic at router/ELB level for near-instant rollback by flipping the virtual IP or load-balancer target set; zero-downtime but requires double capacity.
Canary deployment: progressively shift traffic (e.g., 1%, 5%, 25%, 100%) to new version; useful when capacity is limited and you want fine-grained risk control; requires automated metrics gating.
Automated canary analysis (ACA): compare baseline vs candidate on metrics like error rate, p95 latency, business metrics; use statistical thresholds and time windows to avoid false positives (e.g., require sustained deviation for T minutes).
Rollback types: instant rollback (flip back traffic or redeploy previous image), graceful rollback (drain connections, let sessions expire), and stateful rollback caveats (DB migrations are often irreversible—apply backward-compatible migration patterns).
Health checks and readiness probes: liveness vs readiness in Kubernetes; readiness gating prevents load balancer from sending traffic until service is healthy; build health-check endpoints that are cheap, deterministic, and reflect real user paths.
Observability signals: track error rate, latency percentiles (p50, p95, p99), saturation (CPU, memory), request counts, and key business metrics; compute relative change and signal-to-noise; maintain rolling windows (e.g., compare last 5m vs previous 30m).
Feature flags: separate code rollout from deployment; use feature flags for controlled exposure and fast rollback by toggling flags rather than redeploying; beware flag debt and stale branches.
Idempotency and migration safety: design deploys to be idempotent; for DB schema changes, prefer backward-compatible multi-step migrations (add column → backfill → switch read → remove old code).
Testing strategy: unit tests in CI, integration tests in an isolated environment, smoke tests post-deploy (synthetic requests hitting core flows), and end-to-end checks that anchor to business metrics.
Cost / capacity tradeoffs: Blue-Green doubles runtime capacity; Canary reduces capacity overhead but increases orchestration complexity. Quantify: double capacity cost = 2x active instance count; canary requires fine-grained routing and metrics automation.

Tip: automate gating rules in the pipeline (e.g., fail if error rate increases by >50% for 5 minutes), and expose a manual override for on-call.

Worked example

Design a CI/CD pipeline using Jenkins to support Blue-Green deployments with automated rollback. Frame the problem: ask about target platform (Kubernetes, EC2 behind ELB), traffic-shift mechanism, acceptable downtime, and rollback SLOs. Skeleton answer pillars: 1) build immutable Docker images and push to registry with metadata (git SHA, build ID); 2) automated test stages (unit, integration, contract) then deploy to staging for smoke tests; 3) deploy to green (the new environment) while blue serves traffic, run health checks and smoke tests; 4) switch traffic at ELB/ingress when green passes, monitor key metrics and trigger rollback on predefined conditions. Flag a tradeoff: blue-green gives immediate rollback (flip back) but requires 2x instances; if double capacity is unaffordable, prefer canary. Close with extension: "if I had more time, I'd add automated canary analysis for metric drift, integrate a feature-flag toggle for risky features, and wire the pipeline into on-call runbooks."

A second angle

Implementing a Canary rollout that progressively shifts traffic with automated health gating changes the orchestration details. Instead of two full environments, you run both versions concurrently on the same cluster and use traffic-splitting at the ingress or service mesh (Istio) level. Key constraints: determine canary percentages and cadence (e.g., 1% for 10 minutes → 5% for 15 minutes → 25% for 30 minutes), define statistical tests to compare baseline/canary metrics, and ensure sample sizes are sufficient for signal detection. This framing emphasizes metric sensitivity and automation (ACA), and it forces you to reason about routing, sticky sessions, and test-exposure bias (canary users may not represent all user segments), unlike blue-green which is binary.

Common pitfalls

Pitfall: Underestimating observability needs. Many engineers implement traffic switch logic but fail to define the exact metrics, thresholds, windows, and rollback playbook; this leads to delayed detection or noisy false positives.

Pitfall: Treating DB migrations as reversible. A tempting quick rollback is to redeploy the old binary, but irreversible schema changes will break the old code; always plan backward-compatible migrations and multi-step rollout.

Pitfall: Over-communicating detail or under-scoping responsibilities. As a Software Engineer, own pipeline code, health checks, and artifact immutability; do not assume you'll provision infra or change load-balancer internals without clarifying team boundaries.

Connections

Interviewers may pivot to feature-flag strategies, the design of health-check endpoints and synthetic testing, or how services coordinate schema migrations (backward-compatible migrations and data pipelines). They may also ask about integrating service meshes (Istio) or CD tools (Spinnaker) for advanced traffic control.