Kubernetes, EKS, And Container Operations

What's being tested

Interviewers are probing whether you can design, run, and debug cloud-native applications that behave correctly inside Kubernetes on AWS (`EKS`) without relying on a cluster administrator. Expect questions that test your knowledge of container lifecycle, resource accounting and QoS, deployment strategies (zero-downtime, canary), observability/debugging patterns, and how application design interacts with Kubernetes primitives. Amazon cares because service owners must build resilient microservices that degrade gracefully, autoscale predictably, and minimize operational blast radius.

Core knowledge

Pod / Container lifecycle: A Pod is the atomic deployable unit; container termination receives SIGTERM then SIGKILL after terminationGracePeriodSeconds. Implement graceful shutdown handlers and preStop hooks to drain connections.
Probes: readiness / liveness / startup: readiness probe controls Service endpoints (prevents traffic); liveness probe restarts unhealthy containers; startupProbe avoids premature liveness checks during long init. Misconfiguring these causes traffic drops or crash loops.
Resources, requests, limits, and QoS: CPU is specified in millicores (100m = 0.1 CPU). Requests drive scheduler placement; limits enforce cgroups. QoS classes: Guaranteed (requests==limits), Burstable, BestEffort. OOM kills happen when memory usage > limit.
Deployments, StatefulSets, DaemonSets: Use Deployment for stateless replicas and rolling updates; StatefulSet for stable network IDs/ordered startup and persistent storage; DaemonSet for one pod per node (logging/monitoring).
RollingUpdate and canary controls: RollingUpdate uses maxSurge and maxUnavailable (e.g., maxSurge:1, maxUnavailable:0 for zero-downtime). Canary patterns can be implemented with split Deployments, Ingress weight, or tools like Flagger (progressive delivery).
Service types and ingress: Service ClusterIP for internal DNS, LoadBalancer (creates an AWS ELB/NLB/ALB via controller) for external traffic, NodePort rarely for production. Ingress requires a controller (`AWS Load Balancer Controller` for ALB integration).
Persistent storage: Use PersistentVolume / PersistentVolumeClaim for durability; prefer StatefulSet for single-writer volumes. For shared storage, use appropriate RWX-backed solutions; ephemeral data belongs on emptyDir.
Config & secrets: Use ConfigMap for non-sensitive config, Secret for credentials (note: Secret is base64-encoded, not encrypted by default—use KMS/`sealer` or a secrets operator). Mount vs env var tradeoffs: files allow rotation without restart.
ServiceAccount & IAM: Grant pod AWS permissions with ServiceAccount and IRSA on `EKS` rather than node IAM roles—least-privilege and auditable.
Autoscaling basics: HPA uses metrics (CPU, custom via `metrics-server` / Prometheus). Desired replicas ≈ ceil((currentMetric / targetMetric) * currentReplicas). Balance HPA with pod startup time and target latency.
Observability & logging: Emit stdout/stderr (ingested by node agent). Use metrics (`Prometheus`), structured logs, and tracing (`OpenTelemetry` / `Jaeger`). For `EKS`, forward logs/metrics to `CloudWatch` or a centralized stack via Fluentd/collector.
Image best-practices: Use immutable tags, minimal base images, scan registries (`ECR`) for vulnerabilities, and set imagePullPolicy appropriately. Avoid latest for production.
Networking and policies: Service DNS (<name>.<namespace>.svc.cluster.local) enables discovery. Use NetworkPolicy for pod-level ingress/egress restrictions; expect transient DNS caching and aggressive connection reuse from long-lived clients.

Worked example — "Design a zero-downtime deployment on EKS"

First 30s: clarify assumptions—is the service stateless? Any long-lived TCP connections? Expected RPS and acceptable latency bump? Are DB schema changes involved? Skeleton of the response: (1) Use a Deployment with RollingUpdate and maxSurge:1, maxUnavailable:0 to ensure capacity. (2) Implement robust readiness probes that only return success after warmup, and catch SIGTERM to stop accepting new requests and drain in-flight work. (3) Ensure graceful deregistration from the AWS load balancer (use `AWS Load Balancer Controller` so Kubernetes lifecycle triggers target group deregistration with connection draining). (4) Autoscaling via HPA and pre-warmed capacity to absorb surge. Explicit tradeoff: setting maxUnavailable:0 increases resource usage during deploys (extra pod) but reduces user-visible errors. Close: instrument `p99` latency and error rate, and add an automated canary rollout (or Flagger) for faster rollback if metrics regress—if more time, propose DB migration strategies (expand-contract) and automated health-based rollback.

A second angle — "Investigate intermittent 5xxs and pod restarts"

Frame the problem: get concrete symptoms—are restarts due to crashes (`kubectl get pods` shows restart count) or external 5xxs? Action pillars: (1) check pod events (`kubectl describe pod`) for OOMKilled or probe failures; (2) fetch container logs (`kubectl logs --previous`) and application traces to find exceptions or GC pauses; (3) examine resource metrics (CPU/memory) and node conditions (DiskPressure) via `metrics-server` or Prometheus; (4) verify probes—false liveness kills cause frequent restarts and downstream 5xxs. Tradeoff to call out: increasing memory limits hides leaks but raises cost—better to profile and fix leaking allocations. Finish by proposing mitigations: tune probes, set terminationGracePeriodSeconds, add circuit-breaking and retries at client/ingress, and plan a gradual rollout once root cause fixed.

Common pitfalls

Pitfall: Confusing readiness and liveness probes.

Many candidates treat them interchangeably; the immediate failure mode is removing a pod from rotation vs killing it. Always explain how each affects traffic and restarts.

Pitfall: Proposing cluster-admin fixes for app-level problems.

Suggesting node autoscaling or changing CNI configs is tempting but overkill for application bugs; show you first exhausted app-level fixes (resource tuning, probes, graceful shutdown).

Pitfall: Ignoring shutdown semantics.

A common shallow answer is “handle SIGTERM later.” Better: say how long your terminationGracePeriodSeconds is, what your preStop does, and how the load balancer deregisters the pod to avoid dropped requests.

Connections

Interviewers may pivot to adjacent topics like CI/CD (image promotion, immutable artifacts, deployment pipelines), service meshes and traffic shaping (`Istio`, `AWS App Mesh`), or deeper infra concerns (cluster autoscaling, node groups) — be ready to scope answers to application-level responsibilities.