Kubernetes request path: pods vs nodes, Service types, DNS TTLs and 5xx debugging

The interviews were for an infrastructure team, and every round included a run of fundamentals questions about how traffic reaches a service running on Kubernetes and how to troubleshoot it. Answer them as one connected walkthrough, as an engineer who operates such a cluster. ### Clarifying Questions - Is the cluster on a managed cloud service with a cloud load balancer in front, or on bare metal? - Does external traffic enter through an Ingress controller, a LoadBalancer Service, or a NodePort directly? - Which DNS resolvers sit between the browser and the authoritative servers, and does the cluster run its own DNS cache? ### Part 1 — Nodes and pods What is the difference between a Kubernetes node and a pod? How does a pod end up on a node, and what do the containers in one pod share? ```hint Machine versus unit of scheduling Think about which one is a machine and which one is the smallest thing the scheduler places, and what happens to each when it fails. ``` #### What This Part Should Cover - Precise definitions and how the two relate - What containers in the same pod share - Pod and node lifecycle on failure, and why pod IPs are not stable ### Part 2 — Service IPs: ClusterIP and NodePort Pods come and go, and their IPs change. What is a Service's IP, and how do the ClusterIP and NodePort Service types differ? How does a packet sent to a Service IP reach a pod? ```hint Nobody owns the address The Service IP is not assigned to any network interface. Ask which component on each node makes traffic to it work. ``` #### What This Part Should Cover - The stable virtual IP, label selectors and the list of ready endpoints behind a Service - ClusterIP versus NodePort, and where LoadBalancer fits - The node-level mechanism that rewrites Service traffic to a pod, and what it balances ### Part 3 — From the browser to a pod A user opens a URL in the browser, and the request is served by a pod inside the cluster. Walk through every hop, from name resolution to the container and back. ```hint Name each rewrite List the hops in order, and at each one say what is looked up or rewritten: the name, the IP, the port, the TLS session, the HTTP host and path. ``` #### What This Part Should Cover - DNS resolution and connection setup, including TLS - The hops from the edge load balancer through the cluster to the pod - Where TLS terminates and where each routing decision is made ### Part 4 — DNS TTL and negative TTL What does a DNS record's TTL control, and what is a negative TTL? Give a production situation where each one causes trouble. ```hint Two kinds of cached answers Consider what a resolver caches when a name exists, and what it caches when the name does not exist yet. ``` #### What This Part Should Cover - Positive caching and how TTL length affects failover and migrations - Negative caching: which answers are cached and where the duration comes from - A concrete failure for each, and how to avoid it ### Part 5 — Debugging 5xx errors Users report intermittent 5xx errors from a service running in the cluster. How do you find the cause? ```hint Who generated the error First find which hop produced the error: a proxy that could not get a good answer from its backend leaves different traces from an application that failed. ``` #### Clarifying Questions for this Part - Is it every request or a fraction, one endpoint or all of them, and did it start with a deploy or a traffic change? - Which status codes exactly: 500, 502, 503, 504, or a mix? #### What This Part Should Cover - Localizing the failing hop from the status code, headers and proxy logs - The Kubernetes signals to check and the common causes behind each code - Fixing, verifying and preventing recurrence ### What a Strong Answer Covers - Precise vocabulary for nodes, pods, Services and endpoints - A hop-by-hop request path that names where each lookup, rewrite and TLS termination happens - How DNS caching changes failover and the visibility of newly created records - A systematic, layer-by-layer approach to 5xx errors backed by specific signals - Trade-offs such as kube-proxy modes, NodePort exposure and TTL length ### Follow-up Questions - What does a LoadBalancer Service add on top of NodePort, and what does `externalTrafficPolicy: Local` change? - Pods resolving an external hostname see slow DNS lookups. Why might the `ndots` setting cause extra queries? - During every rolling update, clients see a short burst of 502s. What is the likely cause and how do you prevent it? - How would you move a public service to a new IP address with minimal disruption, given DNS caching?