Nahum LitvinKubernetes, containers and sandboxes, from the production side

Notes from running code we have never seen.

Notes on Kubernetes, containers and sandboxing: the root cause, the fix, and the upstream PR when there is one.

Latest

The Go error recover() cannot catch

concurrent map write is not a panic, so no recover() catches it and the whole process exits.

Go5 min read

Start here

  1. A node that reported Ready and started nothing: four parts, one containerd fixFrom an outage that lasted hours to a client-side timeout in containerd, through a merge, a revert and a second PR.
  2. One lost signal, five days stuck, 45,000 frozen threads: fixing a gVisor hang upstreamOne lost signal hangs pod deletion. The maintainer's smaller fix was the better one.
  3. containerd part 2: 17 lines, 40 days, 30 reviewsWriting the fix was easy. The review was where the engineering happened.

Notes

More writing