drainwatch

When a Kubernetes pod terminates, what actually happens to its established connections?

drainwatch answers that with recorded evidence. It holds open long-lived TCP and UDP flows against a workload, triggers termination, and emits a machine-readable timeline showing exactly when SIGTERM was delivered, when the endpoint left rotation, and when — and how — each flow died or drained.

make demo

That creates a two-node kind cluster, builds and side-loads the probe image, runs one trial with the defaults, prints the table below, and leaves report.json in ./out/.

SUMMARY
  PROTO  TOTAL  DRAINED  SEVERED  READ_TIMEOUT  SURVIVED_WINDOW  NOT_MEASURED
  tcp    10     10       0        0             0                0
  udp    10     0        10       0             0                0

  trigger -> sigterm               63 ms
  sigterm -> endpoint ready:false  -8 ms
  trigger -> endpoint removed      25739 ms
  trigger -> container terminated  25599 ms
  sigterm -> last flow terminal    25006 ms

FLOWS
  ID        PROTO  OUTCOME              T_TERMINAL  LAST_SEQ  DETAIL
  tcp-0001  tcp    drained-clean-close  25066 ms    70        fin after drain announcement
  udp-0001  udp    severed              2382 ms     24        answered by probe instance drainwatch-probe-5497cd7d46-tvb75, was drainwatch-probe-5497cd7d46-nj9t4

(The real output has one row per flow and a TIMELINE section above; 18 flow rows are elided here. The rest is verbatim.)

That is a real recorded run, not a mock-up: it is the drain arm of results/2026-08-31-kind-v1.34.0/, on kind with Kubernetes v1.34.0 and kube-proxy in iptables mode. Two things in it are worth a second look, and both are the point of the tool:

• sigterm -> endpoint ready:false is −8 ms. Not a time machine — the two timestamps come from different hosts' clocks, and the report says so in a note. The honest reading is "simultaneous to within measurement resolution", which already contradicts the common assumption that endpoint removal precedes SIGTERM by a usable margin.
• The UDP flows were severed at 2.38 s while the probe was still serving. They were silently re-homed onto the replacement pod the ReplicaSet created. A graceful shutdown routine bought the TCP connections 25 seconds and bought the UDP flows nothing.

---

What it measures

For every trial, drainwatch records four independent event sources and merges them into one timeline:

Source	What it contributes	Precision
orchestrator	the trigger instant (t=0), settle and observation-window boundaries	exact (one monotonic clock)
client	per-flow terminal events: when and how each flow ended	exact (same clock)
k8s	pod deletionTimestamp, container termination and exit code, EndpointSlice ready:false, endpoint removal	receipt time — an upper bound on when it happened
probe	SIGTERM delivery, readiness flip, drain start/deadline, per-flow close	approximate — cross-host wall clocks, tens of ms

The per-flow outcome is a closed enum. There is no "other", and no empty string:

Outcome	Meaning
drained-clean-close	the server announced the close on the wire, then sent FIN. The only outcome that licenses the word "drained".
severed	the flow ended without the server completing it: RST, an unannounced FIN mid-heartbeat, EPIPE, ICMP port unreachable, or UDP silence. The mechanism is in the flow's detail.
read-timeout	no bytes for the whole flow timeout, and no socket error. The flow went quiet without provably dying.
survived-observation-window	the flow was still alive when the window closed. Not a drained flow — a flow whose fate was not observed.
not-measured	no usable observation exists for this flow.

Three arms, one variable

make reproduce runs three trials that differ only in what the application does on SIGTERM (DRAIN_BEHAVIOR on the probe):

• drain — stop accepting, fail readiness, keep serving established flows, then announce and close them when the drain window expires.
• exit-now — abandon established flows and exit(0).
• ignore — keep serving, never exit; the kubelet sends SIGKILL at the grace-period boundary, which appears in the timeline as container_terminated exitCode=137.

Flow counts, grace period, trigger and settle time are held constant, so any difference between the three reports is attributable to the SIGTERM behaviour and nothing else.

Design rules

These are not aspirations; they are enforced in code and covered by tests.

1. Evidence over assertion. Every run emits a full trial record: environment, configuration, method, timeline, per-flow outcomes. A run that cannot record its environment refuses to start.
2. Preflight, and fail loudly. Six checks run before anything is measured. The first failure aborts the run and names the check, the invariant it enforces, and what to look at. A broken harness never emits a report.
3. Deterministic where possible. No RNG anywhere in the measurement path. Flow IDs are sequential (tcp-0001, udp-0001, …). Timestamps are monotonic wherever the comparison is local. Outcome classification is a pure function of the event history.
4. The absence of an observation is not-measured, never zero. Every optional quantity is a JSON null, in the report and in the table alike.

Where cross-source clock alignment is approximate, the report says so in a clock_note field rather than pretending to sub-millisecond precision, and every affected timeline entry carries "approximate": true.

Install and run

Requirements: Go 1.22+, Docker, and kind. No other runtime dependencies — drainwatch is a single static binary built from the standard library plus k8s.io/client-go.

On macOS, make demo and make reproduce run the orchestrator inside the kind node rather than on your host. Docker Desktop does not reliably forward published UDP ports — in testing the mapping carried the first ~50 datagrams and then stopped, while TCP kept working — so a host-side run fails the udp-flows-replying preflight check. UDP through the same NodePort works fine from inside the cluster, so scripts/dwrun.sh moves the orchestrator across that boundary and copies the reports back. This is automatic; set DRAINWATCH_RUNNER=host to force the host-side path anyway.

git clone https://github.com/jaynirmal15/drainwatch
cd drainwatch
make demo

Individual steps, if you would rather run them yourself:

make build          # bin/drainwatch, with version and commit stamped in
make kind-up        # create the 2-node cluster and side-load the probe image
bin/drainwatch run --out out
make kind-down      # tear the cluster down

Installing the binary on its own

go install github.com/jaynirmal15/drainwatch/cmd/drainwatch@latest

If drainwatch isn't found after installing, add Go's bin directory to your PATH:

export PATH="$PATH:$(go env GOPATH)/bin"

drainwatch run still expects to be started from a clone: it reads deploy/manifests/probe.yaml from the working directory (override with --manifest), and the probe image it deploys is built by make kind-up.

The three subcommands

One binary.

drainwatch run     # the orchestrator: deploy, verify, hold, trigger, observe, report
drainwatch probe   # the in-cluster workload whose termination is measured
drainwatch client  # the flow generator, exposed for manual runs

drainwatch run --help lists every flag. The ones you are most likely to change:

--drain-behavior   drain | exit-now | ignore     (default drain)
--trigger          delete | evict | scale        (default delete)
--grace-period     terminationGracePeriodSeconds (default 30)
--tcp-flows        long-lived TCP flows          (default 10)
--udp-flows        UDP flows                     (default 10)
--settle-seconds   steady state before trigger   (default 10)
--observe-timeout  observation window            (default grace-period + 30)
--repeat           trials into one --out dir     (default 1)
--out              output directory              (default ./out)

The report

report.json is one trial plus the context needed to interpret it.

{
  "drainwatch_version": "0.1.0",
  "git_commit": "abc1234",
  "environment": {
    "kubernetes_version": "v1.31.0",
    "node_count": 2,
    "nodes": [{ "name": "drainwatch-worker", "container_runtime": "containerd://1.7.18" }],
    "kube_proxy_mode": "iptables",
    "cni": "kindnet (identified by DaemonSet name; best effort)",
    "os": "darwin", "arch": "arm64",
    "wall_clock_start": "2026-01-01T12:00:00Z",
    "warnings": []
  },
  "trial": {
    "id": "trial-001",
    "config": { "tcp_flows": 10, "udp_flows": 10, "grace_period_seconds": 30,
                "drain_behavior": "drain", "trigger": "delete" },
    "timeline": [
      { "t_ms": 0,    "source": "orchestrator", "event": "trigger_issued" },
      { "t_ms": 112,  "source": "probe",        "event": "sigterm_received", "approximate": true },
      { "t_ms": 340,  "source": "k8s",          "event": "endpointslice_ready_false" },
      { "t_ms": 25390,"source": "k8s",          "event": "endpointslice_endpoint_removed" }
    ],
    "flows": [
      { "id": "tcp-0001", "proto": "tcp",
        "outcome": "drained-clean-close", "t_terminal_ms": 25133,
        "last_heartbeat_seq": 71 },
      { "id": "udp-0004", "proto": "udp",
        "outcome": "severed", "t_terminal_ms": 3550,
        "last_answered_seq": 17 }
    ],
    "summary": {
      "tcp": { "total": 10, "drained": 10, "severed": 0, "read_timeout": 0,
               "survived_window": 0, "not_measured": 0 },
      "udp": { "total": 10, "drained": 0, "severed": 10, "read_timeout": 0,
               "survived_window": 0, "not_measured": 0 },
      "sigterm_to_ready_false_ms": 228,
      "sigterm_to_last_flow_terminal_ms": 25022
    },
    "clock_note": "t_ms is milliseconds relative to trigger_issued. ..."
  }
}

With --repeat N, each trial lands in out/trial-00N/report.json and a summary.json collects the whole run.

Findings

Recorded results, the full methodology, and the reasoning behind each design choice live in EXPERIMENTS.md. The raw reports they are drawn from are committed under results/.

The headline from the first recorded run — same cluster, same trigger, same 30 s grace period, one variable changed:

SIGTERM behaviour	TCP flows	When they ended	Container exit
exit-now	10/10 severed (RST)	46 ms	exitCode=0
drain	10/10 drained cleanly	25.07 s	exitCode=0
ignore	10/10 severed (SIGKILL)	30.06 s	exitCode=137

UDP was severed in all three arms by being re-homed onto the replacement pod. In exit-now, connections died ~330 ms before the endpoint was removed from the EndpointSlice.

Companion articles:

• Article 1 — placeholder link, forthcoming.
• Article 2 — placeholder link, forthcoming.

Explicitly out of scope for v0.1

These are not oversights. They are stated so that no result from drainwatch is read as covering them.

• Attaching to an arbitrary existing workload (--workload attach). The flag exists, prints "not implemented in v0.1", and exits 2. v0.1 measures a workload it deploys itself, so that the probe's SIGTERM handling and readiness behaviour are known quantities rather than assumptions.
• NLB / cloud load-balancer deregistration timing. drainwatch observes the EndpointSlice, not any cloud provider's target group.
• ECS, service meshes, and Windows nodes.
• Multi-pod and rolling-update scenarios. v0.1 runs exactly one replica, so that a flow's fate is attributable to one pod's termination.
• Any claim about conntrack internals beyond what the flow outcomes show. drainwatch records what happened to the flows; it does not read or interpret conntrack tables.

Development

make test    # unit tests: flow state machine, outcome classification, report schema
make lint    # gofmt check plus go vet
make help    # every target

The test suite includes loopback integration tests that wire the real probe to the real flow generator with no Kubernetes involved, asserting that each of the three SIGTERM behaviours produces the wire events the classifier expects — a drain puts an announcement on the wire before its FIN, and an abrupt exit does not.

Those tests cannot cover kube-proxy, EndpointSlices or the grace-period boundary. Three defects in drainwatch were found only by running against a real cluster — including one where re-homed UDP flows were reported as having survived untouched. They are described in EXPERIMENTS.md.

Licence

MIT. See LICENSE.