toasterson/Anima

Watch

Fork 0

WIP: Test-network subnet collision: something inside dind pins akhmedu-test-net to 172.20.0.0/29 and severs every runner from the daemon #598

Draft

toasterson wants to merge 1 commit from claude/wi-019ff0c1-test-network-subnet-collision-something into main

toasterson commented

2026-08-16 13:41:07 +00:00

Owner

Anima work item 019ff0c1-43b5-75c1-bc85-c8d42b44f744.

Re-filed from akh-medu WI-290 (019fe202-4243-76b3-81a0-8d39aa870858), 2026-08-11. That item was opened against the akh-medu repo on the assumption the subnet was pinned in akh-medu's test harness. A search session on 2026-08-11 08:42Z established that it is not: zero hits in akh-medu for 172.20.0.0 (any prefix), akhmedu-test-net, test-net, require_docker, dind, docker network create / create_network; no bollard / shiplift / docker_api in Cargo.toml; and .solstice/job.sh runs only cargo check + cargo test --lib — no Docker-gated integration tests and no network creation anywhere in the tree. The network is created by something on the Anima runner/dind side, which is why the 2026-08-08 repair was performed on archibald rather than in a repo. akh-medu WI-290 has been cancelled in favour of this item.

Original diagnosis (verbatim from WI-290)

Diagnosed and repaired by hand on archibald 2026-08-08. The repair is not durable — whatever creates the network recreates the collision.

The Anima fleet's compose network anima_default is 172.20.0.0/16; anima-dind-1 sits at 172.20.0.2 and the runners get .3, .4, and so on. Inside dind, a network akhmedu-test-net is created pinned to 172.20.0.0/29. That produced this route table inside dind:

default via 172.20.0.1 dev eth0
172.17.0.0/16 dev docker0            src 172.17.0.1
172.20.0.0/29 dev br-8e09ee52adf7    src 172.20.0.1   <-- more specific, wins
172.20.0.0/16 dev eth0               src 172.20.0.2

172.20.0.0/29 covers .0–.7, exactly where the runners live. anima-runner-1-1 was 172.20.0.4. A runner's SYN arrived on dind's eth0 and dind routed the SYN-ACK into its internal test bridge instead of back out eth0. Every connection to tcp://dind:2375 timed out (curl rc=28) while every surface check looked healthy: DNS resolved, dockerd was listening on 0.0.0.0:2375, TLS was off, both containers were verified on the same host bridge, and iptables was irrelevant (br_netfilter is not loaded on archibald).

Consequences, both silent:

The runner janitor never pruned. anima_runner::janitor failed every pass on volume ls --filter dangling=true and container prune. Found 33 dangling volumes, 1.61 GB — 97% of dind's local volume storage — with nothing reclaiming it. On archibald a full root filesystem crash-loops forgejo-postgres and takes down git, the registry and CI together, so an unbounded leak here is an outage generator.
Every Docker-gated test skipped rather than ran, and the pre-push gate scored that as Pass. Any runner session reporting "tests passed" said nothing about DB-backed behaviour for as long as this persisted.

The failure was self-sustaining: the stale network is precisely the debris the janitor reaps, and the collision is what disabled the janitor.

Removing akhmedu-test-net (no containers attached) restored connectivity immediately — curl http://dind:2375/_ping returns OK, and both janitor commands return rc=0 from inside the runners.

Scope for this item

First, find the creator. It is not in akh-medu. Candidates: the runner-side test harness, a compose file or provisioning script in the Anima deployment, or a leftover fixture in the dind image. Naming the creator is the first deliverable — everything below depends on it.

Then, in preference order:

Let Docker auto-assign the test network's subnet instead of hard-coding one. If a fixed range is genuinely required, move it well away from 172.20.0.0/16 (for example 10.211.0.0/24).
Delete the network on teardown, including on failure paths. It survived because nothing removed it.
Belt-and-braces: set default-address-pools in dind's daemon.json to something like 10.201.0.0/16, so no auto-assigned network inside dind can ever overlap the outer fleet subnet. This needs a dind restart, so schedule it rather than doing it mid-flight.

Acceptance

Running the suite that creates this network to completion, and again after an induced mid-suite failure, leaves no network overlapping 172.20.0.0/16 inside dind, and docker info from anima-runner-1-1 keeps working throughout.

WI-527 (019fe1d3-812e-76d1-bc76-e10c7f6ea3b1) — the runner-side consequences: janitor silence and the skipped-test reporting gap. That item is in testing; this one is the upstream cause.
akh-medu WI-290 (019fe202-4243-76b3-81a0-8d39aa870858) — cancelled, superseded by this item.

Anima work item `019ff0c1-43b5-75c1-bc85-c8d42b44f744`. **Re-filed from akh-medu WI-290 (`019fe202-4243-76b3-81a0-8d39aa870858`), 2026-08-11.** That item was opened against the akh-medu repo on the assumption the subnet was pinned in akh-medu's test harness. A search session on 2026-08-11 08:42Z established that it is not: zero hits in akh-medu for `172.20.0.0` (any prefix), `akhmedu-test-net`, `test-net`, `require_docker`, `dind`, `docker network create` / `create_network`; no `bollard` / `shiplift` / `docker_api` in `Cargo.toml`; and `.solstice/job.sh` runs only `cargo check` + `cargo test --lib` — no Docker-gated integration tests and no network creation anywhere in the tree. The network is created by something on the Anima runner/dind side, which is why the 2026-08-08 repair was performed on archibald rather than in a repo. akh-medu WI-290 has been cancelled in favour of this item. ## Original diagnosis (verbatim from WI-290) Diagnosed and repaired by hand on archibald 2026-08-08. The repair is not durable — whatever creates the network recreates the collision. The Anima fleet's compose network `anima_default` is `172.20.0.0/16`; `anima-dind-1` sits at `172.20.0.2` and the runners get `.3`, `.4`, and so on. Inside dind, a network `akhmedu-test-net` is created pinned to **`172.20.0.0/29`**. That produced this route table inside dind: ``` default via 172.20.0.1 dev eth0 172.17.0.0/16 dev docker0 src 172.17.0.1 172.20.0.0/29 dev br-8e09ee52adf7 src 172.20.0.1 <-- more specific, wins 172.20.0.0/16 dev eth0 src 172.20.0.2 ``` `172.20.0.0/29` covers `.0`–`.7`, exactly where the runners live. `anima-runner-1-1` was `172.20.0.4`. A runner's SYN arrived on dind's eth0 and dind routed the SYN-ACK into its internal test bridge instead of back out eth0. Every connection to `tcp://dind:2375` timed out (`curl` rc=28) while every surface check looked healthy: DNS resolved, dockerd was listening on `0.0.0.0:2375`, TLS was off, both containers were verified on the same host bridge, and iptables was irrelevant (`br_netfilter` is not loaded on archibald). Consequences, both silent: 1. **The runner janitor never pruned.** `anima_runner::janitor` failed every pass on `volume ls --filter dangling=true` and `container prune`. Found 33 dangling volumes, 1.61 GB — 97% of dind's local volume storage — with nothing reclaiming it. On archibald a full root filesystem crash-loops forgejo-postgres and takes down git, the registry and CI together, so an unbounded leak here is an outage generator. 2. **Every Docker-gated test skipped rather than ran**, and the pre-push gate scored that as `Pass`. Any runner session reporting "tests passed" said nothing about DB-backed behaviour for as long as this persisted. The failure was self-sustaining: the stale network is precisely the debris the janitor reaps, and the collision is what disabled the janitor. Removing `akhmedu-test-net` (no containers attached) restored connectivity immediately — `curl http://dind:2375/_ping` returns `OK`, and both janitor commands return rc=0 from inside the runners. ## Scope for this item **First, find the creator.** It is not in akh-medu. Candidates: the runner-side test harness, a compose file or provisioning script in the Anima deployment, or a leftover fixture in the dind image. Naming the creator is the first deliverable — everything below depends on it. Then, in preference order: 1. Let Docker auto-assign the test network's subnet instead of hard-coding one. If a fixed range is genuinely required, move it well away from `172.20.0.0/16` (for example `10.211.0.0/24`). 2. Delete the network on teardown, **including on failure paths**. It survived because nothing removed it. 3. Belt-and-braces: set `default-address-pools` in dind's `daemon.json` to something like `10.201.0.0/16`, so no auto-assigned network inside dind can ever overlap the outer fleet subnet. This needs a dind restart, so schedule it rather than doing it mid-flight. ## Acceptance Running the suite that creates this network to completion, and again after an induced mid-suite failure, leaves no network overlapping `172.20.0.0/16` inside dind, and `docker info` from `anima-runner-1-1` keeps working throughout. ## Related - **WI-527** (`019fe1d3-812e-76d1-bc76-e10c7f6ea3b1`) — the runner-side consequences: janitor silence and the skipped-test reporting gap. That item is in `testing`; this one is the upstream cause. - akh-medu **WI-290** (`019fe202-4243-76b3-81a0-8d39aa870858`) — cancelled, superseded by this item.

toasterson added 1 commit

2026-08-16 13:41:07 +00:00

Test-network subnet collision: something inside dind pins akhmedu-test-net to 172.20.0.0/29 and severs every runner from the daemon df60068c2f

Anima implementation session.

toasterson force-pushed claude/wi-019ff0c1-test-network-subnet-collision-something from df60068c2f to e2841f7c35

2026-08-18 11:07:41 +00:00

Compare

toasterson added 1 commit

2026-08-18 11:42:41 +00:00

Add trailing newline to dind-daemon.json 5f47ea6474

Anima implementation session.

toasterson force-pushed claude/wi-019ff0c1-test-network-subnet-collision-something from 5f47ea6474 to a145fab8ed

2026-08-18 12:21:14 +00:00

Compare

toasterson force-pushed claude/wi-019ff0c1-test-network-subnet-collision-something from a145fab8ed to fbf8912b0c

2026-08-18 20:43:57 +00:00

Compare

This pull request is marked as a work in progress.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin claude/wi-019ff0c1-test-network-subnet-collision-something:claude/wi-019ff0c1-test-network-subnet-collision-something

git switch claude/wi-019ff0c1-test-network-subnet-collision-something

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff claude/wi-019ff0c1-test-network-subnet-collision-something

git switch claude/wi-019ff0c1-test-network-subnet-collision-something

git rebase main

git switch main

git merge --ff-only claude/wi-019ff0c1-test-network-subnet-collision-something

git switch claude/wi-019ff0c1-test-network-subnet-collision-something

git rebase main

git switch main

git merge --no-ff claude/wi-019ff0c1-test-network-subnet-collision-something

git switch main

git merge --squash claude/wi-019ff0c1-test-network-subnet-collision-something