Skip to content

🐛 fix(test): gate the NATS test container on server readiness - #329

Merged
konih merged 1 commit into
mainfrom
lane/flake-nats-01-r2
Aug 23, 2026
Merged

🐛 fix(test): gate the NATS test container on server readiness#329
konih merged 1 commit into
mainfrom
lane/flake-nats-01-r2

Conversation

@konih

@konih konih commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

test-integration intermittently reddened main with nats connect: EOF — the NATS
integration tests connected to a container that was reported ready but whose server had
not yet bound its client port.

Root cause

Not a missing wait strategy, but a silently degraded one.

The nats module's default is wait.ForListeningPort("4222/tcp"). That strategy has two
halves: an external dial of the mapped host port, and an internal in-container check. The
internal check execs /bin/sh to read /proc/net/tcp — and nats:2.11 ships no shell, so
it returns errShellNotFound. WaitUntilReady swallows that error
(wait/host_port.go:240-252) and reports success, leaving readiness resting on the external
dial alone.

That dial is a false positive: Docker's port forwarder accepts the connection as soon as the
port is published, before nats-server binds. The forwarder then closes it, and the client
sees EOF — the exact CI signature.

Fix

Gate on the server's own "Server is ready" log line via the Docker log stream, which the
port forwarder cannot fake, in addition to the existing port wait:

container, err := tc.Run(ctx, "nats:2.11",
    testcontainers.WithWaitStrategy(
        wait.ForLog(natsReadyLog),
        wait.ForListeningPort(natsClientPort),
    ),
)

"Server is ready" is an [INF] line, emitted after "Listening for client connections"
regardless of the debug/trace flags the module passes. If the line ever drifts, the strategy
fails loudlywait.ForLog's 60s default startup timeout surfaces as a tc.Run error
that lands in t.Fatalf (a timeout does not match any IsDockerUnavailable substring, so it
cannot be silently downgraded to a skip).

Verification

An independent reviewer verified all five links of the root-cause chain against the
testcontainers source, reproduced the false positive end-to-end to the literal CI error
signature (client read n=0 err=EOF against a container with nothing listening inside),
confirmed the new wait strategy demonstrably takes effect at runtime, and ran 110 test
executions across 86 real containers with 0 failures and 0 skips
.

It also audited the sibling container helpers: internal/sink/bigquery shares the same
pattern and is protected only incidentally — filed as FLAKE-BQ-01.

Scope

One file, //go:build integration, so it is not compiled into task test / task coverage
and the ./internal/... 90% coverage floor is structurally unmoved. go mod tidy produces
no drift (testcontainers-go is already a direct require).

Note: test-integration is an advisory check here, which is precisely why this flake
could redden main without blocking merges.

The nats testcontainer used the module default wait strategy,
wait.ForListeningPort("4222/tcp"). Its in-container check runs via
/bin/sh, which the nats image does not ship, so testcontainers logs
"Shell not found in container" and falls back to the external check
alone: one TCP dial to the mapped host port. Docker's port forwarder
accepts that dial as soon as the port is published, before nats-server
binds, so the container could be declared ready early and the first
client connection was closed immediately - the EOF that turned
test-integration red on main.

Gate on the server's own "Server is ready" line instead (keeping the
port check as a second condition), which closes the window without a
sleep.
@konih
konih merged commit 4c2ce3d into main Aug 23, 2026
33 checks passed
@konih
konih deleted the lane/flake-nats-01-r2 branch August 23, 2026 17:20
@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant