🐛 fix(test): gate the NATS test container on server readiness - #329
Merged
Conversation
The nats testcontainer used the module default wait strategy,
wait.ForListeningPort("4222/tcp"). Its in-container check runs via
/bin/sh, which the nats image does not ship, so testcontainers logs
"Shell not found in container" and falls back to the external check
alone: one TCP dial to the mapped host port. Docker's port forwarder
accepts that dial as soon as the port is published, before nats-server
binds, so the container could be declared ready early and the first
client connection was closed immediately - the EOF that turned
test-integration red on main.
Gate on the server's own "Server is ready" line instead (keeping the
port check as a second condition), which closes the window without a
sleep.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Problem
test-integrationintermittently reddenedmainwithnats connect: EOF— the NATSintegration tests connected to a container that was reported ready but whose server had
not yet bound its client port.
Root cause
Not a missing wait strategy, but a silently degraded one.
The
natsmodule's default iswait.ForListeningPort("4222/tcp"). That strategy has twohalves: an external dial of the mapped host port, and an internal in-container check. The
internal check execs
/bin/shto read/proc/net/tcp— andnats:2.11ships no shell, soit returns
errShellNotFound.WaitUntilReadyswallows that error(
wait/host_port.go:240-252) and reports success, leaving readiness resting on the externaldial alone.
That dial is a false positive: Docker's port forwarder accepts the connection as soon as the
port is published, before
nats-serverbinds. The forwarder then closes it, and the clientsees
EOF— the exact CI signature.Fix
Gate on the server's own
"Server is ready"log line via the Docker log stream, which theport forwarder cannot fake, in addition to the existing port wait:
"Server is ready"is an[INF]line, emitted after"Listening for client connections"regardless of the debug/trace flags the module passes. If the line ever drifts, the strategy
fails loudly —
wait.ForLog's 60s default startup timeout surfaces as atc.Runerrorthat lands in
t.Fatalf(a timeout does not match anyIsDockerUnavailablesubstring, so itcannot be silently downgraded to a skip).
Verification
An independent reviewer verified all five links of the root-cause chain against the
testcontainers source, reproduced the false positive end-to-end to the literal CI error
signature (
client read n=0 err=EOFagainst a container with nothing listening inside),confirmed the new wait strategy demonstrably takes effect at runtime, and ran 110 test
executions across 86 real containers with 0 failures and 0 skips.
It also audited the sibling container helpers:
internal/sink/bigqueryshares the samepattern and is protected only incidentally — filed as FLAKE-BQ-01.
Scope
One file,
//go:build integration, so it is not compiled intotask test/task coverageand the
./internal/...90% coverage floor is structurally unmoved.go mod tidyproducesno drift (
testcontainers-gois already a direct require).Note:
test-integrationis an advisory check here, which is precisely why this flakecould redden
mainwithout blocking merges.