test: producer-to-collector routing on pg_stat_ch.block_format

[JoshDreamland]

Summary

Closes the producer-side half of the Arrow exporter migration contract: pin pg_stat_ch.block_format emission and the routing-connector match-shape via a generic TAP test.

The unified Arrow exporter (PR #116's commit 4) emits pg_stat_ch.block_format=arrow_events_raw on its OTLP envelope; the legacy ArrowBatchBuilder path emits arrow_ipc. In prod, the central datagres-otelcol deployment's routingconnector is supposed to fan batches between the legacy query_logs_arrow receiver target and a new events_raw receiver target based on that marker. This test pins the producer side of that contract: it doesn't run a real receiver (no CH, no datagres-arrow-exporter) — it runs stock otelcol-contrib with two file exporters and asserts which one received which arm's batches.

What it catches (and what it doesn't)

Catches:

• Producer regression in pg_stat_ch.block_format emission (e.g. unified path silently falling back to arrow_ipc).
• Routingconnector config-shape drift (attribute name typo, OTTL statement that no longer matches).
• OTLP-side serialization regressions that make the marker unreachable to a connector matching on resource attributes.

Does NOT catch:

• The events_raw column-set contract or the receiver's wire-format expectations — those live behind datagres-arrow-exporter in clickgres-platform and are covered by a separate receiver-faithful test there (in flight).
• That t/036_unified_arrow_e2e.pl continues to handle the producer Arrow IPC -> direct CH ingest path.

Mechanics

• New compose profile docker/docker-compose.arrow-route.yml (separate from the existing docker-compose.otel.yml so t/024 and t/037 can coexist; different ports, different container name).
• New collector config docker/otel-routing/collector-config.yaml — stock otel/opentelemetry-collector-contrib:0.120.0, routingconnector keyed on the resource attribute, three file exporters (legacy / events_raw / default fall-through) writing JSONL to a bind-mounted dir.
• t/psch.pm helpers: psch_routing_collector_available / psch_start_routing_collector / psch_stop_routing_collector.
• t/037_arrow_routing.pl runs two arms (legacy + unified) and asserts that the corresponding file grew while the other files didn't. The unified arm also greps the routed JSONL tail for arrow_events_raw to pin the marker value content.

Stacking

Based on unified_arrow_exporter (PR #116). Hard-depends on commits 2 (the GUC) and 4 (the marker swap). Will retarget to main after #116 merges.

Test plan

• Docker / collector image pull succeeds in CI.
• Both arms pass with the producer-side commits already on unified_arrow_exporter.
• Existing TAP suite still green (the new helpers + compose are additive; no shared state with psch_*_otelcol).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a new TAP test that validates the producer-side pg_stat_ch.block_format marker and an otelcol-contrib routingconnector “shape” by routing OTLP logs into different file exporters, ensuring legacy (arrow_ipc) vs unified (arrow_events_raw) batches fan out as expected during the Arrow exporter migration.

Changes:

• Introduces a new TAP test (t/037_arrow_routing.pl) that runs legacy vs unified exporter arms and asserts which routed JSONL file grew.
• Adds Docker Compose + collector config for a standalone routing collector using routingconnector and file exporters.
• Extends TAP helper module (t/psch.pm) with start/stop/availability helpers for the routing collector.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
t/psch.pm	Adds helper functions to manage a dedicated routing-collector docker compose deployment for the new TAP test.
t/037_arrow_routing.pl	New TAP test that drives two producer configurations and asserts routing outcomes via file growth + marker grep.
docker/otel-routing/output/.gitignore	Keeps routed JSONL outputs out of git while preserving the directory.
docker/otel-routing/collector-config.yaml	Minimal collector pipeline using routingconnector and per-route file exporters for assertions.
docker/docker-compose.arrow-route.yml	Standalone compose stack for the routing collector with distinct ports/container name from existing otelcol test stack.

Comments suppressed due to low confidence (1)

t/037_arrow_routing.pl:178

• With cleanup handled via an END block (see earlier comment), the unconditional stop call here should be removed so the test doesn’t tear down a collector it didn’t start.

psch_stop_routing_collector();
done_testing();

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

t/037_arrow_routing.pl:26

• The header comment describes three test arms (unified off, then off+passthrough, then unified on), but the script actually runs only two arms and both set otel_arrow_passthrough=on. This mismatch can confuse future maintainers when updating the test.

#   2. Start a node configured to ship to it; first with the unified GUC OFF,
#      then OFF + with otel_arrow_passthrough so the legacy ArrowBatchBuilder
#      path fires; finally with the unified GUC ON.

t/037_arrow_routing.pl:46

• The test always tears down the routing collector at the end, even if it was already running before the test started. This can disrupt local dev workflows or parallel runs. It’s also safer to register teardown in an END block so cleanup still happens if the test exits early.

    eval { psch_start_routing_collector() };
    if ($@) {
        plan skip_all => "Docker / routing collector not available: $@";
    }
}

t/037_arrow_routing.pl:177

• With teardown handled conditionally in an END block, this unconditional shutdown is redundant and will still bring down a pre-existing collector when the test didn’t start it.

psch_stop_routing_collector();

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (6)

t/037_arrow_routing.pl:200

• wait_for_quiet() can time out and return 0, but the return value is ignored here. If the collector is still writing when the baselines are snapped, the second arm can become flaky (late bytes mis-attributed). Please assert that the collector output actually went quiet before proceeding.

    # Let arm 1's trailing collector writes settle before snapshotting this
    # arm's baselines (see wait_for_quiet).
    wait_for_quiet(1.5, 15,
                   map { "$output_dir/$_.jsonl" } qw(legacy events_raw default));

t/037_arrow_routing.pl:229

• Same as arm 1: after observing growth in events_raw.jsonl, it’s worth waiting for the collector output to go quiet before asserting that the other route files didn’t grow and before tail-inspecting events_raw.jsonl. Without this, late/partial writes can cause flaky or false-pass behavior.

    my $events_raw_after =
        wait_for_growth("$output_dir/events_raw.jsonl", $events_raw_before, 15);
    ok(defined $events_raw_after,
       'arm 2 (arrow_events_raw): events_raw.jsonl received bytes after producer flush');

    my $legacy_after = current_size("$output_dir/legacy.jsonl");
    is($legacy_after, $legacy_before,
       'arm 2 (arrow_events_raw): legacy.jsonl did NOT grow (routing kept it isolated)');

    my $default_after = current_size("$output_dir/default.jsonl");
    is($default_after, $default_before,
       'arm 2 (arrow_events_raw): default.jsonl did NOT grow (no unmatched batches)');

t/psch.pm:276

• Same as psch_routing_collector_available(): add a --max-time to the health polling curl so each iteration can’t block longer than expected.

    for my $i (1..30) {
        my $result = `curl -sf 'http://localhost:23133/' 2>/dev/null`;
        return 1 if $result =~ /Server available/;
        sleep(1);

t/037_arrow_routing.pl:186

• The routing isolation assertions are made immediately after the first observed growth in legacy.jsonl. Because the collector writes asynchronously (and may split a JSON line + newline into separate writes), bytes routed to the wrong file could arrive after these checks and be missed. Waiting for the output files to go quiet before asserting non-growth makes this test deterministic and prevents false passes.

This issue also appears on line 218 of the same file.

    my $legacy_after = wait_for_growth("$output_dir/legacy.jsonl", $legacy_before, 15);
    ok(defined $legacy_after,
       'arm 1 (arrow_ipc): legacy.jsonl received bytes after producer flush');

    my $events_raw_after = current_size("$output_dir/events_raw.jsonl");
    is($events_raw_after, $events_raw_before,
       'arm 1 (arrow_ipc): events_raw.jsonl did NOT grow (routing kept it isolated)');

    my $default_after = current_size("$output_dir/default.jsonl");
    is($default_after, $default_before,
       'arm 1 (arrow_ipc): default.jsonl did NOT grow (no unmatched batches)');

t/013_clickhouse_tls.pl:102

• The comment explains that cumulative backoff can reach 123s (7 failures), but the test only waits 90s. Given PschGetRetryDelayMs() caps per-attempt delay at 60s but allows many consecutive failures, a slow CI restart can still exceed 90s even if ClickHouse is already back. Either adjust the explanation or bump the timeout so it actually covers the documented 123s case.

    # 90s, not 30s: the exporter's reconnect backoff (stats_exporter.cc,
    # PschGetRetryDelayMs) is exponential — base 1s, doubling per
    # consecutive failure, capped at 60s. Failed flush attempts start
    # accumulating the moment CH goes down (well before docker restart's
    # container comes back), and the cumulative wait before the Nth retry
    # is the sum of all prior delays: 1, 3, 7, 15, 31, 63, 123s... A CH
    # restart slow enough to rack up 5-6 failures (very plausible under a
    # loaded CI runner) pushes the bgworker's next attempt past the 30s
    # mark even though CH itself came back much sooner. 90s comfortably
    # covers the 63s (6-failure) cumulative-backoff case plus round-trip.
    my $after = psch_wait_for_clickhouse_query(
        'SELECT count() FROM pg_stat_ch.events_raw', sub { $_[0] >= 1 }, 90);

t/psch.pm:257

• These health-check curl calls have no max-time, so a stalled TCP connect/read can hang the TAP test beyond its intended timeout loop. Adding a small --max-time keeps the polling loop bounded and makes failures fail fast.

This issue also appears on line 273 of the same file.

sub psch_routing_collector_available {
    return 0 unless system("docker ps >/dev/null 2>&1") == 0;
    my $result = `curl -sf 'http://localhost:23133/' 2>/dev/null`;
    return $result =~ /Server available/;

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

t/037_arrow_routing.pl:200

• wait_for_quiet(...) is called but its return value is ignored. If the collector output never settles within the timeout, the test will still snapshot baselines and proceed, which can mis-attribute late bytes from arm 1 to arm 2 and make the routing assertions flaky/incorrect. Please assert that the output actually became quiet (and bail out if it didn’t) before continuing.

    wait_for_quiet(1.5, 15,
                   map { "$output_dir/$_.jsonl" } qw(legacy events_raw default));

docker/docker-compose.arrow-route.yml:26

• The comment claims otel/opentelemetry-collector-contrib:0.120.0 is a distroless image with no wget/curl/shell and that a CMD healthcheck can “never succeed”, but docker/docker-compose.otel.yml uses a CMD-based wget healthcheck with the same image tag. This inconsistency makes it unclear whether the healthcheck is truly impossible or just intentionally omitted here; please soften/rewrite the comment to avoid stating a false constraint.

    # No Docker-level healthcheck: otel/opentelemetry-collector-contrib is a
    # distroless image with no wget/curl/shell, so a CMD-based healthcheck
    # can never succeed and would always report "unhealthy" regardless of
    # actual readiness — misleading in `docker compose ps`, and a landmine
    # for anyone who later adds `--wait` to the start command (would hang
    # forever). Readiness is checked from the host via a real HTTP request
    # in psch_start_routing_collector (t/psch.pm), independent of Docker's
    # own healthcheck machinery.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

t/037_arrow_routing.pl:200

• wait_for_quiet() returns 0 on timeout, but the return value is currently ignored. If the collector never becomes quiet (or is slow), the arm 2 baselines can be snapshotted mid-write, causing late bytes from arm 1 to be mis-attributed to arm 2 and making the test flaky. Please assert that wait_for_quiet() succeeded so failures are explicit and actionable.

    # Let arm 1's trailing collector writes settle before snapshotting this
    # arm's baselines (see wait_for_quiet).
    wait_for_quiet(1.5, 15,
                   map { "$output_dir/$_.jsonl" } qw(legacy events_raw default));

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

t/037_arrow_routing.pl:72

• The END cleanup will run even when the routing collector start failed due to Docker being unavailable (e.g., docker not installed). In that skip path, calling docker compose in psch_stop_routing_collector can add noisy "docker: not found" output after plan skip_all. Gate teardown on Docker being usable to keep skips clean while still cleaning up real partial startups.

    # system() inside END sets $?, which would clobber the script's exit
    # code; preserve it.
    local $?;
    psch_stop_routing_collector() if $started_routing_collector;
}

t/037_arrow_routing.pl:200

• wait_for_quiet() returns 0 on timeout, but the result is ignored. If the collector output never becomes quiet (or takes longer than the timeout), the arm-2 baselines may be captured mid-write and misattribute late bytes to the wrong arm, causing flakes. Assert the quiet period before snapshotting baselines.

    wait_for_quiet(1.5, 15,
                   map { "$output_dir/$_.jsonl" } qw(legacy events_raw default));

t/psch.pm:290

• The routing collector startup loop also runs curl without timeouts; a single hung curl would stall the entire start routine. Add connect/overall timeouts to keep failures bounded and make CI behavior deterministic.

    for my $i (1..30) {
        my $result = `curl -sf 'http://localhost:23133/' 2>/dev/null`;
        return 1 if $result =~ /Server available/;
        sleep(1);

t/psch.pm:282

• psch_start_routing_collector assumes the bind-mounted output directory already exists; if PROJECT_DIR points at a checkout missing docker/otel-routing/output (or the directory was deleted), chmod will die before docker compose even runs. Creating the directory makes the helper more robust.

    chmod 0777, $output_dir
        or die "Failed to chmod routing collector output dir ($output_dir): $!";

t/psch.pm:258

• psch_routing_collector_available runs curl without any connect/overall timeout. If localhost:23133 is blackholed or a proxy intercepts, the TAP suite can hang indefinitely. Add curl timeouts so availability checks fail fast.

This issue also appears on line 287 of the same file.

sub psch_routing_collector_available {
    return 0 unless system("docker ps >/dev/null 2>&1") == 0;
    my $result = `curl -sf 'http://localhost:23133/' 2>/dev/null`;
    return $result =~ /Server available/;

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

docker/docker-compose.arrow-route.yml:26

• The comment claims the otelcol-contrib image has no wget/curl/shell so an in-container CMD healthcheck can never succeed, but this repo’s existing docker-compose.otel.yml defines a wget-based healthcheck for the same image. To avoid misleading future readers, reword this as “we intentionally avoid a Docker healthcheck and poll the host health endpoint instead”, without asserting what tools exist in the image.

    # No Docker-level healthcheck: otel/opentelemetry-collector-contrib is a
    # distroless image with no wget/curl/shell, so a CMD-based healthcheck
    # can never succeed and would always report "unhealthy" regardless of
    # actual readiness — misleading in `docker compose ps`, and a landmine
    # for anyone who later adds `--wait` to the start command (would hang

t/037_arrow_routing.pl:62

• The eval-caught error string ($@) typically includes a trailing newline (and often "at ... line ..."), and embedding that directly in plan skip_all can produce TAP output with embedded newlines; earlier in this PR you note that TAP drops text past the first line. Sanitize/chomp the error before using it as the skip reason so the harness gets a single-line message.

    eval { psch_start_routing_collector() };
    if ($@) {
        plan skip_all => "Docker / routing collector not available: $@";
    }

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit db1f9be. Configure here.}

[cursor]

Shutdown exports bypass routing isolation

Medium Severity

Each arm asserts sink isolation, then stops the Postgres node. Shutdown and in-flight OTLP can still append to the JSONL sinks after those checks. A misroute that lands only during teardown is never compared to the arm baselines, so arm 1 can falsely pass while writing to events_raw.jsonl or default.jsonl.

Additional Locations (1)

• t/037_arrow_routing.pl#L241-L282

 

^{Reviewed by Cursor Bugbot for commit db1f9be. Configure here.}