Skip to content

fix(telemetry): make ApplicationInsights flush await in-flight requests (fixes flaky test)#2373

Open
kgilpin wants to merge 1 commit into
mainfrom
fix/telemetry-ai-flush
Open

fix(telemetry): make ApplicationInsights flush await in-flight requests (fixes flaky test)#2373
kgilpin wants to merge 1 commit into
mainfrom
fix/telemetry-ai-flush

Conversation

@kgilpin

@kgilpin kgilpin commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Problem

tests/backends/application-insights.spec.ts is flaky on main
(e.g. run 28680059992):
the "non-2xx HTTP responses" test intermittently fails with

Expected: "ApplicationInsightsBackend: Failed to send telemetry event. Status: 500"
Received: "Error sending telemetry data to Application Insights", [Error: test error]

plus "Cannot log after tests are done".

Root cause

ApplicationInsightsBackend.sendEvent is fire-and-forget and flush() was a
no-op, and the test's waitForRequest only waits for the request to be sent
(scope.isDone()), not for the client-side res.on('end') / req.on('error')
handlers that console.warn. So the warning fires after the test finishes —
leaking the previous test's error warning into the next test's warnSpy and
racing the assertion.

Fix

  • Track pendingRequests and make flush() resolve once they settle — deleting
    each request from the set after its warning fires — mirroring
    SplunkBackend (which already does this).
  • The two warning tests await flush() before asserting, so the warning is
    guaranteed to have run (deterministic).
  • afterEach restoreAllMocks() isolates the console.warn spy across tests.

Verification

application-insights.spec.ts passes 3× in a row with no late-log warnings;
full telemetry suite 42/42; verify (lint + tsc) clean.

🤖 Generated with Claude Code

The application-insights backend's requests are fire-and-forget and its
flush() was a no-op, so the response/error handlers (which console.warn)
ran after the test finished — leaking warnings into the next test and
racing assertions ("Cannot log after tests are done"; the 500 test saw
the previous test's error warning instead of its own).

Track pending requests and make flush() resolve once they settle (delete
from the set after the warn fires), mirroring SplunkBackend. The two
warning tests now await flush() before asserting — deterministic, because
the warning is guaranteed to have run — and afterEach restoreAllMocks()
isolates the console.warn spy across tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@dividedmind dividedmind left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, but looking at the code I'm not sure this is the correct approach. IMO we don't need flushing in general as we don't batch requests but send them immediately (as alluded to in the comment), and any outstanding requests will run to completion before releasing the event loop on process exit, so in production code this should work as intended.

The problem seems limited to tests, and the flakiness seems to be caused by the fact that waitForRequest doesn't apparently work as intended. Perhaps we can instead return a promise or the request from sendEvent() – test code could await it (or set event handlers) to make sure the request completed, while the production code would simply ignore the return value. (Note if it's a promise it should never reject to avoid bubbling errors to the toplevel.) Splunk backend could work in the same way BTW, flushing is not meaningful there either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants