Skip to content

fix(OpenTelemetry): fuse Activity weakly so never-ended spans are pruned#5393

Open
Ermabo wants to merge 4 commits into
getsentry:mainfrom
Ermabo:fix/otel-spanprocessor-weakref-leak
Open

fix(OpenTelemetry): fuse Activity weakly so never-ended spans are pruned#5393
Ermabo wants to merge 4 commits into
getsentry:mainfrom
Ermabo:fix/otel-spanprocessor-weakref-leak

Conversation

@Ermabo

@Ermabo Ermabo commented Jul 15, 2026

Copy link
Copy Markdown

Fixes #5392. Follow-up to #3198 (from #3166 / #3197).

Problem

SentrySpanProcessor._map accumulates spans that never receive OnEnd (aborted HTTP/2 streams, abandoned gRPC streams, other never-completed requests), so a long running service grows in memory until it runs out.

Root cause

OnStart fuses the Activity onto the span with SetFused(data). SetFused is backed by a ConditionalWeakTable, which holds values strongly (only keys are weak), so _map -> span -> Activity pins the Activity. The activity is null branch in PruneFilteredSpans (added in #3198 to evict collected activities) can therefore never run, and recorded spans that never end stay forever. #3198's description mentions weak references, but SetFused stores them strongly.

Fix

Fuse the Activity via WeakReference<Activity> so _map no longer pins it; the existing prune null-check then evicts orphans once the Activity is collected. IsFiltered still reads the sampling flags lazily from the Activity (works while it is alive).

Evidence

Repro: https://gist.github.com/Ermabo/41e0301e0afabfe4d28cff799d4e79b6 (never-ended spans grow the heap ~1.9 GB over 300k iterations; normal spans stay flat). In production we saw ~91k UnsampledTransaction retained (_isFinished == false, gcroot terminating at _map); gen2 grew ~190 MB to 2.4 GB over ~6 days before an OOM.

Tests

OnStart_FusesActivityWeakly plus the existing PruneFilteredSpans_GarbageCollectedActivity_Pruned; all 44 SentrySpanProcessorTests pass.

Changelog Entry

fix: SentrySpanProcessor no longer leaks spans whose Activity never ends (e.g. aborted requests); the Activity is now held via a WeakReference so orphaned spans are pruned once it is garbage-collected.

Note

One edge remains from making the reference weak: IsFiltered is read at parent-transaction finish, so if a filtered (sampled-out) child's Activity is collected before the parent serializes, the check returns false and that span is included rather than dropped. This is rare, is minor telemetry noise rather than a correctness/stability issue, and is strictly better than the OOM this PR fixes.

An eager snapshot at OnStart was tried and reverted (see commit history): OnStart runs before OTel instrumentation filters (e.g. HttpClient Filter) clear the sampling flags, so it captured filtered spans as unfiltered, a worse trade. The only read point that is both post-filter and GC-safe is OnEnd; capturing there would resolve the edge fully but adds complexity and since this package it deprecated I don't think it warrants the refactor.

@Ermabo
Ermabo requested a review from jamescrosswell as a code owner July 15, 2026 15:00
@github-actions github-actions Bot added the risk: medium PR risk score: medium label Jul 15, 2026
@Ermabo
Ermabo force-pushed the fix/otel-spanprocessor-weakref-leak branch from 1930368 to 6e98220 Compare July 15, 2026 15:00
Comment thread CHANGELOG.md Outdated
…ans are pruned

SentrySpanProcessor fused the Activity onto the span with a strong reference. SetFused stores its ConditionalWeakTable values strongly, so _map kept the Activity alive and PruneFilteredSpans never collected spans whose request ended without OnEnd (aborted or never-completed requests). Those built up until the process ran out of memory.

Fusing a WeakReference instead lets the Activity be collected once the request ends, so the existing prune removes the orphaned span.

Follows up getsentry#3198 (from getsentry#3166 and getsentry#3197), which intended weak references but stored them strongly.

Co-authored-by: Claude <noreply@anthropic.com>
@Ermabo
Ermabo force-pushed the fix/otel-spanprocessor-weakref-leak branch from 302cd49 to 1916445 Compare July 16, 2026 07:46
@Ermabo

Ermabo commented Jul 16, 2026

Copy link
Copy Markdown
Author

Cheers @jamescrosswell! Removed the manual CHANGELOG edit (missed the "do not edit manually" note in CONTRIBUTING) and added a ### Changelog Entry section to the PR description for craft; it's force-pushed.

@Ermabo

Ermabo commented Jul 16, 2026

Copy link
Copy Markdown
Author

Separate from this fix:

The underlying design here (and in #3197) is that _map treats the span as the owner and infers the Activity's lifecycle indirectly. A more robust long-term design might invert that and store the span on the Activity via Activity.SetCustomProperty so its lifetime follows the Activity's. This shrinks _map down to just parent lookups and removes this prune/GC-timing class entirely.

It's not free: remote/ActivityContext parents still need a small lookup. It fixes the memory orphan problem but not never-finished spans, and it likely has cross-SDK implications; so happy to open a discussion to explore rather than scope-creep this PR, if there is interest 😊

@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.71429% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 74.44%. Comparing base (b006e95) to head (1916445).
⚠️ Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
src/Sentry.OpenTelemetry/SentrySpanProcessor.cs 85.71% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5393      +/-   ##
==========================================
+ Coverage   74.23%   74.44%   +0.21%     
==========================================
  Files         509      511       +2     
  Lines       18435    18590     +155     
  Branches     3610     3632      +22     
==========================================
+ Hits        13685    13840     +155     
+ Misses       3875     3874       -1     
- Partials      875      876       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jamescrosswell

Copy link
Copy Markdown
Collaborator

The underlying design here (and in #3197) is that _map treats the span as the owner and infers the Activity's lifecycle indirectly. A more robust long-term design might invert that and store the span on the Activity via Activity.SetCustomProperty so its lifetime follows the Activity's.

Thanks @Ermabo - given that this integration is marked as deprecated, probably not worth burning too many cycles on it.

Longer term, the OTLP integration will be the primary method of integrating with OpenTelemetry.

Still possibly interesting as part of this work - but that's a much bigger project (currently on hold).

Comment thread src/Sentry.OpenTelemetry/SentrySpanProcessor.cs Outdated

@jamescrosswell jamescrosswell left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Ermabo huge thank you for the PR ❤️

I made one non-blocking stylistic suggestion. Let me know whether you'd like to implement that or if you'd prefer I approve and merge as is.

Route both fuse call sites and GetFusedActivity through a shared
ActivityPropertyName const so they can't drift from the Fused
default name. Addresses review feedback on getsentry#5393.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread src/Sentry.OpenTelemetry/SentrySpanProcessor.cs
IsFiltered read IsAllDataRequested/Recorded lazily from the weakly
fused Activity, so once the Activity was garbage-collected the check
returned false and a filtered (sampled-out) span was included in the
transaction instead of dropped. The sampling decision is fixed at span
creation, so capture it as a bool up front. Addresses Cursor Bugbot
finding on getsentry#5393.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0ec4953. Configure here.

Comment thread src/Sentry.OpenTelemetry/SentrySpanProcessor.cs Outdated
@Ermabo

Ermabo commented Jul 17, 2026

Copy link
Copy Markdown
Author

@jamescrosswell happy to contribute!

Both addressed and pushed:

  • SetFusedActivity + shared const (your suggestion): 7eea926
  • Eager filter-decision capture (Cursor Bugbot finding): 0ec4953

Updated the PR description to match. Ready for another look whenever 🙂

EDIT: the eager filter-decision capture (0ec4953) was reverted in 22a607, OnStart runs before OTel instrumentation filters clear the flags, so it captured filtered spans as unfiltered. Back to the lazy read; see follow-up comment below.

This reverts commit 0ec4953.

The eager snapshot read IsAllDataRequested/Recorded in OnStart, which
runs before OTel instrumentation filters (e.g. HttpClient Filter) clear
those flags, so filtered spans were captured as unfiltered. That traded
a minor GC-timing edge for a worse late-filtering regression. Back to
reading the flags lazily from the weakly-fused Activity.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Ermabo

Ermabo commented Jul 17, 2026

Copy link
Copy Markdown
Author

Reverted the eager filter capture (22a607b), Bugbot was right, OnStart runs before OTel instrumentation filters, so it snapshotted too early. Back to the lazy read 😅

PR is now just the leak fix + SetFusedActivity cleanup. Ready for another look @jamescrosswell 😊

@Ermabo
Ermabo requested a review from jamescrosswell July 17, 2026 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: medium PR risk score: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory leak: never-ended Activities accumulate in SentrySpanProcessor._map (sibling of #3197)

2 participants