fix(OpenTelemetry): fuse Activity weakly so never-ended spans are pruned#5393
fix(OpenTelemetry): fuse Activity weakly so never-ended spans are pruned#5393Ermabo wants to merge 4 commits into
Conversation
1930368 to
6e98220
Compare
…ans are pruned SentrySpanProcessor fused the Activity onto the span with a strong reference. SetFused stores its ConditionalWeakTable values strongly, so _map kept the Activity alive and PruneFilteredSpans never collected spans whose request ended without OnEnd (aborted or never-completed requests). Those built up until the process ran out of memory. Fusing a WeakReference instead lets the Activity be collected once the request ends, so the existing prune removes the orphaned span. Follows up getsentry#3198 (from getsentry#3166 and getsentry#3197), which intended weak references but stored them strongly. Co-authored-by: Claude <noreply@anthropic.com>
302cd49 to
1916445
Compare
|
Cheers @jamescrosswell! Removed the manual CHANGELOG edit (missed the "do not edit manually" note in CONTRIBUTING) and added a |
|
Separate from this fix: The underlying design here (and in #3197) is that _map treats the span as the owner and infers the Activity's lifecycle indirectly. A more robust long-term design might invert that and store the span on the Activity via Activity.SetCustomProperty so its lifetime follows the Activity's. This shrinks _map down to just parent lookups and removes this prune/GC-timing class entirely. It's not free: remote/ActivityContext parents still need a small lookup. It fixes the memory orphan problem but not never-finished spans, and it likely has cross-SDK implications; so happy to open a discussion to explore rather than scope-creep this PR, if there is interest 😊 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5393 +/- ##
==========================================
+ Coverage 74.23% 74.44% +0.21%
==========================================
Files 509 511 +2
Lines 18435 18590 +155
Branches 3610 3632 +22
==========================================
+ Hits 13685 13840 +155
+ Misses 3875 3874 -1
- Partials 875 876 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Thanks @Ermabo - given that this integration is marked as deprecated, probably not worth burning too many cycles on it. Longer term, the OTLP integration will be the primary method of integrating with OpenTelemetry. Still possibly interesting as part of this work - but that's a much bigger project (currently on hold). |
jamescrosswell
left a comment
There was a problem hiding this comment.
@Ermabo huge thank you for the PR ❤️
I made one non-blocking stylistic suggestion. Let me know whether you'd like to implement that or if you'd prefer I approve and merge as is.
Route both fuse call sites and GetFusedActivity through a shared ActivityPropertyName const so they can't drift from the Fused default name. Addresses review feedback on getsentry#5393. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
IsFiltered read IsAllDataRequested/Recorded lazily from the weakly fused Activity, so once the Activity was garbage-collected the check returned false and a filtered (sampled-out) span was included in the transaction instead of dropped. The sampling decision is fixed at span creation, so capture it as a bool up front. Addresses Cursor Bugbot finding on getsentry#5393. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 0ec4953. Configure here.
|
@jamescrosswell happy to contribute! Both addressed and pushed:
Updated the PR description to match. Ready for another look whenever 🙂 EDIT: the eager filter-decision capture (0ec4953) was reverted in 22a607, OnStart runs before OTel instrumentation filters clear the flags, so it captured filtered spans as unfiltered. Back to the lazy read; see follow-up comment below. |
This reverts commit 0ec4953. The eager snapshot read IsAllDataRequested/Recorded in OnStart, which runs before OTel instrumentation filters (e.g. HttpClient Filter) clear those flags, so filtered spans were captured as unfiltered. That traded a minor GC-timing edge for a worse late-filtering regression. Back to reading the flags lazily from the weakly-fused Activity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Reverted the eager filter capture (22a607b), Bugbot was right, OnStart runs before OTel instrumentation filters, so it snapshotted too early. Back to the lazy read 😅 PR is now just the leak fix + SetFusedActivity cleanup. Ready for another look @jamescrosswell 😊 |

Fixes #5392. Follow-up to #3198 (from #3166 / #3197).
Problem
SentrySpanProcessor._mapaccumulates spans that never receiveOnEnd(aborted HTTP/2 streams, abandoned gRPC streams, other never-completed requests), so a long running service grows in memory until it runs out.Root cause
OnStartfuses the Activity onto the span withSetFused(data).SetFusedis backed by aConditionalWeakTable, which holds values strongly (only keys are weak), so_map -> span -> Activitypins the Activity. Theactivity is nullbranch inPruneFilteredSpans(added in #3198 to evict collected activities) can therefore never run, and recorded spans that never end stay forever. #3198's description mentions weak references, butSetFusedstores them strongly.Fix
Fuse the Activity via
WeakReference<Activity>so_mapno longer pins it; the existing prune null-check then evicts orphans once the Activity is collected.IsFilteredstill reads the sampling flags lazily from the Activity (works while it is alive).Evidence
Repro: https://gist.github.com/Ermabo/41e0301e0afabfe4d28cff799d4e79b6 (never-ended spans grow the heap ~1.9 GB over 300k iterations; normal spans stay flat). In production we saw ~91k
UnsampledTransactionretained (_isFinished == false,gcrootterminating at_map); gen2 grew ~190 MB to 2.4 GB over ~6 days before an OOM.Tests
OnStart_FusesActivityWeaklyplus the existingPruneFilteredSpans_GarbageCollectedActivity_Pruned; all 44SentrySpanProcessorTestspass.Changelog Entry
fix:
SentrySpanProcessorno longer leaks spans whose Activity never ends (e.g. aborted requests); the Activity is now held via aWeakReferenceso orphaned spans are pruned once it is garbage-collected.Note
One edge remains from making the reference weak:
IsFilteredis read at parent-transaction finish, so if a filtered (sampled-out) child's Activity is collected before the parent serializes, the check returns false and that span is included rather than dropped. This is rare, is minor telemetry noise rather than a correctness/stability issue, and is strictly better than the OOM this PR fixes.An eager snapshot at
OnStartwas tried and reverted (see commit history):OnStartruns before OTel instrumentation filters (e.g. HttpClientFilter) clear the sampling flags, so it captured filtered spans as unfiltered, a worse trade. The only read point that is both post-filter and GC-safe isOnEnd; capturing there would resolve the edge fully but adds complexity and since this package it deprecated I don't think it warrants the refactor.