perf(storage): bound resolveByProviderId fan-out with an ordered sliding window#853
Open
SgtPooki wants to merge 8 commits into
Open
perf(storage): bound resolveByProviderId fan-out with an ordered sliding window#853SgtPooki wants to merge 8 commits into
SgtPooki wants to merge 8 commits into
Conversation
…ing window Evaluate a provider's candidate data sets oldest-first with at most RESOLVE_CONCURRENCY reads in flight, reading metadata before activePieceCount and stopping once the oldest non-empty metadata match is known. Selection is unchanged: oldest non-empty metadata match, else oldest metadata match, else null. Refs #631
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
synapse-dev | bbb1907 | Commit Preview URL Branch Preview URL |
Jun 24 2026, 06:41 PM |
Clear the PQueue in a finally so a rejected metadata/piece-count read stops issuing the remaining reads instead of running the fan-out resolveByProviderId is meant to bound. In-flight reads (<= RESOLVE_CONCURRENCY) run to completion. Add a test pinning oldest-of-several-non-empty selection plus the early-exit guard: the newer non-empty match's getActivePieceCount is never read once the oldest is known.
Replace the enqueue-one-PQueue-task-per-dataset approach with a bounded sliding pool (Set + Promise.race, topping up as each read finishes). The pool never starts datasets newer than the oldest non-empty match it has found, so the read fan-out shrinks to roughly the match position instead of allocating and draining a no-op task per dataset. This makes it a real sliding window and drops the p-queue dependency. Export RESOLVE_CONCURRENCY so the fan-out test derives its bound from it.
Resolve conflict in storage/context.ts: keep the ordered sliding-window resolveByProviderId fan-out from this branch, but adopt master's cheaper hasActivePieces() boolean check (PR #852) in place of getActivePieceCount(). Update the resolve-window tests to mock getActivePieces accordingly. Claude-Session: https://claude.ai/code/session_01HFNuoASCSpUJWSu8XZRjfd
BravoNatalie
approved these changes
Jun 27, 2026
BravoNatalie
left a comment
BravoNatalie
left a comment
There was a problem hiding this comment.
Everything looks solid and clean!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
resolveByProviderIdevaluates a provider's data sets oldest-first through a sliding read pool: at mostRESOLVE_CONCURRENCY(10) reads in flight, topping up the instant one finishes. It reads metadata first andgetActivePieceCountonly on a metadata match, and never starts a data set newer than the oldest non-empty match it has already found. So the read fan-out shrinks to roughly the match position instead of reading every candidate.This replaces the fixed 50-200 batch, which read every candidate before inspecting any result and stalled at batch boundaries for clients with many data sets per provider.
The pool is a bounded
Set+Promise.racethat tops up as each read settles (the same sliding-window shape rvagg asked for, and the one already used inapps/backend/.../ipfs-block.strategy.tson the dealbot side). No new dependency.How to verify
Selection is unchanged: oldest non-empty metadata match, else oldest metadata match, else null. Tests in
storage.test.ts: bounded fan-out for a 196-data-set provider, oldest-non-empty selection across the window, and oldest-of-several-non-empty with the newer match never read.Refs #631.