Add big-request guardrails and honest resource diagnostics - #9669
Open
jhonabreul wants to merge 11 commits into
Open
Add big-request guardrails and honest resource diagnostics#9669jhonabreul wants to merge 11 commits into
jhonabreul wants to merge 11 commits into
Conversation
…f as a corrupt zip
Moves the large/overlapping history request warning state and logic out of the QCAlgorithm partial into a private LargeHistoryRequestDiagnostics nested class with a single entry point, keeping the algorithm class surface small. No behavior change.
Keep the new user-facing warnings and code comments short and direct so they are easy to act on. No behavior change.
jhonabreul
marked this pull request as ready for review
August 12, 2026 17:57
Estimate tick history at 10 data points per second: one per second underestimates liquid symbols by 1-2 orders of magnitude (SPY 2013-10-07 sample data: ~10.7 trade and ~115 quote ticks per second), letting OOM-prone tick history requests slip under the warning threshold. Never throw from the diagnostics checks: failures while checking requests or emitting warnings are logged and disable the checks instead of interfering with the algorithm.
Replace the option-only contract count warning with a check covering
every universe. Because the cost of a selected symbol grows with the
resolution its subscriptions are added at, the flat contract threshold
becomes a per-resolution table ('universe-selection-size-warning-
thresholds', a JSON object like {"Minute": 500}). All universes consume
a single shared budget: each universe contributes selection size over
its resolution's threshold, warning when the combined load reaches one,
so many small selections across resolutions accumulate like a single
wide one. The warning never throws: failures are logged and disable the
check instead of interfering with the algorithm.
One minute is close enough to normal heavy steps (warm up, coarse selection days) to be noisy; three minutes keeps the warning early relative to the twenty minute kill while only firing on genuinely abnormal steps.
The slow time step and long-running scheduled event warnings only went to the engine log, which does not reach the user's log file. Both now also route through the result handler's debug messages via an optional user warning handler, wired in AlgorithmManager.Run and BaseRealTimeHandler.Setup. Only the first scheduled event minute crossing is surfaced to the user; later crossings stay engine-side to avoid flooding user logs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds four independent warn-don't-fail diagnostics for big-request failure modes that today surface as opaque or misleading fatal errors. Runs that would have succeeded are unaffected; all thresholds are config overridable, and the diagnostics never throw: a failure while checking or emitting a warning is logged and disables the check instead of interfering with the algorithm.
ZipDataCacheProvider.Fetch: allocation failures wrapped by Ionic were logged asCorrupt zip file/entryand swallowed. AnOutOfMemoryExceptionanywhere in the chain is now rethrown as a memory diagnostic. Truly corrupt zips keep the old behavior.QCAlgorithm.Historyfunnel, so all overloads): warns once over an estimated data-cell threshold (default 5M), and warns once when 30 consecutive large calls have overlapping time windows (the re-fetch-a-long-lookback-every-day pattern). Tick history is estimated at 10 data points per second of market time, grounded on the repo's SPY sample data (~10.7 trade and ~115 quote ticks per second).AlgorithmTimeLimitManagerlogs a warning once per time step over a threshold (default 3 minutes) instead of staying silent until the isolator kill, andTimeMonitornow names the running scheduled event when it crosses the one-minute mark. Both warnings also reach the user's log file through the result handler's debug messages, not just the engine log.universe-selection-size-warning-thresholds, a JSON object with defaults Tick 100, Second 250, Minute 500, Hour 1000, Daily 2000; a resolution set to a non positive value is disabled). All universe types are covered and consume a single shared budget: each universe contributes its selection size divided by its resolution's threshold, warning when the combined load reaches one, so many small selections across resolutions accumulate like a single wide one. Option chain universes get the narrow-the-filter suggestion (SetFilter/AddOptionContract); other universes are pointed at selecting fewer symbols or a coarser universe resolution.Related Issue
N/A
Motivation and Context
These failure modes cost hours: a wide option universe dying as "corrupt zip" (actually OOM), billion-cell history re-fetches stalling backtests, and 10-minute isolator kills that never name the slow handler. Upfront, named warnings make them self-correcting.
Requires Documentation Change
No. The new config keys only tune warnings and have safe defaults.
How Has This Been Tested?
AlgorithmHistoryTestsfixture.BasicTemplateOptionsAlgorithm(C# + Python) statistics unchanged.Types of changes
Checklist:
bug-<issue#>-<description>orfeature-<issue#>-<description>