Delay reading option chain universe files until close to market open in live trading

[jhonabreul]

Description

In live trading, option chain universes are backed by the daily universe files under Data/option/<market>/universes/. The file for the previous tradable date is read at deployment time and then re-read on every refresh cycle (every 10 minutes) around the clock, even though a new file only appears once per day. Option chain universe files can be big, so this is wasted IO, and reads during the night overlap with the window in which the data pipeline is still writing the new files.

The change:

• LiveCustomDataSubscriptionEnumeratorFactory accepts an optional canRefresh predicate: while it returns false the source is not refreshed or read. The predicate is evaluated at the same cadence as source refreshes, so it is rate limited as well.
• LiveTradingDataFeed.CreateUniverseSubscription passes a gate for OptionChainUniverse subscriptions (GetUniverseFileRefreshGate): the universe file is only read while the exchange is open or within one hour of the next market open (PreOpenUniverseFileRefreshWindow).
• Mid-session deployments still get immediate selection, since the gate is open while the market is open. Other universes served by the same code path (ETF constituents, fundamentals, future chains) are unchanged.

Behavioral effect: for an algorithm deployed while the market is closed, option chain selection now happens between one hour before market open and the open, instead of right at deployment.

Related Issue

N/A

Motivation and Context

Option chain universe files are among the biggest files Lean reads in live trading; reading them around the clock, including overnight while the new files are still being generated, wastes IO for no benefit. Delaying the read until shortly before the market opens ensures the file is read when it is complete and about to be needed.

Requires Documentation Change

No

How Has This Been Tested?

• New LiveCustomDataSubscriptionEnumeratorFactoryTests.RespectsRefreshGate: with the gate closed the source is never read regardless of time passing; gate checks are rate limited like refreshes; once the gate opens the next refresh reads the source.
• New LiveTradingDataFeedTests.OptionChainSelectionIsDelayedUntilCloseToMarketOpen (equity option and index option cases): algorithm deployed at 2 am, asserts the first chain selection happens within [market open - 1 hour, market open]. Red before the change (selection happened right at deployment), green after.
• Full LiveTradingDataFeedTests and LiveCustomDataSubscriptionEnumeratorFactoryTests fixtures with CI category filters: 195 passed, 0 failed, 1 skipped (pre-existing ignored test).
• Live-paper deployment (local Launcher, FakeDataQueue, temporary test-only time provider offset shifting the engine clock to 8:13 am ET pre-open, with a fabricated GOOG universe file for the previous tradable date): the universe refresh gate was checked and blocked at 8:13 and 8:23, and the file was read and the option filter invoked at 8:33 am ET — inside the one hour pre-open window — selecting all 2192 contracts, with zero errors in the log. The instrumentation was discarded after the run.
• Live-paper deployment mid market day (real clock, pristine binary, same fabricated universe file): the universe file was read and the option filter invoked within seconds of deployment (universe subscription created and filter invoked at the same second, 10:24:02 am ET), confirming intraday deployments still get immediate selection.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Refactor (non-breaking change which improves implementation)
• Performance (non-breaking change which improves performance. Please add associated performance test and results)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Non-functional change (xml comments/documentation/etc)

Checklist:

• My code follows the code style of this project.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.
• My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>