Improve pandas history ergonomics: self-describing KeyErrors, DataFrame.get(symbol), lazy DataFrame on typed history - #9668
Draft
jhonabreul wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Improves history/pandas ergonomics in Python, targeting the most common interop failure modes:
Self-describing
PandasMapperKeyErrors. The old error was opaque:The error now names the requested keys, lists the frame's columns and index levels, and adds a targeted hint for the three recurring mistakes: an index level requested as a column (→
get_level_values/reset_index), a symbol that lives in the index (→df.loc/df.xs), and a symbol with no data (→df.get(key)). Any failure while building the description falls back to the legacy message.DataFrame.get(symbol)now returns the symbol's sub-frame, orNonewhen the symbol has no data, instead of raising. Column lookups keep pandas semantics.Typed history results expose
.data_frame. The typedHistory<T>overloads now returnDataHistory<T>instead of a plainIEnumerable, soself.history[TradeBar](symbol, 10).data_frameworks.DataHistory<T>still implementsIEnumerable<T>, the conversion is lazy, and this mirrors the existingOptionHistory/FutureHistorypattern.Deferred: opt-in wide/unstacked frame shapes (breaking default change, needs its own design), and pandas-only attribute hints on typed results (needs an interop-layer extension point).
Related Issue
N/A
Motivation and Context
MultiIndex/typed-result confusion is a dominant Python crash class. Each opaque KeyError costs a full backtest iteration to decode. These changes make the errors self-explanatory and add safe accessors, without changing any default frame shape.
Requires Documentation Change
The typed history return type (
DataHistory<T>) andhistory.get(symbol)/.data_frameaccessors could be mentioned in the history docs.How Has This Been Tested?
PandasIndexingTestscases: each KeyError variant (missing column, index-level key, symbol in index, missing symbol) asserts the new message;DataFrame.gettested for symbol, missing symbol, and column keys. All red before the fix.AlgorithmHistoryTests.TypedHistoryResultsExposeADataFrame(C# and Python): typed results expose the expected frame and remain enumerable.AlgorithmHistoryTests,PandasConverterTests,PandasIndexingTests,PythonUtilTestsfixtures pass.Types of changes
Checklist:
bug-<issue#>-<description>orfeature-<issue#>-<description>