Unified boosting engine: FormulaBoost + WeibullAFT, benchmarked on A100 - #20
Merged
Conversation
Collapse the distributional and standard boosting loops into one multi-channel trainer (fit_boosting) driven by an Objective protocol, so a model is a configuration rather than a separate training fork. - FormulaObjective / FormulaBoost: boost the parameters of an arbitrary differentiable formula f(theta(z), x) with damped generalized Gauss-Newton preconditioning. The off-diagonal GGN term is what a black-box GBDT and an XGBoost custom objective's diagonal-only Hessian cannot express. - WeibullAFTObjective / WeibullAFT: right-censored survival regression that boosts BOTH Weibull scale(z) and shape(z) via an expected-Fisher natural gradient. XGBoost's survival:aft holds the distribution scale as a single global hyperparameter and cannot vary the shape with covariates. - DistributionObjective: NaturalBoost / DistributionalGBDT rebuilt on the shared trainer and the old per-model loop deleted, with device-resident Normal/Poisson steps. GPU path keeps raw scores device-resident when the objective is device-capable and builds one tree per channel with fit_tree_gpu_native; the formula and survival objectives build trees on GPU with host-side preconditioning. Co-authored-by: Cursor <cursoragent@cursor.com>
Add tests/test_formula.py (extrapolation beats a black-box GBDT, full preconditioning beats plain, parameter-surface recovery, eval_set early stopping) and tests/test_survival.py (recovers the shape surface, censoring is actually used, quantile and survival-curve monotonicity, input guards). Extend the CUDA verification and Modal GPU tests to exercise the unified trainer, FormulaBoost, and NaturalBoost CPU/GPU parity, and pin numpy<2.5 on the Modal image to match numba-cuda. Co-authored-by: Cursor <cursoragent@cursor.com>
Add the pivot yardsticks, each with a local CPU path and a Modal A100 entry point, writing JSON to benchmarks/results/: - bench_probabilistic.py: speed and quality vs NGBoost (A100: 1229x faster at 90K rows on a shared 500-tree budget, NLL tied). - bench_formula.py: sales-saturation-curve capability vs global fit, black-box GBDT, and a hand-rolled XGBoost multi-output custom objective (extrapolation ~21x better than black-box; full GGN recovers the saturation-speed surface a diagonal Hessian cannot). - bench_survival.py: Weibull AFT vs XGBoost survival:aft, which cannot vary the shape with covariates (OpenBoost recovers shape(z) at corr 0.997 with a better censored NLL). Also adds the GGN spike that grounds the approach and the unified-engine design document. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Pivot to a parametric-boosting engine where a model is a configuration, not a
fork. One multi-channel trainer (
fit_boosting) driven by anObjectiveprotocol now backs standard GBDT, NaturalBoost/DistributionalGBDT, the new
FormulaBoost, and the newWeibullAFT.DistributionObjective(NaturalBoost,device-resident Normal/Poisson steps),
FormulaObjective(arbitrarydifferentiable
f(theta(z), x)with damped generalized Gauss-Newton), andWeibullAFTObjective(right-censored survival, expected-Fisher naturalgradient). The old per-model distributional loop is deleted.
ob.FormulaBoost) boosts the parameters of a user formula;the off-diagonal GGN term is what a black-box GBDT and an XGBoost custom
objective's diagonal-only Hessian cannot express.
ob.WeibullAFT) boosts both Weibullscale(z)andshape(z)under censoring; XGBoost's
survival:aftholds the scale as a single globalhyperparameter and cannot vary the shape with covariates.
Benchmarks (Modal A100)
bench_probabilistic.py): 1229x faster at 90K rows on ashared 500-tree budget, NLL tied; 900K trains in ~7s.
bench_formula.py): extrapolation ~21x better than ablack-box GBDT;
fullGGN recovers the saturation-speed surface (corr_b 0.88)that a diagonal Hessian / hand-rolled XGBoost custom objective (0.60) cannot.
bench_survival.py): recoversshape(z)at corr0.997 with a better censored NLL (0.76 vs XGBoost 0.83) and C-index parity+.
Test plan
pytest tests/test_formula.py tests/test_survival.py tests/test_distributional.py(65 passed, CPU)ruff check src/openboost/ tests/ benchmarks/modal run tests/modal_gpu_tests.py::unified)Made with Cursor