Skip to content

docs: reposition around distributional regression #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jxucoder merged 1 commit into from
Aug 18, 2026
Merged

docs: reposition around distributional regression #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- `FormulaBoost`: boost parameters of a user formula `y = f(θ, x)` with
finite-difference Jacobian and damped GGN (`precond='full'|'diag'|'plain'`).
- `WeibullAFT`: right-censored Weibull AFT that boosts both scale `λ(z)`
and shape `k(z)` using expected-Fisher natural gradient.
- Unified `fit_boosting` trainer and `Objective` protocol shared by
NaturalBoost, FormulaBoost, and WeibullAFT.
- Capability benchmarks `benchmarks/bench_formula.py` and
`benchmarks/bench_survival.py`; probabilistic speed/quality suite
`benchmarks/bench_probabilistic.py`.

### Changed

- Documentation and package description framed as distributional
regression (GAMLSS / NGBoost) and varying-coefficient models: README,
docs home, quickstart, GPU setup, NaturalBoost guide, XGBoost
migration, and a new benchmarks page. Headline numbers: 1229× vs
NGBoost on A100 at 90K (NLL tied); FormulaBoost ~21× better
extrapolation than black-box GBDT; WeibullAFT recovers `k(z)`.
- Performance documentation now quotes only results backed by committed
benchmark runs (Modal A100 speed/capability, CPU UCI quality).
- Made multi-round `fit_trees_batch` recompute gradients and hessians from targets.
- Consolidated batch configuration and training state into one canonical module.
- Added manually triggered CUDA verification on Modal GPUs.
- Custom distributions now fall back to numerical differentiation when JAX
cannot trace a user NLL, instead of returning placeholder gradients.
- Unsupported `sample_weight` inputs now raise on CUDA, distributed, and
multi-GPU training paths instead of being ignored.
- Performance documentation now quotes only results backed by committed raw
benchmark artifacts.

## [1.0.0rc1] - 2026-01-20

Expand Down
57 changes: 39 additions & 18 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,19 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

## Project Overview

OpenBoost is a GPU-native, all-Python gradient boosting library (~20K lines). It uses Numba JIT for CPU kernels and CuPy/numba-cuda for GPU acceleration. Designed as a research-friendly alternative to XGBoost/LightGBM with full Python source.
OpenBoost is GPU gradient boosting for distributional regression (~20K
lines of Python). Each parameter of the statistical model is a
histogram-tree ensemble, trained with the full K x K natural gradient /
GGN rather than a diagonal approximation. Numba JIT on CPU,
CuPy/numba-cuda on GPU.

Flagship models: `NaturalBoost` (GAMLSS-style distributions),
`FormulaBoost` (varying-coefficient formula), `WeibullAFT` (censored
Weibull). Mean-regression GBDT, GAM, DART, and linear-leaf models share
the same tree engine.

Not a faster XGBoost clone. XGBoost/LightGBM remain the right tool for
ordinary MSE/logloss GBDT.

## Commands

Expand Down Expand Up @@ -44,31 +56,40 @@ uv build # Build wheel/sdist
### Layer Overview

```
Models (_models/) → Core (_core/) → Backends (_backends/)
↓ ↓ ↓
GradientBoosting fit_tree() _cpu.py (Numba JIT)
NaturalBoost histograms _cuda.py (CuPy kernels)
OpenBoostGAM split finding
DART, LinearLeaf growth strategies
Models (_models/) → Trainer (_trainer.py) → Core (_core/) → Backends
↓ Objective protocol ↓ ↓
NaturalBoost fit_boosting() fit_tree() _cpu.py
FormulaBoost _objectives.py histograms _cuda.py
WeibullAFT Distribution / Formula / split finding
GradientBoosting AFT / Loss growth
OpenBoostGAM, DART, LinearLeaf
```

### Data Layer (`_array.py`)
`BinnedArray` is the fundamental data structure — quantile-bins continuous features into uint8 (max 255 bins). Missing values encode as `MISSING_BIN = 255`. Native categorical feature support with auto-detection of string/object columns. All tree-building operates on binned data.
`BinnedArray` is the fundamental data structure: it quantile-bins continuous features into uint8 (max 255 bins). Missing values encode as `MISSING_BIN = 255`. Native categorical feature support with auto-detection of string/object columns. All tree-building operates on binned data.

### Core (`_core/`)
- **`_tree.py`** — `fit_tree()`, `fit_tree_gpu_native()`, `fit_tree_symmetric()`: the main tree-fitting entry points
- **`_primitives.py`** — Low-level histogram building, split finding, sample partitioning
- **`_growth.py`** — Three growth strategies: `LevelWiseGrowth` (XGBoost-style), `LeafWiseGrowth` (LightGBM-style), `SymmetricGrowth` (CatBoost-style)
- **`_tree.py`**: the main tree-fitting entry points `fit_tree()`, `fit_tree_gpu_native()`, `fit_tree_symmetric()`
- **`_primitives.py`**: Low-level histogram building, split finding, sample partitioning
- **`_growth.py`**: Three growth strategies: `LevelWiseGrowth` (XGBoost-style), `LeafWiseGrowth` (LightGBM-style), `SymmetricGrowth` (CatBoost-style)

### Backend Dispatch (`_backends/`)
`get_backend()` / `set_backend()` switch between CPU and CUDA implementations. Same interface, different kernels. Control via `OPENBOOST_BACKEND` env var or `set_backend('cuda')`. Use `backend_context('cpu')` context manager for temporary switches.

### Models (`_models/`)
- **`_boosting.py`** — `GradientBoosting`, `MultiClassGradientBoosting`: main model classes with full callback/eval_set support
- **`_sklearn.py`** — sklearn-compatible wrappers (`OpenBoostRegressor`, `OpenBoostClassifier`, `OpenBoostDARTRegressor`, `OpenBoostGAMRegressor`, `OpenBoostDistributionalRegressor`, `OpenBoostLinearLeafRegressor`)
- **`_distributional.py`** — `NaturalBoost`: distributional GBDT with callbacks/eval_set support
- **`_dart.py`** — `DART`: dropout boosting with callbacks/eval_set support
- **`_linear_leaf.py`**, **`_gam.py`** — Specialized model variants (both support callbacks/eval_set/early stopping; GAM also supports pairwise interactions, smoothing, and monotone shape constraints)
- **`_boosting.py`**: `GradientBoosting`, `MultiClassGradientBoosting`
- **`_distributional.py`**: `NaturalBoost` / `DistributionalGBDT` (facade over the unified trainer)
- **`_formula.py`**: `FormulaBoost` with user `f(θ, x)`, FD Jacobian, GGN preconditioner
- **`_survival.py`**: `WeibullAFT` with censored NLL, expected-Fisher natural gradient
- **`_sklearn.py`**: sklearn wrappers (`OpenBoostRegressor`, `OpenBoostClassifier`, `OpenBoostDARTRegressor`, `OpenBoostGAMRegressor`, `OpenBoostDistributionalRegressor`, `OpenBoostLinearLeafRegressor`)
- **`_dart.py`**, **`_linear_leaf.py`**, **`_gam.py`**: DART, linear-leaf, GAM

### Trainer (`_trainer.py`) + objectives (`_objectives.py`)
Single `fit_boosting(objective, X, y, ...)` loop. Facades supply an
`Objective` (`DistributionObjective`, `FormulaObjective`,
`WeibullAFTObjective`, `LossObjective`). GPU path: device-resident raw
scores when the objective is device-capable; `fit_tree_gpu_native` per
channel.

### Persistence (`_persistence.py`)
`PersistenceMixin` provides `save()`/`load()` on all models. Generic `ob.load(path)` auto-detects model class from saved state.
Expand All @@ -85,12 +106,12 @@ DART, LinearLeaf growth strategies
## Key Conventions

- **Python 3.10+** target. Ruff rules: E, F, I, UP, B, SIM (line length 100; E501, E402, F821 ignored).
- **uv only** for package management — never `pip install` or `conda`.
- **uv only** for package management: never `pip install` or `conda`.
- All Numba-jitted functions use `@njit` or `@cuda.jit`. CPU kernels are in `_backends/_cpu.py`, CUDA in `_backends/_cuda.py`.
- Test environment variable `OPENBOOST_BACKEND=cpu` forces CPU backend in CI.
- Tests use `pytest-xdist` (`-n auto --dist loadfile`) for parallel execution. Shared fixtures are in `tests/conftest.py` (session-scoped datasets, function-scoped gradients).
- **GPU-native builder** (`fit_tree_gpu_native`) does not support missing values or categorical features. The training loop in `_boosting.py` auto-falls back to `fit_tree()` with a warning when the data has NaN or categorical columns.
- **Callbacks**: all models — `GradientBoosting`, `MultiClassGradientBoosting`, `DART`, `NaturalBoost`/`DistributionalGBDT`, `LinearLeafGBDT`, and `OpenBoostGAM` — support `callbacks` and `eval_set` in `fit()`.
- **Callbacks**: every model (`GradientBoosting`, `MultiClassGradientBoosting`, `DART`, `NaturalBoost`/`DistributionalGBDT`, `FormulaBoost`, `WeibullAFT`, `LinearLeafGBDT`, `OpenBoostGAM`) supports `callbacks` and `eval_set` in `fit()`. FormulaBoost eval tuples are `(X, y, model_input)`; WeibullAFT tuples are `(X, y[, event])`.
- **`random_state`**: `GradientBoosting` and `DART` accept `random_state` for reproducibility. Sklearn wrappers pass it through. DART also accepts `seed` (alias).
- **`suggest_params()`**: Returns sklearn-style names by default (`n_estimators`). Pass `style='core'` to get core API names (`n_trees`).
- **Profiling**: `ProfilingCallback` wraps core primitives with timers. Enable via callback or `OPENBOOST_PROFILE=1` env var. Reports go to `logs/` as JSON.
Expand Down
Loading
Loading