Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 13 additions & 19 deletions baseline/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ The audit and qualification documents are:
| **MNIST / MLP3** | 55k optimization / 5k validation; official 10k test monitoring-only | `784 → 512 → 512 → 10`, ReLU | SGD + Nesterov, AdamW, Muon + auxiliary AdamW | [`notebooks/MNIST_MLP3_Baseline_Comparison.ipynb`](notebooks/MNIST_MLP3_Baseline_Comparison.ipynb) |
| **CIFAR-10 / small ViT** | 45k optimization / 5k validation; official 10k test monitoring-only | 4×4 patches, width 192, 6 blocks, 3 heads | SGD + Nesterov, AdamW, Muon + auxiliary AdamW | [`notebooks/CIFAR10_ViT_Optimizer_Baselines.ipynb`](notebooks/CIFAR10_ViT_Optimizer_Baselines.ipynb) |
| **One-head nanoGPT / FineWeb-Edu** | Pinned `sample-10BT`; exact document-disjoint 80M / 1M / 1M GPT-2-BPE splits | 1 block, 1 head, width 128, context 256 | SGD + Nesterov, AdamW, Muon + auxiliary AdamW | [`nanogpt_one_head/README.md`](nanogpt_one_head/README.md) |
| **NGB v4 / FineWeb-Edu** | Same pinned document-disjoint corpus | Separate 1×1 control and 4-block/4-head width-128 model | Two-epoch tuned SGD, AdamW, and Muon protocols | [`ngb/README.md`](ngb/README.md) |
| **nanochat d12** | Native pinned nanochat data/tokenizer pipeline | 12 layers, width 768, context 2048 | Native nanochat Muon + AdamW | [`notebooks/NanoChat_D12_Reference_Baseline.ipynb`](notebooks/NanoChat_D12_Reference_Baseline.ipynb) |
| **nanochat mac_d4** | Separately cached reduced nanochat preparation | 4 layers, width 256, context 512 | Same pinned upstream optimizer mathematics | Same notebook; auto-selected on MPS/CPU |

Expand All @@ -56,35 +57,28 @@ report a `mac_d4` result as d12.
neighborhood for the exact architecture, data, initialization, budget,
optimizer implementation, and runtime policy.

## Environment and persistent paths
## Environment and runtime paths

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[experiment]'
```

When installing from the repository root, use:
Use the currently active conda environment; do not create a repository venv.
From the repository root:

```bash
python -m pip install -e './baseline[experiment]'
python -m pip install -e './baseline/nanogpt_one_head[dev]'
```

Set persistent locations before running long jobs:

```bash
export RG_BASELINE_DATA_DIR="$HOME/rg-optimizer-data"
export RG_BASELINE_RUN_ROOT="$HOME/rg-optimizer-runs"
```

The isolated one-head nanoGPT suite uses:
All local data and long-running experiment artifacts use explicit `/tmp` roots:

```bash
export RG_NANOGPT_ONE_HEAD_ROOT="$HOME/rg-nanogpt-one-head"
export RG_BASELINE_DATA_DIR=/tmp/rg-optimizer-data
export RG_BASELINE_RUN_ROOT=/tmp/rg-optimizer-runs
export RG_NANOGPT_ONE_HEAD_ROOT=/tmp/rg-nanogpt-one-head
export RG_NGB_ROOT=/tmp/rg-ngb
export RG_NGB_DATA_ROOT=/tmp/rg-nanogpt-one-head/data
```

Do not put long-running results in `/tmp`.
NGB v4 keeps its results under `/tmp/rg-ngb/results/<run_slug>` and reuses the
verified FineWeb-Edu token cache under `/tmp/rg-nanogpt-one-head/data`.

## 1. MNIST / MLP3

Expand Down
18 changes: 17 additions & 1 deletion baseline/nanogpt_one_head/src/rg_nanogpt_one_head/__init__.py
Original file line number Diff line number Diff line change
@@ -1,25 +1,33 @@
"""One-block, one-head nanoGPT optimizer baselines on pinned FineWeb-Edu."""
"""Shared nanoGPT optimizer-baseline runtime for v3 and NGB v4."""

from .analysis import (
MATRIX_COLORS,
OPTIMIZER_COLORS,
OPTIMIZER_LABELS,
discover_complete_seeds,
discover_matched_complete_seeds,
final_test_summary,
load_epoch_metrics,
load_layer_metrics,
load_metrics,
load_spectral_summary,
load_test_results,
mean_ci95,
paired_test_differences,
plot_epoch_metric,
plot_layer_metric,
plot_spectral_optimizer_summary,
run_diagnostics_table,
run_status_table,
summarize_run_diagnostics,
)
from .config import (
SUPPORTED_OPTIMIZERS,
canonical_seeds,
expected_transformer_matrix_count,
load_config,
roots,
run_slug,
)
from .data import prepare_fineweb_edu
from .model import GPT, GPTConfig
Expand All @@ -39,20 +47,28 @@
"SUPPORTED_OPTIMIZERS",
"canonical_seeds",
"choose_device",
"discover_complete_seeds",
"discover_matched_complete_seeds",
"expected_transformer_matrix_count",
"final_test_summary",
"load_config",
"load_epoch_metrics",
"load_layer_metrics",
"load_metrics",
"load_spectral_summary",
"load_test_results",
"mean_ci95",
"paired_test_differences",
"plot_epoch_metric",
"plot_layer_metric",
"plot_spectral_optimizer_summary",
"prepare_fineweb_edu",
"roots",
"run_all_replicates",
"run_diagnostics_table",
"run_one",
"run_optimizer_replicates",
"run_slug",
"run_status_table",
"summarize_run_diagnostics",
]
Loading
Loading