Skip to content

Enable load_weights_only in checkpointing - #1996

Open
j316chuck wants to merge 4 commits into
NovaSky-AI:mainfrom
j316chuck:chuck-codex/optimizer-load-state
Open

Enable load_weights_only in checkpointing#1996
j316chuck wants to merge 4 commits into
NovaSky-AI:mainfrom
j316chuck:chuck-codex/optimizer-load-state

Conversation

@j316chuck

@j316chuck j316chuck commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Preserve Tinker's optimizer flag through the load-weights API, engine, and JAX/SkyRL-Train backends.
  • Restore optimizer and scheduler state only for load_state_with_optimizer(); weights-only loads leave the live optimizer untouched.
  • Keep Tinker's public optimizer field for SDK compatibility while using load_optimizer internally.

Fixes #1993.

Testing

uv run --isolated --extra dev --extra tinker --extra jax pytest tests/tinker/test_api_validation.py tests/tinker/test_engine.py tests/tinker/test_api.py::test_training_workflow -q
uv run --isolated --extra skyrl-train --extra dev pytest tests/backends/skyrl_train/test_checkpoint_loading.py -q

The API, engine, SDK workflow, and backend checkpoint tests pass. In a live H200 deployment, a cold one-step job saved a checkpoint and a second job on the same direct endpoint loaded it with load_optimizer=False, completed training, and saved a new checkpoint. The weights-only /load_weights call took 0.841s.

Signed-off-by: Charles Tang <j316chuck@users.noreply.github.com>
@j316chuck
j316chuck marked this pull request as ready for review August 6, 2026 04:57
@j316chuck j316chuck changed the title Honor optimizer choice when loading Tinker state Enable load_weights_only in checkpointing Aug 6, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an "optimizer" boolean flag to the checkpoint loading process across multiple backends (JAX, Ray JAX, and SkyRL Train) and exposes it via the Tinker API, enabling optional restoration of optimizer and scheduler states. The feedback highlights two main areas of improvement: first, the "optimizer" field in "LoadWeightsRequest" and "LoadWeightsInput" should default to "False" to preserve backward compatibility with older clients; second, a safety check should be added in the JAX backend to prevent a potential "KeyError" if "optimizer_state" is missing from the checkpoint data.

Comment thread skyrl/tinker/api.py Outdated
class LoadWeightsRequest(BaseModel):
model_id: str
path: str
optimizer: bool

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Adding optimizer as a required boolean field in LoadWeightsRequest without a default value breaks backward compatibility with older clients or SDKs that do not send this field in their payload. Providing a default value (e.g., False) ensures that the API remains backward-compatible and robust against missing fields.

Suggested change
optimizer: bool
optimizer: bool = False

Comment thread skyrl/tinker/types.py Outdated
Comment thread skyrl/backends/jax.py Outdated
Comment thread skyrl/tinker/api.py Outdated
Comment thread skyrl/tinker/types.py
class LoadWeightsInput(BaseModel):
source_model_id: str
checkpoint_id: str
load_optimizer: bool = True

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clean

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support weights-only checkpoint loading

2 participants