Enable load_weights_only in checkpointing - #1996
Conversation
Signed-off-by: Charles Tang <j316chuck@users.noreply.github.com>
There was a problem hiding this comment.
Code Review
This pull request introduces an "optimizer" boolean flag to the checkpoint loading process across multiple backends (JAX, Ray JAX, and SkyRL Train) and exposes it via the Tinker API, enabling optional restoration of optimizer and scheduler states. The feedback highlights two main areas of improvement: first, the "optimizer" field in "LoadWeightsRequest" and "LoadWeightsInput" should default to "False" to preserve backward compatibility with older clients; second, a safety check should be added in the JAX backend to prevent a potential "KeyError" if "optimizer_state" is missing from the checkpoint data.
| class LoadWeightsRequest(BaseModel): | ||
| model_id: str | ||
| path: str | ||
| optimizer: bool |
There was a problem hiding this comment.
Adding optimizer as a required boolean field in LoadWeightsRequest without a default value breaks backward compatibility with older clients or SDKs that do not send this field in their payload. Providing a default value (e.g., False) ensures that the API remains backward-compatible and robust against missing fields.
| optimizer: bool | |
| optimizer: bool = False |
Signed-off-by: Charles Tang <j316chuck@users.noreply.github.com>
| class LoadWeightsInput(BaseModel): | ||
| source_model_id: str | ||
| checkpoint_id: str | ||
| load_optimizer: bool = True |
Summary
optimizerflag through the load-weights API, engine, and JAX/SkyRL-Train backends.load_state_with_optimizer(); weights-only loads leave the live optimizer untouched.optimizerfield for SDK compatibility while usingload_optimizerinternally.Fixes #1993.
Testing
The API, engine, SDK workflow, and backend checkpoint tests pass. In a live H200 deployment, a cold one-step job saved a checkpoint and a second job on the same direct endpoint loaded it with
load_optimizer=False, completed training, and saved a new checkpoint. The weights-only/load_weightscall took 0.841s.