Skip to content

Keep the LoRA runtime warm between jobs - #1997

Draft
j316chuck wants to merge 4 commits into
NovaSky-AI:mainfrom
j316chuck:chuck-codex/keep-runtime-warm-live
Draft

Keep the LoRA runtime warm between jobs#1997
j316chuck wants to merge 4 commits into
NovaSky-AI:mainfrom
j316chuck:chuck-codex/keep-runtime-warm-live

Conversation

@j316chuck

@j316chuck j316chuck commented Aug 6, 2026

Copy link
Copy Markdown
Contributor
  • Add opt-in engine flags to prewarm inference and retain the shared Ray, worker, inference, and base-model runtime after the final LoRA unload.
  • Unload each job's LoRA and optimizer state while allowing the next compatible create_model to register a fresh adapter against the warm runtime.
  • Keep cold teardown as the default and document the multi-tenant lifecycle.

Testing

Cold Warm
Before 102.512s 1801.271s (failed)
This PR 100.815s 0.152s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant