Put the local-model daemon back on its defaults for an untuned assessment - #4782
Merged
Conversation
…4759, #4763) An untuned assessment only applied its launch tuning when the run HAD knobs, so a daemon the previous run tuned kept serving under that tuning while the reading was stored with `tuningKey: ''` and rendered as "Backend defaults". `compareTunings` then ranked every real tuning against it — the one row a user reads as the baseline was the least trustworthy row in the table. The sweep work made an empty knob set meaningful for llama.cpp behind `reset`, but an ordinary Measure still took the no-applier branch on every runtime. An empty knob set is now a real instruction wherever the manager can honour it (`resetsOnEmpty`), for a plain Measure as well as a sweep baseline: - llama.cpp restores the launch line PortOS displaced when it first tuned the daemon. Deliberately NOT `CLEARED_TUNING`: that clears sweepable knobs the USER may have set on the LLMs page, which only a sweep — having captured them — is entitled to do. Restoring the captured line undoes what PortOS applied and nothing else, so `reset` stays opt-in and sweep-only exactly as documented. - Ollama actively unsets what it exported. `launchctl setenv` writes into a launchd domain that outlives the daemon, so omitting a variable from the next restart does not clear it. The baseline is the env in effect before the tuning, not an empty one — `ensureContextWindow` puts the user's agent context window on the same `OLLAMA_CONTEXT_LENGTH` a tuning uses, and stripping it would undo a setting they chose. - LM Studio reloads through `lms load` with no flags, tracking per model which loads carried them so an already-default model is not cold-loaded for nothing. - MTPLX declares `resetsOnEmpty: false` — its manager refuses an empty knob set rather than relaunching without flags, so an untuned run must not claim it reset anything. `tuningApplied` gains a third state at the manager boundary: `null` means nothing needed to change, which stays the honest answer for a run with nothing to apply — and keeps untuned readings in the recommendations rather than dropping them out via a `false`. A reset that FAILS records `false` with its reason, and both the server's exclusion sentence and the client chip word that case for what it is (the daemon could not be put back on defaults) instead of "tuning was not applied", which contradicts the row's own "backend defaults" label.
atomantic
force-pushed
the
claim/issue-4759
branch
from
August 21, 2026 19:58
e76c295 to
c485bd9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An untuned assessment only applied its launch tuning when the run had knobs. So a daemon the previous run tuned kept serving under that tuning while the reading was stored with
tuningKey: ''and rendered as Backend defaults — andcompareTuningsranked every real tuning against it, making the one row a user reads as the baseline the least trustworthy row in the table. #4775 made an empty knob set meaningful for llama.cpp behindreset, but an ordinary Measure still took the no-applier branch on every runtime.An empty knob set is now a real instruction wherever the manager can honour it (
resetsOnEmpty), for a plain Measure as well as a sweep baseline:CLEARED_TUNING: that clears sweepable knobs the user may have set on the LLMs page, which only a sweep — having captured them — is entitled to do. Restoring the captured line undoes what PortOS applied and nothing else, soresetstays opt-in and sweep-only exactly as documented.launchctl setenvwrites into a launchd domain that outlives the daemon, so omitting a variable from the next restart does not clear it. The baseline is the env in effect before the tuning, not an empty one:ensureContextWindowputs the user's agent context window on the sameOLLAMA_CONTEXT_LENGTHa tuning uses, and stripping it would undo a setting they chose.lms loadwith no flags, tracking per model which loads carried them so an already-default model is not cold-loaded for nothing.resetsOnEmpty: false. Its manager refuses an empty knob set rather than relaunching without flags, so an untuned run must not claim it reset anything.This closes #4763 as well: its three scope items are the Ollama and LM Studio resets and the flag flip.
tuningAppliedgains a third state at the manager boundary:nullmeans nothing needed to change — the honest answer for a run with nothing to apply, and what keeps untuned readings in the recommendations rather than dropping them out via afalse(the regression #4763 warned about). A reset that fails recordsfalsewith its reason, and both the server's exclusion sentence and the client chip word that case for what it is — the daemon could not be put back on defaults — instead of "tuning was not applied", which contradicts the row's own "backend defaults" label.Test plan
localModelAssessments.test.js— every applier that can reset is asked to on an untuned run; a runtime that cannot still claims nothing; a failed reset recordsfalsewith its reason; llama is asked withreset: falseso an ordinary Measure never clears the user's flags; the exclusion reason is worded for which direction failed.llamaServerManager.test.js— restores the pre-tuning launch line; relaunches nothing when already at baseline; keeps the original baseline across successive tunings; drops it when the daemon is stopped, or when a fresh server is started over a crashed one.ollamaManager.test.js— restarts without the env it applied; unsets launchd variables (including ones the next tuning stops naming, and ones exported before a failed restart); restores the pre-tuning env rather than stripping it; drops the baseline however the daemon was stopped; does not claim a variable whosesetenvfailed.lmStudioManager.test.js— reloads without flags; no-ops when already default; refuses when a resident model would not unload; loads a non-resident model without reading its unload error as a refusal; attempts the unload when the loaded-model list could not be trusted.assessmentTuningNotice.test.jsplus both component suites for the reworded rows.Notes
Reviewed locally over six rounds (codex ×3, claude ×3) before the PR opened; 15 findings, 14 fixed. Every guard was mutation-tested — each fix reverted in turn to confirm the corresponding test fails.
One finding was deferred rather than fixed here:
getLlamaServerStatuscollapses "PM2 unreadable" into "not managed", so a transient read failure can drop a model's baseline from the ranked list. It needs a change to that function's status contract, and the refusal itself is the safe direction. Filed as #4780. (mtplxServerManageralready distinguishes the two — that is the pattern to follow.)Closes #4759
Closes #4763