Skip to content

Put the local-model daemon back on its defaults for an untuned assessment - #4782

Merged
atomantic merged 1 commit into
mainfrom
claim/issue-4759
Aug 21, 2026
Merged

Put the local-model daemon back on its defaults for an untuned assessment#4782
atomantic merged 1 commit into
mainfrom
claim/issue-4759

Conversation

@atomantic

@atomantic atomantic commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Summary

An untuned assessment only applied its launch tuning when the run had knobs. So a daemon the previous run tuned kept serving under that tuning while the reading was stored with tuningKey: '' and rendered as Backend defaults — and compareTunings ranked every real tuning against it, making the one row a user reads as the baseline the least trustworthy row in the table. #4775 made an empty knob set meaningful for llama.cpp behind reset, but an ordinary Measure still took the no-applier branch on every runtime.

An empty knob set is now a real instruction wherever the manager can honour it (resetsOnEmpty), for a plain Measure as well as a sweep baseline:

  • llama.cpp — restores the launch line PortOS displaced when it first tuned the daemon. Deliberately not CLEARED_TUNING: that clears sweepable knobs the user may have set on the LLMs page, which only a sweep — having captured them — is entitled to do. Restoring the captured line undoes what PortOS applied and nothing else, so reset stays opt-in and sweep-only exactly as documented.
  • Ollama — actively unsets what it exported. launchctl setenv writes into a launchd domain that outlives the daemon, so omitting a variable from the next restart does not clear it. The baseline is the env in effect before the tuning, not an empty one: ensureContextWindow puts the user's agent context window on the same OLLAMA_CONTEXT_LENGTH a tuning uses, and stripping it would undo a setting they chose.
  • LM Studio — reloads through lms load with no flags, tracking per model which loads carried them so an already-default model is not cold-loaded for nothing.
  • MTPLXresetsOnEmpty: false. Its manager refuses an empty knob set rather than relaunching without flags, so an untuned run must not claim it reset anything.

This closes #4763 as well: its three scope items are the Ollama and LM Studio resets and the flag flip.

tuningApplied gains a third state at the manager boundary: null means nothing needed to change — the honest answer for a run with nothing to apply, and what keeps untuned readings in the recommendations rather than dropping them out via a false (the regression #4763 warned about). A reset that fails records false with its reason, and both the server's exclusion sentence and the client chip word that case for what it is — the daemon could not be put back on defaults — instead of "tuning was not applied", which contradicts the row's own "backend defaults" label.

Test plan

  • localModelAssessments.test.js — every applier that can reset is asked to on an untuned run; a runtime that cannot still claims nothing; a failed reset records false with its reason; llama is asked with reset: false so an ordinary Measure never clears the user's flags; the exclusion reason is worded for which direction failed.
  • llamaServerManager.test.js — restores the pre-tuning launch line; relaunches nothing when already at baseline; keeps the original baseline across successive tunings; drops it when the daemon is stopped, or when a fresh server is started over a crashed one.
  • ollamaManager.test.js — restarts without the env it applied; unsets launchd variables (including ones the next tuning stops naming, and ones exported before a failed restart); restores the pre-tuning env rather than stripping it; drops the baseline however the daemon was stopped; does not claim a variable whose setenv failed.
  • lmStudioManager.test.js — reloads without flags; no-ops when already default; refuses when a resident model would not unload; loads a non-resident model without reading its unload error as a refusal; attempts the unload when the loaded-model list could not be trusted.
  • assessmentTuningNotice.test.js plus both component suites for the reworded rows.
  • Full suites green: server 32600 passed, client 9233 passed, lint clean.

Notes

Reviewed locally over six rounds (codex ×3, claude ×3) before the PR opened; 15 findings, 14 fixed. Every guard was mutation-tested — each fix reverted in turn to confirm the corresponding test fails.

One finding was deferred rather than fixed here: getLlamaServerStatus collapses "PM2 unreadable" into "not managed", so a transient read failure can drop a model's baseline from the ranked list. It needs a change to that function's status contract, and the refusal itself is the safe direction. Filed as #4780. (mtplxServerManager already distinguishes the two — that is the pattern to follow.)

Closes #4759
Closes #4763

…4759, #4763)

An untuned assessment only applied its launch tuning when the run HAD
knobs, so a daemon the previous run tuned kept serving under that tuning
while the reading was stored with `tuningKey: ''` and rendered as
"Backend defaults". `compareTunings` then ranked every real tuning
against it — the one row a user reads as the baseline was the least
trustworthy row in the table. The sweep work made an empty knob set
meaningful for llama.cpp behind `reset`, but an ordinary Measure still
took the no-applier branch on every runtime.

An empty knob set is now a real instruction wherever the manager can
honour it (`resetsOnEmpty`), for a plain Measure as well as a sweep
baseline:

- llama.cpp restores the launch line PortOS displaced when it first
  tuned the daemon. Deliberately NOT `CLEARED_TUNING`: that clears
  sweepable knobs the USER may have set on the LLMs page, which only a
  sweep — having captured them — is entitled to do. Restoring the
  captured line undoes what PortOS applied and nothing else, so `reset`
  stays opt-in and sweep-only exactly as documented.
- Ollama actively unsets what it exported. `launchctl setenv` writes into
  a launchd domain that outlives the daemon, so omitting a variable from
  the next restart does not clear it. The baseline is the env in effect
  before the tuning, not an empty one — `ensureContextWindow` puts the
  user's agent context window on the same `OLLAMA_CONTEXT_LENGTH` a
  tuning uses, and stripping it would undo a setting they chose.
- LM Studio reloads through `lms load` with no flags, tracking per model
  which loads carried them so an already-default model is not cold-loaded
  for nothing.
- MTPLX declares `resetsOnEmpty: false` — its manager refuses an empty
  knob set rather than relaunching without flags, so an untuned run must
  not claim it reset anything.

`tuningApplied` gains a third state at the manager boundary: `null` means
nothing needed to change, which stays the honest answer for a run with
nothing to apply — and keeps untuned readings in the recommendations
rather than dropping them out via a `false`. A reset that FAILS records
`false` with its reason, and both the server's exclusion sentence and the
client chip word that case for what it is (the daemon could not be put
back on defaults) instead of "tuning was not applied", which contradicts
the row's own "backend defaults" label.
@atomantic atomantic changed the title Relaunch the local-model daemon untuned so a baseline assessment measures backend defaults Put the local-model daemon back on its defaults for an untuned assessment Aug 21, 2026
@atomantic
atomantic merged commit bf07b1a into main Aug 21, 2026
7 checks passed
@atomantic
atomantic deleted the claim/issue-4759 branch August 21, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant