Skip to content

Calibrate the CMA-ES stagnation tolerance from the objective, not from a step length - #654

Merged
wshlavacek merged 1 commit into
mainfrom
fix/cmaes-tolfun-objective-scale
Aug 24, 2026
Merged

Calibrate the CMA-ES stagnation tolerance from the objective, not from a step length#654
wshlavacek merged 1 commit into
mainfrom
fix/cmaes-tolfun-objective-scale

Conversation

@wshlavacek

Copy link
Copy Markdown
Collaborator

Closes #653. This is the sibling of #648, in the optimizer #648's own ADR says it mirrors,
and it should have been found and fixed in the same pass.

The defect

ADR-0106 gave cmaes_tolfun its own key because it is a range in objective units while
cmaes_stop_tol is a step length in sampling space. Then it had an unset cmaes_tolfun
fall back to it anyway. The comment sitting on the line above the fallback says the two
"have no common scale and cannot share one well-set value". The line below makes them
share one. cmaes_stop_tol defaults to 1e-11, so the stagnation range was 1e-11 in
objective units.

This is #648 in the mirror.

DE (#648) CMA-ES (this)
inherited stop_tolerance = 0.002, a dimensionless ratio cmaes_stop_tol = 1e-11, a sampling-space step
read as absolute objective range absolute objective range
effect far too loose far too strict
result stops mid-descent, reports a wrong answer as converged TolFun never fires

What the strict direction costs is the trigger the restart battery exists for. Its own
docstring calls it "the trigger the reproduction problems need", and the battery is there
because otherwise a run "polishes a local basin forever and never yields to a restart (the
IPOP/BIPOP machinery silently degenerates to one trapped run)". TolX and ConditionCov are
unaffected.

docs/config_keys.rst already told readers the default was "rarely what you want if you
rely on stagnation restarts". The defect was documented rather than fixed.

Why #648's remedy does not transfer

#648 was repaired by restoring a legacy meaning: stop_tolerance had always been a ratio,
so reading it as one again returned to known-correct behaviour. cmaes_stop_tol was never
an objective quantity, so a default has to be invented rather than restored.

Two candidates were rejected before the third, and the second is worth recording:

  • A fraction of the current objective. What ADR-0106 removed, correctly. On a
    likelihood |f| grows as the fit improves, so the threshold rises fastest where firing
    it costs most.
  • A fraction of the window being tested. Circular, and silently fatal:
    frange <= fraction * frange is never true for a small fraction, so the trigger is
    disabled rather than corrected. I tried this first and ADR-0106's own
    test_cmaes_tolfun_still_fires_on_a_genuinely_flat_history caught it immediately. That
    is the value of having kept that test.

The fix

An unset cmaes_tolfun is 1e-11 times the objective spread across the first scored
generation's population
.

  • Right units, taken from the objective rather than borrowed across a unit boundary.
  • Does not drift with |f|, because it is fixed at the first generation, before
    anything has converged. ADR-0106's objection does not reach it.
  • Not the window under test, so it is not circular.
  • Calibrated once; every IPOP/BIPOP restart reuses it. A later restart starts nearer
    the optimum and would measure a smaller spread, so recalibrating would hold the late,
    large-population restarts to the strictest bar, which is the shape of failure ADR-0106
    fixed.

The fraction is chosen so a problem whose initial population spans one objective unit gets
exactly the 1e-11 this key always defaulted to. A reference-scaled problem is unchanged
by construction; everything else scales in proportion. The run logs the value it picked.

A generation that cannot supply a spread (fewer than two finite scores, or all identical)
keeps the old fallback rather than inventing a number or setting zero. An explicit
cmaes_tolfun is never touched.

Evidence

All three of ADR-0106's regression tests pass unchanged. They construct synthetic
distribution state without scoring a generation, so they still exercise the fallback and
still assert alg.tolfun == alg.stop_tol.

Six new tests: the defect at its decision point (including that the old threshold stays
silent on a run the new one correctly stops), the anchor value, scaling across six decades,
the explicit key, three degenerate populations, and the no-recalibration-on-restart rule.

Full suite including the slow and recovery tiers CI skips: 4778 passed, 13 skipped, none
failed
.

Scope

Only reachable with cmaes_restarts > 0, which is not the default, and nothing in the
shipped corpus sets it. It cannot produce a wrong answer; it weakens the search.

The class, swept

A grep for the same pattern finds exactly two fallbacks in the codebase that cross a unit
boundary this way: de_tolfun and cmaes_tolfun. Both are now resolved. cmaes_run_maxgen
also defaults from unset, but to infinity rather than to another key, so no boundary is
crossed. ADR-0128 records the lesson: treat a defect whose ADR names a sibling as a defect
in a class, and check the sibling in the same pass.

…m a step length (#653)

This is the sibling of #648, in the optimizer #648's own ADR says it mirrors.

ADR-0106 gave cmaes_tolfun its own key because it is a range in objective units
while cmaes_stop_tol is a step length in sampling space, and then had an unset
cmaes_tolfun fall back to it anyway. The comment on the line above the fallback
says the two have no common scale and cannot share one well-set value. The line
below it makes them share one. cmaes_stop_tol defaults to 1e-11, so the
stagnation range was 1e-11 in objective units.

That is #648 in the mirror. There a dimensionless ratio read as an objective
range was far too loose, and fits stopped early reporting a wrong answer. Here a
step length read as an objective range is far too strict, so on an objective of
ordinary magnitude the trigger never fires. What that costs is the trigger the
restart battery exists for: without it a run polishes a local basin and never
yields to a restart, which is the failure the battery was built to prevent. The
documentation already told readers the default was rarely what they wanted,
which described the defect rather than fixing it.

The #648 remedy does not transfer. There, stop_tolerance had always been a ratio,
so reading it as one again returned to known-correct behaviour. cmaes_stop_tol
was never an objective quantity, so a default has to be invented rather than
restored. Two candidates were rejected first. A fraction of the current objective
is what ADR-0106 removed, because a likelihood's magnitude grows as the fit
improves. A fraction of the window being tested is circular, and silently
disables the trigger rather than fixing it; ADR-0106's own flat-history test
caught that attempt, which is the value of having kept it.

An unset cmaes_tolfun is now 1e-11 times the objective spread across the first
scored generation's population. That spread measures how much this objective
varies over the search box, it is in the units the tolerance needs, and it is
taken before anything has converged so it does not drift with the objective. It
is calibrated once and every restart reuses it, so a late restart is not held to
a stricter bar than an early one. The fraction is chosen so a problem whose
initial population spans one objective unit gets exactly the 1e-11 this key
always defaulted to, leaving a reference-scaled problem unchanged. A generation
that cannot supply a spread keeps the old fallback rather than inventing a
number, and an explicit cmaes_tolfun is never touched.

All three of ADR-0106's regression tests pass unchanged: they build synthetic
distribution state without scoring a generation, so they still exercise the
fallback. Six new tests cover the calibration, the anchor value, the scaling
across decades, the explicit key, the degenerate populations and the
no-recalibration-on-restart rule.

A sweep for the same pattern finds exactly two fallbacks in the codebase that
cross a unit boundary this way, de_tolfun and cmaes_tolfun. Both are now
resolved. cmaes_run_maxgen also defaults from unset, but to infinity rather than
to another key, so no boundary is crossed.

Only affects cmaes_restarts > 0, which is not the default. Full suite including
the slow and recovery tiers: 4778 passed, 13 skipped, none failed.
@wshlavacek
wshlavacek merged commit 6f08137 into main Aug 24, 2026
9 checks passed
@wshlavacek
wshlavacek deleted the fix/cmaes-tolfun-objective-scale branch August 24, 2026 03:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An unset cmaes_tolfun inherits a sampling-space step length as an objective range, so the TolFun restart trigger never fires

1 participant