Let a differential evolution fit converge on its own objective's scale - #651
Merged
Conversation
#648) The convergence fix in #561 replaced a dimensionless ratio with an absolute objective range, and let an unset de_tolfun go on using stop_tolerance's number so that existing configurations kept the threshold magnitude they had. ADR-0115 justified reading that number as a range by arguing that an absolute range at the same magnitude is a stricter stop. It is stricter only above an objective of 1. Below that it is looser, and without limit, because the two thresholds separate as the objective falls. At the 2e-05 that a well scaled sum of squares fit reaches, a range of 0.002 stops the run at a spread fifty thousand times wider than the ratio of 0.002 it replaced, so the population satisfies it almost as soon as it is scored. What that costs is a wrong answer that looks right. Tutorial lesson 25 fits three pharmacokinetic rates from a single observed curve. It stopped early and reported k_transit as 11.18 against a true 12.76 and k_abs as 11.50 against a true 9.11, at an objective of 2.19e-05 and with the third rate correct to three digits. Those two rates trade against each other, so the early stop fits the data well and nothing in the output looks wrong. Whether de_tolfun was set now decides what the threshold means. An explicit de_tolfun is an absolute range in the objective's own units and is honoured as written, at any sign and any scale, exactly as before. An unset one falls back to stop_tolerance, which keeps the meaning that key has always had: a dimensionless ratio where the objective is positive, applied as max - min <= tol * min so that an all-zero population never divides, and an absolute range only where the objective is not positive and a ratio would mean nothing. So a positive-objective fit converges as it did in 1.7.0, a likelihood fit keeps the #561 fix, and one value of stop_tolerance means the same thing on fits whose objectives are decades apart. Ignoring failed simulations, and the island guard, are untouched. All six of the #561 regression tests use non-positive fitnesses, so every one of them stays on the range branch and passes unchanged. Four new tests cover this defect, each checked by reverting the fix and watching it fail: a positive population whose spread is small absolutely but large relative to the objective is not converged, an explicit de_tolfun is still read as a range, one value gives the same verdict at objectives six decades apart, and a population straddling zero takes the range branch. ADR-0127 records the correction and ADR-0115 carries a note pointing at it. The algorithms page and the two configuration key entries are rewritten, since both described the threshold as a range in every case. Verified on the full suite including the slow and recovery tiers that continuous integration skips: 4772 passed, 13 skipped, none failed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #648. This is the fix that should land before 1.8.0 is cut.
The false premise
#561 replaced the family's convergence test with an absolute objective range and gave it
a new key,
de_tolfun. Both were right. It also let an unsetde_tolfungo on usingstop_tolerance's number, so existing configurations kept the threshold magnitude theyhad, and ADR-0115 justified reading that number as a range like this:
The last clause is false. The range is stricter only above an objective of 1, and below
it the two separate without limit:
A sum of squares fit on well scaled data lands in the bottom row as a matter of course.
What it cost
Tutorial lesson 25 fits three pharmacokinetic rates from one observed curve. It stopped
early and reported
k_transitas 11.18 against a true 12.76, andk_absas 11.50 againsta true 9.11, at an objective of 2.19e-05, with the third rate correct to three digits.
Those two rates trade against each other, so the early stop fits the data well. Nothing in
the output looks wrong, which is what makes this worth fixing before a release rather than
after one.
The fix
Whether
de_tolfunwas set now decides what the threshold means.de_tolfunis an absolute range in the objective's own units, chosen byan author who can see that scale. Honoured as written, at any sign and any scale,
exactly as before.
stop_tolerance, which keeps the meaning that key hasalways had. Where the objective is positive it is a dimensionless ratio, applied as
max - min <= tol * min, which is themax / min <= 1 + tolthis family used beforede/ade stop after generation 0 on any negative objective: the convergence test is a ratio, not a range (the DE sibling of #550/ADR-0106) #561, written without the division so an all-zero population cannot divide by zero.
Where the objective is not positive a ratio means nothing, so the number is read as an
absolute range, which is the branch de/ade stop after generation 0 on any negative objective: the convergence test is a ratio, not a range (the DE sibling of #550/ADR-0106) #561 needs.
Ignoring failed simulations, and the island guard, are #561's and are untouched.
The narrower reading is the point. #561's defect is not that a ratio is wrong, it is that
a ratio was applied where a ratio has no meaning. Applied where it does, the ratio is the
only one of the two forms that gives a single default the same meaning across fits whose
objectives differ by many decades, which is what a default has to do.
Evidence
All six of #561's regression tests use non-positive fitnesses, so every one stays on the
range branch and passes unchanged. That is the strongest single signal that this gives
nothing back: the fix cannot reach the regime #561 was about.
Four new tests, each checked by reverting the fix and watching it fail rather than only by
watching it pass. Six of the eight parametrized cases fail without it. The two that pass
either way are the straddling-zero boundary guards, which cover the branch that did not
change.
Full suite including the slow and recovery tiers that CI skips: 4772 passed, 13 skipped,
none failed. Before the fix the same run was 4763 passed and 1 failed. Runtime went from
1056 to 1118 seconds, about 6 percent, because fits that were stopping early now run to
an actual convergence.
Documentation
ADR-0127 records the correction. ADR-0115 carries a note pointing at it rather than being
rewritten, since everything else it decided still stands. The algorithms page and both
configuration key entries are rewritten, because all three described the threshold as a
range in every case.