Let a differential evolution fit converge on its own objective's scale

CHANGELOG.md

@@ -778,6 +778,30 @@ All notable changes to PyBNF are documented below. This project adheres to
   control off rather than erroring. `sbml_atol` never had the hole, because
   `parse_atol_setting` has always demanded finiteness. Found while adding the per-model
   `rtol:` field of #586, which would otherwise have inherited the same check.
+- **A differential evolution fit whose objective is small no longer stops before its
+  parameters have separated (#648, ADR-0127).** The convergence fix below replaced a
+  dimensionless ratio with an absolute objective range, and let an unset `de_tolfun` keep
+  using `stop_tolerance`'s number so existing configurations kept the threshold magnitude
+  they had. ADR-0115 justified that by arguing an absolute range at the same magnitude is a
+  stricter stop. It is stricter only above an objective of 1. Below that it is looser, and
+  without limit. At the 2e-05 a well scaled sum of squares fit reaches, a range of 0.002
+  stops the run at a spread fifty thousand times wider than the ratio of 0.002 it replaced,
+  so the population satisfies it almost as soon as it is scored.
+  What that costs is a wrong answer that looks right. Tutorial lesson 25 fits three
+  pharmacokinetic rates from one observed curve, and stopped early reporting `k_transit` as
+  11.18 against a true 12.76 and `k_abs` as 11.50 against a true 9.11, at an objective of
+  2.19e-05 and with the third rate correct to three digits. The two affected rates trade
+  against each other, so the early stop fits the data well and nothing in the output looks
+  wrong.
+  Whether `de_tolfun` was set now decides what the threshold means. An explicit `de_tolfun`
+  is an absolute range in the objective's own units and is honoured as written, at any sign
+  and any scale, exactly as it was. An unset one falls back to `stop_tolerance`, which keeps
+  the meaning that key has always had: a dimensionless ratio where the objective is
+  positive, applied as `max - min <= tol * min` so an all-zero population never divides, and
+  an absolute range only where the objective is not positive and a ratio would mean nothing.
+  So a positive-objective fit converges as it did in 1.7.0, a likelihood fit keeps the fix
+  below, and one value of `stop_tolerance` means the same thing on fits whose objectives are
+  decades apart. Ignoring failed simulations, and the island guard, are unchanged.
 - **`job_type = de` and `job_type = ade` no longer stop after generation 0 on a negative
   objective (#561, ADR-0115).** The Differential Evolution family tested convergence with a
   *ratio* of objectives — `max(fit) / min(fit) < 1 + stop_tolerance` — which reads as

docs/adr/0115-the-de-family-convergence-test-is-an-absolute-objective-range-over-the-finite-population-carried-by-its-own-key-so-a-likelihood-fit-no-longer-stops-after-generation-0.md

@@ -126,10 +126,18 @@ population pass — are unaffected.
   because a restart's arithmetic (range vs window vs tolerance) was otherwise
   unrecoverable from the log; a DE convergence stop has no such hidden arithmetic.
 * **The default magnitude.** `de_tolfun` unset is `stop_tolerance` (default `0.002`).
-  A fit that wants the old *relative* semantics has no equivalent — but that semantics was
+  A fit that wants the old *relative* semantics has no equivalent -- but that semantics was
   only ever meaningful for a positive objective, where an absolute range at the same
   magnitude is a stricter, well-defined stop.
 
+  > **Corrected by ADR-0127 (issue #648).** The last clause is wrong. An absolute range at
+  > the same magnitude is stricter only above an objective of 1, and below it is looser
+  > without limit: at the 2e-05 a well-scaled sum-of-squares fit reaches, a 0.002 range is
+  > fifty thousand times looser than the 0.002 ratio it replaced. An unset `de_tolfun` now
+  > keeps `stop_tolerance`'s ratio meaning where the objective is positive, and reads it as
+  > a range only where a ratio has none. An explicit `de_tolfun` is unchanged, and so is
+  > everything else this ADR decided.
+
 ## Consequences
 
 * **The DE family runs on a likelihood objective.** On the reported fit the run no longer

docs/adr/0127-the-de-family-fallback-threshold-keeps-the-legacy-keys-ratio-meaning-because-a-ratios-magnitude-read-as-a-range-is-looser-without-limit-on-any-objective-below-one.md

@@ -0,0 +1,86 @@
+# The Differential Evolution family's fallback threshold keeps the legacy key's ratio meaning, because a ratio's magnitude read as a range is looser without limit on any objective below one (issue #648)
+
+## Status
+
+Accepted. Corrects one claim in ADR-0115, and changes nothing else it decided.
+
+## The defect
+
+ADR-0115 replaced the family's convergence test with an absolute objective range and gave
+it a new key, `de_tolfun`. Both were right. It also decided that an unset `de_tolfun`
+falls back to `stop_tolerance`, so an existing configuration keeps the number it had, and
+justified reading that number as a range like this:
+
+> A fit that wants the old *relative* semantics has no equivalent — but that semantics was
+> only ever meaningful for a positive objective, where an absolute range at the same
+> magnitude is a stricter, well-defined stop.
+
+The claim that the range is stricter is true only above an objective of 1. Below it the
+range is looser, and without limit, because the two thresholds diverge as the objective
+falls:
+
+| objective the population reaches | ratio 0.002 stops at a spread of | range 0.002 stops at a spread of | range is |
+|---|---|---|---|
+| 1000 | 2 | 0.002 | 1000x stricter |
+| 1 | 0.002 | 0.002 | the same |
+| 2e-05 | 4e-08 | 0.002 | 50000x looser |
+
+A sum of squares fit on well scaled data lands in the bottom row as a matter of course.
+There the population satisfies the stop almost as soon as it is scored, and the run ends
+while its parameters are still far apart.
+
+## Observed
+
+Tutorial lesson 25 fits a transit compartment pharmacokinetic model, observing only the
+central compartment. It reaches an objective of 2.19e-05 and stops there, reporting
+`k_transit` as 11.18 against a true 12.76 and `k_abs` as 11.50 against a true 9.11. The two
+rates trade against each other, so the fit that stops early fits the data well and reports
+a wrong answer with nothing wrong on its face. `k_elim` is recovered to three digits, which
+is what makes the failure hard to see.
+
+The lesson passes at v1.7.0 and at cd6e8531, and fails at 8e006374, which is ADR-0115's
+commit. Adding `de_tolfun = 1e-10` to the unchanged lesson recovers all three rates.
+
+Of the 58 tutorial configurations that use `de` or `ade`, none sets either key, so all 58
+ran on the reinterpreted number. 45 of them are checked for parameter recovery and 44 still
+recovered, so the reach is one fit in 45. The reach is narrow because most fits separate
+their parameters before the stop can cut them short. It is not narrow by design.
+
+## The decision
+
+**Whether `de_tolfun` was set decides what the threshold means.**
+
+* An **explicit** `de_tolfun` is an absolute range in objective units, chosen by an author
+  who can see their own objective's scale. It is honoured as written, at any sign and any
+  scale. ADR-0115's key keeps exactly the meaning it was given.
+* An **unset** `de_tolfun` falls back to `stop_tolerance`, and that key keeps the meaning it
+  has always had. Where the objective is positive it is a dimensionless ratio, applied as
+  `max - min <= tol * min`. That is the `max / min <= 1 + tol` this family used before #561,
+  written without the division so an all-zero population cannot divide by zero. Where the
+  objective is not positive a ratio means nothing, so the number is read as an absolute
+  range, which is the branch ADR-0115 built and the one #561 needs.
+
+The test is still assessed over the finite fitnesses only, and the island guard still holds.
+Both are ADR-0115's, both were correct, and neither changes here.
+
+## Why not a different default
+
+Deriving a range from the objective the population has reached is the same arithmetic as
+the ratio, reached by a longer route and with a new number to explain. Choosing a smaller
+absolute default only moves the problem: any fixed range is wrong by the ratio of two
+objectives' scales, and PyBNF fits objectives spanning many decades.
+
+Keeping the ratio was rejected wholesale by ADR-0115 because it is the source of #561. It
+is not. #561 is a ratio applied where a ratio has no meaning. Applied where it does, a ratio
+is the only one of the two forms that gives one value the same meaning on every fit, which
+is what a default has to do.
+
+## Consequences
+
+* Every fit that does not set `de_tolfun` and whose objective is positive converges exactly
+  as it did at v1.7.0, with #561's fix to failed simulations retained.
+* Every likelihood fit reaching #561's branch is unaffected. All six of ADR-0115's
+  regression tests use non-positive fitnesses and pass unchanged.
+* A configuration that set `de_tolfun` explicitly is unaffected.
+* A configuration that set `stop_tolerance` explicitly gets that number read as a ratio
+  again, which is what it meant when it was written.

docs/algorithms.rst

@@ -131,18 +131,27 @@ Convergence
 """""""""""
 
 A run finishes either at ``max_iterations`` or when the population has converged to
-roughly the same objective value. Convergence is measured as an **absolute range** of the
-objective over the population: the run stops once the spread between the best and worst
-*finite* member falls to ``de_tolfun`` or below. Failed simulations (scored as infinity)
-are ignored, so one dead candidate can neither trigger nor block the stop. The range is in
-the units of your objective function, which is what makes it well-defined on a likelihood
-objective — a negative log-likelihood is unbounded below, so a *ratio* of objectives
-(the pre-#561 test) reads an all-negative population as converged and stops after
-generation 0. ``de_tolfun`` is unset by default and then follows ``stop_tolerance``, so an
-existing config keeps the threshold magnitude it had; set ``de_tolfun`` to control the
-objective range independently of that key. In an island run, convergence is assessed only
-once every island has completed at least one iteration, so a single finished island cannot
-stop the whole search before the others have run.
+roughly the same objective value. Convergence is measured over the *finite* members only,
+so a failed simulation (scored as infinity) can neither trigger nor block the stop.
+
+How the threshold is read depends on where it comes from. By default ``de_tolfun`` is
+unset and the run uses ``stop_tolerance``, which is a **dimensionless ratio**: on a
+positive objective the run stops once the spread between the best and worst member falls
+to that fraction of the best member. Reading it that way is what keeps one value
+meaningful across fits whose objectives differ by many orders of magnitude. On an
+objective that is not positive a ratio means nothing, and reading one there is what made
+an all-negative likelihood population look converged and stop the run after generation 0
+before #561, so the same number is read there as an **absolute range** of the objective.
+
+Setting ``de_tolfun`` states the threshold as an absolute range in your objective's own
+units, on every objective. Use it when you know that scale and want to name the spread you
+will accept. Do not set it to a value borrowed from ``stop_tolerance``: the two carry
+different units, and a ratio's magnitude read as a range is far looser than intended on any
+fit whose objective is below 1 (see #648).
+
+In an island run, convergence is assessed only once every island has completed at least one
+iteration, so a single finished island cannot stop the whole search before the others have
+run.
 
 Applications
 ^^^^^^^^^^^^

docs/config_keys.rst

@@ -1934,7 +1934,9 @@ PyBNF offers two versions of :ref:`differential evoltution <alg-de>`: synchronou
     * ``mutation_factor = 0.7``
 
 **stop_tolerance**
-  Stop the run when the current population has converged to roughly the same objective function value, measured as an **absolute range** of the objective across the finite members of the population: the run ends once :math:`max\_objective - min\_objective \le e`, where *e* is the value of this key. Failed simulations (which score infinity) are ignored, so one dead candidate can neither trigger nor block the stop. This is a range in the units of your objective function; on a likelihood objective (a negative log-likelihood, which is unbounded below) set it to the smallest population spread you still consider unconverged. Prior to #561 this was a *ratio* test (:math:`max/min < 1 + e`), which stopped the run at generation 0 on any objective that can go negative; see ``de_tolfun`` to set the range independently of this key's magnitude.
+  Stop the run when the current population has converged to roughly the same objective function value. This key is a **dimensionless ratio**: where the objective is positive, the run ends once the spread between the best and worst finite member falls to :math:`e \times min\_objective` or below, which is the :math:`max/min \le 1 + e` this family has always used. The threshold therefore scales with the objective the population has reached, so the same value means the same thing on a fit whose objective lands at 1e-05 and one whose objective lands at 1000.
+
+  Where the objective is **not** positive a ratio means nothing, and reading one there is what stopped a likelihood fit at generation 0 before #561. On such an objective this key's value is read instead as an absolute range, :math:`max\_objective - min\_objective \le e`. Failed simulations (which score infinity) are ignored either way, so one dead candidate can neither trigger nor block the stop. To state the threshold as an absolute range on every objective, set ``de_tolfun`` instead.
 
   Default: 0.002
 
@@ -1944,7 +1946,9 @@ PyBNF offers two versions of :ref:`differential evoltution <alg-de>`: synchronou
 
 
 **de_tolfun**
-  The convergence tolerance the run actually uses: the absolute objective range (see ``stop_tolerance``) below which the population counts as converged. It is a range in the units of your objective function, whereas ``stop_tolerance`` was historically a dimensionless ratio; the two are separated so a fit can set a meaningful objective-range stop without reinterpreting the legacy key. Unset, it follows ``stop_tolerance``, so an existing config keeps the threshold magnitude it had.
+  The convergence tolerance the run actually uses: the absolute objective range (see ``stop_tolerance``) below which the population counts as converged. It is a range in the units of your objective function, whereas ``stop_tolerance`` is a dimensionless ratio. The two are separated so a fit can set a meaningful objective-range stop without reinterpreting the legacy key. Set it when you know your objective's scale and want the run to stop at a spread you can name.
+
+  Unset, the run falls back to ``stop_tolerance`` and reads it the way that key has always been read. Where the objective is positive that is a ratio, so the threshold scales with the objective the population has reached. Where the objective is not positive a ratio means nothing, so the same number is read as an absolute range. Setting this key opts out of that and makes the threshold an absolute range everywhere.
 
   Default: unset (follows ``stop_tolerance``)