From f176fcbbd5d2bee0b0e51c4686b6acd7ea332c68 Mon Sep 17 00:00:00 2001 From: Bill Hlavacek Date: Mon, 24 Aug 2026 08:54:08 -0600 Subject: [PATCH 1/2] Base the parallelism report on how many jobs a fit keeps running (#655) The startup parallelism report measured "how many jobs the fit runs at once" from the first batch of jobs the run loop submits. For most fits that is the right number. For scatter search it is not. The first batch is the initialization round, which is init_size parameter sets, ten per free parameter by default and unrelated to the population. Every round after it runs population_size x (population_size - 1) simulations. So a fit with seven free parameters and population_size = 20 on 384 reserved processors was told that 70 jobs would run, that 314 processors would sit idle, and that it should consider lowering population_size. It was about to run 380 simulations at a time. Lowering population_size would have reduced the number of processors it could use, which is the opposite of what the message was for. An algorithm can now say how many parameter sets it keeps out for evaluation once it is under way, through a new expected_parallelism method, and the report uses that number. The default returns None, meaning the first batch is already the right answer, so every fit that was correct before is untouched. Scatter search returns its population pairs, the same number it already prints as "simulations per iteration". When the first round differs from the steady state, the report says so rather than warning about it. The number is scaled by smoothing and parallelize_models, which turn one parameter set into several jobs. Profile likelihood had the same defect from the other end. Its first batch is a single preflight evaluation, so on any cluster it reported a fit that was almost entirely idle. It now reports its directional tracks, which is what it runs for nearly the whole fit. The advice also names the setting each fit actually reads. It named population_size for every fit. Powell and simplex read n_starts and never look at population_size, so they are now told about n_starts. Profile likelihood has no single setting for this, since its concurrency follows how many parameters it profiles, so it is advised only about how many processors to reserve. init_size was not documented in the cluster guide, and its small default is exactly what leaves a large allocation idle during the initialization round. It now has its own section there and a fuller entry in the configuration keys. While writing that, the table of how many simulations each fit type runs at once turned out to list profile likelihood as pl, which is not a valid job_type. It now reads profile_likelihood. Nine new tests cover the sustained count replacing the first batch, the warning quoting it, the jobs-per-parameter-set scaling, the setting each family names, scatter search reporting 380 rather than 70 on the reported cluster, and profile likelihood reporting its tracks rather than its preflight. --- CHANGELOG.md | 27 +++++ docs/cluster.rst | 15 ++- docs/config_keys.rst | 2 +- pybnf/algorithms/base.py | 113 ++++++++++++++---- .../optimizers/concurrent_multistart.py | 9 ++ .../optimizers/profile_likelihood.py | 24 ++++ pybnf/algorithms/optimizers/scatter_search.py | 14 +++ tests/test_profile_likelihood.py | 32 +++++ tests/test_run_loop.py | 85 ++++++++++++- tests/test_scatter.py | 27 +++++ 10 files changed, 314 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8197f4bde..66a1be82f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,33 @@ All notable changes to PyBNF are documented below. This project adheres to ## [Unreleased] +### Fixed +- **The startup parallelism report now describes how busy a fit will actually be, rather + than how busy its first round is (#655).** The report added in v1.8.0 measured "how many + jobs the fit runs at once" from the first batch of jobs submitted. For scatter search + that first batch is the initialization round, which is `init_size` parameter sets + (default ten per free parameter) and has nothing to do with the population. Every round + after it runs `population_size` x (`population_size` - 1) simulations. A fit with seven + free parameters and `population_size = 20` on 384 processors was told 314 processors + would sit idle and that it should lower `population_size`, while it was in fact about to + run 380 simulations at a time. Following that advice would have reduced the number of + processors the fit could use. + An algorithm can now state how many parameter sets it keeps out for evaluation once it + is under way, and the report uses that number. Scatter search reports its population + pairs, which is the same number it already prints as "simulations per iteration". + Profile likelihood reports its directional tracks, since its first batch is a single + preflight evaluation. When the first round differs from the steady state, the report + says so instead of warning about it. The count is scaled by `smoothing` and + `parallelize_models`, which turn one parameter set into several jobs. + The advice also names the setting each fit actually reads. Powell and simplex are told + about `n_starts` rather than `population_size`, which they do not use, and profile + likelihood, whose concurrency follows how many parameters it profiles, is advised only + about how many processors to reserve. + `init_size` is now documented in the cluster guide, with a note that raising it fills a + large allocation during scatter search's initialization round. The table of how many + simulations each fit type runs at once listed profile likelihood as `pl`, which is not a + valid `job_type`. It now reads `profile_likelihood`. + ## [v1.8.1] - 2026-08-23 ### Fixed diff --git a/docs/cluster.rst b/docs/cluster.rst index 84468228d..aba3b9d37 100644 --- a/docs/cluster.rst +++ b/docs/cluster.rst @@ -161,15 +161,15 @@ How many simulations each algorithm runs at once * - ``de``, ``ade``, ``pso``, ``cmaes``, ``dream``, ``p_dream``, ``mh``, ``am``, ``sa``, ``pt`` - ``population_size``. For ``pt`` that is the replicas at all temperatures together; for the Markov chain samplers it is the number of independent chains. * - ``ss`` - - ``population_size`` x (``population_size`` - 1), since every parent-helper pair is a simulation. A reference set of 9 fills 72 processors. + - ``population_size`` x (``population_size`` - 1), since every parent-helper pair is a simulation. A reference set of 9 fills 72 processors. The first round is different; see `The scatter search initialization round`_. * - ``sim`` - min(``population_size``, N - 1) per start, never fewer than 1, where N is the number of free parameters; times ``n_starts`` concurrent starts. * - ``powell`` - One per start, so ``n_starts``: each line search is serial by construction. * - ``trf``, ``lbfgs``, ``gntr`` - ``population_size``, which the gradient optimizers use as their number of concurrent starts. - * - ``pl`` - - Two directional walks per profiled parameter (one per direction), capped by ``profile_likelihood_max_parallel`` -- ``0``, the default, runs all of them at once. + * - ``profile_likelihood`` + - Two directional walks per profiled parameter (one per direction), capped by ``profile_likelihood_max_parallel`` -- ``0``, the default, runs all of them at once. The fit opens with a single evaluation and then a short multi-start polish, both much smaller than this. * - ``hmc`` - None. Its chains are an in-process numeric loop rather than dask jobs (and it runs only on analytical models), so extra nodes do not help. @@ -177,6 +177,15 @@ Multiply any of these by ``smoothing`` and by ``parallelize_models``, if you set The processors an algorithm cannot use are worth reserving only for the memory attached to them. Each worker is a separate process holding its own copy of the models, so a memory-hungry model may need ``parallel_count`` set below the CPU count rather than a bigger population. +The scatter search initialization round +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Scatter search begins by scoring a set of random parameter sets and picking the reference set out of it. That first round runs ``init_size`` simulations, which defaults to ten per free parameter and is unrelated to ``population_size``. Every round after it runs ``population_size`` x (``population_size`` - 1) simulations, and that is the number to size an allocation by. + +The default ``init_size`` is often much smaller than the steady state. Seven free parameters and ``population_size = 20`` gives 70 simulations in the first round and 380 in every round after, so a 384-processor allocation is mostly idle until the first round finishes. Raise ``init_size`` to fill the allocation during it. Nothing is lost by doing so: a larger initial set means a wider search for the starting reference set. ``init_size`` is never allowed below ``population_size``. + +At the start of a cluster fit PyBNF logs how many simulations the fit will run at once against how many workers connected, and warns when the two are far apart. That report uses the steady-state number, so for scatter search it describes the rounds after the first one. + A worked example ^^^^^^^^^^^^^^^^ diff --git a/docs/config_keys.rst b/docs/config_keys.rst index 098fe3cf7..20b8811c3 100644 --- a/docs/config_keys.rst +++ b/docs/config_keys.rst @@ -2020,7 +2020,7 @@ The following options are only available with ``job_type = de``, and serve to ma ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **init_size** - Number of parameter sets to test to generate the initial population. + Number of parameter sets to test to generate the initial population. This first round is the only one of that size: every round after it runs ``population_size`` * (``population_size`` - 1) simulations. On a cluster the default is often far smaller than that, leaving most of the allocation idle until the initial round finishes, so raising it is worthwhile there. It is never allowed below ``population_size``. See :ref:`Running on a cluster `. Default: 10 * number of parameters diff --git a/pybnf/algorithms/base.py b/pybnf/algorithms/base.py index a49bc3135..90003e5c3 100644 --- a/pybnf/algorithms/base.py +++ b/pybnf/algorithms/base.py @@ -114,6 +114,14 @@ class Algorithm(ABC): # base-class flag pattern as _is_simplex, so run() never names a leaf subclass. waits_for_full_generation = False + # Overridable: the name of the setting the parallelism report points at when it + # advises the user (#655). Most fits size their concurrency from population_size, so + # that is the default. A fit that follows a different setting names it instead (the + # local multi-start optimizers name their own n_starts key), and a fit where no + # single setting controls it sets this to None, which leaves the advice talking only + # about how many processors to reserve. + parallelism_setting = 'population_size' + #: The fit's total wall-clock budget (``wall_time_fit``, #529/ADR-0093), or None for #: an unbounded fit. Set by ``pybnf.main()`` on the algorithm it is about to run -- #: and passed on to a refiner / reused across bootstrap replicates -- so one deadline @@ -1295,18 +1303,39 @@ def run(self, client, resume=None, debug=False): # call is exactly the historical one. pool_kwargs = {'timeout': self.budget.remaining()} if self.budget is not None else {} pool = core.as_completed(futures, with_results=True, raise_errors=False, **pool_kwargs) - self._report_parallelism(client, len(futures)) + self._report_parallelism(client, len(futures), len(psets)) self.completed_simulations = self._drain_job_pool(client, pool, pending, backup_every, debug) logger.info("Cancelling %d pending jobs" % len(pending)) client.cancel(list(pending.keys())) self._finalize_run() - def _report_parallelism(self, client, jobs_in_flight): - """Log how many jobs the fit starts with against how many workers connected, and + def expected_parallelism(self): + """How many parameter sets this fit has out for evaluation at once once it is under + way, or None when that is however many the run loop submits in its first batch + (#655). + + The run loop knows only the size of the first batch of jobs it submits. For most + fits that is also how many run at once for the rest of the fit, so the default + here is None and :meth:`_report_parallelism` uses the first batch. A fit whose + first batch is a one-time initialization round of a different size overrides this + and returns the number it settles at, so the parallelism report describes the fit + rather than its opening move. Scatter search is the case that prompted this: it + starts with ``init_size`` parameter sets and then runs + ``population_size * (population_size - 1)`` simulations per iteration forever + after. + + Count parameter sets, not jobs. One parameter set is several jobs under + ``smoothing`` or ``parallelize_models``, and :meth:`_report_parallelism` applies + that multiplier itself. + """ + return None + + def _report_parallelism(self, client, jobs_in_flight, psets_in_flight=None): + """Log how many jobs the fit runs at once against how many workers connected, and warn when the two differ by a large margin (#621). - How many simulations a fit runs at once follows its settings, mainly + How many simulations a fit runs at once follows its settings, usually population_size, not how many processors were reserved. When many more workers connect than there are jobs to run, the extra workers sit idle for the whole run and nothing else would say so, so a user can reserve several machines and quietly @@ -1314,6 +1343,14 @@ def _report_parallelism(self, client, jobs_in_flight): queues up and the larger population buys no extra speed. Either way both numbers go in the log so a finished run can be looked at afterwards. + The number reported is the one the fit sustains, from :meth:`expected_parallelism`, + which is not always the size of the first batch of jobs the run loop submits + (#655). When the two differ, the first batch is a one-time initialization round, so + it is reported as such and the warnings are decided on the sustained number. + ``psets_in_flight`` is how many parameter sets that first batch of jobs came from, + which is how a parameter-set count is converted into a job count; None means one + job per parameter set. + Only cluster runs are reported. A local run's worker count is exactly what the user asked for through parallel_count, so there is nothing to compare it against. The worker count comes from dask; anything that goes wrong reading it is logged and @@ -1332,38 +1369,64 @@ def _report_parallelism(self, client, jobs_in_flight): if n_workers <= 0: return - logger.info('Parallelism: the fit starts with %d job(s) running and %d worker(s) ' - 'connected.' % (jobs_in_flight, n_workers)) - + n_jobs = self.expected_parallelism() + if n_jobs is None or n_jobs <= 0: + n_jobs = jobs_in_flight + elif psets_in_flight: + # expected_parallelism() counts parameter sets, but one parameter set is more + # than one job when smoothing runs replicates of it or parallelize_models + # splits it across models. The first batch shows how many jobs a parameter set + # becomes, and that ratio holds for the rest of the fit. + n_jobs = int(round(n_jobs * jobs_in_flight / psets_in_flight)) + + logger.info('Parallelism: the fit runs %d job(s) at a time and %d worker(s) are ' + 'connected.' % (n_jobs, n_workers)) + + notes = [] + # A first batch of a different size is a one-time initialization round. Say so, so + # that a user watching the start of the fit is not surprised by it and does not + # read it as how busy the fit will be. + if jobs_in_flight != n_jobs: + notes.append('This fit begins with a one-time round of %d job(s) before it ' + 'settles at %d.' % (jobs_in_flight, n_jobs)) # A generational fit drains each generation to almost nothing before starting the # next, so some idle time is expected with one and should not be read as a fault. - note = '' if self.waits_for_full_generation: - note = (' This fit runs one generation at a time and waits for all of it to ' - 'finish before starting the next, so some idle time toward the end of ' - 'each generation is expected.') + notes.append('This fit runs one generation at a time and waits for all of it ' + 'to finish before starting the next, so some idle time toward ' + 'the end of each generation is expected.') + note = (' ' + ' '.join(notes)) if notes else '' + + # Name the setting that sizes the concurrency, when one setting does. A fit where + # none does (profile likelihood, whose concurrency follows how many parameters it + # profiles) leaves the advice to talk about processors only. + setting = self.parallelism_setting + by_setting = ('the fitting settings, mainly %s,' % setting) if setting \ + else 'the fitting settings,' + raise_setting = ('raising %s or ' % setting) if setting else '' + lower_setting = ('lowering %s or ' % setting) if setting else '' # A factor of two in either direction is the "large margin" that draws a warning. - if n_workers >= 2 * jobs_in_flight: - msg = ('The fit starts with only %d job(s) running but %d worker(s) connected, ' + if n_workers >= 2 * n_jobs: + msg = ('The fit runs only %d job(s) at a time but %d worker(s) are connected, ' 'so about %d worker(s) will sit idle. How many jobs run at once is set ' - 'by the fitting settings, mainly population_size, not by how many ' - 'processors were reserved. Consider raising population_size or reserving ' - 'fewer processors.%s' - % (jobs_in_flight, n_workers, n_workers - jobs_in_flight, note)) + 'by %s not by how many processors were reserved. ' + 'Consider %sreserving fewer processors.%s' + % (n_jobs, n_workers, n_workers - n_jobs, by_setting, + raise_setting, note)) logger.warning(msg) print1('Warning: ' + msg) - elif jobs_in_flight >= 2 * n_workers: - msg = ('The fit starts with %d job(s) running but only %d worker(s) connected, ' - 'so jobs will queue and the extra jobs buy no extra speed. How many jobs ' - 'run at once is set by the fitting settings, mainly population_size. ' - 'Consider lowering population_size or reserving more processors.%s' - % (jobs_in_flight, n_workers, note)) + elif n_jobs >= 2 * n_workers: + msg = ('The fit runs %d job(s) at a time but only %d worker(s) are connected, ' + 'so jobs will queue and the extra jobs buy no extra speed. How many ' + 'jobs run at once is set by %s not by how many processors were ' + 'reserved. Consider %sreserving more processors.%s' + % (n_jobs, n_workers, by_setting, lower_setting, note)) logger.warning(msg) print1('Warning: ' + msg) elif note: - # The counts are close, but a generational fit still idles toward the end of - # each generation, so put that on the record for this run. + # The counts are close, but there is still something worth putting on the + # record for this run. logger.info(note.strip()) def _finalize_run(self): diff --git a/pybnf/algorithms/optimizers/concurrent_multistart.py b/pybnf/algorithms/optimizers/concurrent_multistart.py index a69a678dd..fc7ad7f97 100644 --- a/pybnf/algorithms/optimizers/concurrent_multistart.py +++ b/pybnf/algorithms/optimizers/concurrent_multistart.py @@ -112,6 +112,15 @@ class ConcurrentMultiStartOptimizer(StartPointOptimizer): #: local path, ``'stopping'`` for the gradient path) -- cosmetic, preserved verbatim. _stop_verb = 'finished' + @property + def parallelism_setting(self): + """How many jobs one of these fits runs at once is its number of starts, since each + start holds one evaluation in flight, so the parallelism report names the key the + start count is read from rather than the base class's population_size (#655). That + is ``n_starts`` on the local path and ``population_size`` on the gradient path, + which predates the newer key.""" + return self._n_starts_key + def __init__(self, config, refine=False): # A subclass gate that must run *before* the (expensive) network generation in # Algorithm.__init__ -- the gradient path refuses a legacy-edition config here, so diff --git a/pybnf/algorithms/optimizers/profile_likelihood.py b/pybnf/algorithms/optimizers/profile_likelihood.py index 82e12b58d..ba8b10535 100644 --- a/pybnf/algorithms/optimizers/profile_likelihood.py +++ b/pybnf/algorithms/optimizers/profile_likelihood.py @@ -641,6 +641,11 @@ class ProfileLikelihoodAlgorithm(GradientOptimizer): fit_type = 'profile_likelihood' _method_label = 'profile-likelihood polish' + #: No single setting sets how many jobs this fit runs at once, so the parallelism + #: report names none and advises only about processors (#655). See + #: :meth:`expected_parallelism`. + parallelism_setting = None + def __init__(self, config, refine=False): super().__init__(config, refine=refine) self.confidence = config.config['profile_likelihood_confidence'] @@ -696,6 +701,25 @@ def _init_profile_state(self): self._track_queue = [] # remaining (param_idx, direction) tracks not yet launched self._active_tracks = {} # in-flight PSet name -> (param_idx, _ProfileTrack) + def expected_parallelism(self): + """Profiling runs one job per directional track, two tracks per profiled + parameter, up to whatever cap :meth:`_effective_parallel` applies. That is the + concurrency this fit spends nearly all of its time at, so it is what the + parallelism report should describe (#655). + + The run loop's first batch is one job: the preflight evaluation of the box center, + which picks the inner optimizer. The two short phases that follow (the multi-start + polish to the optimum, then profiling) each run more than that. Reporting the + preflight would say a large allocation is almost entirely idle, which is true for + one evaluation and wrong for the rest of the fit. + + No single setting sets this number: it follows how many parameters are profiled + (``profile_likelihood_params``, all of them by default), and + ``profile_likelihood_max_parallel`` can only lower it. Hence + :attr:`parallelism_setting` is None for this fit. + """ + return self._effective_parallel() + def _start_banner(self): return ("Running profile-likelihood analysis at the %g confidence level " "(Delta chi2 = %g, 1 dof) for %i parameter(s)" diff --git a/pybnf/algorithms/optimizers/scatter_search.py b/pybnf/algorithms/optimizers/scatter_search.py index ae3120c3d..faac15d2e 100644 --- a/pybnf/algorithms/optimizers/scatter_search.py +++ b/pybnf/algorithms/optimizers/scatter_search.py @@ -92,6 +92,20 @@ def __init__(self, config): # variables, popsize, maxiters, saveevery): self.local_mins = [] # (Pset, score) pairs that were stuck for 5 gens, and so replaced. self.reserve = [] + def expected_parallelism(self): + """Scatter search runs up to ``population_size * (population_size - 1)`` + simulations at a time, one for every ordered pair in the reference set, and keeps + doing that for the rest of the fit (#655). + + The first batch of jobs the run loop submits is a different number: it is the + ``init_size`` random parameter sets the initialization round scores, which by + default is ten per free parameter and has nothing to do with the population. On a + cluster large enough to matter the two are far apart, so the parallelism report + has to be told which one describes the fit. This is the same number + ``_search_start_run`` already prints as "simulations per iteration". + """ + return self.popsize * (self.popsize - 1) + def reset(self, bootstrap=None): super().reset(bootstrap) self._reset_search_state() diff --git a/tests/test_profile_likelihood.py b/tests/test_profile_likelihood.py index 4579b1dd8..7225c9e87 100644 --- a/tests/test_profile_likelihood.py +++ b/tests/test_profile_likelihood.py @@ -17,6 +17,7 @@ ``theta`` and the u-space Jacobian is ``A`` itself. """ +import logging import os from pathlib import Path @@ -695,6 +696,37 @@ def test_parallel_orchestration_runs_tracks_concurrently(tmp_path): assert (tmp_path / 'profile_p0.txt').is_file() and (tmp_path / 'profile_p1.txt').is_file() +def test_parallelism_report_describes_profiling_not_the_preflight(tmp_path): + """The run loop's first batch for this fit is the single preflight evaluation, but the + fit then spends nearly all its time running one job per directional track. The + parallelism report is given the track count, and since no one setting controls it the + advice names no setting and talks only about processors (#655).""" + A, y, theta_star, f_min, C, names, lower, upper = _lin_model_2d() + alg = _OfflineProfileAlg(A, y, theta_star, f_min, lower, upper, names, str(tmp_path)) + assert alg.expected_parallelism() == 4 # 2 parameters x 2 directions + assert alg.parallelism_setting is None + + class _ClusterClient: + cluster = None + + def scheduler_info(self): + return {'workers': {'tcp://w%d' % i: {} for i in range(40)}} + + records = [] + handler = logging.Handler() + handler.emit = records.append + log = logging.getLogger('pybnf.algorithms') + log.addHandler(handler) + try: + alg._report_parallelism(_ClusterClient(), 1) # 1 = the preflight batch + finally: + log.removeHandler(handler) + text = '\n'.join(r.getMessage() for r in records) + assert 'runs only 4 job(s) at a time but 40 worker(s)' in text + assert 'population_size' not in text + assert 'Consider reserving fewer processors' in text + + def test_max_parallel_cap_serializes_without_truncating_coverage(tmp_path): """A cap only limits concurrency -- the excess tracks queue and run as slots free, never dropped -- so a fully serial (cap = 1) run visits every track and yields byte-identical diff --git a/tests/test_run_loop.py b/tests/test_run_loop.py index 6c93a359d..7055742ec 100644 --- a/tests/test_run_loop.py +++ b/tests/test_run_loop.py @@ -373,14 +373,18 @@ def _scored(name, score): class TestReportParallelism: - """``_report_parallelism`` compares how many jobs a fit starts with against how many + """``_report_parallelism`` compares how many jobs a fit runs at once against how many workers connected, logs both numbers on a cluster run, and warns when they differ by a - large margin (#621). Driven directly with the fake client, which models a local run by + large margin (#621). The number of jobs is what the fit sustains -- ``expected_parallelism`` + when the algorithm defines one, otherwise the size of the first batch the run loop + submitted (#655). Driven directly with the fake client, which models a local run by default and a cluster run when given a worker count.""" - def _algo(self, tmp_path, generational=False): + def _algo(self, tmp_path, generational=False, sustained=None, setting='population_size'): algo = _make_algorithm(tmp_path, [[_pset('a', 1.0)]]) algo.waits_for_full_generation = generational + algo.expected_parallelism = lambda: sustained + algo.parallelism_setting = setting return algo def test_local_run_is_not_reported(self, tmp_path, caplog): @@ -396,7 +400,7 @@ def test_cluster_run_logs_both_numbers(self, tmp_path, caplog): algo = self._algo(tmp_path) with caplog.at_level(logging.INFO, logger='pybnf.algorithms'): algo._report_parallelism(_FakeClient(n_workers=4), 4) - assert 'starts with 4 job(s) running and 4 worker(s) connected' in caplog.text + assert 'runs 4 job(s) at a time and 4 worker(s) are connected' in caplog.text assert not [r for r in caplog.records if r.levelno >= logging.WARNING] def test_idle_workers_warn_and_name_population_size(self, tmp_path, caplog, capsys): @@ -441,6 +445,58 @@ def test_zero_workers_is_not_reported(self, tmp_path, caplog): algo._report_parallelism(_FakeClient(n_workers=0), 4) assert 'Parallelism' not in caplog.text + def test_sustained_count_replaces_the_first_batch(self, tmp_path, caplog): + """A fit whose first batch is a one-time initialization round is judged on the + number it settles at, not on that first batch (#655). Here 20 jobs against 24 + workers is a good match, even though the fit opened with 3 jobs.""" + algo = self._algo(tmp_path, sustained=20) + with caplog.at_level(logging.INFO, logger='pybnf.algorithms'): + algo._report_parallelism(_FakeClient(n_workers=24), 3) + assert 'runs 20 job(s) at a time and 24 worker(s) are connected' in caplog.text + assert 'one-time round of 3 job(s) before it settles at 20' in caplog.text + assert not [r for r in caplog.records if r.levelno >= logging.WARNING] + + def test_sustained_count_is_used_in_the_warning(self, tmp_path, caplog): + """When the sustained number really is too small for the allocation, the warning + quotes it rather than the first batch.""" + algo = self._algo(tmp_path, sustained=6) + with caplog.at_level(logging.WARNING, logger='pybnf.algorithms'): + algo._report_parallelism(_FakeClient(n_workers=40), 100) + assert 'runs only 6 job(s) at a time but 40 worker(s)' in caplog.text + assert 'about 34 worker(s) will sit idle' in caplog.text + assert 'one-time round of 100 job(s) before it settles at 6' in caplog.text + + def test_sustained_count_scales_by_jobs_per_pset(self, tmp_path, caplog): + """expected_parallelism counts parameter sets, and one parameter set is several + jobs under smoothing or parallelize_models. The first batch shows the ratio (here + 12 jobs from 4 parameter sets, so 3 jobs each), so a sustained 10 parameter sets is + 30 jobs (#655).""" + algo = self._algo(tmp_path, sustained=10) + with caplog.at_level(logging.INFO, logger='pybnf.algorithms'): + algo._report_parallelism(_FakeClient(n_workers=32), 12, 4) + assert 'runs 30 job(s) at a time and 32 worker(s) are connected' in caplog.text + assert 'one-time round of 12 job(s) before it settles at 30' in caplog.text + + def test_named_setting_is_the_one_advised(self, tmp_path, caplog): + """A fit whose concurrency follows a setting other than population_size names that + setting instead (#655).""" + algo = self._algo(tmp_path, setting='n_starts') + with caplog.at_level(logging.WARNING, logger='pybnf.algorithms'): + algo._report_parallelism(_FakeClient(n_workers=8), 2) + assert 'mainly n_starts' in caplog.text + assert 'Consider raising n_starts or reserving fewer processors' in caplog.text + assert 'population_size' not in caplog.text + + def test_no_named_setting_advises_only_about_processors(self, tmp_path, caplog): + """A fit where no single setting sets the concurrency (profile likelihood) names + none, and the advice talks only about how many processors to reserve (#655).""" + algo = self._algo(tmp_path, setting=None) + with caplog.at_level(logging.WARNING, logger='pybnf.algorithms'): + algo._report_parallelism(_FakeClient(n_workers=8), 2) + assert 'set by the fitting settings, not by how many processors' in caplog.text + assert 'Consider reserving fewer processors' in caplog.text + assert 'population_size' not in caplog.text + def test_unreadable_worker_count_is_not_fatal(self, tmp_path, caplog): """If reading the worker count from dask fails, the fit is not stopped; the failure is logged and the report is skipped.""" @@ -456,6 +512,25 @@ def scheduler_info(self): assert 'parallelism report is skipped' in caplog.text +def test_local_and_gradient_multistarts_name_their_own_start_setting(): + """The local multi-start optimizers run one job per start, so the parallelism report + advises about the key each of them reads its start count from rather than the base + class's population_size (#655). Powell and simplex read n_starts; the gradient + optimizers predate that key and read population_size.""" + from pybnf.algorithms.optimizers.powell import PowellAlgorithm + from pybnf.algorithms.optimizers.simplex import SimplexAlgorithm + from pybnf.algorithms.optimizers.trf import TRFAlgorithm + from pybnf.algorithms.optimizers.profile_likelihood import ProfileLikelihoodAlgorithm + + # Constructed without __init__: parallelism_setting reads only class attributes, and + # a real construction would build models. + assert object.__new__(PowellAlgorithm).parallelism_setting == 'n_starts' + assert object.__new__(SimplexAlgorithm).parallelism_setting == 'n_starts' + assert object.__new__(TRFAlgorithm).parallelism_setting == 'population_size' + # Profile likelihood inherits from the gradient path but has no single setting. + assert object.__new__(ProfileLikelihoodAlgorithm).parallelism_setting is None + + def test_cluster_run_reports_parallelism_end_to_end(tmp_path, monkeypatch, caplog): """run() calls the parallelism report after submitting the initial jobs: a cluster client with more workers than the one initial job draws the idle-workers warning.""" @@ -464,7 +539,7 @@ def test_cluster_run_reports_parallelism_end_to_end(tmp_path, monkeypatch, caplo algo = _make_algorithm(tmp_path, gens) with caplog.at_level(logging.INFO, logger='pybnf.algorithms'): algo.run(_FakeClient(n_workers=6)) - assert 'starts with 1 job(s) running and 6 worker(s) connected' in caplog.text + assert 'runs 1 job(s) at a time and 6 worker(s) are connected' in caplog.text assert 'will sit idle' in caplog.text diff --git a/tests/test_scatter.py b/tests/test_scatter.py index d0834dd17..6d6439a66 100644 --- a/tests/test_scatter.py +++ b/tests/test_scatter.py @@ -106,6 +106,7 @@ def test_exp10_overflow(self): # The combination's only randomness is add_rand's uniform draw; freezing it to # the midpoint reduces a candidate to the closed form pi - alpha*beta*d. # --------------------------------------------------------------------------- # +import logging import numpy as np import numpy.testing as npt import re as _re @@ -150,6 +151,32 @@ def test_default_init_size_floored_to_population(self, tmp_path): ss = algorithms.ScatterSearch(_ss_config(tmp_path, population_size=40)) assert ss.init_size == 40 + def test_expected_parallelism_is_the_steady_state_not_the_first_round(self, tmp_path): + """Oracle (#655): scatter search runs population_size * (population_size - 1) + simulations per iteration, so that is what it reports as its parallelism, even + though the first batch it submits is the init_size initialization round.""" + ss = algorithms.ScatterSearch(_ss_config(tmp_path, population_size=5, init_size=40)) + assert len(ss.start_run()) == 40 # the one-time initialization round + assert ss.expected_parallelism() == 5 * 4 # what every iteration after it runs + + def test_parallelism_report_uses_the_steady_state(self, tmp_path, caplog): + """Oracle (#655): with more workers than the initialization round but a good match + for the steady state, the report says so and does not warn the user to lower + population_size.""" + ss = algorithms.ScatterSearch(_ss_config(tmp_path, population_size=20, init_size=70)) + + class _ClusterClient: + cluster = None + + def scheduler_info(self): + return {'workers': {'tcp://w%d' % i: {} for i in range(384)}} + + with caplog.at_level(logging.INFO, logger='pybnf.algorithms'): + ss._report_parallelism(_ClusterClient(), 70) + assert 'runs 380 job(s) at a time and 384 worker(s) are connected' in caplog.text + assert 'one-time round of 70 job(s) before it settles at 380' in caplog.text + assert not [r for r in caplog.records if r.levelno >= logging.WARNING] + def test_explicit_reserve_size_used(self, tmp_path): """Oracle (reserve_size config): an explicit reserve_size overrides the default (which is max_iterations).""" From 8f795f53047100c07bac59e632ef4237f6ee74f4 Mon Sep 17 00:00:00 2001 From: Bill Hlavacek Date: Mon, 24 Aug 2026 08:58:32 -0600 Subject: [PATCH 2/2] Fix the short title underline that failed the docs build Sphinx runs with -W, so a section title whose underline is one character short of the title fails the build. --- docs/cluster.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cluster.rst b/docs/cluster.rst index aba3b9d37..391e76748 100644 --- a/docs/cluster.rst +++ b/docs/cluster.rst @@ -178,7 +178,7 @@ Multiply any of these by ``smoothing`` and by ``parallelize_models``, if you set The processors an algorithm cannot use are worth reserving only for the memory attached to them. Each worker is a separate process holding its own copy of the models, so a memory-hungry model may need ``parallel_count`` set below the CPU count rather than a bigger population. The scatter search initialization round -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Scatter search begins by scoring a set of random parameter sets and picking the reference set out of it. That first round runs ``init_size`` simulations, which defaults to ten per free parameter and is unrelated to ``population_size``. Every round after it runs ``population_size`` x (``population_size`` - 1) simulations, and that is the number to size an allocation by.