Base the parallelism report on how many jobs a fit keeps running (#655)

CHANGELOG.md

@@ -5,6 +5,33 @@ All notable changes to PyBNF are documented below. This project adheres to
 
 ## [Unreleased]
 
+### Fixed
+- **The startup parallelism report now describes how busy a fit will actually be, rather
+  than how busy its first round is (#655).** The report added in v1.8.0 measured "how many
+  jobs the fit runs at once" from the first batch of jobs submitted. For scatter search
+  that first batch is the initialization round, which is `init_size` parameter sets
+  (default ten per free parameter) and has nothing to do with the population. Every round
+  after it runs `population_size` x (`population_size` - 1) simulations. A fit with seven
+  free parameters and `population_size = 20` on 384 processors was told 314 processors
+  would sit idle and that it should lower `population_size`, while it was in fact about to
+  run 380 simulations at a time. Following that advice would have reduced the number of
+  processors the fit could use.
+  An algorithm can now state how many parameter sets it keeps out for evaluation once it
+  is under way, and the report uses that number. Scatter search reports its population
+  pairs, which is the same number it already prints as "simulations per iteration".
+  Profile likelihood reports its directional tracks, since its first batch is a single
+  preflight evaluation. When the first round differs from the steady state, the report
+  says so instead of warning about it. The count is scaled by `smoothing` and
+  `parallelize_models`, which turn one parameter set into several jobs.
+  The advice also names the setting each fit actually reads. Powell and simplex are told
+  about `n_starts` rather than `population_size`, which they do not use, and profile
+  likelihood, whose concurrency follows how many parameters it profiles, is advised only
+  about how many processors to reserve.
+  `init_size` is now documented in the cluster guide, with a note that raising it fills a
+  large allocation during scatter search's initialization round. The table of how many
+  simulations each fit type runs at once listed profile likelihood as `pl`, which is not a
+  valid `job_type`. It now reads `profile_likelihood`.
+
 ## [v1.8.1] - 2026-08-23
 
 ### Fixed

docs/cluster.rst

@@ -161,22 +161,31 @@ How many simulations each algorithm runs at once
    * - ``de``, ``ade``, ``pso``, ``cmaes``, ``dream``, ``p_dream``, ``mh``, ``am``, ``sa``, ``pt``
      - ``population_size``. For ``pt`` that is the replicas at all temperatures together; for the Markov chain samplers it is the number of independent chains.
    * - ``ss``
-     - ``population_size`` x (``population_size`` - 1), since every parent-helper pair is a simulation. A reference set of 9 fills 72 processors.
+     - ``population_size`` x (``population_size`` - 1), since every parent-helper pair is a simulation. A reference set of 9 fills 72 processors. The first round is different; see `The scatter search initialization round`_.
    * - ``sim``
      - min(``population_size``, N - 1) per start, never fewer than 1, where N is the number of free parameters; times ``n_starts`` concurrent starts.
    * - ``powell``
      - One per start, so ``n_starts``: each line search is serial by construction.
    * - ``trf``, ``lbfgs``, ``gntr``
      - ``population_size``, which the gradient optimizers use as their number of concurrent starts.
-   * - ``pl``
-     - Two directional walks per profiled parameter (one per direction), capped by ``profile_likelihood_max_parallel`` -- ``0``, the default, runs all of them at once.
+   * - ``profile_likelihood``
+     - Two directional walks per profiled parameter (one per direction), capped by ``profile_likelihood_max_parallel`` -- ``0``, the default, runs all of them at once. The fit opens with a single evaluation and then a short multi-start polish, both much smaller than this.
    * - ``hmc``
      - None. Its chains are an in-process numeric loop rather than dask jobs (and it runs only on analytical models), so extra nodes do not help.
 
 Multiply any of these by ``smoothing`` and by ``parallelize_models``, if you set them: every replicate and every model partition is a separate job. ``n_starts`` adds nothing for the metaheuristics (``de``, ``ade``, ``ss``, ``pso``), whose starts run one after another rather than at the same time.
 
 The processors an algorithm cannot use are worth reserving only for the memory attached to them. Each worker is a separate process holding its own copy of the models, so a memory-hungry model may need ``parallel_count`` set below the CPU count rather than a bigger population.
 
+The scatter search initialization round
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Scatter search begins by scoring a set of random parameter sets and picking the reference set out of it. That first round runs ``init_size`` simulations, which defaults to ten per free parameter and is unrelated to ``population_size``. Every round after it runs ``population_size`` x (``population_size`` - 1) simulations, and that is the number to size an allocation by.
+
+The default ``init_size`` is often much smaller than the steady state. Seven free parameters and ``population_size = 20`` gives 70 simulations in the first round and 380 in every round after, so a 384-processor allocation is mostly idle until the first round finishes. Raise ``init_size`` to fill the allocation during it. Nothing is lost by doing so: a larger initial set means a wider search for the starting reference set. ``init_size`` is never allowed below ``population_size``.
+
+At the start of a cluster fit PyBNF logs how many simulations the fit will run at once against how many workers connected, and warns when the two are far apart. That report uses the steady-state number, so for scatter search it describes the rounds after the first one.
+
 A worked example
 ^^^^^^^^^^^^^^^^
 

docs/config_keys.rst

@@ -2020,7 +2020,7 @@ The following options are only available with ``job_type = de``, and serve to ma
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 **init_size**
-  Number of parameter sets to test to generate the initial population. 
+  Number of parameter sets to test to generate the initial population. This first round is the only one of that size: every round after it runs ``population_size`` * (``population_size`` - 1) simulations. On a cluster the default is often far smaller than that, leaving most of the allocation idle until the initial round finishes, so raising it is worthwhile there. It is never allowed below ``population_size``. See :ref:`Running on a cluster <cluster>`. 
 
   Default: 10 * number of parameters
 

pybnf/algorithms/base.py

@@ -114,6 +114,14 @@ class Algorithm(ABC):
     # base-class flag pattern as _is_simplex, so run() never names a leaf subclass.
     waits_for_full_generation = False
 
+    # Overridable: the name of the setting the parallelism report points at when it
+    # advises the user (#655). Most fits size their concurrency from population_size, so
+    # that is the default. A fit that follows a different setting names it instead (the
+    # local multi-start optimizers name their own n_starts key), and a fit where no
+    # single setting controls it sets this to None, which leaves the advice talking only
+    # about how many processors to reserve.
+    parallelism_setting = 'population_size'
+
     #: The fit's total wall-clock budget (``wall_time_fit``, #529/ADR-0093), or None for
     #: an unbounded fit. Set by ``pybnf.main()`` on the algorithm it is about to run --
     #: and passed on to a refiner / reused across bootstrap replicates -- so one deadline
@@ -1295,25 +1303,54 @@ def run(self, client, resume=None, debug=False):
             # call is exactly the historical one.
             pool_kwargs = {'timeout': self.budget.remaining()} if self.budget is not None else {}
             pool = core.as_completed(futures, with_results=True, raise_errors=False, **pool_kwargs)
-            self._report_parallelism(client, len(futures))
+            self._report_parallelism(client, len(futures), len(psets))
             self.completed_simulations = self._drain_job_pool(client, pool, pending, backup_every, debug)
 
         logger.info("Cancelling %d pending jobs" % len(pending))
         client.cancel(list(pending.keys()))
         self._finalize_run()
 
-    def _report_parallelism(self, client, jobs_in_flight):
-        """Log how many jobs the fit starts with against how many workers connected, and
+    def expected_parallelism(self):
+        """How many parameter sets this fit has out for evaluation at once once it is under
+        way, or None when that is however many the run loop submits in its first batch
+        (#655).
+
+        The run loop knows only the size of the first batch of jobs it submits. For most
+        fits that is also how many run at once for the rest of the fit, so the default
+        here is None and :meth:`_report_parallelism` uses the first batch. A fit whose
+        first batch is a one-time initialization round of a different size overrides this
+        and returns the number it settles at, so the parallelism report describes the fit
+        rather than its opening move. Scatter search is the case that prompted this: it
+        starts with ``init_size`` parameter sets and then runs
+        ``population_size * (population_size - 1)`` simulations per iteration forever
+        after.
+
+        Count parameter sets, not jobs. One parameter set is several jobs under
+        ``smoothing`` or ``parallelize_models``, and :meth:`_report_parallelism` applies
+        that multiplier itself.
+        """
+        return None
+
+    def _report_parallelism(self, client, jobs_in_flight, psets_in_flight=None):
+        """Log how many jobs the fit runs at once against how many workers connected, and
         warn when the two differ by a large margin (#621).
 
-        How many simulations a fit runs at once follows its settings, mainly
+        How many simulations a fit runs at once follows its settings, usually
         population_size, not how many processors were reserved. When many more workers
         connect than there are jobs to run, the extra workers sit idle for the whole run
         and nothing else would say so, so a user can reserve several machines and quietly
         use a fraction of them. The opposite, many more jobs than workers, means work
         queues up and the larger population buys no extra speed. Either way both numbers go
         in the log so a finished run can be looked at afterwards.
 
+        The number reported is the one the fit sustains, from :meth:`expected_parallelism`,
+        which is not always the size of the first batch of jobs the run loop submits
+        (#655). When the two differ, the first batch is a one-time initialization round, so
+        it is reported as such and the warnings are decided on the sustained number.
+        ``psets_in_flight`` is how many parameter sets that first batch of jobs came from,
+        which is how a parameter-set count is converted into a job count; None means one
+        job per parameter set.
+
         Only cluster runs are reported. A local run's worker count is exactly what the user
         asked for through parallel_count, so there is nothing to compare it against. The
         worker count comes from dask; anything that goes wrong reading it is logged and
@@ -1332,38 +1369,64 @@ def _report_parallelism(self, client, jobs_in_flight):
         if n_workers <= 0:
             return
 
-        logger.info('Parallelism: the fit starts with %d job(s) running and %d worker(s) '
-                    'connected.' % (jobs_in_flight, n_workers))
-
+        n_jobs = self.expected_parallelism()
+        if n_jobs is None or n_jobs <= 0:
+            n_jobs = jobs_in_flight
+        elif psets_in_flight:
+            # expected_parallelism() counts parameter sets, but one parameter set is more
+            # than one job when smoothing runs replicates of it or parallelize_models
+            # splits it across models. The first batch shows how many jobs a parameter set
+            # becomes, and that ratio holds for the rest of the fit.
+            n_jobs = int(round(n_jobs * jobs_in_flight / psets_in_flight))
+
+        logger.info('Parallelism: the fit runs %d job(s) at a time and %d worker(s) are '
+                    'connected.' % (n_jobs, n_workers))
+
+        notes = []
+        # A first batch of a different size is a one-time initialization round. Say so, so
+        # that a user watching the start of the fit is not surprised by it and does not
+        # read it as how busy the fit will be.
+        if jobs_in_flight != n_jobs:
+            notes.append('This fit begins with a one-time round of %d job(s) before it '
+                         'settles at %d.' % (jobs_in_flight, n_jobs))
         # A generational fit drains each generation to almost nothing before starting the
         # next, so some idle time is expected with one and should not be read as a fault.
-        note = ''
         if self.waits_for_full_generation:
-            note = (' This fit runs one generation at a time and waits for all of it to '
-                    'finish before starting the next, so some idle time toward the end of '
-                    'each generation is expected.')
+            notes.append('This fit runs one generation at a time and waits for all of it '
+                         'to finish before starting the next, so some idle time toward '
+                         'the end of each generation is expected.')
+        note = (' ' + ' '.join(notes)) if notes else ''
+
+        # Name the setting that sizes the concurrency, when one setting does. A fit where
+        # none does (profile likelihood, whose concurrency follows how many parameters it
+        # profiles) leaves the advice to talk about processors only.
+        setting = self.parallelism_setting
+        by_setting = ('the fitting settings, mainly %s,' % setting) if setting \
+            else 'the fitting settings,'
+        raise_setting = ('raising %s or ' % setting) if setting else ''
+        lower_setting = ('lowering %s or ' % setting) if setting else ''
 
         # A factor of two in either direction is the "large margin" that draws a warning.
-        if n_workers >= 2 * jobs_in_flight:
-            msg = ('The fit starts with only %d job(s) running but %d worker(s) connected, '
+        if n_workers >= 2 * n_jobs:
+            msg = ('The fit runs only %d job(s) at a time but %d worker(s) are connected, '
                    'so about %d worker(s) will sit idle. How many jobs run at once is set '
-                   'by the fitting settings, mainly population_size, not by how many '
-                   'processors were reserved. Consider raising population_size or reserving '
-                   'fewer processors.%s'
-                   % (jobs_in_flight, n_workers, n_workers - jobs_in_flight, note))
+                   'by %s not by how many processors were reserved. '
+                   'Consider %sreserving fewer processors.%s'
+                   % (n_jobs, n_workers, n_workers - n_jobs, by_setting,
+                      raise_setting, note))
             logger.warning(msg)
             print1('Warning: ' + msg)
-        elif jobs_in_flight >= 2 * n_workers:
-            msg = ('The fit starts with %d job(s) running but only %d worker(s) connected, '
-                   'so jobs will queue and the extra jobs buy no extra speed. How many jobs '
-                   'run at once is set by the fitting settings, mainly population_size. '
-                   'Consider lowering population_size or reserving more processors.%s'
-                   % (jobs_in_flight, n_workers, note))
+        elif n_jobs >= 2 * n_workers:
+            msg = ('The fit runs %d job(s) at a time but only %d worker(s) are connected, '
+                   'so jobs will queue and the extra jobs buy no extra speed. How many '
+                   'jobs run at once is set by %s not by how many processors were '
+                   'reserved. Consider %sreserving more processors.%s'
+                   % (n_jobs, n_workers, by_setting, lower_setting, note))
             logger.warning(msg)
             print1('Warning: ' + msg)
         elif note:
-            # The counts are close, but a generational fit still idles toward the end of
-            # each generation, so put that on the record for this run.
+            # The counts are close, but there is still something worth putting on the
+            # record for this run.
             logger.info(note.strip())
 
     def _finalize_run(self):

pybnf/algorithms/optimizers/concurrent_multistart.py

@@ -112,6 +112,15 @@ class ConcurrentMultiStartOptimizer(StartPointOptimizer):
     #: local path, ``'stopping'`` for the gradient path) -- cosmetic, preserved verbatim.
     _stop_verb = 'finished'
 
+    @property
+    def parallelism_setting(self):
+        """How many jobs one of these fits runs at once is its number of starts, since each
+        start holds one evaluation in flight, so the parallelism report names the key the
+        start count is read from rather than the base class's population_size (#655). That
+        is ``n_starts`` on the local path and ``population_size`` on the gradient path,
+        which predates the newer key."""
+        return self._n_starts_key
+
     def __init__(self, config, refine=False):
         # A subclass gate that must run *before* the (expensive) network generation in
         # Algorithm.__init__ -- the gradient path refuses a legacy-edition config here, so

pybnf/algorithms/optimizers/profile_likelihood.py

@@ -641,6 +641,11 @@ class ProfileLikelihoodAlgorithm(GradientOptimizer):
     fit_type = 'profile_likelihood'
     _method_label = 'profile-likelihood polish'
 
+    #: No single setting sets how many jobs this fit runs at once, so the parallelism
+    #: report names none and advises only about processors (#655). See
+    #: :meth:`expected_parallelism`.
+    parallelism_setting = None
+
     def __init__(self, config, refine=False):
         super().__init__(config, refine=refine)
         self.confidence = config.config['profile_likelihood_confidence']
@@ -696,6 +701,25 @@ def _init_profile_state(self):
         self._track_queue = []       # remaining (param_idx, direction) tracks not yet launched
         self._active_tracks = {}     # in-flight PSet name -> (param_idx, _ProfileTrack)
 
+    def expected_parallelism(self):
+        """Profiling runs one job per directional track, two tracks per profiled
+        parameter, up to whatever cap :meth:`_effective_parallel` applies. That is the
+        concurrency this fit spends nearly all of its time at, so it is what the
+        parallelism report should describe (#655).
+
+        The run loop's first batch is one job: the preflight evaluation of the box center,
+        which picks the inner optimizer. The two short phases that follow (the multi-start
+        polish to the optimum, then profiling) each run more than that. Reporting the
+        preflight would say a large allocation is almost entirely idle, which is true for
+        one evaluation and wrong for the rest of the fit.
+
+        No single setting sets this number: it follows how many parameters are profiled
+        (``profile_likelihood_params``, all of them by default), and
+        ``profile_likelihood_max_parallel`` can only lower it. Hence
+        :attr:`parallelism_setting` is None for this fit.
+        """
+        return self._effective_parallel()
+
     def _start_banner(self):
         return ("Running profile-likelihood analysis at the %g confidence level "
                 "(Delta chi2 = %g, 1 dof) for %i parameter(s)"