Base the parallelism report on how many jobs a fit keeps running (#655)

[wshlavacek]

Fixes #655.

The startup parallelism report measured "how many jobs the fit runs at once" from the first batch of jobs the run loop submits. For most fits that is the right number. For scatter search it is not. The first batch is the initialization round, which is init_size parameter sets, ten per free parameter by default and unrelated to the population. Every round after it runs population_size x (population_size - 1) simulations.

So a fit with seven free parameters and population_size = 20 on 384 reserved processors was told that 70 jobs would run, that 314 processors would sit idle, and that it should consider lowering population_size. It was about to run 380 simulations at a time. Lowering population_size would have reduced the number of processors it could use, which is the opposite of what the message was for.

Repro

job_type = ss
population_size = 20
init_size = 70

with seven free parameters, run on a cluster with 384 workers. Before this change the log and console say:

The fit starts with only 70 job(s) running but 384 worker(s) connected, so about 314 worker(s) will sit idle. ... Consider raising population_size or reserving fewer processors.

After it they say:

Parallelism: the fit runs 380 job(s) at a time and 384 worker(s) are connected.
This fit begins with a one-time round of 70 job(s) before it settles at 380. This fit runs one generation at a time and waits for all of it to finish before starting the next, so some idle time toward the end of each generation is expected.

No warning, because there is nothing wrong with that allocation.

What changed

An algorithm can now say how many parameter sets it keeps out for evaluation once it is under way, through a new expected_parallelism method, and the report uses that number. The default returns None, meaning the first batch is already the right answer, so every fit that was correct before is untouched. Scatter search returns its population pairs, the same number it already prints as "simulations per iteration". When the first round differs from the steady state, the report says so rather than warning about it. The number is scaled by smoothing and parallelize_models, which turn one parameter set into several jobs.

Profile likelihood had the same defect from the other end. Its first batch is a single preflight evaluation, so on any cluster it reported a fit that was almost entirely idle. It now reports its directional tracks, which is what it runs for nearly the whole fit.

The advice also names the setting each fit actually reads. It named population_size for every fit. Powell and simplex read n_starts and never look at population_size, so they are now told about n_starts. Profile likelihood has no single setting for this, since its concurrency follows how many parameters it profiles, so it is advised only about how many processors to reserve.

Documentation

init_size was not documented in the cluster guide, and its small default is exactly what leaves a large allocation idle during the initialization round. It now has its own section there and a fuller entry in the configuration keys.

Found while writing that: the table of how many simulations each fit type runs at once listed profile likelihood as pl, which is not a valid job_type. It now reads profile_likelihood.

Tests

Nine new tests cover the sustained count replacing the first batch, the warning quoting it, the jobs-per-parameter-set scaling, the setting each family names, scatter search reporting 380 rather than 70 on the cluster from the issue, and profile likelihood reporting its tracks rather than its preflight. The wording of the report changed, so the existing tests for it were updated to match.

Fast tier: 4593 passed, 13 skipped. The slow and recovery tiers were still running when this was opened, and I will report the result here.

[wshlavacek]

Reporting the test result promised above.

The slow and recovery tiers, which CI does not run, pass on the merged content: 4787 passed, 13 skipped, no failures, in 18 minutes 49 seconds. Run with BNG2.pl on the path and the default marker filter cleared, so every tier ran.