Speed up parallel model import

[FFroehlich]

Parallel model import (AMICI_IMPORT_NPROCS > 1) created a brand-new
spawn worker pool for every single parallelized operation. Since model
import performs dozens of them, and every worker of every pool had to
import sympy and amici from scratch, the process startup overhead often
exceeded the actual work -- for all but the largest models, parallel
import was slower than serial import.

• Create the worker pool lazily and reuse it for all operations, instead
  of creating and tearing one down per operation.
• Use the forkserver start method instead of spawn. Its workers are
  forked from a small, single-threaded server process that imports sympy
  and amici only once, which makes pool creation ~5x cheaper without
  reintroducing the deadlock potential of forking the (potentially
  multi-threaded) main process. spawn is kept only where forkserver
  is unavailable, i.e. on Windows.
• Process small matrices serially, where the inter-process communication
  overhead outweighs any speed-up.
• Compute the jacobian sparsity pattern from the symbols of each row
  instead of calling Basic.has for every (row, variable) pair, which
  re-traverses the full expression tree every time. This also speeds up
  serial import.
• Validate AMICI_IMPORT_NPROCS instead of failing with an opaque error
  further down, and detect unpicklable functions in _parallel_applyfunc
  before dispatching them. The previous PicklingError handler never
  triggered for the most common case (a lambda), which raises
  AttributeError.

Micro-benchmark (4 cores, 12 smart_jacobian calls, ~1.5 s to import
amici): 25.0 s -> 2.8 s. For a workload large enough for parallelization
to pay off, the speed-up over serial import improves from 2.3x to 4.2x
on 4 cores.

[Copilot]

Pull request overview

This PR improves the performance of AMICI’s SymPy-based model import when parallelization is enabled (AMICI_IMPORT_NPROCS > 1) by reducing multiprocessing startup overhead and optimizing Jacobian sparsity preprocessing. It introduces a lazily created, reusable worker pool (preferring forkserver where available), adds early validation and clearer errors for invalid/unpicklable inputs, and adds tests covering both serial and parallel paths.

Changes:

• Reuse a lazily created multiprocessing pool across import operations; prefer forkserver over spawn when available.
• Add serial fast-paths for small workloads and optimize Jacobian sparsity detection for symbol variables.
• Add tests for pool reuse, start method selection, AMICI_IMPORT_NPROCS validation, and parallel applyfunc behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
python/sdist/amici/_symbolic/sympy_utils.py	Implements reusable worker pool, forkserver/spawn selection, serial thresholds, and sparsity/applyfunc optimizations
python/tests/test_sympy_utils.py	Adds unit tests for serial/parallel Jacobian, _parallel_applyfunc, pool reuse, and env var validation
CHANGELOG.md	Documents the parallel import performance improvement and behavior changes

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 96.72131% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.14%. Comparing base (e6e695f) to head (cbeb8b4).

Files with missing lines	Patch %	Lines
python/sdist/amici/_symbolic/sympy_utils.py	96.72%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3232      +/-   ##
==========================================
- Coverage   78.61%   78.14%   -0.48%     
==========================================
  Files         318      318              
  Lines       21122    21166      +44     
  Branches     1487     1487              
==========================================
- Hits        16606    16540      -66     
- Misses       4508     4618     +110     
  Partials        8        8              

Flag	Coverage Δ
cpp	72.21% <96.72%> (+0.16%)	⬆️
cpp_python	36.54% <32.78%> (+<0.01%)	⬆️
petab	48.33% <32.78%> (-0.03%)	⬇️
petab_sciml	16.38% <32.78%> (+0.05%)	⬆️
petab_sciml_benchmarks	14.95% <32.78%> (+0.05%)	⬆️
python	70.58% <96.72%> (+0.16%)	⬆️
sbmlsuite-jax	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
python/sdist/amici/_symbolic/sympy_utils.py	98.26% <96.72%> (+19.38%)	⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[FFroehlich]

Performance check:

Before:

Time for task petab_import is 731.48s, which is within the reference time of 740.00s.

After:

Time for task petab_import is 378.26s, which is within the reference time of 740.00s.