Skip to content

Add NVIDIA cuOpt as a GPU-accelerated solver - #909

Open
mal84emma wants to merge 10 commits into
PyPSA:masterfrom
mal84emma:feat/cuopt-solver
Open

Add NVIDIA cuOpt as a GPU-accelerated solver#909
mal84emma wants to merge 10 commits into
PyPSA:masterfrom
mal84emma:feat/cuopt-solver

Conversation

@mal84emma

Copy link
Copy Markdown
Contributor

Closes #515
Bringing Nvidia speed to linopy.

Why?

  • cuOpt is fast
  • It's free and open-source (Apache 2.0)
  • Did I mention it's fast?

Changes proposed in this Pull Request

First-class support for NVIDIA cuOpt as a linopy solver: LP, MILP and convex QP on the GPU through cuOpt's direct (in-memory) API, with duals for LP/QP, MIP gap and dual bound for MILP, semi-continuous variables, and a Model.to_cuopt() bridge. Installable with pip install "linopy[gpu]" (Linux, CUDA 12 driver ≥ 525.60.13, compute capability ≥ 7.0).

Note

The following content was generated by AI (a supervised multi-agent run; every commit on this branch is marked with its authoring model).

Design points a reviewer will care about

  • Direct API only. File io_apis transparently fall back to the direct build with one warning. The rationale (unverified solution-file parsing, an inverted Q convention between .lp and .mps) is documented at the code site.
  • method=3 (Barrier) is linopy's default, not cuOpt's own concurrent default, which segfaults on repeated solves of models ≳1300 variables (reproduced and documented). An escape hatch to PDLP (method=1) is documented for very large sparse LPs.
  • All cuOpt solves run on one persistent worker thread. cuOpt's bundled LLVM OpenMP runtime leaks per-thread state — the fresh-thread-per-solve pattern aborts the process after a handful of solves. Reported upstream as NVIDIA/cuopt#1768; the worker also owns native object disposal so garbage collection on foreign threads (e.g. dask workers) cannot trigger native teardown off-thread.
  • Maximisation is handled by sign-flipping the objective/Q into an equivalent minimisation and negating the objective, duals and dual bound back — verified against HiGHS on all six sense×constraint combinations plus the presolve path.
  • QP uses 0.5 · M.Q (cuOpt symmetrises internally); a committed deliberate-failure test proves the wrong convention changes the answer by 50% relative, so the guard cannot rot silently.
  • Honest refusals: MIQP raises NotImplementedError up front (cuOpt silently mis-solves it), warm start is refused (usable PDLP warm start needs three simultaneous non-default settings), reduced costs are not populated (upstream sign defect for maximised models with <= rows), SOS/indicator constraints are absent upstream. Each refusal carries a 1–3-line rationale at its enforcement site.
  • GPU probe runs in a subprocess so available_solvers on a GPU-less machine simply omits cuOpt, and the probe cannot poison later os.fork() children.

Performance

Indicative benchmark: LOPF of the PyPSA scigrid-de example network (585 buses, 1423 generators, 852 lines), its 24-hour day tiled to scale the LP; wall time of m.solve(), solvers run sequentially on an otherwise idle Azure VM (8-core EPYC 7V12, Tesla T4 16 GB). HiGHS was given threads=8, parallel=on (measured: no gain over its defaults); cuOpt ran at this branch's defaults (direct API, barrier). Objectives agree to ≤8·10⁻⁴ relative at both solvers' default tolerances.

LP size HiGHS cuOpt (T4) Speedup
60 k vars 4.1 s 2.2 s ~2×
417 k vars 116 s 5.8 s ~20×
1.67 M vars 1526 s 14.7 s ~104×
2.50 M vars 3359 s 20.9 s ~161×

cuOpt times are first-solve-in-process (cold): they include one-off CUDA context initialisation, which repeat solves in the same process avoid, running ~2 s faster. The gap widens with size (HiGHS scales ~n^1.9 here, cuOpt near-linearly). A T4 is the smallest supported datacenter GPU; newer hardware should do better.

Note

To be fair this is a bit of a trust-me-bro benchmark. HiGHS can solve much faster than its default config and even faster with better hardware (test was on a GPU optimized VM, though with not a very good GPU).
Regardless, cuOpt is fast and has been winning competitions.
Particularly now that cards with serious VRAM are available I think it could make solving large ESOMs more accessible without a commercial solver license.

Known upstream issue reviewers should weigh

cuOpt 26.08.00 can destabilise a long-lived host process after any in-process solve: its bundled OpenMP runtime intermittently corrupts glibc malloc metadata, crashing the process later in unrelated code. This was diagnosed to root cause (gdb backtrace into _int_malloc, linopy exonerated by a source audit and by mitigation experiments) and reported with full evidence on NVIDIA/cuopt#1768. Every completed full test-suite run on this branch has a failure set byte-identical to master's baseline — correctness is unaffected — but roughly two thirds of full --run-gpu suite runs abort mid-run from this corruption. The feature is opt-in (install extra + explicit solver choice); whether to ship ahead of an upstream fix is a maintainer decision this PR intentionally surfaces rather than hides.

Verification summary
  • Expected values in the test suite were confirmed by differential testing against live in-process HiGHS 1.15.1 during development, then baked in: LP objective/primal/duals on all six sense×constraint-sense cells (duals agreed to ≤2.5e-9), MILP objective/gap/bound, QP objective to 2e-14 on the pinned 3-variable model, min and max.
  • Status-map verification with forcing recipes for every reachable cuOpt termination status (both LP and MILP enums), including an empty-Solution time-limit path.
  • 20 sequential n=2000 LP solves in one process (regression guard for the upstream thread-state leak; threshold chosen 1.54× above the measured abort boundary).
  • All cuOpt GPU tests pass (pytest test/test_cuopt.py test/test_semi_continuous.py --run-gpu: 102 passed); CPU-only machines skip all of them; full CPU suite and every completed full GPU suite run are failure-set-identical to master's baseline (74 pre-existing failures/errors, all in test/remote and unrelated areas).
  • Interrupt contract (Ctrl-C returns promptly, GPU work completes, worker reusable) covered by CPU-only tests.
  • Docs build warning-identical to master's baseline in a venv without cuOpt installed (the ReadTheDocs condition).
  • mypy: zero new errors (28 pre-existing Xpress errors on master unchanged); ruff clean.

Implementation Approach

This PR was a bit of an experiment at getting a Claude agent system to one-shot new features. If you're interested in the approach I took you can read the agent docs and my thoughts on how it went here.

Checklist

  • Read by a human (Claude wrote the code, but I've read and checked it, all mistakes are mine).
  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

claude and others added 10 commits August 23, 2026 15:56
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add test/test_cuopt.py with the cuOpt differential tests against live HiGHS
(dual sign matrix, presolve maximisation, status mapping, options, refusals,
device probe and fork safety) and test/test_cuopt_interrupt.py for the
KeyboardInterrupt helper. Register cuOpt's time limit option and feature
matrix rows, and skip the warmstart test for it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Trim test/test_cuopt.py to the tests that guard linopy's compensation
code for cuOpt quirks that fail silently (always-minimise duals, 0.5*Q
Hessian, pad-row dual slicing, empty-primal handling, MIQP pre-check)
plus the process-safety machinery. All expectations are now baked
analytic values (confirmed once against HiGHS 1.15.1) instead of live
differential solves. Fold test_cuopt_interrupt.py into test_cuopt.py,
add cuOpt cases to test_semi_continuous.py, document the remaining
quirks in doc/gpu-acceleration.rst and condense the cuOpt comments and
docstrings in solvers.py to the guidance future devs need.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fixes from two adversarial review passes over the cuOpt implementation:

- Widen the pad row for constraint-free models to include 0 so a
  semi-continuous variable keeps its off state (was silently forced on,
  reported Optimal); guarded by a new constraint-free SC test.
- Hand DataModel ownership to the solve worker via a box list so the
  worker's clear is deterministically the final decref for the
  solver-owned reference; docstrings reworded to the actual thread
  contract (solves and unpredictable-thread teardown, not all natives).
- refresh() clears the CUDA probe's own cache; probe failures now log
  the exception and stderr so a broken install is distinguishable from
  a missing GPU.
- Guard a short LP dual like the primal instead of an opaque IndexError;
  MIQP refusal names semi-continuous variables too.
- Default io_api=None fallback logs at info, only an explicitly
  requested file io_api warns; docs updated to match.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Guard cuOpt against silent int32 CSR truncation beyond 2**31-1 nonzeros
- Trim cuOpt helper docstrings to state failures without inline evidence
- Simplify square_equality_model to a fixed one-row fixture with baked values
- Assert cuOpt's GPU_ACCELERATION capability in the feature matrix
- Mark the repeated-solves test `slow` (intrinsic ~135 s GPU cost) and
  register the marker so it can be deselected during development
- Drop the non-discriminating semi-continuous cuOpt test, loosen the
  unbounded-QP status pin to membership, and source the warmstart skip reason
- Slim the GPU documentation

Co-Authored-By: mal84emma <info@materialdifference.earth>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codspeed-hq

codspeed-hq Bot commented Aug 28, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 19.95%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 4 regressed benchmarks
✅ 171 untouched benchmarks
⏩ 175 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Memory test_to_lp[milp-n=50] 2 MB 2.7 MB -24.48%
Memory test_to_lp[masked-n=100] 2.1 MB 2.7 MB -22.43%
Memory test_to_lp[merge_balance-severity=0] 2.6 MB 3.2 MB -17.78%
Memory test_to_lp[expression_arithmetic-n=250] 40.3 MB 47.3 MB -14.76%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mal84emma:feat/cuopt-solver (ff9383d) with master (570ff2e)

Open in CodSpeed

Footnotes

  1. 175 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bindings for cuOpt solver

2 participants