Add a modular workflow runner next to the existing one

This is the main step of a refactoring of MC/bin/o2_dpg_workflow_runner.py,
presented at CHEP 2026. The 2000-line script becomes a package, resource
monitoring moves off the scheduling loop and costs about a tenth of what it
did, and the scheduling policy becomes selectable, with two alternatives to
the original one. Both runners are installed side by side and a dispatcher
picks between them, so nothing changes for a caller that does not ask for the
new one.

- MC/bin/o2_dpg_workflow_runner.py becomes that dispatcher. It reads
  ALIEN_O2DPG_WORKFLOW_RUNNER and defaults to "legacy". The original runner
  moves unchanged to MC/bin/o2dpg_workflow_runner_legacy.py, and the
  o2dpg_workflow_runner.py symlink beside it is untouched, so all ten call
  sites in the repository keep working under either runner.
- MC/workflow_runner/ holds the new package: workflow, graph, resources,
  monitoring, scheduler, executor, cleanup and cache modules.
- Monitoring moves to a background thread with separate CPU and memory
  cadences. Polling psutil synchronously in the scheduling loop cost 10-20 %
  of a core on realistic workflows; this costs 1-2 %.
- --scheduler-policy selects timeframe, which is the default and reproduces
  the original ordering, critical-path, or best-fit.
- --systemd-run supersedes --cgroup for confining a workflow to a CPU and
  memory budget.
- --cache-policy writes a fingerprint of command, environment, software tag
  and dependencies next to the _done marker, so a task whose command changed
  re-runs instead of being skipped. _done remains the skip marker.
- Every flag the original parser accepts, the new one accepts. --cgroup,
  --webhook and --checkpoint-on-failure are accepted, ignored and warned
  about, so a JDL passing them through
  ALIEN_O2DPG_ADDITIONAL_WORKFLOW_RUNNER_ARGS still runs.
- 68 unit tests come with it, run by a new CI job in
  .github/workflows/syntax-checks.yml. No Python test ran in CI before.
- MC/workflow_runner/o2dpg_runner/README.md documents the layout, the
  behavioural differences and the pitfalls found while building it.

https://indico.cern.ch/event/1471803/contributions/6967072/

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: sawenzel

.github/workflows/syntax-checks.yml

@@ -105,6 +105,21 @@ jobs:
           done
           exit "$error"   
 
+  runner-tests:
+    name: Workflow-runner unit tests
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install prerequisites
+        run: pip install pytest psutil
+
+      - name: Run the o2dpg_runner test suite
+        working-directory: MC/workflow_runner
+        run: pytest o2dpg_runner/tests -q
+
   pylint:
     name: Pylint
     runs-on: ubuntu-latest

.gitignore

@@ -2,3 +2,4 @@
 .vscode
 *.pyc
 o2dpg_tests/**
+__pycache__/