From 58f147aca672b61d4e24fe97adfbd7145c88fde8 Mon Sep 17 00:00:00 2001 From: Krstan Vjestica Date: Sun, 26 Jul 2026 11:21:13 +0200 Subject: [PATCH 1/3] docs: warn that upgrading 2026.07.0 stops job execution Anyone upgrading from 2026.06.0 today gets a platform that accepts jobs and runs none of them, with only "not enough available capacity" to explain it. Document both conditions that have to hold for the default queue to work, the verified workaround, and the fact that the workaround does not survive a restart of forail-task. Also spell out that role assignments lost to the 2026.06.0 bug are not recreated by the upgrade. --- docs/RELEASE_NOTES_v2026.07.0.md | 61 ++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/docs/RELEASE_NOTES_v2026.07.0.md b/docs/RELEASE_NOTES_v2026.07.0.md index 1686200..5ff3c1f 100644 --- a/docs/RELEASE_NOTES_v2026.07.0.md +++ b/docs/RELEASE_NOTES_v2026.07.0.md @@ -23,6 +23,63 @@ with the tenancy work; both only drop and re-create PostgreSQL row-level-securit policies, so they apply to an existing database without touching table schemas or rows. +## ⚠️ Known issue — upgrading breaks job execution + +**If you upgrade an existing 2026.06.0 install, jobs stop running: they are +accepted and then stay in `pending` indefinitely.** The only hint is the job's +`job_explanation`, *"This job is not ready to start because there is not enough +available capacity"* — accurate, but it does not point at the cause. Fresh +installs are unaffected. This is open; the workaround below is verified on a +live cluster. + +The `default` instance group has to satisfy two conditions at once for a job to +run locally, and an upgrade breaks both: + +1. **It must contain an instance.** The chart's init Job calls `register_queue + --queuename=default`, which on an upgrade finds the group already there, + prints `Instance Group already registered default` and assigns nothing. +2. **That instance must be able to execute.** The task pod re-registers itself + as `node_type=control` on every start, and a control node only orchestrates. + +Either one alone is enough to hang every launch — both were measured +individually, holding the other fixed. + +Project updates keep working, because they run in `controlplane`, which does +have the instance. The install therefore looks healthy right up until someone +launches a job. + +**Workaround, after `helm upgrade` completes:** + +```bash +kubectl -n forail exec deploy/forail-web -- forail-manage shell -c " +from forail.main.models import Instance, InstanceGroup +i = Instance.objects.get(hostname='forail-node') +i.node_type='hybrid'; i.save(update_fields=['node_type']) +InstanceGroup.objects.get(name='default').instances.add(i)" +``` + +Substitute your own instance hostname if you did not install with the chart +defaults. Any job already sitting in `pending` starts on its own within about a +minute; a fresh short job should return a normal `PLAY RECAP` in a few seconds. + +> **The workaround does not survive a restart of `forail-task`.** That pod +> re-runs `provision_instance` every time it starts — after a node reboot, an +> eviction, or the next `helm upgrade` — and that call resets both `node_type` +> and the group's execution mode. Re-apply it, and re-check job execution, after +> any task-pod restart until the fix ships. + +**Also re-apply your role assignments after upgrading.** Any role assignment +attempted on 2026.06.0 failed silently (the `ScanFinding` / +`TenantIsolationEvent` `FieldDoesNotExist` bug fixed in this release, see +*Fixed*). The upgrade fixes the cause but does not recreate the assignments that +were lost, and *Fixed* saying "no data migration is required" refers to the +schema only. Check the members of every role you rely on and re-grant what is +missing. + +What the upgrade does do correctly: it succeeds, migrations `0209` and `0210` +apply cleanly, and no data is lost — object counts and names are identical +before and after. + ## Security advisories **Upgrade from 2026.06.0 or earlier is strongly recommended.** Several of the @@ -326,6 +383,10 @@ helm upgrade forail oci://ghcr.io/forail-platform/forail-helm \ Before upgrading: +- **Any deployment that runs jobs** — read **Known issue — upgrading breaks job + execution** at the top of these notes, and plan to apply the workaround (and + re-apply role assignments) as part of the upgrade. Without it the platform + comes up healthy but executes nothing. - **SAML deployments** — review **Breaking changes — SAML** above and reconfigure the IdP if needed. - **Any deployment** — review **Breaking changes — deployment defaults**; an From d5e9035168bdc0501a532c6f7aeb671b20511ce8 Mon Sep 17 00:00:00 2001 From: Krstan Vjestica Date: Sun, 26 Jul 2026 20:08:54 +0200 Subject: [PATCH 2/3] docs: record the 2026.06.0 dispatcher crash loop under Fixed Measured on a fresh 2026.06.0 install while validating the upgrade path: the dispatcher exits every ~50s on the unconditional 30s observability schedule, because the task carries Celery's @shared_task rather than Forail's @task(). Every job in flight is cancelled with it, so the release cannot run a job at all. It was fixed in this release without being written down, which left the strongest reason to upgrade undocumented. --- docs/RELEASE_NOTES_v2026.07.0.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/RELEASE_NOTES_v2026.07.0.md b/docs/RELEASE_NOTES_v2026.07.0.md index 5ff3c1f..0000266 100644 --- a/docs/RELEASE_NOTES_v2026.07.0.md +++ b/docs/RELEASE_NOTES_v2026.07.0.md @@ -316,6 +316,18 @@ front of the ingress, or set `forail.cookieSecure: "false"` for a lab install. ## Fixed +- **The task dispatcher crash-looped on 2026.06.0, so no job could finish.** + The periodic schedule runs `update_active_jobs_gauge_task` every 30 seconds + unconditionally, but in 2026.06.0 that function carried Celery's + `@shared_task` instead of Forail's own `@task()`. Dispatching it raised + `ValueError: ... is not decorated with @task()`, the dispatcher exited, and + whatever was running died with *"Task was canceled due to receiving a + shutdown signal"* — typically surfacing as a failed project update and a job + in `error`. Measured on a fresh 2026.06.0 install: the dispatcher restarted + roughly every 50 seconds, indefinitely. **Anyone still on 2026.06.0 should + upgrade**; there is no configuration that avoids this, since the schedule + entry is not conditional. Fixed by registering the task properly + (`@task(queue=get_task_queuename)`). - **In-cluster job execution.** Two pieces were missing from the chart, and each failed a launch on its own. Note this is not "out of the box": project updates and control-plane jobs still run through podman inside the task pod, so they From 39b43a15adb77fb638befdff32ad9aeff5e50e27 Mon Sep 17 00:00:00 2001 From: Krstan Vjestica Date: Sun, 26 Jul 2026 20:31:48 +0200 Subject: [PATCH 3/3] docs: release notes for 2026.07.1 A patch release with one purpose: an upgrade no longer leaves a platform that accepts jobs and runs none of them. Also marks the known issue in the 2026.07.0 notes as fixed, and points at the release rather than the workaround. --- docs/RELEASE_NOTES_v2026.07.0.md | 7 ++- docs/RELEASE_NOTES_v2026.07.1.md | 90 ++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 2 deletions(-) create mode 100644 docs/RELEASE_NOTES_v2026.07.1.md diff --git a/docs/RELEASE_NOTES_v2026.07.0.md b/docs/RELEASE_NOTES_v2026.07.0.md index 0000266..26294d5 100644 --- a/docs/RELEASE_NOTES_v2026.07.0.md +++ b/docs/RELEASE_NOTES_v2026.07.0.md @@ -25,12 +25,15 @@ rows. ## ⚠️ Known issue — upgrading breaks job execution +> **Fixed in 2026.07.1.** Upgrade to it instead of applying the workaround +> below — see the [2026.07.1 release notes](RELEASE_NOTES_v2026.07.1.md). The +> rest of this section describes what happens if you stay on 2026.07.0. + **If you upgrade an existing 2026.06.0 install, jobs stop running: they are accepted and then stay in `pending` indefinitely.** The only hint is the job's `job_explanation`, *"This job is not ready to start because there is not enough available capacity"* — accurate, but it does not point at the cause. Fresh -installs are unaffected. This is open; the workaround below is verified on a -live cluster. +installs are unaffected. The workaround below is verified on a live cluster. The `default` instance group has to satisfy two conditions at once for a job to run locally, and an upgrade breaks both: diff --git a/docs/RELEASE_NOTES_v2026.07.1.md b/docs/RELEASE_NOTES_v2026.07.1.md new file mode 100644 index 0000000..afbb068 --- /dev/null +++ b/docs/RELEASE_NOTES_v2026.07.1.md @@ -0,0 +1,90 @@ +# Forail 2026.07.1 — Release Notes + +**Release date:** 2026-07-26 +**Based on:** Forail 2026.07.0 +**License:** Apache License 2.0 + +--- + +## Overview + +2026.07.1 is a **patch release with one purpose**: an upgrade no longer leaves a +platform that accepts jobs and runs none of them. It fixes the known issue +published with 2026.07.0, and it removes the manual workaround that release +asked for. + +Nothing else changes. There are no migrations, no configuration changes, no +breaking changes. The frontend, the operator and the assistant are unchanged and +keep their 2026.07.0 / 2026.07.1 / 2026.06.0 versions respectively; only the +backend image and the Helm chart move. + +| Component | Version | +|---|---| +| `forail-backend` | **2026.07.1** | +| Helm chart | **2026.7.1** (pins the backend above) | +| `forail-frontend` | 2026.07.0, unchanged | +| `forail-operator` | 2026.07.1, unchanged | +| `forail-assistant` | 2026.06.0, unchanged | + +## Fixed + +### Jobs no longer stop running after an upgrade or a restart + +Upgrading 2026.06.0 → 2026.07.0 left every launch sitting in `pending` +indefinitely, with only `job_explanation` — *"This job is not ready to start +because there is not enough available capacity"* — to explain it. Project +updates kept working, so the install looked healthy right up until someone +launched a job. + +**What was wrong.** For jobs to run on the node itself, the `default` instance +group has to be a regular group that contains an execution-capable instance. +Two things conspired against that, and either one alone was enough to hang every +launch: + +- `register_queue` assigns instances only when it *creates* a group. On an + upgrade the group already exists, so it assigned nothing and left it empty. +- The task pod re-ran `provision_instance` on **every start** — restart, + eviction, rolling upgrade — and that call hardcoded `node_type='control'` and + re-registered `default` as a ContainerGroup, overwriting whatever the + installer had configured. A control node only orchestrates; it does not + execute. + +The second half is why the 2026.07.0 workaround did not stick: the next task-pod +restart quietly undid it. + +**What changed.** Registration now takes its intent from `FORAIL_NODE_TYPE` — +which the Helm chart and the Compose stack already set — and derives the default +queue from it. An execution-capable pod (`hybrid`, `execution`) gets a regular +instance group containing itself; a control-only pod keeps the ContainerGroup, +exactly as before. Both defaults are unchanged when the variable is unset, so a +multi-node install that has no opinion behaves as it did. The chart's init Job +additionally asserts group membership rather than trusting `register_queue`, so +a newer chart paired with an older image still converges. + +**Verified, not assumed.** On a freshly created cluster: install 2026.06.0, +`helm upgrade` to this release, launch a job — successful in 78 s with a normal +`PLAY RECAP`, with no manual intervention at any point. Two subsequent +`kubectl rollout restart deploy/forail-task` left the state untouched and the +next job succeeded as well. + +## Upgrade + +From **2026.07.0**, a straight image re-point. No migrations, no new required +values: + +```bash +helm upgrade forail oci://ghcr.io/forail-platform/forail-helm \ + --version 2026.7.1 -n forail \ + --set secrets.forailAdminPassword='' \ + --set 'forail.allowedHosts=forail.example.com\,127.0.0.1\,localhost' \ + --set task.privileged=true --set task.hostCgroup=true # only if you run jobs in-pod +``` + +**If you applied the 2026.07.0 workaround**, you can leave it in place — it sets +exactly the state this release converges to on its own. Nothing needs to be +undone. + +From **2026.06.0**, read the [2026.07.0 release +notes](RELEASE_NOTES_v2026.07.0.md) first: the breaking changes, the required +admin password and the SAML defaults all still apply. The known issue documented +there no longer does.