ci: probe production health off-host on a schedule

[aterga]

Summary

Item 8 of the rollout plan — "Set up alerts based on the status page".

The status dashboard already runs every check this needs (MCP, the full OAuth suite, TLS, and the Internet Identity linkage). But it runs on the host and it is pull-only: something has to look at it. A host that is down cannot tell you it is down, and that is both the failure mode that matters most and the one the current setup structurally cannot report.

This runs the same suite from GitHub's infrastructure every fifteen minutes. No new service, no new vendor, no dependencies — monitoring/mcp-status has none, so the probe is a checkout and a node invocation.

Related issues

Complements the hosted dashboard rather than replacing it. The longer-term answer is the organisation's Prometheus/OpsGenie stack (dfinity-ops/k8s), which is a separate conversation with #eng-observability.

Changes

• .github/workflows/health.yml — a scheduled probe job.

Design decisions worth review:

• Escalation is labelled honestly, not overstated. A failure fails the workflow, which notifies watchers, and posts to Slack once a webhook exists. Neither is a page and nobody is on call. The header comment says so rather than implying coverage that isn't there.
• The Slack step is conditional on the secret being present. The workflow is useful the day it lands, and gains routing to #eng-identity-imcp2 the moment someone adds SLACK_WEBHOOK_URL — no code change. That deliberately keeps a configuration decision from blocking the signal.
• Deliberately not --strict. That flag escalates warnings to failures, and a third of the suite probes Internet Identity and the IC — infrastructure this team does not operate. Paging on someone else's degradation is how a channel learns to ignore its alerts. Without it the CLI exits non-zero exactly when overall is fail, the same threshold the dashboard uses to serve 503.
• The summary names the failing checks, so the reader is not sent on a log dive at the moment they are least inclined to do one.

Testing

Three defects were found by running it rather than reasoning about it, and all three would have shipped a workflow that looked correct:

1. The probe refuses any origin outside its SSRF allow-list. Production is not id.ai, so it must be named explicitly — otherwise the run reports a usage error instead of a health verdict.
2. Report sections carry title, not label. The summary table would have rendered every section as ?.
3. The Slack step was gated on failure(), which never fires here. The probe swallows its own exit code on purpose, so nothing has failed at that point in the job. The notification would have been silently dead while the run still went red and looked correct from outside.

Verified against three inputs:

• Healthy — live production. Exit 0, overall: pass, 18 checks, summary reads "All checks passed."
• Unhealthy — an origin that answers but is not the service. Exit 1, overall: fail, 8 checks fail, summary table renders real section titles and check labels.
• Unreachable — no report produced at all. Graceful message rather than a traceback, and the Slack line says the host may be unreachable.
• YAML parsed and every step's if: asserted by parsing the file, not by eye.
• $GITHUB_OUTPUT value collapsed to one line and length-capped — that file is line-oriented, so a newline in a value could declare further outputs. Static strings today; free to harden now.
• Not run: cargo / npm — no application code changed.

Checklist

• I have read the Contributing guidelines.
• Docs updated — the rationale and the optional secret are documented in the workflow header.
• No secrets, credentials, or internal-only information are included.

Limits

Fifteen minutes is the cadence, but GitHub's scheduler is best-effort and drifts under load, so treat this as noticed within the hour, not an SLA. Scheduled workflows also disable themselves after 60 days without repository activity.

There is no flap suppression: a sustained outage produces one failure per tick. If that proves noisy, transition-detection is the follow-up — I would rather see the real noise level than guess at it now.

To enable Slack: add a repository secret SLACK_WEBHOOK_URL pointing at an incoming webhook for #eng-identity-imcp2. Nothing else changes.

---

Generated by Claude Code

[Copilot]

Pull request overview

Adds off-host production health monitoring through a scheduled GitHub Actions workflow.

Changes:

• Runs the MCP health suite every 15 minutes.
• Generates failure summaries and optionally notifies Slack.
• Marks unhealthy runs as failed.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (4)

.github/workflows/health.yml:196

• The summary output is derived from server-advertised instance names, and interpolating it directly into a double-quoted shell command permits command substitution (for example, an instance name containing $(...)) when a named II check fails. Pass the output through env so the shell treats it only as data.

          echo "::error::production health check failed (${{ steps.summary.outputs.failed }})"

.github/workflows/health.yml:154

• Collapsing whitespace protects $GITHUB_OUTPUT, but this value is later rendered as Slack mrkdwn. A server-advertised name such as <!channel> or <https://attacker.example|Run log> can therefore trigger mentions or spoof links in the alert. Slack requires &, <, and > to be entity-escaped when inserting untrusted text.

          line = " ".join(line.split())[:400]

.github/workflows/health.yml:122

• This only escapes table separators and actually removes existing backslashes; Markdown constructs such as [Run log](https://attacker.example), images, emphasis, and autolinks remain active. Since the value can contain a server-advertised instance name, it can still spoof the workflow summary despite the comment's stated protection. Escape all Markdown punctuation before rendering the cell.

This issue also appears in the following locations of the same file:

• line 154
• line 196

              return " ".join(t.split()).replace("\\", "").replace("|", "\\|")[:120] or "?"

.github/workflows/health.yml:10

• Repository watchers are not automatically emailed for failed Actions runs. Scheduled-run notifications are associated with the scheduled workflow's actor (typically the user who last changed the cron) and that user's notification settings, so with the optional Slack secret absent this remains pull-only for the rest of the team rather than providing the alert described by the PR. Add or require an explicit team notification route before treating workflow failure as the baseline signal.

# Escalation, honestly labelled: a failure here fails the workflow, which
# notifies watchers by email, and posts to Slack when a webhook is configured.