Skip to content

feat(DEVA11Y-657): composite GitHub Action (v1 scan+report)#1

Open
Crash0v3rrid3 wants to merge 8 commits into
mainfrom
feat/DEVA11Y-463-action-v1
Open

feat(DEVA11Y-657): composite GitHub Action (v1 scan+report)#1
Crash0v3rrid3 wants to merge 8 commits into
mainfrom
feat/DEVA11Y-463-action-v1

Conversation

@Crash0v3rrid3

Copy link
Copy Markdown
Collaborator

DEVA11Y-657 — Composite GitHub Action (v1: scan + report)

Ships the public composite Action for the CI/CD accessibility flow (Epic DEVA11Y-463, ADR-0035/0036).

A developer comments @AccessibilityDevTools on a PR → this Action runs the published @browserstack/accessibility-devtools CLI in PR-scan mode, authenticated by a BrowserStack Service Account key (zero GitHub credentials on the runner) → the CLI sends findings to linter-server, which normalizes + HMAC-signs them (scanProof) → the runner forwards {report, scanProof} (plus a best-effort Actions OIDC token) to the ci-review-service, which verifies and posts as the GitHub App.

What's in this PR

  • action.yml — composite Action with the ADR-0035 Appendix A input surface (username, access-key, comment, check-gate, fail-on-severity, inline-suggestions, sarif, comment-mode, remediation, ai-agent) and outputs (result, error-count, warning-count, findings-count, comment-url, sarif-file).
  • examples/browserstack-a11y.yml — reference consumer workflow (on: issue_comment, permissions: contents/pull-requests read + id-token write).
  • .github/workflows/release.yml — versioned release/bundle pipeline (SHA-pinned CLI, tagged dist).
  • README.md, LICENSE, .gitignore.

Fork-PR safety

"Diff, don't build" — no npm install/postinstall of untrusted head code; degrades to skip when OIDC/secrets are restricted.

Status / dependencies

  • Draft — depends on the linter-server + CLI (a11y-linter #406, story DEVA11Y-655) and ci-review-service (a11y-linter #407, story DEVA11Y-656) landing, plus the real GitHub App credentials + Marketplace listing.
  • CLI version pin / Marketplace publish to be finalized before GA.

🤖 Generated with Claude Code

Crash0v3rrid3 and others added 2 commits July 8, 2026 13:02
Scaffold the public browserstack-accessibility-devtools-action repo per
ADR-0035 Appendix A (GitHub App + Service Account model).

- action.yml: composite action. Inputs username/access-key (required, secret),
  comment, check-gate, fail-on-severity, inline-suggestions, sarif, comment-mode,
  remediation, ai-agent (validated ^[A-Za-z0-9-]+(\[bot\])?$). Outputs result,
  error-count, warning-count, findings-count, comment-url, sarif-file. Hardcoded:
  @AccessibilityDevTools trigger, diff scope, deva11y.config.* inference, commenter
  write/maintain/admin permission gate. Steps implement fork-PR "diff, don't build"
  safety (no install/postinstall of head code; skip when OIDC/secrets restricted),
  pinned npx CLI (placeholder 1.3.0), Actions OIDC mint (aud placeholder
  browserstack-accessibility-devtools), and POST to ci-review-service /v1/findings
  with {platform,repo,prNumber,headSha,findings,scanProof,oidcToken} + X-CI-Platform
  header. The Action holds no GitHub write credential; the App posts server-side.
- examples/browserstack-a11y.yml: reference consumer workflow (issue_comment,
  read-only token + id-token: write, concurrency, if-guard).
- .github/workflows/release.yml: release stub (esbuild dist build guard, action.yml
  validation, manual org-admin Marketplace publish note).
- README.md: public customer README (v1 = scan + report; agent hand-off preview
  off by default; homepage browserstack.com).
- LICENSE (MIT), .gitignore.

Placeholders pending provisioning: CLI version 1.3.0, OIDC audience and service URL
(https://devtools-a11y-linter.browserstack.com), GitHub App. Draft PR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…y token

The commenter permission pre-check called GET /collaborators/{user}/permission,
which requires push access. The documented least-privilege workflow token
(contents: read) cannot call it, so the request 403s, the `|| echo none`
fallback yielded "none", and the action refused to scan for EVERY commenter —
non-functional under its own recommended setup.

Make the client-side check best-effort: fast-refuse only on a DEFINITIVE
non-privileged role; on an inconclusive call (the read-only-token case) proceed
and rely on the authoritative server-side fail-closed permission gate in the
ci-review-service (ADR-0035 §5). The runner is the customer's and was never the
security boundary.

Also correct the preflight fork-safety comments: issue_comment workflows run in
base-repo context with secrets available even for fork PRs, so cred-absence is
not what protects fork PRs — the permission gate + "diff, don't build" are.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Crash0v3rrid3

Copy link
Copy Markdown
Collaborator Author

Code review

Reviewed the composite action (bugs + security; no CLAUDE.md in this repo). Found 1 functional issue — now fixed in 222bd37:

  1. Commenter permission gate hard-failed under the documented read-only token (P1). GET /repos/{owner}/{repo}/collaborators/{username}/permission requires push access; the recommended least-privilege workflow token (contents: read, pull-requests: read) cannot call it, so it 403'd, the || echo none fallback resolved to none, and the action refused to scan for every commenter — non-functional under its own recommended setup.

    Fix: the client-side check is now best-effort — it fast-refuses only on a definitive non-privileged role and, on an inconclusive call (the read-only-token case), proceeds and defers to the authoritative server-side fail-closed permission gate in the ci-review-service (ADR-0035 §5). The runner is the customer's and was never the security boundary.

    # read-only token the call fails (gh exits non-zero) and we must NOT refuse.
    if perm="$(gh api "repos/${repo}/collaborators/${actor}/permission" --jq '.permission' 2>/dev/null)"; then
    echo "Commenter ${actor} has permission: ${perm}"
    case "$perm" in
    admin|maintain|write)
    echo "allowed=true" >> "$GITHUB_OUTPUT" ;;
    *)
    echo "::warning::${actor} lacks write/maintain/admin permission; refusing to scan."
    echo "allowed=false" >> "$GITHUB_OUTPUT" ;;
    esac
    else
    # Could not read the commenter's permission (expected with the documented
    # read-only GITHUB_TOKEN, which cannot call the collaborators API). Do not

Also corrected the preflight fork-safety comments: issue_comment workflows run in base-repo context with secrets available even for fork PRs, so cred-absence is not what protects fork PRs — the permission gate + "diff, don't build" (no PR-head-code execution) are.

Verified clean: OIDC request URL separator (&audience= onto GitHub's already-?-bearing request URL), jq --argjson prNumber body assembly, composite-step failure propagation (scan failure skips oidc/submit; the always() report step fails closed), and no ${{ github.event.* }} interpolated into any run: block (all untrusted input routed via env + jq over GITHUB_EVENT_PATH). All YAML parses; the validate-action structural checks pass.

No remaining high-confidence issues.

Verdict: PASS (draft — placeholders for CLI version, service URL/audience, and the pinned action SHA remain, as expected pre-GA).

🤖 Generated with Claude Code

@Crash0v3rrid3 Crash0v3rrid3 marked this pull request as ready for review July 10, 2026 05:00
Crash0v3rrid3 and others added 2 commits July 10, 2026 10:45
- example workflow + README now reference @v1 (with a note to SHA-pin for
  production) instead of the @<commit-sha> placeholder
- add .github/workflows/ci.yml: PR gate validating action.yml structure +
  YAML well-formedness (the repo previously had no PR checks)

CLI version pin (action.yml) stays a documented release-time value — no
pr-scan CLI version is published to npm yet; the release step pins the real
version. GitHub App id / service URL remain provisioning placeholders.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-URL comments

@browserstack/accessibility-devtools 1.5.0 is published (npmjs `latest`) and
includes PR-scan mode, so pin CLI_VERSION 1.3.0 -> 1.5.0. The OIDC audience
(browserstack-accessibility-devtools) already matches the deployed
ci-review-service verifier config, and SERVICE_URL is the real global ingress;
drop the stale PLACEHOLDER comments accordingly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread examples/browserstack-a11y.yml Outdated
Comment thread README.md
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md Outdated
Comment thread README.md
Comment thread README.md Outdated
Crash0v3rrid3 and others added 2 commits July 13, 2026 11:10
- Title -> "BrowserStack Accessibility DevTools for GitHub"; workflow name ->
  "BrowserStack Accessibility DevTools".
- Learn-more link -> /docs/accessibility-dev-tools/features/remediate-github.
- Drop the internal-sounding "BrowserStack runs no AI model…what works today"
  copy from the intro bullet.
- Remove the Service-Account "no user license…Enterprise plan" aside.
- Inline the full ready-to-copy workflow in Quick start; drop the separate
  examples/ link.
- Agent hand-off: replace the "what works today" internal note with a generic
  preview disclaimer, and add the bot allow-list as a prerequisite.
- Remove Privacy & security and License sections.
- Support -> single https://www.browserstack.com/support link.
- examples/: use @main (always latest) instead of @v1; match workflow name.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tion --head')

The action called a nonexistent `scan` subcommand with --head/--changed-files/
--output and then minted OIDC + POSTed itself. Published CLI 1.5.0 does the whole
PR-scan via `--pr-scan`: resolves the base..head diff locally (git), gets
linter-server to normalize + HMAC-sign, mints the Actions OIDC token, and submits
to the ci-review-service (POST <ci-service-url>/v1/findings) which posts as the App.

- scan step -> `--pr-scan --include '**/*' --base <sha> --head-sha <sha> --repo
  <owner/name> --pr-number <n> --ci-service-url https://devtools-a11y-linter.browserstack.com`
  (--head-sha, not --head; drop --changed-files/--output/--format — the CLI derives
  the diff and submits itself). fail-on-severity=none -> --non-strict.
- Remove the now-redundant `oidc` and `submit` steps (the CLI does both).
- `report` gates on the CLI exit code (0 pass; 1/3/4/other fail).

Known 1.5.0 limits (documented inline): --pr-scan does not forward the comment/
inline/sarif/remediation options or return per-severity counts / comment URL — the
ci-review-service applies defaults; those outputs need a CLI enhancement.

NOTE: --ci-service-url uses the shared host, so POST /v1/findings must be routed to
ci-review-service on devtools-a11y-linter.browserstack.com (currently only /github
is routed) for the submit to land.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@aveek-bs

Copy link
Copy Markdown
  1. Overall the document seems to guide users to @/mention AccessibilityDevtools manually - rather we want to guide users to setup the automatic violation detection flow (with manual commenting as the secondary flow). Please see how you can rework the Prerequisites , Quick start and How it works
  2. comment-mode input - can we have a more intuitive name for this?
  3. inline-suggestions and sarif inputs - The readme should emphasize that the latter being true is a pre-requisite for enabling the former.

@aveek-bs

Copy link
Copy Markdown

Also, the output message formatting should follow the same format as the VS Code formatting

(label): <description>

<how_to_fix>

Crash0v3rrid3 and others added 2 commits July 14, 2026 11:49
…+ severity gate

Addresses review feedback (inputs must be real, not for show; guide users to the
automatic detection flow; VS Code output format) and the flag audit.

action.yml:
- Automatic flow (primary): trigger on pull_request opened/synchronize/reopened/
  ready_for_review (actor = PR author, no commenter gate). Manual @mention stays
  as the secondary flow (with the commenter permission pre-check).
- Trim inputs to the ones that do something: username, access-key,
  fail-on-severity. Removed comment / check-gate / comment-mode /
  inline-suggestions / sarif (now always-on server defaults) and
  remediation / ai-agent (deferred preview). check-gate is folded into
  fail-on-severity (none = never fail).
- Real gate + outputs: parse the CLI's `[deva11y-result]` line into
  error-count / warning-count / findings-count / comment-url, and fail the run
  per fail-on-severity (a real `warning` gate, and `error` actually fails now —
  previously --pr-scan always exited 0 so the check never failed). Operational
  errors (FUP/permission/error) always fail. Dropped the sarif-file output
  (SARIF is uploaded server-side; there is no local file).

README + example:
- Automatic-first quick start (on: pull_request + issue_comment), @main (no v1 pin).
- Inputs/outputs tables reflect reality; describe what actually gets posted
  (sticky comment, inline comments in VS Code format, Check Run, SARIF).
- Inline are review comments, not one-click suggestion blocks (findings carry
  how-to-fix text, not literal replacement code) — corrected the earlier claim.
- Single support link; removed the internal-note remediation section.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nt input

Per review: keep remediation with a generic disclaimer (rather than dropping it).

- action.yml: `ai-agent` input (optional, bare name); a validate-agent step
  (charset ^[A-Za-z0-9-]+(\[bot\])?$, fail fast); passes `--ai-agent` to the CLI
  only when set. Single input — presence enables the hand-off (no separate
  `remediation` boolean).
- README: ai-agent row + a short "AI remediation hand-off (preview)" section with
  the generic disclaimer (supported if your agent accepts bot/App triggers).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Crash0v3rrid3

Copy link
Copy Markdown
Collaborator Author

Thanks @aveek-bs — both of your points are addressed in the latest push:

  1. Automatic detection flow (primary), manual @mention secondary. The example workflow and README now lead with on: pull_request (opened/synchronize/reopened/ready_for_review) as the primary trigger; commenting @AccessibilityDevTools is documented as the on-demand secondary path. On the automatic path the action resolves the PR author as the actor (no commenter gate); the server-side fail-closed permission gate still applies before anything is posted.

  2. VS Code output format. Findings now render as (rule): <description> followed by the how-to-fix text — matching the VS Code extension — in both the summary comment and the inline comments (implemented in ci-review-service render.ts, browserstack-a11y-linter#421).

While here, from a full audit of the inputs (your "are we supporting all of these?" note): the surface is now trimmed to the inputs that actually do something — username, access-key, fail-on-severity, and optional ai-agent. Posting a summary comment, inline comments, a Check Run, and SARIF upload are always-on server defaults. And fail-on-severity now genuinely gates the run (previously --pr-scan always exited 0, so the check never failed regardless of the setting). SARIF is now actually generated + uploaded, and inline comments are deduped across re-runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants