Sentry setup and Heroku performance debugging

[mark-kraemer]

Why

These two skills help with production setup and debugging issues

What changed

• Added a skill for ensuring our sentry setup is correct
• Added a skill for reviewing performance issues that are not captured in sentry

[Copilot]

Pull request overview

Adds two new reusable agent skills to this repository’s skill library: one to audit/provision Sentry org-side setup to a RoleModel standard, and one to triage Heroku performance/availability issues that don’t surface cleanly in Sentry.

Changes:

• Introduces sentry-standard skill with an audit/fix workflow plus reusable dashboard + alert templates.
• Introduces heroku-triage skill with diagnose/apply/verify/review modes and supporting heuristics/signal-gathering references.
• Adds a standalone Papertrail search helper script to support time-series log analysis without loading raw logs into context.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
skills/sentry-standard/SKILL.md	New skill defining the Sentry “standard” audit + provisioning workflow.
skills/sentry-standard/references/dashboard_template.json	Template payload for the shared org-wide “RoleModel SHM” dashboard.
skills/sentry-standard/references/alert_templates.md	CLI templates for required Sentry alert rules and monitor validation guidance.
skills/heroku-triage/SKILL.md	New skill defining Heroku incident triage + tuning workflow across multiple modes.
skills/heroku-triage/scripts/papertrail-search.sh	New script for paginated Papertrail event retrieval and deduped TSV output.
skills/heroku-triage/references/signals.md	Reference recipes for gathering/aggregating signals (Heroku, Papertrail, Sentry, PG).
skills/heroku-triage/references/linear-issue.md	Template + baseline schema for Linear tracking issues used by apply/verify modes.
skills/heroku-triage/references/heuristics.md	Decision trees, sizing rules, and verification criteria/soak periods.

Comments suppressed due to low confidence (4)

skills/heroku-triage/references/signals.md:74

• This example uses scripts/papertrail-search.sh without the skills/heroku-triage/ prefix, which will fail unless the working directory is skills/heroku-triage/. Use the repo-root path consistently like the earlier example in this doc.

scripts/papertrail-search.sh --app $APP --query '"sample#memory_total" "source=web.1"' \

skills/heroku-triage/references/signals.md:82

• This example uses scripts/papertrail-search.sh without the skills/heroku-triage/ prefix, which will fail unless the working directory is skills/heroku-triage/. Use the repo-root path consistently like the earlier example in this doc.

scripts/papertrail-search.sh --app $APP --query '"heroku/router" "service="' \

skills/heroku-triage/references/signals.md:89

• This example uses scripts/papertrail-search.sh without the skills/heroku-triage/ prefix, which will fail unless the working directory is skills/heroku-triage/. Use the repo-root path consistently like the earlier example in this doc.

scripts/papertrail-search.sh --app $APP --query '"duration="' --from <epoch> --to <epoch+3600> --out app.tsv

skills/heroku-triage/references/signals.md:98

• This example uses scripts/papertrail-search.sh without the skills/heroku-triage/ prefix, which will fail unless the working directory is skills/heroku-triage/. Use the repo-root path consistently like the earlier example in this doc.

scripts/papertrail-search.sh --app $APP --query 'web.2' --from <step_epoch-90> --to <step_epoch+90> --out ctx.tsv

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (6)

skills/sentry-standard/references/dashboard_template.json:5

• dashboard_template.json includes a top-level _comment key and pins environment to production, but the skill instructs posting this JSON to the Sentry dashboards API and describes environment selection as a view-time choice. The _comment field is likely to be rejected by the API, and pinning the environment prevents re-scoping at view time. Consider removing _comment entirely and leaving environment empty so the dashboard selector controls it.

{
  "_comment": "RoleModel shared 'RoleModel SHM' dashboard template — ONE org-wide dashboard for all apps, not per-app. No substitution needed: post as-is after removing this _comment key. The empty 'projects' array means 'All Projects' so the dashboard's project selector re-scopes every widget per app at view time; do the same for 'environment' (production is the default filter, chosen at view time). Create it only once per org (audit step 3a). Post: sentry api organizations/rolemodel-software/dashboards/ -X POST --input <file>. Widget set is derived from what SHM report writers actually cite (Almanac product health report summaries, Jan-Jul 2026).",
  "title": "RoleModel SHM",
  "projects": [],
  "environment": ["production"],

skills/sentry-standard/references/alert_templates.md:65

• This section says cron/uptime failures create issues with category cron/uptime, but sentry-standard/SKILL.md notes those legacy categories match nothing and the correct category is outage. Keeping this mismatch will cause agents to build or validate the wrong issue queries.

## 4 & 5. Cron and uptime failures

Cron monitor failures and uptime downtime create **issues** (category `cron` /
`uptime`), so they route through rule 1 automatically. The skill's job is
validation, not new rules:

skills/heroku-triage/references/signals.md:93

• These examples invoke scripts/papertrail-search.sh, but in this repository the script is located at skills/heroku-triage/scripts/papertrail-search.sh. Using the wrong path will cause the cookbook commands to fail when run from the repo root.

```bash
# R14/R15 timeline → hourly buckets: which days, what cadence, which dynos?
scripts/papertrail-search.sh --app $APP --query '"Error R1"' --from $(date -v-7d +%s) --out r1x.tsv
cut -f1 r1x.tsv | cut -c1-13 | sort | uniq -c                    # events per hour
grep -oE 'web\.[0-9]+|worker\.[0-9]+' r1x.tsv | sort | uniq -c   # which process/dyno

# Memory curve for one dyno (needs log-runtime-metrics, §4) → 10-min buckets, max MB
scripts/papertrail-search.sh --app $APP --query '"sample#memory_total" "source=web.1"' \
  --from $(date -v-48H +%s) --out mem.tsv

skills/heroku-triage/references/signals.md:108

• These examples invoke scripts/papertrail-search.sh, but in this repository the script is located at skills/heroku-triage/scripts/papertrail-search.sh. Using the wrong path will cause the cookbook commands to fail when run from the repo root.

# Router latency percentiles for a one-hour window (status/service from heroku router lines)
scripts/papertrail-search.sh --app $APP --query '"heroku/router" "service="' \
  --from <epoch> --to <epoch+3600> --out router.tsv
grep -oE 'service=[0-9]+' router.tsv | cut -d= -f2 | sort -n | \
  awk '{ v[NR] = $1 } END { if (!NR) exit;
    printf "n=%d p50=%d p95=%d p99=%d max=%d\n", NR, v[int(NR*.5)], v[int(NR*.95)], v[int(NR*.99)], v[NR] }'

# App-reported duration for the SAME window (lograge lines) — the queue-time discriminator
scripts/papertrail-search.sh --app $APP --query '"duration="' --from <epoch> --to <epoch+3600> --out app.tsv
grep -oE 'duration=[0-9.]+' app.tsv | cut -d= -f2 | sort -n | \

skills/heroku-triage/references/signals.md:116

• This example invokes scripts/papertrail-search.sh, but in this repository the script is located at skills/heroku-triage/scripts/papertrail-search.sh. Using the wrong path will cause the cookbook commands to fail when run from the repo root.

# What ran just before a memory step / R15? (±90s around a timestamp, all sources)
scripts/papertrail-search.sh --app $APP --query 'web.2' --from <step_epoch-90> --to <step_epoch+90> --out ctx.tsv

skills/heroku-triage/SKILL.md:159

• The reference map still points to scripts/papertrail-search.sh, but the script in this repo is skills/heroku-triage/scripts/papertrail-search.sh. This inconsistency makes it harder to locate/run the script when following the docs from the repo root.

- `references/signals.md` — every data-gathering recipe: heroku CLI census, Papertrail
  API (via `scripts/papertrail-search.sh`), log-runtime-metrics, Sentry MCP intents,
  Postgres, in-repo config. Read at the start of diagnose/review.