Skip to content

Commit 791a529

Browse files
ndemiancclaude
andcommitted
fix(tools): harden the release-notes drafter (PR #38 review)
All five points were real. Two of them made the tool quietly report LESS than the truth, which is the failure mode that matters most for something whose whole job is producing accurate facts. 1. prevTag was interpolated into a shell string. A tag name is repo-controlled but still runtime-discovered input, so this is now defended twice: all git calls go through spawnSync with an argv array (a shell cannot be quoted out of one), and the base tag must match ^vX.Y.Z$ before use. Shape matters independently of injection — a stray v0.9.2-rc1 sorts into the 'v*' glob and would silently produce the wrong range, wrong commit list and wrong compare link. Verified: v0.9.2-rc1, v-wip and "v0.9.2; rm -rf /" are all rejected. 2. PRs were only detected from merge commits, so on a squash-merging repo the list would read "(none detected)" while every subject carried "(#123)". Both shapes are now collected. 3. Suites were keyed by basename, so two extensions with the same test filename would produce an ambiguous "biggest suites" list. Keyed by relative path now. 4. The coverage line printed "N cases in total" while suites whose summary could not be parsed silently contributed 0 — authoritative-sounding and UNDER- reporting. This one bit immediately: 5 of 24 suites do not print a count, so "273 cases in total" was wrong. It now says how many suites were counted and names the ones that were not. 5. The fix/perf heading also carried reverts; renamed to say so. Also fixes a bug I introduced while making #2: prNumbers became a Set, but the renderer still read .length, which is undefined on a Set — so the PR list emptied itself silently. Caught by re-running against the real range (expected #33-36, got "(none detected)"). Materialised to a sorted array. Verified end to end against v0.9.1..HEAD: PRs #33-36 detected, coverage line now honest about the 5 unparsed suites, and all guardrails still fire. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent c56670c commit 791a529

1 file changed

Lines changed: 48 additions & 14 deletions

File tree

scripts/draft-release-notes.mjs

Lines changed: 48 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -25,14 +25,26 @@
2525
* the draft. A tool that silently drops commits is worse than no tool — you cannot review an
2626
* omission you never see. Delete that section once you have checked it.
2727
*--------------------------------------------------------------------------------------------*/
28-
import { execSync, spawnSync } from 'node:child_process';
28+
import { spawnSync } from 'node:child_process';
2929
import { readdirSync, writeFileSync, existsSync } from 'node:fs';
3030
import { join } from 'node:path';
3131

3232
const REPO = process.cwd();
33-
const sh = (cmd) => execSync(cmd, { cwd: REPO, encoding: 'utf8' }).trim();
3433
const die = (msg) => { console.error('draft-release-notes: ' + msg); process.exit(1); };
3534

35+
// git via argv, never a shell string. A tag name is repo-controlled but still UNTRUSTED input here:
36+
// it is discovered at runtime and then used to build a revision range, so passing it through a shell
37+
// would make a tag containing metacharacters an injection. spawnSync with an array cannot be quoted
38+
// out of. (RELEASE_TAG below is the second half of that defence: shape, not just escaping.)
39+
const git = (...args) => {
40+
const r = spawnSync('git', args, { cwd: REPO, encoding: 'utf8' });
41+
if (r.status !== 0) { die(`git ${args.join(' ')} failed: ${(r.stderr || '').trim()}`); }
42+
return r.stdout.trim();
43+
};
44+
45+
// The ONLY tag shape this tool will measure a release against.
46+
const RELEASE_TAG = /^v\d+\.\d+\.\d+$/;
47+
3648
// ---- arguments ---------------------------------------------------------------------------------
3749

3850
const args = process.argv.slice(2);
@@ -42,24 +54,27 @@ if (!version) { die('usage: node scripts/draft-release-notes.mjs <version> [--wr
4254
if (!/^\d+\.\d+\.\d+$/.test(version)) { die(`"${version}" is not a bare semver (expected e.g. 0.9.2, no leading v)`); }
4355

4456
const tag = 'v' + version;
45-
if (sh('git tag --list ' + tag)) {
57+
if (git('tag', '--list', tag)) {
4658
die(`${tag} already exists. Notes are written BEFORE tagging, so the tag contains them.`);
4759
}
4860

4961
// ---- the range ---------------------------------------------------------------------------------
5062

51-
// Newest existing release tag, which is what this release is measured against.
52-
const prevTag = sh("git tag --list 'v*' --sort=-v:refname").split('\n').filter(Boolean)[0];
53-
if (!prevTag) { die('no previous v* tag found — cannot compute a range or a compare link'); }
63+
// Newest existing RELEASE tag — filtered by shape, not merely by the 'v*' glob. A stray tag like
64+
// v0.9.2-rc1 or v-wip sorts into that glob and would silently produce the wrong range (and therefore
65+
// the wrong compare link and the wrong commit list), which is a subtler failure than any injection.
66+
const prevTag = git('tag', '--list', 'v*', '--sort=-v:refname')
67+
.split('\n').map((t) => t.trim()).filter((t) => RELEASE_TAG.test(t))[0];
68+
if (!prevTag) { die('no previous vX.Y.Z release tag found — cannot compute a range or a compare link'); }
5469

55-
const dirty = sh('git status --porcelain').split('\n').filter((l) => l && !l.startsWith('??'));
70+
const dirty = git('status', '--porcelain').split('\n').filter((l) => l && !l.startsWith('??'));
5671
const warnings = [];
5772
if (dirty.length) {
5873
warnings.push(`working tree has ${dirty.length} uncommitted change(s) — the notes may describe code that is not in the tag`);
5974
}
6075

6176
// %x1f separates fields, %x1e separates records: commit subjects contain almost anything else.
62-
const raw = sh(`git log --format=%H%x1f%s%x1f%an%x1e ${prevTag}..HEAD`);
77+
const raw = git('log', '--format=%H%x1f%s%x1f%an%x1e', `${prevTag}..HEAD`);
6378
const commits = raw.split('\x1e').map((r) => r.trim()).filter(Boolean).map((r) => {
6479
const [hash, subject, author] = r.split('\x1f');
6580
return { hash: hash.slice(0, 7), subject, author };
@@ -72,18 +87,27 @@ if (!commits.length) { die(`no commits between ${prevTag} and HEAD — nothing t
7287
// Merge commits are dropped (their PR title is already carried by the squashed/branch commits), but
7388
// their PR numbers are collected so the draft can cite them.
7489

75-
const prNumbers = [];
90+
// PR numbers arrive in one of two shapes depending on the merge strategy, and a tool that only knows
91+
// one of them silently reports "no PRs" on a repo that squash-merges. Collect both:
92+
// merge commit -> "Merge pull request #37 from ..."
93+
// squash commit -> "feat(ai): auto-open the browser (#35)"
94+
const prNumbers = new Set();
7695
const isMerge = (c) => {
7796
const m = /^Merge pull request #(\d+)/.exec(c.subject);
78-
if (m) { prNumbers.push(m[1]); return true; }
97+
if (m) { prNumbers.add(m[1]); return true; }
7998
return /^Merge branch /.test(c.subject);
8099
};
100+
const notePrInSubject = (c) => {
101+
const m = /\(#(\d+)\)\s*$/.exec(c.subject);
102+
if (m) { prNumbers.add(m[1]); }
103+
};
81104

82105
const typeOf = (subject) => (/^(\w+)(\([^)]*\))?!?:/.exec(subject) || [])[1] || 'other';
83106
const USER_FACING = new Set(['feat', 'fix', 'perf', 'revert']);
84107
const INTERNAL = new Set(['ci', 'build', 'chore', 'test', 'docs', 'refactor', 'style']);
85108

86109
const kept = commits.filter((c) => !isMerge(c));
110+
kept.forEach(notePrInSubject); // squash-merge repos carry the PR in the subject, not a merge commit
87111
const features = kept.filter((c) => typeOf(c.subject) === 'feat');
88112
const fixes = kept.filter((c) => ['fix', 'perf', 'revert'].includes(typeOf(c.subject)));
89113
const excluded = kept.filter((c) => INTERNAL.has(typeOf(c.subject)));
@@ -107,7 +131,9 @@ function measureSuites() {
107131
const run = spawnSync('node', [rel], { cwd: REPO, encoding: 'utf8' });
108132
if (run.status !== 0) { failed.push(rel); continue; }
109133
const m = /(\d+) tests? passed/.exec(run.stdout || '');
110-
suites.push({ file, cases: m ? Number(m[1]) : null });
134+
// Keyed by RELATIVE PATH: two extensions may legitimately both have catalog.test.js, and a
135+
// basename would make the "biggest suites" list ambiguous about which one it means.
136+
suites.push({ file: rel, cases: m ? Number(m[1]) : null });
111137
}
112138
}
113139
return { suites, failed };
@@ -118,10 +144,17 @@ if (failed.length) {
118144
die(`these suites FAIL — fix before drafting notes:\n ${failed.join('\n ')}`);
119145
}
120146
const totalCases = suites.reduce((n, s) => n + (s.cases || 0), 0);
147+
// A suite whose summary line we could not parse contributes 0, so the total would quietly UNDER-report
148+
// coverage while sounding authoritative. Say which number we actually have.
149+
const uncounted = suites.filter((s) => s.cases == null);
121150
const biggest = [...suites].sort((a, b) => (b.cases || 0) - (a.cases || 0)).slice(0, 3);
122151

123152
// ---- render -------------------------------------------------------------------------------------
124153

154+
// Set -> sorted array. (A Set has .size, not .length; reading .length silently yields undefined,
155+
// which is how the PR list quietly emptied itself the first time.)
156+
const prList = [...prNumbers].sort((a, b) => Number(a) - Number(b));
157+
125158
const bullet = (c) => `- \`${c.hash}\` ${c.subject}`;
126159
const section = (title, list) => (list.length ? `\n### ${title}\n${list.map(bullet).join('\n')}\n` : '');
127160

@@ -131,7 +164,7 @@ const draft = `# LevelCode v${version}
131164
not the biggest diff. Two features is a fine release; say so plainly. -->
132165
133166
## Highlights
134-
${section('Candidates — feat (write these up, or move them down / delete)', features)}${section('Candidates — fix/perf (usually "Under the hood", unless a user hit the bug)', fixes)}
167+
${section('Candidates — feat (write these up, or move them down / delete)', features)}${section('Candidates — fix/perf/revert (usually "Under the hood", unless a user hit the bug)', fixes)}
135168
<!-- TODO For each thing you keep: say what it does, then the ONE non-obvious property a user should
136169
know (a bound, a tradeoff, a thing it deliberately will not do). That sentence is the whole
137170
value of hand-writing these. -->
@@ -142,7 +175,8 @@ ${section('Candidates — feat (write these up, or move them down / delete)', fe
142175
143176
## Test coverage
144177
145-
- **${suites.length} suites** across the bundled extensions, ${totalCases} cases in total — all green.
178+
- **${suites.length} suites** across the bundled extensions — all green.
179+
- ${totalCases} cases counted${uncounted.length ? ` across ${suites.length - uncounted.length} of them; ${uncounted.length} suite(s) did not report a count (${uncounted.map((s) => s.file).join(', ')}), so the real total is higher` : ' in total'}.
146180
${biggest.map((s) => `- \`${s.file}\`${s.cases != null ? ` (${s.cases} cases)` : ''} — <!-- TODO what does it guard? -->`).join('\n')}
147181
148182
**Full changelog:** https://github.com/levelcodeai/levelcode/compare/${prevTag}...${tag}
@@ -151,7 +185,7 @@ ${biggest.map((s) => `- \`${s.file}\`${s.cases != null ? ` (${s.cases} cases)` :
151185
EVERYTHING BELOW IS SCAFFOLDING — delete it before committing.
152186
153187
Range: ${prevTag}..HEAD (${commits.length} commits, ${kept.length} after dropping merges)
154-
PRs merged: ${prNumbers.length ? prNumbers.map((n) => '#' + n).join(', ') : '(none detected)'}
188+
PRs merged: ${prList.length ? prList.map((n) => '#' + n).join(', ') : '(none detected)'}
155189
${warnings.length ? '\n WARNINGS:\n' + warnings.map((w) => ' - ' + w).join('\n') + '\n' : ''}
156190
EXCLUDED as internal — check this list; anything user-visible in here belongs above:
157191
${excluded.length ? excluded.map((c) => ` ${c.hash} ${c.subject}`).join('\n') : ' (none)'}

0 commit comments

Comments
 (0)