test: add StrykerJS mutation testing foundation#847
Conversation
Sets up mutation testing to measure how effectively the test suite catches bugs, scoped to util/ as the initial baseline (issue lnp2pBot#760). - Add @stryker-mutator core + mocha-runner + typescript-checker - stryker.config.json: mocha runner, perTest coverage, non-blocking (break: null), mutate util/** - npm scripts: mutation-test, mutation-test:util, build:test - Non-blocking CI workflow (weekly + manual) on node:20-bookworm that uploads the HTML report as an artifact - gitignore Stryker outputs The build:test script (full tsc + test tsc) is used as Stryker's build command so the sandbox build is complete: tsconfig.test.json alone does not compile dynamically required files (e.g. bot/modules/block/commands.ts), which only works in CI because it runs a full tsc beforehand. Baseline on util/: 57.69% mutation score on covered code in util/index.ts (the only tested util file); the rest of util/ has no tests yet. Part of lnp2pBot#760
WalkthroughAdds Stryker mutation-testing setup for the util scope, including npm scripts, configuration, ignore rules, and a GitHub Actions workflow that runs weekly or manually and uploads the mutation report. ChangesMutation testing setup
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/mutation-testing.yaml:
- Around line 26-40: The workflow still uses mutable GitHub Action tags in the
mutation-testing job, so update the actions referenced by actions/checkout and
actions/upload-artifact to their full 40-character commit SHAs and keep the
original version as an inline comment for readability. Make this change in the
mutation-testing workflow alongside the existing run steps, preserving the same
behavior while removing tag-based supply-chain risk.
In `@stryker.config.json`:
- Around line 1-19: Stryker is not invoking the installed TypeScript checker
because the configuration lacks the explicit checker activation. Update the
Stryker config to enable the TypeScript checker via the checkers setting in the
existing JSON alongside the current testRunner, mutate, and thresholds options.
Keep the rest of the StrykerJS configuration unchanged so mutation testing still
runs with type checking enabled.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: a33067e2-79d9-478e-940f-d9d29f1cb9c0
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (4)
.github/workflows/mutation-testing.yaml.gitignorepackage.jsonstryker.config.json
| - uses: actions/checkout@v4 | ||
|
|
||
| - run: npm ci | ||
|
|
||
| - name: Run mutation testing on util/ | ||
| run: npm run mutation-test:util | ||
| env: | ||
| NODE_ENV: test | ||
|
|
||
| - name: Upload mutation report | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: mutation-report | ||
| path: reports/mutation/ |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major
Pin both GitHub Actions to full SHAs.
actions/checkout@v4 and actions/upload-artifact@v4 use mutable tags. Pin them to their full 40-character commit SHAs to prevent supply-chain attacks where tags are retargeted to malicious code. Include the tag as a comment for readability.
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
...
- uses: actions/upload-artifact@66337390251171e342b772552734d18b63d645f9 # v4Use Dependabot or Renovate to automate SHA updates when new releases are available.
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 26-26: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 26-26: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 37-37: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/mutation-testing.yaml around lines 26 - 40, The workflow
still uses mutable GitHub Action tags in the mutation-testing job, so update the
actions referenced by actions/checkout and actions/upload-artifact to their full
40-character commit SHAs and keep the original version as an inline comment for
readability. Make this change in the mutation-testing workflow alongside the
existing run steps, preserving the same behavior while removing tag-based
supply-chain risk.
| { | ||
| "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json", | ||
| "packageManager": "npm", | ||
| "testRunner": "mocha", | ||
| "buildCommand": "npm run build:test", | ||
| "coverageAnalysis": "perTest", | ||
| "mutate": ["util/**/*.ts", "!**/*.spec.ts"], | ||
| "mochaOptions": { | ||
| "spec": ["dist/tests/**/*.spec.js"] | ||
| }, | ||
| "thresholds": { | ||
| "high": 80, | ||
| "low": 60, | ||
| "break": null | ||
| }, | ||
| "reporters": ["html", "clear-text", "progress"], | ||
| "timeoutMS": 60000, | ||
| "concurrency": 2 | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major
🧩 Analysis chain
🌐 Web query:
According to the official StrykerJS docs for @stryker-mutator/typescript-checker, what config key enables the checker in stryker.config.json?
💡 Result:
To enable the TypeScript checker in StrykerJS, you must add "typescript" to the checkers configuration array in your stryker.config.json file [1][2][3][4]. You can achieve this by adding the following key to your configuration [1][2]: "checkers": ["typescript"] Additionally, while not strictly required to enable the plugin, you may also need to specify the path to your TypeScript configuration file using the tsconfigFile key if it is not the default "tsconfig.json" [1][2][3][5]. Example configuration: { "checkers": ["typescript"], "tsconfigFile": "tsconfig.json" } Note that the @stryker-mutator/typescript-checker package must be installed in your project [2][5]. By default, Stryker automatically loads plugins that start with "@stryker-mutator/", so you generally do not need to explicitly list it in a "plugins" section unless you have restricted the automatic loading [1][2][5].
Citations:
- 1: https://github.com/stryker-mutator/stryker-js/blob/master/docs/typescript-checker.md
- 2: https://stryker-mutator.io/docs/stryker-js/typescript-checker/
- 3: https://stryker-mutator.io/docs/stryker-js/configuration/
- 4: https://stryker-mutator.io/blog/announcing-faster-typescript-checking/
- 5: https://github.com/stryker-mutator/stryker-js/blob/master/docs/troubleshooting.md
Enable the TypeScript checker explicitly.
The @stryker-mutator/typescript-checker is installed but not activated. StrykerJS requires the "checkers": ["typescript"] property to be defined in stryker.config.json to invoke type-checking during mutation. Without this, type-invalid mutants will proceed to execution, potentially skewing mutation score results.
Suggested change
{
"$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
"packageManager": "npm",
"testRunner": "mocha",
+ "checkers": ["typescript"],
"buildCommand": "npm run build:test",
"coverageAnalysis": "perTest",
"mutate": ["util/**/*.ts", "!**/*.spec.ts"],📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| { | |
| "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json", | |
| "packageManager": "npm", | |
| "testRunner": "mocha", | |
| "buildCommand": "npm run build:test", | |
| "coverageAnalysis": "perTest", | |
| "mutate": ["util/**/*.ts", "!**/*.spec.ts"], | |
| "mochaOptions": { | |
| "spec": ["dist/tests/**/*.spec.js"] | |
| }, | |
| "thresholds": { | |
| "high": 80, | |
| "low": 60, | |
| "break": null | |
| }, | |
| "reporters": ["html", "clear-text", "progress"], | |
| "timeoutMS": 60000, | |
| "concurrency": 2 | |
| } | |
| { | |
| "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json", | |
| "packageManager": "npm", | |
| "testRunner": "mocha", | |
| "checkers": ["typescript"], | |
| "buildCommand": "npm run build:test", | |
| "coverageAnalysis": "perTest", | |
| "mutate": ["util/**/*.ts", "!**/*.spec.ts"], | |
| "mochaOptions": { | |
| "spec": ["dist/tests/**/*.spec.js"] | |
| }, | |
| "thresholds": { | |
| "high": 80, | |
| "low": 60, | |
| "break": null | |
| }, | |
| "reporters": ["html", "clear-text", "progress"], | |
| "timeoutMS": 60000, | |
| "concurrency": 2 | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stryker.config.json` around lines 1 - 19, Stryker is not invoking the
installed TypeScript checker because the configuration lacks the explicit
checker activation. Update the Stryker config to enable the TypeScript checker
via the checkers setting in the existing JSON alongside the current testRunner,
mutate, and thresholds options. Keep the rest of the StrykerJS configuration
unchanged so mutation testing still runs with type checking enabled.
1 participant
Part of #760. First, foundational PR: set up StrykerJS and establish a baseline on
util/. Writing tests for surviving mutants and raising the threshold will follow as separate focused PRs.What this adds
@stryker-mutator/core+mocha-runner+typescript-checkerstryker.config.json— mocha runner,perTestcoverage, non-blocking (break: null), scoped toutil/**mutation-test,mutation-test:util,build:testworkflow_dispatch) onnode:20-bookwormthat uploads the HTML report as an artifactBaseline on
util/util/index.tsutil/This is exactly the signal mutation testing gives: it shows where a bug could be introduced and slip past the suite.
One integration note
Stryker builds in a clean sandbox, which surfaced that
tsconfig.test.json(includetests/**/*) does not compile dynamicallyrequire()d files such asbot/modules/block/commands.ts— it only works normally because CI runs a fulltscfirst. Rather than touch that code, I use abuild:testscript (fulltsc+ testtsc) as Stryker's build command so the sandbox build is complete.Validation
tsc, lint, prettier cleanutil/Summary by CodeRabbit