VxSuite QA Automation Tool

Automates QA testing for VxSuite elections by orchestrating VxAdmin and VxScan apps with mock hardware.

Features

• Clones and sets up VxSuite repository automatically
• Generates test ballots with various vote patterns (blank, overvote, valid votes)
• Automates VxAdmin for election configuration
• Automates VxScan for ballot scanning
• Captures screenshots at key steps
• Generates HTML reports for human review

Installation

# Clone and install
cd vx-qa
pnpm install

# Install Playwright browsers
npx playwright install chromium

Usage

# Create a sample config
pnpm start init -o my-config.json

# Edit the config file, then run
pnpm start run --config my-config.json

CLI Options

pnpm start run [options]

Options:
  -c, --config <path>          Path to configuration file
  -o, --output <dir>           Override output directory
  --vxsuite-version <version>  Override VxSuite version (e.g. v4.0, v4.1)
  -e, --election <path>        Override election source path
  --headless                   Run browser in headless mode (default)
  --no-headless                Run browser in headed mode for debugging

Configuration

Example vx-qa-config.json:

{
  "vxsuite": {
    "repoPath": "~/.vx-qa/vxsuite",
    "version": "v4.0"
  },
  "election": {
    "source": "./election-package-and-ballots.zip"
  },
  "output": {
    "directory": "./qa-output"
  }
}

vxsuite.version selects which VxSuite release to run against. Values match VxSuite's own SoftwareVersion identifiers:

version	git ref	patch
"v4.0"	v4.0.7	vxsuite-v4.0.patch
"v4.1"	v4.1.0	vxsuite-v4.1.patch

The git ref and patch file are derived from the version; you don't set them directly. Older versions (≤ v4.0.4) are not supported.

tallyMode (optional, defaults to "consolidated") controls how VxAdmin tallies CVRs relative to precincts:

mode	behavior
"consolidated"	One VxAdmin session imports and tallies CVRs from every precinct together.
"per-precinct"	Each precinct gets its own configure → scan → import → tally → report → unconfigure cycle, run sequentially (e.g. NH "city" elections by ward).

Output

Each run creates a timestamped directory with an HTML report and the supporting files, including various inputs, outputs, and screenshots.

Troubleshooting

To see the browser as the workflows run, run with --no-headless. You may also consider running with the DEBUG=1 environment variable.

Requirements

• Node.js 20+
• pnpm 8+
• Git

If pnpm install returns a ERR_PNPM_INVALID_WORKSPACE_CONFIGURATION you may need to update your pnpm version. You can gloablly install the latest version of pnpm with npm install -g pnpm@latest

Development

# Build
pnpm build

# Run tests
pnpm test

CI

The project runs automated QA tests in CircleCI using test fixtures in the test-fixtures/ directory. The CI workflow:

1. Runs unit tests with pnpm test
2. Executes a full QA test run per supported VxSuite version, using the vx-qa-config-<version>.json fixtures (e.g. vx-qa-config-v4.0.json, vx-qa-config-v4.1.json). Each version clones/builds its own pinned VxSuite ref and applies that version's patch.
3. Stores output artifacts for review

Each version's fixture references an election package exported from that same VxSuite version (v4.1's package must carry the newer ballotPositions ballot model). Add configurations and packages to test-fixtures/ to expand coverage.

Architecture

src/
├── cli/                 # CLI and TUI
├── config/              # Configuration schema
├── repo/                # Git/repository management
├── apps/                # App orchestration
├── mock-hardware/       # Mock card, USB, scanner control
├── ballots/             # Ballot generation and marking
├── automation/          # Playwright browser automation
├── report/              # HTML report generation
└── utils/               # Logging, paths, processes

License

GPL-3.0