Skip to content

Add standalone OWASP cheat sheet refresh script - #952

Open
Bornunique911 wants to merge 3 commits into
OWASP:mainfrom
Bornunique911:review/issue-471-cheatsheet-refresh-script
Open

Add standalone OWASP cheat sheet refresh script#952
Bornunique911 wants to merge 3 commits into
OWASP:mainfrom
Bornunique911:review/issue-471-cheatsheet-refresh-script

Conversation

@Bornunique911

@Bornunique911 Bornunique911 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR is split out from the larger issue-471 review flow to make review smaller and more focused.

It adds a standalone script for refreshing OWASP Cheat Sheet data and normalizing cheat sheet links in the local cache.

Issue reference:

Problem Fixed

The earlier refresh-scripts review became too large because it was mixed with broader OWASP importer and follow-up work.

For this part of the work, the useful standalone contribution is:

  • rebuilding cheat sheet data into the cache
  • normalizing GitHub-style cheat sheet links to official OWASP Cheat Sheet Series URLs
  • providing that workflow as a reusable script with backup and environment setup

Solution

This PR adds a single standalone script:

  • scripts/update-cheatsheets.sh

The script:

  • prepares the local virtual environment if needed
  • installs Python dependencies if required
  • verifies the target database exists
  • creates a timestamped backup
  • reimports cheat sheet data using --cheatsheets_in
  • normalizes stored cheat sheet links from GitHub paths to official OWASP Cheat Sheet Series URLs

Tests

bash -n scripts/update-cheatsheets.sh

Reviewer Notes

This PR is intentionally narrow because it was split to reduce review size:

  • no frontend changes
  • no parser-stack expansion
  • no unrelated OWASP importer changes
  • no bundle artifacts

This PR is meant to be reviewed as a standalone operational helper.

@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 80b82503-4e18-4440-9af2-606a169483cb

📥 Commits

Reviewing files that changed from the base of the PR and between cdf6c55 and ca9da5b.

📒 Files selected for processing (1)
  • scripts/update-cheatsheets.sh

Summary by CodeRabbit

  • Chores
    • Added an automated process for updating OWASP Cheat Sheet links to official URLs.
    • Creates and verifies a timestamped database backup before applying updates.
    • Checks required configuration and database access before proceeding.
    • Provides clearer failure reporting and handles the update environment automatically for more reliable runs.

Walkthrough

This PR adds scripts/update-cheatsheets.sh. The script prepares a Python environment, backs up the SQLite database, runs cre.py, and normalizes matching OWASP Cheat Sheets links.

Changes

Cheatsheets Update Script

Layer / File(s) Summary
Runtime setup and database backup
scripts/update-cheatsheets.sh
Enables strict Bash handling, prepares the Python virtual environment, installs requirements.txt, validates the database, and creates a verified PID-suffixed backup.
Cheatsheet data update
scripts/update-cheatsheets.sh
Runs cre.py --cheatsheets_in with gap analysis and embedding generation disabled.
OWASP link normalization
scripts/update-cheatsheets.sh
Converts matching GitHub Cheat Sheet links to official .html URLs, commits the database updates, and reports the normalization count.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: northdpole, pa04rth, paoga87

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the new standalone script and its OWASP cheat sheet refresh purpose.
Description check ✅ Passed The description directly explains the script, its workflow, scope, issue references, and syntax check.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
scripts/update-cheatsheets.sh (2)

23-24: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Backup filename collision within the same second.

date +%Y%m%d%H%M%S has 1-second resolution; two runs within the same second silently overwrite each other's backup.

🛡️ Add uniqueness
-BACKUP_FILE="${DB_PATH}.$(date +%Y%m%d%H%M%S).bak"
+BACKUP_FILE="${DB_PATH}.$(date +%Y%m%d%H%M%S)_$$.bak"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/update-cheatsheets.sh` around lines 23 - 24, The backup creation in
the update-cheatsheets script can collide when run multiple times within the
same second because BACKUP_FILE is based only on date +%Y%m%d%H%M%S. Update the
BACKUP_FILE naming logic in the script so each run produces a unique filename,
for example by adding higher-resolution time, the process ID, or another unique
suffix, and keep the cp backup step using that new unique name.

14-16: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Fragile dependency-check proxy.

Using import flask success as a stand-in for "all deps installed" means new/updated packages in requirements.txt won't get installed if flask is already present in the venv from a prior run.

♻️ Simpler and more robust alternative
-if ! python -c "import flask" >/dev/null 2>&1; then
-  pip install -r "$ROOT_DIR/requirements.txt"
-fi
+pip install -q -r "$ROOT_DIR/requirements.txt"

pip install is idempotent and fast when nothing changed, so always running it avoids silently skipping updated dependencies.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/update-cheatsheets.sh` around lines 14 - 16, The dependency check in
the update-cheatsheets.sh script is too narrow because it only tests flask via
the python import gate, so updated or newly added packages in requirements.txt
can be skipped. Remove the import-based conditional around the pip install step
and always run the requirements installation in the script flow so dependency
updates are applied reliably; the relevant logic is the shell block that wraps
pip install -r "$ROOT_DIR/requirements.txt".
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@scripts/update-cheatsheets.sh`:
- Around line 23-24: The backup creation in the update-cheatsheets script can
collide when run multiple times within the same second because BACKUP_FILE is
based only on date +%Y%m%d%H%M%S. Update the BACKUP_FILE naming logic in the
script so each run produces a unique filename, for example by adding
higher-resolution time, the process ID, or another unique suffix, and keep the
cp backup step using that new unique name.
- Around line 14-16: The dependency check in the update-cheatsheets.sh script is
too narrow because it only tests flask via the python import gate, so updated or
newly added packages in requirements.txt can be skipped. Remove the import-based
conditional around the pip install step and always run the requirements
installation in the script flow so dependency updates are applied reliably; the
relevant logic is the shell block that wraps pip install -r
"$ROOT_DIR/requirements.txt".

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro

Run ID: 4a99f980-fd4a-4622-a8a0-b5532012fd97

📥 Commits

Reviewing files that changed from the base of the PR and between 0e16c2e and 716f8b2.

📒 Files selected for processing (1)
  • scripts/update-cheatsheets.sh

@Bornunique911
Bornunique911 force-pushed the review/issue-471-cheatsheet-refresh-script branch from 76283a2 to cdf6c55 Compare July 31, 2026 18:54

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/update-cheatsheets.sh`:
- Around line 8-15: Update the virtual-environment setup in
update-cheatsheets.sh to validate $VENV_DIR/bin/python rather than only the
directory, recreate the environment when that executable is missing, and use the
absolute $VENV_DIR/bin/python for Python and pip operations so system
executables cannot be selected.
- Around line 23-28: Replace the raw cp-based backup in the backup creation flow
with an existing virtualenv Python script using sqlite3.Connection.backup() to
copy DB_PATH to BACKUP_FILE, and wait for that process to complete before
reopening DB_PATH. Afterward, run PRAGMA integrity_check against BACKUP_FILE and
fail if the check is not successful, preserving the existing backup failure
handling.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yml

Review profile: CHILL

Plan: Pro Plus

Run ID: 08927877-75c1-4288-b289-949e2a2a7b1d

📥 Commits

Reviewing files that changed from the base of the PR and between 716f8b2 and cdf6c55.

📒 Files selected for processing (1)
  • scripts/update-cheatsheets.sh

Comment thread scripts/update-cheatsheets.sh Outdated
Comment thread scripts/update-cheatsheets.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant