Skip to content

shellpattern: avoid exponential backtracking for runs of * wildcards, see #2624 - #10174

Open
ThomasWaldmann wants to merge 1 commit into
borgbackup:masterfrom
ThomasWaldmann:patterns-redos-2624
Open

shellpattern: avoid exponential backtracking for runs of * wildcards, see #2624#10174
ThomasWaldmann wants to merge 1 commit into
borgbackup:masterfrom
ThomasWaldmann:patterns-redos-2624

Conversation

@ThomasWaldmann

Copy link
Copy Markdown
Member

Addresses the sh: part of #2624 (pattern evaluation is easily DoS'd).

Problem

sh: patterns were translated 1:1 into SRE regexes ([^/]* per *), so a handful of wildcards is enough for exponential matching time: input/a*a*a*a*a*b takes ~2.4 s to not match a name of 200 as, 10 stars never finish (the issue's reproducer). fm: patterns do not have this problem anymore since Python 3.11: fnmatch.translate() emits atomic groups for * fixed pairs (bpo-40480).

Change

shellpattern.translate() now does the same for sh: patterns: a * FIXED * sequence is emitted as (?>[^/]*?FIXED)[^/]* — FIXED is matched at its leftmost occurrence and the regex engine cannot backtrack into it. This is equivalent to the old regex as long as both neighbours are plain * (also when FIXED contains /): if a match exists using a later occurrence of FIXED, one also exists using the leftmost one, because the left * just absorbs less and the right * absorbs the (slash-free) difference. It is not equivalent when the right neighbour is **/ (*a**/b must match aab) or when FIXED is a {,} alternatives group (*{a,ab}c must match abc), so the atomic group is only used between plain stars. Adjacent * and adjacent **/ are collapsed, they are equivalent to a single one.

* FIXED at the end of a pattern is left as is (linear anyway), **/ chains stay polynomial and {a*,a*}-style chains stay exponential — those and re: patterns are still covered by the warning in borg help patterns, which is updated accordingly.

Numbers

  • input/ + a* × 50 + b vs. 200 as: never finished → 0 ms (new test test_no_exponential_backtracking).
  • 60 000 random patterns (ab/*?{,}[]!\) × random paths: identical results with old and new translate().
  • Existing shellpattern/patterns/archiver pattern tests unchanged and passing.

… see borgbackup#2624

sh: patterns were translated 1:1 into SRE regexes ("[^/]*" per "*"), so
patterns with a handful of wildcards like "input/a*a*a*a*a*a*b" took
exponential time to not match - the issue's reproducer never finished.
fm: patterns already do not have this problem since Python 3.11, because
fnmatch.translate() emits atomic groups.

Do the same for sh: patterns: "* FIXED *" is now emitted as
"(?>[^/]*?FIXED)[^/]*", i.e. FIXED is matched at its leftmost occurrence
and the regex engine can not backtrack into it. This is equivalent as long
as both neighbours are plain "*" (even if FIXED contains path separators),
but not if the right neighbour is "**/" or FIXED is a "{,}" group, so the
atomic group is only used for plain-star neighbours. Adjacent "*" and
adjacent "**/" are collapsed, they are equivalent to a single one.

"input/" + "a*" * 50 + "b" against 200 "a"s: never finished -> 0 ms.
60000 random patterns/paths give identical results with old and new
translate().

Also update the note in "borg help patterns": re: patterns (and sh:
patterns with many "**/" or "{}" alternatives) can still be slow.
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.05%. Comparing base (0c34d24) to head (10fe875).
⚠️ Report is 5 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #10174      +/-   ##
==========================================
- Coverage   87.05%   87.05%   -0.01%     
==========================================
  Files         101      101              
  Lines       18147    18170      +23     
  Branches     2782     2788       +6     
==========================================
+ Hits        15797    15817      +20     
- Misses       1642     1645       +3     
  Partials      708      708              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant