shellpattern: avoid exponential backtracking for runs of * wildcards, see #2624 - #10174
Open
ThomasWaldmann wants to merge 1 commit into
Open
shellpattern: avoid exponential backtracking for runs of * wildcards, see #2624#10174ThomasWaldmann wants to merge 1 commit into
ThomasWaldmann wants to merge 1 commit into
Conversation
… see borgbackup#2624 sh: patterns were translated 1:1 into SRE regexes ("[^/]*" per "*"), so patterns with a handful of wildcards like "input/a*a*a*a*a*a*b" took exponential time to not match - the issue's reproducer never finished. fm: patterns already do not have this problem since Python 3.11, because fnmatch.translate() emits atomic groups. Do the same for sh: patterns: "* FIXED *" is now emitted as "(?>[^/]*?FIXED)[^/]*", i.e. FIXED is matched at its leftmost occurrence and the regex engine can not backtrack into it. This is equivalent as long as both neighbours are plain "*" (even if FIXED contains path separators), but not if the right neighbour is "**/" or FIXED is a "{,}" group, so the atomic group is only used for plain-star neighbours. Adjacent "*" and adjacent "**/" are collapsed, they are equivalent to a single one. "input/" + "a*" * 50 + "b" against 200 "a"s: never finished -> 0 ms. 60000 random patterns/paths give identical results with old and new translate(). Also update the note in "borg help patterns": re: patterns (and sh: patterns with many "**/" or "{}" alternatives) can still be slow.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10174 +/- ##
==========================================
- Coverage 87.05% 87.05% -0.01%
==========================================
Files 101 101
Lines 18147 18170 +23
Branches 2782 2788 +6
==========================================
+ Hits 15797 15817 +20
- Misses 1642 1645 +3
Partials 708 708 ☔ View full report in Codecov by Harness. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the
sh:part of #2624 (pattern evaluation is easily DoS'd).Problem
sh:patterns were translated 1:1 into SRE regexes ([^/]*per*), so a handful of wildcards is enough for exponential matching time:input/a*a*a*a*a*btakes ~2.4 s to not match a name of 200as, 10 stars never finish (the issue's reproducer).fm:patterns do not have this problem anymore since Python 3.11:fnmatch.translate()emits atomic groups for* fixedpairs (bpo-40480).Change
shellpattern.translate()now does the same forsh:patterns: a* FIXED *sequence is emitted as(?>[^/]*?FIXED)[^/]*— FIXED is matched at its leftmost occurrence and the regex engine cannot backtrack into it. This is equivalent to the old regex as long as both neighbours are plain*(also when FIXED contains/): if a match exists using a later occurrence of FIXED, one also exists using the leftmost one, because the left*just absorbs less and the right*absorbs the (slash-free) difference. It is not equivalent when the right neighbour is**/(*a**/bmust matchaab) or when FIXED is a{,}alternatives group (*{a,ab}cmust matchabc), so the atomic group is only used between plain stars. Adjacent*and adjacent**/are collapsed, they are equivalent to a single one.* FIXEDat the end of a pattern is left as is (linear anyway),**/chains stay polynomial and{a*,a*}-style chains stay exponential — those andre:patterns are still covered by the warning inborg help patterns, which is updated accordingly.Numbers
input/+a*× 50 +bvs. 200as: never finished → 0 ms (new testtest_no_exponential_backtracking).ab/*?{,}[]!\) × random paths: identical results with old and newtranslate().shellpattern/patterns/archiver pattern tests unchanged and passing.