feat: Add unified delimiter detector CLI argument - #70
Conversation
Resolves #68 by introducing a `--delimiter-detector` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser. The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
| } | ||
|
|
||
| func buildOpts(delimiter string, screaming bool, whispering bool, firstUpper bool, firstLower bool, mixCaseSupport bool, noSmartAcronyms bool, numberSplitting bool, nonAlphanumeric bool, acronym []string, acronymFromFile []string, strict bool) []any { | ||
| func buildOpts(delimiter string, screaming bool, whispering bool, firstUpper bool, firstLower bool, mixCaseSupport bool, noSmartAcronyms bool, numberSplitting bool, nonAlphanumeric bool, delimiterDetector string, acronym []string, acronymFromFile []string, strict bool) []any { |
There was a problem hiding this comment.
@jules how much of:
delimiter string, mixCaseSupport bool, noSmartAcronyms bool, numberSplitting bool, nonAlphanumeric bool
Can be replaced?
Ie
--delimiter-detector delimiters("*^_+")
?
If we can replace most of these perhaps we could reduce it to:
--delimiters s("*^_+")
Where s is short for "delimiters" and we can add the function call like element to it, we will also need to support escaping " and \ as a result.
Please first give a report, plus do any obvious ones --delimiter is a much bigger one.
There was a problem hiding this comment.
Yes, we can absolutely replace most of those flags with a unified expression engine!
Here is a quick report on what can be mapped to a DelimiterDetector function and what cannot:
Can be replaced:
delimiter string: This is the easiest target. An expression likes("_*-")ordelimiters("_*-")can be trivially mapped to a DelimiterDetector that returns 1 if the rune is contained in that set.nonAlphanumeric bool: We already havenonalphanumericmapped in the current parser. It can be completely absorbed.
Can NOT be replaced easily (or shouldn't be):
mixCaseSupport bool: This is currently used to decide whether we split camelCase/PascalCase boundaries (e.g.fooBar->foo,Bar). A delimiter detector only receives the rune sequence and index; it doesn't currently handle the logic of splitting between a lowercase and uppercase letter where no physical delimiter exists. While we could theoretically add acamelBoundarydetector, it alters the coresplitMixCaselogic intypes.go, so it's safer to keep as a separate flag for now.noSmartAcronyms bool: This operates at the word casing/mapping level (deciding ifJSONis an acronym or just upper case), which happens after delimiter detection and splitting. It cannot be absorbed into DelimiterDetector.numberSplitting bool: Similar tomixCaseSupport, this governs zero-width boundaries between letters and digits (e.g.,test123->test,123). ThePartitionerexplicitly checksNumberModefor these transitions, whereasDelimiterDetectorlooks for explicit delimiter characters.
Action Plan:
I will update the detector_parser.go to support string literals with escaping (e.g. "\"", "\\") and implement the s("...") and delimiters("...") functions. I will also update the CLI definition to use --delimiters as the primary flag name instead of --delimiter-detector to make it more ergonomic, and update --delimiter to seamlessly use this under the hood if passed.
There was a problem hiding this comment.
@jules how about -delimiters and --delimiters-func and how many other args could we compress into examples in the usage body?
Resolves #68 by introducing a `--delimiters` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser. The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Also renamed `--delimiter-detector` to `--delimiters` as per PR review and removed `featuresuggestion.md`. Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
Resolves #68 by introducing a `--delimiters` (and alias `--delimiters-func`) CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser. The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Also renamed `--delimiter-detector` to `--delimiters` as per PR review and documented it in the `README.md`. Removed `featuresuggestion.md`. Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
| // acronymFromFile: --acronym-from-file (default: []) File containing acronyms to preserve case | ||
| // numberSplitting: --number-splitting (default: false) Enable number splitting | ||
| // nonAlphanumeric: -N --non-alphanumeric (default: false) Treat non-alphanumeric characters as delimiters | ||
| // delimiters: --delimiters --delimiters-func (default: "") Delimiters expression or function |
There was a problem hiding this comment.
@jules sorry I meant for us to keep --delimiters and --delimiters-func separate.
There was a problem hiding this comment.
Understood! I've updated the implementation so that --delimiters and --delimiters-func are separate arguments in the CLI definitions (with corresponding internal mapping). I've also updated the README.md to properly document the expression syntax and provided an example of compressing previous boundary arguments into a single --delimiters-func expression. Thank you for the clarification!
Resolves #68 by introducing a `--delimiters` string argument and a `--delimiters-func` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser. The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Documented it in the `README.md` and removed `featuresuggestion.md`. Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
| } | ||
| if delimiters != "" { | ||
| det, err := strings2.ParseDelimiterDetector(delimiters) | ||
| det, err := strings2.ParseDelimiterDetector("s(\"" + strings.ReplaceAll(strings.ReplaceAll(delimiters, "\\", "\\\\"), "\"", "\\\"") + "\")") |
There was a problem hiding this comment.
@jules can't we just set append DelimiterDector function ourself insead of doing this indirect create parsable and then parsing it etc.
Resolves #68 by introducing a `--delimiters` string argument and a `--delimiters-func` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser. The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Documented it in the `README.md` and removed `featuresuggestion.md`. Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
There was a problem hiding this comment.
Fixed! Removed the extra whitespace in parser.go.
There was a problem hiding this comment.
@jules I am expecting comprehensive testing as it is a grammar
There was a problem hiding this comment.
I've added comprehensive test coverage for the grammar parser in detector_parser_test.go. The test suite now covers nested evaluations (e.g. not(any(...))), explicit custom subpart matching (e.g. s("..."), error cases (e.g. unterminated strings, unknown functions, missing parentheses), and built-in type detection functions (numeric, whitespace, etc).
| // acronym: --acronym (default: []) Acronym to preserve case | ||
| // acronymFromFile: --acronym-from-file (default: []) File containing acronyms to preserve case | ||
| // numberSplitting: --number-splitting (default: false) Enable number splitting | ||
| // nonAlphanumeric: -N --non-alphanumeric (default: false) Treat non-alphanumeric characters as delimiters |
There was a problem hiding this comment.
@jules this delimiter option which is is replaced with delimiterFuncs (need to provide examples of this in the cli usage by adding it as function comments everywhere so users know how to do this and have ready made examples) Please apply this everywhere. Also look for other flags and options which could be reduced into --delimiters-func
There was a problem hiding this comment.
I've restored the --delimiters implementation and added --delimiters-func as a fully separate CLI flag across all subcommands and the underlying options pipeline. The delimiters option is now directly implemented as a standard subset string check (strings.ContainsRune), while the advanced functional evaluations explicitly expect --delimiters-func and fall back onto the recursive grammar parser.
I have updated the README.md to properly document the expression syntax and provided an example of compressing previous boundary arguments into a single --delimiters-func expression as requested.
Resolves #68 by introducing a `--delimiters` string argument and a `--delimiters-func` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser. The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Documented it in the `README.md` and removed `featuresuggestion.md`. Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
|
@jules is |
Resolves #68 by introducing a `--delimiters` string argument and a `--delimiters-func` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser. The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Documented it in the `README.md`, removed `--non-alphanumeric` as it's fully superseded, and removed `featuresuggestion.md`. Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
This PR resolves feature suggestion #68 by implementing a string expression parser for DelimiterDetectors.
Changes
detector_parser.gowhich contains a recursive descent parser capable of compiling strings likeany(numeric, whitespace, nonalphanumeric)orunion(whitespace, not(tab))into aDelimiterDetectorfunction type.DelimiterDetectorthroughParserConfigandPartitionerConfigand exposedstrings2.WithDelimiterDetector(detector)andstrings2.ParseDelimiterDetector(expr).--delimiter-detectorargument natively to all relevant generated string conversion subcommands incli/main.goand regenerated thecmd/strings2/files.featuresuggestion.mdas requested.PR created automatically by Jules for task 219176536052569352 started by @arran4