Skip to content

feat: Add unified delimiter detector CLI argument - #70

Merged
arran4 merged 7 commits into
mainfrom
jules-issue-68-219176536052569352
Jul 29, 2026
Merged

feat: Add unified delimiter detector CLI argument#70
arran4 merged 7 commits into
mainfrom
jules-issue-68-219176536052569352

Conversation

@arran4

@arran4 arran4 commented Jul 28, 2026

Copy link
Copy Markdown
Owner

This PR resolves feature suggestion #68 by implementing a string expression parser for DelimiterDetectors.

Changes

  • Expression Parser: Added detector_parser.go which contains a recursive descent parser capable of compiling strings like any(numeric, whitespace, nonalphanumeric) or union(whitespace, not(tab)) into a DelimiterDetector function type.
  • Library API Integration: Plumbed DelimiterDetector through ParserConfig and PartitionerConfig and exposed strings2.WithDelimiterDetector(detector) and strings2.ParseDelimiterDetector(expr).
  • CLI Integration: Exposed the --delimiter-detector argument natively to all relevant generated string conversion subcommands in cli/main.go and regenerated the cmd/strings2/ files.
  • Cleanup: Removed featuresuggestion.md as requested.

PR created automatically by Jules for task 219176536052569352 started by @arran4

Resolves #68 by introducing a `--delimiter-detector` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser.

The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code.

Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
@google-labs-jules

Copy link
Copy Markdown
Contributor

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.


For security, I will only act on instructions from the user who triggered this task.

@arran4 arran4 linked an issue Jul 29, 2026 that may be closed by this pull request
Comment thread cli/main.go Outdated
}

func buildOpts(delimiter string, screaming bool, whispering bool, firstUpper bool, firstLower bool, mixCaseSupport bool, noSmartAcronyms bool, numberSplitting bool, nonAlphanumeric bool, acronym []string, acronymFromFile []string, strict bool) []any {
func buildOpts(delimiter string, screaming bool, whispering bool, firstUpper bool, firstLower bool, mixCaseSupport bool, noSmartAcronyms bool, numberSplitting bool, nonAlphanumeric bool, delimiterDetector string, acronym []string, acronymFromFile []string, strict bool) []any {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jules how much of:

delimiter string, mixCaseSupport bool, noSmartAcronyms bool, numberSplitting bool, nonAlphanumeric bool 

Can be replaced?

Ie

--delimiter-detector delimiters("*^_+")

?

If we can replace most of these perhaps we could reduce it to:

--delimiters s("*^_+")

Where s is short for "delimiters" and we can add the function call like element to it, we will also need to support escaping " and \ as a result.

Please first give a report, plus do any obvious ones --delimiter is a much bigger one.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can absolutely replace most of those flags with a unified expression engine!

Here is a quick report on what can be mapped to a DelimiterDetector function and what cannot:

Can be replaced:

  1. delimiter string: This is the easiest target. An expression like s("_*-") or delimiters("_*-") can be trivially mapped to a DelimiterDetector that returns 1 if the rune is contained in that set.
  2. nonAlphanumeric bool: We already have nonalphanumeric mapped in the current parser. It can be completely absorbed.

Can NOT be replaced easily (or shouldn't be):

  1. mixCaseSupport bool: This is currently used to decide whether we split camelCase/PascalCase boundaries (e.g. fooBar -> foo, Bar). A delimiter detector only receives the rune sequence and index; it doesn't currently handle the logic of splitting between a lowercase and uppercase letter where no physical delimiter exists. While we could theoretically add a camelBoundary detector, it alters the core splitMixCase logic in types.go, so it's safer to keep as a separate flag for now.
  2. noSmartAcronyms bool: This operates at the word casing/mapping level (deciding if JSON is an acronym or just upper case), which happens after delimiter detection and splitting. It cannot be absorbed into DelimiterDetector.
  3. numberSplitting bool: Similar to mixCaseSupport, this governs zero-width boundaries between letters and digits (e.g., test123 -> test, 123). The Partitioner explicitly checks NumberMode for these transitions, whereas DelimiterDetector looks for explicit delimiter characters.

Action Plan:
I will update the detector_parser.go to support string literals with escaping (e.g. "\"", "\\") and implement the s("...") and delimiters("...") functions. I will also update the CLI definition to use --delimiters as the primary flag name instead of --delimiter-detector to make it more ergonomic, and update --delimiter to seamlessly use this under the hood if passed.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jules how about -delimiters and --delimiters-func and how many other args could we compress into examples in the usage body?

google-labs-jules Bot and others added 2 commits July 29, 2026 01:36
Resolves #68 by introducing a `--delimiters` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser.

The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Also renamed `--delimiter-detector` to `--delimiters` as per PR review and removed `featuresuggestion.md`.

Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
Resolves #68 by introducing a `--delimiters` (and alias `--delimiters-func`) CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser.

The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Also renamed `--delimiter-detector` to `--delimiters` as per PR review and documented it in the `README.md`. Removed `featuresuggestion.md`.

Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
Comment thread cli/main.go Outdated
// acronymFromFile: --acronym-from-file (default: []) File containing acronyms to preserve case
// numberSplitting: --number-splitting (default: false) Enable number splitting
// nonAlphanumeric: -N --non-alphanumeric (default: false) Treat non-alphanumeric characters as delimiters
// delimiters: --delimiters --delimiters-func (default: "") Delimiters expression or function

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jules sorry I meant for us to keep --delimiters and --delimiters-func separate.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood! I've updated the implementation so that --delimiters and --delimiters-func are separate arguments in the CLI definitions (with corresponding internal mapping). I've also updated the README.md to properly document the expression syntax and provided an example of compressing previous boundary arguments into a single --delimiters-func expression. Thank you for the clarification!

Resolves #68 by introducing a `--delimiters` string argument and a `--delimiters-func` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser.

The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Documented it in the `README.md` and removed `featuresuggestion.md`.

Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
Comment thread cli/main.go Outdated
}
if delimiters != "" {
det, err := strings2.ParseDelimiterDetector(delimiters)
det, err := strings2.ParseDelimiterDetector("s(\"" + strings.ReplaceAll(strings.ReplaceAll(delimiters, "\\", "\\\\"), "\"", "\\\"") + "\")")

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jules can't we just set append DelimiterDector function ourself insead of doing this indirect create parsable and then parsing it etc.

Resolves #68 by introducing a `--delimiters` string argument and a `--delimiters-func` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser.

The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Documented it in the `README.md` and removed `featuresuggestion.md`.

Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
Comment thread parser.go

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jules strangely adding extra whitespace

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed! Removed the extra whitespace in parser.go.

Comment thread detector_parser_test.go

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jules I am expecting comprehensive testing as it is a grammar

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added comprehensive test coverage for the grammar parser in detector_parser_test.go. The test suite now covers nested evaluations (e.g. not(any(...))), explicit custom subpart matching (e.g. s("..."), error cases (e.g. unterminated strings, unknown functions, missing parentheses), and built-in type detection functions (numeric, whitespace, etc).

Comment thread cli/main.go Outdated
// acronym: --acronym (default: []) Acronym to preserve case
// acronymFromFile: --acronym-from-file (default: []) File containing acronyms to preserve case
// numberSplitting: --number-splitting (default: false) Enable number splitting
// nonAlphanumeric: -N --non-alphanumeric (default: false) Treat non-alphanumeric characters as delimiters

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jules this delimiter option which is is replaced with delimiterFuncs (need to provide examples of this in the cli usage by adding it as function comments everywhere so users know how to do this and have ready made examples) Please apply this everywhere. Also look for other flags and options which could be reduced into --delimiters-func

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've restored the --delimiters implementation and added --delimiters-func as a fully separate CLI flag across all subcommands and the underlying options pipeline. The delimiters option is now directly implemented as a standard subset string check (strings.ContainsRune), while the advanced functional evaluations explicitly expect --delimiters-func and fall back onto the recursive grammar parser.

I have updated the README.md to properly document the expression syntax and provided an example of compressing previous boundary arguments into a single --delimiters-func expression as requested.

Resolves #68 by introducing a `--delimiters` string argument and a `--delimiters-func` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser.

The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Documented it in the `README.md` and removed `featuresuggestion.md`.

Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
@arran4

arran4 commented Jul 29, 2026

Copy link
Copy Markdown
Owner Author

@jules is nonAlphanumeric bool necessary anywhere?

Resolves #68 by introducing a `--delimiters` string argument and a `--delimiters-func` CLI argument that accepts functional string expressions (e.g., `any(numeric, whitespace, nonalphanumeric)`, `s("...")`, `union(s("..."))`) to dynamically generate multi-character boundary matching logic via a custom recursive descent parser.

The feature is fully wired through the `PartitionerConfig` and exposed via the `WithDelimiterDetector` API option, reducing the need for numerous individual boolean flags while enabling infinitely composable custom delimiters without writing Go code. Documented it in the `README.md`, removed `--non-alphanumeric` as it's fully superseded, and removed `featuresuggestion.md`.

Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
@arran4
arran4 merged commit add977f into main Jul 29, 2026
9 checks passed
@arran4
arran4 deleted the jules-issue-68-219176536052569352 branch July 29, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Suggestion: Unified Delimiter Detector CLI Argument

1 participant