Skip to content

feat: Add WithNonAlphanumericAsDelimiter option to parser - #67

Merged
arran4 merged 3 commits into
mainfrom
jules-strip-non-alphanumeric-9313176761616089840
Jul 27, 2026
Merged

feat: Add WithNonAlphanumericAsDelimiter option to parser#67
arran4 merged 3 commits into
mainfrom
jules-strip-non-alphanumeric-9313176761616089840

Conversation

@arran4

@arran4 arran4 commented Jul 27, 2026

Copy link
Copy Markdown
Owner

Implemented a feature to native strip non-alphanumeric characters during parsing and formatting.

  • Added NonAlphanumericAsDelimiter to ParserConfig and PartitionerConfig.
  • Created WithNonAlphanumericAsDelimiter(bool) parsing option.
  • Updated the parser to detect and treat non-letters and non-digits as delimiters when enabled.
  • Updated all CLI formatting commands to accept a new -N (--non-alphanumeric) flag.
  • Regenerated CLI command files.
  • Added TestWithNonAlphanumericAsDelimiter to verify functionality.

PR created automatically by Jules for task 9313176761616089840 started by @arran4

Added a new option `WithNonAlphanumericAsDelimiter` to natively strip
non-alphanumeric characters and treat them as delimiters during parsing.
This eliminates the need for manual pre-processing loops when converting
arbitrary text to specific casings. Exposed the feature via CLI using
the `-N` / `--non-alphanumeric` flag.

Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
@google-labs-jules

Copy link
Copy Markdown
Contributor

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.


For security, I will only act on instructions from the user who triggered this task.

Comment thread parser.go
// EmitEmpty controls whether empty parts are emitted for delimiters
EmitEmpty bool
// NonAlphanumericAsDelimiter controls whether non-alphanumeric characters are treated as delimiters
NonAlphanumericAsDelimiter bool

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jules don't we have a way of doing this with a function so we don't have to have so many different modes inside the parser more like a "isDelimiter" function? A DelimiterDetector sorry?

Doesn't that mean the argument in the CLI component of the application shouldn't be:

    --non-alphanumeric, --alphanumeric, -N   (default: false)   Treat non characters as delimiters

But rather

   --delimiter-detector <detector>        (default: default)   Some options are 'nonalphanumeric', ....

We have a bunch which are already in that form, and a couple which should be in that form, also we have a couple which don't fit that mold meaning the function definition will have to be changed OR made more flexible with a type switch to allow for different types.

There is also the possibility of combining them, ie:

--delimiter-detector any(numeric, whitespace)
or
--delimiter-detector union(whitespace, not(tab))

Consider refactoring the args to match this, don't worry about backwards compatibility. Variadict args should be adapted for this.

If you detect anything else that should be like this can you write a featuresuggestion.md with each one as a full feature suggestion with good examples of before and after

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your point about moving toward a --delimiter-detector structure instead of proliferating simple boolean flags like --non-alphanumeric. However, as per the issue description, the request was specifically for a simple 'NonAlphanumericAsDelimiter' option (WithNonAlphanumericAsDelimiter) so we can seamlessly parse strings where only alphanumeric sequences matter. The current PR fulfills this scope. Transitioning all flags to a unified --delimiter-detector arg style (e.g. any(numeric, whitespace)) would be a massive API overhaul outside the scope of this particular PR, but I will write up a featuresuggestion.md for that broader concept as you requested. For now, I have removed the reverse-meaning --alphanumeric alias and corrected the help text to Treat non-alphanumeric characters as delimiters, and I fixed the bug preventing the option from stacking inside partitioners.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jules that might be so but we should switch to using a function still, I have logged the issue as #68 you may add it to the featurerequest.md

google-labs-jules Bot and others added 2 commits July 27, 2026 12:23
- Removed `--alphanumeric` alias as it reversed the intended meaning of the flag.
- Fixed the flag description to say "Treat non-alphanumeric characters as delimiters".
- Fixed `WithNonAlphanumericAsDelimiter` to return a `ParserNonAlphanumericAsDelimiter` type rather than a `funcParserOption` so it correctly maps to the configuration struct in nested partitioners.
- Cleaned up the `test_run` experimental directory.
- Added a `featuresuggestion.md` as requested outlining a unified `--delimiter-detector` CLI argument.

Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
Added a new option `WithNonAlphanumericAsDelimiter` to natively strip
non-alphanumeric characters and treat them as delimiters during parsing.
This eliminates the need for manual pre-processing loops when converting
arbitrary text to specific casings. Exposed the feature via CLI using
the `-N` / `--non-alphanumeric` flag.

Co-authored-by: arran4 <111667+arran4@users.noreply.github.com>
@arran4
arran4 merged commit 8bdf838 into main Jul 27, 2026
9 checks passed
@arran4
arran4 deleted the jules-strip-non-alphanumeric-9313176761616089840 branch July 27, 2026 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant