iOS swiftpm-1.4.0 prebuilts: HF tokenizers with lookahead regex abort the app (signal 6) — regex_lookahead still missing

[john-rocky]

🐛 Describe the bug

Loading current-generation HF tokenizers through ExecuTorchLLM.TextRunner on iOS aborts the whole app when the tokenizer's pretokenizer regex uses lookahead — which most current models do: \s+(?!\S) appears in the tokenizer.json of Qwen2/3/3.5, LFM2/2.5, and other GPT-2-descended tokenizers.

Console on launch (iPhone 17 Pro, swiftpm-1.4.0 prebuilts, Qwen3.5-0.8B tokenizer.json — identical failure with LFM2.5-1.2B-Instruct):

E0000 re2.cc:237] Error parsing '((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n...': invalid perl operator: (?!
I tokenizers:re2_regex.cpp:27] Re2 failed to compile regex: ((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+), error: invalid perl operator: (?!
E tokenizers:regex.cpp:66] RE2 doesn't support lookahead patterns. Link with `regex_lookahead` to enable support.
E tokenizers:hf_tokenizer.cpp:482] Failed to setup pretokenizer: Error: 9
App terminated due to signal 6.

Two separable problems:

1. regex_lookahead is not shipped in the SwiftPM prebuilts. Package.swift on swiftpm-1.4.0 exposes no product containing lookahead support, so the "Link with regex_lookahead" hint is not actionable for SwiftPM users. This is the same situation reported on swiftpm-1.1.0 in Tokenizer fails on iOS (RE2 lookahead unsupported) – need regex_lookahead static lib or guidance #16391 (closed without a landed fix, as far as I can tell); Error [tokenizers:re2_regex.cpp:26] Failed to compile Regex for Qwen2.5 Model #14432 is the Android neighbor.
2. The failure mode is an abort, not an error. After Failed to setup pretokenizer: Error: 9 the process dies with signal 6, so the host app cannot catch the failure from TextRunner.load() and degrade gracefully.

Workaround we validated

Patching the tokenizer.json pattern \s+(?!\S) → \s+$ makes RE2 accept it. We verified token-for-token identical encodings against the original tokenizer (HF tokenizers on the host) for normal single-spaced text, English and Japanese; runs of ≥2 consecutive spaces tokenize differently, so this is a stopgap, not a fix.

With that patch (plus -all_load in OTHER_LDFLAGS, without which backend/kernel registration silently fails), a Qwen3.5-0.8B 8da4w/XNNPACK export from the official config generates correctly on the device.

What would help

• Ship regex_lookahead (or fold it into executorch_llm) in the SwiftPM prebuilts, or document the supported path for HF tokenizers on iOS.
• Return an error from TextRunner.load() instead of aborting.

Versions

ExecuTorch 1.4.0 (pip) for export; SwiftPM branch swiftpm-1.4.0 prebuilts on device. iPhone 17 Pro (iPhone18,1), iOS 27.0 (24A5380h). Xcode 27.0 beta 3. Export host: macOS 15 / Apple Silicon.