Macro-generate capability registrations; harden runtime timeouts and network egress; add platform + eval CI#5
Merged
Merged
Conversation
JavaScriptCore drains the entire promise graph inside a single
evaluateScript call because bridge calls are synchronous, so the
wall-clock poll loop after evaluation could never interrupt CPU-bound
scripts: while(true){} pinned a worker thread forever and timeoutMs
was never enforced against running JavaScript.
Install a JSContextGroupSetExecutionTimeLimit watchdog per execution
whose callback re-checks the deadline and the cancellation flag every
50ms and terminates the script when either trips. Termination is
classified as EXECUTION_TIMEOUT (or SEARCH_TIMEOUT for search) and
CANCELLED, so call.cancel() now interrupts in-flight JavaScript
instead of only flipping a flag that was polled after completion.
The deadline now starts when evaluation begins rather than after the
script settles, and the dangling-promise poll loop reuses the same
deadline instead of granting a second full timeout budget.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Az9HuLR4Sk6tP5Y924z1zF
network.fetch previously accepted any HTTP(S) destination once the capability was allowlisted, so agent-authored JavaScript could reach loopback services, RFC 1918 hosts, and cloud metadata endpoints such as 169.254.169.254, follow redirects from public URLs to internal ones, and buffer arbitrarily large response bodies into memory. Introduce NetworkAccessPolicy, injected via CodeModeConfiguration.networkAccessPolicy: - .standard (the default) refuses loopback, private-range, link-local, CGNAT, and unique-local IPv4/IPv6 destinations (including encoded literals like 2130706433 and 0x7f000001 via inet_aton, and IPv4-mapped IPv6), plus localhost and .local/.localhost/.internal names, and caps buffered responses at 10 MB. - allowedHosts/blockedHosts support exact and subdomain matching, with explicit allowlist entries able to deliberately re-enable private hosts. .permissive restores the previous unrestricted behavior. - The fetch transfer now runs through a per-task URLSession delegate so redirect targets are re-validated before being followed and oversized bodies are cancelled mid-transfer instead of buffered. - Refusals throw structured NETWORK_POLICY_VIOLATION errors with repair suggestions, and both denials and successful fetch destinations are written to the audit logger. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az9HuLR4Sk6tP5Y924z1zF
The install snippet told users to depend on from: "0.1.0", but the repository has no tags, so Swift Package Manager cannot resolve that requirement. Document the branch-based dependency as the working form until a release is tagged, and keep the versioned form as the preferred path once 0.1.0 exists. Also drop the deprecated name: parameter from the .package declaration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az9HuLR4Sk6tP5Y924z1zF
Compile blocker: JSContextGroupSetExecutionTimeLimit and
JSContextGroupClearExecutionTimeLimit are exported by JavaScriptCore
but declared only in a private WebKit header, so the watchdog would
not compile against the public SDK module. Route both through a new
CCodeModeJSC C shim target that re-declares the exported symbols, and
document the private-API dependency in the watchdog and README.
Correctness (execution):
- The post-evaluation poll loop used 'while Date() < deadline' as its
guard, so a script that had already fulfilled just as the deadline
passed skipped the loop entirely and threw a spurious timeout that
discarded a completed result. Replace both poll loops with a shared
waitForSettlement helper that checks the settled state before the
deadline on every iteration (and re-checks watchdog termination),
and returns an already-settled result even a hair past the deadline.
- Result serialization ran with the watchdog uninstalled, so a result
with a runaway getter/toJSON (e.g. { get x() { while(true){} } })
hung the execution thread forever. Keep the watchdog installed and
rearm it with a fresh bounded budget for the decode phase, so
legitimate serialization still completes while runaway getters are
terminated.
- Fold the duplicated execute/search termination + poll logic into the
shared helper.
Security (network policy):
- Host normalization now strips a trailing root dot, so fully-qualified
spellings like 'localhost.' or 'metadata.google.internal.' can no
longer bypass the loopback/metadata block or the allow/deny lists.
- IPv6 private detection now also covers IPv4-compatible (::a.b.c.d) and
NAT64 (64:ff9b::/96) embeddings of private/loopback IPv4 addresses.
Efficiency:
- Reserve the response buffer from a known Content-Length instead of
growing it through repeated reallocations, and build the success
audit/log summary string once.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Az9HuLR4Sk6tP5Y924z1zF
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az9HuLR4Sk6tP5Y924z1zF
swift test on macOS never compiled the UIKit presenters or the CCodeModeJSC watchdog shim against the iOS/visionOS SDKs, so a platform-specific break (like the private-header symbol the shim resolves) could land undetected. Add a platform-build matrix that runs xcodebuild for iOS and visionOS on every PR and push, plus SwiftPM build caching and toolchain-version logging on the deterministic job. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az9HuLR4Sk6tP5Y924z1zF
CLRegion / CLCircularRegion are unavailable on visionOS, so the region-scoped CLGeocoder.geocodeAddressString(_:in:) overload and the coreLocationRegion helper failed to compile there. The package declared .visionOS(.v2) support but was never actually built for visionOS until the new platform-build CI job exercised it. Use the unscoped geocodeAddressString overload on visionOS and compile the region helper out there; other platforms keep region-scoped geocoding. Pre-existing issue surfaced by the new CI matrix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Az9HuLR4Sk6tP5Y924z1zF
First macOS verification run of the Linux-authored watchdog hung the whole test suite: on macOS 26 JavaScriptCore invokes the termination callback exactly once per installed limit, and returning false permanently disarms the watchdog instead of re-arming it as documented. Every runaway script (while(true), promise-chain loops, runaway getters during serialization) therefore ran forever after the first 50ms check. Root-caused with a minimal C reproduction against JSContextGroupSet- ExecutionTimeLimit. The callback is now a self-referencing C function that re-installs the time limit before returning false, restoring periodic deadline/cancellation checks. All 8 ExecutionWatchdogTests now pass.
…an drift Bridges parse constrained string arguments case-insensitively (lowercased() throughout SystemUI/EventKit/Photos/FileSystem bridges), but the registry's allowedStringValues check was exact-match, so it rejected inputs the bridges accept. Validation now compares case-insensitively, the calendarDelete span list includes the alias spellings EventKitBridge accepts (this_event, future, ...), and the lowercase videoQuality duplicates that worked around the case sensitivity are dropped. Also adds a test pinning that no built-in registration gates on .healthKit via requiredPermissions: the default broker can never report .granted for HealthKit (read authorization is opaque by design), so such a registration would fail closed unconditionally. HealthBridge does per-type authorization itself.
Executions run on a background queue while the host may synchronously block the main thread waiting on the result; DispatchQueue.main.sync from the broker then deadlocks. Use main.async — the existing 10s delegate wait already bounds the request, and if main never runs it the wait times out and returns the current status instead of hanging.
PathPolicy: empty/whitespace paths, scoped roots, appGroup with and without a configured root, absolute paths inside/outside allowed roots, dot-dot escapes vs internal dot-dot, nonexistent nested paths, and symlinks that escape vs symlinks between allowed roots. Plus round-trip/uniqueness tests for InMemoryArtifactStore and ordering/drain/concurrency tests for SyncAuditLogger. SystemPermissionBroker is left untested deliberately: every path ends in a real OS framework call and needs injectable seams first (noted in TODO).
Audit of Tools/CodeModeAuthoring (builds and passes all 14 tests on the current toolchain) against the three built-in registration idioms and the parallel sources of truth behind the metadata drift. Proposes a three-phase path: converge built-ins on the typed-tool protocol with enum-typed constrained arguments (drift-proof by construction, no macro needed), then promote the macro into the core package as @BuiltInCodeMode generating the decode/metadata boilerplate, then migrate per domain and delete the central constraint and type-inference tables. Includes measured swift-syntax build cost (24s clean, ~0 incremental, prebuilts default-on) since that was the original reason the macro package was kept out of the core graph.
… enums Promotes BuiltInCodeModeTool out of SimpleBuiltInCodeModeProviders.swift as the standard registration idiom, extended with capability IDs, JS name aliases, and required permissions. Adds CodeModeStringEnum: a constrained string argument is declared once as an enum, and the advertised allowedStringValues, the case-insensitive/alias-aware decode, and the bridge parsing all derive from that declaration, so they cannot drift. Migrates the EventKit domain (9 registrations, 5 enums) to prove the shape: EventKitBridge and SystemUIBridge now parse span/operation/picker styles through the shared enums, and the four EventKit-domain rows are deleted from the central CapabilityArgumentConstraints.defaults table. Keychain, location, and weather tools converge on the same protocol; the raw-CodeModeRegistration and builtInCapability: glue idioms are gone. Adds constrainedArgumentMetadataIsCoherentForAllRegistrations (every constrained argument must be a declared argument) and reworks the span test to pin enum <-> descriptor <-> bridge agreement. Transitional tools carry a raw passthrough because bridges still consume raw dictionaries; the Phase 2 macro generates decode and the argument metadata, which is where the line-count win lands. 207 tests pass.
Folds Tools/CodeModeAuthoring into the root package: CodeModeMacros becomes a macro target of the main package (swift-syntax 601.0.1, prebuilts default-on; eval CLI cold build measured at 29s) and the host-facing @CodeMode surface ships as the CodeModeAuthoring product. The same compiler plugin now also provides the internal @BuiltInCodeMode + @ToolParam macros. @BuiltInCodeMode generates the mechanical members of a BuiltInCodeModeTool from its Arguments struct: identity statics, the codeModeArguments metadata list, and decode(arguments:). Curated prose, permissions, and call(arguments:context:) stay hand-written. Field types the macro does not recognize as JSON primitives are emitted through CodeModeStringEnum- constrained overloads, so the compiler enforces the semantics on the expansion and constraint metadata derives from the enum. A trailing un-annotated raw: [String: JSONValue] receives the canonicalized passthrough transitional tools need while bridges consume raw dictionaries. The EventKit domain is converted to the macro (its hand-written decode and argument lists are deleted). Adds five expansion tests (generated source + diagnostics), behavioral decode tests, and a metadata baseline test pinning the advertised surface of remindersWrite. CI xcodebuild jobs get -skipMacroValidation for the package-local plugin. 229 tests pass.
CapabilityMetadataGoldenTests pins every registration's full advertised surface (jsNames, prose, permissions, argument lists/types/hints, constraints, resultSummary) against a committed 115-capability JSON baseline; CODEMODE_REGENERATE_GOLDEN=1 regenerates it, and the JSON diff becomes the review artifact for registration refactors. This is the safety rail that makes the remaining domain migrations mechanically verifiable. PHASE3-HANDOFF.md is the self-contained work spec for converting the seven remaining CapabilityRegistrations files to @BuiltInCodeMode: per-registration recipe (including the effective-type rule for inferred/.any arguments and the constrained-enum rewiring pattern), domain order with per-domain notes, the allowed golden-diff categories, and hard rules (no prose rewording, no permission-ownership changes, skip-don't-improvise). 230 tests pass.
Converts all 12 interaction-UI capabilities (share, quicklook, camera capture/scan, mail, messages, print, web, auth, alert, prompt, settings) to macro-authored tools in SystemUICodeModeTools.swift; the registration file is now a thin list. Introduces 8 CodeModeStringEnums for the constrained values (MediaTypeFilter, CameraDevice, CameraFlashMode, CameraVideoQuality, DataScannerMode, DataScannerQualityLevel, PrintOutputType, AlertPreferredStyle) and rewires SystemUIBridge's own lowercased() validations to parse through them, making each enum the single source of truth. The cameraUICapture / cameraUIScanData / printUIPresent / uiAlertPresent rows are deleted from the central CapabilityArgumentConstraints.defaults table. AlertPreferredStyle keeps the all-lowercase 'actionsheet' spelling as an alias because SystemUIBridge accepts it (SystemUIBridge.swift validateAlertArguments); CameraVideoQuality's canonical spellings survive the presenters' lowercasing (UIKitSystemUIPresenter.videoQuality/printOutputType), so raw-passthrough canonicalization is behavior-preserving. Golden diff: one allowed category only — argumentHints gained title/message entries for uiPromptPresent (which advertised no hints for them before); values match the uiAlertPresent wording. 230 tests pass.
Converts the 10 filesystem capabilities (list, read, write, move, copy, delete, stat, mkdir, exists, access) to macro-authored tools in FileSystemCodeModeTools.swift; the filesystem section of +Core.swift is now a thin list. No constrained values, so no enums and no bridge rewiring. networkFetch is left on the flat-init idiom (PHASE3-SKIP comment): it declares nested dotted-path arguments (options.method, options.headers, …) that the flat @BuiltInCodeMode Arguments model cannot express without changing the advertised optionalArguments list, and its options.responseEncoding constraint stays in the central defaults table. Keychain already delegates to the Phase-1 converted builtins and is untouched. Golden diff: empty — every filesystem argument already had a hint, and types match the inferred baseline. 230 tests pass.
…eMode Converts all 13 capabilities (contacts read/search/pick/present/presentNew, photos read/export/pick/limited-picker, documents pick/export/openIn/scan) to macro-authored tools; both registration functions are now thin lists. Constrained values: photosRead and photosUIPick reuse MediaTypeFilter (from domain 1); contactsUIPick gets a new ContactPickerMode enum. Their three rows are deleted from the central defaults table. Rewires the owning parsers to the enums: PhotosBridge.read (mediaType -> MediaTypeFilter), SystemUIBridge validateContactPickerArguments (mode -> ContactPickerMode); photosUIPick's mediaType already validated through MediaTypeFilter via the shared validatePhotoPickerArguments rewired in domain 1. Also decouples constraintValidationMatchesCaseInsensitively from the removed contactsUIPick table row by constructing the constraints inline — it tests the generic validate() matching, which is unchanged. Golden diff: empty — advertised allowedStringValues are identical to the former table rows, all argument types match the inferred/declared baseline, and every argument already had a hint. 230 tests pass.
Converts all 19 vision/notifications/alarm/health/home/media capabilities to macro-authored tools in SystemServicesCodeModeTools.swift; the six registration functions are now thin lists. No constrained values in this domain, so no enums, no bridge rewiring, no table changes. Permission declarations are preserved exactly: notifications (.notifications), alarms (.alarmKit), home (.homeKit); vision/media declare none; and the three health tools declare no requiredPermissions — HealthKit read authorization is opaque so HealthBridge does its own per-type auth, and the noBuiltInRegistrationGatesOnHealthKitPermission invariant test still passes. homeWrite's value keeps its .any type (declared as JSONValue). Golden diff: empty — every argument already had a hint and types match the declared/inferred baseline. 230 tests pass.
…o @BuiltInCodeMode Converts all 15 capabilities in +CloudPushSpeech.swift (6 CloudKit, 5 remote notification, 4 speech) to macro-authored tools; the three sub-functions are now thin lists (the bigTicketAppleRegistrations aggregator is unchanged). The four database ops (queryRecords/saveRecord/deleteRecord/subscribe) share a new CloudKitDatabase enum (private/shared/public); its row is deleted from the central defaults table and SystemCloudKitMapping.databaseName/database now parse through the enum, so the advertised, decoded, and system-client-mapped values share one source. The mapping test (default private, shared, invalid archive) still passes. Golden diff: one allowed category — cloudKitSubscriptionSave gained containerIdentifier and zoneID hints (it advertised none for them); values are copied verbatim from the sibling CloudKit registrations. 230 tests pass.
…nCodeMode Converts all 18 capabilities in +IntentsModelsActivityMaps.swift to macro-authored tools; the four sub-functions are now thin lists. Constrained values: activityEnd gets an ActivityDismissalPolicy enum (default/immediate) and mapsRouteEstimate/mapsOpen share a MapsTransportType enum (automobile/walking/transit/any); both rows are deleted from the central defaults table. transportType parsing in SystemMapsMapping.transportTypeName is rewired through the enum (its allowedTransportTypes constant is removed); dismissalPolicy has no in-tree consumer (host ActivityClient handles it), so the enum gates it at the registry. The type-inference trap: foundationModelsGenerate.prompt is absent from the argument-type table, so its historical advertised type is 'any' — declared as JSONValue to preserve it, not String. Golden diff: one allowed category only — argumentHints gained entries for seven registrations that advertised no hints for those arguments (activityList, activityUpdate, activityEnd, activityPushTokenRead, mapsReverseGeocode, and the departureDate/arrivalDate and transportType gaps on the maps route/open tools). No types, constraints, or existing prose changed. 230 tests pass.
…main) Converts all 18 music/passKit/storeKit capabilities to macro-authored tools; the three sub-functions are now thin lists. Only musicPlaybackControl.action is constrained (a new MusicPlaybackAction enum: play/pause/stop/skipToNext/ skipToPrevious/playCatalog/playLibrary); its row is deleted from the central defaults table. Host clients stay authoritative for playback semantics, so no bridge rewiring — the enum only gates the advertised value set at the registry. The type-inference trap: musicPlaylistWrite.name is absent from the argument- type table, so its historical advertised type is 'any' — declared as JSONValue to preserve it, not String. Also makes registryValidationRejectsDescriptorConstrainedValuesBeforePermissions self-contained: it built a synthetic musicPlaybackControl descriptor that relied on the removed defaults(for:) row, so it now passes an explicit argumentConstraints and still verifies constraint-before-permission ordering. Golden diff: one allowed category only — argumentHints gained entries for five registrations that advertised none for those arguments (musicCatalogDetails countryCode, musicPlaybackControl startPlaying, musicPlaylistWrite description, passKitApplePayPresent timeoutMs, passKitPassPresent identifier+timeoutMs). This completes Phase 3: 114 of 115 capabilities are now macro-authored; networkFetch alone stays on the flat init (dotted-path arguments). Updates TODO.md and PLAN-registration-macros.md to reflect completion. 230 tests pass.
The scheduled (daily) run only fired 'codemode-eval plan', which is pure arithmetic (scenarios x repeat x maxTurns) printed to a log — no live model calls, no compare, no artifact, deterministic output. It burned a macOS runner daily for zero signal. Removed the 'schedule' trigger entirely. Deterministic evals + iOS/visionOS platform builds stay continuous: PRs and pushes to main (merges), unchanged. Replaces the no-op llm-plan job with a real llm-live regression gate, manual only (workflow_dispatch): it previews the budget, then — when a WAVELIKE_API_KEY secret is present and the CLI is built with the private overlay (real codemode-eval llm vs the LLMUnavailable stub) — runs the core/failures suites live and compares each against its committed baseline, failing on pass-rate or exact-capability regression. Absent those it skips the live steps with a warning rather than pretending. This encodes the intended pipeline so it goes fully live once the private Wavelike wiring + secret land, with no further workflow edits. Updates EVALS.md CI Policy to match.
zac
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the critical issues from a repo evaluation: timeouts that couldn't interrupt running JavaScript, an unrestricted
fetch(SSRF), and install docs that couldn't resolve. A high-effort self-review of the fixes then surfaced a compile blocker and several bugs, which are also fixed here.Runtime: real timeouts & cancellation
evaluateScriptcall, so the old wall-clock poll loop never ran while JS executed —while(true){}pinned a worker thread forever andtimeoutMswas never enforced. AddedExecutionWatchdog, which installs a JavaScriptCore execution time limit so CPU-bound scripts are preemptively terminated andcancel()interrupts in-flight JS.JSContextGroupSetExecutionTimeLimit/...Clear...are exported by JavaScriptCore but declared only in a private WebKit header, so they aren't visible through the public SDK module. They're reached through a new thin C shim target,CCodeModeJSC. This is JavaScriptCore's only interruption mechanism; the private-symbol dependency is documented in the README and the watchdog source so a host can make an informed App Store decision.waitForSettlementhelper), and result serialization ran with the watchdog uninstalled so a runaway getter/toJSONhung the thread forever (the watchdog now stays armed with a fresh bounded budget for the decode phase).Security: network egress policy (SSRF)
NetworkAccessPolicyonCodeModeConfiguration. The default blocks loopback, RFC 1918, link-local (including cloud metadata like169.254.169.254), CGNAT, and unique-local destinations — including alternate/encoded IPv4 literals and IPv4-in-IPv6 (IPv4-mapped, IPv4-compatible, and NAT64) forms — pluslocalhost/.local/.internalnames and fully-qualified trailing-dot spellings.allowedHosts/blockedHostsand.permissivegive hosts control. Allowed and denied egress destinations are written to the audit logger.Docs
from: "0.1.0"against a tagless repo (SPM can't resolve it); it documents branch-based install until0.1.0is tagged, and adds sections on the network access policy and the timeout/cancellation semantics.CI
platform-buildmatrix (xcodebuildfor iOS and visionOS) so the UIKit presenters and theCCodeModeJSCshim compile against those SDKs —swift teston macOS never did — plus SwiftPM build caching and toolchain logging.Tests
ExecutionWatchdogTests(infinite loops, loop-in-promise-chain, uncatchable timeout, settles-at-deadline returns result, infinite getter during serialization, explicit cancel, search timeout) andNetworkAccessPolicyTests(private/loopback/link-local/CGNAT, encoded IPv4, IPv6 incl. NAT64, trailing-dot, allow/deny lists, size cap, redirect refusal).Verification note
Authored in a Linux environment without a Swift toolchain or JavaScriptCore, so nothing here was compiled or run locally — this PR is opened specifically to let CI build and test it on macOS/iOS/visionOS. A remaining
TODO.md(committed on this branch) tracks the follow-up work the evaluation surfaced (JS heap cap, concurrency bound, HealthKit-always-denied, calendar-span validation drift, Security-layer tests, and more).🤖 Generated with Claude Code
https://claude.ai/code/session_01Az9HuLR4Sk6tP5Y924z1zF
Generated by Claude Code