fix: avoid tableFromJSON codegen for Cloudflare Workers compatibility - #249
fix: avoid tableFromJSON codegen for Cloudflare Workers compatibility#249ensky wants to merge 4 commits into
Conversation
Co-Authored-By: Paperclip <noreply@paperclip.ing>
docs: fix typo in README ("Analaytics" -> "Analytics")
apache-arrow's `tableFromJSON()` (via `vectorFromArray` → `makeBuilder`
→ `new StructBuilder` → `createIsValidFunction`) calls `new Function(...)`
to generate validity checkers. Cloudflare Workers' runtime forbids
runtime code generation:
EvalError: Code generation from strings disallowed for this context
at new Function (<anonymous>)
at createIsValidFunction
...
at tableFromJSON
at extractAsArrow
The daily-rollup cron has thrown on every fire since deployment under
this restriction, so the R2 `counterscale-daily-rollups` bucket has
never received an object.
Replace `tableFromJSON(records)` with a small `recordsToTable` helper
that builds `Utf8`/`Float64` `Data` directly via `makeData` and assembles
them into a `RecordBatch` — bypassing the Builder/codegen path entirely.
Adds a vitest covering:
- round-trip through `tableToIPC` / `tableFromIPC`
- null/undefined handling
- empty-input edge case
- a guard that proxies `globalThis.Function` and asserts no
`new Function(...)` call happens during table construction or IPC
serialization (proves the CF Workers compatibility property at unit
level)
Co-authored-by: Paperclip <noreply@paperclip.ing>
stordahl
left a comment
There was a problem hiding this comment.
Hey @ensky, thanks for the contribution and fixing this bug! Apologies for the radio silience - I'm picking back up work on counterscale as a volunteer while Ben focuses on Modem. I left a few comments, but overall this looks great
| return makeData({ | ||
| type: new Utf8(), | ||
| length: values.length, | ||
| nullCount: 0, |
There was a problem hiding this comment.
nullCount is always 0, but null/undefined inputs produce empty strings via the ?? new Uint8Array(0) fallback. A downstream reader will see "" for rows that were originally null, which loses the null signal. Either construct a validity bitmap and set nullCount correctly, or add a JSDoc on recordsToTable documenting that it does not preserve Arrow null semantics.
There was a problem hiding this comment.
Implemented a validity bitmap for Utf8 data. It marks non-null rows valid, reports the actual null count, and keeps null/undefined values as Arrow nulls after IPC round-trip. The existing no-codegen guard still passes.
| return makeData({ | ||
| type: new Float64(), | ||
| length: values.length, | ||
| nullCount: 0, |
There was a problem hiding this comment.
Same issue as buildUtf8Data — nullCount is 0 but null/undefined values are written as 0.0 via values[i] ?? 0. This silently converts nulls to zeros.
There was a problem hiding this comment.
Applied the same shared validity bitmap to Float64 data, so null/undefined values no longer decode as 0.0. Placeholder payload values remain internal to the Arrow buffer.
| const table = recordsToTable(records); | ||
| const buf = new Uint8Array(tableToIPC(table, "file")); | ||
| const decoded = tableFromIPC(buf); | ||
| expect(decoded.numRows).toBe(3); |
There was a problem hiding this comment.
This test only verifies row count, not value round-trip fidelity. It should explicitly assert that rows 2 and 3 ({ a: null, b: null } and { a: undefined, b: undefined }) decode with null values for column b.
There was a problem hiding this comment.
Expanded the regression coverage to assert decoded values for both Utf8 and Float64 null/undefined rows. It also verifies that all-valid columns retain an empty validity bitmap.
Keeps nullish rollup fields distinct from empty strings and zeroes while retaining the direct builder required by Cloudflare Workers. Co-Authored-By: Paperclip <noreply@paperclip.ing>
Problem
The daily-rollup scheduled handler (
extractAsArrowinpackages/server/workers/lib/arrow.ts) cannot succeed on CloudflareWorkers as currently written.
apache-arrow'stableFromJSON()internally walks records andtriggers
vectorFromArray→makeBuilder→new StructBuilder→new Builder→createIsValidFunction, which callsnew Function(...)to generate validity-check code at runtime. TheCloudflare Workers runtime forbids this:
I observed this on a real counterscale deployment with CF Workers
compatibility_date = "2025-08-08". The cron has thrown on every firesince the worker was deployed; no Arrow file has ever been written to
the daily-rollups R2 bucket.
Fix
Replace
tableFromJSON(records)with a smallrecordsToTablehelperthat pivots records to columnar form and builds
Utf8/Float64Arrow
Datadirectly viamakeData, then assembles them into aRecordBatch. This bypasses the Builder/codegen path entirely.Column types are inferred per column from the first non-null sample
(
number→ Float64, otherwise → Utf8). This preserves the same columnshape
tableFromJSONwas producing for this consumer — the recordswritten here only ever have string and number fields.
Public API of
extractAsArrowis unchanged (same return shape, samefilename pattern, same R2 write).
Tests
Adds
workers/lib/__tests__/arrow.test.ts:tableToIPC/tableFromIPCand asserts values match.null/undefinedvalues.globalThis.Functionand asserts nonew Function(...)call happens during table construction or IPC serialization — a
unit-level guarantee that the Workers runtime restriction is
satisfied.
Notes
Happy to adjust style/structure to match the project's conventions.
The change is small and self-contained to
workers/lib/arrow.tsplusthe new unit test.