rumcodecs

Warning

This project is retired. rumcodecs was an experiment: could numcodecs.js be reimplemented in Rust/WASM without breaking its API or its byte formats? It could, and that is as far as the experiment goes. The project is no longer maintained.

The published package stays on npm and keeps working, but it is deprecated: expect no further releases, no bug fixes, no security updates, and no support. Use numcodecs.js for anything real. The code below is left as it was — read it, fork it, take what is useful (MIT).

numcodecs.js reimplemented in Rust.

A drop-in replacement for numcodecs.js: the same codec classes with the same API and the same byte formats, backed by Rust compiled to WASM (SIMD128) via blusc. Built for Zarr — chunks written by Python numcodecs decode in the browser, pinned by fixtures Python actually wrote.

📦 The codecs

Codec	Backend	Format
Blosc	blusc (Rust → WASM)	Blosc meta-compressor: blosclz / lz4 / lz4hc / snappy / zlib / zstd with shuffle and bitshuffle filters
LZ4	lz4_flex (Rust → WASM)	numcodecs framing — 4-byte little-endian original size + LZ4 block
Zstd	zstd (Rust → WASM)	Standard Zstd frames
GZip	fflate (pure JS)	gzip
Zlib	fflate (pure JS)	zlib

Every class carries the numcodecs.js surface — static codecId, static fromConfig, the same constructor signatures and static members (Blosc.SHUFFLE, Zstd.MAX_CLEVEL, …) — and test/compat.test.ts asserts it stays that way. Compatibility with the Python side is checked against real chunks: the buffers in test/fixtures/ were written by Python numcodecs, and test/interop.test.ts decodes every one of them. That pins the Python → JavaScript direction; the reverse is the same shared byte format but is not exercised here, since asserting it would mean running Python in CI. The WASM module loads lazily on first encode/decode, so the pure-JS codecs never pay for it.

🚀 Quick start

npm install @fideus-labs/rumcodecs
# fflate is a peer dependency, needed only for GZip and Zlib
npm install fflate

import { Blosc } from "@fideus-labs/rumcodecs";

const codec = new Blosc(5, "zstd", Blosc.SHUFFLE);
const compressed = await codec.encode(data);
const restored = await codec.decode(compressed);

Each codec is also a subpath export, so bundlers ship only what you use:

import Blosc from "@fideus-labs/rumcodecs/blosc";

// The numcodecs config shape works unchanged — e.g. straight from Zarr metadata.
const codec = Blosc.fromConfig({ id: "blosc", cname: "lz4", clevel: 5, shuffle: 1 });

📊 Benchmarks

The benchmark/ workspace package runs rumcodecs head-to-head against numcodecs.js on the same generated arrays (tinybench, multiple sizes and dtypes):

pnpm bench          # console table with speedups
pnpm bench:report   # write JSON + markdown report

🛠️ Development

Prerequisites

• Rust 1.91+ with wasm-pack
• Node 18+ (CI tests 22 and 24) and pnpm

Setup

git clone https://github.com/fideus-labs/rumcodecs
cd rumcodecs
pnpm install          # runs prepare: builds WASM (web + node) and the bundle
pnpm test

Repository structure

rumcodecs/
├── src/                    # TypeScript codec classes (Blosc, GZip, Zlib, LZ4, Zstd)
├── crates/rumcodecs-wasm/  # Rust WASM bindings: blusc, lz4_flex, zstd
├── pkg/                    # wasm-pack output (generated)
├── test/                   # vitest suites: round-trip, numcodecs API, interop fixtures
├── benchmark/              # head-to-head benchmark vs numcodecs.js
└── .github/workflows/      # ci.yml, rust-ci.yml, release.yml

Commands

Command	Purpose
pnpm test	Run all vitest suites (round-trip, per-codec, compat, interop)
pnpm check	Lint, format, and typecheck (oxlint / oxfmt / tsc via vp)
pnpm build	Bundle with vite-plus and emit type declarations
pnpm build:wasm / pnpm build:wasm:node	Rebuild the WASM package for web / Node (SIMD128)
pnpm bench	Benchmark against numcodecs.js
cargo test -- --test-threads=1	Rust unit tests for the WASM crate

🤝 Contributing

Nothing to contribute to — the project is retired, and issues and pull requests are no longer being reviewed or merged. Forking is the way forward: the MIT license covers everything here, and AGENTS.md still has the repository map and conventions if you want to pick the code up. Anyone still interacting here is governed by our Code of Conduct.

📄 License

MIT — see LICENSE.txt. Copyright (c) Fideus Labs LLC.

rumcodecs reimplements the API of numcodecs.js (MIT, © Trevor Manz) so existing Zarr tooling works unchanged. Blosc support comes from the pure-Rust blusc implementation of the Blosc meta-compressor; LZ4 from lz4_flex; Zstd from the zstd crate.