From 18fe1c6510ef43b6f9c3b0ad36f3a2e1addd32ad Mon Sep 17 00:00:00 2001 From: Gonzalo Casas Date: Wed, 19 Aug 2026 00:57:30 +0200 Subject: [PATCH 1/8] docs: document the cross-language serialization architecture Adds an Architecture page describing how one domain model reaches many languages: who owns .proto files, who implements a runtime, and how the two meet at the registry. Four diagrams carry the parts that are hard to convey in prose -- the mirrored per-language structure, the seven AnyData arms (including why `fallback` is the only one that reconstructs an object), the recursive dispatch tree, and the schema distribution model. Mermaid needed enabling: superfences had no custom_fence for it. Co-Authored-By: Claude Opus 5 --- docs/architecture.md | 257 +++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 7 +- 2 files changed, 263 insertions(+), 1 deletion(-) create mode 100644 docs/architecture.md diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..7a919b1 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,257 @@ +# Architecture + +`compas_pb` is the Python reference implementation of a cross-language serialization +architecture. This page documents that architecture so sibling implementations +(`compas_pb_ts` for TypeScript, `compas_pb_csharp` for C#) and packages that own their own +domain models can follow the same contract. + +The goal is a single wire format that carries **domain model objects** — not just the +generated `*Data` protobuf messages — between Python, TypeScript, C# and any other language +that gains a runtime. A `Frame` sent from Grasshopper arrives in a browser agent as a +`Frame`, and an Antikythera `TaskAssignmentMessage` arrives in Python as a +`TaskAssignmentMessage`, without either side knowing anything about the other's language. + +## Two kinds of package + +Every package in this architecture is one of two things. Keeping the distinction sharp is +what makes the system extensible. + +| | Domain model owner | Language runtime | +| --- | --- | --- | +| **Examples** | `compas_pb`[^1], `antikythera`, `compas_timber` | `compas_pb`, `compas_pb_ts`, `compas_pb_csharp` | +| **Owns** | `.proto` files and the domain classes they mirror | registry, discovery, recursive codec | +| **Publishes** | proto bundle + generated bindings, every release | a serialization library for one language | +| **Knows about** | its own types only | no domain types at all | + +[^1]: + `compas_pb` is both, exceptionally. It owns the `.proto` files for COMPAS core types + (`Point`, `Frame`, `Mesh`, `Graph`, …) because protobuf support has not been upstreamed + into `compas` core yet. For every practical purpose, treat `compas_pb`'s ownership of + those schemas as if it were core's own. + +A language runtime never imports a domain package, and a domain package never implements +codec logic. They meet at the registry. + +## How one domain model reaches many languages + +The same three-part structure repeats in every language a domain owner supports: generated +bindings, domain classes that mirror the owner's own model, and a conversions module that +registers the mapping between them. Only the registration mechanism differs. + +```mermaid +flowchart TB + proto["antikythera/proto/*.proto
single source of truth"] + rel["Release vX.Y.Z
proto bundle · typescript · c# · c++ bindings"] + proto ==> rel + + subgraph ts["TypeScript — proposed"] + direction TB + tsm["antikythera_ts
TaskAssignmentMessage
mirrors the Python model"] + tsc["antikythera_ts/conversions
registerSerializer · registerDeserializer"] + tsr["compas_pb_ts
registry · discovery · codec"] + tsm --> tsc + tsc -->|"registers via
register()"| tsr + end + + subgraph py["Python — shipping today"] + direction TB + pym["antikythera.models
TaskAssignmentMessage
subclasses compas.data.Data"] + pyc["antikythera.models.conversions
@pb_serializer · @pb_deserializer"] + pyr["compas_pb
registry · discovery · codec"] + pym --> pyc + pyc -->|"registers via
entry point"| pyr + end + + rel -.->|"bindings"| pym + rel -.->|"bindings"| tsm + + wire["MessageData bytes
one wire format, both directions"] + pyr ==> wire + tsr ==> wire + + classDef ghost stroke-dasharray:4 3 + class tsm,tsc,tsr ghost +``` + +/// caption +One domain model, two languages. Solid borders exist today; dashed borders are proposed. +The proto bundle is what keeps the two columns describing the same wire — without a +published artifact, each language's bindings drift independently. +/// + +The registration arrow is the whole point of the design: `compas_pb` contains no reference +to Antikythera, and Antikythera contains no codec logic, yet +`pb_dump_bts(TaskAssignmentMessage(...))` works. + +## The wire format + +Every message is a `MessageData` envelope carrying a version tag and exactly one `AnyData`. +`AnyData` is a `oneof`, so a value occupies exactly one arm. Which arm it occupies decides +what the reader gets back. + +```mermaid +flowchart LR + md["MessageData"] --> ver["version: string
e.g. 1.1.4"] + md --> any["AnyData
oneof data"] + + any --> m["message: Any
a registered type"] + any --> v["value: Value
null · bool · str · bytes"] + any --> f["fallback: FallbackData"] + any --> iv["int_value: int64"] + any --> dv["double_value: double"] + any --> dict["dict_value: DictData"] + any --> list["list_value: ListData"] + + dict -.->|"map<string, AnyData>
recurses"| any + list -.->|"repeated AnyData
recurses"| any + f -->|"DictData"| decoder["runs DataDecoder
reconstructs the object"] + dict --> plain["stays a plain dict"] + + classDef hot stroke-width:3px + class f,decoder hot +``` + +/// caption +The seven arms of `AnyData`. `dict_value` and `list_value` recurse back into `AnyData`, +which is what makes arbitrary nesting work. The thick path is the one that surprises +people: `fallback` is the **only** arm whose decode runs `DataDecoder` and reconstructs a +COMPAS object. An envelope-shaped dict sent as `dict_value` arrives as a bare dict. +/// + +Two consequences worth stating explicitly, because implementations get them wrong: + +- **`int_value` and `double_value` exist so numbers survive a round trip.** A + `google.protobuf.Value` coerces everything to double, so `3` returns as `3.0`. An integer + must use `int_value`; an integral float must use `double_value`. +- **`bytes` travel as a string** prefixed `base64:` inside `value`, since + `google.protobuf.Value` has no bytes kind. + +## Recursive dispatch + +A runtime's encoder is one recursive function. It is worth reading as a decision tree, +because every arm above corresponds to exactly one branch. + +```mermaid +flowchart TB + start(["serialize(obj)
re-entered by each recursion"]) --> islist{"list
or tuple?"} + islist -->|yes| lv["list_value
← recurse per item"] + islist -->|no| isdict{"dict?"} + isdict -->|yes| dv["dict_value
← recurse per value"] + isdict -->|no| reg{"registered
in registry?"} + reg -->|yes| msg["message: Any
pack with type_url"] + reg -->|no| isdata{"a domain object?
Python: isinstance Data"} + isdata -->|yes| fb["fallback
← serialize its dict form"] + isdata -->|no| prim{"primitive?"} + prim -->|"int"| iv["int_value"] + prim -->|"float"| ddv["double_value"] + prim -->|"bool · str · bytes · None"| val["value"] + prim -->|no| err(["raise TypeError"]) + + + classDef term stroke-width:2px + class lv,dv,msg,fb,iv,ddv,val term +``` + +/// caption +`compas_pb.core._serializer_any` as a decision tree. Registry lookup sits *between* the +container arms and the fallback arm — a registered type gets its native protobuf message, +and only an unregistered domain object degrades to `fallback`. The three container arms +re-enter at the top, once per item. + +Decoding is the mirror image, dispatching on which `oneof` arm is set. +/// + +The registry lookup in the middle is where plugins take effect. In Python the lookup walks +the type's MRO, so registering a serializer for a base class covers its subclasses. + +## Distributing schemas + +A domain model owner publishes its `.proto` bundle and generated bindings as release +artifacts. Consumers pin a version and download it. This replaces the pattern that grew up +by default, where every consumer wrote its own script to scrape `.proto` files out of a git +repository. + +```mermaid +flowchart LR + subgraph before["Today — each consumer rolls its own fetch"] + direction TB + g1["compas_pb
git repo"] + g2["antikythera
git repo"] + c1["compas_pb_ts
proto-sync.mjs"] + c2["compas_pb_csharp
fetch_compas_pb.py"] + c3["antikythera-frontend
update-protos.js"] + g1 -->|"pinned commit"| c1 + g1 -->|"release zip"| c2 + g1 -->|"tag v1.1.4"| c3 + g2 ==>|"branch main
unpinned"| c3 + end + + subgraph after["Proposed — one artifact contract"] + direction TB + r1["compas_pb
Release vX.Y.Z"] + r2["antikythera
Release vX.Y.Z"] + d1["compas_pb_ts"] + d2["compas_pb_csharp"] + d3["antikythera-frontend"] + r1 -->|"pinned"| d1 + r1 -->|"pinned"| d2 + r1 -->|"pinned"| d3 + r2 -->|"pinned"| d3 + end + + before ==>|"consolidate"| after +``` + +/// caption +Three consumers, three hand-written fetchers, three different pinning strategies — one of +which pins nothing at all. Consolidating on published artifacts makes the schema version an +explicit, auditable dependency. +/// + +Rules for the artifact set: + +- **Publish the `.proto` bundle.** It is what every downstream generator needs, and it is + the one artifact that lets a language without an official binding get started. +- **Publish generated bindings per supported language.** Priority languages are TypeScript, + C# and C++. +- **Skip a Python bindings artifact.** The generated `_pb2` modules already ship inside the + wheel, so a separate archive would duplicate what `pip` delivers. +- **Pin by version, never by branch.** A consumer tracking a branch silently adopts wire + changes at build time. + +## The contract for a language runtime + +To call itself a `compas_pb` runtime, a library must provide three things. + +1. **A unified entry point.** `pb_dump_bts` / `pb_load_bts` in Python, adapted to local + naming conventions elsewhere (`pbDump` / `pbLoad`). Callers pass a domain object and get + bytes, or pass bytes and get a domain object. They never branch on type. +2. **Recursive resolution.** Encoding and decoding must recurse through `dict_value` and + `list_value` so arbitrarily nested structures work, and must handle every arm of + `AnyData` — including writing `fallback`, not only reading it. +3. **A registration mechanism.** Third-party packages must be able to add types without + modifying the runtime. Automatic discovery is preferred where the language supports it; + explicit registration is acceptable where it does not. + +On the third point, languages differ in what they can honestly offer. Python enumerates +installed plugins through package metadata, so discovery is genuinely automatic and lazy. +JavaScript has no equivalent registry, and a bundled browser application cannot inspect its +own dependency tree at runtime — so TypeScript uses an explicit `register()` call. Keeping +the registration API separate from the discovery mechanism means a future build-time +discovery step can be added without changing how plugins declare themselves. + +## Status + +| Capability | Python | TypeScript | C# | +| --- | --- | --- | --- | +| Decode, recursive | yes | yes | yes | +| Encode, recursive | yes | **missing** | yes | +| Writes `fallback` | yes | **missing** | yes | +| Extensible registry | yes | **hardcoded map** | partial | +| Third-party registration | yes | **missing** | no | +| Automatic discovery | yes | not possible | no | + +`compas_pb_ts` can encode a single registered wrapper into an envelope via `pbDumpBytes`, +and every wrapper class can serialize itself to its own `*Data` bytes. What it lacks is the +recursive layer above them — the equivalent of `_serializer_any` — so it cannot yet encode a +dict, a list, a nested structure, or a `fallback`. diff --git a/mkdocs.yml b/mkdocs.yml index 956dd19..8d3db58 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -75,7 +75,11 @@ markdown_extensions: - pymdownx.inlinehilite - pymdownx.snippets: check_paths: true - - pymdownx.superfences + - pymdownx.superfences: + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format - toc: permalink: "¤" @@ -146,6 +150,7 @@ nav: - compas_pb.core: reference/compas_pb.core.md - compas_pb.registry: reference/compas_pb.registry.md - compas_pb.conversions: reference/compas_pb.conversions.md + - Architecture: architecture.md - Protobuf Definitions: protobuf.md - Other: - changelog.md From 39e75006d1d203969e6326ac88b9a560b417208c Mon Sep 17 00:00:00 2001 From: Gonzalo Casas Date: Wed, 19 Aug 2026 00:57:41 +0200 Subject: [PATCH 2/8] feat: publish the .proto bundle and TypeScript bindings on release Downstream generators need the schemas themselves, not only the bindings this repository happens to emit, so `create-proto-bundle` zips them for release upload. It is rooted at the include path, keeping the `compas_pb/generated/` prefix intact -- a flat archive would break every import of message.proto. TypeScript joins the generated languages via @bufbuild/protoc-gen-es, installed into a version-keyed cache the same way protoc is. protoc emits no TypeScript natively, so PROTO_PLUGIN_LANGUAGES maps a language to its plugin flag and javascript or go can be added the same way. Plugin-backed assets are pinned by plugin version rather than protoc version, since the plugin is what shapes the generated API. The task identity (package name, generated folder) is now configurable, so any package owning .proto files can reuse this pipeline; antikythera already imports generate_proto_classes from here. Two fixes along the way: - the stale-asset cleanup globbed dist/*.zip while assets are written to dist/proto/*.zip, so it matched nothing and old zips survived into the release upload - an unsupported target language printed a warning and then ran protoc with an empty --_out flag; it now raises Co-Authored-By: Claude Opus 5 --- .gitignore | 5 ++ src/compas_pb/invocations.py | 136 ++++++++++++++++++++++++++++++++--- tasks.py | 2 + 3 files changed, 132 insertions(+), 11 deletions(-) diff --git a/.gitignore b/.gitignore index 9181ddb..7228f48 100644 --- a/.gitignore +++ b/.gitignore @@ -152,3 +152,8 @@ cSharp/**/bin/* cSharp/**/obj/* *.bin + +# Transient per-language protobuf output: create_class_assets generates each language, +# zips it for release upload, then removes the folder. The Python bindings beside them +# are committed on purpose. +src/compas_pb/generated/*/ diff --git a/src/compas_pb/invocations.py b/src/compas_pb/invocations.py index 13e72ee..e4836e2 100644 --- a/src/compas_pb/invocations.py +++ b/src/compas_pb/invocations.py @@ -11,9 +11,18 @@ PROTOC_VERSION = "31.1" PROTOC_GEN_DOCS_VERSION = "1.5.1" -# typescript,javascript, and go need other compiler plugins +PROTOC_GEN_ES_VERSION = "2.14.0" + +# Languages protoc emits natively, with no extra plugin. PROTO_TARGET_LANGUAGES = ["cpp", "csharp", "java", "objc", "php", "ruby"] +# Languages that need a third-party protoc plugin. Maps the language name used in asset +# names to the protoc flag prefix the plugin registers (``--es_out`` for protobuf-es), so +# javascript and go can be added here the same way. +PROTO_PLUGIN_LANGUAGES = {"typescript": "es"} + +ALL_PROTO_TARGET_LANGUAGES = PROTO_TARGET_LANGUAGES + list(PROTO_PLUGIN_LANGUAGES) + _PROTOC_ARCH_MAPPING = { "x86_64": "x86_64", "aarch64": "aarch_64", @@ -97,6 +106,42 @@ def _download_and_extract_protoc(url, extract_path): archive_path.unlink() +def _get_cached_protoc_gen_es_path(): + cache_dir = Path.home() / ".cache" / "protoc-gen-es" / PROTOC_GEN_ES_VERSION + plugin_bin = cache_dir / "node_modules" / ".bin" / "protoc-gen-es" + if platform.system() == "Windows": + plugin_bin = plugin_bin.with_suffix(".cmd") + + return plugin_bin, cache_dir + + +def setup_protoc_gen_es(ctx): + """Install the protobuf-es code generator, and return the path to its executable. + + protoc has no native TypeScript output, so TypeScript bindings come from + ``@bufbuild/protoc-gen-es``. It is an npm package, so this needs node on PATH. The + install is cached per version alongside the protoc cache. + """ + plugin_bin, cache_dir = _get_cached_protoc_gen_es_path() + + if plugin_bin.exists(): + print(f"Using cached protoc-gen-es at: {plugin_bin}") + return plugin_bin + + print(f"protoc-gen-es not found in cache. Installing to: {cache_dir}") + cache_dir.mkdir(parents=True, exist_ok=True) + ctx.run( + f'npm install --silent --no-package-lock --prefix "{cache_dir}" ' + f"@bufbuild/protoc-gen-es@{PROTOC_GEN_ES_VERSION}" + ) + + if not plugin_bin.exists(): + raise FileNotFoundError(f"Failed to find protoc-gen-es after install: {plugin_bin}") + + print("Install complete.") + return plugin_bin + + def setup_protoc(): protoc_bin, cache_dir = _get_cached_protoc_path() plugin_executable = protoc_bin.parent / "protoc-gen-doc" @@ -135,20 +180,33 @@ def setup_protoc(): def generate_proto_classes(ctx, target_language: str = "python"): protoc_path, _ = setup_protoc() - proto_out_folder = "" if target_language == "python": proto_out_folder = Path(ctx.proto_out_folder) - elif target_language in PROTO_TARGET_LANGUAGES: - proto_out_folder = Path(ctx.proto_out_folder) / "compas_pb" / "generated" / target_language + elif target_language in ALL_PROTO_TARGET_LANGUAGES: + proto_out_folder = _generated_root(ctx) / target_language proto_out_folder.mkdir(parents=True, exist_ok=True) else: - print(f"Target language '{target_language}' not supported.") + supported = ", ".join(["python"] + ALL_PROTO_TARGET_LANGUAGES) + raise ValueError(f"Target language '{target_language}' not supported. Choose one of: {supported}") + + # Plugin-backed languages replace the --_out flag with the plugin's own flag, + # and need the plugin binary passed explicitly since it is not on PATH. + plugin_flag = PROTO_PLUGIN_LANGUAGES.get(target_language) + plugin_path = setup_protoc_gen_es(ctx) if plugin_flag == "es" else None for idl_file in ctx.proto_folder.glob("*.proto"): cmd = f"{protoc_path} " cmd += " ".join(f"--proto_path={p}" for p in ctx.proto_include_paths) - cmd += f" --{target_language}_out={proto_out_folder} {idl_file}" + if plugin_flag: + cmd += f' --plugin=protoc-gen-{plugin_flag}="{plugin_path}"' + cmd += f" --{plugin_flag}_out={proto_out_folder}" + if plugin_flag == "es": + # Emit TypeScript rather than JavaScript + .d.ts. + cmd += f" --{plugin_flag}_opt=target=ts" + cmd += f" {idl_file}" + else: + cmd += f" --{target_language}_out={proto_out_folder} {idl_file}" if target_language == "python": cmd += f" --pyi_out={proto_out_folder}" @@ -157,23 +215,79 @@ def generate_proto_classes(ctx, target_language: str = "python"): ctx.run(cmd) +def _package_name(ctx) -> str: + """Distribution name used to label release assets. + + Defaults to compas_pb, so any package that owns .proto files can reuse these tasks by + setting ``package_name`` in its invoke configuration. + """ + return ctx.config.get("package_name", "compas_pb") + + +def _generated_root(ctx) -> Path: + """Folder the per-language bindings are generated into before being zipped.""" + configured = ctx.config.get("generated_folder", None) + if configured: + return Path(configured) + return Path(ctx.proto_out_folder) / _package_name(ctx) / "generated" + + +def _asset_version(language: str) -> str: + """Version marker for a generated-bindings asset. + + Natively-emitted languages are pinned by the protoc that produced them. Plugin-backed + languages are pinned by the plugin version instead, since that is what shapes the + generated API — protoc only feeds it a descriptor. + """ + if language in PROTO_PLUGIN_LANGUAGES: + return PROTOC_GEN_ES_VERSION if PROTO_PLUGIN_LANGUAGES[language] == "es" else PROTOC_VERSION + return PROTOC_VERSION + + +@task() +def create_proto_bundle(ctx): + """Zip the .proto schemas themselves for release upload. + + Every downstream generator needs the schemas, not just the bindings compas_pb happens + to generate. Publishing them is what lets a consumer pin a schema version instead of + scraping this repository at some ref. + """ + dist_dir = Path(ctx.base_folder) / "dist" / "proto" + dist_dir.mkdir(parents=True, exist_ok=True) + + zip_path = dist_dir / f"{_package_name(ctx)}-proto.zip" + zip_path.unlink(missing_ok=True) + + proto_root = Path(ctx.proto_include_paths[0]) + with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as zipf: + for item in sorted(proto_root.rglob("*.proto")): + arcname = str(item.relative_to(proto_root)) + zipf.write(item, arcname) + print(f"Added {arcname}") + + print(f"Proto bundle ready: {zip_path}") + return zip_path + + @task() def create_class_assets(ctx): base_dir = ctx.base_folder dist_dir = base_dir / "dist" dist_dir.mkdir(parents=True, exist_ok=True) - for existing_file in dist_dir.glob("compas_pb-generated-*.zip"): + # Assets are written to dist/proto/, so clean there — globbing dist/ itself never + # matched anything and left stale zips behind for the release upload to pick up. + for existing_file in (dist_dir / "proto").glob(f"{_package_name(ctx)}-generated-*.zip"): existing_file.unlink() print(f"Removed existing asset: {existing_file}") - class_assests = [] + class_assests = [create_proto_bundle(ctx)] - for language in PROTO_TARGET_LANGUAGES: + for language in ALL_PROTO_TARGET_LANGUAGES: generate_proto_classes(ctx, target_language=language) - generated_dir = base_dir / "src" / "compas_pb" / "generated" / language - zip_path = dist_dir / "proto" / f"compas_pb-generated-{language}-{PROTOC_VERSION}.zip" + generated_dir = _generated_root(ctx) / language + zip_path = dist_dir / "proto" / f"{_package_name(ctx)}-generated-{language}-{_asset_version(language)}.zip" zip_path.parent.mkdir(parents=True, exist_ok=True) with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as zipf: for item in generated_dir.rglob("*"): diff --git a/tasks.py b/tasks.py index 7d032b2..db04466 100644 --- a/tasks.py +++ b/tasks.py @@ -7,6 +7,7 @@ from invoke.tasks import task from compas_pb.invocations import create_class_assets +from compas_pb.invocations import create_proto_bundle from compas_pb.invocations import generate_proto_classes from compas_pb.invocations import proto_docs @@ -28,6 +29,7 @@ def pre_build(ctx): pre_build, generate_proto_classes, create_class_assets, + create_proto_bundle, proto_docs, ) From ba4272ed35bfa441f5ed6e0cde4ce4afa7844a0e Mon Sep 17 00:00:00 2001 From: Gonzalo Casas Date: Wed, 19 Aug 2026 01:15:13 +0200 Subject: [PATCH 3/8] fix: accept str or Path in the invoke configuration Reusing these tasks from antikythera surfaced it: compas_pb configures base_folder as a Path, antikythera as a str from os.path.dirname, and create_class_assets did `base_dir / "dist"` on whatever it was given. Same for proto_folder. Both are normalized now, so a consumer can configure either. Co-Authored-By: Claude Opus 5 --- src/compas_pb/invocations.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/compas_pb/invocations.py b/src/compas_pb/invocations.py index e4836e2..f96c41c 100644 --- a/src/compas_pb/invocations.py +++ b/src/compas_pb/invocations.py @@ -194,7 +194,7 @@ def generate_proto_classes(ctx, target_language: str = "python"): plugin_flag = PROTO_PLUGIN_LANGUAGES.get(target_language) plugin_path = setup_protoc_gen_es(ctx) if plugin_flag == "es" else None - for idl_file in ctx.proto_folder.glob("*.proto"): + for idl_file in Path(ctx.proto_folder).glob("*.proto"): cmd = f"{protoc_path} " cmd += " ".join(f"--proto_path={p}" for p in ctx.proto_include_paths) @@ -271,7 +271,7 @@ def create_proto_bundle(ctx): @task() def create_class_assets(ctx): - base_dir = ctx.base_folder + base_dir = Path(ctx.base_folder) dist_dir = base_dir / "dist" dist_dir.mkdir(parents=True, exist_ok=True) From 25777ca4b9a81a0842c15daec46697a4f3329bcc Mon Sep 17 00:00:00 2001 From: Gonzalo Casas Date: Wed, 19 Aug 2026 08:27:06 +0200 Subject: [PATCH 4/8] docs: guide for implementing a runtime in a new language What a compas_pb_rust or compas_pb_go has to do to interoperate, what changes elsewhere when a language is added, and which parts of the wire format are easy to get subtly wrong. The traps section is the part worth having written down: each of the four produced a real bug while compas_pb_ts was being built, and none of them fail loudly. An integer arriving as a float, bytes surfacing as the literal string "base64:...", a domain object sent as dict_value and arriving as a bare dictionary because only the fallback arm reconstructs, and short-name type URL matching that works right up until two packages register the same class name. Also records the rules that came out of the same work: consume schemas from a pinned release artifact rather than vendoring them, resolve shared compas_pb types to the runtime package's copy rather than a second generated one, and test against bytes Python produced, since a round trip passes even when both directions are wrong in the same way. Co-Authored-By: Claude Opus 5 --- docs/implementing-a-runtime.md | 249 +++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 250 insertions(+) create mode 100644 docs/implementing-a-runtime.md diff --git a/docs/implementing-a-runtime.md b/docs/implementing-a-runtime.md new file mode 100644 index 0000000..d8d5b06 --- /dev/null +++ b/docs/implementing-a-runtime.md @@ -0,0 +1,249 @@ +# Implementing a runtime for a new language + +This page is for anyone building `compas_pb` support for a language that does not have it +yet — `compas_pb_rust`, `compas_pb_go`, and so on. It describes what the new package has to +do to interoperate, what has to change elsewhere, and which parts of the wire format are +easy to get subtly wrong. + +Read [Architecture](architecture.md) first for who owns what. In short: a **language +runtime** implements the registry and the codec for one language and knows nothing about +any domain model, while **domain-model owners** own `.proto` files and register their types +into whichever runtimes they support. + +## The contract + +A runtime must provide three things. + +1. **A unified entry point.** `pb_dump_bts` / `pb_load_bts` in Python, adapted to local + naming (`pbDump` / `pbLoad`, `pb_dump` / `pb_load`). A caller passes a domain object and + gets bytes, or passes bytes and gets a domain object back. Callers never branch on type. +2. **Recursive resolution.** Encoding and decoding recurse through `dict_value` and + `list_value`, and handle every arm of `AnyData` — including *writing* `fallback`, not + only reading it. +3. **A registration mechanism.** Third-party packages must be able to add types without + modifying the runtime. Automatic discovery where the language supports it, explicit + registration where it does not. + +Everything below is detail in service of those three. + +## The wire format, in the order you will implement it + +### The envelope + +Every message is a `MessageData` carrying a version string and one `AnyData`. Check the +version before trusting the payload, and reject rather than guess: + +``` +key(version) != key(own version) -> refuse to decode +``` + +The comparison uses a *wire-compatibility key*, not equality. Under `0.x` every minor +release may change the binary schema, so the key is `MAJOR.MINOR`; from `1.0` on, minor +releases stay compatible and the key is `MAJOR`. So `1.0.0` reads `1.2.9`, and `0.5.1` +reads `0.5.7` but not `0.6.0`. Mirror `compas_pb.core._wire_compat_key` exactly. + +A missing version tag is an error, not a default. + +### The seven arms + +`AnyData` is a `oneof`. Dispatch on which arm is set — never probe fields for +presence. Languages that model a `oneof` as a tagged union (Rust enums, protobuf-es +discriminated unions) get this for free; languages that model it as optional sibling fields +make it easy to write a check that silently reads the wrong arm. + +| Arm | Carries | Notes | +| --- | --- | --- | +| `message` | `google.protobuf.Any` | A registered type. Also the legacy home of containers. | +| `value` | `google.protobuf.Value` | null, bool, string — and bytes, see below | +| `fallback` | `FallbackData` | The only arm that reconstructs an object | +| `int_value` | `int64` | | +| `double_value` | `double` | | +| `dict_value` | `DictData` | Recurses | +| `list_value` | `ListData` | Recurses | + +### The four traps + +These are the places a new implementation tends to go wrong. Each one produced a real bug +while `compas_pb_ts` was being built. + +**Integers must not become floats.** `google.protobuf.Value` coerces every number to +double, which is why `int_value` and `double_value` exist. An integer goes in `int_value`; +a float goes in `double_value` *even when it is integral*, so a Python `3.0` comes back a +float rather than an int. + +This is the one rule a language cannot always honour. If your language has a single numeric +type that cannot distinguish `3` from `3.0` (JavaScript), say so in your documentation and +pick a consistent rule — `compas_pb_ts` sends an exact integer as `int_value`, so a Python +float that happens to be integral does not survive a round trip through the browser. A +language with distinct integer and float types (Rust, Go, C#) has no such problem and must +keep them distinct. + +**Bytes travel as a tagged string.** `google.protobuf.Value` has no bytes kind, so bytes +are base64-encoded into `string_value` with a `base64:` prefix. On decode, a string with +that prefix becomes bytes again. A runtime that skips this returns the literal string +`"base64:AAEC/w=="` to its callers. + +**`fallback` is the only arm that reconstructs.** `dict_value` decodes to a plain +dictionary — the decoder does *not* inspect it for a `{dtype, data}` envelope. Only +`fallback` runs the COMPAS decoder and rebuilds the object. So a runtime that sends a +domain object it has no registered serializer for must use `fallback`, or that object +arrives everywhere else as a bare dictionary. + +Python reaches this arm from a live `compas.data.Data` instance. A language with no COMPAS +class hierarchy has to answer the same question differently: `compas_pb_ts` treats a plain +object shaped `{dtype, data}` as the equivalent signal, because in a browser that is the +only form a COMPAS object ever takes. Decide what "this is a domain object" means in your +language, and put the check at the same point in the dispatch — after the registry lookup, +before the plain-dictionary arm. + +**Legacy containers still arrive.** Before the native `dict_value` and `list_value` arms +existed, containers were packed into `message` as `Any`-wrapped `ListData` and `DictData`. +Stored data still contains them, so decoding must handle both. Encoding should only ever +produce the native arms. + +### Type URLs + +A registered type is packed into `Any` with `type.googleapis.com/`. +Match on **everything after the final `/`**, as `type_url.rpartition("/")[2]` does — not on +the last dot-separated segment. Short-name matching appears to work while only `compas_pb` +types are registered, because those names are unique; it breaks the moment a domain-model +owner registers `antikythera.v1.TaskError` alongside some other `TaskError`. + +## Registration and discovery + +Registration is the mechanism a package uses to declare "this class maps to that protobuf +message". Discovery is how the runtime finds those declarations without being told. Keep +them separate: discovery is the part that varies by language, and separating them lets you +add discovery later without changing how plugins declare their types. + +Store *functions*, not a required class shape. Python's registry holds a serializer keyed +by type and a deserializer keyed by protobuf name, which lets a domain model stay free of +protobuf concerns. A registry that instead demands "your class must expose a `bytes` +property" forces every plugin to wrap its own model. + +Lookup on the way out should follow the language's inheritance chain, so registering a base +class covers its subclasses — Python walks the MRO; `compas_pb_ts` walks the prototype +chain; Rust, with no inheritance, needs no such walk. + +For discovery, offer what your language can honestly deliver: + +- **Python** enumerates installed plugins through packaging entry points, so discovery is + automatic and lazy. +- **TypeScript** has no equivalent, and a bundled browser application cannot inspect its + dependency tree at runtime, so registration is an explicit `registerAntikytheraTypes()` + call. Both alternatives — a side-effect import, or build-time codegen — can be silently + dropped by a bundler, which is worse than an explicit call. +- **Rust** can register at link time with a distributed-slice crate (`inventory`, + `linkme`), which is the closest thing to Python's behaviour: a dependent crate declares + its types and the runtime finds them with no call in `main`. +- **C#** can scan loaded assemblies for an attribute, with the usual caveat that an + assembly is not loaded until something touches it. + +Whatever you choose, document how a plugin author declares types and when they take effect. + +## Consuming the schemas + +**Do not vendor `.proto` files, and do not generate them yourself.** The package that owns +a schema publishes, on every release, the `.proto` bundle and generated bindings for each +supported language. Pin a version and download the artifact. Three separate consumers each +wrote their own git-scraping fetcher before this rule existed, and one of them tracked a +branch rather than a tag, so its build silently adopted wire changes. + +**Shared schemas must come from the runtime package, not a second copy.** A domain-model +owner's `.proto` files import `compas_pb`'s, so its generated code refers to +`compas_pb.data` types. Those must resolve to the *same* definitions the runtime uses. In +TypeScript this is load-bearing: protobuf-es links file descriptors by identity, so a +vendored second copy registers a competing descriptor for the same message and the two +halves disagree about types they are supposed to share. `compas_pb_ts` therefore exposes +its generated modules as subpath exports, and `antikythera_ts` rewrites its generated +imports to point at them. + +Check the equivalent in your language before assuming it does not apply: in Rust, two +crates each generating `compas_pb.data` produce two distinct, non-interchangeable types. +The fix is the same — depend on the runtime crate for the shared types. + +## What has to change elsewhere + +```mermaid +flowchart TB + new["compas_pb_rust
new runtime crate"] + + subgraph owner["compas_pb — one change"] + direction TB + lang["add the language to
PROTO_TARGET_LANGUAGES
or PROTO_PLUGIN_LANGUAGES"] + end + + subgraph downstream["Every domain-model owner — no change"] + direction TB + akt["antikythera"] + tmb["compas_timber"] + end + + lang ==>|"bindings now built
on every release"| new + lang -.->|"same task machinery,
so they publish it too"| akt + lang -.-> tmb + + akt ==>|"opt in by writing
a conversions module"| new + tmb ==>|"opt in"| new + + classDef ghost stroke-dasharray:4 3 + class new ghost +``` + +/// caption +Adding a language touches one place. Domain-model owners reuse compas_pb's task machinery, +so they start publishing bindings for the new language without any change of their own — +but a domain model only *reaches* the new language once someone writes its conversions +module. +/// + +**In `compas_pb`**, add the language to the generation task. If `protoc` emits it natively, +append it to `PROTO_TARGET_LANGUAGES` and you are done. If it needs a plugin — as Rust and +TypeScript both do — add it to `PROTO_PLUGIN_LANGUAGES` with the flag prefix its plugin +registers, and cache the plugin binary the way `setup_protoc_gen_es` does. Plugin-backed +languages are pinned by *plugin* version in the asset name, since the plugin is what shapes +the generated API. + +For Rust that means `protoc-gen-prost`, since `protoc` has no native Rust output. + +**In domain-model owners**, nothing has to change for the bindings to appear: they reuse +the same task machinery, so a language added in `compas_pb` is published by every owner on +its next release. Reaching the new language does require someone to write the conversions +module — the counterpart of `antikythera.models.conversions` — mapping that owner's domain +classes to their protobuf messages. + +**Nothing changes in the other runtimes.** They share a wire format, not code. + +## Proving conformance + +The wire format is the contract, so test against bytes rather than against your own +round trips. A round-trip test passes even when both directions are wrong in the same way. + +- **Decode a payload produced by Python.** Generate one with `pb_dump_bts`, commit it as a + base64 constant, and assert your runtime decodes it to the expected values. + `compas_pb_ts` does this with a real Antikythera task message, and it is the single most + useful test in that repository. +- **Have Python decode a payload produced by you.** The other direction catches encoding + bugs that a self-round-trip hides — an integral float sent as `int_value`, an envelope + sent as `dict_value` instead of `fallback`. +- **Cover each arm explicitly**, including the traps above: an integer and a float that + stay distinct, bytes through `base64:`, a `{dtype, data}` envelope landing in `fallback`, + a nested dict inside a list, a legacy `Any`-wrapped container decoding correctly, and an + empty `AnyData` resolving to null rather than throwing. +- **Cover a third-party registration**, so the plugin path is exercised by something other + than the built-in types. + +## Checklist + +- [ ] Envelope written with a version tag; version checked on read using the compatibility key +- [ ] All seven `AnyData` arms encoded and decoded, dispatching on the set arm +- [ ] Integers and floats kept distinct; documented if your language cannot +- [ ] Bytes through the `base64:` convention +- [ ] `fallback` written, not only read +- [ ] Legacy `Any`-wrapped `ListData` and `DictData` still decode +- [ ] Type URLs matched on the full name after the final `/` +- [ ] Registry stores functions and supports third-party registration +- [ ] Unified entry point that callers use without branching on type +- [ ] Generated code consumed from a pinned release artifact, never vendored +- [ ] Shared `compas_pb` types resolve to the runtime package's copy +- [ ] Conformance tests in both directions against Python-produced bytes diff --git a/mkdocs.yml b/mkdocs.yml index 8d3db58..6b9ffad 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -151,6 +151,7 @@ nav: - compas_pb.registry: reference/compas_pb.registry.md - compas_pb.conversions: reference/compas_pb.conversions.md - Architecture: architecture.md + - Implementing a Runtime: implementing-a-runtime.md - Protobuf Definitions: protobuf.md - Other: - changelog.md From bba050d6af8a0540ac09cba36a7739591793c341 Mon Sep 17 00:00:00 2001 From: Gonzalo Casas Date: Wed, 19 Aug 2026 08:33:21 +0200 Subject: [PATCH 5/8] docs: tighten the new-language guide Opens with why anyone would want this rather than with the contract, and cuts roughly a third of the words. The five failure modes are folded into one section, since what matters about each is the symptom and the rule, not the reasoning that led there. Co-Authored-By: Claude Opus 5 --- docs/implementing-a-runtime.md | 281 ++++++++++++--------------------- 1 file changed, 103 insertions(+), 178 deletions(-) diff --git a/docs/implementing-a-runtime.md b/docs/implementing-a-runtime.md index d8d5b06..9e4ea6d 100644 --- a/docs/implementing-a-runtime.md +++ b/docs/implementing-a-runtime.md @@ -1,59 +1,37 @@ # Implementing a runtime for a new language -This page is for anyone building `compas_pb` support for a language that does not have it -yet — `compas_pb_rust`, `compas_pb_go`, and so on. It describes what the new package has to -do to interoperate, what has to change elsewhere, and which parts of the wire format are -easy to get subtly wrong. +COMPAS models are authored in Python, but less and less of what consumes them is: a browser +agent driving a session, a Rhino plugin in C#, a controller in Rust. `compas_pb` is what +lets those exchange the same objects rather than a hand-rolled JSON dialect per pairing — a +`Frame` sent from Grasshopper arrives in a browser as a `Frame`. This page is what it takes +to add a language to that set. Read [Architecture](architecture.md) first for who owns what. In short: a **language -runtime** implements the registry and the codec for one language and knows nothing about -any domain model, while **domain-model owners** own `.proto` files and register their types -into whichever runtimes they support. +runtime** implements the registry and codec for one language and knows nothing about any +domain model; **domain-model owners** own `.proto` files and register their types into the +runtimes they support. ## The contract -A runtime must provide three things. +1. **A unified entry point** — `pb_dump_bts` / `pb_load_bts`, adapted to local naming. + Callers pass a domain object and get bytes, or the reverse. They never branch on type. +2. **Recursive resolution** — recurse through `dict_value` and `list_value`, and handle + every arm of `AnyData`, including *writing* `fallback`. +3. **A registration mechanism** — third parties add types without modifying the runtime. -1. **A unified entry point.** `pb_dump_bts` / `pb_load_bts` in Python, adapted to local - naming (`pbDump` / `pbLoad`, `pb_dump` / `pb_load`). A caller passes a domain object and - gets bytes, or passes bytes and gets a domain object back. Callers never branch on type. -2. **Recursive resolution.** Encoding and decoding recurse through `dict_value` and - `list_value`, and handle every arm of `AnyData` — including *writing* `fallback`, not - only reading it. -3. **A registration mechanism.** Third-party packages must be able to add types without - modifying the runtime. Automatic discovery where the language supports it, explicit - registration where it does not. +## The wire format -Everything below is detail in service of those three. +Every message is a `MessageData`: a version string and one `AnyData`. Check the version +before trusting the payload, using a compatibility key rather than equality. Under `0.x` +the key is `MAJOR.MINOR`; from `1.0` on it is `MAJOR`. So `1.0.0` reads `1.2.9`, and +`0.5.1` reads `0.5.7` but not `0.6.0`. Mirror `compas_pb.core._wire_compat_key`. A missing +version tag is an error, not a default. -## The wire format, in the order you will implement it - -### The envelope - -Every message is a `MessageData` carrying a version string and one `AnyData`. Check the -version before trusting the payload, and reject rather than guess: - -``` -key(version) != key(own version) -> refuse to decode -``` - -The comparison uses a *wire-compatibility key*, not equality. Under `0.x` every minor -release may change the binary schema, so the key is `MAJOR.MINOR`; from `1.0` on, minor -releases stay compatible and the key is `MAJOR`. So `1.0.0` reads `1.2.9`, and `0.5.1` -reads `0.5.7` but not `0.6.0`. Mirror `compas_pb.core._wire_compat_key` exactly. - -A missing version tag is an error, not a default. - -### The seven arms - -`AnyData` is a `oneof`. Dispatch on which arm is set — never probe fields for -presence. Languages that model a `oneof` as a tagged union (Rust enums, protobuf-es -discriminated unions) get this for free; languages that model it as optional sibling fields -make it easy to write a check that silently reads the wrong arm. +`AnyData` is a `oneof`. Dispatch on which arm is set; never probe fields for presence. | Arm | Carries | Notes | | --- | --- | --- | -| `message` | `google.protobuf.Any` | A registered type. Also the legacy home of containers. | +| `message` | `google.protobuf.Any` | A registered type; also the legacy home of containers | | `value` | `google.protobuf.Value` | null, bool, string — and bytes, see below | | `fallback` | `FallbackData` | The only arm that reconstructs an object | | `int_value` | `int64` | | @@ -61,108 +39,69 @@ make it easy to write a check that silently reads the wrong arm. | `dict_value` | `DictData` | Recurses | | `list_value` | `ListData` | Recurses | -### The four traps +## Five things that fail quietly -These are the places a new implementation tends to go wrong. Each one produced a real bug -while `compas_pb_ts` was being built. +Each of these produced a real bug while `compas_pb_ts` was being built. None of them throw. -**Integers must not become floats.** `google.protobuf.Value` coerces every number to -double, which is why `int_value` and `double_value` exist. An integer goes in `int_value`; -a float goes in `double_value` *even when it is integral*, so a Python `3.0` comes back a -float rather than an int. +**Integers becoming floats.** `google.protobuf.Value` coerces every number to double, which +is why the explicit arms exist. An integer goes in `int_value`; a float goes in +`double_value` *even when integral*. A language with distinct numeric types must keep them +distinct. A language without one (JavaScript) cannot fully honour this — say so, and pick a +consistent rule. -This is the one rule a language cannot always honour. If your language has a single numeric -type that cannot distinguish `3` from `3.0` (JavaScript), say so in your documentation and -pick a consistent rule — `compas_pb_ts` sends an exact integer as `int_value`, so a Python -float that happens to be integral does not survive a round trip through the browser. A -language with distinct integer and float types (Rust, Go, C#) has no such problem and must -keep them distinct. - -**Bytes travel as a tagged string.** `google.protobuf.Value` has no bytes kind, so bytes -are base64-encoded into `string_value` with a `base64:` prefix. On decode, a string with -that prefix becomes bytes again. A runtime that skips this returns the literal string -`"base64:AAEC/w=="` to its callers. +**Bytes as a tagged string.** `Value` has no bytes kind, so bytes are base64-encoded into +`string_value` behind a `base64:` prefix, and decoded back on the way in. Skip it and +callers get the literal string `"base64:AAEC/w=="`. **`fallback` is the only arm that reconstructs.** `dict_value` decodes to a plain -dictionary — the decoder does *not* inspect it for a `{dtype, data}` envelope. Only -`fallback` runs the COMPAS decoder and rebuilds the object. So a runtime that sends a -domain object it has no registered serializer for must use `fallback`, or that object -arrives everywhere else as a bare dictionary. - -Python reaches this arm from a live `compas.data.Data` instance. A language with no COMPAS -class hierarchy has to answer the same question differently: `compas_pb_ts` treats a plain -object shaped `{dtype, data}` as the equivalent signal, because in a browser that is the -only form a COMPAS object ever takes. Decide what "this is a domain object" means in your -language, and put the check at the same point in the dispatch — after the registry lookup, -before the plain-dictionary arm. - -**Legacy containers still arrive.** Before the native `dict_value` and `list_value` arms -existed, containers were packed into `message` as `Any`-wrapped `ListData` and `DictData`. -Stored data still contains them, so decoding must handle both. Encoding should only ever -produce the native arms. - -### Type URLs - -A registered type is packed into `Any` with `type.googleapis.com/`. -Match on **everything after the final `/`**, as `type_url.rpartition("/")[2]` does — not on -the last dot-separated segment. Short-name matching appears to work while only `compas_pb` -types are registered, because those names are unique; it breaks the moment a domain-model -owner registers `antikythera.v1.TaskError` alongside some other `TaskError`. +dictionary; the decoder never inspects it for a `{dtype, data}` envelope. A domain object +with no registered serializer must go out as `fallback` or it arrives as a bare dict. +Python reaches this arm from a live `Data` instance; TypeScript, which never holds one, +uses the `{dtype, data}` shape instead. Decide what "this is a domain object" means in your +language and put the check at the same point — after the registry lookup, before the +plain-dictionary arm. + +**Legacy containers.** Before the native container arms existed, containers were packed +into `message` as `Any`-wrapped `ListData` and `DictData`. Stored data still contains them, +so decode both; encode only the native arms. + +**Short type-URL matching.** Types are packed as +`type.googleapis.com/`. Match everything after the final `/`, not the +last dot-separated segment. Short names work until two packages register the same class +name. ## Registration and discovery -Registration is the mechanism a package uses to declare "this class maps to that protobuf -message". Discovery is how the runtime finds those declarations without being told. Keep -them separate: discovery is the part that varies by language, and separating them lets you -add discovery later without changing how plugins declare their types. - -Store *functions*, not a required class shape. Python's registry holds a serializer keyed -by type and a deserializer keyed by protobuf name, which lets a domain model stay free of -protobuf concerns. A registry that instead demands "your class must expose a `bytes` -property" forces every plugin to wrap its own model. - -Lookup on the way out should follow the language's inheritance chain, so registering a base -class covers its subclasses — Python walks the MRO; `compas_pb_ts` walks the prototype -chain; Rust, with no inheritance, needs no such walk. - -For discovery, offer what your language can honestly deliver: - -- **Python** enumerates installed plugins through packaging entry points, so discovery is - automatic and lazy. -- **TypeScript** has no equivalent, and a bundled browser application cannot inspect its - dependency tree at runtime, so registration is an explicit `registerAntikytheraTypes()` - call. Both alternatives — a side-effect import, or build-time codegen — can be silently - dropped by a bundler, which is worse than an explicit call. -- **Rust** can register at link time with a distributed-slice crate (`inventory`, - `linkme`), which is the closest thing to Python's behaviour: a dependent crate declares - its types and the runtime finds them with no call in `main`. -- **C#** can scan loaded assemblies for an attribute, with the usual caveat that an - assembly is not loaded until something touches it. - -Whatever you choose, document how a plugin author declares types and when they take effect. - -## Consuming the schemas - -**Do not vendor `.proto` files, and do not generate them yourself.** The package that owns -a schema publishes, on every release, the `.proto` bundle and generated bindings for each -supported language. Pin a version and download the artifact. Three separate consumers each -wrote their own git-scraping fetcher before this rule existed, and one of them tracked a -branch rather than a tag, so its build silently adopted wire changes. - -**Shared schemas must come from the runtime package, not a second copy.** A domain-model -owner's `.proto` files import `compas_pb`'s, so its generated code refers to -`compas_pb.data` types. Those must resolve to the *same* definitions the runtime uses. In -TypeScript this is load-bearing: protobuf-es links file descriptors by identity, so a -vendored second copy registers a competing descriptor for the same message and the two -halves disagree about types they are supposed to share. `compas_pb_ts` therefore exposes -its generated modules as subpath exports, and `antikythera_ts` rewrites its generated -imports to point at them. - -Check the equivalent in your language before assuming it does not apply: in Rust, two -crates each generating `compas_pb.data` produce two distinct, non-interchangeable types. -The fix is the same — depend on the runtime crate for the shared types. - -## What has to change elsewhere +Keep them separate: discovery is the part that varies by language, and separating them lets +you add it later without changing how plugins declare types. + +Store **functions**, not a required class shape — Python keys a serializer by type and a +deserializer by protobuf name, which keeps domain models free of protobuf concerns. A +registry demanding "expose a `bytes` property" forces every plugin to wrap its own model. +Lookup should follow the language's inheritance chain so a base registration covers +subclasses. + +For discovery, offer what your language can honestly deliver: Python enumerates packaging +entry points; Rust can register at link time (`inventory`, `linkme`); C# can scan loaded +assemblies. TypeScript gets an explicit `register()` call, because a bundler can silently +drop both a side-effect import and generated registration code — worse than a call you can +see. + +## Schemas: consume, never vendor + +Schema owners publish the `.proto` bundle and per-language bindings on every release. Pin a +version and download the artifact. Three consumers each wrote their own git-scraping +fetcher before this rule existed, and one tracked a branch, so its build silently adopted +wire changes. + +**Shared types must resolve to the runtime package's copy.** A domain owner's `.proto` +imports `compas_pb`'s, so its generated code refers to `compas_pb.data` types. In +TypeScript this is load-bearing — protobuf-es links descriptors by identity, so a vendored +second copy registers a competitor and the two halves disagree about types they share. Two +Rust crates each generating `compas_pb.data` likewise produce non-interchangeable types. +Depend on the runtime package for them. + +## What changes elsewhere ```mermaid flowchart TB @@ -192,58 +131,44 @@ flowchart TB /// caption Adding a language touches one place. Domain-model owners reuse compas_pb's task machinery, -so they start publishing bindings for the new language without any change of their own — -but a domain model only *reaches* the new language once someone writes its conversions -module. +so they publish bindings for the new language without any change of their own — but a +domain model only *reaches* it once someone writes its conversions module. /// -**In `compas_pb`**, add the language to the generation task. If `protoc` emits it natively, -append it to `PROTO_TARGET_LANGUAGES` and you are done. If it needs a plugin — as Rust and -TypeScript both do — add it to `PROTO_PLUGIN_LANGUAGES` with the flag prefix its plugin -registers, and cache the plugin binary the way `setup_protoc_gen_es` does. Plugin-backed -languages are pinned by *plugin* version in the asset name, since the plugin is what shapes -the generated API. +In **`compas_pb`**, add the language to the generation task: `PROTO_TARGET_LANGUAGES` if +`protoc` emits it natively, otherwise `PROTO_PLUGIN_LANGUAGES` with its plugin's flag +prefix, caching the binary as `setup_protoc_gen_es` does. Rust needs `protoc-gen-prost`; +plugin-backed languages are pinned by *plugin* version in the asset name, since the plugin +shapes the generated API. -For Rust that means `protoc-gen-prost`, since `protoc` has no native Rust output. +In **domain-model owners**, nothing changes for bindings to appear. Reaching the new +language does need someone to write the conversions module — the counterpart of +`antikythera.models.conversions`. -**In domain-model owners**, nothing has to change for the bindings to appear: they reuse -the same task machinery, so a language added in `compas_pb` is published by every owner on -its next release. Reaching the new language does require someone to write the conversions -module — the counterpart of `antikythera.models.conversions` — mapping that owner's domain -classes to their protobuf messages. - -**Nothing changes in the other runtimes.** They share a wire format, not code. +In **other runtimes**, nothing. They share a wire format, not code. ## Proving conformance -The wire format is the contract, so test against bytes rather than against your own -round trips. A round-trip test passes even when both directions are wrong in the same way. - -- **Decode a payload produced by Python.** Generate one with `pb_dump_bts`, commit it as a - base64 constant, and assert your runtime decodes it to the expected values. - `compas_pb_ts` does this with a real Antikythera task message, and it is the single most - useful test in that repository. -- **Have Python decode a payload produced by you.** The other direction catches encoding - bugs that a self-round-trip hides — an integral float sent as `int_value`, an envelope - sent as `dict_value` instead of `fallback`. -- **Cover each arm explicitly**, including the traps above: an integer and a float that - stay distinct, bytes through `base64:`, a `{dtype, data}` envelope landing in `fallback`, - a nested dict inside a list, a legacy `Any`-wrapped container decoding correctly, and an - empty `AnyData` resolving to null rather than throwing. -- **Cover a third-party registration**, so the plugin path is exercised by something other - than the built-in types. +Test against bytes, not your own round trips — a round trip passes when both directions are +wrong in the same way. + +- Decode a payload produced by Python's `pb_dump_bts`, committed as a base64 constant. + `compas_pb_ts` does this with a real task message; it is the most useful test there. +- Have Python decode a payload produced by you. Catches encoding bugs a self-round-trip + hides. +- Cover each arm, including the five above, and a third-party registration. ## Checklist -- [ ] Envelope written with a version tag; version checked on read using the compatibility key -- [ ] All seven `AnyData` arms encoded and decoded, dispatching on the set arm -- [ ] Integers and floats kept distinct; documented if your language cannot +- [ ] Version tag written and checked with the compatibility key +- [ ] All seven arms encoded and decoded, dispatching on the set arm +- [ ] Integers and floats kept distinct, or the limitation documented - [ ] Bytes through the `base64:` convention - [ ] `fallback` written, not only read -- [ ] Legacy `Any`-wrapped `ListData` and `DictData` still decode +- [ ] Legacy `Any`-wrapped containers still decode - [ ] Type URLs matched on the full name after the final `/` -- [ ] Registry stores functions and supports third-party registration -- [ ] Unified entry point that callers use without branching on type -- [ ] Generated code consumed from a pinned release artifact, never vendored +- [ ] Registry stores functions and accepts third-party registration +- [ ] Unified entry point, no type branching by callers +- [ ] Bindings consumed from a pinned release artifact, never vendored - [ ] Shared `compas_pb` types resolve to the runtime package's copy -- [ ] Conformance tests in both directions against Python-produced bytes +- [ ] Conformance tested in both directions against Python bytes From dc85e98d53414e5e8ed737de96969fa0b616378a Mon Sep 17 00:00:00 2001 From: Gonzalo Casas Date: Wed, 19 Aug 2026 08:38:25 +0200 Subject: [PATCH 6/8] docs: plainer wording, and drop the war stories The failure-modes section read as a list of bugs we once hit, which is not useful to someone starting a runtime. The rules in it are, though -- a runtime that does not know bytes are base64-tagged, or that only `fallback` rebuilds an object, looks fine and quietly corrupts data. They are now four short lines next to the arms they belong to, phrased as rules rather than as history. The same went for the anecdote about consumers scraping git. Reworded throughout for readers who code but are not full-time developers: shorter sentences, fewer clauses, plainer words. Co-Authored-By: Claude Opus 5 --- docs/implementing-a-runtime.md | 245 ++++++++++++++++----------------- 1 file changed, 117 insertions(+), 128 deletions(-) diff --git a/docs/implementing-a-runtime.md b/docs/implementing-a-runtime.md index 9e4ea6d..5964cfe 100644 --- a/docs/implementing-a-runtime.md +++ b/docs/implementing-a-runtime.md @@ -1,107 +1,96 @@ # Implementing a runtime for a new language -COMPAS models are authored in Python, but less and less of what consumes them is: a browser -agent driving a session, a Rhino plugin in C#, a controller in Rust. `compas_pb` is what -lets those exchange the same objects rather than a hand-rolled JSON dialect per pairing — a -`Frame` sent from Grasshopper arrives in a browser as a `Frame`. This page is what it takes -to add a language to that set. - -Read [Architecture](architecture.md) first for who owns what. In short: a **language -runtime** implements the registry and codec for one language and knows nothing about any -domain model; **domain-model owners** own `.proto` files and register their types into the -runtimes they support. - -## The contract - -1. **A unified entry point** — `pb_dump_bts` / `pb_load_bts`, adapted to local naming. - Callers pass a domain object and get bytes, or the reverse. They never branch on type. -2. **Recursive resolution** — recurse through `dict_value` and `list_value`, and handle - every arm of `AnyData`, including *writing* `fallback`. -3. **A registration mechanism** — third parties add types without modifying the runtime. +COMPAS models are written in Python, but more and more of what uses them is not: a browser +agent running a session, a Rhino plugin in C#, a controller in Rust. `compas_pb` is what +lets all of those pass the same objects around, instead of inventing a JSON format for +every new pair of tools. A `Frame` sent from Grasshopper shows up in the browser as a +`Frame`. This page explains how to add a language to that set. + +Have a look at [Architecture](architecture.md) first, for who owns what. The short version: +a **language runtime** implements the registry and the codec for one language, and knows +nothing about any domain model. **Domain-model owners** own `.proto` files and register +their types with the runtimes they care about. + +## What your runtime has to do + +1. **One entry point in, one out.** `pb_dump_bts` and `pb_load_bts`, named to suit your + language. People hand you an object and get bytes, or hand you bytes and get an object. + They should never have to check the type themselves. +2. **Handle nesting.** Walk into `dict_value` and `list_value`, and support every arm of + `AnyData` — including *writing* `fallback`, not just reading it. +3. **Let other packages register types**, without them having to edit your code. ## The wire format -Every message is a `MessageData`: a version string and one `AnyData`. Check the version -before trusting the payload, using a compatibility key rather than equality. Under `0.x` -the key is `MAJOR.MINOR`; from `1.0` on it is `MAJOR`. So `1.0.0` reads `1.2.9`, and -`0.5.1` reads `0.5.7` but not `0.6.0`. Mirror `compas_pb.core._wire_compat_key`. A missing -version tag is an error, not a default. +A message is a `MessageData`: a version string plus one `AnyData`. + +Check the version before you trust anything else. Don't compare the versions directly — +compare their compatibility keys. Under `0.x` the key is `MAJOR.MINOR`, and from `1.0` +onwards it is just `MAJOR`. So `1.0.0` can read `1.2.9`, and `0.5.1` can read `0.5.7` but +not `0.6.0`. `_wire_compat_key` in `compas_pb.core` has the logic to copy. If there is no +version at all, that is an error — don't assume one. -`AnyData` is a `oneof`. Dispatch on which arm is set; never probe fields for presence. +`AnyData` is a `oneof`, so exactly one of these is set. Look at which one it is and switch +on that, rather than testing each field to see if it has a value. -| Arm | Carries | Notes | +| Arm | Holds | Worth knowing | | --- | --- | --- | -| `message` | `google.protobuf.Any` | A registered type; also the legacy home of containers | -| `value` | `google.protobuf.Value` | null, bool, string — and bytes, see below | -| `fallback` | `FallbackData` | The only arm that reconstructs an object | -| `int_value` | `int64` | | -| `double_value` | `double` | | -| `dict_value` | `DictData` | Recurses | -| `list_value` | `ListData` | Recurses | - -## Five things that fail quietly - -Each of these produced a real bug while `compas_pb_ts` was being built. None of them throw. - -**Integers becoming floats.** `google.protobuf.Value` coerces every number to double, which -is why the explicit arms exist. An integer goes in `int_value`; a float goes in -`double_value` *even when integral*. A language with distinct numeric types must keep them -distinct. A language without one (JavaScript) cannot fully honour this — say so, and pick a -consistent rule. - -**Bytes as a tagged string.** `Value` has no bytes kind, so bytes are base64-encoded into -`string_value` behind a `base64:` prefix, and decoded back on the way in. Skip it and -callers get the literal string `"base64:AAEC/w=="`. - -**`fallback` is the only arm that reconstructs.** `dict_value` decodes to a plain -dictionary; the decoder never inspects it for a `{dtype, data}` envelope. A domain object -with no registered serializer must go out as `fallback` or it arrives as a bare dict. -Python reaches this arm from a live `Data` instance; TypeScript, which never holds one, -uses the `{dtype, data}` shape instead. Decide what "this is a domain object" means in your -language and put the check at the same point — after the registry lookup, before the -plain-dictionary arm. - -**Legacy containers.** Before the native container arms existed, containers were packed -into `message` as `Any`-wrapped `ListData` and `DictData`. Stored data still contains them, -so decode both; encode only the native arms. - -**Short type-URL matching.** Types are packed as -`type.googleapis.com/`. Match everything after the final `/`, not the -last dot-separated segment. Short names work until two packages register the same class -name. - -## Registration and discovery - -Keep them separate: discovery is the part that varies by language, and separating them lets -you add it later without changing how plugins declare types. - -Store **functions**, not a required class shape — Python keys a serializer by type and a -deserializer by protobuf name, which keeps domain models free of protobuf concerns. A -registry demanding "expose a `bytes` property" forces every plugin to wrap its own model. -Lookup should follow the language's inheritance chain so a base registration covers -subclasses. - -For discovery, offer what your language can honestly deliver: Python enumerates packaging -entry points; Rust can register at link time (`inventory`, `linkme`); C# can scan loaded -assemblies. TypeScript gets an explicit `register()` call, because a bundler can silently -drop both a side-effect import and generated registration code — worse than a call you can -see. - -## Schemas: consume, never vendor - -Schema owners publish the `.proto` bundle and per-language bindings on every release. Pin a -version and download the artifact. Three consumers each wrote their own git-scraping -fetcher before this rule existed, and one tracked a branch, so its build silently adopted -wire changes. - -**Shared types must resolve to the runtime package's copy.** A domain owner's `.proto` -imports `compas_pb`'s, so its generated code refers to `compas_pb.data` types. In -TypeScript this is load-bearing — protobuf-es links descriptors by identity, so a vendored -second copy registers a competitor and the two halves disagree about types they share. Two -Rust crates each generating `compas_pb.data` likewise produce non-interchangeable types. -Depend on the runtime package for them. - -## What changes elsewhere +| `message` | `google.protobuf.Any` | A registered type. Older data also puts lists and dicts here | +| `value` | `google.protobuf.Value` | null, bool and string. Bytes live here too, base64-encoded behind a `base64:` prefix | +| `fallback` | `FallbackData` | The only arm that rebuilds a real object rather than a dict | +| `int_value` | `int64` | Whole numbers | +| `double_value` | `double` | Floats, even when the value is round, like `3.0` | +| `dict_value` | `DictData` | Nests | +| `list_value` | `ListData` | Nests | + +Four rules that are easy to miss: + +- If your language has separate integer and float types, keep them apart. If it does not + (JavaScript), you cannot fully honour this — pick a rule and write it down. +- Decode lists and dicts in both shapes, old and new, but only ever encode the new ones. +- Match type URLs on everything after the last `/`, not on the last piece after a dot. + Short names seem fine until two packages both register a class called `TaskError`. +- An object with no registered serializer has to go out as `fallback`, or it arrives as a + plain dict. Python knows it has an object because it holds a live `Data` instance; + TypeScript never does, so it looks for the `{dtype, data}` shape instead. Decide what + that question means in your language, and ask it in the same place: after the registry + lookup, before "this is just a dictionary". + +## Registering types, and finding them + +These are two separate jobs. Registering is a package saying "this class maps to that +protobuf message". Discovery is your runtime finding those declarations on its own. Split +them, and you can add discovery later without changing how packages declare things. + +Store **functions**, not a required class shape. Python keeps a serializer per type and a +deserializer per protobuf name, which means a domain model never has to know protobuf +exists. If your registry instead insists that every registered class has, say, a `bytes` +property, then every package has to wrap its own model just to register it. When you look a +type up on the way out, follow your language's inheritance chain, so registering a base +class also covers everything below it. + +For discovery, do whatever your language can actually support. Python lists installed +plugins through packaging entry points. Rust can register at link time with `inventory` or +`linkme`. C# can scan the assemblies it has loaded. TypeScript gets a plain +`register()` call you make yourself, because a bundler can quietly drop a side-effect +import — and a call you can see beats one that vanishes. + +## Getting the schemas + +Don't copy `.proto` files into your repo, and don't generate them yourself. Whoever owns a +schema publishes the `.proto` bundle and the generated bindings for each language on every +release. Pin a version, download the artifact, done. Pulling files straight out of git +works right up until you follow a branch and your build quietly picks up a wire change. + +**Shared types have to come from the runtime package.** A domain owner's `.proto` imports +`compas_pb`'s, so its generated code refers to `compas_pb.data` types. Those need to be the +same types your runtime uses. In TypeScript this really matters: protobuf-es matches file +descriptors by identity, so a second copy registers a rival definition and the two halves +stop agreeing about types they are meant to share. Two Rust crates that each generate +`compas_pb.data` end up with two types that will not talk to each other. Depend on the +runtime package for them. + +## What changes everywhere else ```mermaid flowchart TB @@ -130,45 +119,45 @@ flowchart TB ``` /// caption -Adding a language touches one place. Domain-model owners reuse compas_pb's task machinery, -so they publish bindings for the new language without any change of their own — but a -domain model only *reaches* it once someone writes its conversions module. +Adding a language touches one place. Domain-model owners share compas_pb's build tasks, so +they start publishing bindings for the new language on their own — but a domain model only +*reaches* that language once someone writes its conversions module. /// -In **`compas_pb`**, add the language to the generation task: `PROTO_TARGET_LANGUAGES` if -`protoc` emits it natively, otherwise `PROTO_PLUGIN_LANGUAGES` with its plugin's flag -prefix, caching the binary as `setup_protoc_gen_es` does. Rust needs `protoc-gen-prost`; -plugin-backed languages are pinned by *plugin* version in the asset name, since the plugin -shapes the generated API. +In **`compas_pb`**, add your language to the generation task. If `protoc` can emit it +directly, add it to `PROTO_TARGET_LANGUAGES`. If it needs a plugin, add it to +`PROTO_PLUGIN_LANGUAGES` with the flag its plugin uses, and cache the binary the way +`setup_protoc_gen_es` does. Rust needs `protoc-gen-prost`. Languages that need a plugin are +tagged with the *plugin* version in the asset name, since the plugin is what decides how +the generated code looks. -In **domain-model owners**, nothing changes for bindings to appear. Reaching the new -language does need someone to write the conversions module — the counterpart of -`antikythera.models.conversions`. +In **domain-model owners**, nothing has to change for the bindings to start appearing. For +a domain model to actually reach your language, someone has to write the conversions module +— the equivalent of `antikythera.models.conversions`. -In **other runtimes**, nothing. They share a wire format, not code. +In **other runtimes**, nothing at all. They share a wire format, not code. -## Proving conformance +## Testing it -Test against bytes, not your own round trips — a round trip passes when both directions are -wrong in the same way. +Test against real bytes, not just your own round trips. A round trip still passes if you +get both directions wrong in the same way. -- Decode a payload produced by Python's `pb_dump_bts`, committed as a base64 constant. - `compas_pb_ts` does this with a real task message; it is the most useful test there. -- Have Python decode a payload produced by you. Catches encoding bugs a self-round-trip - hides. -- Cover each arm, including the five above, and a third-party registration. +- Take a payload from Python's `pb_dump_bts`, commit it as a base64 string, and check you + decode it correctly. This is the single most useful test you can write. +- Then go the other way: have Python read something you wrote. +- Cover every arm, plus a type registered by a package other than your own. ## Checklist -- [ ] Version tag written and checked with the compatibility key -- [ ] All seven arms encoded and decoded, dispatching on the set arm -- [ ] Integers and floats kept distinct, or the limitation documented -- [ ] Bytes through the `base64:` convention -- [ ] `fallback` written, not only read -- [ ] Legacy `Any`-wrapped containers still decode -- [ ] Type URLs matched on the full name after the final `/` -- [ ] Registry stores functions and accepts third-party registration -- [ ] Unified entry point, no type branching by callers -- [ ] Bindings consumed from a pinned release artifact, never vendored -- [ ] Shared `compas_pb` types resolve to the runtime package's copy -- [ ] Conformance tested in both directions against Python bytes +- [ ] Version written, and checked with the compatibility key +- [ ] All seven arms read and written, switching on the one that is set +- [ ] Whole numbers and floats stay apart, or the limitation is written down +- [ ] Bytes go through the `base64:` prefix +- [ ] `fallback` is written, not only read +- [ ] Old `Any`-wrapped lists and dicts still decode +- [ ] Type URLs matched on the full name after the last `/` +- [ ] Registry holds functions, and other packages can register types +- [ ] One entry point each way, with no type checking left to callers +- [ ] Bindings come from a pinned release, not copied into the repo +- [ ] Shared `compas_pb` types come from the runtime package +- [ ] Tested both directions against bytes Python produced From d9ed51eab06496080777ce3fc5ec3fc6fe95b38c Mon Sep 17 00:00:00 2001 From: Gonzalo Casas Date: Wed, 19 Aug 2026 09:13:47 +0200 Subject: [PATCH 7/8] refactor: use the release-assets action for the asset check Same four steps as antikythera had, now the shared action. Co-Authored-By: Claude Opus 5 --- .github/workflows/build.yml | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 6c95994..1b9b1c5 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -44,22 +44,11 @@ jobs: needs: build runs-on: ubuntu-latest steps: - - uses: actions/checkout@v7.0.1 - with: - persist-credentials: false - - - uses: compas-dev/compas-actions/setup-python@v1 + - uses: compas-dev/compas-actions/release-assets@v1 with: python-version: "3.12" management-tool: uv extras: dev - - - name: Test asset creation - run: invoke pre-build - - - name: Upload test artifacts - uses: actions/upload-artifact@v7.0.1 - with: - name: protobuf-test-assets - path: dist/proto/*.zip - if-no-files-found: error + invoke-tasks: pre-build + paths: dist/proto/*.zip + artifact-name: protobuf-test-assets From 28a8f933f53ffcc53c5e34b660ed4f00fcf5b773 Mon Sep 17 00:00:00 2001 From: Gonzalo Casas Date: Wed, 19 Aug 2026 10:16:32 +0200 Subject: [PATCH 8/8] docs: record this branch in the changelog Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 34b9f0b..19a902a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +* Added `create-proto-bundle`, which zips the `.proto` schemas for upload as a release asset, so downstream generators can pin a schema version instead of reading files out of this repository. +* Added TypeScript to the generated bindings, via `@bufbuild/protoc-gen-es`. `PROTO_PLUGIN_LANGUAGES` maps a language to the flag its protoc plugin registers, so javascript and go can be added the same way. +* Added an Architecture page describing how one domain model reaches many languages, and who owns `.proto` files versus who implements a runtime. +* Added a guide for implementing a `compas_pb` runtime in a new language. + ### Changed +* Changed the asset tasks to take their package name and output folder from the invoke configuration, so any package that owns `.proto` files can reuse them. `base_folder` and `proto_folder` now accept a string or a `Path`. +* Changed assets built by a protoc plugin to carry the plugin version rather than the protoc version, since the plugin is what shapes the generated API. +* Changed `generate-proto-classes` to raise on an unsupported target language, instead of warning and then running protoc with an empty output flag. +* Changed the asset job to use `compas-dev/compas-actions/release-assets@v1`. + ### Removed