diff --git a/README.md b/README.md index ccb1d64..45eacca 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,7 @@ refiners, so no target's limitations leak backward into the shared representatio [Package layout](#package-layout) · [Design docs](#design-docs) · [Building](#building) · +[Testing and verification](#testing-and-verification) · [License](#license) ## Pipeline @@ -54,8 +55,10 @@ spec ──▶ compiler ──▶ IR ──▶ passes ──▶ IR ──▶ emi - **Compilers** (`compilers/*`) turn one source format into an IR document plus diagnostics. OpenAPI 3.x ships first; Swagger 2.0, TypeSpec, Smithy, GraphQL, AsyncAPI, Protobuf, and Erlang/OTP are planned against the same IR. -- **Passes** (`pass/`) are small, order-explicit IR → IR transforms (validate, dedup, filter, - version-slice, overlay). `validate` — referential integrity — runs by default. +- **Passes** (`pass/`) are small, order-explicit IR → IR transforms. `validate` — referential + integrity — is the only one implemented, and runs by default; `link`, `dedup`, `filter`, + `version-slice` and `overlay` are designed in [architecture.md](docs/architecture.md) §2.2 and + not built yet. - **Emitters** (`emitters/*`, future) turn an IR document into artifacts for one target. SDK runtime policy (retry, timeout, telemetry, error taxonomy) is a *separate* emitter input, not part of the IR. @@ -119,9 +122,26 @@ The flags below are `compile`'s: | `--skip-validate` | Skip the referential-integrity `validate` pass. | | `--explain ` | Report what compiling produced at this source coordinate instead of writing the document. | -Diagnostics print one per line as ` #: `. Exit codes: -`0` clean (and for any help request), `1` a diagnostic reached the `--fail-on` threshold (or the -spec could not be lowered), `2` a usage or I/O error. +Diagnostics print one per line to stderr, in one of two shapes. A diagnostic that names a source +file renders as ` #: `; one that does not — a `validate` +finding, whose pointer is an IR ID rather than a source coordinate — renders as +` : `, rather than fabricating a location in the spec. + +The `` is whatever the stage that raised the diagnostic recorded, so it is not always a +JSON pointer. All three of these come out of a single compile: + +``` +error openapi/validation/validation-allowed-values api.yaml#14:19: [14:19] error ... +error openapi/unresolved-ref api.yaml#: not found -- key Missing not found ... +info openapi/validation-only-keyword api.yaml#/components/schemas/Odd: validation-only keyword ... +``` + +A JSON pointer is the usual case; a `:` appears when the fact came from the upstream +spec validator, which reports positions rather than pointers; an empty pointer appears when the +position is not recoverable. Match on the code, never on the pointer's shape. + +Exit codes: `0` clean (and for any help request), `1` a diagnostic reached the `--fail-on` +threshold (or the spec could not be lowered), `2` a usage or I/O error. ### Library @@ -181,23 +201,138 @@ The design documents are normative — read them before proposing changes to the | [Emitter design](docs/emitter-design.md) | The emitter contract and the plan / refine / emit boundary. | | [Prior art](docs/prior-art.md) | Lessons taken from oagen, Kiota, and TypeSpec/TCGC, and the mistakes each Morphic decision avoids. | | [Reference learnings](docs/reference-learnings.md) | Detailed notes from the reference codebases studied during design. | +| [Micro-compiler design](docs/micro-compiler-design.md) | The restructuring that replaced the OpenAPI compiler's god object with its internal packages, and the caps that stop it regrowing. | +| [Micro-compiler plan](docs/micro-compiler-plan.md) | The order that restructuring landed in, kept as a record of how the pieces blocked each other. | ## Building -Standard Go tooling. These are the same checks the CI `gate` runs, and all must pass before a -change lands: +Standard Go tooling. These are the same checks the CI `gate` job runs +([`.github/workflows/gate.yml`](.github/workflows/gate.yml)), in this order, and all must pass +before a change lands: ```bash -gofmt -l . # must print nothing +gofmt -l $(git ls-files '*.go') # must print nothing go vet ./... golangci-lint run go build ./... -go test ./... -./scripts/check-coverage.sh # enforces 100% statement coverage, overall and per package +./scripts/check-coverage.sh # runs go test ./... and enforces 100% statement coverage +``` + +`gofmt` is scoped to tracked files on purpose: `gofmt -l .` also walks git-ignored directories, so +a checkout carrying any ignored Go tree gets hundreds of lines from a command that is supposed to +print nothing. + +Coverage is a gate at exactly 100%, not a target. `check-coverage.sh` counts statements from the +profile rather than reading `go test`'s rounded percentage — a package at 99.96% prints +`100.0%` — so one uncovered statement fails the build, and `go test ./...` passing locally is not +evidence the gate passes. + +## Testing and verification + +`go test ./...` runs everything. The workflows below are the ones that need a different command, +and each answers a question the plain run does not. + +### One test, one package + +```bash +go test ./ir -run TestNewBigVal_AcceptsDecimalForms # one test +go test ./compilers/openapi -count=1 # one package, test cache bypassed +``` + +### The oracle sweep + +`internal/harness` drives a spec through six oracles in order: no panic, no error diagnostic, +`irverify`'s structural invariants, JSON round-trip, determinism across two compiles, and +order-independence — a recompile with every mapping's entry order reversed, diffed against the +first. That last one is what catches two declarations minting one node at a single pointer, which +a single-order test cannot see. [architecture.md](docs/architecture.md) §5 explains why. + +```bash +go run ./cmd/morphic-harness testdata/conformance/openapi # a directory, walked recursively +go run ./cmd/morphic-harness path/to/spec.yaml # or one spec +``` + +Exit status is `0` when every spec passes, `1` when any fails an oracle, and `2` on a usage or +filesystem error, so it works as a script gate. Run it against whatever spec provoked a compiler +change before submitting it. The same sweep over every committed spec runs as a test: + +```bash +go test ./internal/harness +``` + +### Golden IR snapshots + +`spec → IR → JSON` is snapshot-compared byte for byte. Regenerate after an intentional change: + +```bash +go test ./compilers/openapi -run TestConformance -update +``` + +`-update` is registered by `ir/irtest` rather than by the test framework, so only packages whose +tests import it accept the flag — `go test ./... -update` fails with +`flag provided but not defined`. To see which packages take it: + +```bash +go list -f '{{.ImportPath}} {{.TestImports}} {{.XTestImports}}' ./... | grep ir/irtest | cut -d' ' -f1 +``` + +A golden diff is something to read, not to regenerate reflexively: it is the only place an +unintended IR change surfaces. `TestConformance` will not rewrite a golden whose focused +capability assertion is failing, so `-update` cannot paper over a broken lowering. + +### The conformance corpus + +`testdata/conformance/openapi/` holds one minimal spec per row of +[`docs/ir-spec-matrix.md`](docs/ir-spec-matrix.md), each asserting the IR captures that capability +losslessly. This is what keeps *lossless by default* honest. To add a case: + +1. Write `testdata/conformance/openapi/.yaml`. It must be `.yaml` — nothing else pairs with + a table row. +2. Add `{"", assert}` to the table in `conformanceCases()` + (`compilers/openapi/conformance_test.go`), with a focused assertion for the capability. +3. Run the `-update` command above to mint `.golden.json`. The **first** run reports a + failure, because the corpus/table check reads the directory before the golden lands; re-run to + confirm green. + +`TestConformance_TableNamesEveryCorpusSpec` fails if the table and the directory disagree in +either direction, so a spec cannot land un-asserted and a row cannot outlive its spec. + +### Deliberately broken fixtures + +`testdata/dangling/openapi/` holds reproducers for references the compiler once mishandled. +`compilers/openapi/danglingcheck_test.go` asserts the one property they share: the IR that comes +out is referentially closed. Each reproducer reaches that either by interning the reference +correctly or by dropping it with an error-severity diagnostic, and the table in that test records +which of the two each one is owed. + +The ones that are refused outright are also listed as `knownInvalid` in +`internal/harness/corpus_test.go`, because an error diagnostic is the correct outcome for them and +the sweep would otherwise report it as a finding. That listing has a consequence worth knowing +before you add a fixture: the oracles stop at the first failure, so a spec that trips the +error-diagnostic oracle never reaches `irverify`, round-trip, determinism or order-independence. + +### Fuzzing + +```bash +go test ./compilers/openapi -run '^$' -fuzz FuzzCompile -fuzztime 30s +``` + +`-fuzz` takes exactly one target, and `-run '^$'` stops the package's ordinary tests from running +first. To list the targets rather than trust a count: + +```bash +grep -rn '^func Fuzz' --include='*_test.go' . +``` + +### Atomic output + +```bash +./scripts/verify-atomic-output.sh ``` -Run a single test with `go test ./ir -run TestName`. Golden IR snapshots are regenerated with -the corpus test's `-update` flag after an intentional change. +Drives the built binary against a real filesystem to check that `morphic compile -o` publishes its +output atomically, and pins the four limitations publishing-by-rename brings. It is not part of the +CI gate; run it when touching output writing. ## License diff --git a/docs/architecture.md b/docs/architecture.md index 2fe2ac3..b82ce73 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -16,8 +16,9 @@ between stages. The IR itself is specified in [`ir-design.md`](./ir-design.md). Protobuf ▼ Erlang/OTP ┌── IR passes (IR → IR) ──┐ - │ validate · link · dedup │ - │ filter · version-slice │ + │ validate (implemented) │ + │ link · dedup · filter │ + │ version-slice · overlay │ └──────────┬───────────────┘ ▼ ┌────────────── emitters ───────────────┐ @@ -57,12 +58,12 @@ between stages. The IR itself is specified in [`ir-design.md`](./ir-design.md). One compiler per source format. Each owns its format completely: file loading, reference resolution, format-version normalization, and lowering into IR nodes. -Contract (conceptually — signatures are illustrative, not implementation): +Contract (parameter and result types abbreviated; `compilers/compilers.go` declares it): ```go type Compiler interface { Formats() []SourceFormat // e.g. openapi@3.0, openapi@3.1 - Parse(ctx, sources, Options) (*ir.Document, []ir.Diagnostic, error) + Compile(ctx, sources, Options) (*ir.Document, []ir.Diagnostic, error) } ``` @@ -104,34 +105,35 @@ provenance model, and IR are built for all eight from day one. ### 2.2 IR passes (IR → IR) Small, composable, order-explicit transformations that both the engine and users (via config) -can enable: - -- **validate** — referential integrity (every typed-ID reference resolves to something the document - declares: a `TypeRef` target against the type registry, an `OpID` against the operations the - service tree declares, and so on for every ID class), discriminator mappings point at actual - variants, wire-name uniqueness within a model, binding completeness (every operation parameter - is bound exactly once per binding). Structural errors here are fatal; style issues are warnings. -- **link** — resolve cross-document references when multiple specs are parsed into one document - (multi-service, spec-stitching). -- **dedup** — structurally identical anonymous types are merged (by content hash), with ID - aliases retained so provenance survives. -- **filter** — include/exclude operations and types by pattern (Kiota-style path filtering), - followed by reachability trimming of orphaned types. Filtering serves *surface reduction* - (a smaller SDK). Regenerating only one service of an existing SDK is a different problem and - never uses a filtered document: global decisions (dedup, shared files, naming) must stay - byte-identical across scoped runs, so the emitter consumes the full document plus a scope +can enable. **`validate` is the only one implemented** — `pass/` exports nothing else — and the +rest are design, kept here because the IR has to hold what they will need: + +- **validate** *(implemented)* — referential integrity (every typed-ID reference resolves to + something the document declares: a `TypeRef` target against the type registry, an `OpID` against + the operations the service tree declares, and so on for every ID class), discriminator mappings + point at actual variants, wire-name uniqueness within a model, binding completeness (every + operation parameter is bound exactly once per binding). Structural errors here are fatal; style + issues are warnings. +- **link** *(planned)* — resolve cross-document references when multiple specs are parsed into + one document (multi-service, spec-stitching). +- **dedup** *(planned)* — structurally identical anonymous types are merged (by content hash), + with ID aliases retained so provenance survives. +- **filter** *(planned)* — include/exclude operations and types by pattern (Kiota-style path + filtering), followed by reachability trimming of orphaned types. Filtering serves *surface + reduction* (a smaller SDK). Regenerating only one service of an existing SDK is a different + problem and never uses a filtered document: global decisions (dedup, shared files, naming) must + stay byte-identical across scoped runs, so the emitter consumes the full document plus a scope option and gates emission itself (a lesson oagen recorded after trying the filtered route). -- **version-slice** — project a document carrying availability metadata into a concrete - per-version snapshot (the TypeSpec versioning model: timeline stored, snapshot consumed). -- **overlay** — user-supplied IR patches (rename hints, pagination declarations for specs that - can't express them, doc overrides) applied as data, not code. IR overlays are language-neutral - and each entry carries provenance (user-authored vs tool-inferred, mirroring the `Inferred` - marker) so automated overlay-generation loops stay auditable. Two related hooks live - deliberately elsewhere: *source-document* patching (e.g. the OpenAPI Overlay spec) is a - compiler option applied before lowering — some fixes must land before naming/hoisting - heuristics consume the broken shape — and *per-target-language* naming/compat overlays are a - emitter input keyed by IR ID, so one IR document drives different compat baselines per - language. +- **version-slice** *(planned)* — project a document carrying availability metadata into a + concrete per-version snapshot (the TypeSpec versioning model: timeline stored, snapshot consumed). +- **overlay** *(planned)* — user-supplied IR patches (rename hints, pagination declarations for + specs that can't express them, doc overrides) applied as data, not code. IR overlays are + language-neutral and each entry carries provenance (user-authored vs tool-inferred, mirroring the + `Inferred` marker) so automated overlay-generation loops stay auditable. Two related hooks live + deliberately elsewhere: *source-document* patching (e.g. the OpenAPI Overlay spec) is a compiler + option applied before lowering — some fixes must land before naming/hoisting heuristics consume + the broken shape — and *per-target-language* naming/compat overlays are a emitter input keyed by + IR ID, so one IR document drives different compat baselines per language. The first of those is `openapi.Options.Overlay`: pre-read bytes, applied to the parsed node tree before the model is built, so untouched nodes keep the line and column the parser read @@ -214,13 +216,14 @@ morphic/ │ ├── openapi/ # OpenAPI 3.x → IR (milestone 1). The public face: the │ │ │ # Compiler, its Options, document metadata, and the run that │ │ │ # calls the four lowerings below in order. -│ │ └── internal/ # Its own packages, each below that face: +│ │ └── internal/ # Its own packages, in dependency order: │ │ ├── diag/ # diagnostic codes and the constructor │ │ ├── ids/ # pointer arithmetic; pointer → TypeID │ │ ├── value/ # scalar and BigVal lowering │ │ ├── nodeview/ # the raw source as the resolver reads it │ │ ├── scan/ # pre-lowering refusals (ref cycles, alias fan-out) │ │ ├── annotation/ # what a schema says about itself, not its shape +│ │ ├── overlay/ # source-document patching, before anything is lowered │ │ ├── load/ # parse, validate, resolve │ │ ├── resolve/ # what a $ref names; reference-or-inline entries │ │ ├── merge/ # allOf property reconciliation @@ -229,7 +232,7 @@ morphic/ │ │ ├── auth/ # security schemes and requirements │ │ └── operation/ # path items, webhooks, callbacks, params, content │ ├── typespec/ smithy/ graphql/ asyncapi/ protobuf/ otp/ (future) -├── pass/ # Layer 1 — IR → IR passes (validate, dedup, filter, slice, overlay). +├── pass/ # Layer 1 — IR → IR passes. Only validate exists (§2.2). ├── emitters/ # Layer 2 — emitter contract, plan layer, registry (future). ├── engine/ # Layer 3 — orchestration: sniff format, run compiler, passes, emitters. ├── internal/archtest/ # Layering, grammar, recursion and method-cap rules (tooling). @@ -283,6 +286,9 @@ stderr; the CLI renders diagnostics. ## 5. Testing strategy +What each mechanism is for is below; the commands that run them are in +[`README.md`](../README.md#testing-and-verification). + - **Golden IR snapshots**: each compiler has a corpus of specs; `spec → IR → JSON` is snapshot-compared. IR changes show up as reviewable diffs. - **Capability conformance corpus**: one minimal spec per row of `ir-spec-matrix.md` per format @@ -290,9 +296,11 @@ stderr; the CLI renders diagnostics. keeps "lossless by default" honest as compilers are added. - **Round-trip property**: `parse → serialize → deserialize → deep-equal` for every corpus document. -- **Oracle sweep** (`internal/harness`): every corpus spec is driven through the oracles in order - — no panic, no error diagnostic, `irverify`'s structural invariants, JSON round-trip, - determinism across two compiles, and **order-independence**. The last one compiles the same +- **Oracle sweep** (`internal/harness`, and `cmd/morphic-harness` over arbitrary specs): every + corpus spec is driven through the six oracles in order — no panic, no error diagnostic, + `irverify`'s structural invariants, JSON round-trip, determinism across two compiles, and + **order-independence**. Each stops at the first failure, so a spec that trips the + error-diagnostic oracle never reaches the ones after it. The last one compiles the same source twice with every mapping's entry order reversed and diffs the two documents. It is the general form of the two-order check, and it is what catches an interning collision: two declarations minting one node at a single pointer produce the same document read either way @@ -308,15 +316,17 @@ stderr; the CLI renders diagnostics. diffed. Request-side mismatches block; response-side mismatches inform. The decisive test of a generated SDK is the bytes it puts on the wire, not whether it compiles. -## 6. Milestones - -1. **IR + OpenAPI 3.x compiler** — `ir` package, validate pass, golden corpus, JSON round-trip. -2. **Swagger 2.0 lift** — normalization into the OpenAPI compiler; proves the - format-version-normalization seam. -3. **First emitter** — one language end-to-end; proves the plan/refine/emit boundary and that the - IR retains everything a refiner needs. -4. **Second family compiler (TypeSpec or Smithy)** — proves the spec-agnostic claim: richer-than- - OpenAPI concepts (interfaces, custom scalars, lifecycle visibility, declared pagination) flow - through untouched IR code. -5. **Event-shaped compiler (AsyncAPI)** — proves channels/messages/bindings; then GraphQL, - Protobuf, and Erlang/OTP (the actor-protocol compiler: behaviours → operations + channels). +## 6. What the milestone order proves + +The milestone table — scope and state — lives in [`README.md`](../README.md#status), and only +there. What belongs here is why they run in that order: each one is chosen to falsify a claim the +IR makes, and running them in any other order would let a claim go untested for longer. + +The Swagger 2.0 lift tests the format-version-normalization seam, which has to hold before a +second dialect of anything lands. The first emitter tests the plan/refine/emit boundary and, with +it, whether the IR really retains everything a refiner needs — the acceptance test §2.3 describes. +A second *family* compiler (TypeSpec or Smithy) tests the spec-agnostic claim where it is weakest: +richer-than-OpenAPI concepts — interfaces, custom scalars, lifecycle visibility, declared +pagination — must flow through IR code written before that format existed. An event-shaped +compiler (AsyncAPI) tests channels, messages and bindings, and the Erlang/OTP compiler tests the +same surface from the other direction, lowering behaviours into operations plus channels. diff --git a/docs/emitter-design.md b/docs/emitter-design.md index 0678992..729cd33 100644 --- a/docs/emitter-design.md +++ b/docs/emitter-design.md @@ -29,12 +29,13 @@ compiler (spec → IR) → passes (IR → IR) → emitter (IR → artifacts) └────────────────────────────────────────────────────────────┘ ``` -The engine (Layer 3) runs the compiler, then the passes (`validate · link · dedup · filter · -version-slice · overlay`), then — **once per document** — the shared plan, and finally each -requested emitter. A emitter never sees a source spec, a compiler, another emitter, or the engine -(INV1). Its only structural input is an `ir.Document`; everything else it needs is either derived -from the IR in the plan/refine stages or supplied as a separate, non-IR input (runtime policy, -naming policy, shaping hints). +The engine (Layer 3) runs the compiler, then the passes (`validate` today; `link`, `dedup`, +`filter`, `version-slice` and `overlay` are designed in `architecture.md` §2.2 and not built), +then — **once per document** — the shared plan, and finally each requested emitter. A emitter +never sees a source spec, a compiler, another emitter, or the engine (INV1). Its only structural +input is an `ir.Document`; everything else it needs is either derived from the IR in the +plan/refine stages or supplied as a separate, non-IR input (runtime policy, naming policy, +shaping hints). ### 1.2 The architecture, and why this synthesis @@ -179,18 +180,27 @@ the contract pure and total. ### 2.1 The registry -Keyed by `TargetKey`, populated by each target's `init()` — mirroring the compiler registry -(architecture.md §2.1): +Keyed by `TargetKey`, and — mirroring `compilers.Registry` — a **plain instance the engine +composes explicitly**. No package-level default, no `init()`-time self-registration: ```go -var registry = map[TargetKey]func() Emitter{} +type Registry struct { byTarget map[TargetKey]func() Emitter } -func Register(t TargetKey, ctor func() Emitter) { registry[t] = ctor } -func New(t TargetKey) (Emitter, bool) { c, ok := registry[t]; if !ok { return nil, false }; return c(), true } -func Targets() []TargetKey // sorted — determinism (INV7) +func NewRegistry() *Registry +func (r *Registry) Register(t TargetKey, ctor func() Emitter) error // fails if t is already claimed +func (r *Registry) Lookup(t TargetKey) (func() Emitter, bool) +func (r *Registry) Targets() []TargetKey // sorted — determinism (INV7) ``` -The engine is the only caller of `New`, the only thing that runs `Generate`, and the only thing +A package-level `var registry = map[...]` populated by `init()` is the obvious shape and the wrong +one. It is mutable global state, which INV5 forbids outright; it makes the set of available targets +depend on the import graph rather than on what the caller asked for, so two binaries linking +different subsets get different behaviour from the same call; and it puts registration outside any +test's reach, since `init()` has already run by the time a test could substitute anything. +`compilers/compilers.go` says the same thing in its own words, and its `Register` returns an error +instead of silently overwriting a claimed key — an `init()` registry has nowhere to return one to. + +The engine is the only caller of `Lookup`, the only thing that runs `Generate`, and the only thing that touches the filesystem, prunes, and renders diagnostics. Emitters are I/O-free. ### 2.2 Who computes the plan @@ -936,7 +946,7 @@ emitters/ ├── manifest/ # generation manifest (ID-keyed), header provenance, additive-merge driver (§11) ├── verify/ # neutral Surface projection, differ, injectable per-language severity (§12) ├── golang/ # ◀ FIRST TARGET (TargetKey "go") -│ ├── emitter.go # init() Register("go", …); wires plan→refine→emit +│ ├── emitter.go # New() Emitter for the engine to register; wires plan→refine→emit │ ├── goast/ # typed target AST (sealed sum) + printer │ ├── refine/ # IR+Plan → goast — the ordered lowering pipeline (§4) │ ├── emit/ # goast → []byte via printer + go/format; templates/ (boilerplate only) diff --git a/docs/micro-compiler-design.md b/docs/micro-compiler-design.md index 0400569..31e9fea 100644 --- a/docs/micro-compiler-design.md +++ b/docs/micro-compiler-design.md @@ -1,7 +1,7 @@ # Micro-Compiler Architecture — Design Status: **implemented**. This is a record of a restructuring that happened, not a proposal — -`compilers/openapi` is now its public face over thirteen internal packages, the god object §1 is +`compilers/openapi` is now its public face over its internal packages, the god object §1 is about is gone, and the caps that stop it regrowing are in force. §12 remains the row-by-row record. Scope: `compilers/compile`, `compilers/openapi`, `internal/archtest`, `internal/harness`, `ir/irverify`.