Skip to content

Shape the generator infrastructure for extraction - #188

Merged
matt-edmondson merged 3 commits into
mainfrom
claude/extract-generalize-codegen-hjt2iy
Aug 27, 2026
Merged

Shape the generator infrastructure for extraction#188
matt-edmondson merged 3 commits into
mainfrom
claude/extract-generalize-codegen-hjt2iy

Conversation

@matt-edmondson

Copy link
Copy Markdown
Contributor

Layer 2 of #181 — isolating the parts of Semantics.SourceGenerators that have nothing to do with physical quantities, so moving them into a shared package becomes a mechanical lift.

Closes #182, closes #183, closes #185, closes #186. #184 and #187 are blocked — see below.

What moved

CodeGen/ now holds the reusable layer, with the physics-specific code depending on it and nothing depending back:

File Was
CodeGen/GeneratorBase.cs Generators/GeneratorBase.cs
CodeGen/MetadataFile.cs inline in two generators
CodeGen/DiagnosticCatalog.cs five hand-rolled descriptors across two generators
CodeGen/GeneratedSource.cs GeneratedSource.cs
CodeGen/CSharpKeywords.cs half of Emit.cs

Multi-file metadata (#182)

Two generators had worked around the base's single-file limit, not one. UnitsGenerator carried its own copy of exactly what QuantitiesGenerator had — an Initialize override, a private LoadJson, a CombinedMetadata type, and a dead Generate shim to satisfy an abstract contract it never used. Both are gone; each now just declares what it reads:

protected override IReadOnlyList<string> MetadataFileNames => ["dimensions.json", "units.json"];

Also: the file header is a parameter rather than a hard-coded literal (bound once in SemanticsGeneratorSettings), and metadata discovery matches the whole last path segment instead of EndsWith, which also matched any file whose name merely ended with the wanted one.

Diagnostics (#185)

One DiagnosticCatalog fixes the prefix and the category, which closed three real gaps:

  • CONV001 is gone. The base reported parse failures under id CONV001, category SourceGenerator — a leftover from when it served only ConversionsGenerator — while everything derived from it used SEM00x. Now SEM007.
  • Parse failures are no longer swallowed. The second-metadata-file loader caught JsonException and returned null, so a malformed units.json produced no diagnostic at all and the generator silently emitted identity conversions — factories with no scale factor.
  • A missing metadata file is now reported (SEM006). It used to produce no output and no explanation, indistinguishable from a generator that simply had nothing to emit.

SEM004 is now reported at the position in dimensions.json where the offending unit name is written, so a warning about a name in a 5,000-line file says where the name is. SEM001 and SEM003 still report without a location — they are raised from static collectors that would need the metadata file threaded through six call levels; worth a follow-up, not worth widening this PR.

Test harness (#186)

The CSharpGeneratorDriver setup moves out of SourceGeneratorTests into a standalone GeneratorHarness that supplies every metadata file, the way MSBuild's AdditionalFiles Include="Metadata/*.json" does. Handing a generator only the one file a test cares about is not how it runs for real.

New coverage for things that were untested:

  • GeneratorDiagnosticTests — each of SEM001–SEM007 fires on the input it is meant to catch. Previously the suite only asserted the real metadata produces no diagnostics, which means all five warnings could have stopped firing and every test would still have passed.
  • AnalyzerReleaseTrackingTests — every descriptor is tracked in the analyzer release files, identifiers are unique and consecutive, and the category is consistent. RS2008 no longer surfaces only after a push.
  • Incrementality — every generator reuses its output for unchanged metadata. These are IIncrementalGenerators and nothing checked they behave like one; a generator that recomputes everything on every keystroke passes every output assertion, it just makes the IDE slow.

Verification

  • Generated output under Semantics.Quantities/Generated/ is byte-identical — the whole point for the mechanical half of this change.
  • 1019 passed, 92 failed. The 92 are the same Windows-path tests that already fail on Linux on main (this container); 13 more tests pass than before.

Blocked, not done

Both ship in ktsu-dev/CodeBlocker#87, but this repo pins ktsu.CodeBlocker 1.2.15 from NuGet, so neither can land until that PR merges and publishes. CodeGen/'s WriteFileHeader/WriteSourceFile are private protected purely because SourceFileTemplate is still local; they become protected when the templates come from the package, where they are already public.

Notes for the reviewer

  • Local builds here needed two workarounds that are not committed: dropping ktsu.Sdk.Analyzers (the container's SDK is 10.0.111 / Roslyn 5.0, the analyzer wants 5.9) and temporarily pinning Microsoft.CodeAnalysis.CSharp to 5.0.0 so the generator could load at all. Directory.Packages.props is restored and unchanged in the diff — please confirm CI agrees, since it is the first place a Roslyn-version assumption would show up.
  • CLAUDE.md and docs/physics-generator.md gain the SEM006/SEM007 entries and a diagnostics table.

Generated by Claude Code

Roughly half of Semantics.SourceGenerators has nothing to do with physical
quantities: the metadata-driven IIncrementalGenerator base, metadata loading,
deterministic output handling, and the diagnostics plumbing would work unchanged
for any metadata-driven generator, but were only usable from this repository.
Those parts now sit in CodeGen/ with the physics-specific code depending on them
and nothing depending back, so moving them into a shared package is a mechanical
lift once that package has a home.

The base supports N metadata files. Two generators - not one - had worked around
its single-file limit with their own copy of the same scaffolding: an Initialize
override, a private JSON loader, a combining type, and a shim to satisfy an
abstract contract they never used. Both are gone; QuantitiesGenerator and
UnitsGenerator now just declare the files they read.

The file header is a parameter rather than a hard-coded literal, with this
repository's copyright bound once in SemanticsGeneratorSettings.

Diagnostics are allocated from one catalogue, which fixes the prefix and
category. That closes three gaps:

- The base reported parse failures as CONV001 in category "SourceGenerator", a
  leftover from when it served only ConversionsGenerator, while everything
  derived from it used SEM00x. It is now SEM007.
- The second-metadata-file loader caught JsonException and returned null, so a
  malformed units.json produced no diagnostic at all and the generator silently
  emitted identity conversions. Parse failures are now always reported.
- A declared metadata file that is missing produced no output and no
  explanation. That is now SEM006.

Metadata discovery matches the whole last path segment rather than using
EndsWith, which also matched any file whose name merely ended with the wanted
one. SEM004 is reported at the position in dimensions.json where the offending
unit name is written, so a warning about a name in a large file says where the
name is; SEM001 and SEM003 still report without a location.

Emit.cs is split: the C# vocabulary moves to CodeGen.CSharpKeywords, and what
stays is the part that only means something here - the parameter names this
generator uses and a suppression that is about physics.

The CSharpGeneratorDriver harness moves out of SourceGeneratorTests into a
standalone GeneratorHarness that supplies every metadata file the way MSBuild's
AdditionalFiles item group does, and can drive a generator twice to check it
reuses its output. New tests cover what was untested: each SEM00x firing on the
input it is meant to catch, every descriptor being tracked in the analyzer
release files, and every generator behaving incrementally.

Generated output under Semantics.Quantities/Generated/ is byte-identical.
Test suite: 1019 passed, 92 failed - the same 92 Windows-path tests that
already failed on Linux before this change, and 13 more tests passing.

Refs #182, #183, #185, #186

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015sagQjzFv3cnFNM3G271TJ
Comment thread Semantics.SourceGenerators/CodeGen/GeneratorBase.cs Fixed
Comment thread Semantics.SourceGenerators/CodeGen/GeneratorBase.cs Fixed
Comment thread Semantics.Test/Quantities/GeneratorHarness.cs Fixed
claude added 2 commits August 27, 2026 03:33
IDE0042 on a tuple pattern in the incrementality check, and two using
directives left behind by code that moved into GeneratorHarness.

Both are enforced as build errors by EnforceCodeStyleInBuild, which the local
build had to turn off: the container's SDK ships Roslyn 5.0 and this repository
pins Microsoft.CodeAnalysis 5.9, so the IDE analyzers disagree with CI about an
unrelated pre-existing file. Verified this time by patching that one file
locally and building with style enforcement on, which now reports nothing.

Refs #182, #186

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015sagQjzFv3cnFNM3G271TJ
Three loops flagged by github-code-quality for filtering their sequence
implicitly.

The metadata dictionary is now built by grouping, which states "first duplicate
wins" rather than leaving it to a ContainsKey guard - and avoids the shape the
bot suggested, where a Where predicate would read a dictionary the loop body is
still mutating. The missing-file loop filters with Where. The incrementality
check collapses to a single Any over the flattened output steps, replacing two
nested loops and an early return.

No behaviour change; generated output still byte-identical.

Refs #182, #186

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015sagQjzFv3cnFNM3G271TJ

Copy link
Copy Markdown
Contributor Author

Test on ubuntu-latest is red, and it is not this PR's

main fails the same way. Comparing the two most recent runs:

main @ 3b40272 (run) this PR @ ff7ee56 (run)
Test on ubuntu-latest ❌ 92 failed, 1006 passed ❌ 92 failed, 1019 passed
Test on windows-latest

Same 92 failures, same platform, and 13 more passing tests here — the new diagnostic, analyzer-release-tracking and incrementality tests. This PR adds no failure.

Why now: 8b9df02 ("ci: adopt the unified dotnet workflow") added Linux to the test matrix two days ago — its own commit message calls this out as "new coverage for libraries that ship netstandard and claim to run anywhere but were only ever tested on one operating system." The coverage immediately found what it was looking for. Both main runs since then are red.

What is failing: all 92 are in Semantics.Paths tests that assume Windows path semantics — drive letters, \ separators, and the 259-character limit. A sample: SemanticPath_RootPath, PathOperators_CrossPlatformSeparators_HandleCorrectly, RelativeFilePath_AsAbsoluteWithBase_ResolvesCorrectly, IsChildOf_WithMixedSeparators_WorksCorrectly.

Not fixing it here. No fix exists to port, and writing one means reworking the path-test suite's platform assumptions across roughly a dozen files — entirely unrelated to extracting the code generator, and it would bury this diff. It wants its own issue and its own PR.

The shape of that fix, for whoever picks it up: the failures are in the tests, not in Semantics.Paths itself, so the work is to express each expectation in terms of the running platform — build fixtures with Path.Combine and Path.DirectorySeparatorChar instead of literal C:\... strings, take the root from Path.GetPathRoot(Environment.CurrentDirectory), and gate the 259-character limit behind [TestCategory("OS-Specific")], which CLAUDE.md already prescribes for exactly this. Worth checking as you go whether any failure is a genuine Semantics.Paths bug on Linux rather than a test assumption — that is what the new matrix is for.

I have not re-run the job: a re-run reproduces this identically, and it is already reproduced on main.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants