From 909fb75df73b85584f62a383756894a11bdbfb6b Mon Sep 17 00:00:00 2001 From: bdchatham Date: Mon, 24 Aug 2026 08:56:36 -0700 Subject: [PATCH] docs(specs): add the contract registry and outcome tracking specs The contract registry spec is complete and ready to implement: 23 requirements CDR-001 to CDR-023, 10 acceptance scenarios AS-1.1 to AS-3.3, and 10 success criteria SC-001 to SC-010, each carrying a verifier command. Requirements come first; the design and its tradeoffs follow, so a reader sees what the system must do before how it does it. Beside them sit plan.md, data-model.md, contracts/registry-api.md, quickstart.md, and tasks.md. Tickets PLT-1055 to PLT-1059 source every acceptance criterion from an AS or SC ID in this spec. The transaction outcome tracking spec stays DRAFT. Its clarification session settled the three open decisions, so this commit records them as made and names the alternatives they rejected. Two questions remain, both about the hand-off channel: its queue depth, and whether a drop voids the run. It still needs acceptance scenarios and criteria IDs before it can carry tickets. Verifier: vale specs/ reports 0 errors and 0 warnings across 7 files. Co-Authored-By: Claude Opus 5 (1M context) --- .../contracts/registry-api.md | 148 ++++ .../data-model.md | 123 ++++ specs/contract-deployment-registry/plan.md | 157 +++++ .../quickstart.md | 80 +++ specs/contract-deployment-registry/spec.md | 633 ++++++++++++++++++ specs/contract-deployment-registry/tasks.md | 224 +++++++ specs/transaction-outcome-tracking/spec.md | 267 ++++++++ 7 files changed, 1632 insertions(+) create mode 100644 specs/contract-deployment-registry/contracts/registry-api.md create mode 100644 specs/contract-deployment-registry/data-model.md create mode 100644 specs/contract-deployment-registry/plan.md create mode 100644 specs/contract-deployment-registry/quickstart.md create mode 100644 specs/contract-deployment-registry/spec.md create mode 100644 specs/contract-deployment-registry/tasks.md create mode 100644 specs/transaction-outcome-tracking/spec.md diff --git a/specs/contract-deployment-registry/contracts/registry-api.md b/specs/contract-deployment-registry/contracts/registry-api.md new file mode 100644 index 0000000..b17d2c0 --- /dev/null +++ b/specs/contract-deployment-registry/contracts/registry-api.md @@ -0,0 +1,148 @@ +# Contract: the registry package's exported surface + +Phase 1 output. These are the signatures the generator and the harness call. They +come before implementation, per the constitution's second principle. + +## Loading + +```go +// Load returns the registry the binary carries, with any supplied files layered +// over it. A supplied file naming a chain the binary carries replaces that +// chain's entry, and Load records which source won (CDR-018, CDR-019, CDR-020). +func Load(paths ...string) (*Registry, error) + +// Sources reports which file supplied each chain, for the run to log. +func (r *Registry) Sources() map[string]string +``` + +`Load()` with no paths returns the embedded registry alone. That is the long-lived +chain case, and it touches no disk and no network. + +## Lookup + +```go +// Chain returns the entry matching both the chain id and the genesis hash +// (CDR-015). It reports false when no entry matches either. +func (r *Registry) Chain(chainID int64, genesisHash string) (Chain, bool) + +// Contract returns one named contract. It reports false when the chain carries +// no entry for that name, which is the deploy case (CDR-003). +func (c Chain) Contract(name string) (Contract, bool) +``` + +Two returns rather than an error. A missing chain and a missing contract are both +ordinary, and both lead to a deployment rather than a failure. + +## Verification + +```go +// CodeReader is the one chain call this package makes. *ethclient.Client and +// bind.ContractBackend both satisfy it, and a test supplies a fake. +type CodeReader interface { + CodeAt(ctx context.Context, account common.Address, block *big.Int) ([]byte, error) +} + +// Verify checks that the code at the recorded address hashes to the recorded +// hash (CDR-006, CDR-007). It returns a *MismatchError naming the chain, the +// contract, the address, and both hashes (CDR-008). It never deploys (CDR-009). +func Verify(ctx context.Context, code CodeReader, chain Chain, name string) error + +// MismatchError is the failure CDR-008 describes. It is a distinct type so a +// caller can tell a stale registry from a dial failure. +type MismatchError struct { + ChainName string + ChainID int64 + ContractName string + Address common.Address + Want common.Hash // recorded at deployment + Got common.Hash // observed now; the zero hash means no code +} + +func (e *MismatchError) Error() string +``` + +`Got` holds the zero hash for the absent-code case. A caller therefore needs one +error type, not two, and the message tells the two apart. + +## Recording + +```go +// Record reads the code at a freshly deployed address and returns the entry to +// write (CDR-010). It does not write anything. +func Record(ctx context.Context, code CodeReader, name string, + addr common.Address) (Contract, error) + +// WriteChain writes a chain file for an operator to review and commit. It fails +// if path names a file inside the embedded chains directory (CDR-011). +func WriteChain(path string, chain Chain) error +``` + +`WriteChain` refusing to write into `registry/chains/` is what makes CDR-011 +enforceable rather than a convention. Pointing a run at the committed registry +still does not let it edit that registry. + +## What the generator calls + +`Resolve` answers one question: does this contract already exist on this chain? + +```go +// Resolve returns the address for a named contract, and reports whether the +// caller needs to deploy. It verifies first (CDR-006, CDR-007). +func Resolve(ctx context.Context, code CodeReader, r *Registry, + chainID int64, genesisHash, name string) (common.Address, bool, error) +``` + +It returns `(addr, false, nil)` to bind, `(zero, true, nil)` to deploy, and an +error when a recorded address failed verification. A profile forcing a deployment +(CDR-016) does not call it. + +`Resolve` is the registry's whole contribution. Binding is the caller's, and the +caller does it once for every contract the profile needs, before load generation +starts. + +## The seam load generation sees + +A scenario receives a bound contract and never an address (CDR-021). The +hand-off already exists on `ContractDeployer[T]`: + +```go +GetBindFunc() ContractBindFunc[T] // the caller uses this to bind +SetContract(contract *T) // the caller hands the instance over +``` + +The preparation step lives outside `registry` and outside the scenario. It: + +1. reads the profile and collects the contracts its scenarios need (CDR-022), +2. calls `Resolve` for each, and deploys where the registry has nothing, +3. binds each result with one client (CDR-023), which is not the client that + sends load, +4. calls `SetContract` on each scenario. + +`AttachScenario` goes away. It takes an address, dials its own client, binds, and +panics on failure. This step now owns those three concerns, and turns that +panic into an error. + +A scenario is then a struct holding its bound contracts and its config. It shapes +transactions and knows nothing about where they came from. + +The binding client is not on the load path. `CreateTransactionOpts` sets +`auth.NoSend`, so a bound contract hands the transaction back rather than sending +it, and the sender's own per-endpoint clients do the sending. One client for +binding and code reads is therefore not a throughput concern. + +## The boundary + +```go +// registry imports, in full: +// context, encoding/json, embed, fmt, math/big, os, path/filepath +// github.com/ethereum/go-ethereum/common +// github.com/ethereum/go-ethereum/crypto +// github.com/sei-protocol/sei-load/generator/bindings +``` + +CDR-017 asserts this with `go list -deps`. `generator/bindings` is the one +sei-load import, and it is itself a leaf. + +Note what is absent: no `config`, no `types`, no `generator`, no `sender`, no +`stats`, and no `ethclient`. `Resolve` takes a `CodeReader`, so the caller owns +the client. diff --git a/specs/contract-deployment-registry/data-model.md b/specs/contract-deployment-registry/data-model.md new file mode 100644 index 0000000..4fae0da --- /dev/null +++ b/specs/contract-deployment-registry/data-model.md @@ -0,0 +1,123 @@ +# Data model: the chain file and the registry types + +Phase 1 output. The file format is the one-way door in this feature: once a +committed file exists, changing its shape is a migration. It comes first, and it +needs sign-off before code depends on it. + +## The chain file + +One file per chain. Files the binary knows live in `registry/chains/` and compile +in. A deployment supplies an extra file by path. + +```json +{ + "chainId": 713715, + "chainName": "arctic-1", + "genesisHash": "3f1a…64 hex chars…9c", + "genesisS3URI": "s3://prod-sei-k8s-genesis-artifacts/arctic-1/genesis.json", + "contracts": [ + { + "name": "storagerw", + "address": "0x1234567890123456789012345678901234567890", + "codeHash": "0xabcd…" + } + ] +} +``` + +### Field by field + +| Field | Type | Required | Why it is here | +| -- | -- | -- | -- | +| `chainId` | number | yes | The EVM chain id. Half the identity, and what a transaction signs against. | +| `chainName` | string | yes | For a human and for an error message. Never matched on. | +| `genesisHash` | string | yes | The other half of the identity. Bare hex, no algorithm prefix, matching `SeiNetwork.Status.GenesisHash`. | +| `genesisS3URI` | string | no | Provenance. Where the genesis came from. Never matched on. | +| `contracts` | array | yes | The contracts on this chain, each named. An empty array means the chain carries none yet. | +| `contracts[].name` | string | yes | The contract's name, which a scenario asks for. | +| `contracts[].address` | string | yes | The deployed address. | +| `contracts[].codeHash` | string | yes | Keccak-256 of the runtime code, observed at deployment. | + +### Three decisions inside the format + +**`chainName` and `genesisS3URI` are never matched on.** CDR-015 matches on +`chainId` and `genesisHash`. Both other fields exist for a human reading a +failure. Recording that here stops a later reader from treating either as a key. + +**`codeHash` is Keccak-256, not SHA-256.** The EVM already defines an account's +code hash as Keccak-256 of its runtime code. A reader can therefore check this +value against chain state, not only against `eth_getCode`. `genesisHash` stays +SHA-256: the controller defines it, and this file does not redefine it. Two +hashes in one file use two algorithms, which is worth stating rather than +discovering. + +**`contracts` is a list, not a map keyed by scenario.** CDR-005. A scenario is a +list of contracts, and most files hold a list of one today. The shape does not +change when TokenOps needs three. + +## Go types + +In package `registry`. + +```go +// Contract is one named contract on one chain. A scenario is a list of these, +// and a list of one is ordinary. +type Contract struct { + Name string `json:"name"` + Address common.Address `json:"address"` + CodeHash common.Hash `json:"codeHash"` +} + +// Chain is one chain and the contracts deployed on it. +type Chain struct { + ChainID int64 `json:"chainId"` + ChainName string `json:"chainName"` + GenesisHash string `json:"genesisHash"` + GenesisS3URI string `json:"genesisS3URI,omitempty"` + Contracts []Contract `json:"contracts"` +} + +// Registry holds every chain the binary carries, plus any the deployment +// supplied. +type Registry struct { + chains map[chainKey]Chain +} + +// chainKey is what CDR-015 matches on, and nothing else. +type chainKey struct { + chainID int64 + genesisHash string +} +``` + +`common.Address` and `common.Hash` already marshal to and from hex strings, so +the JSON shape above needs no custom marshaller. + +## What the types deliberately do not hold + +**No ABI, and no bytecode.** Those live in `generator/bindings`, which the +registry imports. Duplicating them in a chain file would let the file and the +binding disagree, and CDR-007 exists to detect exactly that disagreement. + +**No deployer key, and no endpoint.** A chain file describes a chain, not how to +reach it. The run already has endpoints from its profile. + +**No timestamps, and no provenance beyond the S3 URI.** Git holds when an entry +landed and who committed it. + +## Where deployment stays + +The registry looks up, verifies, and records. It does not deploy. + +`ContractScenarioBase.Deploy` already deploys, and moving it would mean moving +`ContractDeployer[T]` and its four implementors into the registry package. That +is a refactor this feature does not need: CDR-003 deploys when no entry exists, +and the scenario already knows how. + +The flow: a preparation step asks the registry for each contract the profile needs. +It deploys where the registry has nothing. It binds every result with one client, +and hands each scenario its bound contracts. The scenario never sees an address +(CDR-021). + +Un-defer when an operator needs to deploy without starting a run. The spec's +Deferred section already names that as the deploy subcommand. diff --git a/specs/contract-deployment-registry/plan.md b/specs/contract-deployment-registry/plan.md new file mode 100644 index 0000000..bd6f0d3 --- /dev/null +++ b/specs/contract-deployment-registry/plan.md @@ -0,0 +1,157 @@ +# Implementation Plan: Contract deployment and the contract registry + +**Branch**: `contract-deployment-registry` | **Date**: 2026-08-22 | **Spec**: `specs/contract-deployment-registry/spec.md` + +**Input**: Feature specification from `specs/contract-deployment-registry/spec.md` + +## Summary + +A run gets a contract's address from a registry compiled into the binary, or from +a chain file the deployment supplies. It deploys only when neither holds one. It +verifies the code at a recorded address before it sends load, and fails rather +than redeploying on a mismatch. + +The registry lives in its own package. It imports no sei-load package other than +the contract bindings. That boundary is the plan's spine. The nightly harness +that creates a chain and deploys contracts cares nothing for offered rate, +scenarios, or metrics. + +## Technical Context + +**Language/Version**: Go 1.25.1, per `go.mod` and the CI matrix. + +**Primary Dependencies**: `go-ethereum`, for `abi`, `bind`, `ethclient` and +`common.Address`. `generator/bindings`, for bytecode and bind functions. No new +module. + +**Storage**: JSON files in the repo, compiled in with `embed`. One file per +chain. A deployment supplies an extra file by path. + +**Testing**: `go test`, `testify/require`, and the repo's mock chain. A package +boundary test using `go list -deps`. + +**Target Platform**: Linux container, run as a Kubernetes Deployment or Job, and +run locally. + +**Project Type**: Single Go module, command plus packages. + +**Performance Goals**: Startup only. Verification costs one `eth_getCode` per +contract, not per account or transaction. + +**Constraints**: The registry package imports no sei-load package other than the +bindings. No Kubernetes API. No S3 fetch. No network call for chain identity. + +**Scale/Scope**: Six contracts exist. Four long-lived chains, plus one ephemeral +chain per nightly run. A scenario holds one contract today. + +## Constitution Check + +*GATE: passed before Phase 0. Re-checked after Phase 1.* + +| Principle | Assessment | +|---|---| +| I. Verify before claiming done | Every requirement has a named test in the spec's Verification section. The boundary in CDR-017 is machine-checkable with `go list -deps`. Each new assertion gets a deliberate break before it counts. | +| II. Interfaces first, two-way doors | Phase 1 delivers the exact types and signatures before any task. **One one-way door: the registry file format.** Once a committed file exists, changing its shape is a migration. It needs sign-off at Phase 1, not at implementation. | +| III. Idiomatic Go, package pattern wins | The registry is a leaf package, which `config` and `generator/bindings` already are. It reuses `ContractBindFunc[T]`, the repo's existing shape, rather than inventing a loader. | +| IV. Smallest slice | The spec's Deferred section names four cuts with un-defer conditions. Phase 1 adds one more: this plan does not change `ContractDeployer[T]`. | +| V. Simplified Technical English | `vale specs/` gates it, and reports no errors and no warnings. | + +**No violations to justify.** Complexity Tracking stays empty. + +### The gate that needed a decision + +`ContractDeployer[T]` is generic over one contract type, so a scenario holds one +address. CDR-005 keys the registry by contract role, which implies more than one. + +This plan does **not** change that interface. The registry stores a role-keyed +map because changing the file format later is the one-way door, and the interface +is not. A scenario with one contract reads the single role. When TokenOps needs +three, it changes `ContractDeployer[T]` and finds the registry already shaped for +it. + +Recorded here so a reader sees the choice rather than inferring it from the code. + +## Project Structure + +### Documentation (this feature) + +```text +specs/contract-deployment-registry/ +├── spec.md # 20 requirements, CDR-001 to CDR-020 +├── plan.md # This file +├── research.md # Phase 0: the four unknowns and how they resolved +├── data-model.md # Phase 1: the file format and the Go types +├── contracts/ # Phase 1: the package's exported surface +├── quickstart.md # Phase 1: adding a chain, and running against one +└── tasks.md # Phase 2, from /speckit-tasks +``` + +### Source code + +```text +registry/ # new leaf package, CDR-017 +├── registry.go # Registry, Lookup, the embedded files +├── chains/ # one JSON file per known chain, embedded +│ ├── arctic-1.json +│ ├── atlantic-2.json +│ └── pacific-1.json +├── deploy.go # deploying a contract and reporting the entry +├── verify.go # the code check, CDR-006 to CDR-008 +├── registry_test.go +└── boundary_test.go # CDR-017, via go list -deps + +config/config.go # Scenario gains the force-deploy flag, CDR-016 +generator/generator.go # deployAll consults the registry before deploying +main.go # a flag for the supplied chain file, CDR-019 +``` + +**Structure Decision**: a new leaf package `registry/`, beside `config/` rather +than under `generator/`. Placing it under `generator/` would put it inside the +tree it is forbidden to depend on, and would hide the boundary. The +chain files sit inside the package so `embed` reaches them. + +## Phase 0: research + +Four unknowns. `research.md` records each, with what settled it. + +1. What a chain file holds, and how a run identifies its chain. Settled by the + clarification session: chain id and genesis hash, with the S3 URI as + provenance. +2. How to get a runtime code hash that a later run reproduces. `Bin` is + creation bytecode, so the registry records what `eth_getCode` returned at + deployment. +3. Whether `embed` reaches a subdirectory of the package. It does, with + `//go:embed chains/*.json`. +4. What the harness fills a chain file from. `SeiNetwork.Status.GenesisHash` and + `GenesisS3URI`, both already published by the controller. + +None remains open. No `NEEDS CLARIFICATION` markers survive into Phase 1. + +## Phase 1: design + +`data-model.md` fixes the file format and the Go types. `contracts/` fixes the +package's exported surface, which is what the generator and the harness both +call. `quickstart.md` states how an operator adds a chain, and how a deployment +supplies one. + +The order matters. The file format is the one-way door, so it gets written and +signed off before any code depends on it. + +## Post-design Constitution re-check + +Run at the end of Phase 1. One gate does not clear on its own. + +| Principle | Post-design | +|---|---| +| I. Verify before claiming done | Holds. `contracts/registry-api.md` gives `MismatchError` a named type, so a test asserts the failure rather than matching a string. | +| II. Interfaces first, two-way doors | **Signatures delivered. The one-way door is now concrete and needs sign-off.** The chain file format in `data-model.md` becomes a migration the moment a committed file exists. Tasks MUST NOT start before an operator approves it. | +| III. Idiomatic Go, package pattern wins | Holds. `Chain`/`Contract` return two values because a miss is ordinary, matching the repo's map-lookup shape. `CodeReader` is one method, so `*ethclient.Client` satisfies it without an adapter. | +| IV. Smallest slice | Holds, and tightened. Phase 1 keeps deployment in `ContractScenarioBase` rather than moving `ContractDeployer[T]` and four implementors. The registry looks up, verifies, and records. | +| V. Simplified Technical English | Holds. `vale` reports no prose findings across all Phase 1 artifacts. | + +**Gate result: proceed to tasks once an operator approves the file format.** +Nothing else blocks. No Complexity Tracking entries. + +## Complexity Tracking + +No Constitution Check violations. This section stays empty. diff --git a/specs/contract-deployment-registry/quickstart.md b/specs/contract-deployment-registry/quickstart.md new file mode 100644 index 0000000..67182ad --- /dev/null +++ b/specs/contract-deployment-registry/quickstart.md @@ -0,0 +1,80 @@ +# Quickstart + +Phase 1 output. Three tasks an operator does, and what each one costs. + +## Run against a chain the binary knows + +Nothing to do. Name the chain in the profile as today. The run finds the entry, +verifies the code, attaches, and sends no deployment. + +The run logs which source supplied each chain, so a reader confirms it used the +compiled registry rather than a file someone left behind. + +## Add a chain the binary does not know + +Two steps, and a human between them. + +**Deploy.** Point a run at the new chain, supplying its chain file: + +```sh +seiload --config profile.json --chain-file ./newchain.json +``` + +The file needs the chain's identity and an empty contract set: + +```json +{ + "chainId": 713714, + "chainName": "devnet-nightly", + "genesisHash": "…from SeiNetwork.Status.GenesisHash…", + "genesisS3URI": "…from SeiNetwork.Status.GenesisS3URI…", + "contracts": {} +} +``` + +The run finds no entry for each role, deploys, and writes a completed file +alongside its report. + +**Commit.** Review the written file. Move it into `registry/chains/` if the binary needs to +carry that chain. A nightly chain is not: it disappears after +the run, and committing its addresses would leave an entry that can never verify. + +## Add a chain from the nightly harness + +The harness holds the `SeiNetwork` it created, so it fills the chain file from +the resource's status rather than by hand: + +```go +chain := registry.Chain{ + ChainID: network.Spec.EVMChainID, + ChainName: network.Name, + GenesisHash: network.Status.GenesisHash, + GenesisS3URI: network.Status.GenesisS3URI, + Contracts: map[registry.Role]registry.Deployment{}, +} +``` + +It writes that file, mounts it into the sei-load pods, and passes `--chain-file`. +The pods deploy on first run. + +This is the case the separation exists for. The harness imports `registry` and +gets no scenario, no sender, and no metrics with it. + +## When a run fails at startup + +A `MismatchError` names the chain, the role, the address, and both hashes. + +**Got is the zero hash.** The address holds no code. Either the chain was +re-created and kept its id, or the entry names an address from a different chain. +Check the genesis hash in the entry against the chain's own. + +**Got differs from Want.** The address holds different code. Either someone +deployed over it, or the entry predates a change to the contract. + +Neither case redeploys, and that is deliberate. A run that repaired itself would +turn a stale entry into a second deployment nobody recorded. + +## What this does not do + +It does not fetch the genesis from S3, so it needs no credentials. It does not +read a Kubernetes API. It does not discover a chain it was not told about. diff --git a/specs/contract-deployment-registry/spec.md b/specs/contract-deployment-registry/spec.md new file mode 100644 index 0000000..89d3d28 --- /dev/null +++ b/specs/contract-deployment-registry/spec.md @@ -0,0 +1,633 @@ +# Contract deployment and the contract registry + +Status: DRAFT +Owner: platform +Linear: TBD + +## Anchors + +EARS · RFC 2119 · Spec Kit · idempotence · fail-closed · one-way door · +content addressing · CREATE2 + +## Why this needs a specification + +A contract scenario deploys its contract at the start of every run. That is +correct on a fresh chain and wrong everywhere else. + +Long-lived chains make that wrong: arctic-1, atlantic-2, pacific-1, and any +standing benchmark environment. Each run leaves another copy of the contract +behind. +Every restart of the prod canary moves its contract to a new address. A metric +series spanning two restarts describes two different contracts. Nothing in the +run says the address changed. + +sei-load already has the mechanism to avoid this. `TxGenerator.Attach` binds a +scenario to an address that already holds the contract, and every contract +scenario implements it. Only `mockDeployAll` calls it, which is the unit-test +path. `generatorBuilder.deployAll` always deploys, and `config.Scenario` carries +no address field. The capability exists, and no profile can reach it. + +The gap is small. The hazards it opens are not, and they are the reason this is a +specification rather than a config field. + +## Clarifications + +### Session 2026-08-22 + +- Q: Where does the registry live, and in what format? → A: A JSON file in the + sei-load repo, beside the bindings that encode against it. +- Q: What identifies a chain in a registry entry? → A: The EVM chain id and the + genesis hash. The genesis S3 URI rides alongside as provenance, not as a key. +- Q: Can a profile force a fresh deployment when the registry has an entry? → + A: Yes, through an explicit per-scenario opt-out. +- Q: How does a new entry reach the repo? → A: The run writes a file of its own. + An operator reviews it and commits it. Refined on 2026-08-23, below. + +### Session 2026-08-23 + +Three review comments on the published spec. + +- Q: Is the type a role or a contract? → A: A contract. A scenario is a list of + contracts, and a list of one is ordinary. `Role` and `Deployment` fold into one + `Contract`, and a chain entry holds a list rather than a map. +- Q: One shared client, or one per scenario? → A: One, for resolution and + binding. It is not on the load path: `auth.NoSend` keeps the transaction in + hand, and the sender's own per-endpoint clients send. +- Q: What does a run write, and do concurrent runs share contracts? → A: It writes + a chain file locally, which an operator commits to make the binary carry the + chain. Concurrent runs on an ephemeral chain each deploy their own contracts, + because sharing one contract would put contention in neither profile. + +## The workflow this serves + +This is one step of a chain, and the chain explains why the pieces divide the way +they do. + +A Go harness creates a `SeiNetwork` through the controller's API. The controller +runs the genesis ceremony and reports the chain id and the genesis hash on the +resource's status. The harness then deploys sei-load against that chain, as a +Helm release of load-generating pods. It runs nightly. + +Two deployment modes fall out, and they need different things. + +**A chain the deployment already knows.** pacific-1, atlantic-2, arctic-1, and any +standing benchmark environment. Their contracts exist. Whoever prepares the +deployment knows which chain it targets before it starts, so the chain's identity +and its contract addresses travel with the binary. The run discovers nothing. + +**A chain the harness just created.** No contracts exist. The harness holds the +chain's identity from the resource it created, and hands it to the run. The run +deploys, and reports what it deployed. + +Neither mode needs sei-load to query a Kubernetes API, and neither needs it to +discover a chain it was not told about. + +## Separation + +The registry and the deployment machinery MUST stay separable from load +generation. They serve the harness above, which has no interest in offered rate, +scenarios, or metrics. Keeping the seam clean now costs a package boundary. +Recovering it later costs a refactor of whatever grew across it. + +CDR-017 states the boundary in a form a test can check. + +## Scope + +This specification covers how a run obtains the address of the contract a +scenario drives: by deploying it, or by reading a recorded deployment. It covers +what the run MUST verify before it trusts a recorded address. + +Out of scope: what the contracts do, how a scenario builds its calls, and how the +run funds the deployer. `funder.Deployer` already resolves the deploying account. + +## Current behaviour + +- `generatorBuilder.deployAll` calls `Scenario.Deploy` for every instance. No + branch reads an address from anywhere. +- `generatorBuilder.mockDeployAll` calls `Scenario.Attach` with a generated + address. `config.LoadConfig.MockDeploy` selects it, and only tests set it. +- `config.Scenario` has no address field. +- `ContractDeployer[T]` is generic over one contract type. A scenario holds one + address, so a scenario needing more than one contract does not fit today. +- A binding's `MetaData.Bin` holds **creation** bytecode. `eth_getCode` returns + **runtime** bytecode. The two differ, so a guard cannot compare them directly. +- `StorageRWv1` exposes a `VERSION` method. No other contract in the repo does, + so an on-chain identity call is not a general mechanism. +- `config.LoadConfig` carries `ChainID`, the EVM chain id, and `SeiChainID`, a + textual label. + +## User stories + +Each story states how to test it on its own, then the scenarios that say what +holds afterwards. + +**Story 1. An operator runs a workload on pacific-1 and deploys nothing.** + +The contracts are already there. The run reads their addresses from the registry +and starts driving load. + +**Independent Test**: run a profile naming a chain the registry covers, against a +mock chain holding the recorded code. The run sends no deployment transaction. + +*Acceptance scenarios* + +**AS-1.1** — the run binds a recorded address. +- GIVEN the registry holds an entry for the run's chain and contract +- AND the code at that address matches the recorded hash +- WHEN the run starts +- THEN it binds that address, and sends no deployment + +**AS-1.2** — the run resolves only what its profile drives. +- GIVEN a profile driving one scenario +- AND a registry entry holding more than one contract +- WHEN the run starts +- THEN it reads the code of that scenario's contract only + +**AS-1.3** — a scenario holds no address. +- GIVEN the run has started +- WHEN a scenario builds a transaction +- THEN it holds a bound contract, and no field of it holds an address + +**Story 2. An operator brings up a new chain and records what the run deployed.** + +No registry entry exists. The run deploys, and it emits an entry an operator can +commit. + +**Independent Test**: run against a chain absent from the registry. The run +deploys, and it writes an entry naming the chain, the contract, the address, and +the code hash it observed. + +*Acceptance scenarios* + +**AS-2.1** — an unknown chain deploys. +- GIVEN no registry entry for the run's chain +- WHEN the run starts +- THEN it deploys each contract its profile needs, and binds what it deployed + +**AS-2.2** — the run records what it deployed. +- GIVEN the run deployed a contract +- WHEN it finishes startup +- THEN it writes a chain file holding the chain id, the genesis hash, the + contract name, the address, and the code hash it read back + +**AS-2.3** — the compiled-in registry stays read-only. +- GIVEN a path inside the compiled-in chains directory +- WHEN the run tries to write there +- THEN it fails, and writes nothing + +**AS-2.4** — a supplied file wins. +- GIVEN a supplied chain file naming a chain the binary also carries +- WHEN the run loads the registry +- THEN the supplied file wins, and the run logs which source it used + +**Story 3. A stale entry fails the run instead of corrupting it.** + +A devnet keeps its chain id across a re-genesis. The recorded address now holds +nothing, or holds different code. + +**Independent Test**: point a run at a chain whose recorded address holds no code. +The run fails at startup and names the chain, the contract, and the address. + +*Acceptance scenarios* + +**AS-3.1** — an empty address fails the run. +- GIVEN a registry entry whose address holds no code +- WHEN the run starts +- THEN it fails, naming the chain, the contract, the address, and both hashes + +**AS-3.2** — wrong code fails the run, distinguishably. +- GIVEN a registry entry whose address holds different code +- WHEN the run starts +- THEN it fails the same way, and the message tells this case from absent code + +**AS-3.3** — a mismatch repairs nothing. +- GIVEN either mismatch +- WHEN the run fails +- THEN it has deployed nothing, and sent no transaction + +## Requirements + +Every requirement carries an ID. A test names the ID it covers. + +### Selecting an address + +**CDR-001.** WHERE a profile names a scenario that needs a contract, the run MUST +get that contract's address from one source only. That source is the registry, or +a deployment the run performs. + +**CDR-002.** IF the registry holds an entry for the run's chain and that +contract, THEN the run MUST use the recorded address. It MUST NOT deploy. + +**CDR-003.** IF the registry holds no entry for the run's chain and that +contract, THEN the run MUST deploy the contract. + +**CDR-004.** A profile MUST be able to override the registry with an explicit +address, for a contract deployed outside this repo. + +**CDR-016.** A profile MUST be able to force a deployment for a scenario, even +where the registry holds an entry. The profile MUST name that choice explicitly. + +Rationale for CDR-016: a run measuring contention from an empty keyspace needs +virgin storage. Registry-first is the default, so a reader can see from the +profile why a run redeploys. + +**CDR-005.** A chain entry MUST hold a list of contracts, each named. A scenario +is a list of contracts, and a list of one is ordinary. + +Rationale for CDR-005: the token and DeFi workloads need three and seven +contracts. A registry shaped around one address per scenario would need a second +breaking change to hold them. + +### Trusting a recorded address + +**CDR-006.** WHEN a run reads an address from the registry, the run MUST verify +that the address holds contract code before it sends any load. + +**CDR-007.** The registry MUST record a hash of the runtime code observed at +deployment. WHEN a run reads an address, it MUST compare the code at that address +against the recorded hash. + +**CDR-008.** IF the code at a recorded address is absent, or its hash differs +from the recorded hash, THEN the run MUST fail at startup. It MUST name the +chain, the contract role, the address, and both hashes. + +**CDR-014.** A registry entry MUST identify its chain by the EVM chain id and the +genesis hash together. An entry MUST also record the genesis S3 URI. + +### Where the values come from + +**CDR-017.** The registry package MUST NOT import any sei-load package other than +the contract bindings. A test MUST assert this. + +Rationale for CDR-017: the harness that creates a chain and deploys contracts has +no interest in offered rate, scenarios, or metrics. `config` and +`generator/bindings` already import no other sei-load package, so a leaf is the +established shape here. `generator/utils` is not a leaf, so the registry builds +its own deployment options instead of borrowing that one. + +**CDR-018.** The binary MUST carry the registry for every chain it already knows, +compiled in. A run against one of those chains MUST NOT read the registry from +disk. It MUST NOT reach a network to find it. + +**CDR-019.** WHERE a deployment targets a chain the binary does not carry, the +deployment MUST supply that chain's file. The run MUST read the chain id, the +genesis hash, and any known contract addresses from that file. + +**CDR-020.** IF a supplied file names a chain the binary also carries, THEN the +supplied file MUST win. The run MUST log which source it used. + +### What load generation receives + +**CDR-021.** Load generation MUST receive bound contracts. A scenario MUST NOT +receive a contract address, and MUST NOT bind a contract itself. + +**CDR-022.** A run MUST resolve only the contracts its profile needs. It MUST NOT +verify or deploy a contract no scenario in the profile drives. + +**CDR-023.** One client MUST serve the whole resolution. A scenario MUST NOT open +its own connection to bind. + +Rationale for CDR-021 to CDR-023: an address is a fact about deployment, and a +scenario's job is to shape a transaction. Today `AttachScenario` takes an +address, dials its own client, binds, and panics if either fails. Every scenario +therefore carries three concerns that belong to one earlier step. + +Moving them makes the boundary in CDR-017 real rather than nominal. A scenario +that never sees an address cannot depend on the registry, so extracting the +registry stays a move rather than a refactor. + +The hand-off already exists. `ContractDeployer[T]` declares `GetBindFunc` and +`SetContract`, so the resolution binds and hands the instance over. What goes +away is `AttachScenario`, which is the part that dials and panics. + +Rationale for CDR-018 to CDR-020: whoever prepares a deployment knows the chain +before it starts. A long-lived chain travels with the binary. A chain the harness +created travels with the deployment. Neither mode discovers a chain, so neither +can point at one nobody chose. + +**CDR-015.** WHEN a run reads an entry, the run MUST match on the chain id and +the genesis hash. It MUST NOT match on the genesis S3 URI. + +Rationale for CDR-014 and CDR-015: `SeiNetwork.Status` already carries both +values, and the controller already pairs them this way. A node booting from the +S3 fallback verifies its download against `GenesisHash`. The hash is the +identity, and the URI is where the artifact lives. A re-genesis changes the +hash. Whether it changes the URI depends on how the controller names the object, +which this specification does not control. + +Rationale for CDR-006 to CDR-008: an EVM chain id does not identify a chain +instance. A devnet keeps its id across a re-genesis, so a recorded address +survives the state that made it meaningful. A registry entry is a cache, and this +is the check that stops a stale cache entry from becoming a silent wrong run. + +The failure is otherwise invisible. Calls to an address holding no code succeed +at the EVM layer and do nothing. Calls to an address holding different code +encode against the wrong ABI. sei-load reads inclusion rather than execution +status, so neither shows up as a failure. See +`specs/transaction-outcome-tracking/spec.md`. + +**CDR-009.** The run MUST NOT repair a mismatch by deploying. It MUST fail. + +Rationale for CDR-009: deploying on mismatch turns a stale entry into a silent +second deployment, which is the behaviour this specification exists to remove. An +operator decides whether the entry is wrong or the chain is. + +### Recording a deployment + +**CDR-010.** WHEN a run deploys a contract, the run MUST emit a registry entry. +The entry MUST hold the chain, the contract role, the address, and the runtime +code hash. + +**CDR-011.** A run MUST NOT modify the compiled-in registry. It MUST write what +it deployed to a chain file of its own. An operator commits that file to make the +binary carry the chain. + +Rationale for CDR-011: the registry is the record other runs trust. A load test +that edits its own source of truth can record an address from a chain that no +longer exists. No human sees it happen. + +### Cost and behaviour + +**CDR-012.** WHILE a run starts, the requests it makes to verify recorded +addresses MUST grow with the contract count. They MUST NOT grow with the account +or transaction count. + +**CDR-013.** WHERE more than one run starts against one ephemeral chain, each run +MUST deploy its own contracts. A run MUST NOT wait for another run's deployment, and +MUST NOT drive a contract another run deployed. + +Rationale for CDR-013: isolation is the intent, not a tolerated outcome. Two runs +driving one StorageRW contract contend on the same slots. That contention is in +neither profile, so both runs measure a workload nobody configured. Separate +contracts keep each run's contention exactly what its profile asked for. + +Sharing one contract set across runs needs a preflight step that deploys once and +injects the addresses. That is orchestration this specification does not add. See +Deferred. + +## Success Criteria + +The specification closes when all of the following hold. A ticket cites these by +ID, so they carry one. + +- **SC-001** A profile naming a chain the registry covers starts and sends no + deployment transaction. *Verifier*: `go test ./registry/... ./generator/...` +- **SC-002** A profile naming an uncovered chain deploys, and writes a + reviewable chain file. *Verifier*: `go test ./generator/...` +- **SC-003** A recorded address holding no code, or unexpected code, fails the + run at startup with both hashes named. *Verifier*: `go test ./registry/...` +- **SC-004** A scenario driving more than one contract records one entry per + contract. *Verifier*: `go test ./registry/...` +- **SC-005** A profile that forces a deployment redeploys, even where an entry + exists. *Verifier*: `go test ./generator/...` +- **SC-006** No scenario holds a contract address, and no scenario opens a + connection to bind. *Verifier*: `go test ./generator/...` and `grep` for the + removed method. +- **SC-007** The registry package imports no sei-load package other than the + contract bindings. *Verifier*: `go test ./registry/ -run TestImportBoundary` +- **SC-008** Every requirement ID above has at least one test that names it. + *Verifier*: review against the traceability table in `tasks.md`. +- **SC-009** Every committed chain file parses, and holds each field the format + names. *Verifier*: the CI gate from T028. +- **SC-010** `make verify` passes. + +## Design + +How we propose to meet the requirements above, and what we considered instead. +The requirements are the contract. This is one way to satisfy them, and a later +reader can replace it without reopening them. +sees the mental model before the rules that constrain it. + +### Three parts, and which one is extractable + +**`registry`** answers one question: does this contract already exist on this chain, +and is the code there the code we recorded? It imports no sei-load package other +than the contract bindings, so it lifts out whole. + +**A preparation step** joins the registry to the scenarios. It reads the profile, +resolves each contract, deploys what is missing, binds the results, and hands +each scenario its contracts. It depends on both sides, so it is the one part that +cannot be extractable, and it stays small for that reason. + +**The scenarios** shape transactions against a bound contract. After this feature +they hold no address and open no connection. + +### The registry's surface + +```go +// Contract is one named contract on one chain. A scenario is a list of these, +// and a list of one is ordinary. +type Contract struct { + Name string + Address common.Address + CodeHash common.Hash +} + +// Chain is one chain and the contracts deployed on it. +type Chain struct { + ChainID int64 + ChainName string + GenesisHash string + GenesisS3URI string + Contracts []Contract +} + +// CodeReader is the one chain call this package makes. An *ethclient.Client +// satisfies it, and a test supplies a fake. +type CodeReader interface { + CodeAt(ctx context.Context, account common.Address, block *big.Int) ([]byte, error) +} + +// Load returns the compiled-in registry, with supplied files layered over it. +func Load(paths ...string) (*Registry, error) +func (r *Registry) Chain(chainID int64, genesisHash string) (Chain, bool) +func (r *Registry) Sources() map[string]string +func (c Chain) Contract(name string) (Contract, bool) + +// Resolve reports the address for a named contract, or that the caller needs to +// deploy. It verifies before it returns an address. +func Resolve(ctx context.Context, code CodeReader, r *Registry, + chainID int64, genesisHash, name string) (addr common.Address, deploy bool, err error) + +// Record reads the code at a freshly deployed address and returns the entry. +func Record(ctx context.Context, code CodeReader, name string, + addr common.Address) (Contract, error) + +// WriteEntry writes a chain file for review. It refuses a path inside the +// compiled-in chains directory. +func WriteEntry(path string, chain Chain) error + +// MismatchError is the failure a stale entry produces. Got is the zero hash when +// the address holds no code. +type MismatchError struct { + ChainName string + ChainID int64 + ContractName string + Address common.Address + Want, Got common.Hash +} +``` + +### What changes on the scenario side + +`ContractDeployer[T]` keeps the two methods the hand-off already uses: + +```go +GetBindFunc() ContractBindFunc[T] // the preparation step binds with this +SetContract(contract *T) // and hands the instance over with this +``` + +`AttachScenario(config, address) common.Address` **goes away.** It takes an +address, dials its own client, binds, and panics if either fails. The +preparation step takes over all three, and returns an error where it panicked. + +### The sequence, at startup + +Load generation does not begin until every step here has run. + +1. `main` reads the profile into a `LoadConfig`. +2. `main` calls `registry.Load(chainFiles...)`. With no files it returns the + compiled-in registry alone, touching no disk and no network. +3. `main` dials **one** client. That client serves every step below, and no + scenario dials its own. + + It is not on the load path. `CreateTransactionOpts` sets `auth.NoSend`, so a + bound contract hands the transaction back rather than sending it, and the + sender's own per-endpoint clients do the sending. This client binds and reads + code. Nothing else. +4. The preparation step collects the contracts the profile's scenarios need. A + contract no scenario drives is never resolved, verified, or deployed. +5. For each contract it calls `registry.Resolve`. + - An entry exists and its code matches: `Resolve` returns the address. + - No entry exists: `Resolve` reports that the caller needs to deploy. + - An entry exists and its code does not match: `Resolve` returns a + `*MismatchError`, and the run stops. +6. For each contract that needs one, the step deploys, then calls + `registry.Record` to read back the code hash. +7. The step binds every address, using `GetBindFunc` and that one client, and + calls `SetContract` on the scenario. +8. If step 6 recorded anything, the step writes a chain file beside the run + report. An operator commits it to make the binary carry that chain. +9. The generator starts. Every scenario holds a bound contract. + +### Where a failure stops the run + +| Failure | Step | What the run does | +| -- | -- | -- | +| A supplied chain file does not parse | 2 | Fails, naming the file. | +| The chain is in no registry, and no file supplies it | 4 | Fails, naming the chain id. | +| A recorded address holds no code, or different code | 5 | Fails with a `*MismatchError`. It does not deploy. | +| A deployment does not mine | 6 | Fails, naming the contract. | +| Binding fails | 7 | Fails, naming the contract. It does not panic. | +| The output path is inside the compiled-in registry | 8 | Fails, refusing to write. | + +Every one of these stops the run before it sends a transaction. That is +deliberate: a run that starts and then discovers its contracts are wrong has +already published numbers nobody can trust. + +### What we considered instead + +Each row is a choice the design makes, the alternative, and what decided it. + +**Binding in the preparation step, not in the scenario.** The scenario could keep +binding, as it does today, and simply receive an address from the registry. That +is a smaller diff. It leaves every scenario dialling its own client and panicking +on failure, and it leaves an address in a type whose job is shaping transactions. +The larger change is what makes CDR-017's boundary real: a scenario that never +holds an address cannot depend on the registry. + +**One shared client, not one per scenario.** Today each `AttachScenario` dials. +Per-scenario clients are simpler to reason about in isolation and cost a +connection each, and they hide a dial failure inside a panic. One client makes +the failure an error at a known step. + +**`Resolve` returns two values, not an error for a miss.** A `NotFound` error +would let one branch handle every outcome. A missing entry is ordinary: it is the +whole of the deploy path. An error would make the common case read as the +exceptional one, and every caller would type-switch to recover it. + +**`CodeReader` is one method, not `*ethclient.Client`.** Taking the concrete type +is less code. It would put `ethclient` in the registry's import list, and it +would force a test to stand up a client or an HTTP fake. One method keeps the +import list short enough to read, and a test supplies a struct. + +**Verify at startup, not lazily on first use.** Lazy verification costs nothing +for a role no scenario reaches. It also means the run has already started, and +already published numbers, when it discovers its contracts are wrong. Startup is +the last point where failing is free. + +**The registry records the code hash it observed, rather than deriving one.** A +binding's `Bin` is creation bytecode and `eth_getCode` returns runtime bytecode, +so there is nothing to derive from. Extracting the runtime portion from creation +bytecode is possible and fragile. The metadata Solidity appends also varies with +compiler settings, so two honest builds can disagree. + +**`ContractDeployer[T]` stays generic over one contract.** CDR-005 keys the +registry by role, which implies more than one. Changing the interface now would +touch four implementors for a case no scenario has yet. The registry's file +format is the one-way door and the interface is not. The format therefore carries +the role map, and the interface changes when TokenOps needs it. + +**A JSON file, not Go source.** Generated Go would fail to compile on a bad +entry, which is a better failure than a startup error. It also means an operator +cannot add a chain without writing Go, and CDR-011's reviewable sidecar becomes +code generation. + +## Decisions this specification does not make + +The clarification session resolved D-1 to D-4. See `## Clarifications`, and +CDR-004, CDR-011, CDR-014, CDR-015 and CDR-016, which carry those answers. + +One decision stays open, because the answer sits in a repository this +specification does not cover. + +**D-5, how a run learns its chain's genesis hash.** Resolved: the deployment +tells the run. A chain the binary knows carries its hash compiled in. A chain the harness created +carries it in the file the deployment supplies, which the harness fills from the +resource's status. sei-load reads no Kubernetes API, and queries no endpoint for +identity. + +CDR-015 therefore compares a value someone supplied, not one the run observed. A +deployment shipping the wrong chain file passes the identity check and fails the +code check at CDR-006 to CDR-008. That is what catches it, and it is why every run +runs the code check. + +## Deferred + +Named so a reader sees the line, and so nobody builds them by accident. + +- **A subcommand that deploys.** The run deploys when it finds no entry, and + writes a file. Separating deployment from load generation is the better shape + and it is not needed to serve the nightly harness. Un-defer when an operator + needs to deploy without starting a run. +- **Reading `SeiNetwork` directly.** The harness holds the resource and fills the + chain file from it. Un-defer if a run ever starts without a harness that knows + its chain. +- **Verifying the genesis against its S3 URI.** The registry records the URI. It + does not fetch it, which would put S3 credentials in a load generator. The + genesis hash already identifies the chain. Un-defer if a chain file's hash and + its URI are ever suspected of disagreeing. +- **A registry covering contracts this repo does not deploy.** CDR-004 lets a + profile name an address. Nothing else about foreign contracts is in scope. +- **A preflight step that deploys once for many runs.** CDR-013 gives each run + its own contracts, which is what keeps each run's contention its own. Sharing + one set needs a step that deploys before any run starts, and injects the + addresses. Un-defer when a workload needs many runs driving one contract on + purpose, rather than by accident. + +## Verification + +A requirement with no test that names its ID is not done. + +- CDR-001 to CDR-005: drive a mock chain and assert which addresses the run binds + and whether it sent a deployment. +- CDR-006 to CDR-009: return absent code, then wrong code, from the mock. Assert + the startup error names the chain, the role, the address, and both hashes. +- CDR-010 and CDR-011: assert the emitted entry's fields, and assert the run left + the committed registry unchanged. +- CDR-014 to CDR-016: assert an entry matches on chain id and genesis hash, and + that a forcing profile deploys where an entry exists. +- CDR-021 to CDR-023: assert no scenario field holds an address, and that a run + with one scenario resolves one role. +- CDR-012: count startup requests against a mock, across two contract counts. +- CDR-013: start two builders against one mock chain and assert both complete. diff --git a/specs/contract-deployment-registry/tasks.md b/specs/contract-deployment-registry/tasks.md new file mode 100644 index 0000000..302aed7 --- /dev/null +++ b/specs/contract-deployment-registry/tasks.md @@ -0,0 +1,224 @@ +# Tasks: Contract deployment and the contract registry + +**Input**: `spec.md` (23 requirements), `plan.md`, `data-model.md`, +`contracts/registry-api.md`, `quickstart.md` + +**Format**: `[ID] [P?] [Story] Description` + +- **[P]**: runs in parallel with its siblings. Different files, no shared edit. +- **[Story]**: US1, US2, US3, or blank for setup and foundation. +- Each task names the requirement IDs it closes. A requirement with no test that + names its ID is not done, so the mapping is the completion check. + +Revised 2026-08-23 after review. `Role` and `Deployment` became one `Contract`. +A task now deletes `AttachScenario` rather than calling it. The preparation step +gained the tasks it never had. + +## A note on delivery order before you read the phases + +US1 is the primary story: run against a chain the binary knows and deploy +nothing. Its code is testable against a mock from the start. + +It is not *useful* until US2 has run. No contracts exist on arctic-1, atlantic-2 +or pacific-1 today, so the compiled-in chain files for those chains cannot hold +addresses yet. Somebody runs US2 against each long-lived chain, reviews the +written file, and commits it. Only then does US1 do anything in production. + +Build US1 first, because US2 and US3 both need its types and its verification. +Expect US2 to be the first phase that changes what an operator sees. + +## Phase 1: Setup + +- [ ] T001 Create `registry/` with `doc.go` stating the package's one job and its + import boundary, and an empty `registry/chains/` holding a `.gitkeep`. +- [ ] T002 [P] Add `registry/boundary_test.go` asserting `go list -deps + ./registry` names no sei-load package except `generator/bindings`. + **Closes CDR-017.** Write it now so it guards every later task. + +T002 passes trivially against an empty package. That is the point: it fails the +moment a later task reaches for `config` or `types`. + +## Phase 2: Foundational + +Blocking prerequisites. No user story starts before these land. + +- [ ] T003 Add `Contract`, `Chain` and `Registry` to `registry/registry.go`, per + `data-model.md`. `Chain.Contracts` is a `[]Contract`, each carrying its + name, address and code hash. `common.Address` and `common.Hash` marshal + themselves, so add no custom marshaller. **Closes CDR-005.** +- [ ] T004 Add `chainKey{chainID, genesisHash}` and `Registry.Chain`, matching on + both fields and on nothing else. **Closes CDR-014, CDR-015.** +- [ ] T005 [P] Add `//go:embed chains/*.json` and `Load(paths ...string)`, + layering supplied files over compiled-in ones. **Closes CDR-018, CDR-019, + CDR-020.** +- [ ] T006 [P] Add `Registry.Sources()` so a run logs which file supplied each + chain. **Closes CDR-020.** +- [ ] T007 Add `registry/testdata/` with a two-chain fixture, one holding a + contract and one empty. Every later test reads it rather than building a + `Chain` inline. + +## Phase 3: User Story 1 - Run against a chain the binary knows (Priority: P1) + +**Goal**: the run finds the entry, verifies the code, hands each scenario its +bound contract, and sends no deployment transaction. + +**Independent test**: run a profile naming a chain in the fixture, against a mock +chain holding the recorded code. Assert the run sent no deployment, and that no +scenario holds an address. + +- [ ] T008 [P] [US1] Add `CodeReader`, and a fake in `registry/registry_test.go` + returning code, no code, or an error. +- [ ] T009 [US1] Add `Chain.Contract(name)`, returning two values because a miss + is ordinary. +- [ ] T010 [US1] Add `Verify(ctx, code, chain, name)`, hashing `eth_getCode` + output with Keccak-256 and comparing it to the recorded hash. + **Closes CDR-006, CDR-007.** +- [ ] T011 [US1] Add `Resolve(...)` returning an address to bind, or a signal to + deploy, and verifying before it returns an address. + **Closes CDR-001, CDR-002, CDR-003.** +- [ ] T012 [US1] Add the preparation step in `generator/`. It reads the profile, + collects the contracts its scenarios need, and dials one client for the + whole step. **Closes CDR-022, CDR-023.** A contract no scenario drives is + never resolved. +- [ ] T013 [US1] Bind each resolved address with `GetBindFunc` and that one + client, then call `SetContract`. **Delete `AttachScenario`** from + `ScenarioDeployer` and its implementors, and return an error where it + panicked. **Closes CDR-021.** +- [ ] T014 [US1] Assert a registry hit issues no deployment, and that no scenario + field holds an address. **Closes CDR-002, CDR-021.** Break the fixture + address to confirm the assertion fails before trusting it. + +T013 is the largest task in the feature. It removes a method from an interface +with four implementors, so it lands on its own rather than beside T012. + +## Phase 4: User Story 2 - Bring up a new chain and record what it deployed (Priority: P2) + +**Goal**: no entry exists, so the run deploys and writes a chain file an operator +reviews. + +**Independent test**: run against a chain absent from the fixture. Assert the run +deployed, and wrote a file naming the chain, the contract, the address, and the +code hash it observed. + +- [ ] T015 [P] [US2] Add `Record(ctx, code, name, addr)`, reading the deployed + code and returning the `Contract`. **Closes CDR-010.** +- [ ] T016 [US2] Add `WriteChain(path, chain)`. It refuses any path inside the + compiled-in `chains/` directory. **Closes CDR-011.** Assert the refusal, + not only the success. +- [ ] T017 [US2] Add a `--chain-file` flag to `main.go`, repeatable, passed into + `Load`. **Closes CDR-019.** +- [ ] T018 [US2] Have the preparation step call `Record` after a scenario + deploys, and write the chain file beside the run report. +- [ ] T019 [US2] Assert a supplied file beats a compiled-in chain of the same + identity. Assert the run logs which one it used. **Closes CDR-020.** + +## Phase 5: User Story 3 - A stale entry fails the run (Priority: P3) + +**Goal**: a recorded address holding nothing, or holding different code, stops +the run at startup instead of corrupting it. + +**Independent test**: point a run at a fixture address the mock serves no code +for. Assert startup fails and the message names the chain, the contract, the +address, and both hashes. + +- [ ] T020 [P] [US3] Add `MismatchError` with `ChainName`, `ChainID`, + `ContractName`, `Address`, `Want` and `Got`, where a zero `Got` means + absent code. **Closes CDR-008.** +- [ ] T021 [US3] Assert `Verify` returns `*MismatchError` for absent code and for + wrong code, and that the message distinguishes them. **Closes CDR-008.** +- [ ] T022 [US3] Assert no path deploys after a mismatch. **Closes CDR-009.** + This is the requirement most likely to erode, because redeploying looks + like a fix. +- [ ] T023 [US3] Assert the run fails before it sends any load, not after. + **Closes CDR-006.** + +## Phase 6: Polish and cross-cutting + +- [ ] T024 [P] Add the force-deploy flag to `config.Scenario`, and honour it in + the preparation step by skipping `Resolve`. **Closes CDR-016.** +- [ ] T025 [P] Add an explicit-address field to `config.Scenario` for a contract + deployed outside this repo. **Closes CDR-004.** +- [ ] T026 [P] Assert startup issues one `CodeAt` per contract, and that the + count does not move with account count. **Closes CDR-012.** +- [ ] T027 [P] Assert two runs against one ephemeral chain each deploy their own + contracts, and that neither drives the other's. **Closes CDR-013.** +- [ ] T028 Add a CI check that every committed chain file parses, and holds each + field the format names. **Closes success criterion 8.** +- [ ] T029 Walk `quickstart.md` end to end against a local chain and correct what + does not match. + +## Dependencies + +```text +Setup (T001, T002) + │ +Foundational (T003 → T004 → T005/T006 → T007) + │ + ├── US1 (T008 → T009 → T010 → T011 → T012 → T013 → T014) + │ │ + │ ├── US2 (T015 → T016 → T017 → T018 → T019) + │ └── US3 (T020 → T021 → T022 → T023) + │ + └── Polish (T024 … T029) +``` + +US2 and US3 both need US1's types and its `Verify`. They do not need each other, +so they run in parallel once US1 lands. + +## Parallel execution + +Within Foundational: T005 and T006 touch different functions in the same file, so +sequence them if one author holds both. + +Within US1: T008 is independent of T009 and T010. T012 and T013 are not parallel +with each other, because T013 consumes what T012 builds. + +Within US3: T020 lands first, then T021, T022 and T023 all consume it and touch +different assertions. + +Within Polish: every task carries `[P]` except T028 and T029. + +## Implementation strategy + +**MVP is US1 plus US2.** US1 alone reads a registry nothing has written. US2 +alone deploys every run, which is today's behaviour. Together they close the +loop: deploy once, record, and reuse. + +**US3 ships with the others despite being third.** Without it a stale entry +produces calls that land, do nothing, and report as included, because sei-load +reads inclusion rather than execution status. US1 and US2 shipping without US3 +would make that failure reachable in production. + +**Polish is genuinely deferrable, with one exception.** T024 to T027 close +requirements nothing depends on yet. T028 is worth pulling forward if a chain +file reaches `main` before the phase does. + +## Traceability + +| Requirement | Task | +| -- | -- | +| CDR-001 | T011 | +| CDR-002 | T011, T014 | +| CDR-003 | T011 | +| CDR-004 | T025 | +| CDR-005 | T003 | +| CDR-006 | T010, T023 | +| CDR-007 | T010 | +| CDR-008 | T020, T021 | +| CDR-009 | T022 | +| CDR-010 | T015 | +| CDR-011 | T016 | +| CDR-012 | T026 | +| CDR-013 | T027 | +| CDR-014 | T004 | +| CDR-015 | T004 | +| CDR-016 | T024 | +| CDR-017 | T002 | +| CDR-018 | T005 | +| CDR-019 | T005, T017 | +| CDR-020 | T005, T006, T019 | +| CDR-021 | T013, T014 | +| CDR-022 | T012 | +| CDR-023 | T012 | + +Every requirement has a task. No task closes nothing. diff --git a/specs/transaction-outcome-tracking/spec.md b/specs/transaction-outcome-tracking/spec.md new file mode 100644 index 0000000..cc96ad7 --- /dev/null +++ b/specs/transaction-outcome-tracking/spec.md @@ -0,0 +1,267 @@ +# Transaction outcome tracking + +Status: DRAFT +Owner: platform +Linear: PLT-466 + +## Anchors + +EARS · RFC 2119 · Spec Kit · goodput · coordinated omission · back-pressure · +conservation identity · fail-closed + +## Why this needs a specification + +sei-load reports how many transactions it offered, and how many reached a block. +It does not report how many did what the workload asked. + +A run can report a million accepted, near-perfect inclusion, and a healthy p99 +while every transaction reverted. Nothing in the output says so. + +One missing metric does not cause this. Two properties of the design do. + +**Two ledgers, and nothing joins them.** `stats.Collector` counts what the sender +submitted. `stats.InclusionTracker` counts what became of those submissions. The +tracker holds no reference to the collector. The collector never learns an +outcome. Neither one can produce a ratio of committed to offered. + +**The tracker observes delivery, not execution.** Its terminal states say whether +a transaction arrived. It derives `Included` from the transaction hashes in a +block, and a hash carries no execution status. `Included` therefore covers a +transaction that committed and one that reverted and burned its gas. + +This lands hardest where a revert is the expected failure. A contract call with a +bad argument reverts. A precompile call from an unassociated account reverts. Each +produces an included transaction and no effect. sei-load is growing exactly those +workloads. + +## Clarifications + +### Session 2026-08-23 + +- Q: Where does the tracker get execution status? → A: Replace the block fetch + with `eth_getBlockReceipts`. Receipts carry the transaction hash and the + status, so one call returns what two would. +- Q: How does the sender hand a transaction to the tracker? → A: Through a + channel. The overflow policy becomes a terminal state. +- Q: Does the tracker own the accounting or feed it? → A: It feeds the collector, + which already holds the per-scenario and per-operation counts. + +## Scope + +This specification covers how sei-load records the outcome of a transaction it +submitted, and how that outcome reaches the run's accounting. + +Out of scope: which workloads sei-load drives, how the sender schedules arrival, +and how the report presents these counts. The report consumes what this produces. + +## Current behaviour + +Stated so a reader sees what changes and what does not. + +- `sender.ShardedSender` calls `InclusionTracker.Register` after a successful + send, which makes `registered` a subset of `accepted`. +- `Register` is a synchronous call. It takes the tracker's mutex, and the tracker + holds that same mutex while it matches a block. +- The tracker subscribes to new heads. It fetches each block's transaction hashes + once per block, not once per transaction. +- An earlier design fetched a receipt per transaction. The rewrite removed that + path, because its cost grew with the throughput the run offered. +- A reap loop evicts a transaction still in flight after `reapAfter`, as + `Expired`. Registration past the registry cap becomes `DroppedAtCap`. +- `sender/doc.go` states a conservation identity: + `registered = included + expired + inflight_at_shutdown`. +- `Collector.RecordTransaction` records the submission and its send latency. + +## User stories + +**Story 1. An operator reads a run and learns whether the workload worked.** + +Today the operator reads an inclusion rate. A run whose transactions all reverted +looks like a run that succeeded. + +**Independent Test**: drive a mock chain that reverts every transaction. The run +report states zero committed. It does not state a healthy inclusion rate alone. + +**Story 2. An engineer compares two runs and trusts the difference.** + +A change that raises the revert rate looks like a change that raised throughput, +because a reverted transaction still counts as delivered. + +**Independent Test**: drive two mock runs with the same offered rate and different +revert rates. The reported goodput differs. The reported inclusion rate does not +carry the difference alone. + +**Story 3. The load generator does not slow itself down while measuring.** + +The sender hands each successful send to the tracker under a shared mutex. It +waits behind a block match in progress. + +**Independent Test**: measure the hand-off latency while the tracker matches a +block. The tail stays within the stated budget. The current design fails this. + +## Requirements + +Every requirement carries an ID. A test names the ID it covers. + +### Outcome states + +**TOT-001.** The tracker MUST distinguish a transaction that committed from one +that reverted. It MUST NOT report one state that covers both. + +**TOT-002.** WHEN the tracker matches a registered transaction in a block, the +tracker MUST resolve it to exactly one terminal state. + +**TOT-003.** The terminal states MUST partition every registered transaction. The +conservation identity MUST hold at shutdown over the new states. + +**TOT-014.** The hand-off from sender to tracker is a channel. IF the channel is +full, THEN the sender MUST NOT block. The run MUST count that transaction under a +terminal state naming the hand-off. + +Rationale for TOT-014: a channel that blocks when full brings back the stall +TOT-010 removes. A channel that drops in silence breaks TOT-003, because a +registered transaction would reach no terminal state. Naming the drop keeps both +true. It also makes the queue size a number an operator sees, rather than a +constant nobody revisits. + +**TOT-004.** IF the tracker matches a transaction but cannot determine its +execution status, THEN the tracker MUST record a state that names the missing +status. It MUST NOT record that transaction as committed. + +Rationale for TOT-004: a status source can fail on its own, while the block fetch +succeeds. Counting an unknown as a success is the defect this specification +removes. Counting it as a revert would invent a chain outcome. + +### Accounting + +**TOT-005.** The run's accounting MUST report offered, accepted, rejected, +committed, reverted, and dropped. + +**TOT-006.** The accounting MUST report those counts per scenario and per +operation. It MUST use the key the send path already labels its metrics with. + +**TOT-007.** The accounting MUST expose the ratio of committed to offered. + +**TOT-016.** The tracker MUST report each terminal outcome to the collector. The +collector MUST own the counts that TOT-005 to TOT-007 report. + +Rationale for TOT-016: the collector already holds per-scenario and +per-operation counts, so one ledger produces the ratio and one report reads it. +The alternative makes the collector key by transaction, which is state it does +not carry. + +**TOT-008.** The accounting MUST separate a reverted transaction from a dropped +one. The system under test produces a revert. The mempool or the run's own +timeout produces a drop. + +### Cost + +**TOT-009.** The tracker MUST NOT issue one request per transaction to determine +execution status. It MUST read status at block granularity. + +**TOT-015.** The tracker MUST get hashes and status from one call per block. It +MUST NOT keep a separate call for hashes. + +Rationale for TOT-015: `eth_getBlockReceipts` returns the transaction hash and +the status together. The call the tracker already makes per block becomes the +call that carries status. Sei serves it, and a `sei_` variant as well. Keeping +the hash-only call beside it would add a round trip, and a second thing to keep +consistent. + +Rationale for TOT-009: the rewrite removed the per-transaction receipt path. One +extra request per transaction scaled the tool's own load with the load it +offered. A block-granular source keeps the fixed per-block cost. + +**TOT-010.** Handing a submitted transaction to the tracker MUST NOT block the +send path behind work the tracker performs for another transaction. + +Rationale for TOT-010: `Register` shares one mutex with block matching. A sender +completing a send waits behind a match. The reported send latency hides that +wait, because the sender measures latency before the call. A load generator that +stalls its own send path understates the number it exists to report. + +**TOT-011.** WHILE a run is in progress, the registry cap MUST bound the +tracker's memory. Memory MUST NOT grow with the number of transactions sent. + +### Reporting + +**TOT-012.** WHERE a run writes a report, the report MUST state which counts +cover execution and which cover delivery only. + +**TOT-013.** IF execution status is unavailable for a whole run, THEN the report +MUST say so. It MUST NOT omit the execution counts silently. + +Rationale for TOT-013: an absent number reads as zero. A run against an endpoint +that serves no status MUST NOT look like a run with no reverts. + +## Success Criteria + +The specification closes when all of the following hold. + +1. A run against a chain that reverts every transaction reports zero committed + and a non-zero reverted count. +2. A run reports a goodput ratio, per scenario and per operation. +3. Every requirement ID above has at least one test that names it. +4. The tracker issues no more requests per block than it issues today. +5. The hand-off tail latency from sender to tracker stays inside a budget the + implementation states and measures. +6. `sender/doc.go` states the conservation identity over the new terminal states, + and a test asserts it. +7. `make verify` passes. + +## Decisions this specification makes + +The clarification session settled three. This section names the alternative each +one rejected, because each alternative reads as reasonable. + +**The status source is `eth_getBlockReceipts`.** Sei serves it, and it carries +the transaction hash beside the status. It replaces the tracker's per-block hash +fetch rather than adding to it, so the request count per block does not move. +TOT-015 states this. + +Rejected: a receipts-root derivation, and a trace call. Both are block-granular. +Both need more from the endpoint than a call Sei already serves. + +**The hand-off is a channel, not a shortened lock hold.** A channel decouples the +sender from block matching. Shortening the lock hold leaves the tail in place, +and that tail is the defect TOT-010 removes. + +The cost is a new terminal state. A full channel MUST NOT block the sender, so a +drop needs a name to keep TOT-003 true. TOT-014 states this. + +**The tracker feeds the collector.** The collector already holds the per-scenario +and per-operation counts. One ledger produces the ratio, and one report reads it. +TOT-016 states this. + +Rejected: the collector reads the tracker at report time. That needs the +collector to key by transaction, which is state it does not carry. + +## Decisions this specification does not make + +Both concern the channel from TOT-014. Neither changes an interface, so neither +blocks the design. + +**D-1. The queue depth.** The depth sets how long a tracker stall the sender +absorbs before it drops. Too small drops under ordinary jitter. Too large hides a +tracker that cannot keep up. + +An implementation MUST state the depth and report the drop count. An operator +then reads a number rather than inheriting a constant. + +**D-2. Whether a hand-off drop voids the run.** A drop means the run cannot +account for a transaction it submitted. A few drops leave the goodput ratio +usable. Many make it a lower bound rather than a measurement. + +The verdict rule needs a threshold. Nothing yet says what that threshold is. + +## Verification + +A requirement with no test that names its ID is not done. + +- TOT-001 to TOT-004 and TOT-008: drive a mock block source that returns a block + holding a reverted transaction. The tracker's tests already use a mock source. +- TOT-005 to TOT-007: assert the counts and the ratio from the collector. +- TOT-009: count the requests the tracker issues per block against a mock. +- TOT-010: measure the hand-off tail while the tracker matches a block. +- TOT-011: register past the cap, then assert the registry size. +- TOT-012 and TOT-013: assert against the report text.