test: add ethdebug/conformance#215
Conversation
|
cc @gnidan |
gnidan
left a comment
gnidan
left a comment
There was a problem hiding this comment.
This conformance suite is awesome! Thank you for putting it together @djolertrk 🙏
Mostly I'm requesting changes to make sure we test against the JSON-Schemas themselves, but maybe you'll agree with my other comments also. Let me know if you run into trouble with the hyperjump stuff.
On a broader note, this stuff might get a bit complicated when we want to add schema versioning, since then suddenly the conformance tests have to grow in dimensionality... Have you thought about this at all? Obviously that conversation remains a bit premature [and definitely shouldn't block getting this PR merged]
| } | ||
|
|
||
| const knownSourceIds = sourceIds(artifact); | ||
| if (knownSourceIds.size > 0) { |
There was a problem hiding this comment.
Can knownSourceIds ever be empty if referencedSourceIds isn't? If so, we probably don't want to silently skip this check.
| const interactive = await runSoldb({ | ||
| executable: soldb!, | ||
| args: [ | ||
| "trace", | ||
| tx.transactionHash, | ||
| "--rpc", | ||
| anvil.rpcUrl, | ||
| "--ethdebug-dir", | ||
| debugDir.spec, | ||
| "--interactive", | ||
| ], | ||
| stdin: "break Counter.sol:8\ncontinue\nq\n", | ||
| }); | ||
|
|
||
| expect(interactive.exitCode).toBe(0); | ||
| expect(interactive.stdout).toContain( | ||
| "Breakpoint set at Counter.sol:8, PC", | ||
| ); | ||
| expect(interactive.stdout).toContain("Breakpoint hit at step"); | ||
| expect(interactive.stdout).toContain("Counter.sol:8, PC"); |
There was a problem hiding this comment.
This stuff gets a bit brittle, no? No way to control soldb as a library for this?
There was a problem hiding this comment.
Yes, agreed. I moved the REPL scripting/output matching behind SolDB adapter helpers, so the test no longer hardcodes the raw interaction inline.
Thanks!
Yes, I’ve thought about it. I think schema versioning can actually make the conformance suite cleaner rather than worse: each fixture can declare the ETHDebug schema version it targets, and the runner can validate it against the matching schema set, and new tests added for a schema bump can target the specific behavior or shape changed by that bump. |
gnidan
left a comment
gnidan
left a comment
There was a problem hiding this comment.
Thanks @djolertrk for handling all my concerns.
This is super exciting that this repo's CI will test against real live solc ethdebug output. Like, holy crap! Kudos for this.
3 participants
Add
conformancetests forethdebugproducers and consumers.