Allow stdin fuzz targets to suppress logs#4646
Conversation
Add an environment-variable switch that lets stdin fuzz targets use the dev-null test logger. This keeps direct invocations verbose by default, while external runners can opt into quieter passing-case replays.
|
🎉 This PR is now ready for review! |
|
Good - the types are compatible. This PR is a clean, mechanical change. Every file follows the identical pattern, the template is updated, and no fuzz targets were missed. The No issues found. |
|
🔔 1st Reminder Hey @TheBlueMatt! This PR has been waiting for your review. |
I guess I'm confused by this - generally once we hit a failure the rest will just be the same issue again and again reached with different inputs. Are you doing something to filter these cases? Otherwise I'm fine with this. |
After working on the fuzzer for a while, it turned out that I need to re-run quite large sets of fuzz strings (1000s) pretty deep inside the impl/test loop to verify that nothing accidentally broke after fixing the failures. Yes indeed, mainly to filter out what the new set of failures is, but also to keep track of all the panic sites, group them, detect timeouts, etc. Oftentimes there are multiple issues. Codex made himself a nice tool for this. |
TheBlueMatt
left a comment
There was a problem hiding this comment.
Mmm, okay, fair enough.
|
👋 The first review has been submitted! Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer. |
I agree that it is a bit messy at the moment with the various ways the fuzzer can run, and the fake/real split didn't make it better. I can imagine that we might want to do a consolidation step some time. |
Add an environment-variable switch that lets stdin fuzz targets use the dev-null test logger. This keeps direct invocations verbose by default, while external runners can opt into quieter passing-case replays.
I use stdin_fuzz also to run big sets of test cases using a runner that parallelizes and also continues on failure to examine the rest. Cargo test and test_cases dir approach do not work for that. The logging is a big slow down.