[WIP] Preset patching and per-trial records - #4118
Draft
peterschmidt85 wants to merge 4 commits into
Draft
Conversation
A trial was one appended line in a shared `trials.jsonl`, mirrored into the session directory by byte offset. An agent rewriting the file moved every offset, and the mirror then committed a torn fragment forever: one verified preset still carries an unparseable trial record. Service attempts had the same shape in `verifications.jsonl`. Every record is now written exactly once and reading is listing a directory: - `trials/<n>/` holds the trial's compiled `task.dstack.yml` and its `trial.json`; the presence of `trial.json` is what marks the trial completed, so in-flight and torn states are visible instead of corrupting. - `service/<k>/` holds each verification attempt's submitted YAML and a `verification.json` written when the attempt ends; an attempt directory without a result is one still in progress. - The byte-offset record mirrors are replaced by a stateless directory mirror that re-lists the source and copies changed files whole, scrubbed, and atomically. A torn read can never be committed; the next pass converges. Trial and attempt directories sort numerically, pinned past 9. The trial contract in the agent prompt is rewritten around the layout, with every per-trial file referenced by its full path and the write order stated where the files are defined. The listing shows the trial being worked on rather than the completed count while trialing, so `trialing (2/3)` cannot read as two finished; `verifying (3/3)` keeps the completed count. Validated end to end: a session on one RTX PRO 4500 produced three contract-exact trial records, a recorded verification attempt, and a saved preset that `dstack preset apply` deploys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The directory mirror decoded each file to text and re-encoded it, which rewrote newlines (CRLF became LF on Unix, LF would become CRLF on Windows) and replaced non-UTF-8 bytes. Harmless for the CLI's own JSON and YAML records, but the mirror now also carries patch files, where the recorded copy must be exactly the bytes that ran. Copy bytes verbatim and redact at the byte level. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The preset agent may now patch the serving framework source code, generate custom kernels, and patch drivers during trials. Patches are recorded as unified diffs under `trials/<n>/patches`, referenced from the trial's `task.dstack.yml` via `files`, and applied with `patch` from its commands, so the recorded configuration reproduces the trial exactly. The verified service records its patches the same way under `service/<k>/patches`, and the saved preset re-roots `files` onto the mirrored session copies, so `dstack preset apply` reproduces the patched configuration after the agent workspace is deleted. When no trial meets the constraints, the best failed trial that has a benchmark is still verified and saved instead of failing the whole creation: the hardware's best is worth keeping even when it falls short of what was asked. To keep that honest, the final report now records the verified trial, presets store the requested `min_context_length` and `max_ttft` alongside the verified values, and `dstack preset apply` warns, instead of refusing, when the preset's verified context is below the requested one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A preset saved from a failed trial shows `*` next to its benchmark, the same mark a running session uses when it has only failed trials to show, computed from the requested constraints the preset now stores. In the trial spark, bars scale from zero so their heights compare as the numbers do, failed trials are red rather than gold, and gold marks the best result only while no trial meets the constraints. While trialing, the `(N/M)` progress counts the trial being worked on rather than the completed ones. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dstack preset createnow records every trial and final-service attempt in its own directory, and the agent may patch the serving framework source code, generate custom kernels, and patch drivers to reach better performance.trials/<n>/task.dstack.yml— a complete task configuration that reproduces the trial's benchmark — plustrials/<n>/trial.jsonwith the benchmark and what the trial taught. Final-service attempts are recorded the same way underservice/<k>/.trials/<n>/patches, delivered into the container viafilesand applied withpatchfrom the task commands. The saved preset carries them, sodstack preset applyreproduces the patched configuration.dstack presetmarks such presets with*next to the benchmark, anddstack preset applywarns when the preset's verified context length is below the requested one.Validated e2e on 1x MI300X with DeepSeek-V4-Flash: the agent authored and recorded an original AITER GEMM-dispatch patch as part of a verified preset ($12.6, 7 trials, +37% over the unpatched baseline).
To be done:
--previous🤖 Generated with Claude Code