Fix out-of-order dotted-key child promoted to a table swallowing following siblings - #588
Conversation
|
This fails to fix this modest variation on the original report: #!/usr/bin/env python3
import tomlkit
doc = tomlkit.parse("a.b.c = 1\na.b.d = 2\na.e = 3\n")
doc["a"]["b"]["c"] = {"x": 9}
output = tomlkit.dumps(doc)
print(output)still with output [a.b.c]
x = 9
a.b.d = 2
a.e = 3 |
|
You're right, thanks. That case fails the same way on The reposition I added only runs in A complete fix has to renormalize the top-level body whenever a nested promotion turns a previously inline fragment into a header, at any depth. That means either giving out-of-order fragments a path to the root container, or normalizing out-of-order groups at render time so ordinary layout is left alone. Patching one more level would just push the counterexample down a level, which is what happened to #558 and #575. I would rather not add a half-measure. If you are open to the root-level normalization approach I will take it in this PR; otherwise I am happy to close this and reopen once that is ready. Which do you prefer? |
1181d18 to
3766cca
Compare
|
Reworked it. Your case is fixed now, at any dotting depth. Instead of moving the promoted fragment in the proxy (which only saw one container), the document body is now rendered with inline entries before the header entries a mutation left after them. That is the order valid TOML always has at a single level: a The diff is smaller than before, since this replaces the proxy-level reposition and the two helpers with one render step. One boundary I want to be explicit about rather than have you find it: an out-of-order group nested inside a table header, e.g. doc = tomlkit.parse("[outer]\np.b = 1\np.c = 2\n")
doc["outer"]["p"]["b"] = {"x": 9}is still wrong, but for a different, pre-existing reason: the promoted table renders as |
|
still fails with this - which is even simpler than the original report #!/usr/bin/env python3
import tomlkit
doc = tomlkit.loads("a.b = 1\nz = 2\n")
doc["a"]["new"] = {"x": 1}
output = tomlkit.dumps(doc)
print(output)output a.b = 1
[a.new]
x = 1
z = 2 |
|
re your example which is "still wrong, but for a different, pre-existing reason". It is wrong for both reasons: the update captures too many keys and it loses the I would expect that a correct fix for #556 should handle the captures-too-many-keys part. |
3766cca to
eeaa445
Compare
|
Fixed, and generalised. That case is not out of order at all: I dropped the gate. The normalisation now keys on the actual violation, a keyed entry rendering inline after something that renders a header at the same level, rather than on any structural signal, so it covers all three shapes (out-of-order promotion, the nested form, and adding a table to a super table). A single cheap pass detects whether anything is trapped; a validly ordered document returns unchanged and renders byte-for-byte the same, so there is no layout change and no real cost for it (200 renders of a 500-table document stayed at roughly 2 ms each). Comments and whitespace carry no key, so they never count as trapped or move, which was the one regression the ungated version briefly caused and is now covered by keeping the check to keyed entries. Your |
|
per crossing update - the example you gave earlier is not only about the missing |
When a mutation makes a table render a `[header]` while a bare or dotted key that is not inside it still follows at the same level, the header swallows that key when the output is parsed again (python-poetry#556). This takes several shapes: assigning a table over a child of an out-of-order dotted-key table (`doc["a"]["b"] = {...}` over `a.b`/`a.c`), the nested `doc["a"]["b"]["c"] = ...` at any depth, adding a table to a super table (`a.b = 1` then `doc["a"]["new"] = {...}`, leaving a following `z = 2` after `[a.new]`), and the same shapes inside another table (`[outer]` containing `p.b`/`p.c`). A `[header]` followed at the same level by a bare or dotted key that is not inside it cannot come from parsing, only from mutation. Both the document and each rendered table now emit such trapped keys before the first header, which is the order valid TOML always has at a single level. A single cheap pass detects whether any keyed entry is trapped after a header; if not, the body renders byte-for-byte unchanged, so validly ordered documents are untouched and pay only one scan. Comments and whitespace carry no key and never move. Fixes python-poetry#556 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
eeaa445 to
ba3795f
Compare
|
You're right, and I was wrong to keep scoping that out. The captures-too-many-keys part belongs in this fix. The normalisation only ran on the document top level; the The lost |
|
please spend more time coming up with examples and testcases yourself - it is silly that I have to do this for you. Here are a couple more to get you started: doc = tomlkit.loads("a.x=1\nq.x=2\nr=3\n")
doc["a"]["new"] = {"v": 1}
doc["q"]["new"] = {"v": 2}(which makes quite the mess) doc = parse(
"""\
[[p]]
a.b = 1
a.c = 2
z = 3
"""
)
doc["p"][0]["a"]["b"] = {"x": 1} (which shows that you also need to take more care of Array-of-Table) Ideally a fix should also take trivia into account - eg consider doc = parse(
"""\
a.b = 1
# Documentation for a.c
a.c = 2
[z]
q = 3
"""
)
doc["a"]["b"] = {"x": 9} |
|
Implemented the requested coverage in The current head now preserves semantic round trips for all three examples:
It also resolves the earlier nested-table half completely: the header is now Fresh verification on this head: 1,064 tests passed, Ruff and formatting clean, compilation and diff checks passed, and 262 valid |
doc = tomlkit.loads("a.b=1\nz=2") # nb no trailing newline
doc["a"]["new"] = {"x": 1}z=2a.b=1
[a.new]
x = 1 |
|
Fixed in The reordered inline partitions were still relying on their original trailing newlines. In the no-trailing-newline case, the final I added the exact regression for |
2 participants
Fixes #556.
The bug
Promoting a dotted-key child from an inline value to a table can leave its new
[header]before keys that still render at the same scope. Re-parsing then captures those keys under the promoted table.The failure is not limited to one out-of-order fragment. A super table can contribute both inline keys and later headers, two mixed super tables can interleave, and the same shape can occur inside a regular table or an array-of-tables element. Moving whole body entries therefore cannot preserve the TOML scope in every case. Leading comment trivia can also become detached from the key it documents.
The fix
Rendering now partitions every container recursively:
This handles root documents, nested tables, arrays of tables, multiple mixed super tables, and arbitrary dotted-key depth without special-casing an example. Headerless super tables retain their existing newline trivia, preserving established exact-output contracts.
Regression coverage
The tests cover:
Verification
python -m pytest -q: 1064 passedpython -m ruff check tomlkit/container.py tests/test_toml_document.py: passedpython -m ruff format --check tomlkit/container.py tests/test_toml_document.py: passedpython -m compileall -q tomlkit/container.py: passedgit diff --check: passedtoml-testfixtures: no mismatchesRelation to prior attempts
#558 and #575 were closed without merge after exposing narrower entry-reordering approaches. This PR operates at the rendering scope where relative inline keys and absolute child headers can be separated consistently.