Verify citations in-stream and record answer provenance for trajectory review - #107
Open
cpsievert wants to merge 14 commits into
Open
Verify citations in-stream and record answer provenance for trajectory review#107cpsievert wants to merge 14 commits into
cpsievert wants to merge 14 commits into
Conversation
|
Preview deployed to Connect ( Deployed from commit 5b49723. |
|
Preview deployed to Connect ( Deployed from commit 5b49723. |
Two fixes to the browser suite added in c2d03a5: - Compare element counts in JS (`.length === N`) rather than returning an integer to `expect_identical()` against a double. - Wait for the fixture's final streamed chunk before clicking the citation pill. The fixture streams five chunks and shinychat re-renders the message on each one, so a pill matched mid-stream was replaced before its popover could open -- the test failed in 2 of 5 runs with a 30s timeout.
collect_appended_tags() and is_tool_result_content() aren't scanning concerns; relocate to utils.R alongside other cross-cutting helpers.
commons_icon_url() and svg_data_uri() both resolved system.file("figs",
...) and checked for a missing file; factor that into commons_icon_path().
The two stay distinct because they serve different renderers: the resource
path only resolves inside an app that ran register_commons_icon_resources()
(commons_ui()), while the data URI is self-contained for the trajectory
review app, which doesn't.
…inal S2 review question: the suite already exercises branched (edited-turn) and truncated/restored conversations, and those tests depend on matching by reconstructed content rather than a recorded from_index ordinal. Document why so the matching doesn't get flagged as dead weight again.
…e table Both rendered the same A/B/C provenance copy from separate hand-written strings that had already drifted (commons_answer_pill()'s "Untrusted" body text no longer matched provenance_aside()'s, and it had a "B" case provenance_aside() deliberately lacks). Extend provenance_display with a "B" entry and pill_class per tag so both renderers read the same label, icon, and body; provenance_aside() still excludes "B" explicitly, since a cited answer's UI is the citation asides, not a pill-shaped aside.
test-openai-citation-smoke-app.R targeted a gitignored sandbox fixture that only ever existed locally for its author, so it skip_if_not()'d for everyone else, including CI. Its assertions didn't add coverage beyond test-citations.R's synthetic-fixture tests of match_citation() and build_citation_corpus() -- it mostly checked the sandbox demo app's own hardcoded config against itself. A test that can never run anywhere but one machine reads as coverage it isn't.
shinychat's history controller reuses one client across new chats, conversation switches, edits, and branch navigation (all via set_turns()), while Commons minted one conversation id per client, so every conversation a session touched collapsed into a single trace conversation and only the last-active one survived trajectory_read() (#106). Two layers: - Commons$stream_async() rotates the conversation id whenever the current turns no longer extend the last streamed state, so divergent histories stop overwriting each other. - commons_server() persists the id in each saved shinychat conversation's app-state values (on_save) and reinstates it when that conversation is reopened (on_restore), so a switch or session restore continues under the conversation's original id instead of minting a fresh one and duplicating its history prefix. This layer is deletable once shinychat records stable conversation identity on its own spans.
cpsievert
force-pushed
the
feat/server-side-citation-verification
branch
from
August 13, 2026 15:59
9339639 to
eb7bed8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR reworks how a commons answer's trustworthiness travels from the model to the user, and
from the user's session to the audit trail. The verification logic itself isn't new — quotes
were already exact-matched against the corpus on the server. What changes is the architecture
around it, in two ways:
rewritten while the answer streams, so only server-authored elements ever reach the
browser. The client-side retrofit protocol is gone.
and citation decisions are written to the trace at the moment they're made, and
trajectory_review()plays them back verbatim instead of re-deriving them heuristically.Together with a conversation-identity fix (#106), the end state is: the pill a user saw and
the pill a reviewer sees are provably the same thing, and every conversation a session touched
is actually present in the audit trail.
The live path: stream → scanner → asides
Today, the model's raw
<citation>markup streams into the browser DOM, and after the turncompletes, the server sends verdicts over a custom message so
commons-chat.jscan replacethe rendered elements positionally. That final hop is fail-open: until the JS runs (or if it
miscounts), the user is looking at unverified markup dressed as citations.
Now
Commons$stream_async()wraps every answer in a streaming scanner (R/citation-scan.R):<commons-citation>— a short explanation plus ablockquote carrying the verbatim excerpt (
inst/prompts/citation-request.md). The blockform replaced the old single-line attribute form because it survives streaming: the scanner
can hold back a partially-arrived block and release everything else immediately.
dictionary entries), but mid-stream. Verified citations are rewritten inline as
<shiny-aside>elements naming their source; failed ones are dropped.dialect never reaches the browser unprojected, and model-spoofed
<shiny-aside>markup isneutralized. Stored turns keep the raw model text — projection is a display concern.
derive_provenance_tag()inR/provenance.R): governed-tool answers are "A", fallback-tool answers with a verifiedcitation "B", without one "C" — same semantics as before, with "B beats A" because the
citation is the thing the user is being asked to trust. A/C answers get a compact
provenance aside appended to the stream; a B answer's provenance UI is its citation asides.
Display rides shinychat's aside component, so the custom pill/citation protocol is deleted:
commons-chat.jsshrinks by ~225 lines,commons_server()sends no custom messages, andR/tagging.R(the old post-hoc provenance derivation) is gone entirely.The recorded path: spans → trajectory read-back → review
Previously the trajectory viewer inferred trust after the fact — tool names implied the tag
(
call_measure⇒ verified,run_sql⇒ cited-or-untrusted) and citations were re-scraped fromanswer text, with footnotes explicitly labeled "unverified". An inference can disagree with
what the user was actually shown, which is the wrong failure mode for an audit surface.
Now the decisions are recorded where they're made:
stream_async()writes the tag(
commons.provenance.tag) and the scanner's accept/reject decisions(
commons.citation.candidates, JSON) onto thecommons_conversation_turnspan.On the way back out,
trajectory_read()(R/trajectory-read.R) reads those attributes offevery recorded call in a conversation and attaches them per-exchange to the reconstructed
turns. The one genuinely subtle design decision is how a recorded call is matched to an
exchange: structurally, by comparing canonicalized exchange-content signatures as a prefix of
the final history — not by ordinal position. Conversation histories aren't append-only once
shinychat's edit and branch-navigation features are in play, so a position recorded by one
call can name the wrong exchange in another call's reality. The comment above
exchange_signature()and the provenance block oftest-trajectories.R(linear, edited,restored, switched, conflicting, incomplete) are the spec for this; I'd start a review of the
read-back side there.
trajectory_review()then becomes a playback device: it reconstructs citation asides from thestored raw turns using the same scanner, resolved by the recorded decisions rather than a
corpus (it never re-verifies), surfaces rejected-citation counts in a review-audit aside, and
takes its pill copy from the same display table the live aside uses so the two can't drift.
Missing or conflicting records fail closed to "no claim" — the viewer never invents a verdict.
Conversation identity (#106)
While wiring the recorded path I found that shinychat's history feature reuses one client
object across new chats, conversation switches, and edits (all via
set_turns()), whilecommons minted one conversation id per client — so every conversation a session touched shared
one id, and
trajectory_read()'s latest-span-wins reconstruction silently kept only the lastone. Fixed in two layers (one commit,
R/commons.R+R/chat.R):stream_async()rotates the conversation id whenever the current turns no longer extendthe last streamed state.
commons_server()persists the id in each saved conversation's app-state values viashinychat's
on_save/on_restorehooks, so reopening a conversation continues under itsoriginal id instead of duplicating its history under a fresh one. This layer is commented
as deletable once shinychat records stable conversation identity on its own spans.
Reviewing guide
R/citation-scan.R, speccedexhaustively in
test-citation-scan.R.stream_async()inR/commons.R; tag semantics and display table inR/provenance.R.R/trajectory-read.R+ the provenance tests intest-trajectories.R.R/trajectory-review.Ris mostly deletion of the inference path plus playback.test-citation-browser.Randtests/testthat/apps/— hence the new Suggests.Trade-offs and notes
show no trust pill in the reviewer. Deliberate: no claim beats an inferred claim, and the
package's pre-release stance means no compat shims.
trajectory_read()now parses one chat span per recorded exchangerather than one per conversation. Fine at review scale; noted in case it ever matters.
the reviewer doesn't yet present them as siblings of the edited conversation — tracked as
the open half of Audit trail silently drops conversations/branches: conversation_id never rotates across shinychat history operations #106.