feat(cli): add docker agent usage to report token and cost spend - #3944
feat(cli): add docker agent usage to report token and cost spend#3944dwin-gharibi wants to merge 4 commits into
docker agent usage to report token and cost spend#3944Conversation
…mmand for being able to see usage outside of tui
… agent usage command
docker-agent
left a comment
There was a problem hiding this comment.
Assessment: 🟡 NEEDS ATTENTION
Two logic issues found in the new aggregation code in pkg/usage/usage.go:
- Per-model call count can be over-counted when an item carries a model name but no usage data — see inline comment at line 141.
- CostIncomplete flag is not triggered for sessions whose only token spend is Reasoning tokens, because
Total()returnsInput + Outputand excludes Reasoning — see inline comment at line 158.
Both findings are in newly added + lines. No issues found in cmd/root/usage.go, cmd/root/root.go, or the test files.
Two review findings, addressed differently because only one is a defect. Reasoning-only spend was genuinely mis-reported. Total() is input+output by design, so a run that burned only reasoning tokens reported Total()==0, the unpriced check never fired, and an understated cost was presented as a genuine $0.00 — exactly what that check exists to prevent. Adds Tokens.AnySpend and keys the check on it; Total keeps its display meaning. Per-model Calls counting a response whose provider reported no usage is not over-counting: the call happened, the tokens were simply never reported (usage tracking off, or an older session). Dropping it would hide a model served entirely by such a provider, which is the worse failure. Kept, but the short token columns are no longer mysterious: ModelRow.Unmetered records how many calls came without usage, and the report warns that token counts are understated — the same treatment unpriced cost already gets.
Pins that an attributed model with no reported usage still counts as a call and is flagged as unmetered, that a reasoning-only session is flagged as unpriced despite Total()==0, and that the understated-tokens note names only the models it applies to.
|
Done. @aheritier |
aheritier
left a comment
There was a problem hiding this comment.
Assessment: 🔴 Request changes — the aggregation misses sub-agent spend, and cost is read from the legacy session field
The shape of this is right: a pure Aggregate over data that is already persisted, no schema change, a thin CLI on top, --json for CI. The gap it closes is real, and the care shows (deterministic tie-breaks, [] rather than null, rune-safe truncation). The unmetered-calls instinct from the last round is also validated by data — in a 145,644-message store only 3 assistant messages carry no usage, so that warning won't be noisy.
But two defects make the reported numbers wrong on the sessions that matter most, and I could only find them by running the command against a real store rather than fixtures.
Verified against PR head 38d3dd12, built and tested on top of current main, and smoke-run against a copy of a real 1.5 GB session store (4,642 sessions, 145,644 messages).
🔴 [blocking] Sub-agent (sub-session) spend is dropped from every breakdown
pkg/usage/usage.go:159-196 walks only s.Messages, and itemModelAndUsage (pkg/usage/usage.go:238-244) handles item.Message and the non-message item fields but never item.SubSession. Sub-sessions are not reachable from the top level either: GetSessions is root-only (pkg/session/store.go:777-779, WHERE parent_id IS NULL OR parent_id = ''). The TUI cost dialog this command mirrors does recurse — pkg/tui/dialog/cost.go:305-308.
One real root session with 46 sub-sessions:
| truth (recursive, as the TUI computes) | this command | |
|---|---|---|
| input tokens | 219,577 | 334 (0.2%) |
| output tokens | 327,351 | 136,234 |
| model calls | 471 | 167 |
| tool calls | 492 | 156 |
Store-wide: 87,982 of 145,644 messages (60%) live in sub-sessions, holding 63.8M input tokens, 2.63B cached-input tokens and $2,273 of recorded cost — none of it reported. transfer_task shows up in the tool table, so multi-agent sessions are the normal case. "Which agent is expensive?" — the use case in #3943 — gets a wrong answer with no warning.
🔴 [blocking] Cost comes from the legacy session-level field, producing $0.00 plus a false "no pricing" warning
pkg/usage/usage.go:161 uses Cost: s.Cost. pkg/session/session.go:1518-1521 documents that field as existing "only for backward-compatible persistence", with TotalCost() as the canonical figure — and every other consumer uses TotalCost(): pkg/server/session_manager.go:527, pkg/app/export/html.go:119, pkg/tui/dialog/cost.go:347, pkg/tui/components/sidebar/sidebar.go:972.
Combined with the AnySpend() && Cost == 0 heuristic at pkg/usage/usage.go:198-204, here is the actual output of the built binary for session 6047f54d:
6047f54d ... 334 24.8M 136.2K $0.0000+ <title>
! Cost is understated: no pricing for anthropic/claude-opus-4-8
That session's TotalCost() is $44.45, and its own messages carry $34.82 of recorded per-message cost — the model is priced. Store-wide the total is understated ~25% ($7,224.17 from the cost column vs ~$9,611 actually recorded). This inverts the principle the PR argues for: rather than refusing to understate, it asserts something false and attributes it to a priced model.
🟠 [should-fix] The package doc states a constraint that isn't true — per-model cost is attributable
pkg/usage/usage.go:22-25 says cost is "persisted per session and per non-message item, never per message". chat.Message.Cost (pkg/chat/chat.go:91-92) is persisted inside message_json and reloads intact (I read $34.82 back out of the store), is summed by session.TotalCost() (pkg/session/session.go:1529), and is already attributed per model by pkg/tui/dialog/cost.go:303. The msg.Cost = 0 at pkg/session/session.go:1978 only clears a request-assembly copy. So the doc encodes a false data-model claim, and it is the justification for the finding above and for an omitted feature: fix the cost source and per-model cost falls out for free.
🟠 [should-fix] No test touches the sub-session path
25 tests, none constructing session.NewSubSessionItem, and nothing exercising loadUsageSessions or a real store. That gap is why both blocking findings shipped; two tests close it.
🟠 [should-fix] The whole store is loaded into memory before --since is applied
cmd/root/usage.go:104-108 calls GetSessions (every session, every item) and filters afterwards. Measured: --since 24h on a 1.5 GB store took 11.7 s and 2.70 GB peak RSS to print 3 sessions — enough to OOM a 2 GB CI runner, which is the advertised --json-in-CI use case. GetSessionSummaries (pkg/session/store.go:812-815) exists for exactly this reason; filter on metadata, then load items for the survivors.
🟠 [should-fix] --session rejects the ID the command prints
The table prints 8 chars via shortSessionID (cmd/root/usage.go:245-250), but --session 6047f54d fails with session not found. Accepting a unique prefix would make the output self-consistent.
🟠 [should-fix] Docs not updated
docs/features/cli/index.md documents every user-facing command (### docker agent eval at line 450, models at 192, toolsets at 216, plus share and serve *). A new top-level command belongs there in the same change.
🔵 [optional] A read-only report opens the store read-write and runs migrations
sqlitestore.New (pkg/session/sqlitestore/sqlitestore.go:24-60) runs migrations and, on failure, moves the database aside to .bak; it also takes a write lock, so usage can contend with a running agent. A read-only open would be proportionate for a reporting command.
🔵 [optional] Fourth divergent aggregation site
pkg/tui/dialog/cost.go:296-320 already walks a session for these exact numbers. Extracting that walker into pkg/usage and having the TUI consume it would stop the two from disagreeing — they currently disagree in both ways described above.
Worth doing: yes. The gap is genuine (figures reachable only from the TUI and OTel), the approach is proportionate, and no schema change is needed. It just needs the aggregation to see sub-sessions and to take cost from TotalCost(); do that and the per-model cost breakdown you documented as impossible becomes available too.
Token and cost figures were reachable only from the TUI's cost dialog and from OTel, so every
headless path —
--execin CI, the API server, MCP/A2A — was blind to spend. This adds a CLI reportover the session store.
Closes #3943.
--jsonfor CI,--session <id>for one session,--since <dur>for a window.No schema change
This is aggregation over data already persisted:
chat.Message.Usageand.Modelper message,session.Item.Cost/.Usage/.Modelfor non-message spend such as compaction, andsession.Session.Costfor the cumulative total.The
CACHEDcolumn is the reason token figures are summed from per-message usage rather than thesession's scalar counters — cached-input and cache-write are only broken out per message, and
without them prompt-caching wins are invisible.
Cost is reported honestly, including where it can't be
Three deliberate limitations, all surfaced rather than papered over:
Unpriced models are flagged, not shown as
$0.00. A model missing from the pricing cataloguerecords zero cost against real tokens — the runtime logs it, and it happens routinely. A session
that moved tokens but recorded no cost is marked
CostIncomplete, its cost prints with a trailing+, and the report ends with:A report that quietly understates spend is worse than no report.
No per-model cost. Cost is persisted per session and per non-message item, never per message, so
attributing it across the models used inside one session is not possible without a schema change.
Tokens per model are attributable and are reported; cost per model is simply absent rather than
guessed. The package doc says so.
Tokens and cost come from different sources and can disagree. An older session whose messages
carry no usage reports zero tokens while still reporting a cost. Documented in the package doc.
Structure
pkg/usage—Aggregate([]*session.Session) Report, a pure function with no I/O, so theaggregation is unit-testable without a database or a CLI.
Reportis JSON-tagged and is exactlywhat
--jsonemits.cmd/root/usage.go— flags, store access, and two more pure functions (filterSessions,renderUsage) that are tested directly againstbytes.Bufferrather than through a live command.Sessions are newest-first; models and tools are ordered by size descending so the cost drivers come
first. Ties break on name so output is deterministic.
Tests
pkg/usage/usage_test.go(9 tests): per-session token/cost sums including the cached andcache-write breakdown; per-model attribution with descending order; per-tool call counts;
unpriced-model flagging; newest-first ordering; non-message (compaction) item usage counted;
tolerance for items with no message, no usage, and no model;
Tokens.Total()semantics.cmd/root/usage_test.go(8 tests):--sincefiltering including that an undated session iskept (an unknown timestamp is not evidence of age) and that filtering does not mutate the caller's
slice; text rendering; the unpriced warning and the
+marker; the empty case; JSON round-trip;JSON emitting
[]rather thannullsojq '.sessions[]'works on a quiet day; tokenabbreviation; rune-safe title truncation.
Two details worth noting because they came from testing rather than from design:
filterSessionsoriginally filtered in place; the "does not mutate the caller's slice" test iswhy it clones.
"sessions": nullfor an empty report, which would break the--json | jquse case the help text advertises. Caught by smoke-running the real command.Smoke-tested end to end against a real (empty) SQLite store for the text, JSON, and
--helppaths.Verification
Toolchain
go1.26.5, darwin/arm64.go test ./pkg/usage/ ./cmd/root/golangci-lint run ./pkg/usage/... ./cmd/root/...(v2.12.2, CI's pin)go run ./lint .go build ./...,gofmt -lgo test ./...pkg/teamloaderfails — pre-existing (Google Cloud ADC), unrelatedLint initially reported 5 issues in the new code (
gofumpt,perfsprint,recvcheck,testifylint,unparam) — all fixed. Therecvcheckone was worth having:Tokenshad avalue-receiver
Total()alongside a pointer-receiveradd(), and sinceTokensis embedded inJSON-serialized rows a mixed receiver set is a genuine trap. It is now a free
addUsagefunctionwith value receivers throughout.
One flake to be aware of:
pkg/tools/mcp'sTestOAuthTransport_DoesNotForwardBearerAcrossOriginsfailed once during a full-suite run with
HTTP/1.x transport connection broken: CloseIdleConnections called. It passes in isolation and this PR does not touch MCP or HTTP — noting it because it willbite someone else on a loaded machine.