Skip to content

feat(cli): add docker agent usage to report token and cost spend - #3944

Open
dwin-gharibi wants to merge 4 commits into
docker:mainfrom
dwin-gharibi:feat/usage-reporting
Open

feat(cli): add docker agent usage to report token and cost spend#3944
dwin-gharibi wants to merge 4 commits into
docker:mainfrom
dwin-gharibi:feat/usage-reporting

Conversation

@dwin-gharibi

Copy link
Copy Markdown
Contributor

Token and cost figures were reachable only from the TUI's cost dialog and from OTel, so every
headless path — --exec in CI, the API server, MCP/A2A — was blind to spend. This adds a CLI report
over the session store.

Closes #3943.

$ docker agent usage --since 24h
SESSION   CREATED           INPUT   CACHED  OUTPUT  COST   TITLE
a1b2c3d4  2026-08-06 12:04  128.4K  96.2K   4.1K    $0.42  fix the failing cache test
3 session(s)                412.7K  310.1K  11.9K   $1.28

MODEL                    CALLS  INPUT   CACHED  OUTPUT
anthropic/claude-opus-5     41  380.2K  295.0K  10.4K

TOOL             CALLS
read_file           58
shell               21

--json for CI, --session <id> for one session, --since <dur> for a window.

No schema change

This is aggregation over data already persisted: chat.Message.Usage and .Model per message,
session.Item.Cost/.Usage/.Model for non-message spend such as compaction, and
session.Session.Cost for the cumulative total.

The CACHED column is the reason token figures are summed from per-message usage rather than the
session's scalar counters — cached-input and cache-write are only broken out per message, and
without them prompt-caching wins are invisible.

Cost is reported honestly, including where it can't be

Three deliberate limitations, all surfaced rather than papered over:

Unpriced models are flagged, not shown as $0.00. A model missing from the pricing catalogue
records zero cost against real tokens — the runtime logs it, and it happens routinely. A session
that moved tokens but recorded no cost is marked CostIncomplete, its cost prints with a trailing
+, and the report ends with:

! Cost is understated: no pricing for test/fake-root

A report that quietly understates spend is worse than no report.

No per-model cost. Cost is persisted per session and per non-message item, never per message, so
attributing it across the models used inside one session is not possible without a schema change.
Tokens per model are attributable and are reported; cost per model is simply absent rather than
guessed. The package doc says so.

Tokens and cost come from different sources and can disagree. An older session whose messages
carry no usage reports zero tokens while still reporting a cost. Documented in the package doc.

Structure

pkg/usageAggregate([]*session.Session) Report, a pure function with no I/O, so the
aggregation is unit-testable without a database or a CLI. Report is JSON-tagged and is exactly
what --json emits.

cmd/root/usage.go — flags, store access, and two more pure functions (filterSessions,
renderUsage) that are tested directly against bytes.Buffer rather than through a live command.

Sessions are newest-first; models and tools are ordered by size descending so the cost drivers come
first. Ties break on name so output is deterministic.

Tests

pkg/usage/usage_test.go (9 tests): per-session token/cost sums including the cached and
cache-write breakdown; per-model attribution with descending order; per-tool call counts;
unpriced-model flagging; newest-first ordering; non-message (compaction) item usage counted;
tolerance for items with no message, no usage, and no model; Tokens.Total() semantics.

cmd/root/usage_test.go (8 tests): --since filtering including that an undated session is
kept (an unknown timestamp is not evidence of age) and that filtering does not mutate the caller's
slice; text rendering; the unpriced warning and the + marker; the empty case; JSON round-trip;
JSON emitting [] rather than null so jq '.sessions[]' works on a quiet day; token
abbreviation; rune-safe title truncation.

Two details worth noting because they came from testing rather than from design:

  • filterSessions originally filtered in place; the "does not mutate the caller's slice" test is
    why it clones.
  • The JSON path originally emitted "sessions": null for an empty report, which would break the
    --json | jq use case the help text advertises. Caught by smoke-running the real command.

Smoke-tested end to end against a real (empty) SQLite store for the text, JSON, and --help paths.

Verification

Toolchain go1.26.5, darwin/arm64.

Check Result
go test ./pkg/usage/ ./cmd/root/ ok
golangci-lint run ./pkg/usage/... ./cmd/root/... (v2.12.2, CI's pin) 0 issues
go run ./lint . 1770 files, no offenses
go build ./..., gofmt -l clean
go test ./... only pkg/teamloader fails — pre-existing (Google Cloud ADC), unrelated

Lint initially reported 5 issues in the new code (gofumpt, perfsprint, recvcheck,
testifylint, unparam) — all fixed. The recvcheck one was worth having: Tokens had a
value-receiver Total() alongside a pointer-receiver add(), and since Tokens is embedded in
JSON-serialized rows a mixed receiver set is a genuine trap. It is now a free addUsage function
with value receivers throughout.

One flake to be aware of: pkg/tools/mcp's TestOAuthTransport_DoesNotForwardBearerAcrossOrigins
failed once during a full-suite run with HTTP/1.x transport connection broken: CloseIdleConnections called. It passes in isolation and this PR does not touch MCP or HTTP — noting it because it will
bite someone else on a loaded machine.

@dwin-gharibi
dwin-gharibi requested a review from a team as a code owner August 7, 2026 05:42
@aheritier aheritier added area/cli CLI commands, flags, output formatting kind/feat PR adds a new feature (maps to feat:). Use on PRs only. labels Aug 7, 2026
@aheritier aheritier self-assigned this Aug 7, 2026
@aheritier
aheritier requested a review from docker-agent August 7, 2026 06:09

@docker-agent docker-agent left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: 🟡 NEEDS ATTENTION

Two logic issues found in the new aggregation code in pkg/usage/usage.go:

  1. Per-model call count can be over-counted when an item carries a model name but no usage data — see inline comment at line 141.
  2. CostIncomplete flag is not triggered for sessions whose only token spend is Reasoning tokens, because Total() returns Input + Output and excludes Reasoning — see inline comment at line 158.

Both findings are in newly added + lines. No issues found in cmd/root/usage.go, cmd/root/root.go, or the test files.

Comment thread pkg/usage/usage.go
Comment thread pkg/usage/usage.go
Two review findings, addressed differently because only one is a defect.

Reasoning-only spend was genuinely mis-reported. Total() is input+output by
design, so a run that burned only reasoning tokens reported Total()==0, the
unpriced check never fired, and an understated cost was presented as a
genuine $0.00 — exactly what that check exists to prevent. Adds
Tokens.AnySpend and keys the check on it; Total keeps its display meaning.

Per-model Calls counting a response whose provider reported no usage is not
over-counting: the call happened, the tokens were simply never reported
(usage tracking off, or an older session). Dropping it would hide a model
served entirely by such a provider, which is the worse failure. Kept, but the
short token columns are no longer mysterious: ModelRow.Unmetered records how
many calls came without usage, and the report warns that token counts are
understated — the same treatment unpriced cost already gets.
Pins that an attributed model with no reported usage still counts as a call
and is flagged as unmetered, that a reasoning-only session is flagged as
unpriced despite Total()==0, and that the understated-tokens note names only
the models it applies to.
@dwin-gharibi

Copy link
Copy Markdown
Contributor Author

Done. @aheritier

@aheritier aheritier removed their assignment Aug 7, 2026

@aheritier aheritier left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assessment: 🔴 Request changes — the aggregation misses sub-agent spend, and cost is read from the legacy session field

The shape of this is right: a pure Aggregate over data that is already persisted, no schema change, a thin CLI on top, --json for CI. The gap it closes is real, and the care shows (deterministic tie-breaks, [] rather than null, rune-safe truncation). The unmetered-calls instinct from the last round is also validated by data — in a 145,644-message store only 3 assistant messages carry no usage, so that warning won't be noisy.

But two defects make the reported numbers wrong on the sessions that matter most, and I could only find them by running the command against a real store rather than fixtures.

Verified against PR head 38d3dd12, built and tested on top of current main, and smoke-run against a copy of a real 1.5 GB session store (4,642 sessions, 145,644 messages).


🔴 [blocking] Sub-agent (sub-session) spend is dropped from every breakdown

pkg/usage/usage.go:159-196 walks only s.Messages, and itemModelAndUsage (pkg/usage/usage.go:238-244) handles item.Message and the non-message item fields but never item.SubSession. Sub-sessions are not reachable from the top level either: GetSessions is root-only (pkg/session/store.go:777-779, WHERE parent_id IS NULL OR parent_id = ''). The TUI cost dialog this command mirrors does recurse — pkg/tui/dialog/cost.go:305-308.

One real root session with 46 sub-sessions:

truth (recursive, as the TUI computes) this command
input tokens 219,577 334 (0.2%)
output tokens 327,351 136,234
model calls 471 167
tool calls 492 156

Store-wide: 87,982 of 145,644 messages (60%) live in sub-sessions, holding 63.8M input tokens, 2.63B cached-input tokens and $2,273 of recorded cost — none of it reported. transfer_task shows up in the tool table, so multi-agent sessions are the normal case. "Which agent is expensive?" — the use case in #3943 — gets a wrong answer with no warning.

🔴 [blocking] Cost comes from the legacy session-level field, producing $0.00 plus a false "no pricing" warning

pkg/usage/usage.go:161 uses Cost: s.Cost. pkg/session/session.go:1518-1521 documents that field as existing "only for backward-compatible persistence", with TotalCost() as the canonical figure — and every other consumer uses TotalCost(): pkg/server/session_manager.go:527, pkg/app/export/html.go:119, pkg/tui/dialog/cost.go:347, pkg/tui/components/sidebar/sidebar.go:972.

Combined with the AnySpend() && Cost == 0 heuristic at pkg/usage/usage.go:198-204, here is the actual output of the built binary for session 6047f54d:

6047f54d  ...  334  24.8M  136.2K  $0.0000+  <title>
! Cost is understated: no pricing for anthropic/claude-opus-4-8

That session's TotalCost() is $44.45, and its own messages carry $34.82 of recorded per-message cost — the model is priced. Store-wide the total is understated ~25% ($7,224.17 from the cost column vs ~$9,611 actually recorded). This inverts the principle the PR argues for: rather than refusing to understate, it asserts something false and attributes it to a priced model.

🟠 [should-fix] The package doc states a constraint that isn't true — per-model cost is attributable

pkg/usage/usage.go:22-25 says cost is "persisted per session and per non-message item, never per message". chat.Message.Cost (pkg/chat/chat.go:91-92) is persisted inside message_json and reloads intact (I read $34.82 back out of the store), is summed by session.TotalCost() (pkg/session/session.go:1529), and is already attributed per model by pkg/tui/dialog/cost.go:303. The msg.Cost = 0 at pkg/session/session.go:1978 only clears a request-assembly copy. So the doc encodes a false data-model claim, and it is the justification for the finding above and for an omitted feature: fix the cost source and per-model cost falls out for free.

🟠 [should-fix] No test touches the sub-session path

25 tests, none constructing session.NewSubSessionItem, and nothing exercising loadUsageSessions or a real store. That gap is why both blocking findings shipped; two tests close it.

🟠 [should-fix] The whole store is loaded into memory before --since is applied

cmd/root/usage.go:104-108 calls GetSessions (every session, every item) and filters afterwards. Measured: --since 24h on a 1.5 GB store took 11.7 s and 2.70 GB peak RSS to print 3 sessions — enough to OOM a 2 GB CI runner, which is the advertised --json-in-CI use case. GetSessionSummaries (pkg/session/store.go:812-815) exists for exactly this reason; filter on metadata, then load items for the survivors.

🟠 [should-fix] --session rejects the ID the command prints

The table prints 8 chars via shortSessionID (cmd/root/usage.go:245-250), but --session 6047f54d fails with session not found. Accepting a unique prefix would make the output self-consistent.

🟠 [should-fix] Docs not updated

docs/features/cli/index.md documents every user-facing command (### docker agent eval at line 450, models at 192, toolsets at 216, plus share and serve *). A new top-level command belongs there in the same change.

🔵 [optional] A read-only report opens the store read-write and runs migrations

sqlitestore.New (pkg/session/sqlitestore/sqlitestore.go:24-60) runs migrations and, on failure, moves the database aside to .bak; it also takes a write lock, so usage can contend with a running agent. A read-only open would be proportionate for a reporting command.

🔵 [optional] Fourth divergent aggregation site

pkg/tui/dialog/cost.go:296-320 already walks a session for these exact numbers. Extracting that walker into pkg/usage and having the TUI consume it would stop the two from disagreeing — they currently disagree in both ways described above.


Worth doing: yes. The gap is genuine (figures reachable only from the TUI and OTel), the approach is proportionate, and no schema change is needed. It just needs the aggregation to see sub-sessions and to take cost from TotalCost(); do that and the per-model cost breakdown you documented as impossible becomes available too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cli CLI commands, flags, output formatting kind/feat PR adds a new feature (maps to feat:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No way to see what an agent spent outside the TUI

3 participants