otel: adopt gen_ai.* semantic conventions on tool + agent spans - #422
Conversation
Forge emitted a partial GenAI attribute set and published tool telemetry under proprietary forge.tool.* keys that GenAI-native backends don't recognize. This adopts the OTel GenAI semantic conventions. Tool spans (tool.<name>) now emit, replacing forge.tool.* (which never shipped to production, so no dual-emit window): - gen_ai.operation.name=execute_tool, gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.type (function | extension for MCP-backed tools) - gen_ai.tool.call.arguments / .result / .description (opt-in, CaptureContent) - error.type on failure (replaces forge.tool.error) - MCP tools (namespaced "<server>__<tool>") additionally carry mcp.method.name=tools/call. mcp.session.id / mcp.protocol.version need the MCP manager plumbed to the executor and are deferred. agent.execute now emits gen_ai.provider.name, gen_ai.agent.id/.name/.version (from forge.yaml), gen_ai.conversation.id (Forge session / A2A task id), and gen_ai.tool.definitions (opt-in). llm.completion now emits gen_ai.operation.name=chat, gen_ai.provider.name, gen_ai.response.id, and gen_ai.response.model (wired via a new ChatResponse.Model populated from the Anthropic/OpenAI response bodies; falls back to the request model). gen_ai.system is kept one release as a deprecated alias of gen_ai.provider.name. attrs.go constants updated in one sweep. Tests assert the new keys on tool/agent/llm spans (incl. the MCP extension path); docs/core-concepts/observability-tracing.md updated. Refs #421
7015862 to
ee89ad9
Compare
initializ-mk
left a comment
There was a problem hiding this comment.
Approve-grade — clean, correct semconv mapping with comprehensive tests. Traced every claim to the branch source.
Verified correct
- Tool spans —
gen_ai.operation.name=execute_tool,.tool.name,.tool.call.id(fromtc.ID),.tool.type; MCP tools additionally carrymcp.method.name=tools/calland function tools omit it. Both branches (incl. the absence assertion) are pinned byTestExecuteStampsGenAIToolAttrs/TestExecuteMCPToolSpanUsesExtensionType. error.type="tool_execution_error"replacingforge.tool.erroris a semconv improvement:error.typeis meant to be a low-cardinality classification, and the full message is still preserved viaRecordError+SetStatus(codes.Error, …)— nothing lost.llm.completion—operation.name=chat, dualgen_ai.system+gen_ai.provider.name,response.id, andresponse.modelwired through the newChatResponse.Model. Confirmed bothparseAnthropicResponseandparseOpenAIResponsepopulate it from the response body's"model", with a safe fallback to the request model when a provider does not echo one.agent.execute—provider.name,agent.id/.name/.version(fromforge.yamlvia the new runner wiring), andconversation.id=task.ID— correct: the persistent session/thread key, not a synthesized id.- Content gating intact —
tool.call.arguments/.result/.description+tool.definitionsall behindCaptureContent, through the samePrepareSpanContentredact+truncate pipeline, and absent (not empty) by default. The redaction tests were updated to the new keys and still assert the AWS-key scrub. forge.tool.nameretained correctly — not orphaned; live consumer atguardrails_tracing.go:56(theguardrail.<gate>span cross-references the evaluated tool).- Single span path —
ExecuteStreamdelegates toExecute, so streaming and non-streaming share the one emission path; no divergent path silently missing the new attrs.
Minor, non-blocking notes
error.typeis a single constant for all tool failures — fine per semconv, but a natural future refinement is distinguishing timeout / auth / not-found.gen_ai.tool.definitionsrides the 4 KiB content cap, which can truncate a large catalog to invalid JSON — acceptable for telemetry and documented as opt-in / potentially large.
All 10 CI checks green. Nicely scoped, docs updated in lockstep. Inline note on the MCP-type heuristic below.
| // tool type (an agent-side bridge to an external system); all | ||
| // others are "function". | ||
| toolType := observability.ToolTypeFunction | ||
| isMCPTool := strings.Contains(tc.Function.Name, "__") |
There was a problem hiding this comment.
The strings.Contains(name, "__") heuristic for typing a tool as extension is consistent with how the rest of the runtime identifies namespaced ops (e.g. pdpGoverns uses the same __ test), so I am comfortable with it. The one edge worth a mental footnote: a builtin or skill tool whose name happened to contain __ would be mistyped as extension and pick up a spurious mcp.method.name=tools/call. In practice the <server>__<tool> namespacing convention makes that a non-issue, but if the executor ever gains a first-class "is this an MCP-backed tool?" signal (the same plumbing the deferred mcp.session.id / mcp.protocol.version need), routing this off that signal instead of the name would make the tool-type classification exact. Non-blocking.
There was a problem hiding this comment.
Good catch — addressed in 6a1a949. Rather than route off the deferred #423 plumbing, I added Registry.IsMCPTool(name) (type-asserts the MCPSource marker) to the ToolExecutor interface and now classify off that. This also fixes a real edge you implied: NamespacedSource API per-op tools (<api>__<op>) share the __ shape but aren't MCP — they'd have been mistyped as extension with a spurious mcp.method.name. New test TestExecuteNamespacedNonMCPToolNotTypedAsExtension pins that. gen_ai.tool.type is now exact.
Review feedback on #422: classifying a tool as `extension` via strings.Contains(name, "__") mistypes NON-MCP namespaced tools — NamespacedSource API per-op tools ("<api>__<op>") share the "__" shape but are not MCP, so they would get gen_ai.tool.type=extension and a spurious mcp.method.name=tools/call. Add Registry.IsMCPTool(name), which type-asserts the registered tool to the MCPSource marker (the authoritative signal), and expose it on the runtime.ToolExecutor interface. The loop now types tools off that instead of the name shape. Adds TestExecuteNamespacedNonMCPToolNotTypedAsExtension asserting a "<api>__<op>" tool that is not MCP is typed "function" with no mcp.method.name; existing MCP/function span tests updated to drive the marker explicitly. Refs #422
initializ-mk
left a comment
There was a problem hiding this comment.
Fix verified — complete, correct, and it closed a genuine latent bug rather than just the hypothetical I raised. ✅
Commit 6a1a949 replaces the strings.Contains(name, "__") heuristic with the registry's authoritative MCPSource marker. Tracing it against source confirmed the edge was real, not theoretical: the registration guard (registry.go:35-42) requires any __-named tool to implement either MCPSource or NamespacedSource, and NamespacedSource is a live sibling (per-op API tools materialized from OpenAPI, <api>__<op>). The old heuristic would have typed those as extension and stamped a spurious mcp.method.name=tools/call on every OpenAPI-tool deployment.
Why the fix is airtight
Registry.IsMCPTooltype-assertst.(MCPSource)— RLock-guarded, false for unknown/non-MCP. The two markers have distinct method names (MCPSource()vsNamespacedSource(),tool.go:46/56), so aNamespacedSourcetool cannot satisfyMCPSource; classification is now exact. TheNamespacedSourcedoc comment states the intent verbatim — "without falsely claiming to be an MCP tool."MCPToolimplementsMCPSource()(mcp_tool.go:160), so real MCP tools still typeextension.- Call site is nil-guarded (
e.tools != nil && e.tools.IsMCPTool(...)) and drives bothgen_ai.tool.typeand themcp.method.namegate off the single signal. ToolExecutor.IsMCPTooladded to the interface;Registry+ all three test mocks updated.
Test coverage tightened
- New
TestExecuteNamespacedNonMCPToolNotTypedAsExtensionpins the exact edge — a__-named tool with the MCP marker off isfunctionwith nomcp.method.name. genAIToolRunnow threads an explicitisMCPflag, so the function/extension paths are no longer entangled with the name shape.
All 10 CI checks green on the new head. No new issues, nothing else regressed. LGTM — good to merge.
Implements the committed scope of #421 — adopt the OTel GenAI semantic conventions on Forge's spans, with tool attributes first.
What changed
Tool spans (
tool.<name>) — replace the proprietaryforge.tool.*keys (never shipped, so a straight swap, no dual-emit):gen_ai.operation.name=execute_tool,gen_ai.tool.name,gen_ai.tool.call.id,gen_ai.tool.type(function|extension)CaptureContent):gen_ai.tool.call.arguments,gen_ai.tool.call.result,gen_ai.tool.descriptionerror.typeon failure (replacesforge.tool.error)<server>__<tool>) →gen_ai.tool.type=extension+mcp.method.name=tools/callagent.execute—gen_ai.provider.name,gen_ai.agent.id/.name/.version(fromforge.yaml),gen_ai.conversation.id(Forge session / A2A task id), and opt-ingen_ai.tool.definitions.llm.completion—gen_ai.operation.name=chat,gen_ai.provider.name,gen_ai.response.id,gen_ai.response.model(newChatResponse.Model, populated from the Anthropic/OpenAI response bodies; falls back to the request model).gen_ai.systemis retained one release as a deprecated alias ofgen_ai.provider.name.attrs.goconstants updated in one sweep;forge.tool.namekept only for the guardrail spans that reference it.Deferred (noted in-code + on #421)
mcp.session.id/mcp.protocol.version— need the MCP manager plumbed to the executor.system_instructions, memory/retrieval/embeddings).Verification
gofmtclean;forge-core+forge-clibuild.extensionpath and thatmcp.method.nameis absent on function tools. Updated the content-capture tests to the new keys.go test ./runtime/ ./llm/providers/ ./observability/(core) and./runtime/(cli) all pass.Refs #421