Skip to content

otel: adopt gen_ai.* semantic conventions on tool + agent spans - #422

Merged
initializ-mk merged 2 commits into
mainfrom
feat/otel-genai-tool-attrs
Aug 26, 2026
Merged

otel: adopt gen_ai.* semantic conventions on tool + agent spans#422
initializ-mk merged 2 commits into
mainfrom
feat/otel-genai-tool-attrs

Conversation

@initializ-mk

@initializ-mk initializ-mk commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Implements the committed scope of #421 — adopt the OTel GenAI semantic conventions on Forge's spans, with tool attributes first.

What changed

Tool spans (tool.<name>) — replace the proprietary forge.tool.* keys (never shipped, so a straight swap, no dual-emit):

  • gen_ai.operation.name=execute_tool, gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.type (function | extension)
  • opt-in (CaptureContent): gen_ai.tool.call.arguments, gen_ai.tool.call.result, gen_ai.tool.description
  • error.type on failure (replaces forge.tool.error)
  • MCP-backed tools (<server>__<tool>) → gen_ai.tool.type=extension + mcp.method.name=tools/call

agent.executegen_ai.provider.name, gen_ai.agent.id/.name/.version (from forge.yaml), gen_ai.conversation.id (Forge session / A2A task id), and opt-in gen_ai.tool.definitions.

llm.completiongen_ai.operation.name=chat, gen_ai.provider.name, gen_ai.response.id, gen_ai.response.model (new ChatResponse.Model, populated from the Anthropic/OpenAI response bodies; falls back to the request model).

gen_ai.system is retained one release as a deprecated alias of gen_ai.provider.name. attrs.go constants updated in one sweep; forge.tool.name kept only for the guardrail spans that reference it.

Deferred (noted in-code + on #421)

  • mcp.session.id / mcp.protocol.version — need the MCP manager plumbed to the executor.
  • Secondary GenAI gaps (usage cache/reasoning tokens, request params, system_instructions, memory/retrieval/embeddings).

Verification

  • gofmt clean; forge-core + forge-cli build.
  • New tests assert the semconv keys on tool / agent / llm spans, incl. the MCP extension path and that mcp.method.name is absent on function tools. Updated the content-capture tests to the new keys.
  • go test ./runtime/ ./llm/providers/ ./observability/ (core) and ./runtime/ (cli) all pass.

Refs #421

Forge emitted a partial GenAI attribute set and published tool telemetry
under proprietary forge.tool.* keys that GenAI-native backends don't
recognize. This adopts the OTel GenAI semantic conventions.

Tool spans (tool.<name>) now emit, replacing forge.tool.* (which never
shipped to production, so no dual-emit window):
- gen_ai.operation.name=execute_tool, gen_ai.tool.name, gen_ai.tool.call.id,
  gen_ai.tool.type (function | extension for MCP-backed tools)
- gen_ai.tool.call.arguments / .result / .description (opt-in, CaptureContent)
- error.type on failure (replaces forge.tool.error)
- MCP tools (namespaced "<server>__<tool>") additionally carry
  mcp.method.name=tools/call. mcp.session.id / mcp.protocol.version need
  the MCP manager plumbed to the executor and are deferred.

agent.execute now emits gen_ai.provider.name, gen_ai.agent.id/.name/.version
(from forge.yaml), gen_ai.conversation.id (Forge session / A2A task id), and
gen_ai.tool.definitions (opt-in).

llm.completion now emits gen_ai.operation.name=chat, gen_ai.provider.name,
gen_ai.response.id, and gen_ai.response.model (wired via a new
ChatResponse.Model populated from the Anthropic/OpenAI response bodies;
falls back to the request model).

gen_ai.system is kept one release as a deprecated alias of
gen_ai.provider.name. attrs.go constants updated in one sweep.

Tests assert the new keys on tool/agent/llm spans (incl. the MCP
extension path); docs/core-concepts/observability-tracing.md updated.

Refs #421
@initializ-mk
initializ-mk force-pushed the feat/otel-genai-tool-attrs branch from 7015862 to ee89ad9 Compare August 26, 2026 18:14

@initializ-mk initializ-mk left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve-grade — clean, correct semconv mapping with comprehensive tests. Traced every claim to the branch source.

Verified correct

  • Tool spansgen_ai.operation.name=execute_tool, .tool.name, .tool.call.id (from tc.ID), .tool.type; MCP tools additionally carry mcp.method.name=tools/call and function tools omit it. Both branches (incl. the absence assertion) are pinned by TestExecuteStampsGenAIToolAttrs / TestExecuteMCPToolSpanUsesExtensionType.
  • error.type="tool_execution_error" replacing forge.tool.error is a semconv improvement: error.type is meant to be a low-cardinality classification, and the full message is still preserved via RecordError + SetStatus(codes.Error, …) — nothing lost.
  • llm.completionoperation.name=chat, dual gen_ai.system + gen_ai.provider.name, response.id, and response.model wired through the new ChatResponse.Model. Confirmed both parseAnthropicResponse and parseOpenAIResponse populate it from the response body's "model", with a safe fallback to the request model when a provider does not echo one.
  • agent.executeprovider.name, agent.id/.name/.version (from forge.yaml via the new runner wiring), and conversation.id=task.ID — correct: the persistent session/thread key, not a synthesized id.
  • Content gating intacttool.call.arguments/.result/.description + tool.definitions all behind CaptureContent, through the same PrepareSpanContent redact+truncate pipeline, and absent (not empty) by default. The redaction tests were updated to the new keys and still assert the AWS-key scrub.
  • forge.tool.name retained correctly — not orphaned; live consumer at guardrails_tracing.go:56 (the guardrail.<gate> span cross-references the evaluated tool).
  • Single span pathExecuteStream delegates to Execute, so streaming and non-streaming share the one emission path; no divergent path silently missing the new attrs.

Minor, non-blocking notes

  1. error.type is a single constant for all tool failures — fine per semconv, but a natural future refinement is distinguishing timeout / auth / not-found.
  2. gen_ai.tool.definitions rides the 4 KiB content cap, which can truncate a large catalog to invalid JSON — acceptable for telemetry and documented as opt-in / potentially large.

All 10 CI checks green. Nicely scoped, docs updated in lockstep. Inline note on the MCP-type heuristic below.

Comment thread forge-core/runtime/loop.go Outdated
// tool type (an agent-side bridge to an external system); all
// others are "function".
toolType := observability.ToolTypeFunction
isMCPTool := strings.Contains(tc.Function.Name, "__")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The strings.Contains(name, "__") heuristic for typing a tool as extension is consistent with how the rest of the runtime identifies namespaced ops (e.g. pdpGoverns uses the same __ test), so I am comfortable with it. The one edge worth a mental footnote: a builtin or skill tool whose name happened to contain __ would be mistyped as extension and pick up a spurious mcp.method.name=tools/call. In practice the <server>__<tool> namespacing convention makes that a non-issue, but if the executor ever gains a first-class "is this an MCP-backed tool?" signal (the same plumbing the deferred mcp.session.id / mcp.protocol.version need), routing this off that signal instead of the name would make the tool-type classification exact. Non-blocking.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — addressed in 6a1a949. Rather than route off the deferred #423 plumbing, I added Registry.IsMCPTool(name) (type-asserts the MCPSource marker) to the ToolExecutor interface and now classify off that. This also fixes a real edge you implied: NamespacedSource API per-op tools (<api>__<op>) share the __ shape but aren't MCP — they'd have been mistyped as extension with a spurious mcp.method.name. New test TestExecuteNamespacedNonMCPToolNotTypedAsExtension pins that. gen_ai.tool.type is now exact.

Review feedback on #422: classifying a tool as `extension` via
strings.Contains(name, "__") mistypes NON-MCP namespaced tools —
NamespacedSource API per-op tools ("<api>__<op>") share the "__" shape
but are not MCP, so they would get gen_ai.tool.type=extension and a
spurious mcp.method.name=tools/call.

Add Registry.IsMCPTool(name), which type-asserts the registered tool to
the MCPSource marker (the authoritative signal), and expose it on the
runtime.ToolExecutor interface. The loop now types tools off that instead
of the name shape.

Adds TestExecuteNamespacedNonMCPToolNotTypedAsExtension asserting a
"<api>__<op>" tool that is not MCP is typed "function" with no
mcp.method.name; existing MCP/function span tests updated to drive the
marker explicitly.

Refs #422

@initializ-mk initializ-mk left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix verified — complete, correct, and it closed a genuine latent bug rather than just the hypothetical I raised.

Commit 6a1a949 replaces the strings.Contains(name, "__") heuristic with the registry's authoritative MCPSource marker. Tracing it against source confirmed the edge was real, not theoretical: the registration guard (registry.go:35-42) requires any __-named tool to implement either MCPSource or NamespacedSource, and NamespacedSource is a live sibling (per-op API tools materialized from OpenAPI, <api>__<op>). The old heuristic would have typed those as extension and stamped a spurious mcp.method.name=tools/call on every OpenAPI-tool deployment.

Why the fix is airtight

  • Registry.IsMCPTool type-asserts t.(MCPSource) — RLock-guarded, false for unknown/non-MCP. The two markers have distinct method names (MCPSource() vs NamespacedSource(), tool.go:46/56), so a NamespacedSource tool cannot satisfy MCPSource; classification is now exact. The NamespacedSource doc comment states the intent verbatim — "without falsely claiming to be an MCP tool."
  • MCPTool implements MCPSource() (mcp_tool.go:160), so real MCP tools still type extension.
  • Call site is nil-guarded (e.tools != nil && e.tools.IsMCPTool(...)) and drives both gen_ai.tool.type and the mcp.method.name gate off the single signal.
  • ToolExecutor.IsMCPTool added to the interface; Registry + all three test mocks updated.

Test coverage tightened

  • New TestExecuteNamespacedNonMCPToolNotTypedAsExtension pins the exact edge — a __-named tool with the MCP marker off is function with no mcp.method.name.
  • genAIToolRun now threads an explicit isMCP flag, so the function/extension paths are no longer entangled with the name shape.

All 10 CI checks green on the new head. No new issues, nothing else regressed. LGTM — good to merge.

@initializ-mk
initializ-mk merged commit 854e042 into main Aug 26, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request epic:eval Cross-repo observability + evaluation epic

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant