Skip to content

feat(llmhubs): support tool calling and structured output in the Gemi…#45

Open
xinkuleee wants to merge 1 commit into
microsoft:mainfrom
xinkuleee:feat/gemini-tool-calling
Open

feat(llmhubs): support tool calling and structured output in the Gemi…#45
xinkuleee wants to merge 1 commit into
microsoft:mainfrom
xinkuleee:feat/gemini-tool-calling

Conversation

@xinkuleee

Copy link
Copy Markdown

Summary

Adds Google Gemini function/tool calling and JSON-schema structured output to the
LLMHub Gemini adapter, so Gemini models can drive the chat agent (tools) and
structured routing (intent check).

  • Tool calling: map internal (OpenAI-shaped) tools / function_call /
    function_result to Gemini functionDeclarations / functionCall /
    functionResponse (role mapping assistant→model / tool→user, and
    call_id↔function-name correlation since Gemini responses carry no call id);
    parse Gemini functionCall back into function_call outputs.
  • Schema sanitizer: rewrite tool/response JSON Schema into Gemini's
    OpenAPI-subset — inline $ref/$defs, drop additionalProperties/$schema,
    flatten single-element allOf, and convert Optional anyOf + {"type":"null"}
    to nullable.
  • Structured output: map response_format.type=json_schema to Gemini
    responseMimeType + responseSchema (fixes chat intent-check routing on Gemini).
  • Capability flags: advertise supports_tools / supports_structured_output
    for the Gemini provider (core io_profile derivation + backend equivalent).
  • Tests: unit coverage for request building, input/response mapping, the
    sanitizer (refs/defs/additionalProperties/allOf/anyOf-null/recursion), and
    structured output.

Validation

  • make precommit-run
  • cd backend && go test ./...
  • cd core && uv run pytest
  • Frontend build/lint, if working from a frontend source checkout
  • Not run; explain below

Frontend is untouched, so its build/lint does not apply.

Checklist

  • Linked relevant issues or explained why there is no issue.
  • Updated docs and examples where behavior changed.
  • Updated CHANGELOG.md under ## [Unreleased] for user-facing changes.
  • Regenerated and committed generated files after proto, Wire, or OpenAPI changes.
  • Confirmed no secrets, tokens, credentials, or sensitive logs are included.

Notes for reviewers

  • Streaming is not implemented for Gemini yet — it falls back to the base
    non-streaming path (generate_stream yields a single chunk). Tool calling works,
    but without incremental token streaming; a follow-up can add
    streamGenerateContent?alt=sse.
  • The schema sanitizer targets the keywords Gemini currently rejects; verified
    against the full builtin tool set and the structured intent-check schema.
  • No secrets/config included: the local Gemini model YAML (with API key) and the
    local default-model switch are gitignored and intentionally excluded from this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant