Skip to content

feat: catch invalid model params before the provider does - #169

Closed
brunobuddy wants to merge 1 commit into
mainfrom
long-tail-api
Closed

feat: catch invalid model params before the provider does#169
brunobuddy wants to merge 1 commit into
mainfrom
long-tail-api

Conversation

@brunobuddy

Copy link
Copy Markdown
Member

💭 Why

The catalog has always recorded which parameter combinations providers reject. Nothing acted on it. parseParams even said so in a comment: cross-parameter rules were "not yet enforced".

So the data that makes this catalog different from a model-metadata list was display-only.

✨ What changed

  • POST /api/v1/validate returns the problems plus a corrected safeParams payload.
  • New modelparams-mcp package: 4 read-only MCP tools, stdio, catalog bundled in.
  • New llm-model-parameters agent skill (skills.sh format).
  • modelparams gains checkApplicability, isApplicable, dropUnsupported, resolveModelId.
  • parseParams now enforces cross-parameter rules.
  • Both packages release in lockstep; modelparams-mcp pins modelparams exactly.
  • CONTRIBUTING.md: new directories, and that applicability rules are now enforced.

👤 For users

Three issue codes come back: unknown_parameter, invalid_value, not_applicable (with conflictsWith). safeParams is always a payload that passes validation, so a caller can spread it and move on.

Catches the cases people currently hit as provider 400s: top_p with a non-default temperature on Anthropic, sampling knobs on reasoning models, a thinking budget without thinking enabled.

🔧 For operators

modelparams-mcp needs its own npm trusted publisher before the first release, pointing at release-modelparams.yml (org mnfst, repo modelparams.dev). Without it that publish step fails. Renaming the workflow file breaks trusted publishing for both packages.

Submitting the server to the MCP registry is still manual.

📝 Notes

  • No backend. The 239-model catalog compiles into the function from the committed generated data.
  • Verified with a local vercel build: the function builds and runs, and static /api/v1/*.json is unaffected. Header rules layer, so the no-store rule for /api/v1/validate has to stay after the general /api/(.*) s-maxage rule or verdicts get cached across callers.
  • 245 tests (165 site, 62 package, 18 MCP), including a stdio smoke test in CI.
  • Applicability evaluation falls back to a parameter's catalog default when the request omits it, since that is what the provider applies instead.

The catalog has always recorded which parameter combinations providers
reject, but nothing acted on it. parseParams even said so in a comment.

Adds three ways to consume it:

- POST /api/v1/validate returns the problems plus a corrected payload.
  Stateless Vercel Function with the catalog compiled in, no backend.
- modelparams-mcp, an MCP server with four read-only tools over the
  catalog, so an agent can look parameters up instead of guessing.
- An llm-model-parameters agent skill in skills.sh format.

Underneath, the modelparams package gains the engine all three share:
checkApplicability, isApplicable, dropUnsupported and resolveModelId.
parseParams now enforces cross-parameter rules. When a rule references a
parameter the request omitted, evaluation falls back to that parameter's
catalog default, since that is what the provider applies instead.

Both packages ship in lockstep and modelparams-mcp pins modelparams
exactly, so a given server version carries one known catalog. That also
avoids a caret trap: for 0.0.x, ^0.0.1 resolves to >=0.0.1 <0.0.2, which
would have frozen the server on the first catalog forever.
@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
modelparams.dev Ready Ready Preview Jul 30, 2026 8:00am

Request Review

@github-actions github-actions Bot added site Website code or tooling meta Repo docs, CI, and config labels Jul 30, 2026
@brunobuddy brunobuddy closed this Jul 30, 2026
@brunobuddy brunobuddy reopened this Jul 30, 2026
@brunobuddy

Copy link
Copy Markdown
Member Author

Superseded by a stack of four, one per feature, so each can be reviewed on its own:

Same work, plus two fixes the split surfaced: resolveModelId was returning empty suggestion lists because it matched a bare query against the full provider/model id, and the lockfile here disagreed with the MCP package's own package.json.

@brunobuddy brunobuddy closed this Jul 30, 2026
@brunobuddy
brunobuddy deleted the long-tail-api branch July 30, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

meta Repo docs, CI, and config site Website code or tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant