Plan in-app AI chat and Fix It assistant

[ejsmith]

Goal

Add an in-app AI assistant for Exceptionless that lets users ask project questions in plain English and powers a Fix It / Explain button on events and stacks.

The in-app assistant should reuse the same core capabilities as the hosted MCP server, but it should be implemented as a first-party backend feature rather than calling MCP over HTTP from the app.

Product rules

• AI is available only on selected paid plans.
• Usage is token-based because provider billing is token-based.
• Start with included monthly token budgets only.
• Do not support overage billing in v1.
• Track provider cost internally from day one.
• Start read-only; any write actions must require explicit user confirmation.

Initial plan fields:

• ai_enabled
• ai_monthly_token_limit
• ai_max_tokens_per_request
• ai_max_tool_calls_per_request
• ai_max_context_bytes
• ai_allowed_models
• ai_overage_enabled = false

Phase 1: Shared AI tool layer

Extract reusable backend tool logic so MCP and in-app AI use the same permission checks, validation, result shaping, and truncation behavior.

Suggested services:

• AiToolCatalog
• AiToolExecutor
• AiContextBuilder
• AiUsageTracker
• AiPlanLimitService

The in-app chat should call shared internal services directly. It should not call /mcp over HTTP.

Phase 2: Usage metering and limits

Add persistent usage tracking per organization.

Track:

• organization id
• user id
• operation: chat, fix_it, explain_event, explain_stack
• model
• input tokens
• output tokens
• total tokens
• estimated provider cost
• created date / month bucket

Enforce before and during requests:

• plan allows AI
• monthly token budget remaining
• per-request max token cap
• max tool calls
• max context size

Use a reservation or capped request budget so a single request cannot materially overspend the remaining monthly allowance.

Phase 3: Backend chat API

Add API endpoints such as:

• POST /api/v2/ai/conversations
• GET /api/v2/ai/conversations
• GET /api/v2/ai/conversations/{id}
• POST /api/v2/ai/conversations/{id}/messages
• optional streaming endpoint for assistant responses

The backend owns:

• permission checks
• plan checks
• usage reservation
• model calls
• tool calls
• persisted conversation history

Phase 4: Chat UI

Add a project-scoped AI chat surface in the app.

Potential entry points:

• Account AI Tools page
• project settings
• client setup page
• profile menu
• later: sidebar/global command menu

UI should show:

• current project / organization context
• token usage this month
• remaining token budget
• limit reached state
• conversation history

Phase 5: Fix It / Explain buttons

Add Fix It / Explain buttons on stack and event detail pages.

Clicking starts a conversation with preloaded context:

• project
• stack/event id
• stack trace
• request info
• environment
• recent occurrences
• version/release info if available
• related stacks/events if useful

Assistant response should be structured:

• likely cause
• evidence
• suggested fix
• assumptions
• next debugging steps
• related Exceptionless links

Phase 6: Confirmed write actions later

After read-only chat is useful, add confirmed write actions:

• mark fixed in version
• snooze
• mark future events critical
• add reference link

The assistant may propose an action, but the UI should execute it only after explicit user confirmation. All write actions should be audited.

Phase 7: Overage billing later

After real usage data exists, add pay-as-you-go.

Add:

• ai_overage_enabled
• ai_monthly_spend_limit_cents
• ai_overage_price_per_million_tokens
• internal provider cost tracking
• markup calculation

User options:

• stop at included token limit
• allow overage up to a monthly cap

Recommended first implementation slice

Start with foundation work only:

1. Add plan AI limit fields.
2. Add AI usage tracking.
3. Add backend limit enforcement service.
4. Add provider abstraction, with model calls behind config/feature flag.
5. Keep the first slice read-only.
6. Add minimal admin/system visibility for usage and limits.

A later PR can add the first user-facing read-only chat and Fix It experience.