Ingestion Analytics API and Eval Hub Integration (issue 29 of 29)

[gabemontero]

Labels: ready-to-code
Depends on: Issue 5 (#4043 health data model), Issue 21 (#4059 audit logging)
RHIDP Stories: RHIDP-15280 (analytics scope)
Feature: RHDHPLAN-1508 — Epic RHIDP-15277 (consolidated from RHDHPLAN-1513 RHIDP-15333)

Implement analytics REST API endpoints (sync history, quality scores, match coverage) with RBAC gating and pagination, quality score storage (DB table + store), match coverage calculator, Neo4j sync status embedding, Eval Hub client for external quality score ingestion, and multi-eval backend support (LightEval, IBM Clear, GuideLLM).

Tasks

From openspec/changes/ingestion-audit-metrics/tasks.md group 5 (RHIDP-15344):

• 5.1 Define analytics API routes
• 5.2 Add GET /api/boost/admin/analytics/sync-history endpoint
• 5.3 Add GET /api/boost/admin/analytics/quality-scores endpoint
• 5.4 Add GET /api/boost/admin/analytics/match-coverage endpoint
• 5.5 Add RBAC gating to all endpoints
• 5.6–5.10 Pagination, date range filtering, connector filtering, eval source filtering, route registration
• 5.11 Add integration tests

From openspec/changes/ingestion-audit-metrics/tasks.md group 6:

• 6.1–6.7 Sync history service: AnalyticsService, getSyncHistory(), pagination, date range filtering, connector filtering, duration calculation, unit tests

From openspec/changes/ingestion-audit-metrics/tasks.md group 7:

• 7.1–7.8 Quality score storage: DB table schema, migration, QualityScoresStore, insertScore(), getScoresBySkill(), getLatestScores(), indexes, unit tests

From openspec/changes/ingestion-audit-metrics/tasks.md group 8:

• 8.1–8.6 Quality score analytics: getQualityScores(), per-skill scores, aggregate distribution, normalization, eval source filtering, unit tests

From openspec/changes/ingestion-audit-metrics/tasks.md group 9:

• 9.1–9.8 Match coverage calculator: agent capability extraction, skill ID extraction, matching logic, coverage ratio, unmatched capabilities list, caching, unit tests

From openspec/changes/ingestion-audit-metrics/tasks.md group 10:

• 10.1–10.6 Neo4j sync status: connection check, last sync timestamp, entity count, embed in analytics responses, graceful unavailable handling, unit tests

From openspec/changes/ingestion-audit-metrics/tasks.md group 11:

• 11.1–11.8 Eval Hub client: interface, fetchQualityScores(), app-config schema, client init, HTTP client, error handling, score normalization, unit tests

From openspec/changes/ingestion-audit-metrics/tasks.md group 12:

• 12.1–12.8 Eval Hub ingestion service: background task, refresh interval config, score fetching/storage, graceful unavailable handling, manual trigger endpoint, status logging, integration tests

From openspec/changes/ingestion-audit-metrics/tasks.md group 13:

• 13.1–13.6 Multi-eval backend support: eval_source field, source normalization, source filtering, per-source aggregation, LightEval/IBM Clear/GuideLLM support, unit tests

Specifications

• openspec/changes/ingestion-audit-metrics/specs/analytics-api/spec.md
• openspec/changes/ingestion-audit-metrics/specs/eval-hub-integration/spec.md

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 8:05 PM UTC · Completed 8:08 PM UTC
Commit: e03b281 · View workflow run →

[fullsend-ai-triage]

Triage: Blocked on Prerequisites

This issue is well-specified (detailed task breakdown across 9 task groups with spec references) but explicitly depends on two open issues that must land first:

• Ingestion Health — API, Data Model, Error Classification (issue 5 of 29) #4043 — Ingestion Health — API, Data Model, Error Classification (issue 5 of 29) — provides the health data model that analytics endpoints (sync history, quality scores) build upon.
• RBAC Audit Logging (issue 21 of 29) #4059 — RBAC Audit Logging (issue 21 of 29) — provides the RBAC gating infrastructure required by all analytics API endpoints (tasks 5.5).

Both dependencies are still open with no PRs in progress. This issue should be unblocked once those prerequisites are resolved.

Scope summary: Implements analytics REST API endpoints (sync history, quality scores, match coverage) with RBAC gating and pagination, quality score DB storage, match coverage calculator, Neo4j sync status embedding, Eval Hub client for external score ingestion, and multi-eval backend support (LightEval, IBM Clear, GuideLLM). Covers task groups 5–13 from the ingestion-audit-metrics task plan.

Blocked by:

• Ingestion Health — API, Data Model, Error Classification (issue 5 of 29) #4043
• RBAC Audit Logging (issue 21 of 29) #4059