feat(eval): ondemand simulate — replay a dataset, evaluate synchronously

[jariy17]

What

eval ondemand simulate — replay a dataset against a runtime, then evaluate the sessions synchronously, client-side (scores print inline). The on-demand twin of batch-evaluation simulate.

Pipeline: invokeDataset (replay) → getTracesForAgent (CloudWatch) → evaluate (Evaluate API).

Follows the batch-evaluation simulate pattern

• Same invoke flags + --ingestion-wait-ms (default 180000; 0 to skip)
• Same SIGINT abort + invokeDataset composition
• Refuses when 0 invoked, naming the first failure (failures[0])
• Output adds sessions[] (exampleId ↔ sessionId) + failures[]

Differs by design: no --name / --kms-key-arn (no async job); output is inline scores, not a job id.

Output

{
  "sessionsEvaluated": 1,
  "results": [ { "evaluatorId": "Builtin.Helpfulness", "value": 0.83, "label": "Very Helpful", "explanation": "…", "tokenUsage": {…} } ],
  "examplesInvoked": 1,
  "examplesFailed": 0,
  "sessions": [ { "exampleId": "greet", "sessionId": "…" } ],
  "failures": []
}

Tests (batch pattern)

• Edge tests — ondemand.test.tsx (TestCoreClient): required-flag validation, refuse-when-nothing-invoked, --ingestion-wait-ms passthrough.
• Fixture golden — ondemand.fixture.test.tsx: real invoke → CloudWatch traces → Evaluate, recorded against a live agent, replayed offline via matchGolden. Deterministic via the injected newSessionId seam plus a now() clock seam — on-demand's trace-query window is otherwise Date.now()-based, so its StartQuery fixture key would drift between record and replay (batch has no client query; evaluate pins an explicit --start-time/--end-time).
• Live-validated across 10 dataset scenarios against a real agent in the EXPLORE account (all real Builtin.Helpfulness scores; edge/negative paths handled).

Notes

• Builtin.Helpfulness ignores ground-truth refs (ignoredReferenceInputFields); the handler still forwards them (adapter exercised).
• Rebased onto current refactor (was stacked on the pre-merge batch-simulate work).

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.20%. Comparing base (3ef3f23) to head (c29ad31).
⚠️ Report is 1 commits behind head on refactor.

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2071      +/-   ##
============================================
+ Coverage     97.19%   97.20%   +0.01%     
============================================
  Files           471      472       +1     
  Lines         28731    28845     +114     
============================================
+ Hits          27925    28039     +114     
  Misses          806      806              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)