feat: add opt-in verbose_logs to surface GEval judge criteria, steps,… - #301
feat: add opt-in verbose_logs to surface GEval judge criteria, steps,…#301x86girl wants to merge 1 commit into
Conversation
… and rubrics GEval metrics ask the judge LLM to generate evaluation steps and rubrics internally, but these were never exposed in the output — making it hard to understand how a score was derived. This adds a `verbose: true` flag to any GEval metric config that captures the judge's criteria, evaluation steps, and rubric from DeepEval's verbose_logs and propagates them through the full pipeline to the JSON/CSV output. Changes: - GEvalConfig: new `verbose` bool field (default False), parsed from metadata - GEvalHandler: capture `metric.verbose_logs` at turn and conversation level - DeepEvalMetrics: propagate last_verbose_logs from the GEval handler - MetricResult: new `verbose_logs` optional field, inherited by EvaluationResult - MetricsEvaluator: read handler verbose_logs after evaluation, attach to result - Serializer: include verbose_logs in JSON output only when non-None - constants: add verbose_logs to SUPPORTED_CSV_COLUMNS - system.yaml: document the verbose option in metrics_metadata example Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
WalkthroughChangesGEval verbose log propagation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant GEvalConfig
participant GEvalHandler
participant MetricsEvaluator
participant MetricResult
participant result_to_json_dict
GEvalConfig->>GEvalHandler: provide verbose configuration
GEvalHandler->>GEvalHandler: capture verbose_logs
GEvalHandler->>MetricsEvaluator: expose last_verbose_logs
MetricsEvaluator->>MetricResult: assign verbose_logs
result_to_json_dict->>MetricResult: read verbose_logs
result_to_json_dict->>result_to_json_dict: include non-null logs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lightspeed_evaluation/pipeline/evaluation/evaluator.py`:
- Around line 230-236: Update the GEval/DeepEval verbose-log propagation in the
evaluator to use logs from the handlers created by _create_handler_for_judge
rather than self.handlers[framework]. Carry the selected or aggregated panel
logs through JudgeOrchestrator and assign them to MetricResult for the
corresponding panel run, avoiding stale primary-handler state. Add a test
covering verbose-log propagation from a panel-created handler.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 51c7c3aa-fd25-42c7-8778-aa22a8d6ec65
📒 Files selected for processing (13)
config/system.yamlsrc/lightspeed_evaluation/core/constants.pysrc/lightspeed_evaluation/core/metrics/deepeval.pysrc/lightspeed_evaluation/core/metrics/geval.pysrc/lightspeed_evaluation/core/models/data.pysrc/lightspeed_evaluation/core/models/llm.pysrc/lightspeed_evaluation/core/output/serializers.pysrc/lightspeed_evaluation/pipeline/evaluation/evaluator.pytests/unit/core/metrics/test_geval.pytests/unit/core/models/test_data.pytests/unit/core/models/test_system.pytests/unit/core/output/test_final_coverage.pytests/unit/pipeline/evaluation/test_evaluator.py
| # Capture verbose_logs from GEval handler if enabled | ||
| if framework in ("geval", "deepeval"): | ||
| handler = self.handlers.get(framework) | ||
| if handler is not None: | ||
| verbose_logs = getattr(handler, "last_verbose_logs", None) | ||
| if isinstance(verbose_logs, str): | ||
| metric_result.verbose_logs = verbose_logs |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Propagate verbose logs from panel-created handlers.
Panel GEval runs use DeepEvalMetrics instances created by _create_handler_for_judge, not necessarily self.handlers[framework]. This reads the primary handler’s state after orchestration, so panel verbose logs are dropped (or stale logs can be attached). Carry the selected/aggregated panel logs through JudgeOrchestrator into MetricResult, and add a panel-path test.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lightspeed_evaluation/pipeline/evaluation/evaluator.py` around lines 230
- 236, Update the GEval/DeepEval verbose-log propagation in the evaluator to use
logs from the handlers created by _create_handler_for_judge rather than
self.handlers[framework]. Carry the selected or aggregated panel logs through
JudgeOrchestrator and assign them to MetricResult for the corresponding panel
run, avoiding stale primary-handler state. Add a test covering verbose-log
propagation from a panel-created handler.
asamal4
left a comment
asamal4
left a comment
There was a problem hiding this comment.
Thanks !!
Please address coderabbit's comment. and PTAL my comments..
Thread safety — shared mutable state between write and read
Judge panel path — verbose logs not captured
Only GEval — but verbose logs are available on all DeepEval metrics
Only file backends — missing from SQL, MLflow, Langfuse
Inconsistent JSON schema — conditionally omitted unlike other optional fields
| """ | ||
| self.deepeval_llm_manager = deepeval_llm_manager | ||
| self.metric_manager = metric_manager | ||
| self.last_verbose_logs: str | None = None |
There was a problem hiding this comment.
last_verbose_logs is mutable instance state on a shared object. Handlers are created once and shared by all ThreadPoolExecutor threads. Between the write here and the read at evaluator.py, another thread can reset or overwrite the value. Same issue exists on deepeval.py.
|
|
||
| # Capture verbose_logs from GEval handler if enabled | ||
| if framework in ("geval", "deepeval"): | ||
| handler = self.handlers.get(framework) |
There was a problem hiding this comment.
When the judge panel path is used, fresh DeepEvalMetrics instances per judge are created. But this line reads from self.handlers.get(framework); the primary handler, not the panel handler. Panel verbose_logs are never captured.
| default=None, | ||
| description="Per-judge scores when using judge panel (for transparency)", | ||
| ) | ||
| verbose_logs: Optional[str] = Field( |
There was a problem hiding this comment.
This field is added to MetricResult and flows to CSV/JSON, but EvaluationResultDB has no verbose_logs column for other storage. Data is silently dropped on any non-file storage backend.
| "agent_latency": r.agent_latency, | ||
| "tokens_per_second": r.tokens_per_second, | ||
| } | ||
| if r.verbose_logs is not None: |
There was a problem hiding this comment.
Every other optional field in this dict (judge_scores, time_to_first_token, etc.) is always present, set to None when absent.
| turn_data=scope.turn_data, | ||
| is_conversation=scope.is_conversation, | ||
| ) | ||
| self.last_verbose_logs = self.geval_handler.last_verbose_logs |
There was a problem hiding this comment.
it's a DeepEval-wide feature, this code limit it to geval only, standard deepeval metric wont capture the verbose log.
| # expected_outcome: "Technically correct and follows best practices." | ||
| threshold: 0.7 | ||
| description: "General technical accuracy of provided commands, code, or technical information" | ||
| # verbose: true # optional: include verbose logs (criteria, evaluation steps, rubric) in output |
There was a problem hiding this comment.
Why does verbose need to be an individual metric property? either we would want the log for all deepeval metrics or none at all..
2 participants
Currently, lightspeed-evaluation does not report when creating rubrics using a Judge (LLM).
For correctness or consistency, for example, it asks the judge to define the steps to get this rubric, and lightspeed-evaluation is
not reporting anywhere how it has been done.
This PR create a new opt-in verbose_logs arg to surface GEval judge criteria, steps, and rubrics.
Summary by CodeRabbit
New Features
Tests