save both old and new file structure for 1x1

[asamal4]

Description

save both old and new file structure for 1x1
Add additional metadata

Type of change

• Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up service version
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Unit tests improvement

Tools used to create PR

Identify any AI code assistants used in this PR (for transparency and review context)

• Assisted-by: Claude

Related Tickets & Documents

• Related Issue #
• Closes #

Checklist before requesting a review

• I have performed a self-review of my code.
• PR has passed all pre-merge test jobs.
• If it is a core feature, I have added thorough tests.

Testing

• Please provide detailed steps to perform tests related to this code change.
• How were the fix/results from this change verified? Please provide relevant screenshots or results.

Summary by CodeRabbit

• Improvements
  • Evaluation outputs now use a consistent nested directory structure across single- and multi-run evaluations.
  • Evaluation summaries provide clearer reporting for offline and agent-based runs, including token usage totals and report locations.
  • Multi-agent results can also be copied to a flat output location for easier access.
• Bug Fixes
  • Evaluations with no matching conversation groups now return complete totals, including token usage information.

[coderabbitai]

Walkthrough

The change standardizes run output directories under timestamped agent/run paths and refactors evaluation reporting to aggregate token-aware totals across offline and agent modes, including legacy flat-output copying and updated 1x1 path assertions.

Changes

Evaluation output and reporting

Layer / File(s)	Summary
Nested output layout src/lightspeed_evaluation/pipeline/behavioral/orchestrator.py, tests/unit/pipeline/behavioral/test_orchestrator.py	All runs use eval_<timestamp>/<agent>/run_<idx> output paths, and the 1x1 test verifies the nested layout.
Summary aggregation and output compatibility src/lightspeed_evaluation/runner/evaluation.py	Shared helpers aggregate evaluation and token totals, print available report locations, and copy nested agent output to a legacy flat directory when applicable.
Mode-specific totals and completion reporting src/lightspeed_evaluation/runner/evaluation.py	Empty, offline, and agent paths return token-aware totals and use the shared completion summary routine.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant run_evaluation
  participant RunResult
  participant _aggregate_totals
  participant _print_run_summary
  run_evaluation->>RunResult: collect evaluation summaries
  run_evaluation->>_aggregate_totals: aggregate run_results
  _aggregate_totals-->>run_evaluation: return totals
  run_evaluation->>_print_run_summary: print totals and report locations

Loading

Possibly related PRs

• lightspeed-core/lightspeed-evaluation#294: Updates the same orchestrator output layout and removes 1x1 path special-casing.
• lightspeed-core/lightspeed-evaluation#297: Modifies the same agent-mode orchestration and evaluation summary aggregation flow.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: preserving both old and new file structures for 1x1 runs.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lightspeed_evaluation/runner/evaluation.py`:
- Around line 88-106: Replace the direct summary/report-location print calls in
the evaluation completion flow with structured logger.info events, including
totals, mode, run count, and output directories as structured fields. Preserve
the existing CLI-formatted output separately only where it is an established
console contract, and retain the current single-run and multi-run directory
behavior.
- Around line 315-319: Add pytest coverage around the evaluation runner flow
containing _aggregate_totals and _copy_flat_output, mocking run_evaluation as
needed. Verify 1x1 runs copy flat output, multi-run executions do not copy it,
and token fields are aggregated correctly; ensure the tests invoke the runner
paths directly rather than relying on the orchestrator test.

In `@tests/unit/pipeline/behavioral/test_orchestrator.py`:
- Around line 404-405: Update the output_dir assertions in the orchestrator test
to normalize or inspect path components using platform-independent path
utilities, rather than matching the hardcoded "model_a/run_1" separator.
Preserve validation that the path contains the model/run layout and the "eval_"
component.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 8e3038bb-cb57-4810-a957-ccc0063c2087

📥 Commits

Reviewing files that changed from the base of the PR and between 590f807 and 0b8a929.

📒 Files selected for processing (3)

• src/lightspeed_evaluation/pipeline/behavioral/orchestrator.py
• src/lightspeed_evaluation/runner/evaluation.py
• tests/unit/pipeline/behavioral/test_orchestrator.py

[xmican10]

LGTM, thanks!