Skip to content

feat(eval-author): scaffold the nemo eval-author CLI surface - #968

Open
aleckhoury wants to merge 1 commit into
mainfrom
ase-685-eval-author-cli-scaffolding/akhoury
Open

feat(eval-author): scaffold the nemo eval-author CLI surface#968
aleckhoury wants to merge 1 commit into
mainfrom
ase-685-eval-author-cli-scaffolding/akhoury

Conversation

@aleckhoury

@aleckhoury aleckhoury commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Closes ASE-685.

Summary

  • Registers nemo eval-author with discover, audit, propose, run, and doctor. Simply scaffolding spaces for commands so we can parallelize more easily
  • Follows the two sibling optimizer CLIs rather than inventing a shape: a single cli.py holding a NemoCLI subclass whose get_cli() declares commands inline, matching ExperimentalistCLI and InsightsCLI down to the no-op root callback.

Summary by CodeRabbit

  • New Features

    • Added the nemo eval-author command with discover, audit, propose, run, and doctor subcommands.
    • Added CLI help output describing the available evaluation-authoring commands.
  • Bug Fixes

    • Not applicable; the subcommands currently indicate that implementation is pending.
  • Tests

    • Added coverage for command discovery, help output, pending-command behavior, and CLI registration.

@aleckhoury
aleckhoury requested review from a team as code owners July 29, 2026 15:55
@github-actions github-actions Bot added the feat label Jul 29, 2026
Base automatically changed from ase-eval-author-plugin/akhoury to main July 29, 2026 21:33
Register discover, audit, propose, run, and doctor so the command tree
exists and each child ticket has a landing spot. Every body exits non-zero
naming the ticket that owns it, because a placeholder that exits 0 reads as
a run that did nothing. Flags belong to those tickets, so no verb declares
options yet.

The CLI module imports nothing from eval_author. Those agents build their
LLM client while the class body executes, so importing one here would make
the whole CLI require AUTHOR_* credentials, including the doctor verb whose
job is to report them missing. Experimentalist keeps its runner behind a
lazily assigned module global for the same reason.

Declare nemo-platform-plugin directly instead of leaning on the transitive
path through Experimentalist, which is the coupling this plugin is shedding.

Cover the entry-point wiring in tests. A typo in the key or the import path
does not fail an import, it just makes nemo eval-author quietly missing from
the CLI, which no unit test of the module itself would catch.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
@aleckhoury
aleckhoury force-pushed the ase-685-eval-author-cli-scaffolding/akhoury branch from bcd8cd3 to f875731 Compare July 29, 2026 21:54
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Adds the nemo eval-author CLI plugin, registers five placeholder verbs, adds the nemo-platform-plugin dependency, and tests command help, failure responses, entry-point metadata, and class loading.

Changes

Eval Author CLI

Layer / File(s) Summary
CLI registration and command scaffolding
plugins/nemo-eval-author/pyproject.toml, plugins/nemo-eval-author/src/nemo_eval_author_plugin/cli.py
Registers eval-author to EvalAuthorCLI, adds the platform dependency, and defines discover, audit, propose, run, and doctor placeholder commands.
CLI and entry-point validation
plugins/nemo-eval-author/tests/test_cli.py
Tests help output, placeholder exit codes and ticket messages, entry-point metadata, and loading of EvalAuthorCLI.

Suggested reviewers: a2bondar, aahunt-nv, ajaythorve

Sequence Diagram(s)

sequenceDiagram
  participant NemoCLI
  participant EvalAuthorCLI
  participant TyperApp
  NemoCLI->>EvalAuthorCLI: load eval-author entry point
  EvalAuthorCLI->>TyperApp: register CLI verbs
  TyperApp->>EvalAuthorCLI: invoke selected placeholder
  EvalAuthorCLI-->>TyperApp: return ticket error and exit code 1
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: scaffolding the nemo eval-author CLI surface.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ase-685-eval-author-cli-scaffolding/akhoury

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor
Suite Lines Covered Line Rate Branch Rate
Unit Tests 28180/36015 78.2% 62.6%
Integration Tests 16417/34733 47.3% 19.7%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants