Skip to content

test: stabilize ARM runner setup#433

Merged
rapids-bot[bot] merged 1 commit into
NVIDIA:mainfrom
willkill07:test/arm-ci-stability
Jul 15, 2026
Merged

test: stabilize ARM runner setup#433
rapids-bot[bot] merged 1 commit into
NVIDIA:mainfrom
willkill07:test/arm-ci-stability

Conversation

@willkill07

@willkill07 willkill07 commented Jul 15, 2026

Copy link
Copy Markdown
Member

Overview

This isolates two unrelated ARM CI failures observed while validating #428: a sidecar-publication timing flake and transient Ubuntu ports package-download failures.

  • I confirm this contribution is my own work, or I have the right to submit it under this project license.
  • I searched existing issues and open pull requests, and this does not duplicate existing work.

Details

  • Allow up to 30 seconds for test sidecars to publish ownership state on slower ARM runners.
  • Use HTTPS for Ubuntu ports package sources on ARM.
  • Retry apt metadata and package downloads up to five times.

No production runtime behavior changes.

Validation:

  • Repeated the previously flaky CLI test 10 times.
  • cargo clippy -p nemo-relay-cli --test cli_tests -- -D warnings
  • just test-rust
  • uv run pre-commit run --all-files

Where should the reviewer start?

Start with the Install musl tools step in .github/workflows/ci_rust.yml, then review the test-only timeout constant in crates/cli/tests/cli_tests.rs.

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

Summary by CodeRabbit

  • Chores

    • Improved build reliability for ARM64 and musl-based environments by adding package-manager retries and platform-specific setup handling.
  • Tests

    • Increased the sidecar publication wait period to 30 seconds, reducing failures caused by slower startup or publishing conditions.

Signed-off-by: Will Killian <wkillian@nvidia.com>
@willkill07 willkill07 requested a review from a team as a code owner July 15, 2026 15:42
@github-actions github-actions Bot added size:S PR is small Test Test related lang:rust PR changes/introduces Rust code labels Jul 15, 2026
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 80952dac-f548-47b6-b346-f6c08572cf54

📥 Commits

Reviewing files that changed from the base of the PR and between 45831fc and 2178097.

📒 Files selected for processing (2)
  • .github/workflows/ci_rust.yml
  • crates/cli/tests/cli_tests.rs
📜 Recent review details
⏰ Context from checks skipped due to timeout. (1)
  • GitHub Check: Check / Run
🧰 Additional context used
📓 Path-based instructions (11)
.github/workflows/*.{yml,yaml}

📄 CodeRabbit inference engine (.agents/skills/maintain-ci/SKILL.md)

.github/workflows/*.{yml,yaml}: Put permissions: on each job that needs token access.
Avoid workflow-level permissions: unless the repository intentionally centralizes them and the inheritance tradeoff is documented.
Keep third-party actions pinned to full commit SHAs and preserve the readable version comment after the SHA.
Prefer action-native or ecosystem-native caching over generic actions/cache.
Use lockfiles or dependency manifests to drive cache invalidation.
Keep deploy and publish permissions isolated to the jobs that need them.
Read both caller and callee when a workflow uses workflow_call.
Put release-tag validation in the earliest practical caller job when the pipeline has tag-based publish behavior.
Keep pure-Python plugin SDK packaging as a single wheel artifact instead of duplicating it across every platform matrix entry.
contents: read is the default minimum for checkout-based build, test, docs, and packaging jobs.
pull-requests: read is required for PR metadata lookup jobs.
pages: write and id-token: write should be limited to Pages deployment jobs and any caller that invokes them through a reusable workflow.
For reusable workflows, the caller must grant every permission the called jobs require; the callee cannot elevate beyond what the caller provides.
Prefer astral-sh/setup-uv cache support with cache-dependency-glob anchored to uv.lock.
Prefer Swatinem/rust-cache with explicit shared-key and workspaces instead of ad hoc target-directory caching.
Avoid caching generated outputs that can hide stale behavior unless the repo already relies on them deliberately.

Keep CI workflow commands and package references in GitHub Actions workflows aligned with the current install, build, and example commands.

Files:

  • .github/workflows/ci_rust.yml
{justfile,codecov.yml,codecov.yaml,.github/workflows/**/*.yml,.github/workflows/**/*.yaml}

📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)

justfile, Codecov, and CI package/test workflows must include new plugin crates and packages.

Files:

  • .github/workflows/ci_rust.yml
{.github/**,.gitlab-ci.yml,.pre-commit-config.yaml,justfile,scripts/**}

⚙️ CodeRabbit configuration file

{.github/**,.gitlab-ci.yml,.pre-commit-config.yaml,justfile,scripts/**}: Review automation changes for reproducibility, pinned versions where appropriate, secret handling, and consistency with the documented validation matrix.
Pay attention to commands that need generated native artifacts, FFI libraries, or platform-specific environment variables.

Files:

  • .github/workflows/ci_rust.yml
**/*.rs

📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)

**/*.rs: Any Rust change must run just test-rust
Any Rust change must run cargo fmt --all
Any Rust change must run cargo clippy --workspace --all-targets -- -D warnings

**/*.rs: Run cargo fmt --all for all FFI work since it is Rust work
Run just test-rust to validate FFI changes
Run cargo clippy --workspace --all-targets -- -D warnings to enforce strict linting on FFI work

When Rust files changed as part of Go work, also run cargo fmt --all, just test-rust, and cargo clippy --workspace --all-targets -- -D warnings

**/*.rs: Run cargo fmt --all when Rust files are changed as part of Node work
Run cargo clippy --workspace --all-targets -- -D warnings when Rust files are changed as part of Node work
Run just test-rust when Rust files are changed as part of Node work

When changing the core Rust runtime or Rust-facing API surface, format Rust code with cargo fmt (rustfmt defaults), keep cargo clippy -- -D warnings clean, and satisfy cargo deny check per deny.toml.

**/*.rs: If any Rust code changed, always run just test-rust.
If any Rust code changed, also run cargo fmt --all.
If any Rust code changed, also run cargo clippy --workspace --all-targets -- -D warnings.
For Rust changes headed for review, run cargo fmt --all and cargo clippy --workspace --all-targets -- -D warnings even if relying on pre-commit.

Files:

  • crates/cli/tests/cli_tests.rs
**/*.{rs,py}

📄 CodeRabbit inference engine (AGENTS.md)

Follow binding naming conventions in Rust and Python: use snake_case.

Files:

  • crates/cli/tests/cli_tests.rs
**/*.{rs,py,js,mjs,cjs,ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.{rs,py,js,mjs,cjs,ts,tsx}: Use Json = serde_json::Value in Rust-facing runtime APIs where the existing code expects JSON payloads.
Use Result<T> with FlowError in core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.

Files:

  • crates/cli/tests/cli_tests.rs
**/*.{rs,py,go,js,ts,c,h}

📄 CodeRabbit inference engine (CONTRIBUTING.md)

Use language-appropriate naming conventions: Rust snake_case, C FFI exports prefixed nemo_relay_, Go PascalCase, Node.js camelCase, and Python snake_case.

Files:

  • crates/cli/tests/cli_tests.rs
**/*.{rs,go,js,ts}

📄 CodeRabbit inference engine (CONTRIBUTING.md)

Add the SPDX license header to all Rust, Go, JavaScript, and TypeScript source files using the corresponding // comment form.

Files:

  • crates/cli/tests/cli_tests.rs
**/*

📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)

**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, use maintain-dynamic-plugins and include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, prefer uv run pre-commit run --files <changed files...>.
Before review or handoff, run uv run pre-commit run --all-files.

Files:

  • crates/cli/tests/cli_tests.rs
**/*.{rs,py,go,js,ts}

📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)

If a language surface changed, always run that language's test target even when Rust core did not change.

Files:

  • crates/cli/tests/cli_tests.rs
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}

⚙️ CodeRabbit configuration file

{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.

Files:

  • crates/cli/tests/cli_tests.rs
🧠 Learnings (1)
📚 Learning: 2026-05-03T04:23:07.497Z
Learnt from: willkill07
Repo: NVIDIA/NeMo-Flow PR: 46
File: .github/workflows/ci_rust.yml:31-64
Timestamp: 2026-05-03T04:23:07.497Z
Learning: In GitHub Actions workflow YAML, it’s valid to conditionally disable a service container by setting the service container’s `image` to an empty string (`''`) via a matrix variable (e.g., `redis_service_image: ''`). This intentionally makes the runner skip service initialization for that matrix entry rather than failing the job. When reviewing workflows, don’t flag this as an error if the workflow uses an empty `image` to disable the service on specific matrix entries (e.g., OS-specific setups); verify the `image` is sourced from the matrix variable and that the service is only expected to be available when a non-empty image is provided.

Applied to files:

  • .github/workflows/ci_rust.yml
🪛 zizmor (1.26.1)
.github/workflows/ci_rust.yml

[warning] 241-241: code injection via template expansion (template-injection): may expand into attacker-controllable code

(template-injection)

🔇 Additional comments (2)
crates/cli/tests/cli_tests.rs (1)

27-27: LGTM!

Also applies to: 1254-1254

.github/workflows/ci_rust.yml (1)

240-249: LGTM!


Walkthrough

The changes harden musl package installation in the Rust CI workflow and replace a hardcoded sidecar publication test timeout with a shared 30-second constant.

Changes

CI package installation resilience

Layer / File(s) Summary
Harden musl tool installation
.github/workflows/ci_rust.yml
The musl installation step enables strict shell handling, rewrites apt sources for ARM64 runners, and configures apt retries.

Sidecar publication test timeout

Layer / File(s) Summary
Centralize sidecar publication timeout
crates/cli/tests/cli_tests.rs
Defines a 30-second timeout constant and uses it in wait_for_owned_sidecar.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title follows Conventional Commits, matches the ARM CI stabilization focus, and stays within the length and punctuation rules.
Description check ✅ Passed The description includes all required sections: Overview with checkboxes, Details, reviewer start point, and a related issue reference.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@willkill07 willkill07 added this to the 0.6 milestone Jul 15, 2026
@willkill07 willkill07 self-assigned this Jul 15, 2026
@willkill07

Copy link
Copy Markdown
Member Author

/merge

@rapids-bot rapids-bot Bot merged commit d1d70c2 into NVIDIA:main Jul 15, 2026
103 of 105 checks passed
rapids-bot Bot pushed a commit that referenced this pull request Jul 15, 2026
#### Overview

This isolates an unrelated Linux CI flake observed while validating #433. The worker cancellation contract succeeded, but its client task exceeded a two-second test deadline under a fully loaded Rust matrix.

- [x] I confirm this contribution is my own work, or I have the right to submit it under this project license.
- [x] I searched existing issues and open pull requests, and this does not duplicate existing work.

#### Details

Use one named 10-second deadline for both worker cancellation tests. The tests still fail boundedly when cancellation stalls, while allowing scheduling delays on loaded CI hosts. Production behavior is unchanged.

Validation:
- Repeated worker_service_cancels_unary_and_stream_invocations_by_id 30 times.
- cargo clippy -p nemo-relay-worker --test worker_sdk_tests -- -D warnings
- just test-rust
- uv run pre-commit run --all-files

#### Where should the reviewer start?

Review WORKER_TEST_TIMEOUT and its two cancellation-test call sites in crates/worker/tests/worker_sdk_tests.rs.

#### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

- Relates to #433



## Summary by CodeRabbit

* **Tests**
  * Standardized timeout handling in cancellation-related asynchronous tests.
  * Preserved existing test behavior while improving consistency and reliability.

Authors:
  - Will Killian (https://github.com/willkill07)

Approvers:
  - Zhongxuan (Daniel) Wang (https://github.com/zhongxuanwang-nv)

URL: #436
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lang:rust PR changes/introduces Rust code size:S PR is small Test Test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants