CSCS MI300 CI: select the ROCm provider explicitly - #1031
Merged
Conversation
The MI300 job set JULIA_AMDGPU_DISABLE_ARTIFACTS=1 to pick up the uenv's ROCm. That variable no longer exists after the discovery refactor, so the job silently resolved to the TheRock artifacts instead, and every test died with hipErrorOutOfMemory out of hipStreamCreateWithPriority (3167 occurrences; no hipMalloc ever failed). The .rocm-miopen merge and MIOPEN_PREFIX symlinking the script does were inert as a result. Select the provider through ROCm_Runtime's "local" preference instead, set before Pkg.instantiate() since it is a compile-time preference that also gates artifact resolution, and dev the two workspace packages the way .buildkite/pipeline.yml now does. The resolved provider is asserted against the requested one afterwards, so a future change of default fails loudly instead of quietly swapping the ROCm underneath the job. The script is parameterised by ROCM_LOCAL so both providers can be run. Only the uenv variant is enabled: this is the project's only gfx942 CI, so covering the default artifact path here is worthwhile, but TheRock ships a ROCm 7.14 userspace that cannot create a HIP stream at all against the 6.12.12 amdgpu driver on this machine, so that job is left commented out until the mismatch is resolved. Verified on beverin (2x MI300A, gfx942, Julia 1.12.6) at c644fc5: uenv ROCm, full suite, --jobs=32: 16162 pass, 0 OOM (was 3167 OOM) A/B in one allocation, only the preference differing: local=true HIP 7.2.53211 hipStreamCreateWithPriority OK local=false HIP 7.14.60850 hipErrorOutOfMemory Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment blamed a ROCm 7.14 userspace / 6.12.12 amdgpu driver mismatch. That was wrong: the driver side is fine (rocminfo enumerates all agents, hipMalloc succeeds). What actually fails is ROCclr's blit-kernel build on first stream creation, which cannot resolve __amd_streamOpsIncrement and __amd_streamOpsDecrement, and is then reported as hipErrorOutOfMemory -- which is what made this look like memory exhaustion for so long. Reported upstream as ROCm/TheRock#7426, so the comment now just points there rather than restating the analysis. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
luraess
force-pushed
the
lr/cscs-mi300-rocm-provider
branch
from
August 17, 2026 19:20
1d2535b to
4fafd6c
Compare
Member
Author
|
cscs-ci run |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1028 for the CSCS MI300 job, which currently fails every test.
ci/cscs-mi300.ymlusedJULIA_AMDGPU_DISABLE_ARTIFACTS=1to pick up the uenv's ROCm. That variable no longer exists after the discovery refactor, so the job silently resolved to the artifacts instead, and every test died withhipErrorOutOfMemoryout ofhipStreamCreateWithPrioritywith nohipMallocever failing.This selects the provider through
ROCm_Runtime'slocalpreference, set beforePkg.instantiate()since it also gates artifact resolution, and devs the two workspace packages the way.buildkite/pipeline.ymlnow does. The resolved provider is asserted against the requested one, so a change of default can't silently swap the ROCm underneath the job again.The script is parameterised by
ROCM_LOCALso both providers can be run, but only the uenv variant is enabled. The artifact path can't currently get off the ground on gfx942, see ROCm/TheRock#7426, That job is left commented out, to be re-enabled (withallow_failure) once that is resolved.