Skip to content

refactor discovery, ship therock tarballs as artifacts (2) - #1028

Draft
simeonschaub wants to merge 14 commits into
mainfrom
sds/rocm_libs
Draft

refactor discovery, ship therock tarballs as artifacts (2)#1028
simeonschaub wants to merge 14 commits into
mainfrom
sds/rocm_libs

Conversation

@simeonschaub

Copy link
Copy Markdown
Member

(Same as #1024, reopened from the upstream branch to run MI300 CI)

Some of the AI comments still need to be cleaned up but the logic looks reasonable from a first review, so I thought I'd put this up as a draft for feedback.

It turns out MIOpen needs all of libllvm and libclang, so unlike discussed in #919 (comment), I went back to shipping therock artifacts directly as done in that PR, since we don't gain much by repackaging anymore.

Otherwise, do a split similar to CUDA.jl of having ROCm_Runtime be a pseudo-jll providing automatic downloads through artifacts and having manual discovery live in ROCm_Runtime_Discovery. Either one of those is then loaded depending on a preference

@simeonschaub

Copy link
Copy Markdown
Member Author

Hmm, looks like an out of memory error. @luraess have you seen that before? The question is whether it's just an intermittent issue, whether ROCm 7.14 is really using more memory, or if this surfaced some other issue with the 7.14 vendor libs

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

cscs-ci run

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

Did something change from #1024 as this was passing Buildkite CI - like we're pulling in some newer jll or so? Else these failure may be MI300 specific.

@simeonschaub

Copy link
Copy Markdown
Member Author

No, nothing should have changed. I think this might be MI300 specific

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

The PR amends the Buildkite yml, and we may need to port those changes to the CSCS CI one. I am checking it rn.

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

Ok, so the issue on MI300 was the fact we did not update the yml file. I am working on a fix and possibly testing twice, once using local ROCm install, once using the local install. Is running not only from artifact but also testing one local ROCm install something we want to have also in Buildkite?

@simeonschaub

Copy link
Copy Markdown
Member Author

Those buildkite changes should only really matter on 1.10 though, on 1.12 Julia should respect the sources entry in the Project.toml file.

We should definitely test both configurations, but not sure we need to do this in buildkite too, seeing how the runners are already struggling too keep up

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

Ok. Yeah so using the new preference way I can now select back to use local ROCm install on MI300 and CI passes. However, it fails when using the new approach at the first HIP stream creation which may require investigation.

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

With the ROCm 7.14 TheRock artifact bundle, the first hipStreamCreateWithPriority makes ROCclr JIT-compile its internal blit kernels, and that link fails on undefined __amd_streamOpsIncrement/__amd_streamOpsDecrement. Those are present in the bundle only as weak symbols in opencl.bc, which isn't in the HIP device-library link set. Then ROCclr reports the failure misleadingly as hipErrorOutOfMemory. I'll dig in further.

@luraess

luraess commented Aug 17, 2026

Copy link
Copy Markdown
Member

On gfx942 (MI300A) the downloaded ROCm 7.14 artifacts don't seem usable as every HIP application we tried fails at its first hipStreamCreateWithPriority with hipErrorOutOfMemory. It doesn't look like memory exhaustion: ROCclr builds its blit kernels on first stream creation, that link fails on undefined __amd_streamOpsIncrement/__amd_streamOpsDecrement, and the error appears to surface as OOM. Reproducible in plain C++ against the bundle with no Julia involved. Filed upstream as ROCm/TheRock#7426.

RDNA looks unaffected as therock-dist-linux-gfx110X-all-7.14.0.tar.gz works, with the same device-library layout, so it may be specific to the gfx94X bundle.

For this PR it mainly matters that an MI300 user taking the new default would get an error pointing at memory rather than at the cause. Setting the local preference works fine, but that isn't discoverable from the message. Maybe we should decide what to do is such cases as well in the future, make a clearer error message or possibly fall back to system install if one is discovered?

Note that JULIA_AMDGPU_DISABLE_ARTIFACTS has no more effect.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMDGPU.jl Benchmarks

Details
Benchmark suite Current: 35abdbb Previous: 58dd394 Ratio
amdgpu/synchronization/context/device 497.5 ns 540 ns 0.92
amdgpu/synchronization/stream/blocking 225 ns 227.5 ns 0.99
amdgpu/synchronization/stream/nonblocking 282.5 ns 310 ns 0.91
array/accumulate/Float32/1d 78716 ns 78001 ns 1.01
array/accumulate/Float32/dims=1 261693.75 ns 275066.5 ns 0.95
array/accumulate/Float32/dims=1L 90371.25 ns 82516.25 ns 1.10
array/accumulate/Float32/dims=2 95676.5 ns 81193.75 ns 1.18
array/accumulate/Float32/dims=2L 2616077 ns 2758054.75 ns 0.95
array/accumulate/Int64/1d 87476.25 ns 79148.5 ns 1.11
array/accumulate/Int64/dims=1 247931 ns 243553.5 ns 1.02
array/accumulate/Int64/dims=1L 100421.5 ns 84101.25 ns 1.19
array/accumulate/Int64/dims=2 96364 ns 86991.25 ns 1.11
array/accumulate/Int64/dims=2L 2892270.75 ns 2895916.75 ns 1.00
array/broadcast 49340.75 ns 74006 ns 0.67
array/construct 2410 ns 2335 ns 1.03
array/copy 35708 ns 37160.75 ns 0.96
array/copyto!/cpu_to_gpu 89211.25 ns 111339.25 ns 0.80
array/copyto!/gpu_to_cpu 89283.75 ns 111596.5 ns 0.80
array/copyto!/gpu_to_gpu 35073 ns 59505.75 ns 0.59
array/iteration/findall/bool 144339.5 ns 139364.75 ns 1.04
array/iteration/findall/int 146909.5 ns 152744.75 ns 0.96
array/iteration/findfirst/bool 151657 ns 145527.25 ns 1.04
array/iteration/findfirst/int 146269.5 ns 145997.25 ns 1.00
array/iteration/findmin/1d 132011.75 ns 124581.75 ns 1.06
array/iteration/findmin/2d 117439.25 ns 98116.25 ns 1.20
array/iteration/logical 232818.25 ns 252929 ns 0.92
array/iteration/scalar 297279.25 ns 309789.75 ns 0.96
array/permutedims/2d 70678.5 ns 72443.5 ns 0.98
array/permutedims/3d 70218.5 ns 71971 ns 0.98
array/permutedims/4d 72523.5 ns 75061 ns 0.97
array/random/rand/Float32 44275.75 ns 45963 ns 0.96
array/random/rand/Int64 53338.25 ns 54770.75 ns 0.97
array/random/rand!/Float32 41418 ns 44300.75 ns 0.93
array/random/rand!/Int64 49123.25 ns 73566 ns 0.67
array/random/randn/Float32 79568.5 ns 76658.75 ns 1.04
array/random/randn!/Float32 55983.25 ns 81353.5 ns 0.69
array/reductions/mapreduce/Float32/1d 107894 ns 101831.5 ns 1.06
array/reductions/mapreduce/Float32/dims=1 91901.5 ns 91791.25 ns 1.00
array/reductions/mapreduce/Float32/dims=1L 841879.25 ns 834629.75 ns 1.01
array/reductions/mapreduce/Float32/dims=2 98316.25 ns 88776.5 ns 1.11
array/reductions/mapreduce/Float32/dims=2L 143004.5 ns 144237 ns 0.99
array/reductions/mapreduce/Int64/1d 109051.5 ns 101501.5 ns 1.07
array/reductions/mapreduce/Int64/dims=1 93616.25 ns 90426.25 ns 1.04
array/reductions/mapreduce/Int64/dims=1L 841434.5 ns 834257.25 ns 1.01
array/reductions/mapreduce/Int64/dims=2 96831.25 ns 88551.5 ns 1.09
array/reductions/mapreduce/Int64/dims=2L 143574.5 ns 145772.25 ns 0.98
array/reductions/reduce/Float32/1d 110661.5 ns 102909 ns 1.08
array/reductions/reduce/Float32/dims=1 93353.75 ns 91134 ns 1.02
array/reductions/reduce/Float32/dims=1L 841414.5 ns 835889.75 ns 1.01
array/reductions/reduce/Float32/dims=2 98279 ns 88548.75 ns 1.11
array/reductions/reduce/Float32/dims=2L 143287 ns 145259.5 ns 0.99
array/reductions/reduce/Int64/1d 110349 ns 101701.5 ns 1.09
array/reductions/reduce/Int64/dims=1 93531.5 ns 88971.5 ns 1.05
array/reductions/reduce/Int64/dims=1L 843162 ns 831467.25 ns 1.01
array/reductions/reduce/Int64/dims=2 96711.5 ns 88508.75 ns 1.09
array/reductions/reduce/Int64/dims=2L 142859.5 ns 144032.25 ns 0.99
array/reverse/1d 45173.25 ns 46513.25 ns 0.97
array/reverse/1dL 75358.75 ns 73918.5 ns 1.02
array/reverse/1dL_inplace 58006 ns 81293.75 ns 0.71
array/reverse/1d_inplace 38320.5 ns 53618.5 ns 0.71
array/reverse/2d 49428 ns 51231 ns 0.96
array/reverse/2dL 96011.5 ns 84231.25 ns 1.14
array/reverse/2dL_inplace 49048.25 ns 92779 ns 0.53
array/reverse/2d_inplace 40780.5 ns 42753 ns 0.95
array/sorting/1d 333802.25 ns 334009.75 ns 1.00
integration/byval/reference 38741 ns 39821 ns 0.97
integration/byval/slices=1 40321 ns 40321 ns 1
integration/byval/slices=2 150262 ns 147452 ns 1.02
integration/byval/slices=3 250694 ns 240194 ns 1.04
integration/volumerhs 5004631 ns 5007121 ns 1.00
kernel/indexing 35155.5 ns 58288.5 ns 0.60
kernel/indexing_checked 36755.5 ns 50148.25 ns 0.73
kernel/launch 1347.5 ns 1445 ns 0.93
kernel/rand 90091.25 ns 101319 ns 0.89
latency/import 1808372911 ns 1689718608 ns 1.07
latency/precompile 38418239792 ns 38551162270 ns 1.00
latency/ttfp 2393580787 ns 2308017436 ns 1.04

This comment was automatically generated by workflow using github-action-benchmark.

@evetion

evetion commented Aug 20, 2026

Copy link
Copy Markdown

Tested on my 6900XT. Only one failure, which is fixed in #1042, but some interesting warnings, like

Output generated during execution of 'hip_dnn/conv':
┌ MIOpen(HIP): Warning [ParseAndLoadDb] File is unreadable: "/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/share/miopen/db/gfx1030_40.HIP.fdb.txt"
│ MIOpen(HIP): Warning [OpenRuntimeLibraryForDevice] CK grouped conv library not found for device gfx1030: libMIOpenCKGroupedConv_gfx1030.so: cannot open shared object file: No such file or directory
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ MIOpen(HIP): Warning [IsEnoughWorkspace] [EvaluateInvokers] Solver <GemmFwdRest>, workspace required: 4032, provided ptr: 0 size: 0
└ MIOpen Error: localhost.localdomain:/__w/rockrel/rockrel/rocm-libraries/projects/miopen/src/convolution.cpp:249: Channels do not match for the filter

Full run:

Details

(AMDGPU) pkg> test
      Status `~/.julia/dev/AMDGPU/test/Project.toml`
  [21141c5a] AMDGPU v2.7.3 `.`
  [79e6a3ab] Adapt v4.7.0
  [ab4f0b2a] BFloat16s v0.6.1
  [7a1cc6ca] FFTW v1.10.0
  [1a297f60] FillArrays v1.17.0
  [f6369f11] ForwardDiff v1.4.5
  [0c68f7d7] GPUArrays v11.5.12
  [61eb1bfa] GPUCompiler v2.2.1
  [033835bb] JLD2 v0.6.5
  [63c18a36] KernelAbstractions v0.9.42
  [929cbde3] LLVM v9.13.0
  [d3525ed8] ParallelTestRunner v2.8.1
  [08abe8d2] PrettyTables v3.4.8
  [0bf8d700] ROCm_Runtime_Discovery v0.1.0 `ROCm_Runtime_Discovery`
  [a0a7dd2c] SparseMatricesCSR v0.6.12
  [276daf66] SpecialFunctions v2.9.0
  [90137ffa] StaticArrays v1.9.19
  [10745b16] Statistics v1.11.1
  [8ba89e20] Distributed v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [37e2e46d] LinearAlgebra v1.12.0
  [44cfe95a] Pkg v1.12.1
  [9a3f8284] Random v1.11.0
  [2f01184e] SparseArrays v1.12.0
  [fa267f1f] TOML v1.0.3
  [8dfed614] Test v1.11.0
      Status `~/.julia/dev/AMDGPU/Manifest.toml`
  [21141c5a] AMDGPU v2.7.3 `.`
  [621f4979] AbstractFFTs v1.5.0
  [6a4ca0a5] AcceleratedKernels v0.4.3
  [79e6a3ab] Adapt v4.7.0
  [dce04be8] ArgCheck v2.5.0
  [4fba245c] ArrayInterface v7.30.0
  [a9b6321e] Atomix v1.1.3
  [ab4f0b2a] BFloat16s v0.6.1
  [62783981] BitTwiddlingConvenienceFunctions v0.1.6
  [fa961155] CEnum v0.5.0
  [2a0fbf3d] CPUSummary v0.2.7
  [0b6fb165] ChunkCodecCore v1.0.1
  [4c0bbee4] ChunkCodecLibZlib v1.1.0
  [55437552] ChunkCodecLibZstd v1.0.0
  [fb6a15b2] CloseOpenIntervals v0.1.13
  [bbf7d656] CommonSubexpressions v0.3.1
  [f70d9fcc] CommonWorldInvalidations v1.1.2
  [34da2185] Compat v4.18.1
  [9db33cc3] CompilerCaching v0.4.2
  [adafc99b] CpuId v0.3.1
  [a8cc5b0e] Crayons v4.2.0
  [9a962f9c] DataAPI v1.16.0
  [e2d170a0] DataValueInterfaces v1.0.0
  [163ba53b] DiffResults v1.1.0
  [b552c78f] DiffRules v1.16.0
  [ffbed154] DocStringExtensions v0.9.5
  [e2ba6199] ExprTools v0.1.11
  [7a1cc6ca] FFTW v1.10.0
  [5789e2e9] FileIO v1.20.0
  [1a297f60] FillArrays v1.17.0
  [f6369f11] ForwardDiff v1.4.5
  [0c68f7d7] GPUArrays v11.5.12
  [46192b85] GPUArraysCore v0.2.0
  [61eb1bfa] GPUCompiler v2.2.1
  [096a3bc2] GPUToolbox v3.0.0
  [076d061b] HashArrayMappedTries v0.2.0
  [b5f81e59] IOCapture v1.0.0
  [615f187c] IfElse v0.1.1
  [92d709cd] IrrationalConstants v0.2.6
  [82899510] IteratorInterfaceExtensions v1.0.0
  [033835bb] JLD2 v0.6.5
  [692b3bcd] JLLWrappers v1.8.0
  [63c18a36] KernelAbstractions v0.9.42
  [929cbde3] LLVM v9.13.0
  [b964fa9f] LaTeXStrings v1.4.1
  [10f19ff3] LayoutPointers v0.1.17
  [2ab3a3ac] LogExpFunctions v1.0.1
  [1914dd2f] MacroTools v0.5.16
  [36869731] Malt v1.4.1
  [d125e4d3] ManualMemory v0.1.8
  [77ba4419] NaNMath v1.1.4
  [bac558e1] OrderedCollections v2.0.1
  [d3525ed8] ParallelTestRunner v2.8.1
  [f517fe37] Polyester v0.7.19
  [1d0040c9] PolyesterWeave v0.2.2
  [aea7be01] PrecompileTools v1.3.4
  [21216c6a] Preferences v1.5.2
  [08abe8d2] PrettyTables v3.4.8
  [3129f4d2] ROCm_Runtime v7.14.0 `ROCm_Runtime`
  [0bf8d700] ROCm_Runtime_Discovery v0.1.0 `ROCm_Runtime_Discovery`
  [74087812] Random123 v1.7.1
  [e6cf234a] RandomNumbers v1.6.0
  [189a3867] Reexport v1.2.2
  [05181044] RelocatableFolders v1.0.1
  [ae029012] Requires v1.3.1
  [94e857df] SIMDTypes v0.1.0
  [431bcebd] SciMLPublic v1.2.4
  [7e506255] ScopedValues v1.6.2
  [6c6a2e73] Scratch v1.3.0
  [a0a7dd2c] SparseMatricesCSR v0.6.12
  [276daf66] SpecialFunctions v2.9.0
  [aedffcd0] Static v1.4.6
  [0d7ed370] StaticArrayInterface v1.10.0
  [90137ffa] StaticArrays v1.9.19
  [1e83bf80] StaticArraysCore v1.4.4
  [10745b16] Statistics v1.11.1
  [7792a7ef] StrideArraysCore v0.5.9
  [892a3eda] StringManipulation v0.5.0
  [3783bdb8] TableTraits v1.0.1
  [bd369af6] Tables v1.13.0
  [8290d209] ThreadingUtilities v0.5.6
  [e689c965] Tracy v0.1.6
  [013be700] UnsafeAtomics v0.3.2
  [cc5c0156] AMDGPU_LLVM_Backend_jll v22.1.8+2
  [f5851436] FFTW_jll v3.3.12+0
  [1d5cc7b8] IntelOpenMP_jll v2025.2.0+0
  [dad2f222] LLVMExtra_jll v0.0.46+0
  [ad6e5548] LibTracyClient_jll v0.13.1+0
  [856f044c] MKL_jll v2025.2.0+0
  [efe28fd5] OpenSpecFun_jll v0.5.6+0
⌅ [873c0968] ROCmDeviceLibs_jll v6.2.1+2
  [3161d3a3] Zstd_jll v1.5.7+1
  [1317d2d5] oneTBB_jll v2022.3.0+0
  [8b5cbfcf] tree_sitter_gcn_jll v0.1.0+0
  [44208993] tree_sitter_llvm_jll v1.1.0+0
  [71e3f6e6] tree_sitter_ptx_jll v0.1.0+0
  [f0e86581] tree_sitter_spirv_jll v0.1.0+0
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [8ba89e20] Distributed v1.11.0
  [f43a241f] Downloads v1.7.0
  [7b1f6079] FileWatching v1.11.0
  [b77e0a4c] InteractiveUtils v1.11.0
  [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0
  [4af54fe1] LazyArtifacts v1.11.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [37e2e46d] LinearAlgebra v1.12.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [a63ad114] Mmap v1.11.0
  [ca575930] NetworkOptions v1.3.0
  [44cfe95a] Pkg v1.12.1
  [de0858da] Printf v1.11.0
  [3fa0cd96] REPL v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization v1.11.0
  [6462fe0b] Sockets v1.11.0
  [2f01184e] SparseArrays v1.12.0
  [f489334b] StyledStrings v1.11.0
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test v1.11.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.3.0+1
  [deac9b47] LibCURL_jll v8.15.0+0
  [e37daf67] LibGit2_jll v1.9.0+0
  [29816b5a] LibSSH2_jll v1.11.3+1
  [14a3606d] MozillaCACerts_jll v2025.11.4
  [4536629a] OpenBLAS_jll v0.3.29+0
  [05823500] OpenLibm_jll v0.8.7+0
  [458c3c95] OpenSSL_jll v3.5.4+0
  [bea87d4a] SuiteSparse_jll v7.8.3+2
  [83775a58] Zlib_jll v1.3.1+2
  [8e850b90] libblastrampoline_jll v5.15.0+0
  [8e850ede] nghttp2_jll v1.64.0+1
  [3f19e933] p7zip_jll v17.7.0+0
        Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading.
Precompiling for configuration --code-coverage=none --color=yes --check-bounds=yes --warn-overwrite=yes --depwarn=yes --inline=yes --startup-file=no --track-allocation=none
Precompiling packages finished.
  3 dependencies successfully precompiled in 29 seconds. 89 already precompiled.
     Testing Running tests...
[ Info: System information:
Julia Version 1.12.6
Commit 15346901f00 (2026-04-09 19:20 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 24 × AMD Ryzen 9 5900X 12-Core Processor
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, znver3)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 24 virtual cores)
Environment:
  JULIA_LOAD_PATH = @:/home/evetion/.julia/dev/AMDGPU/test
AMDGPU versioninfo
ROCm provider: downloaded artifacts
┌───────────┬──────────────────┬────────────┬─────────────────────────────────────────────────────────────────────────────────────────────┐
│ Available │ Name             │ Version    │ Path                                                                                        │
├───────────┼──────────────────┼────────────┼─────────────────────────────────────────────────────────────────────────────────────────────┤
│     +     │ LLD              │ -/home/evetion/.julia/artifacts/e21d5c5990876ecba40cb3d23b0b6fc6126709ec/bin/lld             │
│     +     │ Device Libraries │ -/home/evetion/.julia/artifacts/0a0cad46b7dacc66387ed19e8c0f45b097f46691/amdgcn/bitcode      │
│     +     │ HIP              │ 7.14.60850/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/libamdhip64.so  │
│     +     │ rocBLAS          │ 5.5.0/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/librocblas.so   │
│     +     │ rocSOLVER        │ 3.35.0/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/librocsolver.so │
│     +     │ rocSPARSE        │ 4.7.0/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/librocsparse.so │
│     +     │ rocRAND          │ 4.5.0/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/librocrand.so   │
│     +     │ rocFFT           │ 1.0.38/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/librocfft.so    │
│     +     │ MIOpen           │ 3.5.2/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/lib/libMIOpen.so    │
└───────────┴──────────────────┴────────────┴─────────────────────────────────────────────────────────────────────────────────────────────┘

AMDGPU devices
┌────┬───────────────────────┬──────────┬───────────┬────────────┬───────────────┐
│ Id │                  Name │ GCN arch │ Wavefront │     Memory │ Shared Memory │
├────┼───────────────────────┼──────────┼───────────┼────────────┼───────────────┤
│  1 │ AMD Radeon RX 6900 XT │  gfx1030 │        3215.984 GiB │    64.000 KiB │
└────┴───────────────────────┴──────────┴───────────┴────────────┴───────────────┘

┌ Info: Available tests:
│ core/codegen
│ core/core_tests
│ core/discovery_tests
│ core/graph_tests
│ core/rocarray_base
│ core/rocarray_broadcast
│ core/tls
│ device/array
│ device/exceptions
│ device/execution_control
│ device/hostcall
│ device/indexing
│ device/launch
│ device/math
│ device/memory
│ device/output
│ device/random
│ device/regression
│ device/synchronization
│ device/vadd
│ device/wavefront
│ enzyme_tests
│ external/forwarddiff
│ gpuarrays/alloc cache
│ gpuarrays/base
│ gpuarrays/broadcasting
│ gpuarrays/constructors
│ gpuarrays/ext/jld2
│ gpuarrays/indexing find
│ gpuarrays/indexing multidimensional
│ gpuarrays/indexing reshaped wrappers
│ gpuarrays/indexing scalar
│ gpuarrays/linalg/NaN_false
│ gpuarrays/linalg/core
│ gpuarrays/linalg/diagonal
│ gpuarrays/linalg/kron
│ gpuarrays/linalg/kron_diagonal
│ gpuarrays/linalg/mul!/integer-accumulate
│ gpuarrays/linalg/mul!/matrix-matrix
│ gpuarrays/linalg/mul!/mixed-eltype
│ gpuarrays/linalg/mul!/strided-views
│ gpuarrays/linalg/mul!/vector-matrix
│ gpuarrays/linalg/norm
│ gpuarrays/math/intrinsics
│ gpuarrays/math/power
│ gpuarrays/random
│ gpuarrays/reductions/== isequal
│ gpuarrays/reductions/and or xor
│ gpuarrays/reductions/any all count
│ gpuarrays/reductions/mapreduce
│ gpuarrays/reductions/mapreducedim!
│ gpuarrays/reductions/mapreducedim!_large
│ gpuarrays/reductions/minimum maximum extrema
│ gpuarrays/reductions/sum prod
│ gpuarrays/sparse
│ gpuarrays/statistics
│ gpuarrays/uniformscaling
│ gpuarrays/vectors
│ hip_core_tests
│ hip_dnn/activations
│ hip_dnn/batchnorm
│ hip_dnn/conv
│ hip_dnn/descriptors
│ hip_dnn/pool
│ hip_rocarray/blas
│ hip_rocarray/fft
│ hip_rocarray/indexing
│ hip_rocarray/multi_gpu
│ hip_rocarray/random
│ hip_rocarray/reverse
│ hip_rocarray/solver
│ hip_rocarray/sorting
│ hip_rocsparse/broadcast
│ hip_rocsparse/conversions
│ hip_rocsparse/device
│ hip_rocsparse/generic
│ hip_rocsparse/interfaces
│ hip_rocsparse/linalg
│ hip_rocsparse/preconditioners
│ hip_rocsparse/sparse_matrices_csr
│ kernelabstractions_tests
│ wmma_rdna3_tests
└ wmma_rdna4_tests
Running 80 tests using 24 parallel jobs. If this is too many concurrent jobs, specify the `--jobs=N` argument to the tests, or set the `JULIA_CPU_THREADS` environment variable.
                                                   │   Test   │ ──────────────── CPU ──────────────── │
Test                                      (Worker) │ time (s) │ GC (s) │ GC % │ Alloc (MB) │ RSS (MB) │
core/codegen                                  (10) │    27.94 │   failed at 2026-08-20T14:48:47.469
gpuarrays/indexing find                       (23) │   116.572.001.75019.561403.18 │
gpuarrays/linalg/diagonal                     (20) │   142.702.231.65127.821750.55 │
gpuarrays/ext/jld2                            (21) │   162.812.161.35452.501330.11 │
gpuarrays/math/power                          (10) │   137.852.952.16862.151720.43 │
gpuarrays/reductions/mapreducedim!_large      (12) │   192.534.322.27224.991571.34 │
hip_rocarray/fft                              (18) │   193.712.921.54827.511706.62 │
device/wavefront                              (22) │   198.422.031.04563.271496.95 │
gpuarrays/constructors                        (24) │   216.902.821.36425.591405.97 │
gpuarrays/statistics                          (19) │   238.233.541.59063.441969.02 │
hip_rocarray/sorting                          (17) │   253.053.271.37987.501696.25 │
gpuarrays/reductions/== isequal               (16) │   256.343.451.38920.671576.09 │
hip_rocsparse/broadcast                       (23) │   134.701.210.95681.462215.89 │
gpuarrays/indexing multidimensional           (11) │   278.193.691.39617.091806.81 │
hip_rocsparse/generic                         (20) │   142.761.371.03640.912205.06 │
gpuarrays/base                                (21) │   130.612.471.95643.061905.07 │
hip_rocsparse/interfaces                      (13) │   304.453.981.38886.391916.57 │
kernelabstractions_tests                      (10) │   133.572.632.05293.042209.82 │
core/graph_tests                              (21) │    11.620.080.7472.331957.14 │
gpuarrays/indexing reshaped wrappers          (22) │   118.151.651.45139.431823.52 │
gpuarrays/linalg/mul!/strided-views           (19) │    80.150.891.13870.192283.23 │
gpuarrays/random                              (24) │   102.710.730.73069.241755.69 │
device/random                                 (18) │   147.031.451.03695.302031.11 │
hip_rocsparse/linalg                          (17) │    99.700.720.73100.952351.42 │
hip_rocarray/reverse                          (24) │    30.280.351.21297.201888.52 │
gpuarrays/reductions/and or xor               (16) │   100.600.780.83133.901845.16 │
core/rocarray_broadcast                       (13) │    53.420.490.92077.332135.75 │
external/forwarddiff                          (22) │    37.080.962.61595.591969.10 │
hip_rocarray/random                           (19) │    41.020.280.71218.512530.88 │
gpuarrays/linalg/mul!/vector-matrix           (12) │   178.581.821.05721.892236.48 │
core/rocarray_base                            (21) │    52.220.340.61802.452369.13 │
gpuarrays/linalg/NaN_false                    (20) │    81.870.610.72726.322340.18 │
hip_rocsparse/preconditioners                 (23) │   117.180.510.42539.152353.46 │
hip_rocsparse/conversions                     (13) │    15.650.070.5519.122284.49 │
core/core_tests                               (22) │    15.500.000.018.541969.29 │
gpuarrays/linalg/mul!/mixed-eltype            (11) │   107.091.791.75095.982074.33 │
gpuarrays/reductions/any all count            (18) │    40.230.320.81473.762152.79 │
hip_rocsparse/sparse_matrices_csr             (24) │    25.730.110.4496.552273.71 │
gpuarrays/indexing scalar                     (10) │    78.970.370.51766.172371.20 │
gpuarrays/linalg/kron_diagonal                (12) │    15.540.080.5398.312410.23 │
gpuarrays/linalg/mul!/integer-accumulate      (19) │    21.870.160.7818.132614.25 │
device/math                                   (17) │    35.370.210.61077.952463.01 │
gpuarrays/math/intrinsics                     (20) │    16.200.070.5572.882480.91 │
hip_dnn/batchnorm                             (21) │    18.790.000.0232.182758.63 │
device/launch                                 (13) │    12.100.070.6413.112343.81 │
hip_rocarray/indexing                         (23) │    16.000.100.6570.912493.31 │
device/exceptions                             (18) │     8.660.000.0314.282236.98 │
gpuarrays/uniformscaling                      (16) │    39.230.250.61063.512017.14 │
device/memory                                 (11) │    12.460.000.0267.542111.79 │
gpuarrays/alloc cache                         (21) │     1.880.000.092.692758.63 │
device/synchronization                        (12) │     7.300.000.0185.972410.23 │
hip_rocsparse/device                          (19) │     7.500.000.074.282708.81 │
device/execution_control                      (13) │     4.470.000.0134.852343.81 │
device/array                                  (20) │     5.850.000.0173.092480.91 │
core/discovery_tests                          (18) │     1.180.000.045.872236.98 │
hip_dnn/descriptors                           (10) │    10.820.000.0192.492371.20 │
device/indexing                               (23) │     5.870.000.0105.312493.31 │
device/vadd                                   (16) │     3.960.000.0115.752017.14 │
core/tls                                      (11) │     2.870.000.0132.392239.95 │
hip_core_tests                                (21) │     2.100.000.073.392758.63 │
wmma_rdna4_tests                              (13) │     0.120.000.01.542343.81 │
gpuarrays/vectors                             (19) │     0.820.000.023.552721.56 │
hip_rocarray/multi_gpu                        (20) │     0.210.000.01.542480.91 │
device/regression                             (24) │    20.950.100.5515.742427.00 │
wmma_rdna3_tests                              (18) │     0.550.000.023.322236.98 │
gpuarrays/sparse                              (10) │     0.020.000.01.062371.20 │
gpuarrays/linalg/mul!/matrix-matrix            (6) │   428.644.921.112205.982186.73 │
hip_rocarray/solver                            (4) │   428.785.971.414613.912703.00 │
hip_dnn/activations                           (17) │    23.190.000.0306.902933.34 │
hip_dnn/conv                                  (22) │    41.560.070.2464.752917.51 │
gpuarrays/linalg/kron                         (14) │   438.369.092.114433.012193.75 │
hip_dnn/pool                                  (12) │    23.440.000.096.362723.57 │
gpuarrays/reductions/mapreduce                 (5) │   449.505.521.213483.011973.34 │
gpuarrays/reductions/sum prod                  (2) │   488.366.541.316708.232367.82 │
gpuarrays/reductions/mapreducedim!             (1) │   497.156.921.418441.332449.36 │
hip_rocarray/blas                              (3) │   514.969.051.825136.482838.43 │
gpuarrays/reductions/minimum maximum extrema   (7) │   533.728.041.523260.852736.15 │
gpuarrays/linalg/core                         (15) │   535.899.391.826925.573194.47 │
gpuarrays/linalg/norm                          (8) │   583.768.381.428219.413381.32 │
gpuarrays/broadcasting                         (9) │   645.0410.691.735856.753771.40 │

Output generated during execution of 'gpuarrays/linalg/diagonal':
┌ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Float32, 1}, ::AMDGPU.Device.ROCDeviceVector{Float32, 1}, ::Tuple{Int64, Int64}, ::Int64, ::UnitRange{Int64}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
└ └ Otherwise, performance might degrade if they keep running in the background.

Output generated during execution of 'gpuarrays/indexing multidimensional':
┌ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Int16, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Int16, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Int16, 1}, ::AMDGPU.Device.ROCDeviceArray{Int16, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Int32, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Int32, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Int32, 1}, ::AMDGPU.Device.ROCDeviceArray{Int32, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Int64, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Int64, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Int64, 1}, ::AMDGPU.Device.ROCDeviceArray{Int64, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Float16, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float16, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Float16, 1}, ::AMDGPU.Device.ROCDeviceArray{Float16, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Float32, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float32, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Float32, 1}, ::AMDGPU.Device.ROCDeviceArray{Float32, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Float64, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float64, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Float64, 1}, ::AMDGPU.Device.ROCDeviceArray{Float64, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{ComplexF16, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF16, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF16, 1}, ::AMDGPU.Device.ROCDeviceArray{ComplexF16, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{ComplexF32, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF32, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF32, 1}, ::AMDGPU.Device.ROCDeviceArray{ComplexF32, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{ComplexF64, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF64, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF64, 1}, ::AMDGPU.Device.ROCDeviceArray{ComplexF64, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int16}, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int16}, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int16}, 1}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int16}, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int32}, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int32}, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int32}, 1}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int32}, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int64}, 3, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int64}, 1}, ::Tuple{Int64, Int64, Int64}, ::Int64, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{2, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}, CartesianIndices{2, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Complex{Int64}, 1}, ::AMDGPU.Device.ROCDeviceArray{Complex{Int64}, 3, 1}, ::Tuple{Int64, Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_setindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceMatrix{Float32, 1}, ::AMDGPU.Device.ROCDeviceVector{Float32, 1}, ::Tuple{Int64, Int64}, ::Int64, ::Int64, ::Base.Slice{Base.OneTo{Int64}})
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{Float32, 1}, ::LinearAlgebra.Adjoint{Float32, AMDGPU.Device.ROCDeviceMatrix{Float32, 1}}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
└ └ Otherwise, performance might degrade if they keep running in the background.

Output generated during execution of 'kernelabstractions_tests':
┌ Precompiling packages...6826.4 ms  ✓ AMDGPU  AMDGPUSpecialFunctionsExt
│   1 dependency successfully precompiled in 7 seconds. 98 already precompiled.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{Float64, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float64, 1}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
└ [ Info: Stopped global hostcall: `malloc_hostcall`.

Output generated during execution of 'hip_rocsparse/sparse_matrices_csr':
┌ Precompiling packages...7155.6 ms  ✓ ArrayInterface  ArrayInterfaceAMDGPUExt
│   1 dependency successfully precompiled in 7 seconds. 96 already precompiled.
│ Precompiling packages...4694.5 ms  ✓ AMDGPU  AMDGPUSparseMatricesCSRExt
└   1 dependency successfully precompiled in 5 seconds. 118 already precompiled.

Output generated during execution of 'device/launch':
┌ ┌ Warning: Only 1 GPU detected; skipping multi-GPU tests
└ └ @ Main.var"##device/launch#1208" ~/.julia/dev/AMDGPU/test/device/launch.jl:91

Output generated during execution of 'core/discovery_tests':
┌ Warning: HIP_PLATFORM=hcc is deprecated.Please use HIP_PLATFORM=amd.
└ Warning: HIP_PLATFORM=hcc is deprecated.Please use HIP_PLATFORM=amd.

Output generated during execution of 'wmma_rdna4_tests':
[ [ Info: Skipping WMMA_RDNA4 tests (requires RDNA4+ / gfx1200+)

Output generated during execution of 'hip_rocarray/multi_gpu':
[ [ Info: Skipping Multi-GPU tests (requires more than 1 GPU)

Output generated during execution of 'wmma_rdna3_tests':
[ [ Info: Skipping WMMA_RDNA3 tests (requires RDNA3+)

Output generated during execution of 'hip_rocarray/solver':
┌ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{Float32, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float32, 1}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{Float64, 1}, ::AMDGPU.Device.ROCDeviceMatrix{Float64, 1}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{ComplexF32, 1}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF32, 1}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
│ └ Otherwise, performance might degrade if they keep running in the background.
│ ┌ Info: Global hostcalls detected!
│ │ - Source: MethodInstance for GPUArrays.gpu_getindex_kernel(::KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicCheck, Nothing, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, KernelAbstractions.NDIteration.NDRange{1, KernelAbstractions.NDIteration.DynamicSize, KernelAbstractions.NDIteration.DynamicSize, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}, CartesianIndices{1, Tuple{Base.OneTo{Int64}}}}}, ::AMDGPU.Device.ROCDeviceVector{ComplexF64, 1}, ::AMDGPU.Device.ROCDeviceMatrix{ComplexF64, 1}, ::Tuple{Int64, Int64}, ::Base.Slice{Base.OneTo{Int64}}, ::Int64)
│ │ - Hostcalls: [:malloc_hostcall, :malloc_hostcall]
│ │
│ │ Use `AMDGPU.synchronize(; stop_hostcalls=true)` to synchronize and stop them.
└ └ Otherwise, performance might degrade if they keep running in the background.

Output generated during execution of 'hip_dnn/conv':
┌ MIOpen(HIP): Warning [ParseAndLoadDb] File is unreadable: "/home/evetion/.julia/artifacts/41050555b43ef6fdfd8c0984835c37aa659595b2/share/miopen/db/gfx1030_40.HIP.fdb.txt"MIOpen(HIP): Warning [OpenRuntimeLibraryForDevice] CK grouped conv library not found for device gfx1030: libMIOpenCKGroupedConv_gfx1030.so: cannot open shared object file: No such file or directory
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ warning: xnack 'Off' was requested for a processor that does not support it!
│ MIOpen(HIP): Warning [IsEnoughWorkspace] [EvaluateInvokers] Solver <GemmFwdRest>, workspace required: 4032, provided ptr: 0 size: 0
└ MIOpen Error: localhost.localdomain:/__w/rockrel/rockrel/rocm-libraries/projects/miopen/src/convolution.cpp:249: Channels do not match for the filter

Test Summary:                                    |  Pass  Fail  Broken  Total      Time
  Overall                                        | 16857     1      16  16874  11m07.9s
    core/codegen                                 |     3     1              4     27.8s
      Synchronization                            |     1                    1     24.5s
      Trapping                                   |     2                    2      1.7s
      Hardware FP atomics                        |           1              1      1.4s
    gpuarrays/indexing find                      |    67                   67   1m56.4s
    gpuarrays/linalg/diagonal                    |    82                   82   2m22.5s
    gpuarrays/ext/jld2                           |   108                  108   2m42.6s
    gpuarrays/math/power                         |    72                   72   2m17.8s
    gpuarrays/reductions/mapreducedim!_large     |    50                   50   3m12.4s
    hip_rocarray/fft                             |   261                  261   3m13.5s
    device/wavefront                             |    83             9     92   3m18.2s
    gpuarrays/constructors                       |   992                  992   3m36.7s
    gpuarrays/statistics                         |    78                   78   3m58.0s
    hip_rocarray/sorting                         |   384                  384   4m12.9s
    gpuarrays/reductions/== isequal              |   290                  290   4m16.2s
    hip_rocsparse/broadcast                      |    65                   65   2m14.7s
    gpuarrays/indexing multidimensional          |   101                  101   4m38.0s
    hip_rocsparse/generic                        |   596                  596   2m22.8s
    gpuarrays/base                               |   151                  151   2m10.6s
    hip_rocsparse/interfaces                     |   625                  625   5m04.3s
    kernelabstractions_tests                     |  2236             6   2242   2m13.6s
    core/graph_tests                             |    12                   12     11.6s
    gpuarrays/indexing reshaped wrappers         |    42                   42   1m58.2s
    gpuarrays/linalg/mul!/strided-views          |    36                   36   1m20.1s
    gpuarrays/random                             |    62                   62   1m42.7s
    device/random                                |   312                  312   2m27.0s
    hip_rocsparse/linalg                         |   168                  168   1m39.7s
    hip_rocarray/reverse                         |    88                   88     30.3s
    gpuarrays/reductions/and or xor              |    99                   99   1m40.6s
    core/rocarray_broadcast                      |    23                   23     53.4s
    external/forwarddiff                         |    18                   18     37.1s
    hip_rocarray/random                          |   141                  141     41.0s
    gpuarrays/linalg/mul!/vector-matrix          |   168                  168   2m58.6s
    core/rocarray_base                           |    74                   74     52.2s
    gpuarrays/linalg/NaN_false                   |    54                   54   1m21.9s
    hip_rocsparse/preconditioners                |   312                  312   1m57.2s
    hip_rocsparse/conversions                    |    34                   34     15.6s
    core/core_tests                              |    27                   27     15.5s
    gpuarrays/linalg/mul!/mixed-eltype           |    60                   60   1m47.1s
    gpuarrays/reductions/any all count           |   101                  101     40.2s
    hip_rocsparse/sparse_matrices_csr            |    53                   53     25.7s
    gpuarrays/indexing scalar                    |   477                  477   1m19.0s
    gpuarrays/linalg/kron_diagonal               |    16                   16     15.5s
    gpuarrays/linalg/mul!/integer-accumulate     |     6                    6     21.9s
    device/math                                  |    27                   27     35.4s
    gpuarrays/math/intrinsics                    |    12                   12     16.2s
    hip_dnn/batchnorm                            |    12                   12     18.8s
    device/launch                                |    11             1     12     12.1s
    hip_rocarray/indexing                        |     3                    3     16.0s
    device/exceptions                            |     6                    6      8.7s
    gpuarrays/uniformscaling                     |    56                   56     39.2s
    device/memory                                |     7                    7     12.5s
    gpuarrays/alloc cache                        |    37                   37      1.9s
    device/synchronization                       |   176                  176      7.3s
    hip_rocsparse/device                         |    10                   10      7.5s
    device/execution_control                     |     5                    5      4.5s
    device/array                                 |    22                   22      5.9s
    core/discovery_tests                         |    14                   14      1.2s
    hip_dnn/descriptors                          |   125                  125     10.8s
    device/indexing                              |     2                    2      5.9s
    device/vadd                                  |     1                    1      4.0s
    core/tls                                     |    13                   13      2.9s
    hip_core_tests                               |     4                    4      2.1s
    wmma_rdna4_tests                             |                          0      0.1s
    gpuarrays/vectors                            |    10                   10      0.8s
    hip_rocarray/multi_gpu                       |                          0      0.2s
    device/regression                            |     3                    3     20.9s
    wmma_rdna3_tests                             |                          0      0.6s
    gpuarrays/sparse                             |                          0      0.0s
    gpuarrays/linalg/mul!/matrix-matrix          |   444                  444   7m08.5s
    hip_rocarray/solver                          |  1089                 1089   7m08.6s
    hip_dnn/activations                          |    10                   10     23.2s
    hip_dnn/conv                                 |    27                   27     41.6s
    gpuarrays/linalg/kron                        |   192                  192   7m18.2s
    hip_dnn/pool                                 |                          0     23.4s
    gpuarrays/reductions/mapreduce               |   900                  900   7m29.3s
    gpuarrays/reductions/sum prod                |   526                  526   8m08.2s
    gpuarrays/reductions/mapreducedim!           |   576                  576   8m17.0s
    hip_rocarray/blas                            |  1040                 1040   8m34.8s
    gpuarrays/reductions/minimum maximum extrema |   414                  414   8m53.6s
    gpuarrays/linalg/core                        |  1120                 1120   8m55.7s
    gpuarrays/linalg/norm                        |   876                  876   9m43.6s
    gpuarrays/broadcasting                       |   460                  460  10m44.9s
    FAILURE

Error in testset Hardware FP atomics:
Test Failed at /home/evetion/.julia/dev/AMDGPU/test/core/codegen.jl:47
  Expression: occursin("global_atomic_add_$(fp)", String(take!(iob)))
   Evaluated: occursin("global_atomic_add_f32", "\t.amdgcn_target \"amdgcn-amd-amdhsa--gfx1030\"\n\t.amdhsa_code_object_version 6\n\t.text\n\t.globl\t_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE ; -- Begin function _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE\n\t.p2align\t8\n\t.type\t_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE,@function\n_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE: ; @_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE\n; %bb.0:                                ; %conversion\n\ts_add_u32 s12, s12, s17\n\ts_addc_u32 s13, s13, 0\n\ts_setreg_b32 hwreg(HW_REG_FLAT_SCR_LO), s12\n\ts_setreg_b32 hwreg(HW_REG_FLAT_SCR_HI), s13\n\ts_mov_b64 s[22:23], s[2:3]\n\ts_mov_b64 s[20:21], s[0:1]\n\ts_load_dwordx2 s[0:1], s[8:9], 0x28\n\ts_add_u32 s20, s20, s17\n\ts_addc_u32 s21, s21, 0\n\ts_waitcnt lgkmcnt(0)\n\ts_cmp_lg_u64 s[0:1], 0\n\ts_cbranch_scc0 .LBB0_6\n; %bb.1:                                ; %L14\n\ts_mov_b32 s4, exec_lo\n\ts_mov_b32 s2, 0\n\tv_mbcnt_lo_u32_b32 v0, s4, 0\n\ts_mov_b32 s3, exec_lo\n\tv_cmpx_eq_u32_e32 0, v0\n\ts_cbranch_execz .LBB0_4\n; %bb.2:\n\ts_load_dwordx2 s[0:1], s[8:9], 0x30\n\ts_bcnt1_i32_b32 s4, s4\n\tv_mov_b32_e32 v3, 0\n\tv_cvt_f32_ubyte0_e32 v2, s4\n\ts_mov_b32 s4, 0\n\ts_waitcnt lgkmcnt(0)\n\ts_load_dword s5, s[0:1], 0x0\n\ts_waitcnt lgkmcnt(0)\n\tv_mov_b32_e32 v1, s5\n.LBB0_3:                                ; %atomicrmw.start\n                                        ; =>This Inner Loop Header: Depth=1\n\tv_add_f32_e32 v0, v1, v2\n\tglobal_atomic_cmpswap v0, v3, v[0:1], s[0:1] glc\n\ts_waitcnt vmcnt(0)\n\tbuffer_gl1_inv\n\tbuffer_gl0_inv\n\tv_cmp_eq_u32_e32 vcc_lo, v0, v1\n\tv_mov_b32_e32 v1, v0\n\ts_or_b32 s4, vcc_lo, s4\n\ts_andn2_b32 exec_lo, exec_lo, s4\n\ts_cbranch_execnz .LBB0_3\n.LBB0_4:                                ; %Flow3\n\ts_or_b32 exec_lo, exec_lo, s3\n\ts_andn2_b32 vcc_lo, exec_lo, s2\n\ts_cbranch_vccz .LBB0_7\n; %bb.5:                                ; %UnifiedReturnBlock\n\ts_endpgm\n.LBB0_6:\n.LBB0_7:                                ; %L10\n\ts_load_dwordx2 s[0:1], s[8:9], 0x0\n\ts_lshl_b32 s3, s14, 16\n\ts_mov_b32 s2, 0\n\ts_add_i32 s15, s15, 1\n\ts_mov_b64 s[4:5], src_private_base\n\ts_lshl_b32 s4, s16, 16\n\ts_add_i32 s3, s3, 0x10000\n\ts_and_b32 s7, s15, 0xffff\n\ts_mov_b32 s6, s2\n\ts_add_i32 s4, s4, 0x10001\n\ts_or_b64 s[2:3], s[6:7], s[2:3]\n\ts_waitcnt lgkmcnt(0)\n\ts_cmp_eq_u32 s1, s5\n\ts_cselect_b32 s2, -1, 0\n\ts_andn2_b32 vcc_lo, exec_lo, s2\n\ts_mov_b32 s2, -1\n\ts_cbranch_vccz .LBB0_9\n; %bb.8:                                ; %atomicrmw.global\n\tv_mov_b32_e32 v2, 0\n\tv_mov_b32_e32 v5, s1\n\tv_mov_b32_e32 v0, s4\n\tv_mov_b32_e32 v1, s3\n\tv_mov_b32_e32 v4, s0\n\tv_mov_b32_e32 v3, v2\n\ts_mov_b32 s2, 0\n\tflat_atomic_cmpswap_x2 v[4:5], v[0:3]\n\ts_waitcnt lgkmcnt(0)\n\ts_waitcnt_vscnt null, 0x0\n\tbuffer_gl1_inv\n\tbuffer_gl0_inv\n.LBB0_9:                                ; %Flow4\n\ts_andn2_b32 vcc_lo, exec_lo, s2\n\ts_cbranch_vccnz .LBB0_11\n; %bb.10:                               ; %atomicrmw.private\n\ts_cmp_lg_u64 s[0:1], 0\n\ts_cselect_b32 s0, s0, -1\n\tv_mov_b32_e32 v2, s0\n\ts_clause 0x1\n\tbuffer_load_dword v0, v2, s[20:23], 0 offen\n\tbuffer_load_dword v1, v2, s[20:23], 0 offen offset:4\n\ts_waitcnt vmcnt(0)\n\tv_cmp_eq_u64_e32 vcc_lo, 0, v[0:1]\n\tv_cndmask_b32_e64 v0, v0, s4, vcc_lo\n\tv_cndmask_b32_e64 v1, v1, s3, vcc_lo\n\tbuffer_store_dword v0, v2, s[20:23], 0 offen\n\tbuffer_store_dword v1, v2, s[20:23], 0 offen offset:4\n.LBB0_11:                               ; %atomicrmw.phi\n\ts_endpgm\n\t; divergent unreachable\n\ts_endpgm\n\t.section\t.rodata,\"a\",@progbits\n\t.p2align\t6, 0x0\n\t.amdhsa_kernel _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE\n\t\t.amdhsa_group_segment_fixed_size 0\n\t\t.amdhsa_private_segment_fixed_size 0\n\t\t.amdhsa_kernarg_size 320\n\t\t.amdhsa_user_sgpr_count 14\n\t\t.amdhsa_user_sgpr_private_segment_buffer 1\n\t\t.amdhsa_user_sgpr_dispatch_ptr 1\n\t\t.amdhsa_user_sgpr_queue_ptr 1\n\t\t.amdhsa_user_sgpr_kernarg_segment_ptr 1\n\t\t.amdhsa_user_sgpr_dispatch_id 1\n\t\t.amdhsa_user_sgpr_flat_scratch_init 1\n\t\t.amdhsa_user_sgpr_private_segment_size 0\n\t\t.amdhsa_wavefront_size32 1\n\t\t.amdhsa_uses_dynamic_stack 0\n\t\t.amdhsa_system_sgpr_private_segment_wavefront_offset 0\n\t\t.amdhsa_system_sgpr_workgroup_id_x 1\n\t\t.amdhsa_system_sgpr_workgroup_id_y 1\n\t\t.amdhsa_system_sgpr_workgroup_id_z 1\n\t\t.amdhsa_system_sgpr_workgroup_info 0\n\t\t.amdhsa_system_vgpr_workitem_id 2\n\t\t.amdhsa_next_free_vgpr 6\n\t\t.amdhsa_next_free_sgpr 24\n\t\t.amdhsa_reserve_vcc 1\n\t\t.amdhsa_reserve_flat_scratch 1\n\t\t.amdhsa_float_round_mode_32 0\n\t\t.amdhsa_float_round_mode_16_64 0\n\t\t.amdhsa_float_denorm_mode_32 3\n\t\t.amdhsa_float_denorm_mode_16_64 3\n\t\t.amdhsa_dx10_clamp 1\n\t\t.amdhsa_ieee_mode 1\n\t\t.amdhsa_fp16_overflow 0\n\t\t.amdhsa_workgroup_processor_mode 1\n\t\t.amdhsa_memory_ordered 1\n\t\t.amdhsa_forward_progress 1\n\t\t.amdhsa_shared_vgpr_count 0\n\t\t.amdhsa_exception_fp_ieee_invalid_op 0\n\t\t.amdhsa_exception_fp_denorm_src 0\n\t\t.amdhsa_exception_fp_ieee_div_zero 0\n\t\t.amdhsa_exception_fp_ieee_overflow 0\n\t\t.amdhsa_exception_fp_ieee_underflow 0\n\t\t.amdhsa_exception_fp_ieee_inexact 0\n\t\t.amdhsa_exception_int_div_zero 0\n\t.end_amdhsa_kernel\n\t.text\n.Lfunc_end0:\n\t.size\t_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE, .Lfunc_end0-_Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE\n                                        ; -- End function\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.num_vgpr, 6\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.num_agpr, 0\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.numbered_sgpr, 24\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.num_named_barrier, 0\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.private_seg_size, 0\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.uses_vcc, 1\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.uses_flat_scratch, 1\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.has_dyn_sized_stack, 0\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.has_recursion, 0\n\t.set _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.has_indirect_call, 0\n\t.section\t.AMDGPU.csdata,\"\",@progbits\n; Kernel info:\n; codeLenInByte = 424\n; TotalNumSgprs: 26\n; NumVgprs: 6\n; ScratchSize: 0\n; MemoryBound: 0\n; FloatMode: 240\n; IeeeMode: 1\n; LDSByteSize: 0 bytes/workgroup (compile time only)\n; SGPRBlocks: 0\n; VGPRBlocks: 0\n; NumSGPRsForWavesPerEU: 26\n; NumVGPRsForWavesPerEU: 6\n; Occupancy: 16\n; WaveLimiterHint : 0\n; COMPUTE_PGM_RSRC2:SCRATCH_EN: 0\n; COMPUTE_PGM_RSRC2:USER_SGPR: 14\n; COMPUTE_PGM_RSRC2:TRAP_HANDLER: 0\n; COMPUTE_PGM_RSRC2:TGID_X_EN: 1\n; COMPUTE_PGM_RSRC2:TGID_Y_EN: 1\n; COMPUTE_PGM_RSRC2:TGID_Z_EN: 1\n; COMPUTE_PGM_RSRC2:TIDIG_COMP_CNT: 2\n\t.text\n\t.p2alignl 6, 3214868480\n\t.fill 48, 4, 3214868480\n\t.section\t.AMDGPU.gpr_maximums,\"\",@progbits\n\t.set amdgpu.max_num_vgpr, 0\n\t.set amdgpu.max_num_agpr, 0\n\t.set amdgpu.max_num_sgpr, 0\n\t.set amdgpu.max_num_named_barrier, 0\n\t.text\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.ident\t\"clang version 18.1.8 (Fedora 18.1.8-5.fc41)\"\n\t.section\t\".note.GNU-stack\",\"\",@progbits\n\t.amdgpu_metadata\n---\namdhsa.kernels:\n  - .args:\n      - .name:           state\n        .offset:         0\n        .size:           40\n        .value_kind:     by_value\n      - .name:           'x::ROCDeviceArray'\n        .offset:         40\n        .size:           24\n        .value_kind:     by_value\n      - .offset:         64\n        .size:           4\n        .value_kind:     hidden_block_count_x\n      - .offset:         68\n        .size:           4\n        .value_kind:     hidden_block_count_y\n      - .offset:         72\n        .size:           4\n        .value_kind:     hidden_block_count_z\n      - .offset:         76\n        .size:           2\n        .value_kind:     hidden_group_size_x\n      - .offset:         78\n        .size:           2\n        .value_kind:     hidden_group_size_y\n      - .offset:         80\n        .size:           2\n        .value_kind:     hidden_group_size_z\n      - .offset:         82\n        .size:           2\n        .value_kind:     hidden_remainder_x\n      - .offset:         84\n        .size:           2\n        .value_kind:     hidden_remainder_y\n      - .offset:         86\n        .size:           2\n        .value_kind:     hidden_remainder_z\n      - .offset:         104\n        .size:           8\n        .value_kind:     hidden_global_offset_x\n      - .offset:         112\n        .size:           8\n        .value_kind:     hidden_global_offset_y\n      - .offset:         120\n        .size:           8\n        .value_kind:     hidden_global_offset_z\n      - .offset:         128\n        .size:           2\n        .value_kind:     hidden_grid_dims\n      - .offset:         144\n        .size:           8\n        .value_kind:     hidden_hostcall_buffer\n      - .offset:         152\n        .size:           8\n        .value_kind:     hidden_multigrid_sync_arg\n      - .offset:         160\n        .size:           8\n        .value_kind:     hidden_heap_v1\n      - .offset:         168\n        .size:           8\n        .value_kind:     hidden_default_queue\n      - .offset:         176\n        .size:           8\n        .value_kind:     hidden_completion_action\n      - .offset:         264\n        .size:           8\n        .value_kind:     hidden_queue_ptr\n    .group_segment_fixed_size: 0\n    .kernarg_segment_align: 8\n    .kernarg_segment_size: 320\n    .language:       OpenCL C\n    .language_version:\n      - 2\n      - 0\n    .max_flat_workgroup_size: 1024\n    .name:           _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE\n    .private_segment_fixed_size: 0\n    .sgpr_count:     26\n    .sgpr_spill_count: 0\n    .symbol:         _Z14atomic_fp_ker_14ROCDeviceArrayI7Float32Li1ELi1EE.kd\n    .uses_dynamic_stack: false\n    .vgpr_count:     6\n    .vgpr_spill_count: 0\n    .wavefront_size: 32\n    .workgroup_processor_mode: 1\namdhsa.target:   amdgcn-amd-amdhsa--gfx1030\namdhsa.version:\n  - 1\n  - 2\n...\n\n\t.end_amdgpu_metadata\n")

ERROR: LoadError: Test run finished with errors
in expression starting at /home/evetion/.julia/dev/AMDGPU/test/runtests.jl:103
ERROR: Package AMDGPU errored during testing

@simeonschaub

Copy link
Copy Markdown
Member Author

Thanks for the report! These warnings should all be harmless, MIOpen just has quite a verbose log level by default and the tests still pass, so I don't think there's anything for us to do here. Some of these warnings are already tracked in upstream. I'll attach the robot's analysis in case you're curious:

Breakdown 🤖
  1. MIOpen Error: … convolution.cpp:249: Channels do not match for the filter

Not a failure. It comes from the last testset in test/hip_dnn/conv.jl:86-91 ("Check MIOpenException"), which deliberately passes a 1‑channel filter against a 3‑channel input and asserts MIOpenException is thrown. MIOpen logs the
MIOPEN_THROW message to stderr before returning the bad-param status. The test passes; the log line is cosmetic. Could be worth a comment in the test so reporters don't mistake it for a failure.

  1. File is unreadable: …/share/miopen/db/gfx1030_40.HIP.fdb.txt

Upstream/expected. MIOpen ships no system find-db for RDNA parts in the TheRock builds; it JIT-tunes and writes a user db to ~/.config/miopen. An AMD maintainer confirmed this in ROCm/TheRock#5105 (closed): "The Warning [ParseAndLoadDb]
File is unreadable warning is a little misleading here as we don't expect the db files to be there." First-run convs will be slow; subsequent runs use the cached user db.

  1. CK grouped conv library not found for device gfx1030: libMIOpenCKGroupedConv_gfx1030.so

Upstream TheRock packaging/MIOpen. MIOpen 7.x dlopens a per-arch CK grouped-conv library; TheRock doesn't build one for gfx103x (CK's grouped conv needs MFMA/WMMA), so it falls back to GemmFwdRest. Same warning is tracked for other
archs:

No gfx1030-specific issue exists; #7409 is the closest to "please ship/suppress this for archs without CK support".

  1. warning: xnack 'Off' was requested for a processor that does not support it! (×4)

Upstream MIOpen. In src/comgr.cpp, under WORKAROUND_ISSUE_3001, MIOpen appends -mno-xnack when assembling its .s kernels (the 4 repeats = 4 asm kernel builds) whenever the target doesn't have xnack enabled — including gfx1030, which
doesn't support xnack at all, hence LLVM's warning. Tracked in ROCm/rocm-libraries#6812 (open, "MIOpen still sends old/removed -mno-xnack argument to clang"), continuation of ROCm/MIOpen#2851. Harmless for us (it's a warning, not an
error).

  1. [IsEnoughWorkspace] Solver , workspace required: 4032, provided ptr: 0 size: 0

Upstream MIOpen behavior. We do allocate miopenConvolutionForwardGetWorkSpaceSize bytes before Find (src/dnn/convolution.jl:103-104), but with no find-db the size query and the Find-time solver list don't agree, so a solver gets skipped
with this warning. Identical warnings (same solver, ptr 0) are reported from PyTorch in ROCm/rocm-libraries#4071 (open) and ROCm/MIOpen#3957 (open); MIOPEN_FIND_MODE=FAST suppresses them.

Caveat: I couldn't reproduce (no GPU in this sandbox); the analysis is from the MIOpen source in rocm-libraries/develop and the issue trackers.

If you want to quiet this for users, the cheap option is setting MIOPEN_LOG_LEVEL=2 (errors only; default release level is 3 = warnings) in the test runner or documenting it — the xnack line is an LLVM warning and wouldn't be affected
by that, though. I can add a comment to the Check MIOpenException test and/or an env default if you'd like.

@luraess

luraess commented Aug 22, 2026

Copy link
Copy Markdown
Member

I'll need to rework slightly the Bk pipeline now that #1045 landed.

@simeonschaub

Copy link
Copy Markdown
Member Author

I can also do it, I need to add hiptensor to this anyways

simeonschaub and others added 12 commits August 22, 2026 17:31
…ry subpackages

By default the ROCm runtime (HIP, HSA) and vendor libraries (rocBLAS,
rocSPARSE, rocSOLVER, rocRAND, rocFFT, hipTENSOR, MIOpen) now come from the
new ROCm_Runtime subpackage, which ships AMD's TheRock distribution tarballs
directly as lazy artifacts, selected by GPU architecture and ROCm version
through platform augmentation. The former src/discovery/ code moves into the
ROCm_Runtime_Discovery subpackage, which replicates ROCm_Runtime's API for a
local ROCm installation; it is only loaded when that is requested through
ROCm_Runtime's "local" preference, settable via the new
AMDGPU.set_rocm_version!([version]; local_rocm) (mirroring
CUDA.set_runtime_version!).

hipTENSOR is part of the same TheRock tarballs (only for the CDNA bundles,
gfx908/gfx90a/gfx94X/gfx950; TheRock excludes it on RDNA targets), so
libhiptensor is provided by both packages alongside the other libraries.

ld.lld now always comes from AMDGPU_LLVM_Backend_jll (it has to match the
LLVM version device code is generated with), and the device bitcode libraries
keep coming from ROCmDeviceLibs_jll until local ones can be downgraded.
libMIOpen_path is renamed to libMIOpen for consistency with the provider
packages.

Also adds a manually-dispatched workflow to regenerate the Artifacts.toml
hashes from repo.amd.com.

Assisted-by: Claude Code (claude-fable-5)
Both ROCm_Runtime and ROCm_Runtime_Discovery export the same library path
globals, so AMDGPU can simply `using` whichever the "local" preference
selects: the imported bindings alias the provider's globals and pick up the
values assigned by its `__init__`, which runs before AMDGPU loads. AMDGPU's
`libhip`/`libhsaruntime` names are renamed to the providers'
`libamdhip64`/`libhsa_runtime64` so no aliasing layer is needed, and
`lld_path`/`libdevice_libs` are imported directly from their JLLs, which are
assumed to always be available. The discovery package gains libhiprtc and
libamd_comgr lookups for API parity with ROCm_Runtime.

As a side effect the device libraries are now available while AMDGPU itself
precompiles (the JLL's `__init__` runs then), so the warming compile links
them for real; the DEVICE_LIBS reset stays to keep absolute artifact paths
and bitcode blobs out of the package image.

Assisted-by: Claude Code (claude-fable-5)
The subpackages are wired up through [sources], which only the active
project's Project.toml provides and which Pkg supports only on Julia 1.11+.
Steps running on Julia 1.10 (like the GPU-less environment check) or
resolving from another project (docs, perf) therefore failed with
"expected package ROCm_Runtime to be registered". Explicitly Pkg.develop'ing
the subpackages records path entries in the manifest, which works everywhere.

Verified by simulating the CI resolve in a fresh clone on Julia 1.10 and
nightly, including a GPU-less load (arch = "none") on 1.10.

Assisted-by: Claude Code (claude-fable-5)
The previous fix dev'ed the subpackages in the step command, but the
julia-test plugin instantiates the project in its pre-command hook, which
runs before the step's command -- so Julia 1.10 (no [sources] support) still
failed with "expected package ROCm_Runtime to be registered". Follow
CUDA.jl's approach instead: on 1.10 (tests, Enzyme, GPU-less check) drop the
julia-test plugin and run Pkg.develop + Pkg.test manually; on 1.11+ the
plugin works as-is since Pkg honors [sources] there, so those steps go back
to their plain Pkg.update() command.

Verified on a fresh clone with Julia 1.10: develop + update resolves, and
Pkg.test constructs its sandbox with the dev'ed subpackages from the parent
manifest.

Assisted-by: Claude Code (claude-fable-5)
The MI300 job set JULIA_AMDGPU_DISABLE_ARTIFACTS=1 to pick up the uenv's
ROCm. That variable no longer exists after the discovery refactor, so the
job silently resolved to the TheRock artifacts instead, and every test
died with hipErrorOutOfMemory out of hipStreamCreateWithPriority (3167
occurrences; no hipMalloc ever failed). The .rocm-miopen merge and
MIOPEN_PREFIX symlinking the script does were inert as a result.

Select the provider through ROCm_Runtime's "local" preference instead,
set before Pkg.instantiate() since it is a compile-time preference that
also gates artifact resolution, and dev the two workspace packages the
way .buildkite/pipeline.yml now does. The resolved provider is asserted
against the requested one afterwards, so a future change of default
fails loudly instead of quietly swapping the ROCm underneath the job.

The script is parameterised by ROCM_LOCAL so both providers can be run.
Only the uenv variant is enabled: this is the project's only gfx942 CI,
so covering the default artifact path here is worthwhile, but TheRock
ships a ROCm 7.14 userspace that cannot create a HIP stream at all
against the 6.12.12 amdgpu driver on this machine, so that job is left
commented out until the mismatch is resolved.

Verified on beverin (2x MI300A, gfx942, Julia 1.12.6) at c644fc5:

  uenv ROCm, full suite, --jobs=32:  16162 pass, 0 OOM  (was 3167 OOM)
  A/B in one allocation, only the preference differing:
    local=true   HIP 7.2.53211   hipStreamCreateWithPriority OK
    local=false  HIP 7.14.60850  hipErrorOutOfMemory

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment blamed a ROCm 7.14 userspace / 6.12.12 amdgpu driver mismatch.
That was wrong: the driver side is fine (rocminfo enumerates all agents,
hipMalloc succeeds). What actually fails is ROCclr's blit-kernel build on
first stream creation, which cannot resolve __amd_streamOpsIncrement and
__amd_streamOpsDecrement, and is then reported as hipErrorOutOfMemory --
which is what made this look like memory exhaustion for so long.

Reported upstream as ROCm/TheRock#7426, so the
comment now just points there rather than restating the analysis.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants