Skip to content

use device libs from AMDGPU_LLVM_Backend_jll (2) - #1044

Open
simeonschaub wants to merge 11 commits into
mainfrom
sds/downgrade_devlibs
Open

use device libs from AMDGPU_LLVM_Backend_jll (2)#1044
simeonschaub wants to merge 11 commits into
mainfrom
sds/downgrade_devlibs

Conversation

@simeonschaub

Copy link
Copy Markdown
Member

We downgrade these into a scratch space to be compatible with Julia's LLVM version

ref #919 (comment)

Reopened to run MI300 CI

The device libraries now ship with AMDGPU_LLVM_Backend_jll, built with a
newer LLVM than Julia's. Downgrade their bitcode once with LLVMDowngrader_jll
into a Scratch.jl-managed directory (keyed by the artifact tree hashes and
the downgrade target), and drop the ROCmDeviceLibs_jll dependency.

Assisted-by: Claude Code (claude-fable-5)
Julia 1.10 does not support opaque pointers, which our 15 target emits
(LLVM 15 supported both opaque and typed). Alternative is we use typed
pointers for 15 as a default as well
The 15 bitcode target emits opaque pointers, which is why it was disabled
for Julia 1.10 (LLVM 15, typed pointers). Julia 1.11 ships LLVM 16, which
supports opaque pointers, so use the 15 target there and keep 14 for 1.10.

A fresh LLVM 16 context still defaults to typed pointers, so `hipcompile`
now asks for an opaque-pointer context on LLVM 16, otherwise the
downgraded bitcode fails to load ("Opaque pointers are only supported in
-opaque-pointers mode"). GPUCompiler and our own IR generation already
handle both modes.

The 15 target is also the more faithful one: it can downgrade every
device lib (the 14 target rejects asanrtl.bc, which made us fall back to
the system libs) and needs far fewer synthesized pointer casts.

Require LLVMDowngrader_jll 0.9.1, which fixes the typed signatures of
the AMDGPU pointer intrinsics; 0.9.0 produced modules that fail the
verifier.

Assisted-by: Claude Code (claude-fable-5)
This partially reverts 87fc233. Julia 1.11's own codegen uses opaque
pointers, but GPUCompiler still uses typed pointers on LLVM 16, and it
creates contexts for reflection (`code_llvm`, `code_native`, the
`@device_code_*` macros) itself, so the opaque device libs produced by
the 15 target could only be loaded there with a new GPUCompiler hook.
That is not worth it: with LLVMDowngrader_jll 0.9.1 the typed output of
the 14 target passes the verifier for all device libs, so just use it on
Julia 1.11 as well.

Keep requiring LLVMDowngrader_jll 0.9.1, since 0.9.0 mistyped the AMDGPU
pointer intrinsics.

Assisted-by: Claude Code (claude-fable-5)

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMDGPU.jl Benchmarks

Details
Benchmark suite Current: e093932 Previous: 58dd394 Ratio
amdgpu/synchronization/context/device 637.5 ns 540 ns 1.18
amdgpu/synchronization/stream/blocking 262.5 ns 227.5 ns 1.15
amdgpu/synchronization/stream/nonblocking 370 ns 310 ns 1.19
array/accumulate/Float32/1d 73528.5 ns 78001 ns 0.94
array/accumulate/Float32/dims=1 279739 ns 275066.5 ns 1.02
array/accumulate/Float32/dims=1L 81218.5 ns 82516.25 ns 0.98
array/accumulate/Float32/dims=2 80671 ns 81193.75 ns 0.99
array/accumulate/Float32/dims=2L 2764466.25 ns 2758054.75 ns 1.00
array/accumulate/Int64/1d 79143.5 ns 79148.5 ns 1.00
array/accumulate/Int64/dims=1 246286 ns 243553.5 ns 1.01
array/accumulate/Int64/dims=1L 84186.25 ns 84101.25 ns 1.00
array/accumulate/Int64/dims=2 86363.75 ns 86991.25 ns 0.99
array/accumulate/Int64/dims=2L 2893263 ns 2895916.75 ns 1.00
array/broadcast 73411 ns 74006 ns 0.99
array/construct 2327.5 ns 2335 ns 1.00
array/copy 36795.5 ns 37160.75 ns 0.99
array/copyto!/cpu_to_gpu 111546.5 ns 111339.25 ns 1.00
array/copyto!/gpu_to_cpu 111924 ns 111596.5 ns 1.00
array/copyto!/gpu_to_gpu 48595.75 ns 59505.75 ns 0.82
array/iteration/findall/bool 140069.5 ns 139364.75 ns 1.01
array/iteration/findall/int 151072.25 ns 152744.75 ns 0.99
array/iteration/findfirst/bool 146529.75 ns 145527.25 ns 1.01
array/iteration/findfirst/int 145699.5 ns 145997.25 ns 1.00
array/iteration/findmin/1d 121736.75 ns 124581.75 ns 0.98
array/iteration/findmin/2d 110106.5 ns 98116.25 ns 1.12
array/iteration/logical 252573.5 ns 252929 ns 1.00
array/iteration/scalar 300054.25 ns 309789.75 ns 0.97
array/permutedims/2d 71786 ns 72443.5 ns 0.99
array/permutedims/3d 71283.5 ns 71971 ns 0.99
array/permutedims/4d 72348.5 ns 75061 ns 0.96
array/random/rand/Float32 45133.25 ns 45963 ns 0.98
array/random/rand/Int64 54388.25 ns 54770.75 ns 0.99
array/random/rand!/Float32 65518.5 ns 44300.75 ns 1.48
array/random/rand!/Int64 73301 ns 73566 ns 1.00
array/random/randn/Float32 73111 ns 76658.75 ns 0.95
array/random/randn!/Float32 80838.5 ns 81353.5 ns 0.99
array/reductions/mapreduce/Float32/1d 100979 ns 101831.5 ns 0.99
array/reductions/mapreduce/Float32/dims=1 89686.25 ns 91791.25 ns 0.98
array/reductions/mapreduce/Float32/dims=1L 832701.75 ns 834629.75 ns 1.00
array/reductions/mapreduce/Float32/dims=2 87598.75 ns 88776.5 ns 0.99
array/reductions/mapreduce/Float32/dims=2L 144267 ns 144237 ns 1.00
array/reductions/mapreduce/Int64/1d 100358.75 ns 101501.5 ns 0.99
array/reductions/mapreduce/Int64/dims=1 89498.75 ns 90426.25 ns 0.99
array/reductions/mapreduce/Int64/dims=1L 830816.75 ns 834257.25 ns 1.00
array/reductions/mapreduce/Int64/dims=2 86561.25 ns 88551.5 ns 0.98
array/reductions/mapreduce/Int64/dims=2L 145214.5 ns 145772.25 ns 1.00
array/reductions/reduce/Float32/1d 99724 ns 102909 ns 0.97
array/reductions/reduce/Float32/dims=1 89713.75 ns 91134 ns 0.98
array/reductions/reduce/Float32/dims=1L 832726.75 ns 835889.75 ns 1.00
array/reductions/reduce/Float32/dims=2 87591.25 ns 88548.75 ns 0.99
array/reductions/reduce/Float32/dims=2L 144577.25 ns 145259.5 ns 1.00
array/reductions/reduce/Int64/1d 100024 ns 101701.5 ns 0.98
array/reductions/reduce/Int64/dims=1 88136.25 ns 88971.5 ns 0.99
array/reductions/reduce/Int64/dims=1L 831411.75 ns 831467.25 ns 1.00
array/reductions/reduce/Int64/dims=2 87443.75 ns 88508.75 ns 0.99
array/reductions/reduce/Int64/dims=2L 144917.25 ns 144032.25 ns 1.01
array/reverse/1d 45500.5 ns 46513.25 ns 0.98
array/reverse/1dL 73598.5 ns 73918.5 ns 1.00
array/reverse/1dL_inplace 79738.5 ns 81293.75 ns 0.98
array/reverse/1d_inplace 52068.25 ns 53618.5 ns 0.97
array/reverse/2d 50195.75 ns 51231 ns 0.98
array/reverse/2dL 84441.25 ns 84231.25 ns 1.00
array/reverse/2dL_inplace 69383.5 ns 92779 ns 0.75
array/reverse/2d_inplace 63363.25 ns 42753 ns 1.48
array/sorting/1d 333994.5 ns 334009.75 ns 1.00
integration/byval/reference 39871 ns 39821 ns 1.00
integration/byval/slices=1 40820 ns 40321 ns 1.01
integration/byval/slices=2 140692 ns 147452 ns 0.95
integration/byval/slices=3 237083 ns 240194 ns 0.99
integration/volumerhs 5011010 ns 5007121 ns 1.00
kernel/indexing 48793.25 ns 58288.5 ns 0.84
kernel/indexing_checked 48423.25 ns 50148.25 ns 0.97
kernel/launch 1480 ns 1445 ns 1.02
kernel/rand 100961.5 ns 101319 ns 1.00
latency/import 1693628312 ns 1689718608 ns 1.00
latency/precompile 39192093618 ns 38551162270 ns 1.02
latency/ttfp 2314342244 ns 2308017436 ns 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@simeonschaub
simeonschaub requested a review from luraess August 22, 2026 08:00
@simeonschaub

Copy link
Copy Markdown
Member Author

OK, I think this should be good to go now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant