Add true HDR output, RT shadows, and PCSS shadows for vulkan

[The-E]

Note that this is based on @notimaginative's PR (#7553) and should only be merged afterwards.
This also adapts the work done in @BMagnu's Shadow Overhaul PR (#7529), which is therefore another path dependency.

Summary

Adds optional HDR10 (PQ / ST.2084 + BT.2020) swap chain output to the Vulkan renderer, with paper-white / peak-luminance controls and an in-game calibration screen. When disabled (the default) or when running on a non-HDR display / the OpenGL renderer, behavior is unchanged.

This PR has since grown to also bring the Vulkan renderer's shadow rendering up to par with the shadow-rendering overhaul merged in from lafiel/shadow_overhaul_2 (dynamic cascade counts, PCF/PCSS, cockpit shadow cascades, and an optional raytraced-shadow path), since both landed on this branch together. Unlike the HDR work, the shadow overhaul itself affects both the OpenGL and Vulkan backends.

What's included

Swap chain & metadata (VulkanRendererSetup.cpp)

• Enables VK_EXT_swapchain_colorspace (instance) and VK_EXT_hdr_metadata (device, when available).
• When HDR is requested, negotiates an A2B10G10R10 swap chain with the HDR10_ST2084 color space, falling back cleanly to SDR/sRGB otherwise.
• Submits VkHdrMetadataEXT (paper white / peak luminance, BT.2020 primaries) and re-applies it on swap chain recreation (resize / fullscreen toggle).

Frame composition refactor (VulkanRenderer.cpp, VulkanRendererLoop.cpp)

• Introduces an intermediate RGBA16F composition buffer at window resolution. All rendering (scene, UI, ImGui) now targets this buffer.
• A final encode pass converts the composition buffer to the swap chain: SDR passthrough when HDR is off, HDR10 PQ encode when on.

Tonemap / output shaders (tonemapping-f.sdr, gamma.sdr)

• New hdr_mode uniform: 0 = existing SDR tonemap, 1 = HDR scene tonemap (exposure + headroom clamp relative to paper white, stored as extended sRGB in the fp16 composition buffer), 2 = HDR10 output encode (linearize, scale to nits, BT.709 -> BT.2020, PQ encode).
• LDR intermediate targets (Scene_ldr, Scene_luminance) widen to fp16 when HDR is active so highlights above paper white survive.

Post-process anti-aliasing (VulkanPostProcessingSMAA.cpp, VulkanPostProcessingLDR.cpp, smaa-*.sdr, fxaapre-f.sdr)

• Ported SMAA to Vulkan. It previously had no Vulkan implementation at all — Graphics.AAMode listed the SMAA presets unconditionally, but selecting one while running Vulkan silently produced zero anti-aliasing with no warning. All three passes (edge detection, blending-weight calculation, neighborhood blending) are now implemented, plus the area/search lookup texture uploads; the shared SMAA.sdr algorithm body needed no changes, only thin per-backend wrapper shaders.
• FXAA and SMAA both now work correctly while HDR10 output is active, instead of being unconditionally disabled there. Their fixed luma thresholds assume [0,1] LDR input, which breaks once Scene_ldr carries extended-range values; fixed by tonemapping a second, properly-compressed proxy buffer for edge/luma detection only, while the actual blended output still reads the real extended-range colors — so anti-aliasing works without sacrificing HDR headroom.

Options (2d.cpp / 2d.h)

• Graphics.HDR (bool, requires restart), Graphics.HDRPaperWhite (nits, live), Graphics.HDRPeakLuminance (nits, live), persisted via the options system.
• Gr_hdr_output_active reflects whether the renderer actually negotiated an HDR10 swap chain (distinct from the request flag).

Calibration screen (ingame_options_ui.cpp / .h)

• New "HDR Calibration" entry in the SCP Options menu.
• Status banner (active / enabled-but-inactive / disabled), live paper-white and peak-luminance sliders bound to the persistent options, a paper-white-relative grayscale ramp, and R/G/B/white primary patches for sanity-checking the BT.709 -> BT.2020 mapping.

OpenGL (gropenglpostprocessing.cpp)

• New tonemap UBO fields are explicitly zero-initialized so the OpenGL backend's behavior is unchanged.

HDR framebuffer readback (1c3f8b7f9)

• Adds an fp16 -> BGRA8 conversion path for framebuffer readback, so screenshots and other systems that assume 8-bit output keep working when HDR display output is enabled.

Vulkan raytraced shadows — TLAS/BLAS (63f8fa059)

• New VulkanRaytracingManager: BLAS caching keyed to model lifecycle, per-frame TLAS build/rebuild (VulkanRaytracingTlas.cpp, VulkanRaytracingBlas.cpp) with dynamic capacity growth.
• Vulkan buffer/device-address plumbing needed for acceleration structures (VulkanBuffer, VulkanMemory).
• Descriptor manager support for binding the TLAS and RT-shadow resources, plus a capability query for raytracing support (gr_vulkan.cpp).

Shadow rendering overhaul, OpenGL side (merged from shadow_overhaul_2)

• Extracted shadow rendering into its own shader/render pass and consolidated render-queue code — model_draw_list and the new shadow_render_list now share a common render_queue<Derived, DrawEntryT> CRTP base (code/graphics/render_queue.h).
• Switched from VSM to hardware PCF + PCSS with proper depth textures; smoothness exposed as a mod-table option.
• Shadow cascade count is now dynamic (mod-table configurable) instead of a hardcoded 4, and cockpit geometry gets its own dedicated shadow cascades instead of fighting with the main scene's.
• Assorted correctness fixes: depth-clamped shadow geometry to avoid artifacts, FOV multiplier fix, shadow cascade override handling, inter-frame shadow buffer clearing.

Vulkan shadow parity port (f35864fda)

• shader_get_shadow_cascade_defines() shared between both backends' shader compilers so cascade-count defines stay in sync.
• New VulkanDrawManager::renderShadowDraw(): builds the SDR_TYPE_SHADOW_MAP_GEN pipeline, binds ShadowMapData/ShadowCascadeParams UBOs, and does a single instanced draw across all active cascades, replacing the old "spawn N instances" hack.
• VulkanShadowMap rewritten: dropped the VSM color target entirely, dynamic layer count (Num_shadow_cascades + Num_cockpit_shadow_cascades), and a depth-compare sampler for hardware PCF via sampler2DArrayShadow.
• New descriptor bindings (GlobalBinding::ShadowCascadeParams, MaterialBinding::ShadowMapData), and VulkanDeletionQueue::queueAccelerationStructure() so TLAS rebuilds defer destruction of the old acceleration structure instead of destroying it synchronously out from under an in-flight command buffer.
• Shaders unified across both backends (single transformToShadowMap(), matching sampler2DArrayShadow usage, matching vertex/fragment interface blocks, shared shadowCascadeParams UBO layout).
• Fixed a startup crash (shadow_cascade_params_init() needed to run eagerly, not lazily, since shadows_start_render() indexes Shadow_frustums before the lazy init point is reached) and a validation error from building the shadow TLAS inside an active render pass.
• Fixed a Vulkan-only regression where destroyed/blown-off submodels (e.g. a ship's turret rotator after its subsystem is destroyed) stayed visible, stuck in place with an identity transform, instead of being hidden — the shader computed the "clip this submodel" flag but the code that acted on it was accidentally wrapped in #ifndef VULKAN.

Logging

• Vulkan logging calls unified on mprintf/scoped nprintf categories for consistency with the rest of the codebase, plus a new in-game log filter UI for toggling categories and browsing session logs.

Fixes picked up along the way

• Fixed a Vulkan startup crash: gr_uniform_buffer_managers_init() was only ever called from the OpenGL init path, so UniformBufferManager stayed null under Vulkan and the first draw needing a uniform buffer (title screen rendering) dereferenced it (SIGSEGV).
• Fixed all four Mac CI configs failing with ccache: No such file or directory — the workflow hardcodes the Intel Homebrew ccache path (/usr/local/bin/ccache), which doesn't exist on Apple Silicon-hosted GitHub runners (Homebrew installs under /opt/homebrew there). configure_cmake.sh now falls back to a PATH-resolved ccache when the configured path isn't executable.

Notes / limitations

• The HDR10 output path itself is Vulkan only; OpenGL is intentionally untouched there. The shadow-rendering overhaul affects both backends, with this PR bringing Vulkan up to parity with the OpenGL-side changes.
• HDR requires a restart to take effect (swap chain format negotiation).
• ImGui patches cannot exceed paper white, so peak luminance is best tuned in-game by raising it until bright highlights/explosions stop clipping.

[BMagnu]

Check out SDR_FLAG_TONEMAPPING_LINEAR_OUT.
It's basically poor-man's HDR output in the OpenGL pass, currently only used for the OpenXR pass, since the headset swapchain seems to do its own tonemap, or at the very least expects non-SDR input (it's highly possible that proper HDR input is in fact the correct thing to forward here, instead of truly linear data).
This should very likely be merged into this work, even if it means supporting OpenGL to some degree, rather than exist as a weird, second, half-baked HDR pass. Especially since proper handling across not only gameplay but also menus will fix #7181.
I'm not yet fully familiar with most of the Vulkan PR, but I assume that before this change, there is no handling of Cmdline_window_res, which this new buffer could then do in the future as well? I also assume that this PR is going to be the proper place to mirror #7484 for the SDL3 upgrade. If so, we should at least prepare / design this in a way that makes it easy to retrofit.

[notimaginative]

Gave this a try on my Mac. Most of my known Vulkan bugs appear to be gone (tex flicker, MSAA, fog, loadout flicker). The texture corruption in massive_battle2 looks to be substantially reduced, if not entirely eliminated.

New issues that I noticed:

Shadows do not appear to work correctly. Or at least not like they did before. I'm not seeing/noticing self-shadowing in the techroom or lab. I see shadows in-mission, but they can move or completely vanish depending on my view direction even if nothing else in the mission is moving. I cleaned out the cache just to be sure but there was no visual change.

HDR works but certain graphical elements flicker (like text and highlighted ui controls), and there is some easily producible yet kind of random graphics corruption. The corruption was visible on some ui screens but not others. In the techroom when viewing the Ulysses there is corruption, but not when viewing the Herc. I have not yet played a mission with HDR enabled.

[BMagnu]

Another known bug, for the record:
The Orbradar is basically a disco ball at the moment

[The-E]

Gave this a try on my Mac. Most of my known Vulkan bugs appear to be gone (tex flicker, MSAA, fog, loadout flicker). The texture corruption in massive_battle2 looks to be substantially reduced, if not entirely eliminated.

New issues that I noticed:

Shadows do not appear to work correctly. Or at least not like they did before. I'm not seeing/noticing self-shadowing in the techroom or lab. I see shadows in-mission, but they can move or completely vanish depending on my view direction even if nothing else in the mission is moving. I cleaned out the cache just to be sure but there was no visual change.

HDR works but certain graphical elements flicker (like text and highlighted ui controls), and there is some easily producible yet kind of random graphics corruption. The corruption was visible on some ui screens but not others. In the techroom when viewing the Ulysses there is corruption, but not when viewing the Herc. I have not yet played a mission with HDR enabled.

can you post logs (with the vulkan log filter enabled) and screenshots of this?

[notimaginative]

Here are logs, screenshots, and a vid. I'm not sure if the screenshot or vid will actually show the HDR, but in the game it's very noticeable and nice looking on my MBP screen.

For the log I made sure to enable "Vulkan" and "vulkanstate", launched the game, went to the techroom, switch the model from Ulysses to Hec, went back to mainhall and exited. The pilot select screen had ui flickering and corruption, the mainhall had ui flickering but no real corruption, the Ulysses had both, the Herc had ui flicker only.

vulkan-hdr.log

I had trouble with the screenshots as one frame appears to have corruption but show none of the ui elements that flicker, or show the ui elements but not the corruption. So most of the screenshots, particularly from the techroom, never showed any corruption. As such I'm just including a single screenshot that demonstrated the corruption as well as a shot of my in-game settings.

The video is just enough to show the various issues. However while recording the video the Herc actually showed corruption, so I just cut it before the switch.

vulkan-hdr.mp4

I should probably also note that I'm unable to use fullscreen here. I tried, to see if there was any difference, but it always crashes. On the Mac it must do a mode switch to work, but that was disabled in the Vulkan PR. Mode switching should work better when SDL3 hits though so we can probably re-enable it.

The only things that jumped out at me during a cursory review are fixed. I'm going to leave actual approval to BMagnu however, since I'm not qualified to properly review the bulk of the changes.