Respect VmallocTotal from /proc/meminfo as virtual address space limit

[janvorli]

While .NET runtime and GC takes into account the virtual memory limit extracted from the RLIMIT_AS, there is another limiting factor on Linux that it doesn't consider - the VmallocTotal in /proc/meminfo. This value exposes a kernel compile time constant and we've got a report of a problem when it was mere 256GB.

Add minipal_get_virtual_address_space_limit() in src/native/minipal/vmlimit.c that computes the effective virtual address space limit by taking the minimum of RLIMIT_AS and VmallocTotal from /proc/meminfo. The result is cached since these values don't change during process lifetime.

All existing RLIMIT_AS call sites in coreclr (GC, PAL, minipal doublemapping) now use this single consolidated function instead of duplicating the getrlimit logic.

Close #85556

[azure-pipelines]

Azure Pipelines:
Successfully started running 4 pipeline(s).
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR centralizes virtual address space limit detection on Unix by introducing a minipal helper that computes an effective limit as min(RLIMIT_AS, VmallocTotal) (Linux) and updates CoreCLR call sites (GC, PAL, minipal double-mapping) to use it instead of duplicating getrlimit logic.

Changes:

• Add minipal_get_virtual_address_space_limit() in src/native/minipal and cache the computed value.
• Update CoreCLR consumers (GC, PAL virtual memory, double-mapping) to use the consolidated minipal helper.
• Wire the new minipal source file into the native build.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/native/minipal/vmlimit.h	Declares new minipal helper for effective virtual address space limit.
src/native/minipal/vmlimit.c	Implements limit computation (RLIMIT_AS + Linux VmallocTotal) with caching.
src/native/minipal/CMakeLists.txt	Adds vmlimit.c to minipal sources on Unix hosts.
src/coreclr/pal/src/map/virtual.cpp	Switches executable memory reservation sizing to use minipal VM limit helper.
src/coreclr/minipal/Unix/doublemapping.cpp	Clips double-mapped memory sizing using minipal VM limit helper.
src/coreclr/gc/unix/gcenv.unix.cpp	Uses minipal VM limit helper for GC VM limit and virtual load computation.
src/coreclr/gc/unix/cgroup.cpp	Uses minipal VM limit helper when computing restricted physical memory limit.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/map/virtual.cpp:1642

• initialReserveLimit is an int32_t, but it’s being assigned from a size_t computation (virtualAddressSpaceLimit * percent / 100). When the VM limit is large (e.g. 256GB VmallocTotal), this can overflow to a negative value and then be used as an allocation size later (see line 1720), potentially turning into a huge reservation after integer conversion.

    size_t virtualAddressSpaceLimit = minipal_get_virtual_address_space_limit();
    if (virtualAddressSpaceLimit != SIZE_MAX)
    {
        // By default reserve max 20% of the available virtual address space
        size_t initialExecMemoryPerc = 20;
        CLRConfigNoCache defInitialExecMemoryPerc = CLRConfigNoCache::Get("InitialExecMemoryPercent", /*noprefix*/ false, &getenv);
        if (defInitialExecMemoryPerc.IsSet())
        {
            DWORD perc;
            if (defInitialExecMemoryPerc.TryAsInteger(16, perc))
            {
                initialExecMemoryPerc = perc;
            }
        }

        initialReserveLimit = virtualAddressSpaceLimit * initialExecMemoryPerc / 100;
        if (initialReserveLimit < sizeOfAllocation)
        {
            sizeOfAllocation = initialReserveLimit;
        }

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/map/virtual.cpp:1642

• initialReserveLimit is an int32_t, but it's currently assigned from a size_t calculation (virtualAddressSpaceLimit * percent / 100). With the new VmallocTotal-based limit, this can be a large finite value (e.g., hundreds of GB) and overflow/truncate when stored into int32_t, which can then flow into ReserveVirtualMemory as a negative/garbage size. Since this logic is only intended to reduce the default reservation, only set initialReserveLimit (and shrink sizeOfAllocation) when the computed value is smaller than the current sizeOfAllocation, and compute in size_t to avoid overflow.

    size_t virtualAddressSpaceLimit = minipal_get_virtual_address_space_limit();
    if (virtualAddressSpaceLimit != SIZE_MAX)
    {
        // By default reserve max 20% of the available virtual address space
        size_t initialExecMemoryPerc = 20;
        CLRConfigNoCache defInitialExecMemoryPerc = CLRConfigNoCache::Get("InitialExecMemoryPercent", /*noprefix*/ false, &getenv);
        if (defInitialExecMemoryPerc.IsSet())
        {
            DWORD perc;
            if (defInitialExecMemoryPerc.TryAsInteger(16, perc))
            {
                initialExecMemoryPerc = perc;
            }
        }

        initialReserveLimit = virtualAddressSpaceLimit * initialExecMemoryPerc / 100;
        if (initialReserveLimit < sizeOfAllocation)
        {
            sizeOfAllocation = initialReserveLimit;
        }

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/map/virtual.cpp:1643

• initialReserveLimit is int32_t, but it's assigned from virtualAddressSpaceLimit * initialExecMemoryPerc / 100 (a size_t). If the VM limit is large (e.g. VmallocTotal=256GB) this overflows/truncates to a negative int32_t, and later sizeOfAllocation = initialReserveLimit can pass a negative size into ReserveVirtualMemory. Only compute/store initialReserveLimit when the computed limit is actually smaller than the current sizeOfAllocation (and therefore fits in int32_t).

    size_t virtualAddressSpaceLimit = minipal_get_virtual_address_space_limit();
    if (virtualAddressSpaceLimit != SIZE_MAX)
    {
        // By default reserve max 20% of the available virtual address space
        size_t initialExecMemoryPerc = 20;
        CLRConfigNoCache defInitialExecMemoryPerc = CLRConfigNoCache::Get("InitialExecMemoryPercent", /*noprefix*/ false, &getenv);
        if (defInitialExecMemoryPerc.IsSet())
        {
            DWORD perc;
            if (defInitialExecMemoryPerc.TryAsInteger(16, perc))
            {
                initialExecMemoryPerc = perc;
            }
        }

        initialReserveLimit = virtualAddressSpaceLimit * initialExecMemoryPerc / 100;
        if (initialReserveLimit < sizeOfAllocation)
        {
            sizeOfAllocation = initialReserveLimit;
        }
    }

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

[MichalPetryka]

volatile is not correct here, in c it doesnt have any threading guarantees.

[janvorli]

That is not for threading guarantees, but for compiler to understand that this value can change in a way that the compiler doesn't see. We use this pattern at multiple places.

[MichalPetryka]

this value can change in a way that the compiler doesn't see

I don't see how that'd be supposed to happen here?

Even if this doesn't care about threading, the possibility of multiple threads accessing it at once is itself UB in C:

The execution of a program contains a data race if it contains two conflicting actions in different threads, at least one of which is not atomic, and neither happens before the other. Any such data race results in undefined behavior.

where normal or volatile accesses are not guaranteed atomic.

It'd also trigger TSAN errors, not sure if @jkoritzinsky plans to add that to sanitized pipelines at some point.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

[am11]

Suggested change

	#if !defined(__APPLE__) && !defined(__HAIKU__)
	#include "minipalconfig.h"
	#if defined(TARGET_LINUX)

[am11]

Suggested change

	#endif // !defined(__APPLE__) && !defined(__HAIKU__)
	#endif // defined(TARGET_LINUX)

[am11]

Suggested change

	#if !defined(__APPLE__) && !defined(__HAIKU__)
	#if defined(TARGET_LINUX)