Skip to content

nvidia-drm: return -ENOSYS when PRIME sg_table is unavailable - #1317

Open
VaggelisGian wants to merge 1 commit into
NVIDIA:mainfrom
VaggelisGian:fix-prime-sgt-vidmem-errno
Open

nvidia-drm: return -ENOSYS when PRIME sg_table is unavailable#1317
VaggelisGian wants to merge 1 commit into
NVIDIA:mainfrom
VaggelisGian:fix-prime-sgt-vidmem-errno

Conversation

@VaggelisGian

Copy link
Copy Markdown

Fixes the misleading error reported in #1302.

Problem

__nv_drm_gem_nvkms_memory_prime_get_sg_table() returns ERR_PTR(-ENOMEM) for NvKmsKapiMemory backed by video memory (pages_count == 0, no struct pages). The value propagates verbatim through dma_buf_map_attachment() to the importing driver: a cross-device PRIME import of such an object reports an out-of-memory condition that never happened. As reported in #1302, kwin_wayland treats this as real memory exhaustion and aborts instead of falling back or reporting an unsupported operation.

Change

One line: return ERR_PTR(-ENOSYS) instead of ERR_PTR(-ENOMEM) in that case.

-ENOSYS is what DRM core itself uses when a GEM object has no sg_table implementation:

  • v5.11 commit d693def4fd1c ("drm: Remove obsolete GEM and PRIME callbacks from struct drm_driver"): WARN_ON(!obj->funcs->get_sg_table); return ERR_PTR(-ENOSYS);
  • v6.4 commit 207395da5a97 ("drm/prime: reject DMA-BUF attach when get_sg_table is missing"): attach rejected with -ENOSYS.

The errno is the only change:

  • No caller under kernel-open/nvidia-drm branches on the old value; DRM core forwards PTR_ERR unchanged.
  • Real allocation failures on this path still return -ENOMEM from their own sites.
  • pages_count == 0 occurs only for vidmem objects: sysmem objects whose page list cannot be built fail at object init (__nv_drm_nvkms_gem_obj_init).

Test Plan

Prepared on Windows without Linux kernel toolchain or NVIDIA GPU, so no build and no runtime test were possible; verification was static:

git diff --stat
 kernel-open/nvidia-drm/nvidia-drm-gem-nvkms-memory.c | 2 +-

git diff --check        # clean

Traced all four constructors of nvkms-memory GEM objects (dumb_create, import_nvkms_memory_ioctl, alloc_nvkms_memory_ioctl, prime_dup) to confirm pages_count == 0 implies vidmem; grepped all callers of nv_drm_gem_prime_get_sg_table for errno-specific branching (none). A hardware test on any multi-GPU system importing a vidmem-backed nvidia-drm buffer from another driver would exercise the changed path.

Vidmem-backed NvKmsKapiMemory has no struct pages, so
__nv_drm_gem_nvkms_memory_prime_get_sg_table() can never build an
sg_table for it. It returned ERR_PTR(-ENOMEM), which propagates
unchanged through dma_buf_map_attachment() to the importing driver,
so a cross-device PRIME import of a vidmem GEM object reports an
out-of-memory condition that never happened. On multi-GPU systems
this makes compositors treat an unsupported export as memory
exhaustion and abort (NVIDIA#1302).

Return -ENOSYS instead, matching the value drm_prime.c uses when a
GEM object has no get_sg_table implementation. The errno is the only
change: no caller in nvidia-drm switches on the old value, and real
allocation failures on this path still return -ENOMEM from their
own sites.

Test Plan:
  No build or runtime test possible in the preparation environment
  (no Linux kernel toolchain, no NVIDIA GPU). Verified statically:
  git diff --stat : 1 file changed, 1 insertion(+), 1 deletion(-)
  pages_count == 0 occurs only for vidmem-backed objects; sysmem
  objects whose page list cannot be built fail at object init.
  No caller under kernel-open/nvidia-drm branches on the errno
  value; DRM core forwards PTR_ERR unchanged to the importer.
@CLAassistant

CLAassistant commented Aug 24, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants