Context
When launching Apache Spark workloads in containers on Dataproc GPU clusters, the container executor runs under unprivileged user contexts (such as the job submitting user or the system yarn user).
By default, NVIDIA driver installations restrict access to the GPU device nodes in /dev/ (such as /dev/nvidia* and /dev/nvidia-uvm ) to the video and render groups with 0660 permissions.
The Problem
The gpu/install_gpu_driver.sh script in the initialization-actions repository does not configure group mappings to allow YARN Node Manager or job execution users to belong to the video and render groups.
As a result:
- The YARN auto-discovery script
getGpusResources.sh (which executes nvidia-smi as the yarn user) fails with a permission denial when trying to access /dev/nvidia*.
- The YARN Node Manager daemon crashes on startup during node registration, leading to cluster creation timeouts (Error code 13).
- Containerized workloads launched by YARN are blocked from accessing GPU hardware because the container processes run without membership in the host's
video or render groups.
Proposed Fix
Modify the gpu/install_gpu_driver.sh script to configure these group mappings (for example, executing usermod -aG video,render yarn ) during the YARN configuration phase, or ensure they are captured in the generated first-boot deferred configuration script.
Context
When launching Apache Spark workloads in containers on Dataproc GPU clusters, the container executor runs under unprivileged user contexts (such as the job submitting user or the system
yarnuser).By default, NVIDIA driver installations restrict access to the GPU device nodes in
/dev/(such as/dev/nvidia*and/dev/nvidia-uvm) to thevideoandrendergroups with0660permissions.The Problem
The
gpu/install_gpu_driver.shscript in theinitialization-actionsrepository does not configure group mappings to allow YARN Node Manager or job execution users to belong to thevideoandrendergroups.As a result:
getGpusResources.sh(which executesnvidia-smias theyarnuser) fails with a permission denial when trying to access/dev/nvidia*.videoorrendergroups.Proposed Fix
Modify the
gpu/install_gpu_driver.shscript to configure these group mappings (for example, executingusermod -aG video,render yarn) during the YARN configuration phase, or ensure they are captured in the generated first-boot deferred configuration script.