Use fuser instead of lsof command in run-command-shim to prevent run commands hanging due to lsof command hanging

[Vivek Lingaiah (viveklingaiah)]

Problem(s):

• lsof command hanging during execution of run-command-shim for 3P customers Chevron, CapGemini. This hang has caused timeouts with Run Command v1 Linux (even before executing the user's script)
  ICM incidents mentioned here Task 34627514: Investigate RCv2 Linux NOT using lsof command OR using alternatives (lsof command hanging)
• "lsof command not found" error due to its package not installed

Background:

• lsof command is being used currently in shim file of extensions (Run command v1, v2, custom script) to prevent issues caused by race conditions
• lsof command in shim is used to "list open files by the run command binary run-command-handler". If there are any open files already, we do not run the script, we wait and retry today.
• lsof is performance-intensive (lists processes, examines all file descriptors and symbolic links. It can get stuck if there are issues examining file descriptors on an unavailable/unresponsive mount, Network file system , DNS timeouts on name resolution of IPs). This can cause hang issues on high load servers of our customers

Solution

• Use fuser command instead to detect any conflicts like "text file busy" error before executing RC (do not use lsof)
• fuser command also outputs what has the run command binary open in write mode
• If fuser command not available on VM, it executes the script anyways as best-effort execution.

Sample execution from log:

Simulation of "Text file busy" error (RC binary is open in write mode)
sleeping for 3 seconds before retry, attempt 9 of 10
bash has access mode 'aw' file handle on run-command-handler
sleep has access mode 'aw' file handle on run-command-handler
waiting for process(es) with write handle on run-command-handler
sleeping for 3 seconds before retry, attempt 10 of 10

• nohup /var/lib/waagent/Microsoft.CPlat.Core.RunCommandHandlerLinux-1.3.28/bin/run-command-handler enable
  nohup: failed to run command '/var/lib/waagent/Microsoft.CPlat.Core.RunCommandHandlerLinux-1.3.28/bin/run-command-handler': Text file busy
  [stderr]

Successful case (RC executed fine )
2026-08-04T03:14:15.300891Z INFO ExtHandler [Microsoft.CPlat.Core.RunCommandHandlerLinux.RC0803_4-1.3.28] Command: bin/run-command-shim enable
[stdout]
t.CPlat.Core.RunCommandHandlerLinux-1.3.28/bin/run-command-handler enable
time=2026-08-04T03:14:13Z version=v/git@- operation=enable event=start
time=2026-08-04T03:14:13Z version=v/git@- operation=enable message="processing command"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable message="getting required initial variables"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4
time=2026-08-04T03:14:13Z version=v/git@- operation=enable seq=0 seqNum=0
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event=pre-check
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="comparing seqnum" path=RC0803_4.mrseq
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="seqnum saved" path=RC0803_4.mrseq
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 message="processing command for extensionName: RC0803_4 and seqNum: 0"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 message="creating json to report status"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 message="reporting status by writing status file locally"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 message="Run Command status was written to file successfully."
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="reading configuration from /var/lib/waagent/Microsoft.CPlat.Core.RunCommandHandlerLinux-1.3.28/config/RC0803_4.0.settings"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="read configuration"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="parsing configuration json"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="parsed configuration json"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="validating configuration logically"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="validated configuration"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 message="extension policy settings file does not exist. No policy applied." error="stat /var/lib/waagent/Microsoft.CPlat.Core.RunCommandHandlerLinux-1.3.28/config/waagent_runtime_policy.json: no such file or directory"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="creating output directory" path=/var/lib/waagent/run-command-handler/download/RC0803_4/0
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="created output directory"
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 scriptUri=
time=2026-08-04T03:14:13Z version=v/git@- operation=enable extensionName=RC0803_4 event="executing command" output=/var/lib/waagent/run-command-handler/download/RC0803_4/0
time=2026-

Already addressed.