policy: close memoized policy filesystems

[crazy-max]

This fixes policy filesystem cleanup so a policy filesystem returned by newPolicyPathFS releases the backend roots it acquired during one logical use.

On Windows, the old implementation can leave the policy context directory open after loadPolicyData succeeds. This test reproduces the failure:

func TestLoadPolicyDataReleasesPolicyDir(t *testing.T) {
	dir := t.TempDir()
	require.NoError(t, os.WriteFile(filepath.Join(dir, "policy.rego"), []byte("package docker\n"), 0600))

	provider := newPolicyPathFS(context.Background(), nil, policyOpt{
		ContextDir: dir,
	})

	_, ok, err := loadPolicyData(provider, "policy.rego")
	require.NoError(t, err)
	require.True(t, ok)
	require.NoError(t, os.RemoveAll(dir))
}

The old implementation fails because Windows cannot remove the directory while the policy root is still open.

--- FAIL: TestLoadPolicyDataReleasesPolicyDir (1.79s)
    policy_loader_test.go:106:
        	Error Trace:	...
        	Error:      	Received unexpected error:
        	            	unlinkat ...: The process cannot access the file because it is being used by another process.
        	Test:       	TestLoadPolicyDataReleasesPolicyDir
FAIL

newPolicyPathFS now returns a per-use wrapper around the shared policy path filesystem. That wrapper lazily acquires the cwd and context backends at most once, records the matching release functions, and closes only the backends acquired by that wrapper.

loadPolicyData calls Stat and then ReadFile, and both operations can resolve through the memoized backend filesystem. The old close path only released one memoized reference for the whole logical policy filesystem use, so the backend root could remain open.

[tonistiigi]

I don't get this. The previous function does reference counting. The new one just closes directly, so should be unsafe for multiple users. I also don't see what is left using the previous function.

[crazy-max]

I don't get this. The previous function does reference counting. The new one just closes directly, so should be unsafe for multiple users. I also don't see what is left using the previous function.

Yes this repro was not the right because it asserted an internal close callback and fails on Linux too. The actual user-visible failure is Windows-specific because the leaked policy root keeps the policy directory open.

Here is a new repro:

func TestLoadPolicyDataReleasesPolicyDir(t *testing.T) {
	dir := t.TempDir()
	require.NoError(t, os.WriteFile(filepath.Join(dir, "policy.rego"), []byte("package docker\n"), 0600))

	provider := newPolicyPathFS(context.Background(), nil, policyOpt{
		ContextDir: dir,
	})

	_, ok, err := loadPolicyData(provider, "policy.rego")
	require.NoError(t, err)
	require.True(t, ok)
	require.NoError(t, os.RemoveAll(dir))
}

On Windows without this PR, it fails with this output:

--- FAIL: TestLoadPolicyDataReleasesPolicyDir (1.79s)
    policy_loader_test.go:106:
        	Error Trace:	...
        	Error:      	Received unexpected error:
        	            	unlinkat ...: The process cannot access the file because it is being used by another process.
        	Test:       	TestLoadPolicyDataReleasesPolicyDir
FAIL

The path is loadPolicyData, then root.Stat, then fs.ReadFile. Both calls resolve through policyPathFS, so the memoized context filesystem is acquired more than once during a single logical use of the policy filesystem. The old policyPathFS.Close only released one memoized reference, which leaves the directory-backed root open on Windows.

But your point about closeAll being too broad as a general replacement for the ref-counted close is right. I will rework the implementation to preserve the ref-count abstraction more explicitly.