fix(stages): recover if /var/lib/cloud/instance is a directory

[maburlik]

Proposed Commit Message

fix(stages): recover if /var/lib/cloud/instance is a directory

/var/lib/cloud/instance is only ever supposed to be a symlink into
instances/<iid>, or absent. Several places in Init unconditionally
call util.del_file() on it -- a plain os.unlink() -- which raises an
uncaught IsADirectoryError if the path is ever found to be a real
directory instead, aborting the boot stage before any further
modules can run.

This exact symptom has recurred more than once (GH-3710 /
LP:#1883903, fixed narrowly for one call site in 20.3; GH-4282, an
unresolved recurrence closed for lack of a reproduction). Rather
than chasing every possible writer that could promote this path,
this hardens every call site in Init that consumes the "symlink or
absent" invariant (purge_cache(), _get_data_source(), and
_reflect_cur_instance()) through one shared helper: if the path is
found to be a directory, log a warning and remove it recursively so
it can be recreated, instead of crashing.

The removal is scoped defensively: it never touches a mount point
(falls back to the original, non-destructive del_file() for that one
case instead of recursing into it), and shutil.rmtree() itself already
guarantees it won't follow a top-level symlink or any symlink nested
inside the directory being removed into its target.

Fixes GH-6979

Additional Context

Filed as GH-6979 first, with the full symptom writeup (traceback,
status.json, and a structural observation about when in the boot the
directory-promotion must occur, since init-local's own
_reflect_cur_instance() call always succeeds moments before the
crashing one). This PR is the candidate fix offered at the end of that
issue.

While investigating, I found the same "unguarded write_file() under
the not-yet-created instance_link" anti-pattern that caused GH-3710
still exists today in cc_snap.py's snapd.assertions write (it never
got the ensure_dir_exists=False fix cc_final_message received in
20.3). Not fixed here since it's an independent, narrower issue and I
have no evidence it's actually implicated in GH-6979 specifically --
flagging it here in case a maintainer wants a separate small PR for it,
or wants it folded into this one instead.

This PR intentionally does not identify or fix whatever causes the
directory promotion in the first place -- seeing that mechanism was
already effectively out of scope for GH-4282's own multi-year, never-
resolved attempt at the same question. It makes cloud-init resilient
to the symptom regardless of cause, at the one place in the codebase
that can make that guarantee for the whole bug class.

Safety review of the removal itself: the new helper uses
shutil.rmtree() (via util.del_dir()) rather than os.unlink() for
the directory case, so I specifically checked it can't be tricked into
deleting more than intended. Confirmed independently that rmtree()
already refuses to operate on a top-level symlink and never follows
symlinks nested inside the directory being removed into their
targets (only the symlink entries themselves are unlinked) -- both
pinned with regression tests now. Also found and fixed one real gap:
rmtree() has no filesystem-boundary awareness, so if
instance_link were ever a mount point rather than an ordinary
directory, it would have recursed into and deleted the mounted
filesystem's contents before failing on the mount point itself -- a
strictly larger blast radius than the crash this PR replaces. Added an
os.path.ismount() guard so that case falls through to the original
del_file() behavior instead (the same IsADirectoryError as before,
for that one case, rather than deleting anything).

Test Steps

Added regression tests in tests/unittests/test_stages.py
(TestInit_ReflectCurInstance) that:

• Reproduce the exact reported traceback against the unpatched code
  (verified locally: both _reflect_cur_instance() and
  purge_cache(rm_instance_lnk=True) fail with IsADirectoryError: [Errno 21] Is a directory: '.../instance' without this change,
  matching the field report byte for byte) and confirm the fix heals
  them.
• Cover the shared helper (_remove_stale_instance_link) directly and
  in isolation: real-directory (heals + warns), valid symlink including
  a dangling one (silent, unchanged behavior), and absent (no-op).
• Cover both _get_data_source() call sites via lighter "is it wired to
  the tested helper" tests (mocking the datasource-discovery machinery
  those call sites depend on, rather than re-deriving full filesystem
  behavior that's already covered above).
• Add baseline coverage for the normal cases, none of which had any
  direct test coverage before this change.
• Pin the two safety properties above: a nested symlink inside the
  healed directory survives untouched, and a (mocked) mount point at
  instance_link is never recursed into.

Run with:

tox -e py3 -- tests/unittests/test_stages.py -v -k ReflectCurInstance

Full suite comparison against an unmodified main: 5754 passed with
this change vs. 5742 passed on baseline -- exactly the 12 tests added
here, zero regressions. black, isort, and pylint (this repo's
currently pinned versions) all clean; pylint 10.00/10.

Merge type

• Squash merge using "Proposed Commit Message"
• Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)