feature: add ublk-based block device frontend (overlaybd-ublk) - #429
Open
haolianglh wants to merge 10 commits into
Open
feature: add ublk-based block device frontend (overlaybd-ublk)#429haolianglh wants to merge 10 commits into
haolianglh wants to merge 10 commits into
Conversation
The existing TCMU frontend requires a single resident daemon for all devices on a host: netlink only notifies the registered handler process, so one crash takes down every overlaybd disk, and the code carries several netlink workarounds as the price of that architecture. Introduce overlaybd-ublk, an alternative frontend based on the kernel's io_uring-backed ublk driver, with no SCSI overhead and no single point of failure: one process serves exactly one device, so device create/delete collapses into process start/stop and failures are isolated to a single disk. * add/del/list CLI following ublk ecosystem conventions; add returns only after the device is usable and prints the device path * per-queue event loop driven by photon: ring fd polled via epoll, CQEs reaped with the non-blocking libublksrv API, IO served by coroutines calling ImageFile directly * IO mapping mirrors the TCMU frontend (READ/WRITE/FLUSH/DISCARD); writable devices honestly advertise a volatile cache, read-only devices do not * works on kernels with partial io_uring feature sets: queue setup retries without IORING_SETUP_COOP_TASKRUN, device deletion prefers DEL_DEV_ASYNC and falls back to a bounded sync del * libublksrv v1.7 and liburing 2.8 are fetched at configure time and statically linked (dual MIT/LGPL, used under MIT); building needs autotools, opt out with -DBUILD_UBLK_FRONTEND=off * verified end-to-end on ublk-capable kernels: startup sequence, idle wakeup, full-queue fio pressure with crc32c verify, FLUSH path, add/del lifecycle, and read-only semantics; 13 unit tests cover the IO mapping and CLI layers Signed-off-by: haolianglh <haolianglh@sina.com>
With one process per device, daemons sharing one cache directory can race: the file cache's eviction (truncate+unlink) and refill locking is in-process only, so concurrent daemons risk accounting drift, cache thrashing, and a small window of serving stale zeroes when eviction truncates a file another daemon is reading. Introduce `add --cache-dir <dir>`, giving each device its own registry and gzip cache directories. The override is applied by pointing create_image_service at a patched temporary copy of the service config, leaving the shared image_service code untouched. Document that multiple devices must use separate cache directories. Signed-off-by: haolianglh <haolianglh@sina.com>
The file cache's eviction and refill locking is in-process only, so
two daemons sharing a cache directory can corrupt space accounting,
thrash each other's evictions, and in a small window serve stale
zeroes. With one process per device, the unsafe configuration must
not be reachable by default.
Give every device its own cache directory, always:
<base>/<image-key>/<instance>/{registry_cache,gzip_cache}
* <base> defaults to /opt/overlaybd/ublk_cache; --cache-dir overrides
the base only
* <image-key> is a hash of the image config's realpath, so remounting
an image reuses its warm cache and distinct images never collide;
a "source" file records the origin path
* <instance> is claimed via flock (inst0, inst1, ...): mounting the
same read-only image repeatedly is a legitimate pattern and gets
automatically numbered instances, or a caller-pinned name via
--instance-id
* writable images (a config with an effective upper) are restricted
to a single exclusive mount: concurrent RW mounts would corrupt the
upper layer itself, not just the cache
Also fold the six file-scope globals of ublk_device.cpp into a
UblkDevice class (libublksrv callbacks reach it through the ctrl-dev
priv-data slot), removing the last single-device assumption from the
code in preparation for a possible multi-device daemon mode.
Unit tests cover the config patching, image key derivation, writable
detection and the CLI additions (19 cases); runtime verified on a
ublk-capable kernel: automatic slots, writable-mount rejection,
--instance-id, warm directory reuse, and the full add/IO/del
lifecycle.
Signed-off-by: haolianglh <haolianglh@sina.com>
Add a daemon that hosts all ublk devices in one process and shares a single ImageService, complementing the one-process-per-device CLI. The control plane is HTTP over a root-only unix socket (docker.sock style, curl-debuggable): POST /v1/add and /v1/del, GET /v1/list and /v1/ping, POST /v1/shutdown. add replies only when the device is usable; del replies after the device is fully torn down; SIGTERM or shutdown stops all devices in parallel (STOP first, then join) so total teardown time stays close to the slowest single device. To host several devices per process, split UblkDevice::run() into start()/wait()/stop()/teardown() with a second construction path that injects a shared ImageService, and derive the ImageService registration tag from pid plus a sequence number (a bare pid tag collides on the second add). Also fix a latent fd-0 bug the daemon exposed: on kernels without IORING_SETUP_COOP_TASKRUN, the old init-then-retry flow made every queue creation fail once, and libublksrv's failure path closes fd 0 (ublksrv_queue_deinit runs on a zero-initialized queue whose epollfd/efd are 0 behind >= 0 guards). With one process per device the victim was stdin and the retried ring landed on fd 0, working by luck; in the daemon, fd 0 was the previous device's queue ring, so every add silently wedged the device added before it. Probe the flag once at startup with a private throwaway ring instead, so the failing path is never reached; the daemon additionally parks fd 0 on /dev/null as a guard. Protocol parsing/encoding is a standalone photon-free module with unit tests (23 cases in total). Verified on a ublk-capable kernel: mixed RO/RW devices, cross-device regression probes, concurrent dual-device fio with data verification, del, and parallel shutdown. Signed-off-by: haolianglh <haolianglh@sina.com>
Land the safety items of the daemon milestone M2 (ADR-0006):
* The daemon now patches the service config to a private cache tree
<base>/daemon/{registry_cache,gzip_cache} guarded by an exclusive
flock (--cache-dir overrides the base; the service config's cacheDir
is deliberately ignored so the default never collides with tcmu's
cache directory). A second daemon on the same base is refused.
* Writable images are excluded across processes: the daemon takes the
per-image inst0 lock file (lock-only) before opening the upper, so
the daemon and single-device CLI processes cannot double-mount one
RW image; the lock is released when the device is deleted.
* The CLI `del -n N` now refuses devices owned by overlaybd-ublkd and
prints the equivalent curl command: the pidfile of such a device
points at the daemon, so the old SIGTERM semantics tore down ALL
daemon devices at once (found the hard way). `list` tags these
devices with [ublkd-managed].
* Concurrent control requests are made safe: a per-image in-flight
guard rejects duplicate adds, and a stopping state machine rejects
concurrent deletes of one device, which previously raced on the
device-table iterator across coroutine yields.
* Failure reasons of a daemonized CLI add now travel to the console
over the ready pipe (the child has no stderr and the log is not yet
redirected during early setup), and the patched service config copy
stays alive until teardown -- deleting it early silently dropped the
global default download section from image configs.
Verified on a ublk-capable kernel: dual-daemon refusal, bidirectional
RW mount exclusion, RO cross-mounting, concurrent add/del races, del
rejection with a live daemon, and full teardown via SIGTERM.
Signed-off-by: haolianglh <haolianglh@sina.com>
Add POST /v1/resize for growing a writable image online: the image layer goes through ImageFile::resize() in-process (optionally growing the ext4 inside), then the kernel is told the new capacity with UBLK_U_CMD_UPDATE_SIZE. libublksrv v1.7 has no wrapper for that command and its generic sender is private, so the uring_cmd is sent on a private throwaway SQE128 ring against /dev/ublk-control, assembled the same way as the library does. Support is negotiated: UBLK_F_UPDATE_SIZE is probed once via GET_FEATURES and requested at ADD_DEV, since the kernel only accepts UPDATE_SIZE on devices created with the flag. Only growing is allowed and only for writable images; kernels without the feature get a clear 501. The error paths (feature missing, read-only device, unknown device, bad parameters) are runtime verified; the success path needs a mainline >= 6.11 kernel and is covered by the pending mainline regression. Also ship a systemd unit template for the daemon (generous TimeoutStopSec: deletions may take seconds per device on kernels without DEL_DEV_ASYNC, and killing a flushing daemon risks a writeback deadlock) and document the daemon in the README. Signed-off-by: haolianglh <haolianglh@sina.com>
Rework `del` to drive the kernel directly instead of signaling the owner process. STOP_DEV/DEL_DEV are per-dev_id control commands that any root process may send through /dev/ublk-control, so deletion no longer depends on a pidfile and works on orphans whose owner was killed -9. GET_DEV_INFO identifies the owner as the kernel sees it; a live single-device owner is stopped externally and drains without self-deleting (teardown skips DEL when the stop was external), then del issues DEL. Devices owned by overlaybd-ublkd are still refused with a pointer to the daemon API. This also cures the writeback-deadlock window of the old path: del no longer relies on the owner tearing itself down under SIGTERM, and a normal del now takes ~0.2s instead of ~2.2s (it no longer hits the bounded self-DEL fallback). `list` now takes the kernel as source of truth: it scans /sys/block and /dev/ublkc for devices, reports each as running / [ublkd-managed] / ORPHAN, and additionally flags stopped-but-not-deleted residuals (this kernel auto-stops a device when its owner dies but does not free the id) and stale pidfiles. Such residuals are reclaimable with the same `del`, retiring the old rmmod workaround. Add `del --force` for an unresponsive owner: it reads the owner pid from the pidfile (never from the kernel -- once STOP_DEV was sent to a device whose owner is frozen, every further control command blocks forever, so --force must not touch the control plane before the owner is gone), SIGCONTs then SIGKILLs it, and only then reclaims the device. Without --force a wedged owner is reported and refused rather than risking a machine hang on backported kernels. Verified on a ublk-capable kernel: orphan reclamation (single-device and daemon), residual listing and cleanup, id reuse after reclaim, --force on a frozen owner, live-daemon refusal, and that the daemon's own del and SIGTERM teardown paths are unchanged. Signed-off-by: haolianglh <haolianglh@sina.com>
Add POST /v1/acquire and /v1/release to the daemon, a lease-style
counterpart to the ownership-style add/del. Acquiring the same
read-only image in shared mode returns the same device with an
incremented refcount instead of creating a new one, so N consumers of
a golden image share a single ublk device (and a single ImageFile and
cache) rather than N. Release decrements the refcount and tears the
device down only when it reaches zero.
`acquire {"mode":"exclusive"}` always creates a fresh device; it is
equivalent to add today and is the hook for a future warm pool, so
the protocol will not change when that lands. Shared mode is refused
for writable images (concurrent writers would corrupt the upper) and
the half-created device is rolled back. Cross-verb guards: del refuses
a shared device and points at release; release on an exclusive device
is allowed (equivalent to del).
list now reports mode and refcount per device. Protocol parsing and
encoding are covered by unit tests (26 total); runtime verified on a
ublk-capable kernel: shared reuse and refcounting, release teardown at
zero, writable rejection with rollback, exclusive/shared orthogonality,
cross-verb guards, and shutdown cleanup of shared devices.
Signed-off-by: haolianglh <haolianglh@sina.com>
Serve `acquire {"mode":"exclusive"}` from a pool of pre-created
devices when enabled, hot-swapping the requested image into a device
that is already up instead of paying the ~8ms ADD_DEV + queue init +
START_DEV. Pooling is off by default and needs --pool-low with
--pool-size-gb; --pool-high caps the idle set.
Hot-swapping is done by the queue thread itself at a moment when no
IO is in flight: it is the only reader of the dispatch target and runs
on a single photon vcpu, so no atomics or reader/writer coordination
are needed. A control coroutine posts the request and waits with a
bound. It also has to wake the queue loop, which otherwise sleeps up
to a second on the ring fd -- that made the first version of the swap
take a second, i.e. slower than creating a device. Cross-thread
signalling now goes through an eventfd and the interrupt happens
inside the queue thread's own vcpu; a swap takes ~28us.
Pooled devices are parked on placeholder images the daemon builds
itself (sparse LSMT layer plus a minimal config, one set of files per
pooled device -- sharing one would let two ImageFiles open the same
layer read-write and corrupt it). A device may only serve an image
whose virtual size, block size and writability match, since the
kernel fixes those at creation time; anything else falls back to
creating a device.
Recycling a device requires care our reference point does not need:
the device stays LIVE across tenants, so the kernel never drops its
page cache. Release therefore flushes and invalidates the block
device (BLKFLSBUF) before parking it, and refuses to recycle a device
that is still mounted (it is torn down instead).
Verified on a ublk-capable kernel: prewarming to the watermark, pool
hits with correct data after the swap, successful cache invalidation
on recycle, refusal to recycle a mounted device, fallback on
mismatch, unchanged behaviour with pooling off, and clean shutdown of
pooled devices. Note that the measured end-to-end acquire time is
dominated by opening the image, which pooling does not address.
Signed-off-by: haolianglh <haolianglh@sina.com>
Sharing one device between all consumers of a read-only image changes
the expectation that one acquire yields one device, and a caller that
forgets to release keeps that device alive for everyone. Make it an
explicit choice: --enable-shared-devices turns it on, and it is off by
default, matching how the warm pool is gated.
With the flag off, `acquire {"mode":"shared"}` is not an error -- it
degrades to an independent device per caller, exactly what add would
give -- and the response reports mode "exclusive". A caller therefore
needs no deployment-specific logic, yet can still tell from the
response whether sharing took effect. Everything downstream (the
shared map, refcounting, the read-only check, the cross-verb guards)
keys off one boolean, so there is no path where the flag is bypassed.
The daemon logs the setting at startup and logs each degraded acquire.
Verified on a ublk-capable kernel: with the flag off, two shared
acquires of one read-only image return two different devices reporting
exclusive mode; with it on, the previous shared behaviour is unchanged.
Signed-off-by: haolianglh <haolianglh@sina.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR adds a new block device frontend
overlaybd-ublk, which exposes an overlaybd image (config.v1.json) as/dev/ublkbNvia the kernel's ublk driver (io_uring based), as an alternative to the existing TCMU/SCSI path. The TCMU frontend is not touched by this PR.Motivation
TCMU forces a single resident daemon (netlink only notifies the registered handler process): one crash takes down every overlaybd disk on the host, and the codebase carries several netlink workarounds (SO_RCVBUFFORCE, NETLINK_NO_ENOBUFS, block_netlink restart protection) as the price of that architecture. ublk has no such constraint and no SCSI protocol overhead, and is the de-facto successor to TCMU for userspace block devices.
Design
One process per device: process lifetime == device lifetime. Device create/delete collapses into process start/stop, and a failure is isolated to a single disk. CLI follows the ublk ecosystem conventions:
overlaybd-ublk add --config /path/config.v1.json # prints /dev/ublkbN when ready
overlaybd-ublk list
overlaybd-ublk del -n 0
add daemonizes and only returns success after the image is open and the block device is usable; the device path on stdout is the sync-ready contract for future snapshotter integration. Each device can write to its own log file via add --log-path ... (recommended when running multiple devices; daemons otherwise share the global log file and are tagged ublk- for disambiguation).
Single-thread hybrid event loop: the queue thread runs a photon environment and owns the loop -- the io_uring ring fd is polled via photon epoll, CQEs are reaped with the non-blocking public API ublksrv_queue_reap_events(), and IO runs in photon coroutines calling ImageFile directly (ublksrv_complete_io() is always followed by an explicit io_uring_submit(), since commit SQEs are queued but not submitted by libublksrv). On teardown the loop drains through ublksrv_process_io() until -ENODEV, matching the official daemons' exit semantics, so no io command is ever left inflight in the kernel. The IO dispatch layer is decoupled from the event loop and unit-testable in isolation.
IO mapping mirrors the TCMU frontend: READ -> preadv (retry-forever), WRITE -> pwritev (EROFS special case), FLUSH -> fdatasync, DISCARD/WRITE_ZEROES -> fallocate(PUNCH_HOLE|KEEP_SIZE). Writable devices honestly advertise UBLK_ATTR_VOLATILE_CACHE so the kernel issues FLUSH as needed; read-only devices advertise no cache (nothing volatile to flush).
Kernel compatibility: the frontend runs on any kernel exposing /dev/ublk-control, including ones where ublk was backported with a partial io_uring feature set:
queue setup retries without IORING_SETUP_COOP_TASKRUN (a pure optimization, mainline 5.19+) when the kernel rejects it; device deletion prefers UBLK_CMD_DEL_DEV_ASYNC (kernel 6.5+) and falls back to a sync del bounded at 2s; if the kernel still holds device references at that point the daemon exits and process teardown lets the kernel complete the deletion, so del never hangs and dev ids are never leaked.
Testing
Full build passes with the option on and off; the off state produces no new targets and introduces no new build requirements. 13 unit tests (ublk_dispatch_test) cover the op-mapping layer (read/write roundtrip, EIO/EROFS/short-read paths, flush, discard punch hole, unknown op) and CLI parsing -- no kernel or photon dependency. Verified end-to-end on kernels with the ublk driver available: device startup sequence, first IO after long idle (no lost wakeups), full-queue fio pressure (4k randread iodepth 128 x 4 jobs; randwrite + crc32c verify with zero errors; 256k sequential mixed), FLUSH path via fsync, full add/del lifecycle (bounded del, clean process exit, dev id reuse), and read-only semantics (getro=1, raw writes rejected, committed layer mounts ro and reads back written data).
Known limitations / follow-ups
v1 scope: single queue, no zero-copy, no UBLK_F_USER_RECOVERY (process death == device removal, by design), no snapshotter integration. End-to-end DISCARD depends on the kernel: some kernels with ublk backported apply the discard queue limits but lack the pre-5.19 QUEUE_FLAG_DISCARD glue, so fstrim/blkdiscard report "operation not supported" there (graceful degradation; the userspace path is covered by unit tests).
Hardening backlog: orphan-device cleanup in del/list (direct /dev/ublk-control DEL without pidfile).
Unmount filesystems on /dev/ublkbN before del: the daemon serves the device's IO, so tearing it down with a mounted filesystem aborts in-flight journal writes (documented in README).