snrt_barrier potential livelock

[jpf-h]

@Aquaticfuller @DiyouS

Found this while programming a kernel, but I'm not sure if it is a real issue. At least I did not see it causing problems yet, but Claude Code flagged it as a potential issue. Can you check this?

The reset barr->barrier = 0 is a plain volatile store and the release __atomic_add_fetch(&barrier_iteration, 1, __ATOMIC_RELAXED) is a relaxed RMW. C11 gives no happens-before edge between them as observed from another thread — a waiter that observes the iteration bump has no guarantee of seeing the counter reset. volatile orders nothing across threads; relaxed orders nothing, period.

By the ISA: under RVWMO the two writes are to different addresses with no fence between them and no .aq/.rl — unordered, so a conforming RISC-V implementation may make the AMO visible first. And on this platform specifically, even upgrading the RMW to __ATOMIC_RELEASE would not fix it, because Snitch ignores the .rl bit on AMOs (the platform's own known erratum) — only an explicit fence works.

The failure when it fires: a waiter released by the early-visible bump sprints to the next barrier round and amoadds the stale counter; the late-arriving reset clobbers that increment; the new round's count can then never reach n, and every participant spins forever.

Plausibility on this hardware: the two writes demonstrably travel different physical paths — the plain store through the cache datapath, the AMO through the insitu AMO shim at the bank. The window is narrow (a few dozen cycles inside the last arriver's call) and needs tight barrier reuse to exploit — which is why it hasn't bitten anyone — but tight reuse is exactly what back-to-back barrier calls do, and our interval loop crosses the same object four times per interval with nothing in between.

All three counting barriers in barrier.c share the defect — snrt_barrier, snrt_cluster_sw_barrier, and snrt_global_barrier are the same reset-then-bump shape. snrt_cluster_hw_barrier is unaffected.

The fix is one fence per function: between the counter reset and the iteration increment, asm volatile("fence rw, rw" ::: "memory") (a release thread-fence also works and lowers to a fence; a .rl on the RMW does not, per the Snitch erratum).

A secondary observation, separable: these barriers provide no memory ordering for user data at all (everything is relaxed, no fences) — they are pure rendezvous. Most callers assume "barrier" implies "my writes before it are visible after it," and on this platform that assumption silently fails. Either document it loudly or give the barrier acquire/release semantics

[DiyouS]

Currently we are not using software barrier at all. We use the hardware barrier snrt_cluster_hw_barrier which bases on the following mechanism:

1. Cores send a read/write request to the hardware barrier address (an address reserved in memory-mapped register space)
2. The tile-level barrier block will NOT ack the request, which will cause the core stalling and waiting
3. After all participants reach the tile barrier, the FSM inside tile barrier will issue a one-bit signal to the upper level for global synchronization
4. The cluster-level barrier will release the lock after all participating tiles reach it.
5. Tile barrier will ack the ld/st and give a fake rsp then

[jpf-h]

Understood. However, with the current HW barrier, it is not possible to synchronize different "software groups" of cores, is it? Say I have core 1-3 working on one problem and core 4-7 on a completely different one. And both groups need barriers, but independently? I think in this case we need SW barriers.

So I think i the SW implementation is not used, that's fair. But if it is potentially buggy on this architecture, I'd rather remove it from the runtime than have a potentially broken implementation there.

[DiyouS]

Understood. However, with the current HW barrier, it is not possible to synchronize different "software groups" of cores, is it? Say I have core 1-3 working on one problem and core 4-7 on a completely different one. And both groups need barriers, but independently? I think in this case we need SW barriers.

So I think i the SW implementation is not used, that's fair. But if it is potentially buggy on this architecture, I'd rather remove it from the runtime than have a potentially broken implementation there.

No it is still possible. In the dev/multi-group branch, we can configure the partial barrier in the following way:

1. A function in runtime can generate a mask based on the software group of cores participating in the barrier, e.g. core 1-3 and core 4-7
2. The cluster level barrier can be configrued via memory-mapped register on which tile to participate: e.g. tile 0 and 1
3. Then the partial barrier will wait only for core 1-3 in tile 0, core 4-7 in tile 1, and tile 0 and 1 in cluster level.

The SW barrier is taken from spatz cluster, which is the initial platform we adapted from. We will have some cleaning work to remove all unused or broken functions