[RDMA] Send/recv buffers share one registered memory pool → silent stall risk at large scale

[KanPlus]

Summary

In bRPC RDMA, all connections' send-serialization buffers and receive pre-posted buffers come from a single process-wide registered memory pool (AllocBlock, block_pool.cpp:426), pinned via ibv_reg_mr. Unlike TCP (independent, pageable, kernel-managed per-socket buffers), this creates a structural send/recv coupling that can cause a silent, whole-pool stall under large-scale, mixed-load, under-provisioned deployments.

Root cause (with code pointers)

• Global IOBuf allocator is replaced by the registered pool: rdma_helper.cpp:578.
• Send buffer is released only after the peer ACKs (IMM): rdma_endpoint.cpp:965. The peer can only receive if it has already posted a recv buffer — which also comes from the same pool.
• On pool exhaustion, PostRecv(1) fails (rdma_endpoint.cpp:983, AllocBlockFrom returns NULL at block_pool.cpp:374).
• Cycle: pool exhausted → recv re-post fails → no one can receive → in-flight sends never complete → ACK never returns → send buffers (same pool) never freed → pool never recovers → all connections sharing the pool stall.
• The per-connection credit window (_window_size) is congestion control only and is independent of AllocBlock; it provides no memory isolation.

When it happens (scenarios)

Three conditions together:

1. Mixed load — large messages (64KB/2MB blocks) and small RPCs (8KB blocks) share one pool.
2. High fan-out — many concurrent connections under load.
3. Under-provisioned pool — especially since, by default buckets=4, the 8KB pool (block_type 0, carrying all normal RPC send/recv buffers) cannot grow at runtime (CanExtendBlockRuntime, block_pool.cpp:139-141 requires buckets==1 or region_num<1; InitBlockPool already set region_num[0]=1). It is locked at the initial 1GB.

Typical deployments at risk: gateways/proxies mixing big transfers and small control RPCs, multi-tenant access layers, long-lived large-message streams.

Impact

• Whole-pool stall, not single-request failure. All RDMA connections sharing the pool hang simultaneously (availability-level).
• Silent and hard to diagnose. Only occasional Fail to extend region / Fail to allocate rbuf in logs — no "shared-pool deadlock" signal. Easy to misattribute to network/kernel (Fail to ibv_post_send: Cannot allocate memory #3202 showed ibv_post_send: Cannot allocate memory, rooted in kernel SQ memory, while the block pool had already triggered extend).
• No per-connection fairness/isolation. AllocBlockFrom is LIFO + per-thread TLS hoarding + random bucket, biasing toward recently active (large-flow) threads.

Questions for the community

1. Is the send/recv shared-pool coupling at scale considered expected behavior, or a real reliability risk?
2. Should docs state explicit RDMA applicable / not-applicable boundaries for large-scale mixed load?
3. Should CanExtendBlockRuntime's buckets==1 restriction be relaxed for block_type 0 so the 8KB pool can grow under default config?
4. Should a per-direction (at least receive-side) reserved/guaranteed memory mechanism be introduced to avoid recv refill being starved by sends?
5. Should docs (max_regions default 16 vs code 3; buckets restriction) be aligned with code?

Static source analysis only; recommend a repro with mixed large/small flows + high fan-out + gradually shrinking pool.