Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
# Changelog

## Unreleased

- Add `SolidObjects::WakeUpAdapters::Postgresql`, an optional cross-process
wake-up using PostgreSQL notifications. In-process signalling cannot reach a
worker process, so reactive delivery waited out `polling_interval`. With the
adapter configured, measured cross-process wake-up latency drops from 103.7 ms
to 2.9 ms at p50. The polling interval remains the upper bound, and neither
signalling nor waiting raises into its caller. `WakeUpAdapters.for` selects
notifications on PostgreSQL and the in-process default elsewhere; it is not
the default, because the adapter opens a connection per waiting thread
outside the pool and `LISTEN` does not survive a transaction-pooling proxy.

## 0.7.3 - 2026-08-09

- Coordinate batched component refreshes by revision as well as scope and batch
Expand Down
15 changes: 15 additions & 0 deletions docs/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,21 @@ Cable delivery, or browser rendering, which dominate wall-clock time in a real
deployment and make the request-count difference matter more than it appears
here. End-to-end latency against a deployed application has not been measured.

## Cross-process wake-up

Measured 2026-08-09 against PostgreSQL 17, 30 samples, with `polling_interval`
at its 100 ms default and a signal sent 2 ms after the waiter began.

| Wake-up strategy | p50 | p95 |
| --- | ---: | ---: |
| In-process `WakeUp` | 103.7 ms | 105.1 ms |
| `WakeUpAdapters::Postgresql` | 2.9 ms | 5.1 ms |

The in-process wake-up cannot reach another process, so a worker waits out the
full polling interval no matter how quickly the web process committed. The
notification adapter removes that floor rather than shrinking it, and the
polling interval remains the upper bound if a notification is missed.

## Durable row growth

The storage cost is deterministic even when latency is not:
Expand Down
30 changes: 30 additions & 0 deletions docs/realtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,36 @@ applications discover the namespaced engine asset. Applications created with
explicitly serve the module. Turbo's normal morph rules still apply; use
`data-turbo-permanent` for elements that must never be changed.

## Cross-process wake-up

Runtime roles poll for work and are woken early by an in-process signal. That
signal cannot cross process boundaries, so a commit in a Puma process does not
wake a broadcast executor in a worker process, and delivery waits out
`polling_interval`, 100 ms by default.

On PostgreSQL, install the notification adapter to remove that delay:

```ruby
# config/initializers/solid_objects.rb
configuration.wake_up_adapter = SolidObjects::WakeUpAdapters.for
```

`WakeUpAdapters.for` returns notifications on PostgreSQL and the in-process
default on SQLite and MySQL, so the same line is safe across adapters. Name
`SolidObjects::WakeUpAdapters::Postgresql.new` directly to require it.

MySQL has no notification primitive, so MySQL applications keep polling and tune
`polling_interval`.

Measured latency for a cross-process wake-up drops from 103.7 ms to 2.9 ms at
p50. The adapter keeps `polling_interval` as the upper bound: a missed or failed
notification costs latency, never correctness, and signalling never raises into
the caller that committed. `LISTEN` needs its own connection, so the adapter
opens one outside the pool and releases it on `stop`.

Applications on SQLite or MySQL, or that do not configure the adapter, keep the
existing polling behaviour.

## Batched component refreshes

A component refresh costs one browser request. When one actor mutation changes
Expand Down
8 changes: 8 additions & 0 deletions lib/generators/solid_objects/templates/solid_objects.rb
Original file line number Diff line number Diff line change
Expand Up @@ -58,4 +58,12 @@
# configuration.authorize_administration = lambda do |authorization_context:, **|
# authorization_context.is_a?(Hash) && authorization_context[:source] == "cli"
# end

# Runtime roles poll for work and are woken early by an in-process signal,
# which cannot reach another process. On PostgreSQL, notifications remove that
# delay. This opens a connection per waiting thread outside the pool and does
# not work through a transaction-pooling proxy such as PgBouncer, so it is
# opt-in:
#
# configuration.wake_up_adapter = SolidObjects::WakeUpAdapters.for
end
2 changes: 2 additions & 0 deletions lib/solid_objects.rb
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@
require "solid_objects/actor_channel"
require "solid_objects/action_cable_broadcast_adapter"
require "solid_objects/wake_up"
require "solid_objects/wake_up_adapters/postgresql"
require "solid_objects/wake_up_adapters"
require "solid_objects/effect_registry"
require "solid_objects/commit_action_registry"
require "solid_objects/lease"
Expand Down
28 changes: 23 additions & 5 deletions lib/solid_objects/supervisor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -44,17 +44,35 @@ def start
def stop
return unless @started

components.each(&:request_shutdown)
join_until_timeout
components.reject(&:stopped?).each(&:stop)
@started = false
SolidObjects.instrument(:"supervisor.stopped", component_count: components.length)
begin
components.each(&:request_shutdown)
join_until_timeout
components.reject(&:stopped?).each(&:stop)
ensure
# Connections held outside the pool must be released even when a
# component fails to stop, or they accumulate across restarts.
release_wake_up
@started = false
SolidObjects.instrument(:"supervisor.stopped", component_count: components.length)
end
end

private

attr_reader :components, :threads

# A wake-up adapter may hold connections outside the pool, which would
# otherwise accumulate across restarts in one process.
# @rbs () -> void
def release_wake_up
wake_up = SolidObjects.wake_up
return unless wake_up.respond_to?(:stop)

wake_up.stop
rescue
nil
end

# @rbs (worker_count: Integer, effect_worker_count: Integer, broadcast_worker_count: Integer, reminder_scheduler_count: Integer) -> Array[Worker | EffectExecutor | ReminderScheduler | BroadcastExecutor]
def build_components(
worker_count:,
Expand Down
23 changes: 23 additions & 0 deletions lib/solid_objects/wake_up_adapters.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# rbs_inline: enabled

module SolidObjects
module WakeUpAdapters
module_function

# Returns the best wake-up strategy for a connection: cross-process
# notifications where the database provides them, and the in-process
# default everywhere else.
#
# This is deliberately not the default. A notification adapter opens a
# connection per waiting thread outside the pool, and `LISTEN` does not
# survive a transaction-pooling proxy such as PgBouncer, so adopting it is
# a deployment decision rather than an upgrade side effect.
#
# @rbs (?untyped) -> untyped
def for(connection = Record.connection)
return Postgresql.new if connection.adapter_name.match?(/postgres/i)

WakeUp.new
end
end
end
136 changes: 136 additions & 0 deletions lib/solid_objects/wake_up_adapters/postgresql.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# rbs_inline: enabled

module SolidObjects
module WakeUpAdapters
# Wakes runtime roles across processes using PostgreSQL notifications.
#
# The in-process wake-up cannot reach another process, so a commit in a web
# process leaves a worker waiting out its polling interval. This adapter
# keeps that polling interval as the upper bound and delivers a notification
# when one is available, so a missed or failed notification costs latency
# rather than correctness.
class Postgresql
CHANNEL = "solid_objects_wake_up"
FAILED_WAIT_INTERVAL = 0.05

# @rbs @channel: String
# @rbs @mutex: Thread::Mutex
# @rbs @connections: Array[untyped]

attr_reader :channel

# @rbs (?channel: String) -> void
def initialize(channel: CHANNEL)
@channel = channel
@mutex = Thread::Mutex.new
@connections = []
end

# @rbs () -> bool
def signal
notify_channel
true
rescue => error
instrument_failure(:signal, error)
false
end

# @rbs (timeout: Numeric) -> bool
def wait(timeout:)
connection = listening_connection

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Shared connection breaks wake-up broadcast

When the supervisor runs multiple runtime roles, the process-wide adapter returns the same cached PostgreSQL connection to every waiter and calls wait_for_notify outside the mutex. A notification is consumed through that one connection instead of independently waking every role, while concurrent connection use can also fail and leave roles asleep until their polling timeout.

Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/solid_objects/wake_up_adapters/postgresql.rb
Line: 40

Comment:
**Shared connection breaks wake-up broadcast**

When the supervisor runs multiple runtime roles, the process-wide adapter returns the same cached PostgreSQL connection to every waiter and calls `wait_for_notify` outside the mutex. A notification is consumed through that one connection instead of independently waking every role, while concurrent connection use can also fail and leave roles asleep until their polling timeout.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct on both halves, and the test half is the part I got wrong.

The adapter is memoized process-wide and shared across runtime roles, so every waiter used one cached connection and called wait_for_notify on it concurrently. Whichever waiter won consumed the notification and the other roles slept until their polling timeout, which defeats the purpose of the adapter, and concurrent use of a single connection is unsafe regardless.

Fixed in 9a6ca78: a listening connection is opened and tracked per thread, and stop disconnects all of them.

On the test: you were right that it only demonstrated multiple waiters with separate adapters. I rewrote it to share one adapter, and it still passed against the broken code, because it asserted only that each waiter woke, not when. Two of the three were waking at the 5 second timeout, which the assertion happily accepted. It now asserts every waiter wakes in under a second against a 5 second timeout. Against the shared connection that test hangs; with the per-thread connection all seven pass.

Added a second test asserting two threads hold distinct listening connections.

!connection.raw_connection.wait_for_notify(timeout.to_f).nil?
rescue => error
instrument_failure(:wait, error)
pace_after_failure(timeout)
false
end

# Starts listening before a caller blocks, so a notification sent between
# startup and the first wait is not missed.
# @rbs () -> bool
def listen
listening_connection
true
rescue => error
instrument_failure(:listen, error)
false
end

# @rbs () -> bool
def stop
open = mutex.synchronize do
listening = connections.dup
connections.clear
listening
end
Thread.current[thread_key] = nil
open.each { |connection| disconnect(connection) }
open.any?
end

private

attr_reader :mutex, :connections

# @rbs () -> void
def notify_channel
Record.connection_pool.with_connection do |connection|
connection.execute("NOTIFY #{connection.quote_table_name(channel)}")
end
end

# A listening connection is dedicated and per thread. `LISTEN` is per
# connection, a blocking wait must not hold a connection the rest of the
# runtime needs, and one connection cannot serve concurrent waiters: the
# supervisor shares one adapter across roles, and a notification consumed
# by one waiter would leave the others asleep until their poll expired.
# @rbs () -> untyped
def listening_connection
connection = Thread.current[thread_key]
return connection if connection&.active?

open_listening_connection
end

# @rbs () -> untyped
def open_listening_connection
connection = Record.connection_pool.send(:new_connection)
connection.execute("LISTEN #{connection.quote_table_name(channel)}")
Thread.current[thread_key] = connection
mutex.synchronize { connections << connection }
Comment on lines +97 to +100

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Listener connections survive shutdown

When a configured PostgreSQL adapter has opened per-thread listeners and the supervisor is restarted in the same process, Supervisor#stop never invokes the adapter's stop, so those dedicated connections remain open and accumulate across restarts, eventually consuming PostgreSQL connection capacity.

Prompt To Fix With AI
This is a comment left during a code review.
Path: lib/solid_objects/wake_up_adapters/postgresql.rb
Line: 97-100

Comment:
**Listener connections survive shutdown**

When a configured PostgreSQL adapter has opened per-thread listeners and the supervisor is restarted in the same process, `Supervisor#stop` never invokes the adapter's `stop`, so those dedicated connections remain open and accumulate across restarts, eventually consuming PostgreSQL connection capacity.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. The adapter holds connections outside the pool and Supervisor#stop never released them, so restarting the supervisor in one process accumulated listeners until PostgreSQL ran out of capacity.

Fixed in f61792e: shutdown stops the wake-up adapter when it supports stop, guarded so the default in-process WakeUp, which has no stop, is unaffected, and rescued so a failure there cannot break shutdown.

Regression test: configure the adapter, open a listener, start and stop a supervisor, and assert no connections remain tracked.

connection
end

# @rbs () -> Symbol
def thread_key
:"solid_objects_wake_up_#{object_id}"
end

# @rbs (untyped) -> void
def disconnect(connection)
connection.disconnect!
rescue
nil
end

# @rbs (Numeric) -> void
def pace_after_failure(timeout)
interval = [ timeout.to_f, FAILED_WAIT_INTERVAL ].min
return unless interval.positive?

sleep interval
end

# @rbs (Symbol, Exception) -> void
def instrument_failure(operation, error)
SolidObjects.instrument(
:"wake_up.failed",
adapter: "postgresql",
operation: operation.to_s,
error_class: error.class.name,
error_message: error.message
)
end
end
end
end
5 changes: 5 additions & 0 deletions sig/generated/lib/solid_objects/supervisor.rbs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ module SolidObjects

attr_reader threads: untyped

# A wake-up adapter may hold connections outside the pool, which would
# otherwise accumulate across restarts in one process.
# @rbs () -> void
def release_wake_up: () -> void

# @rbs (worker_count: Integer, effect_worker_count: Integer, broadcast_worker_count: Integer, reminder_scheduler_count: Integer) -> Array[Worker | EffectExecutor | ReminderScheduler | BroadcastExecutor]
def build_components: (worker_count: Integer, effect_worker_count: Integer, broadcast_worker_count: Integer, reminder_scheduler_count: Integer) -> Array[Worker | EffectExecutor | ReminderScheduler | BroadcastExecutor]

Expand Down
17 changes: 17 additions & 0 deletions sig/generated/lib/solid_objects/wake_up_adapters.rbs
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Generated from lib/solid_objects/wake_up_adapters.rb with RBS::Inline

module SolidObjects
module WakeUpAdapters
# Returns the best wake-up strategy for a connection: cross-process
# notifications where the database provides them, and the in-process
# default everywhere else.
#
# This is deliberately not the default. A notification adapter opens a
# connection per waiting thread outside the pool, and `LISTEN` does not
# survive a transaction-pooling proxy such as PgBouncer, so adopting it is
# a deployment decision rather than an upgrade side effect.
#
# @rbs (?untyped) -> untyped
def self?.for: (?untyped) -> untyped
end
end
Loading
Loading