Merge pull request #39 from cardmagic/agent/idle-polling-backoff

Reduce idle polling CPU for 0.13.1

Author: cardmagic

CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 0.13.1 - 2026-08-16
+
+- Back idle actor, effect, reminder, and broadcast polling off exponentially
+  from the configured fast interval to a new one-second idle ceiling. Any
+  processed work or wake-up resets the role immediately, and actor polling
+  remains capped by the lease-renewal interval.
+- Expose each role's `current_polling_interval` and emit
+  `solid_objects.polling.interval_changed` instrumentation for every idle,
+  work, and wake-up transition.
+- Warn once when live processes share the database without a configured
+  cross-process wake-up adapter.
+- Make the in-process wake-up generation-aware so a signal committed between
+  an empty claim and the wait is not missed. PostgreSQL and Redis adapters now
+  expose the same watch contract.
+- Add a reproducible four-role SQLite idle benchmark and repair the benchmark
+  schema setup for the current operation columns.
+- **Behavior change:** `polling_interval` is now the fast interval after
+  activity, not a constant idle cadence. Existing explicit values back off to
+  `idle_polling_interval`, which defaults to one second. Set both options to
+  the same value to preserve a fixed cadence. Existing custom wake-up adapters
+  that return `nil` remain at the fast cadence until they return `false` for a
+  timeout and `true` for a notification.
+
 ## 0.13.0 - 2026-08-15
 
 - **Breaking:** make observables invalidation-only by default. An ordinary

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    solid_objects (0.13.0)
+    solid_objects (0.13.1)
       actioncable (>= 8.0)
       actionpack (>= 8.0)
       actionview (>= 8.0)
@@ -384,7 +384,7 @@ CHECKSUMS
   rubocop-rails-omakase (1.1.0) sha256=2af73ac8ee5852de2919abbd2618af9c15c19b512c4cfc1f9a5d3b6ef009109d
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   securerandom (0.4.1) sha256=cc5193d414a4341b6e225f0cb4446aceca8e50d5e1888743fac16987638ea0b1
-  solid_objects (0.13.0)
+  solid_objects (0.13.1)
   sqlite3 (2.9.5-aarch64-linux-gnu) sha256=78075b6337d3d182c6d2b4691049ed45cd220826160c9ea18946bf6a1de200dc
   sqlite3 (2.9.5-aarch64-linux-musl) sha256=18c801185deb4adc01ddb281e8f672a39e3d1729979ca91e39439cd3eac0402d
   sqlite3 (2.9.5-arm-linux-gnu) sha256=1bdfca0c7d63998c60b0f4a8e3c8df2d33800ccc4abd2d612eddbbbc92a4c48b

README.md

@@ -1014,6 +1014,7 @@ Important defaults:
 | Setting | Default |
 | --- | ---: |
 | `polling_interval` | 0.1 seconds |
+| `idle_polling_interval` | 1 second |
 | `sync_polling_interval` | 0.05 seconds |
 | `lease_duration` | 30 seconds |
 | `lease_renewal_interval` | 10 seconds |
@@ -1038,6 +1039,14 @@ Payload, state, and result limits; retry delay; table prefix; logging; wake-up;
 broadcast; database; and authorization adapters are also configurable. Invalid
 lease intervals, component counts, and size limits fail fast at boot.
 
+`polling_interval` is the fast interval after work or a wake-up. Consecutive
+empty passes double it up to `idle_polling_interval`. Actor workers never wait
+longer than `lease_renewal_interval`. Set the fast and idle values equal for a
+fixed cadence. The default wake-up reaches only the current Ruby process;
+configure PostgreSQL notifications or optional Redis Pub/Sub when separate
+processes need low-latency delivery. The runtime warns once when it sees that
+topology without an adapter.
+
 ## Workers and operations
 
 `solid_objects start` runs actor, effect, reminder, and broadcast roles under
@@ -1299,8 +1308,8 @@ Partially implemented:
 
 - the supervisor starts and drains roles but does not replace a crashed role or
   run periodic maintenance automatically;
-- cross-process wake-up uses polling; PostgreSQL notifications and optional
-  Redis acceleration are not implemented;
+- PostgreSQL notifications and optional Redis acceleration are implemented,
+  but adapter selection remains explicit and polling is the durable fallback;
 - live observable and component replacement work, while Turbo append actions
   remain future work;
 - local admission limits exist, but distributed rate limits and global

benchmark/idle_polling.rb

@@ -0,0 +1,98 @@
+# frozen_string_literal: true
+
+require "json"
+require "rbconfig"
+require_relative "support"
+
+class CountingWakeUp < SolidObjects::WakeUp
+  def initialize
+    super
+    @count_mutex = Mutex.new
+    @poll_count = 0
+  end
+
+  def wait(timeout:, generation: nil)
+    @count_mutex.synchronize { @poll_count += 1 }
+    super
+  end
+
+  def reset_count
+    @count_mutex.synchronize { @poll_count = 0 }
+  end
+
+  def poll_count
+    @count_mutex.synchronize { @poll_count }
+  end
+end
+
+def measure(interval:, warmup:, duration:, wake_up:)
+  SolidObjects.configuration.polling_interval = interval
+  SolidObjects.configuration.idle_polling_interval = 1.0
+  components = [
+    SolidObjects::Worker.new,
+    SolidObjects::EffectExecutor.new,
+    SolidObjects::ReminderScheduler.new,
+    SolidObjects::BroadcastExecutor.new
+  ]
+  threads = components.map { |component| Thread.new { component.run } }
+
+  sleep warmup
+  wake_up.reset_count
+  cpu_started_at = Process.times
+  wall_started_at = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+  sleep duration
+  wall_elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - wall_started_at
+  cpu_finished_at = Process.times
+  cpu_elapsed = cpu_finished_at.utime + cpu_finished_at.stime -
+    cpu_started_at.utime - cpu_started_at.stime
+  poll_count = wake_up.poll_count
+
+  {
+    polling_interval: interval,
+    idle_polling_interval: SolidObjects.configuration.idle_polling_interval,
+    current_intervals: components.map(&:current_polling_interval),
+    polls: poll_count,
+    polls_per_second: (poll_count / wall_elapsed).round(3),
+    idle_cpu_percent: ((cpu_elapsed / wall_elapsed) * 100).round(3)
+  }
+ensure
+  components&.each(&:request_shutdown)
+  threads&.each { |thread| thread.join(2) }
+  components&.each(&:stop)
+end
+
+intervals = ENV.fetch("INTERVALS", "0.02,0.1,0.5").split(",").map do |value|
+  Float(value).tap { |interval| raise ArgumentError, "intervals must be positive" unless interval.positive? }
+end
+warmup = Float(ENV.fetch("WARMUP", "3"))
+duration = Float(ENV.fetch("DURATION", "10"))
+raise ArgumentError, "warmup must be positive" unless warmup.positive?
+raise ArgumentError, "duration must be positive" unless duration.positive?
+
+SolidObjectsBenchmark.setup
+wake_up = CountingWakeUp.new
+SolidObjects.configuration.wake_up_adapter = wake_up
+database_version = ActiveRecord::Base.connection.select_value("SELECT sqlite_version()")
+results = intervals.map { |interval| measure(interval:, warmup:, duration:, wake_up:) }
+puts JSON.pretty_generate(
+  measured_at: Time.now.utc.iso8601,
+  package_version: SolidObjects::VERSION,
+  runtime: {
+    ruby: RUBY_DESCRIPTION,
+    platform: RUBY_PLATFORM,
+    cpu: RbConfig::CONFIG.fetch("host_cpu")
+  },
+  database: {
+    adapter: "sqlite",
+    version: database_version,
+    path: SolidObjectsBenchmark::DATABASE_PATH
+  },
+  methodology: {
+    roles: %w[actors effects reminders broadcasts],
+    warmup_seconds: warmup,
+    duration_seconds: duration,
+    cpu_percent: "process user plus system CPU time divided by wall time"
+  },
+  results:
+)
+SolidObjectsBenchmark.teardown

benchmark/support.rb

@@ -402,8 +402,10 @@ def establish_connection
     def migrate
       require_relative "../db/migrate/20260805000000_create_solid_objects_tables"
       require_relative "../db/migrate/20260806000000_add_state_revision_to_solid_objects_instances"
+      require_relative "../db/migrate/20260813000000_rename_message_dispatch_columns"
       CreateSolidObjectsTables.new.migrate(:up)
       AddStateRevisionToSolidObjectsInstances.new.migrate(:up)
+      RenameMessageDispatchColumns.new.migrate(:up)
     end
 
     # @rbs () -> void

docs/adr/0011-wake-up-strategy.md

@@ -11,6 +11,10 @@ Polling adds latency and database queries. PostgreSQL notifications are transact
 
 Database rows remain the only durable source of work and results. Wake-up
 adapters only prompt workers and synchronous waiters to re-query those rows.
+Actor, effect, reminder, and broadcast roles double consecutive empty waits
+from `polling_interval` to `idle_polling_interval`. Work and notifications reset
+the wait immediately. Actor workers clamp the ceiling to
+`lease_renewal_interval`.
 
 The interface supports:
 
@@ -40,4 +44,6 @@ Timeout does not cancel durable work.
 - A reconnecting PostgreSQL listener must commit `LISTEN`, inspect current state, and then wait.
 - Redis loss only increases latency and never loses durable work.
 - Every adapter retains periodic polling to close startup, reconnect, and missed-message races.
+- A process that returns `false` from a timed wait participates in backoff; an older custom adapter that returns `nil` keeps the fast cadence.
+- A multi-process deployment without an adapter trades idle database load for up to the current idle polling interval of notification latency and logs that topology once.
 - Notification payloads never contain actor arguments or results.

docs/architecture.md

@@ -365,8 +365,8 @@ it drains earlier messages and the target through the same activation and
 executor used by workers. If another process owns the actor, the caller waits
 for the row to become completed, rejected, dead-lettered, destroyed, or timed
 out. Every wait re-queries durable rows. The implemented wake-up interface
-provides same-process signaling, bounded polling, and dependency injection.
-PostgreSQL `LISTEN/NOTIFY` and optional Redis Pub/Sub are planned adapters.
+provides generation-aware same-process signaling, adaptive bounded polling,
+PostgreSQL `LISTEN/NOTIFY`, and optional Redis Pub/Sub.
 
 The normal path does not wait for a worker polling interval because the caller
 assists execution immediately. End-to-end latency still includes earlier

docs/benchmarks.md

@@ -5,6 +5,44 @@ They include the runtime's Active Record and database query overhead and will
 vary with hardware, schema size, connection pools, durability settings, and
 contention.
 
+## Idle SQLite polling
+
+Run the four-role idle harness with:
+
+```bash
+bundle exec ruby -Ilib benchmark/idle_polling.rb
+```
+
+It warms each interval for three seconds, measures for ten seconds, and reports
+process user plus system CPU time divided by wall time. Measured August 16,
+2026 on an Apple M5 with Ruby 4.0.6 and SQLite 3.53.2. The before run used
+0.13.0; the after run used the prepared 0.13.1 tree.
+
+| Fast interval | Before polls/s | Before CPU | After polls/s | After CPU |
+| ---: | ---: | ---: | ---: | ---: |
+| 20 ms | 165.340 | 8.401% | 3.998 | 0.947% |
+| 100 ms | 38.396 | 2.925% | 3.999 | 0.482% |
+| 500 ms | 7.998 | 2.061% | 3.996 | 0.283% |
+
+The after run reached the one-second ceiling for the actor, effect, reminder,
+and broadcast roles. These are developer-laptop measurements, not a CPU
+guarantee; timer scheduling, YJIT, the SQLite file, and unrelated host activity
+affect short samples.
+
+Five SQLite samples measured durable enqueue through committed completion after
+2.5 seconds of idleness. The polling-only multi-process harness submits just
+after an empty pass, so it measures approximately the full polling wait rather
+than average arrival latency.
+
+| Topology | 0.13.0 p50 | Prepared 0.13.1 p50 |
+| --- | ---: | ---: |
+| One process, in-process wake-up | 43.360 ms | 50.339 ms |
+| Two processes, polling only | 117.787 ms | 1,028.006 ms |
+
+The local wake-up keeps the one-process path prompt after backoff. The
+polling-only row is the explicit tradeoff: use PostgreSQL notifications or
+optional Redis Pub/Sub when separate processes need low-latency delivery.
+
 ## Production-shaped adoption measurement
 
 An adoption evaluation measured Solid Objects 0.2.0 from a macOS Rails process

docs/development.md

@@ -140,6 +140,7 @@ COUNT=500 CONCURRENCY=4 bundle exec ruby -Ilib benchmark/concurrent_actors.rb
 COUNT=100 bundle exec ruby -Ilib benchmark/sync_latency.rb
 COUNT=500 bundle exec ruby -Ilib benchmark/activation_cache.rb
 bundle exec ruby -Ilib benchmark/query_count.rb
+bundle exec ruby -Ilib benchmark/idle_polling.rb
 ```
 
 SQLite is the default. Set `SOLID_OBJECTS_DATABASE_URL` to benchmark a dedicated

docs/operations.md

@@ -69,6 +69,7 @@ Important controls include:
 - `lease_duration`
 - `lease_renewal_interval`
 - `polling_interval`
+- `idle_polling_interval`
 - `max_mailbox_length`
 - payload, state, and result byte limits
 - retry attempts and delay
@@ -81,6 +82,29 @@ Keep lease duration comfortably above renewal interval and expected database
 pause time. A handler can exceed the pass-duration budget because Ruby code is
 not safely preempted; alert on message duration and isolate untrusted work.
 
+## Polling and wake-up adapters
+
+`polling_interval` is the fast interval after work or a wake-up. Consecutive
+empty actor, effect, reminder, and broadcast passes double that role's wait up
+to `idle_polling_interval`, which defaults to one second. Actor workers clamp
+the ceiling to `lease_renewal_interval` while they may hold cached activations.
+Set the fast and idle values equal for a fixed cadence.
+
+The default wake-up interrupts waits only in the current Ruby process. When a
+live process record shows that the database is shared across processes and no
+adapter is configured, the runtime logs
+`solid_objects.polling_only_cross_process_wake_up` once. Configure
+`WakeUpAdapters::Postgresql` or `WakeUpAdapters::Redis` when separate processes
+need prompt delivery. Without one, newly committed work can wait up to the
+current idle polling interval.
+
+Each role exposes `current_polling_interval`.
+`solid_objects.polling.interval_changed` reports the role, reason, previous
+interval, and current interval. The polling-only warning is also emitted as
+`solid_objects.polling.only_cross_process_wake_up` instrumentation. Custom
+adapters should return `true` for a notification and `false` for a timeout; an
+older adapter that returns `nil` remains compatible and keeps the fast cadence.
+
 ## Graceful shutdown
 
 The supervisor requests shutdown, stops new claims, lets active loops return,