-
Notifications
You must be signed in to change notification settings - Fork 0
feat: add a Redis cross-process wake-up #20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,183 @@ | ||
| # rbs_inline: enabled | ||
|
|
||
| require "timeout" | ||
|
|
||
| module SolidObjects | ||
| module WakeUpAdapters | ||
| # Wakes runtime roles across processes using Redis publish/subscribe. | ||
| # | ||
| # MySQL has no notification primitive, so this is the cross-process option | ||
| # for applications that cannot use PostgreSQL notifications. It is optional | ||
| # in every sense: the `redis` gem is not a dependency of this gem, and the | ||
| # polling interval remains the upper bound, so a missed or failed | ||
| # notification costs latency rather than correctness. | ||
| class Redis | ||
| CHANNEL = "solid_objects_wake_up" | ||
| FAILED_WAIT_INTERVAL = 0.05 | ||
| SUBSCRIBE_TIMEOUT = 5.0 | ||
|
|
||
| # @rbs @channel: String | ||
| # @rbs @url: String? | ||
| # @rbs @client: untyped | ||
| # @rbs @mutex: Thread::Mutex | ||
| # @rbs @condition: Thread::ConditionVariable | ||
| # @rbs @subscriber: Thread? | ||
| # @rbs @subscription: untyped | ||
| # @rbs @signalled: Integer | ||
|
|
||
| attr_reader :channel | ||
|
|
||
| # @rbs (?channel: String, ?url: String?, ?client: untyped) -> void | ||
| def initialize(channel: CHANNEL, url: nil, client: nil) | ||
| @channel = channel | ||
| @url = url | ||
| @client = client | ||
| @mutex = Thread::Mutex.new | ||
| @condition = Thread::ConditionVariable.new | ||
| @subscriber = nil | ||
| @subscription = nil | ||
| @signalled = 0 | ||
| validate_client! | ||
| end | ||
|
|
||
| # @rbs () -> bool | ||
| def signal | ||
| publisher.publish(channel, "1") | ||
| true | ||
| rescue => error | ||
| instrument_failure(:signal, error) | ||
| false | ||
| end | ||
|
|
||
| # The counter is snapshotted before subscribing, and re-checked before | ||
| # blocking, so a signal delivered while this caller was still getting | ||
| # ready is observed rather than absorbed into the new baseline. | ||
| # @rbs (timeout: Numeric) -> bool | ||
| def wait(timeout:) | ||
| signalled = mutex.synchronize { @signalled } | ||
| return paced_failure(timeout) unless listen | ||
|
|
||
| mutex.synchronize do | ||
| return true unless @signalled == signalled | ||
|
|
||
| condition.wait(mutex, timeout.to_f) | ||
| @signalled != signalled | ||
| end | ||
| end | ||
|
|
||
| # Redis delivers to a subscribed connection only, and a subscribed | ||
| # connection cannot serve other callers, so one background subscription | ||
| # per process fans out to every waiting role in memory. Subscribing | ||
| # eagerly also closes the window where a signal sent during startup would | ||
| # be missed. | ||
| # @rbs () -> bool | ||
| def listen | ||
| mutex.synchronize do | ||
| return true if @subscriber&.alive? | ||
|
|
||
| ready = Queue.new | ||
| @subscriber = Thread.new { subscribe_loop(ready) } | ||
| Timeout.timeout(SUBSCRIBE_TIMEOUT) { ready.pop } == :subscribed | ||
| end | ||
| rescue => error | ||
| instrument_failure(:listen, error) | ||
| false | ||
| end | ||
|
|
||
| # @rbs () -> bool | ||
| def stop | ||
| subscriber = mutex.synchronize do | ||
| thread = @subscriber | ||
| @subscriber = nil | ||
| thread | ||
| end | ||
| return false unless subscriber | ||
|
|
||
| disconnect(@subscription) | ||
| subscriber.join(SUBSCRIBE_TIMEOUT) | ||
| subscriber.kill if subscriber.alive? | ||
| true | ||
| end | ||
|
|
||
| private | ||
|
|
||
| attr_reader :mutex, :condition, :url | ||
|
|
||
| # @rbs (Queue) -> void | ||
| def subscribe_loop(ready) | ||
| connection = build_client | ||
| @subscription = connection | ||
| connection.subscribe(channel) do |on| | ||
| on.subscribe { ready << :subscribed } | ||
| on.message { broadcast } | ||
| end | ||
| rescue => error | ||
| instrument_failure(:subscribe, error) | ||
| ready << :failed | ||
| end | ||
|
|
||
| # @rbs () -> void | ||
| def broadcast | ||
| mutex.synchronize do | ||
| @signalled += 1 | ||
| condition.broadcast | ||
| end | ||
| end | ||
|
|
||
| # @rbs (Numeric) -> bool | ||
| def paced_failure(timeout) | ||
| pace_after_failure(timeout) | ||
| false | ||
| end | ||
|
|
||
| # @rbs () -> untyped | ||
| def publisher | ||
| @publisher ||= build_client | ||
| end | ||
|
|
||
| # @rbs () -> untyped | ||
| def build_client | ||
| return @client.call if @client.respond_to?(:call) | ||
|
|
||
| require "redis" | ||
| url ? ::Redis.new(url:) : ::Redis.new | ||
| rescue LoadError | ||
| raise ArgumentError, | ||
| "the redis gem is required for SolidObjects::WakeUpAdapters::Redis" | ||
| end | ||
|
|
||
| # @rbs () -> void | ||
| def validate_client! | ||
| return if @client.nil? || @client.respond_to?(:call) | ||
|
|
||
| raise ArgumentError, "client must respond to call and return a Redis client" | ||
| end | ||
|
|
||
| # @rbs (untyped) -> void | ||
| def disconnect(connection) | ||
| connection&.close | ||
| rescue | ||
| nil | ||
| end | ||
|
|
||
| # @rbs (Numeric) -> void | ||
| def pace_after_failure(timeout) | ||
| interval = [ timeout.to_f, FAILED_WAIT_INTERVAL ].min | ||
| return unless interval.positive? | ||
|
|
||
| sleep interval | ||
| end | ||
|
|
||
| # @rbs (Symbol, Exception) -> void | ||
| def instrument_failure(operation, error) | ||
| SolidObjects.instrument( | ||
| :"wake_up.failed", | ||
| adapter: "redis", | ||
| operation: operation.to_s, | ||
| error_class: error.class.name, | ||
| error_message: error.message | ||
| ) | ||
| end | ||
| end | ||
| end | ||
| end | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a Redis notification arrives after
listenreleases the mutex but beforewaitsnapshots@signalled, the increment becomes the new baseline and the condition broadcast occurs before the waiter is registered, causing the runtime role to sleep untilpolling_intervalexpires.Prompt To Fix With AI
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Valid, and it is the same class of bug as the one my own test caught earlier in this adapter: a window where a signal is delivered to nobody.
A notification arriving after
listenreleased the mutex but beforewaitsnapshotted@signalledbecame the new baseline, and its broadcast reached no registered waiter, so the role slept out its polling interval.The counter is now snapshotted before subscribing and re-checked before blocking, so a signal delivered while a waiter was still getting ready is observed rather than swallowed. A test publishes from inside
listento model exactly that interleaving; it fails against the previous code and passes now.