ActiveJob and :exponentially_longer

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

ActiveJob and :exponentially_longer

Anthony Ross
We're seeing issues using :exponentially_longer where failures in downstream systems are causing a thundering herd effect.  A great many jobs fail, they all get enqueued to try again in a static interval and bring down the downstream system yet again.  This repeats until the retry limit is exhausted.

We've gotten around this using a custom `wait` block but since exponential backoff with randomness is builtin to some adapters, like Sidekiq , I'm propsing to have it baked into to :exponentially_longer.

I submitted a PR here: https://github.com/rails/rails/pull/31872

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveJob and :exponentially_longer

Anthony Ross

Hello - Is anyone available to review this PR?  I believe I've addressed all feedback.  Thank you!  https://github.com/rails/rails/pull/31872

Anthony


On Friday, February 2, 2018 at 9:04:30 AM UTC-8, Anthony Ross wrote:
We're seeing issues using :exponentially_longer where failures in downstream systems are causing a <a href="https://en.wikipedia.org/wiki/Thundering_herd_problem" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fen.wikipedia.org%2Fwiki%2FThundering_herd_problem\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFkiADwIeAJan2d7vDD75sA12K90g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fen.wikipedia.org%2Fwiki%2FThundering_herd_problem\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFkiADwIeAJan2d7vDD75sA12K90g&#39;;return true;">thundering herd effect.  A great many jobs fail, they all get enqueued to try again in a static interval and bring down the downstream system yet again.  This repeats until the retry limit is exhausted.

We've gotten around this using a custom `wait` block but since exponential backoff with randomness is builtin to some adapters, like <a href="https://github.com/mperham/sidekiq/issues/480" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fmperham%2Fsidekiq%2Fissues%2F480\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHQhcKg7Kb54eDfPbvCuDo3NxnzRA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fmperham%2Fsidekiq%2Fissues%2F480\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHQhcKg7Kb54eDfPbvCuDo3NxnzRA&#39;;return true;">Sidekiq , I'm propsing to have it baked into to :exponentially_longer.

I submitted a PR here: <a href="https://github.com/rails/rails/pull/31872" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fpull%2F31872\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFspUUut4HMtRN8OUQe0-uKZeu6Gg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fpull%2F31872\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFspUUut4HMtRN8OUQe0-uKZeu6Gg&#39;;return true;">https://github.com/rails/rails/pull/31872

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-core/64203323-af8e-41aa-8e9a-724d485b1a26%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveJob and :exponentially_longer

Tom Rossi
+1 I had a similar issue and would welcome it...

On Fri, Nov 22, 2019 at 6:33 PM Anthony Ross <[hidden email]> wrote:

Hello - Is anyone available to review this PR?  I believe I've addressed all feedback.  Thank you!  https://github.com/rails/rails/pull/31872

Anthony


On Friday, February 2, 2018 at 9:04:30 AM UTC-8, Anthony Ross wrote:
We're seeing issues using :exponentially_longer where failures in downstream systems are causing a thundering herd effect.  A great many jobs fail, they all get enqueued to try again in a static interval and bring down the downstream system yet again.  This repeats until the retry limit is exhausted.

We've gotten around this using a custom `wait` block but since exponential backoff with randomness is builtin to some adapters, like Sidekiq , I'm propsing to have it baked into to :exponentially_longer.


--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-core/64203323-af8e-41aa-8e9a-724d485b1a26%40googlegroups.com.
--
Sent from mobile

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-core/CAJAxyrdTk_pudJ820y%2BEJfrQz%3DN4_GC48sSr5h9Kp-FoCmEiOg%40mail.gmail.com.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveJob and :exponentially_longer

Santiago Bartesaghi
In reply to this post by Anthony Ross
+1 to get this feature into Rails

On Friday, February 2, 2018 at 2:04:30 PM UTC-3, Anthony Ross wrote:
We're seeing issues using :exponentially_longer where failures in downstream systems are causing a <a href="https://en.wikipedia.org/wiki/Thundering_herd_problem" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fen.wikipedia.org%2Fwiki%2FThundering_herd_problem\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFkiADwIeAJan2d7vDD75sA12K90g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fen.wikipedia.org%2Fwiki%2FThundering_herd_problem\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFkiADwIeAJan2d7vDD75sA12K90g&#39;;return true;">thundering herd effect.  A great many jobs fail, they all get enqueued to try again in a static interval and bring down the downstream system yet again.  This repeats until the retry limit is exhausted.

We've gotten around this using a custom `wait` block but since exponential backoff with randomness is builtin to some adapters, like <a href="https://github.com/mperham/sidekiq/issues/480" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fmperham%2Fsidekiq%2Fissues%2F480\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHQhcKg7Kb54eDfPbvCuDo3NxnzRA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fmperham%2Fsidekiq%2Fissues%2F480\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHQhcKg7Kb54eDfPbvCuDo3NxnzRA&#39;;return true;">Sidekiq , I'm propsing to have it baked into to :exponentially_longer.

I submitted a PR here: <a href="https://github.com/rails/rails/pull/31872" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fpull%2F31872\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFspUUut4HMtRN8OUQe0-uKZeu6Gg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fpull%2F31872\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFspUUut4HMtRN8OUQe0-uKZeu6Gg&#39;;return true;">https://github.com/rails/rails/pull/31872

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-core/9dd33afd-f746-4e68-a382-00f2f352d222%40googlegroups.com.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveJob and :exponentially_longer

Kasper Timm Hansen
It’s merged. Thanks everyone 🙌

--
Kasper
On 28 Nov 2019, 19.08 +0100, Santiago Bartesaghi <[hidden email]>, wrote:
+1 to get this feature into Rails

On Friday, February 2, 2018 at 2:04:30 PM UTC-3, Anthony Ross wrote:
We're seeing issues using :exponentially_longer where failures in downstream systems are causing a thundering herd effect.  A great many jobs fail, they all get enqueued to try again in a static interval and bring down the downstream system yet again.  This repeats until the retry limit is exhausted.

We've gotten around this using a custom `wait` block but since exponential backoff with randomness is builtin to some adapters, like Sidekiq , I'm propsing to have it baked into to :exponentially_longer.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-core/9dd33afd-f746-4e68-a382-00f2f352d222%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/rubyonrails-core/25021fa2-d30f-4477-bee9-6460d5575726%40Spark.