[feature] Add sentinel support to ActionCable Redis subscription adapter

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[feature] Add sentinel support to ActionCable Redis subscription adapter

Alistair Israel
I've been trying to deploy a small, sample Rails 5 app that uses ActionCable to send some real-time updates via WebSockets to the front-end Javascript. It works fine when running locally, or on single node test environments running single-node Redis as well.

When I tried to deploy it on to Kubernetes, however, I ran into a small stumbling block because to deploy Redis on to Kubernetes in a highly available manner it seems the recommended way is through use of Redis Sentinels:

* https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis
* https://redis.io/topics/sentinel

redis-rails already supports sentinels, so use of Redis with sentinels for cache, sessions, etc. is not a problem: https://github.com/redis-store/redis-rails#usage-with-redis-sentinel

However, the ActionCable Redis subscription adapter is configured independently, and when I dived into the code it became apparent that the configuration did not currently support specifying Redis sentinel configuration (even though the underlying redis gem powering both ActionCable and redis-rails already does).

It looks like it should only take a couple of lines of additional code at https://github.com/rails/rails/blob/master/actioncable/lib/action_cable/subscription_adapter/redis.rb#L13 to support an additional `sentinels: [...]` configuration parameter (if present).

I realize that I could simply supply my own custom lambda to ActionCable::SubscriptionAdapter::Redis.redis_connector in an initializer to support sentinels there, but am wondering if this is something the rest of the community might appreciate. If so, I'll gladly raise a PR.

Cheers!
- alistair

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [feature] Add sentinel support to ActionCable Redis subscription adapter

blaed
Hi Alistair, 
This is something that I need, care to share any pointers about how you solved this?

Thanks, 
- Blaed

On Wednesday, April 26, 2017 at 1:00:40 PM UTC-5, Alistair Israel wrote:
I've been trying to deploy a small, sample Rails 5 app that uses ActionCable to send some real-time updates via WebSockets to the front-end Javascript. It works fine when running locally, or on single node test environments running single-node Redis as well.

When I tried to deploy it on to Kubernetes, however, I ran into a small stumbling block because to deploy Redis on to Kubernetes in a highly available manner it seems the recommended way is through use of Redis Sentinels:

* <a href="https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkubernetes%2Fkubernetes%2Ftree%2Fmaster%2Fexamples%2Fstorage%2Fredis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFtsG_VjvPPmfvsv0itCIKn7U5kOA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkubernetes%2Fkubernetes%2Ftree%2Fmaster%2Fexamples%2Fstorage%2Fredis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFtsG_VjvPPmfvsv0itCIKn7U5kOA&#39;;return true;">https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis
* <a href="https://redis.io/topics/sentinel" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fredis.io%2Ftopics%2Fsentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFDDXI-H5H3je9C0_TSlxroVzjmkQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fredis.io%2Ftopics%2Fsentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFDDXI-H5H3je9C0_TSlxroVzjmkQ&#39;;return true;">https://redis.io/topics/sentinel

redis-rails already supports sentinels, so use of Redis with sentinels for cache, sessions, etc. is not a problem: <a href="https://github.com/redis-store/redis-rails#usage-with-redis-sentinel" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fredis-store%2Fredis-rails%23usage-with-redis-sentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGOpGD99ZfVl_5xVhIS3bp0BEPWNg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fredis-store%2Fredis-rails%23usage-with-redis-sentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGOpGD99ZfVl_5xVhIS3bp0BEPWNg&#39;;return true;">https://github.com/redis-store/redis-rails#usage-with-redis-sentinel

However, the ActionCable Redis subscription adapter is configured independently, and when I dived into the code it became apparent that the configuration did not currently support specifying Redis sentinel configuration (even though the underlying redis gem powering both ActionCable and redis-rails already does).

It looks like it should only take a couple of lines of additional code at <a href="https://github.com/rails/rails/blob/master/actioncable/lib/action_cable/subscription_adapter/redis.rb#L13" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fblob%2Fmaster%2Factioncable%2Flib%2Faction_cable%2Fsubscription_adapter%2Fredis.rb%23L13\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE8tq-ixyxPIhvoL3peceMsh4CJAw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fblob%2Fmaster%2Factioncable%2Flib%2Faction_cable%2Fsubscription_adapter%2Fredis.rb%23L13\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE8tq-ixyxPIhvoL3peceMsh4CJAw&#39;;return true;">https://github.com/rails/rails/blob/master/actioncable/lib/action_cable/subscription_adapter/redis.rb#L13 to support an additional `sentinels: [...]` configuration parameter (if present).

I realize that I could simply supply my own custom lambda to ActionCable::SubscriptionAdapter::Redis.redis_connector in an initializer to support sentinels there, but am wondering if this is something the rest of the community might appreciate. If so, I'll gladly raise a PR.

Cheers!
- alistair

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [feature] Add sentinel support to ActionCable Redis subscription adapter

Alistair Israel
Hi, Blaed. Apologies that I only saw this now (since Google groups doesn't actually notify you if someone replies to your email thread).

Anyway, yes, it's relatively easy. I ended up just supplying a custom lambda to ActionCable::SubscriptionAdapter::Redis.redis_connector. I made a file named config/initializers/action_cable.rb and in there wrote:

require 'action_cable/subscription_adapter/redis'

ActionCable::SubscriptionAdapter::Redis.redis_connector = lambda do |config|
  redis_opts
= { url: config[:url] }
  redis_opts
[:sentinels] = config[:sentinels] if config.key?(:sentinels)
 
::Redis.new(redis_opts)
end


Basically, check if an optional "sentinels" key was placed in config/cable.yml, and use that to configure the Redis connector accordingly. For example:

production:
  adapter
: redis
  url
: redis://mymaster
  sentinels
:
 
- host: redis-sentinel
    port
: 26379


Hope this helps!

On Friday, August 25, 2017 at 10:41:27 PM UTC+8, [hidden email] wrote:
Hi Alistair, 
This is something that I need, care to share any pointers about how you solved this?

Thanks, 
- Blaed

On Wednesday, April 26, 2017 at 1:00:40 PM UTC-5, Alistair Israel wrote:
I've been trying to deploy a small, sample Rails 5 app that uses ActionCable to send some real-time updates via WebSockets to the front-end Javascript. It works fine when running locally, or on single node test environments running single-node Redis as well.

When I tried to deploy it on to Kubernetes, however, I ran into a small stumbling block because to deploy Redis on to Kubernetes in a highly available manner it seems the recommended way is through use of Redis Sentinels:

* <a href="https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkubernetes%2Fkubernetes%2Ftree%2Fmaster%2Fexamples%2Fstorage%2Fredis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFtsG_VjvPPmfvsv0itCIKn7U5kOA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkubernetes%2Fkubernetes%2Ftree%2Fmaster%2Fexamples%2Fstorage%2Fredis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFtsG_VjvPPmfvsv0itCIKn7U5kOA&#39;;return true;">https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis
* <a href="https://redis.io/topics/sentinel" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fredis.io%2Ftopics%2Fsentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFDDXI-H5H3je9C0_TSlxroVzjmkQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fredis.io%2Ftopics%2Fsentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFDDXI-H5H3je9C0_TSlxroVzjmkQ&#39;;return true;">https://redis.io/topics/sentinel

redis-rails already supports sentinels, so use of Redis with sentinels for cache, sessions, etc. is not a problem: <a href="https://github.com/redis-store/redis-rails#usage-with-redis-sentinel" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fredis-store%2Fredis-rails%23usage-with-redis-sentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGOpGD99ZfVl_5xVhIS3bp0BEPWNg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fredis-store%2Fredis-rails%23usage-with-redis-sentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGOpGD99ZfVl_5xVhIS3bp0BEPWNg&#39;;return true;">https://github.com/redis-store/redis-rails#usage-with-redis-sentinel

However, the ActionCable Redis subscription adapter is configured independently, and when I dived into the code it became apparent that the configuration did not currently support specifying Redis sentinel configuration (even though the underlying redis gem powering both ActionCable and redis-rails already does).

It looks like it should only take a couple of lines of additional code at <a href="https://github.com/rails/rails/blob/master/actioncable/lib/action_cable/subscription_adapter/redis.rb#L13" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fblob%2Fmaster%2Factioncable%2Flib%2Faction_cable%2Fsubscription_adapter%2Fredis.rb%23L13\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE8tq-ixyxPIhvoL3peceMsh4CJAw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fblob%2Fmaster%2Factioncable%2Flib%2Faction_cable%2Fsubscription_adapter%2Fredis.rb%23L13\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE8tq-ixyxPIhvoL3peceMsh4CJAw&#39;;return true;">https://github.com/rails/rails/blob/master/actioncable/lib/action_cable/subscription_adapter/redis.rb#L13 to support an additional `sentinels: [...]` configuration parameter (if present).

I realize that I could simply supply my own custom lambda to ActionCable::SubscriptionAdapter::Redis.redis_connector in an initializer to support sentinels there, but am wondering if this is something the rest of the community might appreciate. If so, I'll gladly raise a PR.

Cheers!
- alistair

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [feature] Add sentinel support to ActionCable Redis subscription adapter

blaed
Hey Alistair, 
No problem at all! That was nearly the exact solution I ended up using, so hopefully this is helpful to any other googlers

On Fri, Apr 27, 2018 at 8:39 PM Alistair Israel <[hidden email]> wrote:
Hi, Blaed. Apologies that I only saw this now (since Google groups doesn't actually notify you if someone replies to your email thread).

Anyway, yes, it's relatively easy. I ended up just supplying a custom lambda to ActionCable::SubscriptionAdapter::Redis.redis_connector. I made a file named config/initializers/action_cable.rb and in there wrote:

require 'action_cable/subscription_adapter/redis'

ActionCable::SubscriptionAdapter::Redis.redis_connector = lambda do |config|
  redis_opts
= { url: config[:url] }
  redis_opts
[:sentinels] = config[:sentinels] if config.key?(:sentinels)
 
::Redis.new(redis_opts)
end


Basically, check if an optional "sentinels" key was placed in config/cable.yml, and use that to configure the Redis connector accordingly. For example:

production:
  adapter
: redis
  url
: redis://mymaster
  sentinels
:
 
- host: redis-sentinel
    port
: 26379


Hope this helps!

On Friday, August 25, 2017 at 10:41:27 PM UTC+8, [hidden email] wrote:
Hi Alistair, 
This is something that I need, care to share any pointers about how you solved this?

Thanks, 
- Blaed

On Wednesday, April 26, 2017 at 1:00:40 PM UTC-5, Alistair Israel wrote:
I've been trying to deploy a small, sample Rails 5 app that uses ActionCable to send some real-time updates via WebSockets to the front-end Javascript. It works fine when running locally, or on single node test environments running single-node Redis as well.

When I tried to deploy it on to Kubernetes, however, I ran into a small stumbling block because to deploy Redis on to Kubernetes in a highly available manner it seems the recommended way is through use of Redis Sentinels:

* https://redis.io/topics/sentinel

redis-rails already supports sentinels, so use of Redis with sentinels for cache, sessions, etc. is not a problem: https://github.com/redis-store/redis-rails#usage-with-redis-sentinel

However, the ActionCable Redis subscription adapter is configured independently, and when I dived into the code it became apparent that the configuration did not currently support specifying Redis sentinel configuration (even though the underlying redis gem powering both ActionCable and redis-rails already does).

It looks like it should only take a couple of lines of additional code at https://github.com/rails/rails/blob/master/actioncable/lib/action_cable/subscription_adapter/redis.rb#L13 to support an additional `sentinels: [...]` configuration parameter (if present).

I realize that I could simply supply my own custom lambda to ActionCable::SubscriptionAdapter::Redis.redis_connector in an initializer to support sentinels there, but am wondering if this is something the rest of the community might appreciate. If so, I'll gladly raise a PR.

Cheers!
- alistair

--
You received this message because you are subscribed to a topic in the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rubyonrails-core/-Wrpc51Es4E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
zac
Reply | Threaded
Open this post in threaded view
|

Re: [feature] Add sentinel support to ActionCable Redis subscription adapter

zac
This is fantastic, thank you both!

On Tuesday, July 10, 2018 at 9:40:53 AM UTC-4, Blaed Johnston wrote:
Hey Alistair, 
No problem at all! That was nearly the exact solution I ended up using, so hopefully this is helpful to any other googlers

On Fri, Apr 27, 2018 at 8:39 PM Alistair Israel <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="fj42sv_1CQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ais...@...> wrote:
Hi, Blaed. Apologies that I only saw this now (since Google groups doesn't actually notify you if someone replies to your email thread).

Anyway, yes, it's relatively easy. I ended up just supplying a custom lambda to ActionCable::SubscriptionAdapter::Redis.redis_connector. I made a file named config/initializers/action_cable.rb and in there wrote:

require 'action_cable/subscription_adapter/redis'

ActionCable::SubscriptionAdapter::Redis.redis_connector = lambda do |config|
  redis_opts
= { url: config[:url] }
  redis_opts
[:sentinels] = config[:sentinels] if config.key?(:sentinels)
 
::Redis.new(redis_opts)
end


Basically, check if an optional "sentinels" key was placed in config/cable.yml, and use that to configure the Redis connector accordingly. For example:

production:
  adapter
: redis
  url
: redis://mymaster
  sentinels
:
 
- host: redis-sentinel
    port
: 26379


Hope this helps!

On Friday, August 25, 2017 at 10:41:27 PM UTC+8, [hidden email] wrote:
Hi Alistair, 
This is something that I need, care to share any pointers about how you solved this?

Thanks, 
- Blaed

On Wednesday, April 26, 2017 at 1:00:40 PM UTC-5, Alistair Israel wrote:
I've been trying to deploy a small, sample Rails 5 app that uses ActionCable to send some real-time updates via WebSockets to the front-end Javascript. It works fine when running locally, or on single node test environments running single-node Redis as well.

When I tried to deploy it on to Kubernetes, however, I ran into a small stumbling block because to deploy Redis on to Kubernetes in a highly available manner it seems the recommended way is through use of Redis Sentinels:

* <a href="https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkubernetes%2Fkubernetes%2Ftree%2Fmaster%2Fexamples%2Fstorage%2Fredis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFtsG_VjvPPmfvsv0itCIKn7U5kOA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkubernetes%2Fkubernetes%2Ftree%2Fmaster%2Fexamples%2Fstorage%2Fredis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFtsG_VjvPPmfvsv0itCIKn7U5kOA&#39;;return true;">https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis
* <a href="https://redis.io/topics/sentinel" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fredis.io%2Ftopics%2Fsentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFDDXI-H5H3je9C0_TSlxroVzjmkQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fredis.io%2Ftopics%2Fsentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFDDXI-H5H3je9C0_TSlxroVzjmkQ&#39;;return true;">https://redis.io/topics/sentinel

redis-rails already supports sentinels, so use of Redis with sentinels for cache, sessions, etc. is not a problem: <a href="https://github.com/redis-store/redis-rails#usage-with-redis-sentinel" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fredis-store%2Fredis-rails%23usage-with-redis-sentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGOpGD99ZfVl_5xVhIS3bp0BEPWNg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fredis-store%2Fredis-rails%23usage-with-redis-sentinel\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGOpGD99ZfVl_5xVhIS3bp0BEPWNg&#39;;return true;">https://github.com/redis-store/redis-rails#usage-with-redis-sentinel

However, the ActionCable Redis subscription adapter is configured independently, and when I dived into the code it became apparent that the configuration did not currently support specifying Redis sentinel configuration (even though the underlying redis gem powering both ActionCable and redis-rails already does).

It looks like it should only take a couple of lines of additional code at <a href="https://github.com/rails/rails/blob/master/actioncable/lib/action_cable/subscription_adapter/redis.rb#L13" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fblob%2Fmaster%2Factioncable%2Flib%2Faction_cable%2Fsubscription_adapter%2Fredis.rb%23L13\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE8tq-ixyxPIhvoL3peceMsh4CJAw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fblob%2Fmaster%2Factioncable%2Flib%2Faction_cable%2Fsubscription_adapter%2Fredis.rb%23L13\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNE8tq-ixyxPIhvoL3peceMsh4CJAw&#39;;return true;">https://github.com/rails/rails/blob/master/actioncable/lib/action_cable/subscription_adapter/redis.rb#L13 to support an additional `sentinels: [...]` configuration parameter (if present).

I realize that I could simply supply my own custom lambda to ActionCable::SubscriptionAdapter::Redis.redis_connector in an initializer to support sentinels there, but am wondering if this is something the rest of the community might appreciate. If so, I'll gladly raise a PR.

Cheers!
- alistair

--
You received this message because you are subscribed to a topic in the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this topic, visit <a href="https://groups.google.com/d/topic/rubyonrails-core/-Wrpc51Es4E/unsubscribe" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/topic/rubyonrails-core/-Wrpc51Es4E/unsubscribe&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/topic/rubyonrails-core/-Wrpc51Es4E/unsubscribe&#39;;return true;">https://groups.google.com/d/topic/rubyonrails-core/-Wrpc51Es4E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="fj42sv_1CQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">rubyonrails-co...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="fj42sv_1CQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">rubyonra...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/rubyonrails-core" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/rubyonrails-core&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/rubyonrails-core&#39;;return true;">https://groups.google.com/group/rubyonrails-core.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.