[Rails Scaling] Add ability to easily have separate DB for engines

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[Rails Scaling] Add ability to easily have separate DB for engines

Kelly Stannard
Hello,

Summary

After watching the @eileencodes RailsConf keynote, I decided to resurrect an idea that Rails engines were the key to easy Rails scaling in larger companies since it promotes separation while being easier than SOA. Part of that separation is the need to have engines that can connect to their own database, promoting the portability of the engine.

Dan Manges articulates the gist of the engines only idea pretty well.
This is my original, functioning project from 2 years ago.
Current progress towards this goal.

Details of how an engine will work

Start with a separate, but otherwise normal, database.yml stored in the engine. The engine's namespaced ApplicationRecord will establish a connection using that database.yml. Table prefix is also dropped since it is in its own database.

I think there is probably a better, less verbose way to have runtime stuff happen, but just haven't figured it out yet.

In addition, we should expose new rake tasks, something like this from my original project.

Details of how the plugin generator will work

Right now I just added a `separate_database` flag so the full command would be:

be rails plugin --full --mountable --separate_database bar_service

Thanks for your time,

Kelly Wolf Stannard

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Rails Scaling] Add ability to easily have separate DB for engines

James Adam

It’s an interesting idea, but I’m not sure that a lot of people will need or even want to isolate engines into their own databases — in most cases, I’d expect people to want to link their application models to engine models (e.g. users to posts), which means they’d naturally sit in the same database.


I think you’ll need to make a very clear use-case for separating databases on a per-engine basis to get traction.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Rails Scaling] Add ability to easily have separate DB for engines

Kelly Stannard
Thank you for your response!

You are right in that single database is the easiest and most natural form of application. I think people reach for services way too early so don't expect an impassioned defense of SOA from me. That said, I think that multiple database is a feature that will have buy in for much the same reason SOA has buy in. Some people can't handle all their requests with a single database. For reference, this is Eileen's keynote:

https://youtu.be/8evXWvM4oXM?t=1130

Basically, think of this as SOA, but the applications are engines, and instead of communicating via HTTP, the applications communicate via method calls.

On Wednesday, June 20, 2018 at 11:03:19 AM UTC-4, James Adam wrote:

It’s an interesting idea, but I’m not sure that a lot of people will need or even want to isolate engines into their own databases — in most cases, I’d expect people to want to link their application models to engine models (e.g. users to posts), which means they’d naturally sit in the same database.


I think you’ll need to make a very clear use-case for separating databases on a per-engine basis to get traction.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Rails Scaling] Add ability to easily have separate DB for engines

Sachin Mudgal
I second Kelly's response. Even one of my current applications is having requirement of interacting with multiple database for different purposes.

The product(http://fanmojo.in) is itself on micro-service based architecture, where a few parts are developed as individual micro-services in Rails. Now, one Rails micro-service itself needs to interact with multiple databases for different purposes.

Say, Admin-Panel is micro-service in Rails, now it needs to interact in the following ways:
  1. MySQL for User-base type of data.
  2. DynamoDB/MongoDB for NoSQL type data.
  3. Redis for current state of Live/Volatile type of data.
So, a namespace based models could have easily separated out things.

Even while migrating our data via application interface(had to use application interface due to certain schema changes), we had do some little hack-type tricks to make it done(though it was easy stuff too).

On Wed, Jun 20, 2018 at 8:46 PM Kelly Stannard <[hidden email]> wrote:
Thank you for your response!

You are right in that single database is the easiest and most natural form of application. I think people reach for services way too early so don't expect an impassioned defense of SOA from me. That said, I think that multiple database is a feature that will have buy in for much the same reason SOA has buy in. Some people can't handle all their requests with a single database. For reference, this is Eileen's keynote:

https://youtu.be/8evXWvM4oXM?t=1130

Basically, think of this as SOA, but the applications are engines, and instead of communicating via HTTP, the applications communicate via method calls.

On Wednesday, June 20, 2018 at 11:03:19 AM UTC-4, James Adam wrote:

It’s an interesting idea, but I’m not sure that a lot of people will need or even want to isolate engines into their own databases — in most cases, I’d expect people to want to link their application models to engine models (e.g. users to posts), which means they’d naturally sit in the same database.


I think you’ll need to make a very clear use-case for separating databases on a per-engine basis to get traction.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Rails Scaling] Add ability to easily have separate DB for engines

Greg Barendt
For what it's worth, we'd be interested in per-engine database configuration as well.

We have an (admittedly unusually complicated) architecture in which it would be great for us to be able to package all of our authentication-related code into an engine so that each of our applications can have it isolated from the rest of their code. We're also using Oracle and storing our authentication-related code in a separate database schema, which means the per-engine isolation makes even more sense.

On Wednesday, June 20, 2018 at 6:31:25 PM UTC-4, Sachin Mudgal wrote:
I second Kelly's response. Even one of my current applications is having requirement of interacting with multiple database for different purposes.

The product(<a href="http://fanmojo.in" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffanmojo.in\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFaCcMWpiqWVWUk9-96fQGy0uVLDA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Ffanmojo.in\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFaCcMWpiqWVWUk9-96fQGy0uVLDA&#39;;return true;">http://fanmojo.in) is itself on micro-service based architecture, where a few parts are developed as individual micro-services in Rails. Now, one Rails micro-service itself needs to interact with multiple databases for different purposes.

Say, Admin-Panel is micro-service in Rails, now it needs to interact in the following ways:
  1. MySQL for User-base type of data.
  2. DynamoDB/MongoDB for NoSQL type data.
  3. Redis for current state of Live/Volatile type of data.
So, a namespace based models could have easily separated out things.

Even while migrating our data via application interface(had to use application interface due to certain schema changes), we had do some little hack-type tricks to make it done(though it was easy stuff too).

On Wed, Jun 20, 2018 at 8:46 PM Kelly Stannard <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="bHt-WQioAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">kwsta...@...> wrote:
Thank you for your response!

You are right in that single database is the easiest and most natural form of application. I think people reach for services way too early so don't expect an impassioned defense of SOA from me. That said, I think that multiple database is a feature that will have buy in for much the same reason SOA has buy in. Some people can't handle all their requests with a single database. For reference, this is Eileen's keynote:

<a href="https://youtu.be/8evXWvM4oXM?t=1130" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://youtu.be/8evXWvM4oXM?t\x3d1130&#39;;return true;" onclick="this.href=&#39;https://youtu.be/8evXWvM4oXM?t\x3d1130&#39;;return true;">https://youtu.be/8evXWvM4oXM?t=1130

Basically, think of this as SOA, but the applications are engines, and instead of communicating via HTTP, the applications communicate via method calls.

On Wednesday, June 20, 2018 at 11:03:19 AM UTC-4, James Adam wrote:

It’s an interesting idea, but I’m not sure that a lot of people will need or even want to isolate engines into their own databases — in most cases, I’d expect people to want to link their application models to engine models (e.g. users to posts), which means they’d naturally sit in the same database.


I think you’ll need to make a very clear use-case for separating databases on a per-engine basis to get traction.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bHt-WQioAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">rubyonrails-co...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="bHt-WQioAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">rubyonra...@googlegroups.com.
Visit this group at <a href="https://groups.google.com/group/rubyonrails-core" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/group/rubyonrails-core&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/group/rubyonrails-core&#39;;return true;">https://groups.google.com/group/rubyonrails-core.
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/d/optout&#39;;return true;">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Rails Scaling] Add ability to easily have separate DB for engines

Dan Manges-2
In reply to this post by Kelly Stannard
This is an interesting idea, thanks for sharing it. We use multiple databases in our Rails app at Root using https://github.com/ankane/multiverse

Having the database.yml inside the engine directory itself might be problematic. For us at Root, that'd work well since all of our engines are inside our project. How would this work for engines loaded via gems? It'd be hard for somebody to include an engine in their app via a gem and then modify the database.yml. I wonder if it'd be better to have the top-level config/database.yml define the credentials for the engine database somehow. An engine-owned file could be built to use environment variables, but we've also defined other database connection parameters in the yaml file, like disabling prepared statements and setting statement timeouts and the connection pool size.

Figuring out how to manage the connections should be solvable. After that though, Rails will still need to solve all of the migration issues mentioned in the @eileencodes RailsConf keynote. From my perspective, that's the bigger challenge. Rails' current approach to migrations assumes one database, and that's why gems like multiverse are necessary to use multiple. Using establish_connection is pretty elegant, but the migrations code is too tightly coupled to one database connection set on ActiveRecord::Base. From my perspective it'd be nice to resolve that first. Ideally, all of the migration code should be able to work by passing in a database connection, rather than needing to modify the global state set on ActiveRecord::Base.

Dan


On Sunday, May 27, 2018 at 9:43:18 PM UTC-4, Kelly Stannard wrote:
Hello,

Summary

After watching the @eileencodes RailsConf keynote, I decided to resurrect an idea that Rails engines were the key to easy Rails scaling in larger companies since it promotes separation while being easier than SOA. Part of that separation is the need to have engines that can connect to their own database, promoting the portability of the engine.

<a href="https://medium.com/@dan_manges/the-modular-monolith-rails-architecture-fb1023826fc4" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fmedium.com%2F%40dan_manges%2Fthe-modular-monolith-rails-architecture-fb1023826fc4\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH5RI6i679byUNg6pDNKNnayze1ug&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fmedium.com%2F%40dan_manges%2Fthe-modular-monolith-rails-architecture-fb1023826fc4\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH5RI6i679byUNg6pDNKNnayze1ug&#39;;return true;">Dan Manges articulates the gist of the engines only idea pretty well.
<a href="https://github.com/kwstannard/engine_only_app_test" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkwstannard%2Fengine_only_app_test\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGnur5Nmuq_l_RXEqK1DzRPahmSGg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkwstannard%2Fengine_only_app_test\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGnur5Nmuq_l_RXEqK1DzRPahmSGg&#39;;return true;">This is my original, functioning project from 2 years ago.
<a href="https://github.com/rails/rails/compare/master...kwstannard:separate_db_engine" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fcompare%2Fmaster...kwstannard%3Aseparate_db_engine\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGrNAwj7mrhSQs4e6C2ouSHWYCSYA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fcompare%2Fmaster...kwstannard%3Aseparate_db_engine\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGrNAwj7mrhSQs4e6C2ouSHWYCSYA&#39;;return true;">Current progress towards this goal.

Details of how an engine will work

Start with a separate, but otherwise normal, database.yml stored in the engine. The engine's namespaced ApplicationRecord will establish a connection using that database.yml. Table prefix is also dropped since it is in its own database.

I think there is probably a better, less verbose way to have runtime stuff happen, but just haven't figured it out yet.

In addition, we should expose new rake tasks, <a href="https://github.com/kwstannard/engine_only_app_test/blob/master/engine_only_helpers/lib/tasks/engine_only_helpers/db.rake" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkwstannard%2Fengine_only_app_test%2Fblob%2Fmaster%2Fengine_only_helpers%2Flib%2Ftasks%2Fengine_only_helpers%2Fdb.rake\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEG6pQjevVtLCoZJobpfCv9wcG3bQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkwstannard%2Fengine_only_app_test%2Fblob%2Fmaster%2Fengine_only_helpers%2Flib%2Ftasks%2Fengine_only_helpers%2Fdb.rake\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEG6pQjevVtLCoZJobpfCv9wcG3bQ&#39;;return true;">something like this from my original project.

Details of how the plugin generator will work

Right now I just added a `separate_database` flag so the full command would be:

be rails plugin --full --mountable --separate_database bar_service

Thanks for your time,

Kelly Wolf Stannard

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Rails Scaling] Add ability to easily have separate DB for engines

Andrew Kaspick
I believe rails 6 will have multiple DB support baked in by default... https://speakerdeck.com/eileencodes/railsconf-2018-the-future-of-rails-6-scalable-by-default

On Mon, Jun 25, 2018, 6:26 AM Dan Manges, <[hidden email]> wrote:
This is an interesting idea, thanks for sharing it. We use multiple databases in our Rails app at Root using https://github.com/ankane/multiverse

Having the database.yml inside the engine directory itself might be problematic. For us at Root, that'd work well since all of our engines are inside our project. How would this work for engines loaded via gems? It'd be hard for somebody to include an engine in their app via a gem and then modify the database.yml. I wonder if it'd be better to have the top-level config/database.yml define the credentials for the engine database somehow. An engine-owned file could be built to use environment variables, but we've also defined other database connection parameters in the yaml file, like disabling prepared statements and setting statement timeouts and the connection pool size.

Figuring out how to manage the connections should be solvable. After that though, Rails will still need to solve all of the migration issues mentioned in the @eileencodes RailsConf keynote. From my perspective, that's the bigger challenge. Rails' current approach to migrations assumes one database, and that's why gems like multiverse are necessary to use multiple. Using establish_connection is pretty elegant, but the migrations code is too tightly coupled to one database connection set on ActiveRecord::Base. From my perspective it'd be nice to resolve that first. Ideally, all of the migration code should be able to work by passing in a database connection, rather than needing to modify the global state set on ActiveRecord::Base.

Dan


On Sunday, May 27, 2018 at 9:43:18 PM UTC-4, Kelly Stannard wrote:
Hello,

Summary

After watching the @eileencodes RailsConf keynote, I decided to resurrect an idea that Rails engines were the key to easy Rails scaling in larger companies since it promotes separation while being easier than SOA. Part of that separation is the need to have engines that can connect to their own database, promoting the portability of the engine.

Dan Manges articulates the gist of the engines only idea pretty well.
This is my original, functioning project from 2 years ago.
Current progress towards this goal.

Details of how an engine will work

Start with a separate, but otherwise normal, database.yml stored in the engine. The engine's namespaced ApplicationRecord will establish a connection using that database.yml. Table prefix is also dropped since it is in its own database.

I think there is probably a better, less verbose way to have runtime stuff happen, but just haven't figured it out yet.

In addition, we should expose new rake tasks, something like this from my original project.

Details of how the plugin generator will work

Right now I just added a `separate_database` flag so the full command would be:

be rails plugin --full --mountable --separate_database bar_service

Thanks for your time,

Kelly Wolf Stannard

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Rails Scaling] Add ability to easily have separate DB for engines

Kelly Stannard
In reply to this post by Dan Manges-2
Thank you for joining in Dan.

Most likely, we need namespaced ENV variables. If you think of each engine as a separate application the most obvious thing would be to control each connection like you would a normal application's connection, except you just namespace it.

I linked to my database migration solution in the OP. Since the engines are basically separate applications, the migrations will be run separately and possibly even concurrently.

On Monday, June 25, 2018 at 6:26:47 AM UTC-4, Dan Manges wrote:
This is an interesting idea, thanks for sharing it. We use multiple databases in our Rails app at Root using <a href="https://github.com/ankane/multiverse" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fankane%2Fmultiverse\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHLuS4a931CPVIdC3PTeLkPmPftfA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fankane%2Fmultiverse\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHLuS4a931CPVIdC3PTeLkPmPftfA&#39;;return true;">https://github.com/ankane/multiverse

Having the database.yml inside the engine directory itself might be problematic. For us at Root, that'd work well since all of our engines are inside our project. How would this work for engines loaded via gems? It'd be hard for somebody to include an engine in their app via a gem and then modify the database.yml. I wonder if it'd be better to have the top-level config/database.yml define the credentials for the engine database somehow. An engine-owned file could be built to use environment variables, but we've also defined other database connection parameters in the yaml file, like disabling prepared statements and setting statement timeouts and the connection pool size.

Figuring out how to manage the connections should be solvable. After that though, Rails will still need to solve all of the migration issues mentioned in the @eileencodes RailsConf keynote. From my perspective, that's the bigger challenge. Rails' current approach to migrations assumes one database, and that's why gems like multiverse are necessary to use multiple. Using establish_connection is pretty elegant, but the migrations code is too tightly coupled to one database connection set on ActiveRecord::Base. From my perspective it'd be nice to resolve that first. Ideally, all of the migration code should be able to work by passing in a database connection, rather than needing to modify the global state set on ActiveRecord::Base.

Dan


On Sunday, May 27, 2018 at 9:43:18 PM UTC-4, Kelly Stannard wrote:
Hello,

Summary

After watching the @eileencodes RailsConf keynote, I decided to resurrect an idea that Rails engines were the key to easy Rails scaling in larger companies since it promotes separation while being easier than SOA. Part of that separation is the need to have engines that can connect to their own database, promoting the portability of the engine.

<a href="https://medium.com/@dan_manges/the-modular-monolith-rails-architecture-fb1023826fc4" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fmedium.com%2F%40dan_manges%2Fthe-modular-monolith-rails-architecture-fb1023826fc4\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH5RI6i679byUNg6pDNKNnayze1ug&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fmedium.com%2F%40dan_manges%2Fthe-modular-monolith-rails-architecture-fb1023826fc4\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH5RI6i679byUNg6pDNKNnayze1ug&#39;;return true;">Dan Manges articulates the gist of the engines only idea pretty well.
<a href="https://github.com/kwstannard/engine_only_app_test" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkwstannard%2Fengine_only_app_test\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGnur5Nmuq_l_RXEqK1DzRPahmSGg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkwstannard%2Fengine_only_app_test\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGnur5Nmuq_l_RXEqK1DzRPahmSGg&#39;;return true;">This is my original, functioning project from 2 years ago.
<a href="https://github.com/rails/rails/compare/master...kwstannard:separate_db_engine" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fcompare%2Fmaster...kwstannard%3Aseparate_db_engine\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGrNAwj7mrhSQs4e6C2ouSHWYCSYA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Frails%2Frails%2Fcompare%2Fmaster...kwstannard%3Aseparate_db_engine\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGrNAwj7mrhSQs4e6C2ouSHWYCSYA&#39;;return true;">Current progress towards this goal.

Details of how an engine will work

Start with a separate, but otherwise normal, database.yml stored in the engine. The engine's namespaced ApplicationRecord will establish a connection using that database.yml. Table prefix is also dropped since it is in its own database.

I think there is probably a better, less verbose way to have runtime stuff happen, but just haven't figured it out yet.

In addition, we should expose new rake tasks, <a href="https://github.com/kwstannard/engine_only_app_test/blob/master/engine_only_helpers/lib/tasks/engine_only_helpers/db.rake" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkwstannard%2Fengine_only_app_test%2Fblob%2Fmaster%2Fengine_only_helpers%2Flib%2Ftasks%2Fengine_only_helpers%2Fdb.rake\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEG6pQjevVtLCoZJobpfCv9wcG3bQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2Fkwstannard%2Fengine_only_app_test%2Fblob%2Fmaster%2Fengine_only_helpers%2Flib%2Ftasks%2Fengine_only_helpers%2Fdb.rake\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEG6pQjevVtLCoZJobpfCv9wcG3bQ&#39;;return true;">something like this from my original project.

Details of how the plugin generator will work

Right now I just added a `separate_database` flag so the full command would be:

be rails plugin --full --mountable --separate_database bar_service

Thanks for your time,

Kelly Wolf Stannard

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Rails Scaling] Add ability to easily have separate DB for engines

Greg Navis
In reply to this post by Kelly Stannard
Could this be address with ConnectionHandler? I imagine models in each engine could call with "establish_connection :my_engine" (or inherit from a base class a'la ApplicationRecord) and then database.yml could provide per-engine connection settings. If an engine needs to use the "default" database then it'd just inherit the settings for production.

Best regards
Greg Navis

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

Re: [Rails Scaling] Add ability to easily have separate DB for engines

Kelly Stannard
Yes, I didn't do the ApplicationRecord trick in my original project, but that plus connection handler seems like the best way to handle the connection on a per-engine basis.

On Monday, July 9, 2018 at 3:22:43 PM UTC-4, Greg Navis wrote:
Could this be address with <a href="http://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionHandler.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fapi.rubyonrails.org%2Fclasses%2FActiveRecord%2FConnectionAdapters%2FConnectionHandler.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFHnv7Y7OyFjwWTAIyFlNqFB_cWdQ&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fapi.rubyonrails.org%2Fclasses%2FActiveRecord%2FConnectionAdapters%2FConnectionHandler.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFHnv7Y7OyFjwWTAIyFlNqFB_cWdQ&#39;;return true;">ConnectionHandler? I imagine models in each engine could call with "establish_connection :my_engine" (or inherit from a base class a'la ApplicationRecord) and then database.yml could provide per-engine connection settings. If an engine needs to use the "default" database then it'd just inherit the settings for production.

Best regards
Greg Navis

--
You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/rubyonrails-core.
For more options, visit https://groups.google.com/d/optout.