Anything wrong with shelling out to jruby?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Anything wrong with shelling out to jruby?

John Joseph Bachir
I have an application where I would like to shell out to jruby. A very simple script that essentially does this a dozen times serially:

begin
  `rails runner scripts/foo.rb`
rescue
  # .. report error
end

I remember hearing about problems invoking jruby from jruby. Some initial basic tests in my dev environment don't show any problems.

My guess at what the problem is: if jruby is configured to take an application's worth of RAM (say 512M), then the small out script will take that and then each individual sub process will take that as well.

I wonder if in practice this won't matter much, because the OS will move most of the outer instance's memory into VM and just keep it there?

Thanks for any insights or pointers to other info about this!

John
Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

keithrbennett
I don’t know of any problems shelling out to JRuby, but it will require the memory and startup time of starting a whole new JVM for each time you do.  I would not be comforted by the fact that the OS might save your first JVM’s RAM to disk and restore it later; that would take time.  Is speed not important in your case?

If I were you, I’d look at minimum to shelling out once rather than a dozen times.  Why do you want to shell out so many times?

Also, your script looks like it shells out only once, no?

- Keith

> On Mar 16, 2015, at 6:45 PM, John Joseph Bachir <[hidden email]> wrote:
>
> I have an application where I would like to shell out to jruby. A very simple script that essentially does this a dozen times serially:
>
> begin
>   `rails runner scripts/foo.rb`
> rescue
>   # .. report error
> end
>
> I remember hearing about problems invoking jruby from jruby. Some initial basic tests in my dev environment don't show any problems.
>
> My guess at what the problem is: if jruby is configured to take an application's worth of RAM (say 512M), then the small out script will take that and then each individual sub process will take that as well.
>
> I wonder if in practice this won't matter much, because the OS will move most of the outer instance's memory into VM and just keep it there?
>
> Thanks for any insights or pointers to other info about this!
>
> John


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email


Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

Ted Han

On Tue, Mar 17, 2015 at 9:07 AM, Keith Bennett <[hidden email]> wrote:
I don’t know of any problems shelling out to JRuby, but it will require the memory and startup time of starting a whole new JVM for each time you do.  I would not be comforted by the fact that the OS might save your first JVM’s RAM to disk and restore it later; that would take time.  Is speed not important in your case?

If I were you, I’d look at minimum to shelling out once rather than a dozen times.  Why do you want to shell out so many times?

Also, your script looks like it shells out only once, no?

- Keith

> On Mar 16, 2015, at 6:45 PM, John Joseph Bachir <[hidden email]> wrote:
>
> I have an application where I would like to shell out to jruby. A very simple script that essentially does this a dozen times serially:
>
> begin
>   `rails runner scripts/foo.rb`
> rescue
>   # .. report error
> end
>
> I remember hearing about problems invoking jruby from jruby. Some initial basic tests in my dev environment don't show any problems.
>
> My guess at what the problem is: if jruby is configured to take an application's worth of RAM (say 512M), then the small out script will take that and then each individual sub process will take that as well.
>
> I wonder if in practice this won't matter much, because the OS will move most of the outer instance's memory into VM and just keep it there?
>
> Thanks for any insights or pointers to other info about this!
>
> John


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email



Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

keithrbennett
Ted -

That text mentions Nailgun; is Nailgun use still supported/recommended?  I thought no.

Thanks,
Keith

On Mar 17, 2015, at 9:10 AM, Ted Han <[hidden email]> wrote:


On Tue, Mar 17, 2015 at 9:07 AM, Keith Bennett <[hidden email]> wrote:
I don’t know of any problems shelling out to JRuby, but it will require the memory and startup time of starting a whole new JVM for each time you do.  I would not be comforted by the fact that the OS might save your first JVM’s RAM to disk and restore it later; that would take time.  Is speed not important in your case?

If I were you, I’d look at minimum to shelling out once rather than a dozen times.  Why do you want to shell out so many times?

Also, your script looks like it shells out only once, no?

- Keith

> On Mar 16, 2015, at 6:45 PM, John Joseph Bachir <[hidden email]> wrote:
>
> I have an application where I would like to shell out to jruby. A very simple script that essentially does this a dozen times serially:
>
> begin
>   `rails runner scripts/foo.rb`
> rescue
>   # .. report error
> end
>
> I remember hearing about problems invoking jruby from jruby. Some initial basic tests in my dev environment don't show any problems.
>
> My guess at what the problem is: if jruby is configured to take an application's worth of RAM (say 512M), then the small out script will take that and then each individual sub process will take that as well.
>
> I wonder if in practice this won't matter much, because the OS will move most of the outer instance's memory into VM and just keep it there?
>
> Thanks for any insights or pointers to other info about this!
>
> John


---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email




Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

John Joseph Bachir

On Tue, Mar 17, 2015 at 9:07 AM, Keith Bennett <[hidden email]> wrote:
Is speed not important in your case?

It's not important. These are batch jobs that run nightly. This master script runs a few sub scripts, each of which do a few calculations and then queue up many (more efficient and well-structured) jobs in my background job system (sidekiq). Eventually I'd like to make the work in the sub scripts be done in workers as well, but for the time being I'd like to just keep things as-is and bring all the invocations into one place, and then start refactoring.


If I were you, I’d look at minimum to shelling out once rather than a dozen times.  Why do you want to shell out so many times?

As described above, I have many sub scripts which work and I don't want to refactor them in this first phase.

 
Also, your script looks like it shells out only once, no?

That's an example of the code that I'll be doing about a dozen times in the script.



On Tue, Mar 17, 2015 at 9:10 AM, Ted Han <[hidden email]> wrote:

“we will attempt to run it in the same JVM using a new JRuby instance”

Great news, thanks! FYI if others are interested, this feature can be set at the commandline[1]:

    JRUBY_OPTS="-J-Djruby.launch.inproc=true" rails runner scripts/foo.rb

Or within your ruby code[2]:

    # scripts/foo.rb
    require 'jruby'
    JRuby.runtime.instance_config.run_ruby_in_process = true

Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

John Joseph Bachir
Some simple tests at the heroku console show that this feature doesn't help performance at all:

> JRuby.runtime.instance_config.run_ruby_in_process = false
=> false
> Benchmark.measure{`ruby -e"puts 'foo'"`}
=>   0.020000   0.010000  14.980000 ( 17.739000)

> JRuby.runtime.instance_config.run_ruby_in_process = true
=> true
> Benchmark.measure{`ruby -e"puts 'foo'"`}
=>   0.010000   0.000000  14.740000 ( 17.536000)


I wonder if the in-process feature isn't being used for some reason? Any way to inspect this in either the inner or outer process?
Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

John Joseph Bachir
I made some more observations. I did an experiment similar to my March 17 email, but this time with `ruby -e"sleep 1-"`, and then I observed my process list to see if a new "java" was created and how much memory it took.

with JRuby.runtime.instance_config.run_ruby_in_process set to true or false, a second java instance was always created. The outer Java instance (running pry) took about 180 megs, and the inner about 80 megs.

Is this behavior expected? Let me know if there is more info I can provide.

John

Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

John Joseph Bachir

On Sat, Mar 21, 2015 at 1:46 PM, John Joseph Bachir <[hidden email]> wrote:
`ruby -e"sleep 1-"`

whoops, i mean  `ruby -e"sleep 10"`
Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

John Joseph Bachir
I also tried invoking pry like so:

  JRUBY_OPTS="-J-Djruby.launch.inproc=true" pry

Same behavior.
Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

Tim Uckun
Any word from the Jruby maintainers about this?

It seems to me that it could be a huge win for jruby if a new java instance was not launched both for speed and memory utilization.

On Sun, Mar 22, 2015 at 6:54 AM, John Joseph Bachir <[hidden email]> wrote:
I also tried invoking pry like so:

  JRUBY_OPTS="-J-Djruby.launch.inproc=true" pry

Same behavior.

Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

Christian MICHON
Regarding pry debugging, you should use pry-remote instead of pry. Indeed, you may have launched the jruby job as a service without a console attached to it.

On Sun, Mar 22, 2015 at 9:50 PM, Tim Uckun <[hidden email]> wrote:
Any word from the Jruby maintainers about this?

It seems to me that it could be a huge win for jruby if a new java instance was not launched both for speed and memory utilization.

On Sun, Mar 22, 2015 at 6:54 AM, John Joseph Bachir <[hidden email]> wrote:
I also tried invoking pry like so:

  JRUBY_OPTS="-J-Djruby.launch.inproc=true" pry

Same behavior.




--
Christian
Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

John Joseph Bachir

On Mon, Mar 23, 2015 at 3:02 AM, Christian MICHON <[hidden email]> wrote:
Regarding pry debugging, you should use pry-remote instead of pry. Indeed, you may have launched the jruby job as a service without a console attached to it.

Good thinking. I did another test, but am still finding a new JVM being created.

I wrote a script, shell-out-test.rb:

puts "starting script and sleeping for 5 seconds"
sleep 5
puts "shelling out for 10 seconds"
`ruby -e 'sleep 10'`
puts "done shelling out, sleeping another 5 seconds"
sleep 5
 

Then opened two terminals side-by-side. In one I ran:

  watch 'ps aux |grep [j]ava'

In the other I ran my script:

  ruby shell-out-test.rb

And I observed two java processes running while shelling out.

Reply | Threaded
Open this post in threaded view
|

Re: Anything wrong with shelling out to jruby?

John Joseph Bachir
Okay, in the IRC channel headius requested I make a ticket for this:


So I suppose it's considered broken for the time being.