Quantcast

[ruby-core:19846] [Bug #744] memory leak in callcc?

classic Classic list List threaded Threaded
141 messages Options
1234 ... 8
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:19846] [Bug #744] memory leak in callcc?

Dennis Ranke-2
Bug #744: memory leak in callcc?
http://redmine.ruby-lang.org/issues/show/744

Author: Roger Pack
Status: Open, Priority: Normal

from http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7896&group_id=426
this code
require 'generator'
loop {  g = Generator.new {|x| (1..3).each {|i| x.yield i}} }

seems to leak for me--I'm not sure if this is expected or not.

Thanks.


----------------------------------------
http://redmine.ruby-lang.org

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ruby-core:19846] [Bug #744] memory leak in callcc?

Brent Roman
Roger,

I've run into a number of issues related to Continuations and MRI's garbage collector,
so I thought I'd have a look at this one.  I investigated the equivalent (non-generator)
example described at:

http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7896&group_id=426

This:

   loop {@x = callcc {|c| c}}

quickly consumes all of memory.

One of my x86 Linux machines crashed after a couple minutes of running this
loop with a Segmentation Fault.  My guess here is that the stack, which during
GC is unchecked,  got too deep.

What I saw is that the stack during garbage collection became rediculously deep.
(>15000 frames deep in the GC)  Here's a bit of the backtrace:

#11768 0x0806401a in mark_locations_array (x=0xa887004, n=1228) at gc.c:437
#11769 0x0805dceb in thread_mark (th=0xa8860f8) at eval.c:7403
#11770 0x0806438e in rb_gc_mark (ptr=59) at gc.c:881
#11771 0x0806401a in mark_locations_array (x=0xa88924c, n=1228) at gc.c:437
#11772 0x0805dceb in thread_mark (th=0xa888340) at eval.c:7403
#11773 0x0806438e in rb_gc_mark (ptr=59) at gc.c:881
#...

This looks like rb_gc_mark() got passed a bogus VALUE pointer.
I cannot even unwind the stack to the point where this happened without
gdb itself segfaulting.

Interestinly, the very same Ruby interpreter running on an ARM9 under Linux handles
this case without leaking memory or segfaulting.  So, in answer to your original
question:

I don't think this behavior is intentional.

And, I plan to spend a bit more time looking into it.
Any hints would be appreciated...

- brent


Roger Pack-5 wrote
Bug #744: memory leak in callcc?
http://redmine.ruby-lang.org/issues/show/744

Author: Roger Pack
Status: Open, Priority: Normal

from http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7896&group_id=426
this code
require 'generator'
loop {  g = Generator.new {|x| (1..3).each {|i| x.yield i}} }

seems to leak for me--I'm not sure if this is expected or not.

Thanks.


----------------------------------------
http://redmine.ruby-lang.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ruby-core:19846] [Bug #744] memory leak in callcc?

Brent Roman
Roger,

This "leak" appears to be an artifact of MRI's conservative garbage collector.
Depending upon the compiler options and the target CPU, there may be
unused references to these continuation objects left on the thread's stack when
it is copied in whole to by the callcc method.

In your example, these unused references form a linked list of continuations
across the loop iterations.  When the GC tries to mark such a recursive structure, it
consumes a lot of stack space.  With Ruby 1.68, this leads to a segmentation
fault when the stack size exceeds the max allowed (see ulimit -s)
With later versions of Ruby, the mark phase of the GC will "give up"
when the stack grows too large.  This avoids the segfault, as GC silently
stops working instead.  At least that's what I think I see in v1.8.7 p72

When I built x86 Ruby using gcc without optimization  (CFLAGS=-g),
even this caused a memory leak:

  loop { callcc {|c| c}}

However, when I rebuilt it with CFLAGS=-O2, the memory leak only appeared
when the continuations returned by callcc where assigned to a variable.

When compiled for the ARM9 with gcc CFLAGS=-Os or CFLAGS=-O2, everything works as
it should.  No leaks observed.  However, when I changed to CFLAGS=-g or CFLAGS=-O3,
the original example leaks badly.  These tests were performed using x86 gcc v3.3.5 and
ARM gcc v3.4.5.

Can anyone suggest debugging techniques to help determine what is leaving the dangling
references to these continuations on Ruby's 'C' call stack?

If we knew what wrote these, we might be able to explicitly clear them once
they go out of scope.  This should result in better significantly GC performance all around.

- brent

Brent Roman wrote
Roger,

I've run into a number of issues related to Continuations and MRI's garbage collector,
so I thought I'd have a look at this one.  I investigated the equivalent (non-generator)
example described at:

http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7896&group_id=426

This:

   loop {@x = callcc {|c| c}}

quickly consumes all of memory.

One of my x86 Linux machines crashed after a couple minutes of running this
loop with a Segmentation Fault.  My guess here is that the stack, which during
GC is unchecked,  got too deep.


Roger Pack-5 wrote
Bug #744: memory leak in callcc?
http://redmine.ruby-lang.org/issues/show/744

Author: Roger Pack
Status: Open, Priority: Normal

from http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7896&group_id=426
this code
require 'generator'
loop {  g = Generator.new {|x| (1..3).each {|i| x.yield i}} }

seems to leak for me--I'm not sure if this is expected or not.

Thanks.


----------------------------------------
http://redmine.ruby-lang.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:19945] [Bug #744] memory leak in callcc?

Dennis Ranke-2
In reply to this post by Dennis Ranke-2
Issue #744 has been updated by Roger Pack.


> If we knew what wrote these, we might be able to explicitly clear them
> once
> they go out of scope.  This should result in better significantly GC
> performance all around.

Yeah I've wondered that too.  Maybe we can have a hackfest for it some saturday :)
http://redmine.ruby-lang.org/issues/show/649
is related [and somewhat frustrating to be honest].  My thought is that maybe there's a way to "clear the stack" of data that isn't currently "useful" and thus clear it of old references [I realize this may be hard].

Thoughts?
-=R
----------------------------------------
http://redmine.ruby-lang.org/issues/show/744

----------------------------------------
http://redmine.ruby-lang.org

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:19967] Re: [Bug #744] memory leak in callcc?

Ken Bloom-2
On Sun, 16 Nov 2008 02:59:18 +0900, Roger Pack wrote:

> Issue #744 has been updated by Roger Pack.
>
>
>> If we knew what wrote these, we might be able to explicitly clear them
>> once
>> they go out of scope.  This should result in better significantly GC
>> performance all around.
>
> Yeah I've wondered that too.  Maybe we can have a hackfest for it some
> saturday :) http://redmine.ruby-lang.org/issues/show/649 is related [and
> somewhat frustrating to be honest].  My thought is that maybe there's a
> way to "clear the stack" of data that isn't currently "useful" and thus
> clear it of old references [I realize this may be hard].

I can't reproduce http://redmine.ruby-lang.org/issues/show/649
in Debian's Ruby 1.8 or 1.9.

ruby1.8        1.8.7.72-1
ruby1.9        1.9.0.2-8

(The callcc thing, on the other hand, is broken on Debian's Ruby 1.8)

--
Chanoch (Ken) Bloom. PhD candidate. Linguistic Cognition Laboratory.
Department of Computer Science. Illinois Institute of Technology.
http://www.iit.edu/~kbloom1/


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ruby-core:19945] [Bug #744] memory leak in callcc?

Brent Roman
In reply to this post by Dennis Ranke-2
Roger,

Well, I just summarized the result of this Saturday's hackfest at:

http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7896&group_id=426

The main problem seems to be that 'C'/C++ compilers do not
initialize automatic variables, so one is bound to have old, unused, but
valid pointers left on the stack from previous at any point in time.
 By the way, even this will fix the example leak:

  loop {@x = callcc {|c| c}; 2*6+4}

Pretty silly, but it works for me.  And, the fact it works proves that
the issue is unused references left on the stack.

The behavior is a fundamental design weakness of conservative GC.  
We notice it most when managing large and/or highly connected objects
like threads, continuations and large arrays.

One could hack the gcc to force it to initialize automatic variables to zero
even though this violates the 'C' langauage spec.  But
I can't help feeling that there must be a better way.

In my on again off again quest to put Ruby on a diet, I'll probably
hack at this a bit more over to coming weeks, initially
on my patched v1.68 interpreter.  Thanks for the redmine link.
Found this hanging off it.  Looks better than the "reachability"
patches to GC:

http://softwareverify.com/ruby/customBuild/memtrack/index.html

Here are some interesting posts about this problem from outside
the Ruby world.  The first is especially relevant:

http://gcc.gnu.org/ml/java/2005-05/msg00265.html
http://www.red-bean.com/guile/guile/new/msg01070.html
http://www.digitalmars.com/rtl/gcdescr.html

- brent


Roger Pack-5 wrote
Issue #744 has been updated by Roger Pack.


> If we knew what wrote these, we might be able to explicitly clear them
> once
> they go out of scope.  This should result in better significantly GC
> performance all around.

Yeah I've wondered that too.  Maybe we can have a hackfest for it some saturday :)
http://redmine.ruby-lang.org/issues/show/649
is related [and somewhat frustrating to be honest].  My thought is that maybe there's a way to "clear the stack" of data that isn't currently "useful" and thus clear it of old references [I realize this may be hard].

Thoughts?
-=R
----------------------------------------
http://redmine.ruby-lang.org/issues/show/744

----------------------------------------
http://redmine.ruby-lang.org
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:19969] Re: [Bug #744] memory leak in callcc?

"Martin J. Dürst"
At 12:54 08/11/17, Brent Roman wrote:

>One could hack the gcc to force it to initialize automatic variables to zero
>even though this violates the 'C' langauage spec.

I haven't read the spec, but my guess (having worked on other specs)
is that the only thing that the 'C' language spec says is that the
value is undefined. A value that happens to be zero would still be
undefined, as far as I understand.

Regards,    Martin.



#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[hidden email]    


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ruby-core:19969] Re: [Bug #744] memory leak in callcc?

Brent Roman
Martin,

Well.  Ummm.  If a compiler writes zeros then is it not setting the
value of variables that the spec says should remain undefined until
explicitly initialized?

Whether or not it violates the 'C' language spec, I don't know any way to
make gcc do this with existing compiler options or pragmas.

Does anyone else?

- brent


Martin Duerst wrote
At 12:54 08/11/17, Brent Roman wrote:

>One could hack the gcc to force it to initialize automatic variables to zero
>even though this violates the 'C' langauage spec.

I haven't read the spec, but my guess (having worked on other specs)
is that the only thing that the 'C' language spec says is that the
value is undefined. A value that happens to be zero would still be
undefined, as far as I understand.

Regards,    Martin.



#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp    

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:19972] Re: [Bug #744] memory leak in callcc?

Kurt Stephens-2
A common technique is to allocate a reasonably sized array (256-bytes) on the C stack and zero it before and after each allocation.  This reduces garbage left on the stack before and after allocation and possible GC:

void *my_alloc(size_t size)
{
   char zeros[256];
   void *ptr;
   memset(zeros, 0, sizeof(zeros));
   ptr = my_alloc_inner(size);
   memset(zeros, 0, sizeof(zeros));
   return ptr;
}

void *my_alloc_inner(size_t size)
{
   /* may call GC */
}

Might need to put my_alloc_inner() in a separate compilation unit to avoid inlining.

Kurt

Brent Roman wrote:
Martin,

Well.  Ummm.  If a compiler writes zeros then is it not setting the
value of variables that the spec says should remain undefined until
explicitly initialized? 

Whether or not it violates the 'C' language spec, I don't know any way to 
make gcc do this with existing compiler options or pragmas.

Does anyone else?

- brent



Martin Duerst wrote:
  
At 12:54 08/11/17, Brent Roman wrote:

    
One could hack the gcc to force it to initialize automatic variables to
      
zero 
  
even though this violates the 'C' langauage spec.
      
I haven't read the spec, but my guess (having worked on other specs)
is that the only thing that the 'C' language spec says is that the
value is undefined. A value that happens to be zero would still be
undefined, as far as I understand.

Regards,    Martin.



#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       [hidden email]     




    

  

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Promising C coding techniques to reduce MRI's memory use

Brent Roman
After a couple weeks of long nights and false starts, I feel I may have come up with
a fix for a large class of Ruby memory leak.  The basic technique is a refinement of the
one Kurt Stephens suggested.  It not only eliminates the leaks in this one liner:

  loop {@x=callcc{|c|c}}

but also in our multi-threaded robotics application.  Our Ruby process used to grow
to 20+ MB during a day long run.  The same run now stays smaller than 10MB.
On an embedded ARM Linux machine with only 32MB of DRAM, this is a great result!

The central problem is that gcc (and other compilers) tend to create
sparse stack frames such that, when a new frame is pushed onto the stack, it does not
completely overwrite the one that had been previously stored there.  The new frame gets
activated with old VALUE pointers preserved inside its holes.  These become "live" again
as far as any conservative garbage collector is concerned.  And, viola, a leak is born!

I implemented a scheme for recording the maximum depth of the C stack in
xmalloc and during garbage collection itself.  However, I realized that
there was no point in clearing the stack when it is near its maximum depth.
Instead, stack clearing is deferred until CHECK_INTS, as this tends to happen
between evaluation of nodes, when the stack is likely to be shallower.

At this point
a tight loop quickly zeros the region between the current top of stack, as
returned by alloca(0), and the maximum recorded stack extent.  It also updates
the stack extent so no memory is cleared repeatedly if the stack contracts further.

This paper discusses this and similar techniques:
http://www.hpl.hp.com/personal/Hans_Boehm/gc/papers/pldi93.ps.Z

Another related issue is that the style of rb_eval() in eval.c in the
1.8 and 1.6 series causes gcc to emit a especially large and sparse stack frames.
Consider that gcc allocates two pair of stack slots for r and l in constructs like this:

    switch (nd_type(node)) {
  /* nodes for speed-up(literal match) */
      case NODE_MATCH2:
        {
            VALUE l = rb_eval(self,node->nd_recv);
            VALUE r = rb_eval(self,node->nd_value);
            result = rb_reg_match(l, r);
        }
        break;

        /* nodes for speed-up(literal match) */
      case NODE_MATCH3:
        {
            VALUE r = rb_eval(self,node->nd_recv);
            VALUE l = rb_eval(self,node->nd_value);
....

By the time the compiler's optimizer is allocating stack frame slots, all the block structure
of the original code has been lost in various transformations.
As a result, each rb_eval() call ends up pushing about 4k bytes onto the C stack,
of which less than 20% is even initialized.    This means that:

1)  There is a high probability that old VALUEs from previous frames
     will be resurrected as the stack grows,

2)  The GC must scan a sparse, large stack and mark the many dead object pointers it contains.

3)  callcc and thread context switches must copy needlessly large stacks

4)  recursive Ruby programs run out of stack space much earlier than than they might otherwise.

When I simply re-factored rb_eval() such that it calls a (non-inline) function
for each node type it encounters, the total observed C stack size for my application
was reduced by more than two thirds.  Not surprisingly, threading and continuation
micro benchmarks and run about 3 - 4 times faster.  However, I expect that benchmarks that
operate repeatedly on a few large, long lived objects will run slower.

Keep in mind that these techniques should improve the performance of *any* garbage
collector that scans the unstructured C stack for valid object pointers.  It may
even be relevant for the 1.9 series Ruby, but I'll leave that for those more
qualified to determine.

Today, this is implemented only in my heavily patched version of Ruby 1.6.8.
In the short term, if there's interest,
I can quickly post my hacked 1.6.8 Ruby to an FTP site for others to test.

Longer term,
The stack clearing could be supplied as a small patch to the 1.8 series, however the
refactoring of rb_eval() is probably too large to be attached to an email message
on this list.  I will take the time to produce these patches only if at least a few people
commit to testing them,  reporting detailed results and suggestions for improvement here.

- brent

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:20168] Re: Promising C coding techniques to reduce MRI's memory use

Nobuyoshi Nakada-2
Hi,

At Fri, 28 Nov 2008 18:54:45 +0900,
Brent Roman wrote in [ruby-core:20149]:
> Longer term,
> The stack clearing could be supplied as a small patch to the 1.8 series,
> however the
> refactoring of rb_eval() is probably too large to be attached to an email
> message
> on this list.  I will take the time to produce these patches only if at
> least a few people
> commit to testing them,  reporting detailed results and suggestions for
> improvement here.

In shorter, if you use gcc, can't you try -mpreferred-stack-boundary=2 option?

--
Nobu Nakada

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ruby-core:20168] Re: Promising C coding techniques to reduce MRI's memory use

Brent Roman
Before hacking rb_eval(), I first tried finding some compiler
options that would fill the stack holes.  

Decreasing the stack slot alignment requirements
does pack stack somewhat, however, the very sparse stack
frame generated by the huge switch statement in rb_eval() remains
largely unaffected by any compiler options I could find.
These holes still caused the GC to preform poorly for my app and
to fail utterly when presented with:  @x=loop {callcc {|c| c}}

Just have a look at the generated assembler code for rb_eval:
from "gcc -S -O2 eval.c".  The function preamble decrements stack pointer by
566 bytes.  Which of those bytes is actually written is determined
by the node type processed.  Most of them remain uninitialized in *all* cases.
With -mpreferred-stack-boundary=2, rb_eval() starts by decrementing
the stack pointer by 548 bytes.  No much difference.

After factoring, rb_eval() decriments the stack pointer by
only about 20 bytes.  I got best results with these options on x86 gcc 4.3.2:

gcc -mpreferred-stack-boundary=2 -fno-stack-protector -fno-inline-functions-called-once

Nobu, these are not just 2%-5% memory and time reductions.
For multithreaded applications, the both time and space performance
are significantly improved.  I suspect that some large single threaded
apps will also benefit.  (Maybe even rails?! :-)

There's an opportunity here.  I hope that
the core developers will find time to seriously explore it.

- brent

Nobuyoshi Nakada-2 wrote
Hi,

At Fri, 28 Nov 2008 18:54:45 +0900,
Brent Roman wrote in [ruby-core:20149]:
> Longer term,
> The stack clearing could be supplied as a small patch to the 1.8 series,
> however the
> refactoring of rb_eval() is probably too large to be attached to an email
> message
> on this list.  I will take the time to produce these patches only if at
> least a few people
> commit to testing them,  reporting detailed results and suggestions for
> improvement here.

In shorter, if you use gcc, can't you try -mpreferred-stack-boundary=2 option?

--
Nobu Nakada
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:20170] Re: Promising C coding techniques to reduce MRI's memory use

rogerdpack
In reply to this post by Brent Roman
> After a couple weeks of long nights and false starts, I feel I may have come
> up with
> a fix for a large class of Ruby memory leak.  The basic technique is a
> refinement of the
> one Kurt Stephens suggested.  It not only eliminates the leaks in this one
> liner:

Wow thanks for doing that. I'd say please create a redmine bug for it
[or attach it to an existing].  A patch to 1.8.7 would be sweet :)
A patch for 1.9 would be great too :)

I'd imagine that your system is "better" than just blindly doing a
garbage_collect()
{
clear_stack();
....do normal gc
}
void clear_stack()
{
  a = char[10000];
  memclear(a);
}
?

Thanks!
-=R
Note that I use gcc 3.4.5 I assume that won't be a problem though.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:20175] Re: Promising C coding techniques to reduce MRI's memory use

Brian Candler
In reply to this post by Nobuyoshi Nakada-2
The problem can be demonstrated with a very simple program (attached), and
it looks to me like a bug in gcc - surely it should overlap stack
assignments for automatic variables which aren't in scope simultaneously?

One solution to rb_eval() might be an ugly union at the top of the function
(second attachment). But it seems wrong to have to do this just to code
around an implementation problem with one particular compiler, albeit a
ubiquitous one.

Regards,

Brian.

sf.c (795 bytes) Download Attachment
sf2.c (1018 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ruby-core:20175] Re: Promising C coding techniques to reduce MRI's memory use

Brent Roman
Brian,

Thanks for the very clear demo program to illustrate the problem.
Is there anyone who can run look at the assembler code generated
for this demo by a recent Microsoft or Intel 'C' compiler?

In any case,
I doubt that the gcc maintainers would consider this behavior a bug.
It's been with them from before v3.3.5.  They've known about it for many
years.  They view it is an limitation of their register optimization techniques
and are more concerned about speeding up the code than shrinking
its stack footprint.  However, for us, larger stacks = slower code due to
stack copying and the conservative GC.

The "ugly union" solution would not be sufficient because much of the
stack is occupied by compiler generated temporaries that have no
representation in the 'C' input source.  I did consider such wholesale
code changes, but resisted because they would have been, as you say,
quite ugly and difficult to maintain.

What I did come up with was not ugly at all.  Factor the unwieldy switch
statement of rb_eval() into separate functions to handle each node
type and clear the stack at a few opportune times.  rb_eval() becomes
smaller and more likely to be optimized.  I buried the stack clearing
into macros that already exist.

- brent


Brian Candler wrote
The problem can be demonstrated with a very simple program (attached), and
it looks to me like a bug in gcc - surely it should overlap stack
assignments for automatic variables which aren't in scope simultaneously?

One solution to rb_eval() might be an ugly union at the top of the function
(second attachment). But it seems wrong to have to do this just to code
around an implementation problem with one particular compiler, albeit a
ubiquitous one.

Regards,

Brian.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [ruby-core:20170] Re: Promising C coding techniques to reduce MRI's memory use

Brent Roman
In reply to this post by rogerdpack
Roger,

I already responded in detail to this bug:

http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7896&group_id=426

I just bang on Ruby 1.6.8 for our robotics application.

You seem to already be doing a lot of excellent Ruby testing with current versions.
If I spent a couple days developing these two patches for Ruby 1.8.7,
would you be willing to run
regression tests against them and to report the results here?

I think the small stack clearing patch should improve the GC behavior,
but, by itself, it will likely slow down some apps due to its having
to clear large areas of stack.  I'd expect to see that
slow down mitigated by the larger patch that would refactor rb_eval()
and thereby keep the stack smaller.

The combined patches will likely be large, so I'll just post links to them here.

Would anyone else be willing to test them? ...
Particularly those who have large apps, and/or apps that use multiple threads or
continuations that seem to leak memory?

- brent

P.S.  I use gcc 3.4.5 for generating code for our embedded ARM targets.
The older compiler generates fewer stack temporaries than the newer ones.
Don't rush to update :-)

P.P.S.  The way GC is currently invoked causes it to occur when that stack
is already near its maximum depth.  This patch tries to make GC normally
occur is part of CHECK_INTS, when the stack tends to be shallower.
At that point, clearing the stack can be much more effective.


Roger Pack wrote
Wow thanks for doing that. I'd say please create a redmine bug for it
[or attach it to an existing].  A patch to 1.8.7 would be sweet :)
A patch for 1.9 would be great too :)

I'd imagine that your system is "better" than just blindly doing a
garbage_collect()
{
clear_stack();
....do normal gc
}
void clear_stack()
{
  a = char[10000];
  memclear(a);
}
?

Thanks!
-=R
Note that I use gcc 3.4.5 I assume that won't be a problem though.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:20185] Re: Promising C coding techniques to reduce MRI's memory use

Brian Candler
In reply to this post by Brent Roman
> What I did come up with was not ugly at all.  Factor the unwieldy switch
> statement of rb_eval() into separate functions to handle each node
> type

Did you replace the whole switch statement with a dispatch table? That
sounds like a sensible thing to do anyway.

OTOH, if this is for ruby 1.8.x, I'm afraid you may not find much interest
in such changes while the focus is all on 1.9.

Perhaps worth checking how 1.9's bytecode interpreter stacks up under the
same conditions?

OTOH, 1.9 doesn't have callcc anyway, so maybe your application code would
need a lot of restructuring to use Fiber instead. I don't know if it's
possible to implement callcc in terms of Fiber.

Regards,

Brian.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:20186] Re: Promising C coding techniques to reduce MRI's memory use

Paul Brannan
On Mon, Dec 01, 2008 at 06:29:00PM +0900, Brian Candler wrote:
> OTOH, 1.9 doesn't have callcc anyway, so maybe your application code would
> need a lot of restructuring to use Fiber instead. I don't know if it's
> possible to implement callcc in terms of Fiber.

1.9 does have callcc (require 'continuation').  It's probably not good
to use it, though.

Paul


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:20192] Re: Promising C coding techniques to reduce MRI's memory use

Ezra Zygmuntowicz-3
In reply to this post by Brent Roman
Brent-

        I would love to see a version of these patches against 1.8.6 or  
1.8.7. I can test them on a few hundred servers to see what kind of  
resource consumption these changes have in larger deployments.

        Awesome work on this. I'm very interetsed in testing this for you.  
You can contact me off list if you like or if you want servers to use  
to test this on.

Thanks

Ezra Zygmuntowicz
[hidden email]


On Nov 30, 2008, at 11:34 AM, Brent Roman wrote:

>
> Roger,
>
> I already responded in detail to this bug:
>
> http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7896&group_id=426
>
> I just bang on Ruby 1.6.8 for our robotics application.
>
> You seem to already be doing a lot of excellent Ruby testing with  
> current
> versions.
> If I spent a couple days developing these two patches for Ruby 1.8.7,
> would you be willing to run
> regression tests against them and to report the results here?
>
> I think the small stack clearing patch should improve the GC behavior,
> but, by itself, it will likely slow down some apps due to its having
> to clear large areas of stack.  I'd expect to see that
> slow down mitigated by the larger patch that would refactor rb_eval()
> and thereby keep the stack smaller.
>
> The combined patches will likely be large, so I'll just post links  
> to them
> here.
>
> Would anyone else be willing to test them? ...
> Particularly those who have large apps, and/or apps that use multiple
> threads or
> continuations that seem to leak memory?
>
> - brent
>
> P.S.  I use gcc 3.4.5 for generating code for our embedded ARM  
> targets.
> The older compiler generates fewer stack temporaries than the newer  
> ones.
> Don't rush to update :-)
>
> P.P.S.  The way GC is currently invoked causes it to occur when that  
> stack
> is already near its maximum depth.  This patch tries to make GC  
> normally
> occur is part of CHECK_INTS, when the stack tends to be shallower.
> At that point, clearing the stack can be much more effective.
>
>
>
> Roger Pack wrote:
>>
>>
>> Wow thanks for doing that. I'd say please create a redmine bug for it
>> [or attach it to an existing].  A patch to 1.8.7 would be sweet :)
>> A patch for 1.9 would be great too :)
>>
>> I'd imagine that your system is "better" than just blindly doing a
>> garbage_collect()
>> {
>> clear_stack();
>> ....do normal gc
>> }
>> void clear_stack()
>> {
>>  a = char[10000];
>>  memclear(a);
>> }
>> ?
>>
>> Thanks!
>> -=R
>> Note that I use gcc 3.4.5 I assume that won't be a problem though.
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p20761320.html
> Sent from the ruby-core mailing list archive at Nabble.com.
>
>






Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[ruby-core:20193] Re: Promising C coding techniques to reduce MRI's memory use

Ezra Zygmuntowicz-3
In reply to this post by Brian Candler

On Dec 1, 2008, at 1:29 AM, Brian Candler wrote:

>> What I did come up with was not ugly at all.  Factor the unwieldy  
>> switch
>> statement of rb_eval() into separate functions to handle each node
>> type
>
> Did you replace the whole switch statement with a dispatch table? That
> sounds like a sensible thing to do anyway.
>
> OTOH, if this is for ruby 1.8.x, I'm afraid you may not find much  
> interest
> in such changes while the focus is all on 1.9.




        Actually I think you will find a *ton* of interest in this for the  
1.8.* branch. There are thousands of production apps that are not  
going to move to 1.9 anytime soon and any improvements to 1.8.* thread  
and callcc handling like this would be very welcome.

Thanks
Ezra Zygmuntowicz
[hidden email]




1234 ... 8
Loading...