mysql mem-tables vs. memcached

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

mysql mem-tables vs. memcached

Peter Ertl-3
Could someone please elaborate on the technical differences and practical
impact of whether choosing memory-based tables in MySQL or using memcached.

I got this far on my own:
It seems that MySQL uses the NDB engine for transaction-safe memory access
in a cluster. the memory storage engine seems to be faster but not
synchronizable by any means in a cluster. memcached seems to be ultimately  
fast
but requires extra effort in your code.

Who got experience on that?

Best regards
Peter
_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

Christophe Vigny
You can use pure ruby memory structure like hash or array and access
them via drb server.
it use lot off memory (but that cheap) and you have good performance,
and pure ruby object, no sql needed (like memcache).
If you have complex processing you can do that in the drb server unlike
with memcached, u dont touch the database, and its quite
fast for processing, but you depend on drb. if u need load balancing and
fail over, you can write a pool to do that .


Pete wrote:

> Could someone please elaborate on the technical differences and practical
> impact of whether choosing memory-based tables in MySQL or using
> memcached.
>
> I got this far on my own:
> It seems that MySQL uses the NDB engine for transaction-safe memory
> access
> in a cluster. the memory storage engine seems to be faster but not
> synchronizable by any means in a cluster. memcached seems to be
> ultimately  fast
> but requires extra effort in your code.
>
> Who got experience on that?
>
> Best regards
> Peter
> _______________________________________________
> Rails mailing list
> [hidden email]
> http://lists.rubyonrails.org/mailman/listinfo/rails
>
>

"Ce message et toutes les pièces jointes sont des informations
strictement confidentielles et réservées au(x) destinataire(s). Ce
courriel n'a pas de valeur contractuelle et son contenu ne constitue ni
une acceptation, ni un engagement de la part de l'auteur et des sociétés
du groupe Serveur et Artprice, sauf dans le cas où cela aurait été prévu
avec le destinataire par un accord écrit. Le contenu de ce message et
les pièces jointes ne peuvent constituer une preuve au sens de l'article
1316-1 du Code Civil. L.auteur et les sociétés du groupe Serveur et
Artprice déclinent toute responsabilité au titre de ce courriel s'il a
été altéré, déformé, falsifié ou indûment utilisé par des tiers ou
encore s'il a causé tout dommage ou perte de toute nature. Si vous
n'êtes pas le bon destinataire, merci de nous contacter et de ne pas le
divulguer."

"This message including any attachments are confidential and privileged
material intended solely for the addressees. Its contents do not
constitute a commitment by groupe Serveur sas and Artprice SA, except
when provided for in a written agreement with the addressees. The
contents of this message cannot constitute neither the proof nor the
acceptance of any agreement as per article 1316-1 of the French civil code.
Groupe Serveur sas and Artprice SA shall not be rendered liable in any
manner whatsoever for the delay and/or loss in transit of this message,
for corruption, alteration, falsification, misuse or fraudulent use
(which may be made) of this message. If you receive this message in
error, please delete it and immediately notify the sender. If the reader
of this message is not the intended recipient, you are hereby notified
that any unauthorized use, copying or dissemination is prohibited."
_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

Jason Edgecombe
In reply to this post by Peter Ertl-3
Pete wrote:

> Could someone please elaborate on the technical differences and practical
> impact of whether choosing memory-based tables in MySQL or using
> memcached.
>
> I got this far on my own:
> It seems that MySQL uses the NDB engine for transaction-safe memory
> access
> in a cluster. the memory storage engine seems to be faster but not
> synchronizable by any means in a cluster. memcached seems to be
> ultimately  fast
> but requires extra effort in your code.
>
> Who got experience on that?

Hi,

mysql memory tables aren't likely to lose data.

Memcache WILL lose data. It's a cache, that's what it does. memcache's
purpose is to store a copy of frequently used data for fast access.
Don't store anything that you don't want to lose in memcache.

Jason

_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

WeaZeLb0y
mySQL memory tables hold the information until the connection is
closed or the table is dropped.  These are useful if you have a large
process and want to use a stored procedure to move/update data to keep
the network traffic low.

memCache was created as a proxy to the database for performance
reasons.  Its to fill requests that would normally go to a database.
So, if every request on your website requires a row from the 'members'
table it would be smart to cache that with memCache.

Why not just use in memory tables and forget memCache altogether?
Because when a site has so much traffic, like livejournal, and you
only have 1 database server to perform the transactions you need to
have a minimum number of requests to that server.

Plus, memCache is optimized for exactly this purpose, there is
probably less de/serializing work that has to occur than if you were
to use a database.


On 12/16/05, Jason Edgecombe <[hidden email]> wrote:

> Pete wrote:
>
> > Could someone please elaborate on the technical differences and practical
> > impact of whether choosing memory-based tables in MySQL or using
> > memcached.
> >
> > I got this far on my own:
> > It seems that MySQL uses the NDB engine for transaction-safe memory
> > access
> > in a cluster. the memory storage engine seems to be faster but not
> > synchronizable by any means in a cluster. memcached seems to be
> > ultimately  fast
> > but requires extra effort in your code.
> >
> > Who got experience on that?
>
> Hi,
>
> mysql memory tables aren't likely to lose data.
>
> Memcache WILL lose data. It's a cache, that's what it does. memcache's
> purpose is to store a copy of frequently used data for fast access.
> Don't store anything that you don't want to lose in memcache.
>
> Jason
>
> _______________________________________________
> Rails mailing list
> [hidden email]
> http://lists.rubyonrails.org/mailman/listinfo/rails
>
_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

Eric Hodel
In reply to this post by Peter Ertl-3
On Dec 16, 2005, at 1:48 AM, [hidden email] wrote:

> Could someone please elaborate on the technical differences and  
> practical
> impact of whether choosing memory-based tables in MySQL or using  
> memcached.
>
> I got this far on my own:
> It seems that MySQL uses the NDB engine for transaction-safe memory  
> access
> in a cluster. the memory storage engine seems to be faster but not
> synchronizable by any means in a cluster. memcached seems to be  
> ultimately
> fast but requires extra effort in your code.
>
> Who got experience on that?

I have no experience with MySQL memory tables, but.

memcached doesn't have query parsing overhead.
memcached doesn't have database adapter overhead.
memcached can be run across multiple machines painlessly.
memcached is easier to set up.
memcached can be painlessly used with unstructured data.
memcached becomes transparent with proper abstraction.

--
Eric Hodel - [hidden email] - http://segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com


_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

Eric Hodel
In reply to this post by Peter Ertl-3
On Dec 16, 2005, at 5:35 AM, [hidden email] wrote:

> You can use pure ruby memory structure like hash or array and access
> them via drb server.
> it use lot off memory (but that cheap) and you have good performance,
> and pure ruby object, no sql needed (like memcache).

memcached does not use SQL, it is more like a distributed  
dictionary.  Memcached can store pure ruby objects (via Marshal, just  
like DRb).

> If you have complex processing you can do that in the drb server  
> unlike
> with memcached, u dont touch the database, and its quite

memcached is not an RDBMS, it is just a dictionary.  It is faster  
than DRb and will (potentially) consume less memory because of the  
way it allocates space for cache items.

> fast for processing, but you depend on drb. if u need load  
> balancing and
> fail over, you can write a pool to do that .

memcached does load balancing painlessly.

--
Eric Hodel - [hidden email] - http://segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com


_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

Chuck Remes-3
In reply to this post by Eric Hodel
On Dec 16, 2005, at 2:07 PM, Eric Hodel wrote:

> On Dec 16, 2005, at 1:48 AM, [hidden email]  
> wrote:
>
>> Could someone please elaborate on the technical differences and  
>> practical
>> impact of whether choosing memory-based tables in MySQL or using  
>> memcached.
>>
>> I got this far on my own:
>> It seems that MySQL uses the NDB engine for transaction-safe  
>> memory access
>> in a cluster. the memory storage engine seems to be faster but not
>> synchronizable by any means in a cluster. memcached seems to be  
>> ultimately
>> fast but requires extra effort in your code.
>>
>> Who got experience on that?
>
> I have no experience with MySQL memory tables, but.
>
> memcached doesn't have query parsing overhead.
> memcached doesn't have database adapter overhead.
> memcached can be run across multiple machines painlessly.
> memcached is easier to set up.
> memcached can be painlessly used with unstructured data.
> memcached becomes transparent with proper abstraction.

Can you point to any sample code where someone is using memcached  
with ruby? I took a look around all the wikis and came up dry. The  
best examples I could find were in the wikipedia entry for memcached;  
the examples were not in ruby but it was easy to parse through them.

While lurking on this list I think I saw a note or two implying that  
Rails uses memcached internally (as a configurable option). If you  
can confirm or deny this then I'd bet that code would be a perfect  
example.

Thanks for any help and pointers.

_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

Eric Hodel
In reply to this post by Peter Ertl-3
On Dec 16, 2005, at 8:43 PM, [hidden email] wrote:

> Can you point to any sample code where someone is using memcached
> with ruby?

43 Things uses memcached extensively (but not as extensively as  
LiveJournal).

> I took a look around all the wikis and came up dry. The
> best examples I could find were in the wikipedia entry for memcached;
> the examples were not in ruby but it was easy to parse through them.
>
> While lurking on this list I think I saw a note or two implying that
> Rails uses memcached internally (as a configurable option). If you
> can confirm or deny this then I'd bet that code would be a perfect
> example.

Here's how to use memcache to store sessions, I put it in config/
environment.rb.  You will need the Ruby-MemCache gem installed.

memcache_options = {
   :c_threshold => 10_000,
   :compression => true,
   :debug => false,
   :namespace => 'your_namespace',
   :readonly => false,
   :urlencode => false
}

CACHE = MemCache.new memcache_options
CACHE.servers = 'localhost:11211'

session_options = {
   :database_manager => CGI::Session::MemCacheStore,
   :cache => CACHE,
   :session_domain => 'example.com'
}

ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS.update  
session_options

Note that there is a bug [1] but you won't be able to tell you're  
experiencing it [2].

Since you've got a CACHE global constant, you can use it to store  
things that take a long time to build:

CACHE.set 'yourkey', 'something that takes a long time to build'

then later:

value = CACHE.get 'yourkey'

[1] http://dev.rubyonrails.org/ticket/3214
[2] http://dev.rubyonrails.org/ticket/3216

--
Eric Hodel - [hidden email] - http://segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com


_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

Jason Edgecombe
Eric Hodel wrote:

> On Dec 16, 2005, at 8:43 PM, [hidden email] wrote:
>
>> Can you point to any sample code where someone is using memcached
>> with ruby?
>
>
> 43 Things uses memcached extensively (but not as extensively as  
> LiveJournal).
>
>> I took a look around all the wikis and came up dry. The
>> best examples I could find were in the wikipedia entry for memcached;
>> the examples were not in ruby but it was easy to parse through them.
>>
>> While lurking on this list I think I saw a note or two implying that
>> Rails uses memcached internally (as a configurable option). If you
>> can confirm or deny this then I'd bet that code would be a perfect
>> example.
>
>
> Here's how to use memcache to store sessions, I put it in config/
> environment.rb.  You will need the Ruby-MemCache gem installed.
>
> memcache_options = {
>   :c_threshold => 10_000,
>   :compression => true,
>   :debug => false,
>   :namespace => 'your_namespace',
>   :readonly => false,
>   :urlencode => false
> }
>
> CACHE = MemCache.new memcache_options
> CACHE.servers = 'localhost:11211'
>
> session_options = {
>   :database_manager => CGI::Session::MemCacheStore,
>   :cache => CACHE,
>   :session_domain => 'example.com'
> }
>
> ActionController::CgiRequest::DEFAULT_SESSION_OPTIONS.update  
> session_options
>
> Note that there is a bug [1] but you won't be able to tell you're  
> experiencing it [2].
>
> Since you've got a CACHE global constant, you can use it to store  
> things that take a long time to build:
>
> CACHE.set 'yourkey', 'something that takes a long time to build'
>
> then later:
>
> value = CACHE.get 'yourkey'
>
> [1] http://dev.rubyonrails.org/ticket/3214
> [2] http://dev.rubyonrails.org/ticket/3216
>
Has anyone subclasses ActiveRecord to provide transparent caching of
records using memcache?

How would someone go about doing this?

Jason
_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

Eric Hodel
In reply to this post by Peter Ertl-3
On Dec 17, 2005, at 2:21 PM, [hidden email] wrote:

> Eric Hodel wrote:
>
>> On Dec 16, 2005, at 8:43 PM, [hidden email]  
>> wrote:
>>
>>> Can you point to any sample code where someone is using memcached
>>> with ruby?
>>
>>
>> 43 Things uses memcached extensively (but not as extensively as
>> LiveJournal).
>
> Has anyone subclasses ActiveRecord to provide transparent caching of
> records using memcache?

Yes.  43 Things uses this.

> How would someone go about doing this?

I can probably package it up for release after the new year.  It is  
not difficult to implement, we override #find and #find_by_sql and  
add some hacks to override the single table inheritance silliness.

--
Eric Hodel - [hidden email] - http://segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

Joshua Sierles
Eric,

How do you handle cache expiry?


On 12/17/05, Eric Hodel <[hidden email]> wrote:

> On Dec 17, 2005, at 2:21 PM, [hidden email] wrote:
>
> > Eric Hodel wrote:
> >
> >> On Dec 16, 2005, at 8:43 PM, [hidden email]
> >> wrote:
> >>
> >>> Can you point to any sample code where someone is using memcached
> >>> with ruby?
> >>
> >>
> >> 43 Things uses memcached extensively (but not as extensively as
> >> LiveJournal).
> >
> > Has anyone subclasses ActiveRecord to provide transparent caching of
> > records using memcache?
>
> Yes.  43 Things uses this.
>
> > How would someone go about doing this?
>
> I can probably package it up for release after the new year.  It is
> not difficult to implement, we override #find and #find_by_sql and
> add some hacks to override the single table inheritance silliness.
>
> --
> Eric Hodel - [hidden email] - http://segment7.net
> This implementation is HODEL-HASH-9600 compliant
>
> http://trackmap.robotcoop.com
>
> _______________________________________________
> Rails mailing list
> [hidden email]
> http://lists.rubyonrails.org/mailman/listinfo/rails
>
_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails
Reply | Threaded
Open this post in threaded view
|

Re: mysql mem-tables vs. memcached

Christophe Vigny
In reply to this post by Eric Hodel


Eric Hodel wrote:

> On Dec 16, 2005, at 5:35 AM, [hidden email] wrote:
>
>> You can use pure ruby memory structure like hash or array and access
>> them via drb server.
>> it use lot off memory (but that cheap) and you have good performance,
>> and pure ruby object, no sql needed (like memcache).
>
>
> memcached does not use SQL, it is more like a distributed  
> dictionary.  Memcached can store pure ruby objects (via Marshal, just  
> like DRb).
>
>> If you have complex processing you can do that in the drb server  unlike
>> with memcached, u dont touch the database, and its quite
>
>
> memcached is not an RDBMS, it is just a dictionary.  It is faster  
> than DRb and will (potentially) consume less memory because of the  
> way it allocates space for cache items.


yes but the chache cant handle any programming logic, if u have to sort
for exemple u must fetch the object, (this could be long for thousand of
them),
and sort localy. uless a drb server can do the trick remotely !
it depends you needs .

>
>> fast for processing, but you depend on drb. if u need load  balancing
>> and
>> fail over, you can write a pool to do that .
>
>
> memcached does load balancing painlessly.
>
yes and you dont even had to write code for that !
memchache is great for proxy fast storing !

"Ce message et toutes les pièces jointes sont des informations
strictement confidentielles et réservées au(x) destinataire(s). Ce
courriel n'a pas de valeur contractuelle et son contenu ne constitue ni
une acceptation, ni un engagement de la part de l'auteur et des sociétés
du groupe Serveur et Artprice, sauf dans le cas où cela aurait été prévu
avec le destinataire par un accord écrit. Le contenu de ce message et
les pièces jointes ne peuvent constituer une preuve au sens de l'article
1316-1 du Code Civil. L.auteur et les sociétés du groupe Serveur et
Artprice déclinent toute responsabilité au titre de ce courriel s'il a
été altéré, déformé, falsifié ou indûment utilisé par des tiers ou
encore s'il a causé tout dommage ou perte de toute nature. Si vous
n'êtes pas le bon destinataire, merci de nous contacter et de ne pas le
divulguer."

"This message including any attachments are confidential and privileged
material intended solely for the addressees. Its contents do not
constitute a commitment by groupe Serveur sas and Artprice SA, except
when provided for in a written agreement with the addressees. The
contents of this message cannot constitute neither the proof nor the
acceptance of any agreement as per article 1316-1 of the French civil code.
Groupe Serveur sas and Artprice SA shall not be rendered liable in any
manner whatsoever for the delay and/or loss in transit of this message,
for corruption, alteration, falsification, misuse or fraudulent use
(which may be made) of this message. If you receive this message in
error, please delete it and immediately notify the sender. If the reader
of this message is not the intended recipient, you are hereby notified
that any unauthorized use, copying or dissemination is prohibited."
_______________________________________________
Rails mailing list
[hidden email]
http://lists.rubyonrails.org/mailman/listinfo/rails