Have not got around to doing performance tests, but based on general AppEngine performance, I guess it will not come close to a dedicated Memcache machine. Should be good for everything else.
Just to give you some rough range, Redis is in the range of 10k query/second on an entry level laptop. As you can guess the main bottleneck is the networking I/O with things like Memcached or Redis.
About the speed of dumping the DB. 1 million keys DB with 16 bytes string inside every key takes something like 1 second to be saved on the same HW. For the next release I hope to have replication and LZO compression of (compressible) big values working.
Since it's in C, it probably has an edge, but as I've mentioned to antirez a couple of times, I'd love to see someone take a crack at this in Erlang - this is exactly the kind of thing it was built for.
Apart from speed what I like of C for this kind of projects is the self-contain-ness and the fact that most developers that may help in the development are probably able to read/write C but not Erlang.
Nice project, I already read about it. Still note that one of the major points of Redis is to support more complex types as values. Tokyo Cabinet is basically an on-disk hash table library, with an optional networking server using this lib to export the functionality.
For example if you want to use Redis in order to store logs from a lot of different computers, and you want to only take the last 1000 entries for every computer, all you need to do is for every computer something like this:
RPUSH computer_5 "... your log line ..."
LTRIM computer_5 0 999
Redis offers atomic LPOP/RPOP too, this makes pretty easy to create queues where different computers can pop/push values with the guarantee the operation is atomic.
Another trivial example: think about implementing something like YC with a key-value DB. Support for lists and sets will make your life much easier :)
I've knocked up a quick prototype ruby client library for redis. So far it supports GET, SET, EXISTS, DELETE and KEYS. Should be able to have a fully compliant library in a few days.
Thanks! your implementation is cool and if you agree will just be the Ruby implementation distributed with Redis. INCRBY/DECRBY are still missing but in a week I plan to have this, all the Set operations inside and release another beta. Then will be the time of replication and compression. And finally I plan to write a fail-over daemon that will handle masters and slaves, check who is running and who appears to be down and act accordingly.
If you like this, check out Datastore (http://github.com/ociule/gae-datastore/tree/master). It will allow you to put up a simple hash-value database on AppEngine.
Live demo at http://gae-datastore.appspot.com/
Have not got around to doing performance tests, but based on general AppEngine performance, I guess it will not come close to a dedicated Memcache machine. Should be good for everything else.
Just to give you some rough range, Redis is in the range of 10k query/second on an entry level laptop. As you can guess the main bottleneck is the networking I/O with things like Memcached or Redis.
About the speed of dumping the DB. 1 million keys DB with 16 bytes string inside every key takes something like 1 second to be saved on the same HW. For the next release I hope to have replication and LZO compression of (compressible) big values working.
Since it's in C, it probably has an edge, but as I've mentioned to antirez a couple of times, I'd love to see someone take a crack at this in Erlang - this is exactly the kind of thing it was built for.
There are a lot of projects similar to Redis in Erlang actually: http://www.metabrew.com/article/anti-rdbms-a-list-of-distrib...
Apart from speed what I like of C for this kind of projects is the self-contain-ness and the fact that most developers that may help in the development are probably able to read/write C but not Erlang.
Just tried to wikify the documentation a bit:
http://code.google.com/p/redis/wiki/README
Not very well formatted but if you want to know a bit more about Redis without to uncompress the tar.gz it should help.
Tokyo Cabinet: http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-va... http://sourceforge.net/projects/tokyocabinet/
Nice project, I already read about it. Still note that one of the major points of Redis is to support more complex types as values. Tokyo Cabinet is basically an on-disk hash table library, with an optional networking server using this lib to export the functionality.
For example if you want to use Redis in order to store logs from a lot of different computers, and you want to only take the last 1000 entries for every computer, all you need to do is for every computer something like this:
Redis offers atomic LPOP/RPOP too, this makes pretty easy to create queues where different computers can pop/push values with the guarantee the operation is atomic.
Another trivial example: think about implementing something like YC with a key-value DB. Support for lists and sets will make your life much easier :)
There is also memcachedb. (http://memcachedb.org/) Same interface as memcache, with replication features and persistent storage.
I've knocked up a quick prototype ruby client library for redis. So far it supports GET, SET, EXISTS, DELETE and KEYS. Should be able to have a fully compliant library in a few days.
http://github.com/ezmobius/redis-rb/tree/master
Umm redis is really cool. Nice job on this antirez. I've just finished implementing the entire spec here:
http://github.com/ezmobius/redis-rb/blob/0951204e6a84ca335ac...
And I am really enjoying being able to store lists as values and pushing/popping/indexing. Looking forward to where you are going with this.
Although the current release doesn't seem to support all the documented features yet. incrby and decrby don't work yet.
Can you get in touch? I'd love to pick your brain a little about redis.
Thanks! your implementation is cool and if you agree will just be the Ruby implementation distributed with Redis. INCRBY/DECRBY are still missing but in a week I plan to have this, all the Set operations inside and release another beta. Then will be the time of replication and compression. And finally I plan to write a fail-over daemon that will handle masters and slaves, check who is running and who appears to be down and act accordingly.