This is really inspiring. Reminds me of the quote "A complex system that works is invariably found to have evolved from a simple system that worked."
The interesting question of course is how to evolve a simple system into a successful complex system. One thing that seems important is nailing the right architecture early on - the simple system must contain all the right ideas in it and it is important to weed out bad ideas before they are entrenched in the code base. Still, planning the long-term evolution of a software system remains an arcane art to me. "Listen to your users and do what is necessary at the moment" is too simplistic.
On the contrary I don't think you can plan architectural perfection from the beginning. IMO you have to throw it in a prod-like environment early (preferably as an internal app with a minimal SLO) so that you can iterate early and often.
That's what I talked about in the last sentence of my post. It is important to do this, but it is not the whole story. How do you decide in which direction to iterate?
Similar to my sibling comment I think the value in sw architects isn't the ability to predict the future from experience, rather they should build systems that can accept reasonable change.
Some practical examples would be to implement build systems into their stack that accept database migrations and rollbacks easily. Or realizing they are in a fast-paced team that could benefit from specific frameworks and languages (experience would play a big role here).
Make it feature-driven. When Facebook came out they didn't know photo sharing would be a definitive feature. When they put it out there they took a slug to the shoulder for not realizing it sooner and then implemented relevant features (like tagging).
Similarly when Redis came out I doubt the creator realized everybody and their grandmother would use lists as a defacto message queue but now look at all the commands list has that facilitate just that.
Key point is to accept and welcome that slug to the shoulder that pushes you in the right direction, to abuse the metaphor.
All you have to do is listen to your users and do what's necessary if you have scars to remind you that sticking IO here will bring the system to its knees on a real user load, or that this customer is going to ask for LDAP integration next week so you better build your authentication accordingly.
I agree that experience helps, but experience alone is insufficient. Experience should be properly reflected upon, or you can stick to suboptimal practices justifying them by saying "that's how I always do it, worked for me in the past".
Experience can even be detrimental - the well-known second system syndrome comes to mind!
If you are able to rewrite the code base in a short amount of time, the infrastructure problems are easier to handle. That way you can move forward faster and fix problems early on.
Yes, when the code base is small, evolving it is much simpler. One of the challenges is deciding on the exact right moment to introduce big changes - too early and the changes are in the wrong direction or even useless, too late and the amount of work is prohibiting and you are forced to live with suboptimal architecture.
Yeah, I agree. I think the real challange is to keep the code base in such a good shape that a rewrite of the whole thing or parts of is always doable. Someone talked about never writing modules that can't be rewritten in two weeks.
Yes, when the code base is small, evolving it is much simpler.
There's obviously some truth to that. But I also think it leaves or an important angle: Often larger, more complicated, features are easier to add if you have a lot of infrastructure pieces.
Say, in a database, if you have no durability (journaling), resource management (errors, locks), error checking (deadlocks, days checksumming), it'll be a lot more work to add support for some new query language [feature]. Now databases are my thing, but the same it's true for a lot of areas.
Disregarding that is IMO one of the reasons ground up rewrites fail so often: You'll get bogged down with infrastructure for a long time, without proper non toy users of the facility.
antirez mentions that this forks to save, but I'm not seeing that. The backgroundsave procedure makes a copy of db to dbcopy, but then never saves dbcopy. I wonder how much further the TCL implementation was extended before antirez switched to C.
BTW, I hadn't heard of LLOOGG (the L in LMDB) before. It was this:
"LLOGG was web service I (Salvatore Sanfilippo) and my co-founder Fabio Pitrola ran for seven years for free."
You are right, the intention was that but now I'm reminding that, in the middle of doing it, I thought, ok I reached the limits of the Tcl prototype, let's rewrite in C.
As someone who worked for a company for a few years on a vast Tcl codebase, a company in which almost every engineer lamented the fact that they were working in an unused and unknown language, it warms my heart to see that it was used for this.
I dabbled with it myself, writing a simple TCL interpreter in C, but I always find myself comparing it to FORTH a language I still find it easy to love.
This is really inspiring. Reminds me of the quote "A complex system that works is invariably found to have evolved from a simple system that worked."
The interesting question of course is how to evolve a simple system into a successful complex system. One thing that seems important is nailing the right architecture early on - the simple system must contain all the right ideas in it and it is important to weed out bad ideas before they are entrenched in the code base. Still, planning the long-term evolution of a software system remains an arcane art to me. "Listen to your users and do what is necessary at the moment" is too simplistic.
On the contrary I don't think you can plan architectural perfection from the beginning. IMO you have to throw it in a prod-like environment early (preferably as an internal app with a minimal SLO) so that you can iterate early and often.
That's what I talked about in the last sentence of my post. It is important to do this, but it is not the whole story. How do you decide in which direction to iterate?
I think it takes experience. That's why I still believe that sw architects are really essential.
Similar to my sibling comment I think the value in sw architects isn't the ability to predict the future from experience, rather they should build systems that can accept reasonable change.
Some practical examples would be to implement build systems into their stack that accept database migrations and rollbacks easily. Or realizing they are in a fast-paced team that could benefit from specific frameworks and languages (experience would play a big role here).
Make it feature-driven. When Facebook came out they didn't know photo sharing would be a definitive feature. When they put it out there they took a slug to the shoulder for not realizing it sooner and then implemented relevant features (like tagging).
Similarly when Redis came out I doubt the creator realized everybody and their grandmother would use lists as a defacto message queue but now look at all the commands list has that facilitate just that.
Key point is to accept and welcome that slug to the shoulder that pushes you in the right direction, to abuse the metaphor.
Iirc Zuckerberg allowed profile pics but realized people kept changing them all the time so they implemented photos
The ability to plan falls out of experience.
All you have to do is listen to your users and do what's necessary if you have scars to remind you that sticking IO here will bring the system to its knees on a real user load, or that this customer is going to ask for LDAP integration next week so you better build your authentication accordingly.
I agree that experience helps, but experience alone is insufficient. Experience should be properly reflected upon, or you can stick to suboptimal practices justifying them by saying "that's how I always do it, worked for me in the past".
Experience can even be detrimental - the well-known second system syndrome comes to mind!
Yet lack of experience won't help either...
so, experience and keeping up with what customers need
If you are able to rewrite the code base in a short amount of time, the infrastructure problems are easier to handle. That way you can move forward faster and fix problems early on.
Yes, when the code base is small, evolving it is much simpler. One of the challenges is deciding on the exact right moment to introduce big changes - too early and the changes are in the wrong direction or even useless, too late and the amount of work is prohibiting and you are forced to live with suboptimal architecture.
Yeah, I agree. I think the real challange is to keep the code base in such a good shape that a rewrite of the whole thing or parts of is always doable. Someone talked about never writing modules that can't be rewritten in two weeks.
There's obviously some truth to that. But I also think it leaves or an important angle: Often larger, more complicated, features are easier to add if you have a lot of infrastructure pieces.
Say, in a database, if you have no durability (journaling), resource management (errors, locks), error checking (deadlocks, days checksumming), it'll be a lot more work to add support for some new query language [feature]. Now databases are my thing, but the same it's true for a lot of areas.
Disregarding that is IMO one of the reasons ground up rewrites fail so often: You'll get bogged down with infrastructure for a long time, without proper non toy users of the facility.
As a partial answer, here is the document that outlines the redis philosophy: http://oldblog.antirez.com/post/redis-manifesto.html
I bet that coming up with this points and having them in mind (even subconsciously) has helped antirez evolve redis in a coherent direction.
antirez mentions that this forks to save, but I'm not seeing that. The backgroundsave procedure makes a copy of db to dbcopy, but then never saves dbcopy. I wonder how much further the TCL implementation was extended before antirez switched to C.
BTW, I hadn't heard of LLOOGG (the L in LMDB) before. It was this:
"LLOGG was web service I (Salvatore Sanfilippo) and my co-founder Fabio Pitrola ran for seven years for free."
https://github.com/antirez/lloogg/blob/master/README.md
Looks like the intention was there, they pull in TclX with a comment about it.
https://gist.github.com/antirez/6ca04dd191bdb82aad9fb241013e...
Guessing he just found some interim version that wasn't the one he actually wanted to post.
You are right, the intention was that but now I'm reminding that, in the middle of doing it, I thought, ok I reached the limits of the Tcl prototype, let's rewrite in C.
This naming is so confusing.
For those not aware, LMDB is the memory mapped database that's powering OpenLDAP. https://symas.com/lightning-memory-mapped-database/
LMDB is one of the backends that you can choose. BDB is more common in my experience. http://www.openldap.org/doc/admin24/backends.html
To be fair, LLOOGG Memory DB is apparently earlier than Lightning Memory-Mapped Database
As someone who worked for a company for a few years on a vast Tcl codebase, a company in which almost every engineer lamented the fact that they were working in an unused and unknown language, it warms my heart to see that it was used for this.
Antirez has long-been a TCL fan, as you can see by his piece here:
http://antirez.com/articoli/tclmisunderstood.html
I dabbled with it myself, writing a simple TCL interpreter in C, but I always find myself comparing it to FORTH a language I still find it easy to love.
Nice to see Tcl used here.
If you enjoyed reading this, you might enjoy reading git's first self-hosted commit: https://github.com/git/git/commit/e83c5163316f89bfbde7d9ab23...