Bang up article! As someone who doesn't get to do enough (almost any) calculus in my daily programming assignments, I thoroughly enjoyed reading about Kevin's dive into that derivation (linked in the supplemental article). All the people being negative here can swallow raisins
The only Rust section is the one on storage improvements about the struct that stores the hash, but do they really need that many hashes that 2 bytes makes that big of a difference? Article doesn't expand, but I guess it's a hash for every task on every computer, so maybe yes.
That's exactly his point/the area of cost saving - that they didn't actually need as many hashes as they had started with. The trick was in finding out how many hashes they could cull without degrading load balance.
These optimizations are impressive, but it gets me thinking: at what point does a company become a collection of impenetrable siloes, where nothing really does what you expect? Maybe know with AI this is less of an issue as exploring a codebase is also much faster.
I feel like any company whose products have RESTful interfaces are already there…
One wants to turn on an indicator on a remote device. A simple Boolean value. But we need networking, TLS, authentication plugins, certificate validation, distributed logging, containers, orchestration, HTTP client/server, interprocess communication, daemon dependency management, …
Sure, one can say each of these layers and abstractions has an important and justifiable purpose. But one can also step back and start wondering - what the hell are we really doing???
At some level, it seems like each layer of abstraction has to manage others, only simply because they exist.
Imagine the simplicity of 1800s telegraph signaling - no software!
Too often we build systems with Fortune-50 style hierarchies when a 5-person team could do the whole job.
And at what point does a company start to care about performance? 100TB of RAM is expensive as hell, but getting products to market faster was worth the cost
Cloudflare is truly amazing, they have made so much possible for my main side-project at a price and performance that I can’t really take credit for (http://sourcelibrary.org), I don’t care if their text was written with AI, I just wish I could get my own AI to sing so well about hashing… but wait.. today I noticed Claude trying to use hashing when a timestamp would honestly do, and now I’m really doubting myself, hmm…
This is pleasant coincidence. Really like your site and it's queued to send in my newsletter in the morning! Just happened to see your comment here while I was reading. Great work.
I don’t care if their text was written with AI, I just wish I could get my own AI to sing so well about hashing… but wait.. today I noticed Claude trying to use hashing when a timestamp would honestly do, and now I’m really doubting myself, hmm…
Tried out the first 1000 words in Pangram, and it seemed happy it was human written. Not surprised either, it has been some of the better writing I've seen out of Cloudflare recently.
CPU DRAM can't really be used for inference efficiently -- inference mostly wants memory bandwidth, not memory capacity, and GPU DRAM has >10x more bandwidth. The fabs can switch between them but you can't switch after the fact.
Those machines with GPUs still need RAM of their own, and they generally want large caches to avoid SSD penalties. You even see this spill out in the form of costs for KV cache in <1min, 5m, 1hr rates etc.
Bang up article! As someone who doesn't get to do enough (almost any) calculus in my daily programming assignments, I thoroughly enjoyed reading about Kevin's dive into that derivation (linked in the supplemental article). All the people being negative here can swallow raisins
Thanks! Maybe dial it back or people are going to think I paid you
The only Rust section is the one on storage improvements about the struct that stores the hash, but do they really need that many hashes that 2 bytes makes that big of a difference? Article doesn't expand, but I guess it's a hash for every task on every computer, so maybe yes.
That's exactly his point/the area of cost saving - that they didn't actually need as many hashes as they had started with. The trick was in finding out how many hashes they could cull without degrading load balance.
These optimizations are impressive, but it gets me thinking: at what point does a company become a collection of impenetrable siloes, where nothing really does what you expect? Maybe know with AI this is less of an issue as exploring a codebase is also much faster.
My intuition around larger companies is they are already impenetrable silos, and AI makes it worse
I feel like any company whose products have RESTful interfaces are already there…
One wants to turn on an indicator on a remote device. A simple Boolean value. But we need networking, TLS, authentication plugins, certificate validation, distributed logging, containers, orchestration, HTTP client/server, interprocess communication, daemon dependency management, …
Sure, one can say each of these layers and abstractions has an important and justifiable purpose. But one can also step back and start wondering - what the hell are we really doing???
At some level, it seems like each layer of abstraction has to manage others, only simply because they exist.
Imagine the simplicity of 1800s telegraph signaling - no software!
Too often we build systems with Fortune-50 style hierarchies when a 5-person team could do the whole job.
Build a system as simple as possible but no simpler.
An 1800s telegraph system doesnt work in the modem world, there is far too much communication and the system would just collapse into molten slag.
All those things you've listed are because we live in an adversarial world and I'd steal all your money off the telegraph wire if you tried it.
Having worked in hardware for a moment, everything we do in software is like this. Even C.
There are a whole series of blog posts from the Fishworks guys explaining why it could possibly be so hard to turn on one LED.
But Oracle probably deleted then so you'll have to find them on archive.org.
And at what point does a company start to care about performance? 100TB of RAM is expensive as hell, but getting products to market faster was worth the cost
Cloudflare is truly amazing, they have made so much possible for my main side-project at a price and performance that I can’t really take credit for (http://sourcelibrary.org), I don’t care if their text was written with AI, I just wish I could get my own AI to sing so well about hashing… but wait.. today I noticed Claude trying to use hashing when a timestamp would honestly do, and now I’m really doubting myself, hmm…
This is pleasant coincidence. Really like your site and it's queued to send in my newsletter in the morning! Just happened to see your comment here while I was reading. Great work.
Tried out the first 1000 words in Pangram, and it seemed happy it was human written. Not surprised either, it has been some of the better writing I've seen out of Cloudflare recently.
An AI would have known that saying, “Hi, mom” in a professional post was a bad idea.
anyone remember 100tb hosting company?
Someone really needed a few hundred TB to waste on inference and went looking under the rugs…
CPU DRAM can't really be used for inference efficiently -- inference mostly wants memory bandwidth, not memory capacity, and GPU DRAM has >10x more bandwidth. The fabs can switch between them but you can't switch after the fact.
Those machines with GPUs still need RAM of their own, and they generally want large caches to avoid SSD penalties. You even see this spill out in the form of costs for KV cache in <1min, 5m, 1hr rates etc.
I definitely enjoyed this writing style more than a lot of the recent cf blog posts. Cool article!
does this mean RAM prices can go down now? Please?