- 26comments
- 392comments
- 101comments
- 228comments
- 62comments
- 330comments
- 9comments
- 241comments
- 246comments
- 49comments
- 318comments
- 6comments
- 24comments
- 1comments
- 28comments
- 63comments
- 75comments
- 17comments
- 60comments
- 38comments
- 1comments
- 58comments
- 77comments
- 481comments
- 39comments
- 195comments
- 22comments
- 8comments
- 37comments
- —discuss
This tracks perfectly with Winrar being more profitable than OpenAI... coincidence? I think not!
winrar is profitable? sure? well, on the other hand, they sure don't make losses
They are a German GmbH and must publicly state their financials: https://www.northdata.de/win%C2%B7rar%20GmbH,%20Berlin/Amtsg...
Looks pretty profitable to me.
They're one of the few companies that actually manage to sell "boxed software" (i.e. has not changed much in years but new customers keep buying it)
That said, Windows users should use 7-Zip. Better compression format, unpacks more kinds of archives
That’s surprising. Seems there is a niche for everything.
One moment, how are we supposed to know how well that search was done? There is no way to search a meaningful part of the search space.
So the result only gives us some lower bound of how well gzip works as a "plausibility tester" of a continuation of a text. The space of possible sequences is many orders of magnitude larger than what was searched. So there might be sequences in there that compress much better.
The text mentions beamsearch, but I don't see a discussion about how well beamsearch performs in finding the global optima when it comes to gzip compressibility of a text?
That's a fair question. Suppose we have a way to find a byte sequence x that globally minimises len(gzip(context + prompt + x)) over all sequences x of length n. Here + denotes string concatenation.
It's unclear if this is very useful.
The reason it may not be very useful is that one of Deflate's ingredients is a pass that replaces repeated substrings with backreferences to the earlier occurrence in the plaintext input stream.
E.g. suppose we want to find an n=200 byte sequence x that minimises len(gzip(context+prompt+x)).
If there exists any 200 byte sequence y such that prompt+y is a substring of context, then Deflate can encode prompt+y as a backreference to that earlier sequence - it needs to store a match-length & a distance-length, encoded using its Huffman trees. This candidate solution y may not be a global minima to our stated objective function, but if not, it's probably going to be a very good near-optimal approximate solution.
Taking a step back, repeating huge chunks of the input context produces something that's great for minimising compressed output size but doesn't seem particularly helpful as a generative model.
Not without attention or something approximating it.
The fact that gzip is relatively fast should be your first clue that something important is missing.
Gzip is great at predicting the next token for one very specific narrative. LLMs can predict next tokens for entire universes of narratives. Searching for the correct next token across this space scales ~quadratically with the input size. Gzip scales linearly. I can gzip a one terabyte file. Imagine feeding that much into an LLM. These are wildly different animals that happen to overlap in a very small way. Equating compression to intelligence looks increasingly silly to me.
If we must compare language models to compression, they are much more like jpeg and mp3 than they are gzip and flac. I can go fuck with a jpeg file pretty severely at the bitstream level and still have something resembling performance on the other side. Gzip cannot remotely approach this.
Perhaps a better question is if LLMs are used as compressors, how well is that expected to work.
Extremely well, aside from speed.
And energy consumption.
Quite well. This project[1], by Fabrice Bellard of ffmpeg fame, is quite old in AI years and uses an ancient LLM, but still beats xz by a solid margin.
[1]: https://bellard.org/ts_zip/
In part because gzip only has a 32KiB window size, and I think it'd be at least quadratic within that window if you were going for optimal compression.
Match-finding does not need to be quadratic. However, truly optimal gzip block splitting is very slow, indeed.
I'll concede the window part, but Gzip runs within the physical confines of a single cpu core and is typically entirely resident in local caches. The point is not just the quadratic scaling but also what it scales with.
Show me an LLM that can run at 300 megabytes per second. Even dedicated ASICs with weights burned in will never move this fast.
I agree, but also equating LLMs with intelligence is wrong.
Previous discussions of that specific page https://news.ycombinator.com/item?id=48557691
I'm more interested in the converse question: how well does an LLM perform as a compressor, compared to gzip (ignoring its insanely lower speed)?
Top contestant in the Hutter Prize uses a neural network for compression. So fair to say, LLMs would perform pretty well compared to gzip.
Even ignoring speed per GP, the Hutter Prize's metric includes the size of the decompressor. LLMs would be disqualified for being larger than 1GB.
lossless vs lossy is the question.
Interesting approach, I wonder how this could be used as a classifier. :)
I've used LZO as a spam classifier on chat. Spam tends to be very content-less and repetitive...
[delayed]
.
This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.
3blue1brown did a series on this topic: https://www.youtube.com/watch?v=l6DKRf-fAAM https://www.youtube.com/watch?v=GlYgs6v2YfU (i think one more is yet to release)