Can Gzip Be a Language Model? A Deep Dive into Compression-Based Text Generation

Compression Meets Language Modeling

In the world of artificial intelligence, language models are synonymous with massive neural networks, billions of parameters, and staggering computational costs. But what if a standard compression tool—the kind that ships with your operating system—could perform language modeling too? That's the provocative question developer Nathan Barry explores in a recent project called gzipt, which uses gzip's DEFLATE algorithm to generate text.

The experiment, detailed in Barry's blog post and subsequently discussed across Hacker News and Reddit, isn't just a parlor trick. It's a practical demonstration of a deep theoretical principle: compression is prediction. Every compressor assigns probabilities to symbols, and every prediction model compresses data. The two are mathematically equivalent, a concept formalized in the paper Language Modeling is Compression.

The Compression-Prediction Equivalence

To understand how gzip can generate text, you first need to grasp the core insight. A compressor spends few bytes on data it 'expects' and many bytes on data it doesn't. A file with a million repeated 'A' characters compresses to almost nothing, while a million random bytes barely compress at all. This isn't a coincidence—it's rooted in information theory.

The number of bits needed to encode a symbol is -log2(p), where p is the probability the model assigns to it. High probability means few bits. So, any compressor has a probability model hiding inside it, whether or not anyone wrote one down. For gzip, that model is DEFLATE, which finds matches against recent text in a 32 KiB sliding window.

If a continuation echoes something already in the window, DEFLATE encodes it as a cheap back-reference instead of literal bytes. This gives a scoring mechanism: the smaller the compressed size of context plus candidate, the more 'predicted' the candidate is. Barry's tool uses this score to generate text.

How Gzipt Works: Beam Search Over Bytes

Scoring individual bytes fails because gzip only outputs integer byte lengths, creating quantization noise—many candidates tie. The solution is to look ahead a whole span before committing. Gzipt runs a beam search over byte sequences, scoring partial continuations by their compressed length.

The algorithm works in a loop: start with a prompt, show gzip the corpus plus the recent tail of generated text, then search for the most compressible continuations. At each step, it extends partial continuations by every byte that occurs in the corpus, scores them all, and prunes down to the best beam width. After a horizon of bytes, it commits to the best full span.

One critical detail: only the last 'tail' bytes of generated output stay in the scoring context. If gzip could see its entire history, it would fall into verbatim loops, endlessly copying text it just emitted. The tail constraint prevents this, forcing more varied output.

Real Output: Shakespeare Through a Compressor

Barry primed gzipt on the tiny Shakespeare corpus and prompted it with 'MENENIUS:'. The output, while not coherent prose, clearly echoes the training distribution:

MENENIUS: 'Though all at once canq

MARCIUS: Pray now, nocamest thou to a morsel .

LARTIUS: Hence, and I' the end admire, where G again; and after it ag .

The text captures character names, dialogue patterns, and vocabulary—far more than one would expect from a compressor. It's not going to win any literary awards, but it demonstrates that gzip's sliding window mechanism implicitly encodes a language model.

Beyond Text Generation: Classification and More

The implications extend beyond text generation. As noted in a Hacker News comment, you can classify a test file by topic using gzip: the file belongs to the topic with the smallest compressed size. This technique has been used for zero-shot text classification, leveraging the same compression-prediction equivalence.

This isn't meant to replace neural language models. The patterns gzip can capture are far simpler than those learned by gradient descent. But the demonstration raises fundamental questions about what constitutes a language model. Both gzip and neural models predict and compress—the difference lies in the complexity of patterns each can capture.

Why It Matters

The gzipt experiment is more than a curiosity. It highlights the thin boundary between compression and prediction, a concept with deep implications for AI research. If compression is prediction, then improving compression algorithms could lead to better language models, and vice versa.

For practitioners, it's a reminder that the tools we take for granted—like the gzip bundled with every operating system—contain hidden capabilities. The entire project is one file of pure standard-library Python (just zlib), available on GitHub for anyone to experiment with.

As Barry notes, the name 'GziPT' was too good to pass up, even though the code actually uses zlib under the hood. Both use the same DEFLATE algorithm, so the spirit is intact. The experiment doesn't suggest gzip should replace neural language models, but it illustrates that the boundary between compression and prediction has always been thinner than it appears.