I Built a Python Compressor Using OpenAI's Tokenizer. Here's What Happened

LLMs spent billions of dollars learning the best subword dictionary that has ever existed. Somewhere around 200,000 pieces of language, ranked by how much they compress. And I kept looking at that dictionary and thinking: nobody is allowed…