When you send a prompt to an LLM, the model never reads your text. Before anything else, a tokenizer breaks your sentence into pieces and swaps each piece for a number. That is what the model receives: a list of integers. Never the letters.

For a single prompt, this step is instant. You don’t even notice it happened.

But you almost never tokenize just one thing. Think about RAG: you take your company’s documents and tokenize every one of them before indexing. Or a fine-tune: you tokenize the entire dataset before training. Or an eval running over millions of examples. That invisible step turns into machine-hours.

And nobody times it. We accept slow tokenization as if it were a law of physics - “that’s the price of working with LLMs at scale”. It isn’t. When I finally measured it, there was a 200x sitting there, waiting for someone to look.

That 200x lives in a specific place: pipelines that tokenize in bulk and write the ids to disk. If your case is a user’s prompt in production, the math changes. And between those two ends sits TTFT, the time to first token, where tokenization shows up again. We’ll get to all of them.

First: how much of the time is tokenization?

Before selling a solution, it’s worth measuring the problem. Optimizing something that carries no weight is wasted time - we’ll come back to that at the end.

I set up the simplest possible scenario: 281,664 real text documents, 863 MB from the fineweb dataset. Read from disk, run through the Qwen3-8B tokenizer, write out the token ids. That’s it - no model running anywhere in the middle.

nanoGPT’s prepare.py does exactly this: it tokenizes the corpus once and writes the ids to a .bin file that training reads later. Tokenizing again every epoch would be waste.

Then I timed each step. The result:

step	time
read 863 MB from disk	0.8 s
tokenize	146 s
write 760 MB of ids	0.1 s

Tokenization was 99.4% of the time. It wasn’t a bottleneck among others. It was the whole pipeline. Reading and writing nearly a gigabyte each was noise next to it.

That changes the game. When one step eats 99% of the time, optimizing it stops being polish and becomes priority number one.

SO: killing tokenization speeds up the entire pipeline

Enter Gigatoken, a tokenizer Marcel Roed wrote in Rust. Its promise is aggressive: up to ~1000x faster than HuggingFace’s tokenizers.

Too round a number, so I got suspicious. I ran my own benchmark - same corpus, same tokenizer, on my machine (a MacBook M4 Max). Tokenization only:

tool	time	throughput
HuggingFace tokenizers	146 s	0.01 GB/s
Gigatoken	0.74 s	~1.2 GB/s

About 200x on my machine. Not the 1000x from their README, but that 1000x came from a 144-core server - the more cores, the wider the gap opens. On my laptop, 200x is enough to hurt.

And the whole pipeline, I/O and all? It dropped from 147 seconds to under 2. 88x faster end to end. Two and a half minutes of waiting became less than two seconds, on the same corpus, with the same result.

NOTE: the most important number in those tables isn’t the time. It’s that both produced exactly the same 190,429,497 tokens. Byte for byte. One of them is 200x faster doing identical work. That’s not a shortcut, that’s engineering.

This is not premature optimization

An objection popped into my head before it could pop into yours: isn’t swapping tokenizers to shave off seconds the very picture of premature optimization?

That’s what I used to think, and it’s why I had never timed this step. HuggingFace’s tokenizers helps you not look, too: it’s written in Rust, it’s multithreaded, it’s maintained by serious people. Any reasonable person glances at it and says “this is optimized”. And there were still 200x on the table.

Premature optimization would have been tuning the I/O that took 0.8 seconds. Attacking the step that eats 99% of the time is the obvious move, and it only looks bold because nobody had checked the clock before.

What does exist, and nobody has named it, is the inverse problem: late optimization - accepting a bottleneck as untouchable without ever having timed it. “Tokenization is slow” was exactly that. A wall everybody saw, until someone saw a door.

“But didn’t you pick the slowest tokenizer on purpose?”

Fair question. If I had tested only Qwen, you’d have every right to be suspicious. So I tested more.

With GPT-2, the classic: HuggingFace took 129 seconds, Gigatoken took 0.22. The 193,502,159 tokens, once again, identical. Switching tokenizers within HuggingFace doesn’t close the gap - it widens it.

But the test that really matters is against tiktoken, from OpenAI. It’s the tokenizer with a reputation for speed, the one running behind GPT-4. And it is fast: it did the same corpus in 12 seconds, about 10x faster than HuggingFace.

Gigatoken did it in 0.22 seconds. 57x faster than the fast tokenizer, with token-for-token identical output.

And here’s the interesting part: HuggingFace being slow doesn’t mean “tokenization is slow”. tiktoken does the same tokenization 10x faster just by swapping one internal mechanism. The bottleneck has a specific name, and it isn’t BPE.

How? SIMD in the part nobody was looking at

The bottleneck’s name is pretokenization.

Before the actual tokenization algorithm (BPE), there’s a mundane step: splitting the text into pre-chunks. Almost everybody does it with a regular expression. And a regex scanning gigabytes of text is slow - it crawls along more or less character by character, hunting for the pattern.

What Marcel did was replace the regex with an implementation that does the same thing with SIMD. And this is where the magic falls apart.

Imagine you need to find every comma in a thousand-page text. You can read it word by word, looking. Or you can open the text into 64 parallel columns and ask all at once: “is there a comma in any of these 64 characters right now?”. Same result. Very different time.

That’s SIMD. Single Instruction, Multiple Data. One instruction, many pieces of data. The processor grabs 16, 32, 64 bytes and runs the same operation on all of them in a single cycle.

NOTE: if you want to pull on this thread, SIMD is an entire axis of concurrency most of us ignore - it isn’t threads or processes, it’s the processor’s own ALU working in batches. Paul Butcher dedicates a chapter to it in Seven Concurrency Models in Seven Weeks (data parallelism), and Mitchell Hashimoto has a great essay arguing that every developer should know SIMD. But for what matters here, one thing is enough: this is the technique that turns 146 seconds into 0.7.

The honest part: 200x on one step is not 200x on your life

Now the warning that separates this post from a sales brochure.

I said tokenization was 99% of the time. Notice: 99% of the time of THAT test. That holds for any pipeline where no model runs alongside: you read text, tokenize, write the ids. That’s all I measured, and there tokenization dominates on its own.

The moment a model enters the same pipeline, the math changes. In a RAG you tokenize and then push each chunk through an embedding model - that forward pass is real work, and tokenization’s slice shrinks.

And if you’re tokenizing a user’s prompt before calling a model in production, the picture flips completely. During generation itself, tokenization is a crumb next to the forward pass - a 200x there changes almost nothing, because it was never the problem.

The narrow exception is the time to first token, TTFT. Gigatoken’s own author showed, in the Hacker News discussion, that on smaller models you can shave 5-10% off TTFT. It’s real - but it’s a different universe from the 88x in preprocessing. The size of the prize depends on where tokenization sits in your equation. That’s Amdahl’s Law.

The total gain from optimizing one part is capped by how much that part weighed in the whole. Optimizing 99% of the time transforms everything. Optimizing 1% changes nothing, no matter how spectacular the 200x looks.

I learned that the hard way. This year I entered the rinha de backend (a Brazilian backend performance contest) and, for the first time in my life, wrote a SIMD kernel by hand - the application did a lot of repetitive math, the perfect candidate. The math got much faster. And the final gain was disappointing.

Because the math was never my bottleneck. The time lived in everything running around it. I had brilliantly accelerated a piece that carried no weight. The same law that makes Gigatoken look like magic tripped me up in the rinha - just in the opposite direction.

What remains

The real work isn’t “install Gigatoken”. It’s measuring where your time lives before you start optimizing.

If you do data preprocessing for LLMs and have never timed tokenization, time it. If your path looks like the test’s - read text, tokenize, write the ids - it may be eating almost everything. And if it is, there’s a 200x sitting there:

pip install gigatoken

Its API mimics the tokenizers you already use, so the swap itself is one line. How much of that gain shows up on your end is another conversation, and it depends on how much of your time was in tokenization. Only the stopwatch can answer that.

But if tokenization is a crumb in your pipeline, save Gigatoken for the right day and go hunt the real bottleneck. Because in the end it’s always the same story: time lives somewhere specific, and it’s almost never where we think it does.

Thanks for reading!