Tokens and n-grams: the basics nobody explains, and the bill nobody sees coming

Ask an AI service a question, and the bill that follows never quite matches the number of words you typed. That's not a bug. It's because you never paid per word. You paid per token, a unit most people use daily without ever seeing it explained, even though it's exactly where the ongoing cost of AI in daily operation actually comes from.

What a token actually is

A token isn't a word. It's the smallest chunk of text a language model actually handles internally: sometimes a whole word, sometimes a word stem, sometimes just a few characters. "Cat" might be one token. A long, unusual compound word will likely become several. The technique behind the split is called subword tokenization (the most common variant is called Byte Pair Encoding, BPE), and the principle is simple: build a vocabulary of the text chunks that occur most often in the training data, and split everything else into smaller pieces of that vocabulary.

This has a concrete, often overlooked consequence for non-English users. English generally tokenizes more efficiently than many other languages, since most AI models' vocabularies were built with English text as the dominant source. Languages with long compound words, or scripts and structures less represented in training data, often get split into more tokens than an English equivalent would need. Same content, same meaning, but more tokens, and therefore a higher bill, simply because of which language it's written in.

N-grams: the idea is a century old, only the tool is new

Predicting the next word from the ones before it isn't a new idea invented with ChatGPT. It traces back to Claude Shannon's work on information theory in the 1940s, where he showed that language could be modeled statistically: count how often a given sequence of words is followed by a particular next word, and use those probabilities to predict or generate text. That's exactly what an n-gram model does, a model that looks at the N-1 preceding words (or tokens) and calculates the probability of what comes next, based on how often that combination has appeared in a body of text. This kind of model is covered in detail in Speech and Language Processing by Daniel Jurafsky and James Martin, the standard textbook in natural language processing, where n-grams still serve as the first and most concrete example of how language can be modeled statistically, before the book moves on to neural methods.

The point isn't that n-gram models were particularly good. They weren't; they forgot everything beyond their small window of preceding words, and could never capture context further back in a sentence or paragraph. The point is that today's LLMs solve the same underlying task, predicting the next token based on what came before, just with a neural network instead of a frequency table, and with the ability to weigh an entire context instead of just the last two or three words. That ability, to determine which earlier tokens actually matter no matter how far back they sit, is what attention solved, and it's the subject of the next article in this series.

N-grams aren't dead, they just changed jobs

Here's the part of the story rarely told: n-gram techniques never disappeared, they just changed jobs. Instead of being the language model, they're now used as a tool to clean the enormous amount of text a modern LLM is trained on, before training even begins.

Three concrete places where they're still at work:

Deduplication. Web-scraped training data contains enormous amounts of near-identical documents: reprinted articles, copied pages, repeated content. Research has shown that such duplicates make models worse and increase the risk that they memorize and later reproduce text verbatim. The standard solution is to represent each document as a set of n-grams and compare them (often using a technique called MinHash) to find and remove the ones that are nearly identical.

Contamination control. When OpenAI built GPT-3, their own paper described how they actively searched for and removed training examples that had an overlap of 13 or more consecutive tokens with the datasets that would later be used to evaluate the model. Otherwise the model could, in effect, have "studied for the test."

Quality filtering. Before a piece of text is even allowed into the training data, a separate, much simpler n-gram model (the tool KenLM and its 5-gram models are a common standard) is often trained to score how "natural" the language in the text is. Text that scores low, such as code fragments, log files, and spam, gets filtered out before it reaches the actual model.

In other words: next time you hear that n-grams are an outdated technique, that's only true if you mean as the model itself. As a tool in the machine room, they're still keeping the job running.

Why this matters for the bill

Back to the invoice. Every AI service you use is priced per token, typically with different rates for tokens you send in (your question, the entire conversation history, any documents you attach) and tokens the model sends back. A long, unstructured conversation where the entire history gets resent with every new question grows in cost with every turn, not just linearly but often noticeably, since the entire context window gets recalculated each time. In daily operation, across thousands or millions of calls, that's the difference between a marginal cost and a significant line item in the budget, and it's a difference determined by how you design your AI workflows, not just which model you choose.

Understanding tokens, then, isn't a technical side interest. It's the single most concrete, measurable cost lever in the entire operation of AI, and the one most often overlooked in decisions that are otherwise weighed carefully on every other line of the budget.