A Mathematical Theory of Communication

Claude Shannon's 'A Mathematical Theory of Communication' from 1948 is one of the most influential papers ever published in science and engineering. Here, the concept of 'information' was introduced in a way that could be measured and quantified, and Shannon showed that language could be modeled as a probabilistic system.

For modern AI, Shannon's insight is critical: it is the thinking behind everything from n-gram models to today's language models. Without this fundamental understanding that language is statistically constrained, it is impossible to understand why a language model works the way it does.