How Large Language Models Actually Work (No Math Required)
Tokens, embeddings, attention, training and why models hallucinate — the complete mental model of an LLM, explained in plain English for developers.
Elena Ruiz
AI & Tools Writer
Hugo Nina
Fact Checked
You use large language models every day now — in your editor, your terminal, your search engine. And if you’re like most developers, you have a working intuition that somewhere inside there’s “a neural network predicting text,” plus a vague cloud of terms: transformers, attention, parameters, tokens.
This article turns that cloud into a clear mental model. No linear algebra, no training runs — just the concepts, connected in the right order, at the depth a working developer actually needs. By the end, behaviors that seem mysterious (hallucinations, context limits, weird arithmetic) will look like obvious consequences of the design.
One job: predict the next token
Strip away everything else and an LLM is a function:
Given a sequence of text, output a probability for every possible next chunk.
Feed it “The capital of France is” and it assigns high probability to ” Paris”, low probability to ” Madrid”, nearly zero to ” refrigerator”. Generation is just running this in a loop: pick a next token, append it, predict again, thousands of times per response.
Everything an LLM appears to do — answering, coding, translating, reasoning — is this one operation. There is no separate “code module” or “facts database.” The startling discovery of the last decade is that predicting text extremely well requires internally modeling the things the text is about: grammar, facts, code semantics, even rudimentary logic. Competence emerges from compression.
Tokens: the atoms of the system
Models don’t see words or letters. Text is first split into tokens — frequent chunks of characters from a fixed vocabulary (typically 50,000–250,000 entries). “The developer wrote tests” might become The, developer, wrote, tests — while a rare word like “hydroxyapatite” shatters into several fragments.
This single fact explains a lot of everyday weirdness:
- Why spelling games trip models up: the model literally doesn’t see letters, so “how many r’s in strawberry” is asking about the insides of atoms it can’t inspect.
- Why arithmetic is shaky: “12345 × 6789” is token patterns, not numbers in registers. Models memorize small results and approximate the rest — which is why serious tools have the model call a calculator instead.
- Why you’re billed and limited in tokens: the context window (128K, 200K, 1M tokens depending on model) and API pricing count these units. As a rule of thumb in English, one token ≈ ¾ of a word.
Each token is then mapped to an embedding — a long list of numbers representing its meaning. Words used in similar contexts get similar numbers. Meaning becomes geometry: “king” and “queen” end up near each other; both far from “carburetor”.
Attention: the mechanism that changed everything
Older text models read left to right with limited memory, forgetting the start of a paragraph by its end. The 2017 transformer architecture introduced self-attention, and it’s the reason the current AI era exists.
The intuition: as the model processes each token, attention lets it look back at every other token in the context and decide which ones matter right now. Processing “it” in “The server crashed because it ran out of memory,” attention links “it” strongly to “server”. Processing the closing brace of a function, attention can reach back to the signature hundreds of lines earlier.
Each layer runs many attention “heads” in parallel — one might track syntax, another coreference, another code structure. Stack dozens of such layers, and the model builds representations of remarkable depth: this token is a variable name, defined 200 lines ago, inside an async function, in TypeScript.
Attention is also why context windows have limits: naively, every token attends to every other token, so cost grows roughly with the square of length. The very long windows of modern models come from clever engineering around that quadratic wall — and why “needle in a haystack” retrieval in huge contexts still sometimes degrades.
Training: compression, then civilization
Pre-training is the expensive part: show the model trillions of tokens (web text, books, code) and endlessly play “guess the next token,” adjusting billions of internal parameters slightly after each guess. Months of GPU time later, you have a base model — a formidable text-completion engine with no manners: ask it a question and it might respond with more questions, because that’s a plausible continuation of text.
Turning that into a useful assistant takes a second phase:
- Instruction tuning: fine-tune on curated examples of instructions and good responses, teaching the format of being helpful.
- Preference optimization (RLHF and successors): humans (and increasingly AI judges) rank candidate responses; the model is adjusted toward preferred ones — safer, clearer, better organized.
This second phase is tiny compared to pre-training, but it’s what you actually interact with. When a model politely declines, structures an answer with headers, or says “Great question!” — that’s sculpted behavior, not emergent physics.
Why models hallucinate (and always will, a little)
A language model always outputs a plausible next token. Plausibility is the only currency; truth enters only insofar as true statements were more common in training data. Ask about an obscure library and the model may produce an API that should exist — perfectly shaped, entirely fictional. Nothing inside natively distinguishes “I know this” from “this pattern is likely.”
Understanding this reframes hallucination from “bug to be patched” to “failure mode to be engineered around”:
- Give the model the facts at inference time instead of relying on its memory — retrieval-augmented generation, which we cover step by step in RAG Explained.
- Give it tools — calculators, search, code execution — so it can check instead of guess.
- Ask for verifiable output and verify it: run the code it writes, check the citations it gives.
Modern training substantially reduces confident fabrication, and models increasingly say “I’m not sure.” But a system whose fundamental operation is produce plausible text will never be a database, and treating it like one is the root of most AI-integration failures.
From single answers to “reasoning”
You’ll also see models that “think before answering” — producing long internal chains of reasoning before the final response. Under the hood it’s still next-token prediction; the model has been trained to spend tokens working through the problem (exploring, backtracking, checking) before answering, which measurably improves math, code and logic performance. The lesson generalizes: an LLM’s quality scales with the relevant text in its context — which is why prompting techniques matter, as we detail in Prompt Engineering for Developers.
The parts around the model
What you call “an AI” in production is a system: the model, plus a context-assembly layer (system prompt, conversation history, retrieved documents), plus tool integrations, plus output validation. The model is the engine, not the car. That’s also why running models yourself is more approachable than it sounds — the engine is downloadable, as we show in Running Local LLMs.
A quick vocabulary consolidation:
| Term | Plain meaning |
|---|---|
| Parameters | The learned numbers inside the model (billions of them) |
| Tokens | Text chunks the model reads and writes |
| Embedding | A token’s meaning, encoded as numbers |
| Attention | Mechanism letting tokens use other tokens as context |
| Context window | Maximum tokens the model can consider at once |
| Temperature | Randomness dial for picking among probable tokens |
| Fine-tuning | Further training to specialize behavior |
| RLHF | Training that aligns output with human preferences |
Final Thoughts
A large language model is a next-token predictor of extraordinary quality — built from tokens and embeddings, powered by attention, educated by compressing a huge slice of human text, and finished with training that shapes raw prediction into helpful behavior. Hold that model in your head and the practical rules follow naturally: give it context instead of trusting its memory, give it tools instead of trusting its arithmetic, verify what matters, and treat its fluency as what it is — the texture of the system, not proof of certainty.
The developers getting the most out of AI right now aren’t the ones who think it’s magic, or the ones who think it’s a stochastic parrot. They’re the ones with an accurate mental model, building systems that lean on what LLMs do brilliantly and scaffold what they don’t.
