Murray State University · Part one · What AI is

How a Language Model Works

Murray State University
Demo 2 of 17 · Part one · Stage 1

A model learns to predict what comes next

How can changing thousands of numbers turn random text into recognizable patterns?

Before you start. Demo 1 handed forward one idea — a model is a representation that can be trained — and ended on the transformer, the one network family behind text, image and audio models alike. This page trains one. It reads text as tokens (here, single characters), and everything it knows sits in its parameters, the numbers training changes. Nothing on this page is pretrained: the model starts as seeded random numbers, on your device.

Predict

This model has 42,458 parameters and trains on one short story. After all four eras, can it do simple math — say, 2 + 2?

Try

Press Train · era 1 and watch the loss fall, then read the new sample against Era 0. In Try next-token prediction, change the prompt and the temperature — then type 2+2= as the prompt and read what comes back.

Takeaway

Training changes the parameters; inference runs the frozen parameters forward to score every possible next token. A small model trained on a little text learns the shape of the language long before it learns to say anything true.

Live backpropagation on this device

Train the model

untrained

This small transformer begins with random parameters. Its only training text is one original story.

The untrained sample will appear here.

step 0 loss — speed —/s

No loss samples yet.

Open the model’s training text

Live forward pass, frozen parameters

Try next-token prediction

Choose an era, give the model some text, and inspect the eight most likely next characters.

Model era

Lower temperature favors the strongest probability. Higher temperature makes lower-ranked tokens more likely. Other characters are omitted, so the eight displayed percentages need not sum to 100%.

Live read off this model’s own parameters

Inspect the live model

Each view says which live checkpoint it reads.

This teaching model uses one character per token. Type to see each character’s numeric ID.

Its vocabulary contains 26 characters. Production language models commonly use larger word-piece vocabularies.

Check yourself

Era 4 writes much more like the training story than Era 0 did. What changed?

Answer from what you just watched in Train the model and Inspect the live model.

Bryant Harrison · Murray State University
One HTML file · runs offline

Guidehow to use this demo

Follow one idea across three stages: a language model predicts tokens, worked steps can support computation, and an agent loop connects text to tools.

Demo 2 of 17, Part one. Demo 1 handed forward that a model is a representation that can be trained, and ended on the transformer; this demo trains one. It hands forward that a model reads its context (Demo 5) and that a score is a measurement under conditions (Demo 3).

  1. LLM. Read the Era 0 sample, press Train · era 1, and compare eras. In the next-token workbench, change the prompt, era and temperature. Use Tokenize text, Inspect parameters and Trace attention for deeper inspection.
  2. Reasoning. Press Run experiment · train both and compare the live exam results. Then try more frozen-model attempts and Fine-tune on accepted examples.
  3. Agents. Choose You choose the actions or Watch the scripted policy. Change Visible context, restart, and observe what the activity can no longer see.

Getting around

The LLM, Reasoning and Agents tabs switch stages. Details opens the evidence and limits for the current stage. The Guide button reopens this card; Settings contains Open Presenter Notes, Presentation mode and Reset.

Allow time for training

LLM eras can take several seconds. The Reasoning experiment can take a few minutes on a phone. Progress measurements update while the page works.

Settings

Presenter notes

The printable session plan, on screen: timings, the beats to hit, the lines that land, the misconceptions, and the numbers this seeded run produces.

Presentation mode

Scales the whole page up one notch for projecting to a room, and puts a Notes button in the header for the stage cues.

Reset

Back to the start, without reloading the page: the transformer is untrained again, every era is gone, the two Reasoning models and their exam tables are cleared, and the Agents run is back at turn one with the full context. Presentation mode stays as you set it.

DetailsLLM

What is live

A small transformer you can inspect

Live every number on this stage

The model is created, trained, sampled and inspected in your browser. The loss, the era samples, the next-token probabilities, the parameter heatmaps and the attention maps are all computed here, on this device, while you watch. Nothing is replayed from a recording.

The page creates an untrained transformer in JavaScript: character embeddings, position embeddings, two attention-plus-feed-forward blocks, normalization and an output layer. Its 42,458 parameters begin as seeded random numbers and update during training.

Training repeatedly asks the model to predict the next character in one small original story. Backpropagation computes gradients; Adam uses them to update the parameters. Inference then uses a selected frozen era to produce next-token probabilities.

How the pieces fit

Tokens become numeric vectors. Causal attention lets each position combine information from earlier positions. Feed-forward layers transform those combined values. The output layer produces one score per possible next token.

Attention weights are real intermediate values from this model. They show a routing pattern for one computation, not a complete explanation of why the model produced an answer.

Sourced the GPT-2 and GPT-3 bars

This page’s own parameter count is computed from the model it just built. The two comparison bars are published figures, cited below. Parameter count alone does not describe training data, architecture, capability or quality — the bars are there for scale, and for nothing else.

Assumptions and simplifications

  • One character per token. Production models use much larger word-piece vocabularies, so the tokenizer here is the simplest honest version of the idea, not a miniature of a real one.
  • One short original story as the entire training corpus. Real models train on corpora many orders of magnitude larger, which is most of why they can say true things and this one cannot.
  • Two attention-plus-feed-forward blocks. The shape of the computation is the same as a large decoder-only model; the scale is not, and scale is not a detail.
  • Training is deterministic here (fixed seeds), so the numbers reproduce exactly. A real training run is not reproducible in this way.
  • Attention weights show a routing pattern for one computation. They are not an explanation of why the model produced an answer, and reading them as intent is the most common way to over-interpret this panel.

Sources

Boundary: model initialization, training, loss, samples, probabilities, parameter heatmaps and attention maps are computed on this device. The page makes no runtime network request and loads no pretrained weights. This tiny character model demonstrates selected mechanisms; it is not a miniature claim about every production language model.

Presenter Notesthe printable guide, on screen

Murray State University

How a Language Model WorksDemo 2 of 17 · Part one · 20 minutes

You need: one browser and index.html. It runs offline. For projection use Settings → Presentation mode. Start fresh: the untrained model is useful evidence.

Learning goals

What a learner knows at the end of each stage — these are the three Takeaway cards, verbatim.

  • LLM. Training changes the parameters; inference runs the frozen parameters forward to score every possible next token. A small model trained on a little text learns the shape of the language long before it learns to say anything true.
  • Reasoning. Room to write intermediate steps lets the same machinery reach answers it cannot produce in one jump — a real, measurable gain on this task, and not the same thing as understanding arithmetic.
  • Agents. Tools and a loop turn prediction into action, and the context window bounds what the loop can still see — a run can fail for want of memory, not for want of ability.

Where this sits on the path

Demo 1 showed models that play and handed forward one idea, in its own words: a model is a representation that can be trained, and architecture, training method and model type are three separate questions. Its last stage ended on the transformer; this demo trains one. It shows a model that predicts text, then what it takes to reason and to act. It hands forward two things: a model reads its context (Demo 5, Prompting) and a score is a measurement under conditions — the exam rows here — which is how Demo 3 asks you to read published benchmarks.

One question for the session

What changes when a text predictor is given more structure, more checking, or tools? This page shows three bounded mechanisms: a tiny language model learning next-character prediction, arithmetic experiments with written work and checking, and a tool loop with a scripted action selector. Use Details for the deeper architecture and implementation notes.

“Keep asking what the program learned, what it is calculating now, and what part is a teaching fixture.”

LLM · Language model — 6 minutes

Question

How can adjustable numbers turn recent text into a next-character prediction?

Predict prompt

After all four eras, can it do simple math, like 2 + 2? The answer is no: its vocabulary has no digits, so 2+2= reaches it as blank spaces. Have someone type 2+2= into the prompt box to see it. The reveal says so on the card once Era 4 finishes, and Stage 2 trains the same kind of model on sums.

Activity

0:00
Open LLM. Use Tokenize text and Inspect parameters on an untrained prompt. The numbers begin as random values.
1:30
Choose Train the model. Read one untrained sample, then narrate the loss and samples as the first era completes. Use Try next-token prediction to compare likely next characters; change Temperature only after the audience sees the ranked choices.
4:30
Choose Trace attention. Select a character and show which earlier positions receive weight for that prediction. It is a live intermediate value, not evidence that the model has a mind or intent.

Expected observation

Training changes the weights, loss declines, and the model’s next-character distribution changes. Its samples can become more patterned while still being limited by one short, original story.

Check-yourself answer key

Era 4 writes more like the story than Era 0. What changed? Its parameters. Not a lookup of stored passages (nothing is stored to retrieve — the whole model is 42,458 numbers) and not a bigger vocabulary (Tokenize text shows the same 26 characters at every era).

Teaching limit

This is a character-level transformer trained from scratch on a tiny corpus. It demonstrates a transformer learning mechanism, not general language ability or a claim about every frontier system.

Reasoning · Worked steps — 8 minutes

Question

How do written intermediate steps and checking change a small model’s behavior on addition?

Predict prompt

Same exam, which scores higher — answers-only or worked steps? Take the vote before pressing Run experiment; the card echoes each guess and settles it from the two scores on screen.

Activity

6:00
Open Reasoning and choose Train and compare. Compare direct-answer training with worked column arithmetic. The models share core width, depth, and heads, but their contexts, seeds, and update counts differ to fit their different text formats.
8:30
Choose Compare exam answers. Read a live attempt and an exam row. The useful comparison is the presence of written intermediate work, not a universal performance promise.
10:30
Optional slow extension: Try more attempts. Run 1, 2, 4, and 8 attempts with the same frozen under-trained model. The arithmetic checker rejects malformed work and accepted chains vote. More attempts add inference-time computation, not training.

Expected observation

Written steps provide smaller targets for the model to predict. Sampling plus checking can improve the chosen answer even when weights stay fixed.

Check-yourself answer key

The worked-steps model scores higher. What does that show? That generating intermediate steps gives the same machinery more room to compute. The card’s two wrong options are wrong in opposite directions and both are common in a room: “it understands arithmetic” (too strong — both are next-token predictors, and the fine-tuning is supervised, not reinforcement learning) and “nothing changed, both just predict the next token” (too weak — the two exam scores on the page are the evidence against it). Give both corrections equal time.

Teaching limit

The checker validates column arithmetic from the addends. The optional accepted-example round also retains chains matching the generated answer, then uses ordinary next-token updates on those examples. That is rejection-selected supervised fine-tuning, not a policy-gradient reinforcement-learning update.

Agents · Tool-using loop — 5 minutes

Question

What does a system need in addition to text prediction to take actions and use results?

Predict prompt

Same task with Visible context set to Tiny — does the run still finish? Take the vote before the tiny run; the card settles it when the run ends.

Activity

14:00
Open Agents, then choose You choose the actions. The calculator, search, reading, notebook, observations, and distractor consequences all run locally.
16:00
Choose Watch the scripted policy. Point out the calculator’s deliberate bad syntax and its genuine error message, then the later corrected call. Open the raw context to see what the action selector can read.
18:00
Choose Tiny · last 450 characters, Restart run, and Watch the scripted policy again. The goal leaves the visible context and the automatic run stops.

Expected observation

Tool results become new text in the context. The small window changes what the selector can use, so the same task can fail when its goal is no longer visible.

Check-yourself answer key

On Tiny the same policy stops without finishing. Why? The goal scrolled out of the visible context. Not a weak policy (it is the same hand-written policy and it finishes on the full context) and not the calculator error (that error is real, deliberate, and recovered from in both runs).

Teaching limit

The action selector in this activity is a scripted policy, including the calculator mistake and retry; it is not a learned model. The loop, local tools, observations, and context truncation are the genuine mechanisms on display.

Optional slow experiments and close

Let Train the model and Train and compare continue while you discuss their live displays; each can take a few minutes. Try more attempts and Fine-tune on accepted examples are optional extensions. Do not promise a particular score or curve: the page uses seeded, bounded demonstrations.

“A language model predicts. Reasoning experiments add written work, checking, or extra attempts. An agent adds a loop and tools; in this activity, its action choice is deliberately scripted.”

  • Which claim came from a learned model, a fixed calculation, or a scripted policy?
  • Why can extra attempts help without changing weights?
  • What would need to change before this agent activity used a learned action selector?

Misconceptions to expect

It is just predicting the next token, so it cannot really compute anything.It does only predict the next token — and the worked-steps model still scores measurably higher on the identical exam. Prediction with room to write steps is computation. The two exam scores are the evidence; show them, do not argue it.
The worked-steps model understands arithmetic.Same architecture, different training text. It reaches more right answers; nothing on the page supports comprehension, and the fine-tuning is ordinary supervised training on its own accepted examples, not reinforcement learning.
Attention shows what the model is thinking.Attention weights are real intermediate values from this computation — a routing pattern for one prediction. They are not intent and not an explanation of why the answer came out as it did.
Frontier models are this recipe plus scale.The shape of the computation is the same as a decoder-only transformer. The scale, the data, and much of the training pipeline are not, and scale is not a detail. This page demonstrates selected mechanisms, not a miniature of every production model.
The agent got smarter or dumber between the two runs.It is the same scripted policy both times. What changed is what the loop could still see. A run can fail for want of memory, not for want of ability.

Presenter controls

Guide opens from the header and on entry. Settings contains Open Presenter Notes, Presentation mode, and Reset. Reset puts the whole demonstration back to its fresh state in place, without reloading the page: the transformer is untrained again, every era is gone, the Reasoning models and exam tables are cleared, and the Agents run is back at turn one on the full context. Presentation mode stays as you set it. Open printable guide opens the rendered copy of this same guide.

Bryant Harrison · Murray State University · How a Language Model Works · Offline, single-file demonstration · no account or runtime network

Open the printable copy (demo-guide.html) — the same document, laid out for A4; the PDF beside it is a render of that file.
Bryant Harrison · Murray State University