Demo 3 of 17 · Part two · 22–28 min full, 10 min cut
The Pace of AI Progress — presenter guide
One HTML file, offline, no AI at runtime. You need a screen to project or share and a mouse or
touchscreen for the drawing in stage 1. Nothing else.
Learning goals
What the activity itself demonstrably shows, one per stage:
- On the tests the field cites most, the change since 2023 is measured in multiples, not
points — and every series stops where its published record stops, conditions
attached.
- A benchmark is retired when it stops measuring what it was meant to; being beaten is one of
four routes, and the replacement often counts something new — so a rising line across
generations is not one trend.
- An achievement is an event with an acceptance status and a stated AI role; read the rung
before the headline — every rung from Nobel Prize to retracted has a real
2023–2026 occupant.
- A gap in published history can be estimated; a future cannot — three rules fit the same
two points, and only the next measurement chooses between them.
Where this sits on the path
Demo 2 showed a language model being measured — a score is a measurement, not a property of
a mind. This demo hands that forward to published measurements: read one with its
conditions, and do not extend it past where it stops. Demo 4 takes the question this one raises
— progress this fast invites “is it a bubble?” — and answers it with
history rather than a script.
One question for the session
How far has AI actually come, and how would you know? The honest answer is large and
specific: multiples, on named tests, under stated conditions, with the boundary on the same
slide.
The one sentence to land: the change is bigger than most rooms guess, and a number is only
a result when it comes with its conditions. Both halves, in that order.
The three facts worth a projector
Ninety seconds each, no benchmark literacy needed.
Fourth, if a room wants money: ARC-AGI-1 at 87.5% cost about $4,560 per task in December
2024; today 98.5% costs 28 cents, and ARC-AGI-2 — the harder successor —
reaches 95% at $1.12. A harder test, a higher score, roughly four thousand times less money.
The shape of the session
Measure 10 min (two series drawn with the room, three offered) · Retire 6 (one family
opened properly) · Achieve 6 (top rung, then bottom) · Forecast 4 (drag to year 8,
then the boundary card).
The ten-minute cut: stage 1 with the METR series only, stage 3 filtered to Nobel Prize
and to contested, then the last paragraph of stage 4 — magnitude, acceptance and the
anti-extrapolation point, without the benchmark-family detail.
Stage 1 — Measure
Question. How much did the most-cited AI measurements move, and could you have guessed
the size of it?
Predict prompt. “GPT-4 managed tasks that take a person three and a half minutes.
Where is that by late 2025 — ten minutes? An hour?” Take numbers out loud, then draw
the room's median, not your own.
Activity. Do the GDPval series yourself so the room sees the mechanic, then hand over the
METR series and take a show of hands before Reveal. Drag across the shaded region, or
focus the chart and use Up/Down. Choose a series jumps between the five.
Expected observation. Almost every room draws below the published line, on every series.
Name that as a fact about the measurements, not a failing of the room — and do not let it
become “draw higher next time”, the error stage 4 exists to kill. The reveal's
what would have broken this line list is the payload: a successor released inside the
gap, a harness change, a problem set corrected in 42% of its items.
Second half. Ask what the hatched gap between the last dot and the snapshot date means:
nothing was measured there, by this page.
Check-yourself answer key. “Your line was below the published result. What does
that show?” — That the results moved faster than the room guessed, on these tests
under these conditions. Not that the room can forecast by drawing further.
Teaching limit. Five series are a teaching selection, not a survey. The ratio at reveal
compares an endpoint with one selected report; it does not grade anyone's reasoning.
Stage 2 — Retire
Question. Every widely cited test gets replaced. Were they all simply beaten?
Predict prompt. “SWE-bench Verified was the standard AI coding benchmark, and it
has been retired. What was the top score when its own co-author stopped reporting it?” Let
the room guess high: 76.80%, and the reason is not the score.
Activity. Open Software engineering first; it breaks the tidy story. Read
OpenAI's own words off the card: 59.4% of a hard 27.6% subset had flawed tests, and
every frontier model could reproduce the human bug fix from memory. Then ARC-AGI, for the
generation that changed the unit.
Expected observation. Four routes, and only one is “beaten”. Rooms assume
contamination is rare; the Terminal-Bench leaderboard carries a canary string begging not to be
trained on, which lands it better than any argument.
Check-yourself answer key. “Terminal-Bench went 2.0 → 3.0 → 4.0. What
does the higher number tell you?” — Only that the old version stopped measuring what
it was meant to: 4.0 has 66 tasks to 2.1's 89, and one model scores higher on 4.0.
Teaching limit. Four families, not the field. The status and route chips are labelled
Reasoned because they are this page's reading of the quotation beside them — a
reader is invited to disagree, which is why the quotation is shown.
Stage 3 — Achieve
Question. Which of these happened the way the headline says, and how would you tell?
Predict prompt. “The news said AI solved a Millennium Prize Problem. On a ladder
from Nobel Prize down to retracted, where does that sit?” Take answers before filtering.
Activity. Filter to Nobel Prize — one card, and it took years. Then
Announced, contested — one card, the biggest headline on the page; read its
Who checked it line aloud, in full. Then Retracted: the same organisation nine
months earlier.
Expected observation. The rung ordering inverts the news ordering: the loudest claims
sit at the bottom, and the one with a prize is two years old and not a language model.
Say this about Navier–Stokes, precisely. An AI system produced a proof, checked in
Lean, that closed the last gap in a strategy two human mathematicians had built — for one
of the four statements Clay accepts as resolving the problem. Clay's page still reads
“Active”; it is not peer-reviewed; Clay's rules need publication plus two years plus
general acceptance, so no prize was possible before 2028 whatever anyone thinks of the proof;
OpenAI states it does not intend to claim it; the priority dispute is unresolved; and statement
(C) leaves (A), the unforced case everyone means, open.
That long version is not hedging — it is a better story than the headline, and it survives
the one person in the room who knows the field. If you shorten it, shorten it to: a real,
checkable advance on one of the four statements Clay accepts, with AI closing a specific gap
inside a human proof strategy. Never to “AI solved a Millennium Prize Problem”.
Check-yourself answer key. “Which rung does the September 2026
Navier–Stokes result sit on?” — Announced, contested.
Teaching limit. Fourteen cases, weighted toward mathematics because that is where the
strongest and the most contested recent claims are. The ladder ranks how thoroughly a claim has
been checked, not how important it is.
Stage 4 — Forecast
Question. What can two measurements tell you about next year?
Predict prompt. “A score went 20 to 30 in a year. Eight years on, how far apart can
three rules that fit those points be — a few points, or a factor of ten?”
Activity. Drag to year 8: the compound rule leaves the scale entirely, the saturation
rule is still under 75. Then read the boundary card — five sentences, deliberately
short.
Expected observation. A room that has just watched five series move by multiples wants to
extrapolate; this is the stage that refuses. Say the reconciliation out loud: stage 1 filled a
gap in published history, where an answer existed. A forecast has no answer yet, and two points
do not choose the shape.
Check-yourself answer key. “Which rule fits the two points?” — All
three, and by year 8 they disagree by a factor of ten.
Teaching limit. The scenario is invented, ceiling included. It shows that two points
underdetermine a curve; it says nothing about which curve AI is on.
Misconceptions
“The benchmarks just got easier, so the numbers are meaningless.”
The opposite, mostly: ARC-AGI-2 was built to resist scale and went 6.5% to 95% in
seventeen months, and on a harder mathematics set released this month the same model gets 2 of
68. Read the task count, not the version number (stages 1–2).
“A benchmark gets replaced because it was beaten.”
One of four. SWE-bench Verified was retired at 76.80% for flawed tests and contamination;
Terminal-Bench 2.1 was an environment fix; multilingual variants exist for coverage
(stage 2).
“A score is a property of the model.”
One model on one benchmark spans 18 points across harnesses, and 45 on ARC-AGI-3 at a
fixed effort setting. Effort is not even monotone: 35.2% at “none” beats 17.5% at
“low” (stages 1–2).
“If AI won a Nobel and solved a Millennium problem, it can do the
work.”
The Nobel was for AlphaFold, not a chatbot, after years of checking; the
Navier–Stokes result is unrecognised and contested. Meanwhile the one randomised trial of
experienced developers measured them 19% slower, and the best score on real paid
freelance projects is 16.1% (stages 3–4).
“Saturation means progress is slowing.”
Saturation is a property of bounded scores. METR's horizon has no ceiling, has forced no
successor, and its doubling time is shortening (stages 1–2).
Discussion questions
- Pick one series. What would you need to know to say whether it is still rising in two years
— and could you have known it in February 2025?
- A vendor shows one number for its model on a benchmark. What five things do you now ask for
before believing it?
- GDPval says professionals rate the machine's work as good as or better seven times in ten;
the developer trial says experienced developers were 19% slower. Both careful. What question
would distinguish them for your own work?
Questions you will get
“So did AI solve a Millennium Prize Problem or not?”
Give the precise version in stage 3 and do not flinch from its length. Short form: a real,
checkable advance on one of Clay's four statements, with AI closing a gap inside a human
strategy; not peer-reviewed; not recognised; no prize; disputed on priority. Pushed for yes or
no: “not as the news said it, and more than a sceptic would expect.”
“Aren’t these numbers from the companies selling it?”
Some are, and the page labels them. GDPval is OpenAI's — but the graders are independent
professionals judging blind, and Epoch AI republished the scores. METR is a non-profit with no
product; the ARC Prize and Terminal-Bench boards are third-party and publish cost beside score.
FrontierMath is the sharpest case: OpenAI funded it and holds exclusive access to part of it,
and Epoch says so on its own page, and Epoch's own run reproduces OpenAI's 97.6% exactly.
“Why does that one card have two different numbers for GPQA?”
Because OpenAI published two. GPT-4o's GPQA Diamond pass@1 is 50.6% in the o1 post's table; the
May 2024 launch chart, labelled plain “GPQA”, shows 53.6% for the same model. Two
runs, two pages, one vendor. Showing both rather than picking one is the whole lesson in a
row.
“Where did you get this?”
Open Settings → Run data integrity checks in front of them. It re-derives live that
every reference resolves to a source dated to the snapshot, that every point carries its
conditions, that every series says where it stops, and that every label word is from its closed
vocabulary. Then open Details on any stage: each lists its sources with the fetch date and
how the page was read — fetched, rendered in a browser, or downloaded as a dataset.
“Is this current?”
Every source was opened on the snapshot date printed on the chart, and each series is drawn only
as far as its published record goes — that is what the hatched stretch means. Re-check the
sources each semester: a stale page here would teach the opposite of its own lesson.
Presenter controls
- Settings → Presentation mode for projector-sized type; it also enlarges the
chart and puts a Notes button in the header.
- Run the data integrity checks before the room arrives, so you know they are green.
- Between rooms, Settings → Reset restarts the demo without reloading — drawn
lines cleared, first series, stage 1, filters and scenario back to their first state —
and keeps Presentation mode as set.
- Nothing is stored, so an ordinary reload resets it too. No network is needed at any
point.
- Top right: ? is the Guide (the on-load how-to, not this document);
⚙ is Settings, holding this document, Presentation mode, Reset and the
checks.