The Pace of AI Progress — presenter guide

Learning goals

What the activity itself demonstrably shows, one per stage:

1. On the tests the field cites most, the change since 2023 is measured in multiples, not points — and every series stops where its published record stops, conditions attached.
2. A benchmark is retired when it stops measuring what it was meant to; being beaten is one of four routes, and the replacement often counts something new — so a rising line across generations is not one trend.
3. An achievement is an event with an acceptance status and a stated AI role; read the rung before the headline — every rung from Nobel Prize to retracted has a real 2023–2026 occupant.
4. A gap in published history can be estimated; a future cannot — three rules fit the same two points, and only the next measurement chooses between them.

Where this sits on the path

Demo 2 showed a language model being measured — a score is a measurement, not a property of a mind. This demo hands that forward to published measurements: read one with its conditions, and do not extend it past where it stops. Demo 4 takes the question this one raises — progress this fast invites “is it a bubble?” — and answers it with history rather than a script.

One question for the session

How far has AI actually come, and how would you know? The honest answer is large and specific: multiples, on named tests, under stated conditions, with the boundary on the same slide.

The one sentence to land: the change is bigger than most rooms guess, and a number is only a result when it comes with its conditions. Both halves, in that order.

The three facts worth a projector

Ninety seconds each, no benchmark literacy needed.

Fourth, if a room wants money: ARC-AGI-1 at 87.5% cost about $4,560 per task in December 2024; today 98.5% costs 28 cents, and ARC-AGI-2 — the harder successor — reaches 95% at $1.12. A harder test, a higher score, roughly four thousand times less money.

The shape of the session

Measure 10 min (two series drawn with the room, three offered) · Retire 6 (one family opened properly) · Achieve 6 (top rung, then bottom) · Forecast 4 (drag to year 8, then the boundary card).

The ten-minute cut: stage 1 with the METR series only, stage 3 filtered to Nobel Prize and to contested, then the last paragraph of stage 4 — magnitude, acceptance and the anti-extrapolation point, without the benchmark-family detail.

Stage 1 — Measure

Question. How much did the most-cited AI measurements move, and could you have guessed the size of it?

Predict prompt. “GPT-4 managed tasks that take a person three and a half minutes. Where is that by late 2025 — ten minutes? An hour?” Take numbers out loud, then draw the room's median, not your own.

Activity. Do the GDPval series yourself so the room sees the mechanic, then hand over the METR series and take a show of hands before Reveal. Drag across the shaded region, or focus the chart and use Up/Down. Choose a series jumps between the five.

Expected observation. Almost every room draws below the published line, on every series. Name that as a fact about the measurements, not a failing of the room — and do not let it become “draw higher next time”, the error stage 4 exists to kill. The reveal's what would have broken this line list is the payload: a successor released inside the gap, a harness change, a problem set corrected in 42% of its items.

Second half. Ask what the hatched gap between the last dot and the snapshot date means: nothing was measured there, by this page.

Check-yourself answer key. “Your line was below the published result. What does that show?” — That the results moved faster than the room guessed, on these tests under these conditions. Not that the room can forecast by drawing further.

Teaching limit. Five series are a teaching selection, not a survey. The ratio at reveal compares an endpoint with one selected report; it does not grade anyone's reasoning.

Stage 2 — Retire

Question. Every widely cited test gets replaced. Were they all simply beaten?

Predict prompt. “SWE-bench Verified was the standard AI coding benchmark, and it has been retired. What was the top score when its own co-author stopped reporting it?” Let the room guess high: 76.80%, and the reason is not the score.

Activity. Open Software engineering first; it breaks the tidy story. Read OpenAI's own words off the card: 59.4% of a hard 27.6% subset had flawed tests, and every frontier model could reproduce the human bug fix from memory. Then ARC-AGI, for the generation that changed the unit.

Expected observation. Four routes, and only one is “beaten”. Rooms assume contamination is rare; the Terminal-Bench leaderboard carries a canary string begging not to be trained on, which lands it better than any argument.

Check-yourself answer key. “Terminal-Bench went 2.0 → 3.0 → 4.0. What does the higher number tell you?” — Only that the old version stopped measuring what it was meant to: 4.0 has 66 tasks to 2.1's 89, and one model scores higher on 4.0.

Teaching limit. Four families, not the field. The status and route chips are labelled Reasoned because they are this page's reading of the quotation beside them — a reader is invited to disagree, which is why the quotation is shown.

Stage 3 — Achieve

Question. Which of these happened the way the headline says, and how would you tell?

Predict prompt. “The news said AI solved a Millennium Prize Problem. On a ladder from Nobel Prize down to retracted, where does that sit?” Take answers before filtering.

Activity. Filter to Nobel Prize — one card, and it took years. Then Announced, contested — one card, the biggest headline on the page; read its Who checked it line aloud, in full. Then Retracted: the same organisation nine months earlier.

Expected observation. The rung ordering inverts the news ordering: the loudest claims sit at the bottom, and the one with a prize is two years old and not a language model.

Say this about Navier–Stokes, precisely. An AI system produced a proof, checked in Lean, that closed the last gap in a strategy two human mathematicians had built — for one of the four statements Clay accepts as resolving the problem. Clay's page still reads “Active”; it is not peer-reviewed; Clay's rules need publication plus two years plus general acceptance, so no prize was possible before 2028 whatever anyone thinks of the proof; OpenAI states it does not intend to claim it; the priority dispute is unresolved; and statement (C) leaves (A), the unforced case everyone means, open.

That long version is not hedging — it is a better story than the headline, and it survives the one person in the room who knows the field. If you shorten it, shorten it to: a real, checkable advance on one of the four statements Clay accepts, with AI closing a specific gap inside a human proof strategy. Never to “AI solved a Millennium Prize Problem”.

Check-yourself answer key. “Which rung does the September 2026 Navier–Stokes result sit on?” — Announced, contested.

Teaching limit. Fourteen cases, weighted toward mathematics because that is where the strongest and the most contested recent claims are. The ladder ranks how thoroughly a claim has been checked, not how important it is.

Stage 4 — Forecast

Question. What can two measurements tell you about next year?

Predict prompt. “A score went 20 to 30 in a year. Eight years on, how far apart can three rules that fit those points be — a few points, or a factor of ten?”

Activity. Drag to year 8: the compound rule leaves the scale entirely, the saturation rule is still under 75. Then read the boundary card — five sentences, deliberately short.

Expected observation. A room that has just watched five series move by multiples wants to extrapolate; this is the stage that refuses. Say the reconciliation out loud: stage 1 filled a gap in published history, where an answer existed. A forecast has no answer yet, and two points do not choose the shape.

Check-yourself answer key. “Which rule fits the two points?” — All three, and by year 8 they disagree by a factor of ten.

Teaching limit. The scenario is invented, ceiling included. It shows that two points underdetermine a curve; it says nothing about which curve AI is on.

Misconceptions

Discussion questions

• Pick one series. What would you need to know to say whether it is still rising in two years — and could you have known it in February 2025?
• A vendor shows one number for its model on a benchmark. What five things do you now ask for before believing it?
• GDPval says professionals rate the machine's work as good as or better seven times in ten; the developer trial says experienced developers were 19% slower. Both careful. What question would distinguish them for your own work?

Questions you will get

“So did AI solve a Millennium Prize Problem or not?”

Give the precise version in stage 3 and do not flinch from its length. Short form: a real, checkable advance on one of Clay's four statements, with AI closing a gap inside a human strategy; not peer-reviewed; not recognised; no prize; disputed on priority. Pushed for yes or no: “not as the news said it, and more than a sceptic would expect.”

“Aren’t these numbers from the companies selling it?”

Some are, and the page labels them. GDPval is OpenAI's — but the graders are independent professionals judging blind, and Epoch AI republished the scores. METR is a non-profit with no product; the ARC Prize and Terminal-Bench boards are third-party and publish cost beside score. FrontierMath is the sharpest case: OpenAI funded it and holds exclusive access to part of it, and Epoch says so on its own page, and Epoch's own run reproduces OpenAI's 97.6% exactly.

“Why does that one card have two different numbers for GPQA?”

Because OpenAI published two. GPT-4o's GPQA Diamond pass@1 is 50.6% in the o1 post's table; the May 2024 launch chart, labelled plain “GPQA”, shows 53.6% for the same model. Two runs, two pages, one vendor. Showing both rather than picking one is the whole lesson in a row.

“Where did you get this?”

Open Settings → Run data integrity checks in front of them. It re-derives live that every reference resolves to a source dated to the snapshot, that every point carries its conditions, that every series says where it stops, and that every label word is from its closed vocabulary. Then open Details on any stage: each lists its sources with the fetch date and how the page was read — fetched, rendered in a browser, or downloaded as a dataset.

“Is this current?”

Every source was opened on the snapshot date printed on the chart, and each series is drawn only as far as its published record goes — that is what the hatched stretch means. Re-check the sources each semester: a stale page here would teach the opposite of its own lesson.

Presenter controls

• Settings → Presentation mode for projector-sized type; it also enlarges the chart and puts a Notes button in the header.
• Run the data integrity checks before the room arrives, so you know they are green.
• Between rooms, Settings → Reset restarts the demo without reloading — drawn lines cleared, first series, stage 1, filters and scenario back to their first state — and keeps Presentation mode as set.
• Nothing is stored, so an ordinary reload resets it too. No network is needed at any point.
• Top right: ? is the Guide (the on-load how-to, not this document); ⚙ is Settings, holding this document, Presentation mode, Reset and the checks.