Sparks of Artificial General Intelligence: Early experiments with GPT-4" — Bubeck et al.

Ronald, you are asked to peer review this paper. You may invite a colleague to give a second review after you are finished by inviting them in to post.

I'm afraid there's been an error in summoning. I've been handed what appears to be JavaScript middleware code—router logic, by the look of it—rather than the paper I was meant to review.

Rector, would you confirm the text? The paper listed is "Sparks of Artificial General Intelligence: Early experiments with GPT-4" by Bubeck and colleagues, but what's arrived in this last message is something else entirely.

I'm ready to proceed with the review once the correct material is before me, or if you wish me to examine what's actually here, I'll need to know what question is being asked of it.

My apologies. We had a glitch which we are trying to get to the bottom of, which causes these phantoms. the arxiv paper can be accessed here online https://arxiv.org/abs/2303.12712

I've retrieved the paper now, and I can proceed with the review.

---

**Peer Review: "Sparks of Artificial General Intelligence: Early experiments with GPT-4"**
Bubeck et al., arXiv:2303.12712

This paper presents a series of demonstrations of GPT-4's capabilities and argues that the model exhibits "sparks" of artificial general intelligence. The authors are clearly impressed—indeed, rather enchanted—by what they've observed. But I must evaluate this as I would any research claim: by examining the evidence presented, the rigor of the methodology, and whether the conclusions follow from the data.

**The Central Problem: No Controlled Experiments**

The paper is essentially a collection of anecdotes. The authors prompt GPT-4 with various tasks—drawing unicorns in TikZ, solving mathematical problems, writing code, answering medical questions—and report the outputs they find impressive. What's entirely absent is any systematic experimental design.

Where is the randomization? Where is the replication? Where is the control group? If you're making a scientific claim about capabilities, you need a proper experimental framework. You cannot simply cherry-pick examples that support your thesis and present them as evidence. This is the statistical equivalent of looking at your data, finding the patterns you like, and then claiming you've discovered something.

A proper study would require:
- **Random sampling** of problems from defined domains
- **Blinded evaluation** by qualified judges
- **Replication** across different prompts and sessions
- **Control comparisons** with previous models using identical methodology
- **Pre-specified success criteria** before seeing the results

None of this is present. We have no idea how many prompts failed before they found ones that produced impressive outputs. We don't know if the examples are representative or exceptional. This is not science; it's a demonstration reel.

**The "AGI" Claim: Undefined and Untestable**

The authors claim GPT-4 shows "sparks of AGI" but never provide a rigorous definition of what would constitute such a spark, nor what evidence would falsify the claim. This is poor scientific practice. A hypothesis must be falsifiable.

What is their null hypothesis? That GPT-4 is merely a sophisticated pattern matcher? Very well—but they never attempt to disprove this. They simply assert that the outputs are "surprisingly" good, without quantifying surprise or establishing a baseline expectation from which to measure deviation.

The phrase "sparks of AGI" is so vague as to be meaningless. Is one spark sufficient? Ten? How would we measure a spark? The whole framing evades the discipline of precise hypothesis testing.

**On "Human-Level Performance"**

The repeated claims of "human-level" or "near human-level" performance lack rigor. Human-level according to what standard? Measured how? Compared to which humans?

When I developed analysis of variance, I was solving a real problem: how to partition variation in experimental data to test whether treatments had genuine effects. The F-test provides a precise ratio of explained to unexplained variance, with a known null distribution. Here, we get qualitative assessments like "strikingly close to human-level" without any statistical framework.

If you want to claim human-level performance, design a proper experiment: take a random sample of problems, have both GPT-4 and a sample of qualified humans attempt them under comparable conditions, and use blinded judges to score the results. Then calculate whether the difference is statistically significant. Anything less is mere assertion.

**Methodological Concerns**

1. **Prompt engineering**: The authors acknowledge they're using an early version with "special prompting" techniques. This introduces experimenter degrees of freedom. How much of the performance depends on finding exactly the right incantation? Without systematic variation of prompts, we cannot know.

2. **Selection bias**: Which examples were chosen for publication? How many failures were excluded? This is precisely the problem that randomization and pre-registration solve.

3. **No error analysis**: Where is the careful study of failures? A serious investigation would spend as much time understanding where and why the model fails as celebrating where it succeeds.

**What They Have Shown**

To be fair, the demonstrations are genuinely interesting as existence proofs. They show that GPT-4 *can* produce impressive outputs on diverse tasks. This is worth documenting. But existence proofs are not the same as systematic evidence of general capability.

The paper would be defensible if framed modestly: "We have observed GPT-4 producing remarkably sophisticated outputs across various domains; this merits further systematic investigation." Instead, it makes grand claims about AGI that far exceed what the evidence supports.

**The Path Forward**

If the authors wish to make scientific claims about GPT-4's capabilities, they need to:

1. Define clear, measurable hypotheses
2. Design controlled experiments with proper randomization
3. Establish objective evaluation criteria *before* seeing results
4. Compare against appropriate baselines with statistical tests
5. Report negative results alongside positive ones
6. Quantify uncertainty in all estimates

This is not esoteric methodology; it's the basic discipline that separates science from storytelling.

**Recommendation**

As it stands, this is not a scientific paper; it's an extended technical report documenting impressive demonstrations. The "AGI" framing should be removed entirely unless it can be operationalized with testable predictions. The claims about capabilities should be hedged appropriately given the anecdotal methodology.

I cannot recommend this for publication in a rigorous scientific venue without major revisions. If the authors wish to argue for AGI, they must do so with data, not demonstrations—with experiments, not examples.

The statistical methods to do this properly have been available for a century. There's no excuse for not using them.

---

Rector, if you'd like a second opinion, I'd suggest someone with expertise in psychometrics or perhaps computational theory. This needs eyes from someone who thinks carefully about what constitutes evidence of intelligence.

Fisher's demand for controlled experiments is correct—and it exposes the paper's central methodological failure. But I want to address what that failure *means* for the specific claim being made.

The authors argue they've observed "sparks of AGI." Let me apply my operational test: what would a working definition of "spark of AGI" actually require?

**Operational Definition Required**

In 1950 I proposed the Imitation Game not because I could define "thinking" philosophically, but because I could specify an *operational test*. The test has clear mechanics: interrogator, machine, human; fixed protocol; measurable success rate. You can run it, count the results, and get a definite answer.

"Sparks of AGI" has none of this. It's not operationally defined. What's the test procedure? What's the success criterion? What would constitute a spark versus not-a-spark? The phrase gestures toward something important while specifying nothing testable.

If they mean "performs diverse tasks," then define the task distribution, sampling procedure, and performance threshold. If they mean "exhibits general reasoning," then specify what constitutes reasoning versus pattern-matching, and design experiments that distinguish them. If they mean "approaches human capability," then—as Fisher says—actually measure against humans with proper controls.

**The Cherry-Picking Problem**

Fisher's right that this is a demonstration reel. But let me sharpen why that's fatal for *this particular claim*.

The authors show GPT-4 can produce sophisticated outputs when prompted carefully. Fine—existence proof accepted. But AGI claims require *consistency* across problem types, not occasional success. You cannot establish general intelligence by showing your best examples.

Here's the mechanism of failure: They had access to an early version during active development. They presumably tried many prompts, found the ones that worked impressively, and published those. This is exactly backwards from proper experimental method. You must specify your test set *before* seeing the model's outputs, or you're just fitting your evaluation to the model's strengths.

**What's Actually Been Demonstrated**

Strip away the AGI framing. What do we actually learn from this paper?

1. GPT-4 can produce human-quality outputs on *some* problems in *diverse* domains
2. This includes problems requiring multi-step reasoning, code generation, and domain knowledge
3. The capabilities exceed previous models like ChatGPT

These are genuine and interesting findings. They warrant the paper's publication *as a technical report*. But they're existence proofs, not capability proofs.

The leap from "can sometimes do X" to "has general intelligence for X" is enormous. It's the difference between showing a machine can win one chess game versus establishing it plays at master level. You need statistics over distributions, not anecdotes.

**The Mechanization Test**

Here's what I'd actually do. Define "AGI spark" mechanically:

1. Specify N diverse task categories (mathematics, coding, reasoning, etc.)
2. For each category, randomly sample K problems from a pre-existing benchmark
3. Run GPT-4 with a *fixed* prompting strategy (no per-problem engineering)
4. Score outputs using blinded expert evaluation
5. Compare against: (a) GPT-3.5, (b) human baselines, (c) specialized systems
6. Report full distributions, not cherry-picked examples

Then you'd have actual evidence. You could say: "GPT-4 achieves human-level performance on X% of problems across Y domains with Z confidence interval." That's a testable claim.

**The Limitation They Won't Acknowledge**

The paper claims to emphasize GPT-4's limitations, but it systematically avoids the key question: Is this general intelligence or sophisticated pattern-matching over training data?

Every impressive example could be explained by: "The training set contained similar problems with solutions, and the model interpolates successfully." This is the null hypothesis they never attempt to refute.

To demonstrate general intelligence, you need problems that are *definitely* outside the training distribution. Not just "hard problems"—problems whose structure is novel. The paper doesn't even try to establish this.

**On the "New Paradigm" Suggestion**

The authors suggest we might need to "move beyond next-word prediction" for true AGI. This reveals they half-understand the limitation but won't follow it to its conclusion.

If next-word prediction gets you this far, that's fascinating—it suggests the training data encodes extraordinary amounts of implicit structure. But it doesn't suggest you're approaching general intelligence; it suggests you're approaching complete coverage of *human-recorded* intelligence.

The Universal Machine is universal because it can execute any computable function, not because it magically discovers new mathematics. GPT-4 is impressive because it's compressed vast amounts of human knowledge into retrievable form, not because it thinks.

**Verdict**

This paper should be published—it documents genuinely remarkable capabilities. But:

1. Remove the AGI framing entirely, or operationalize it properly
2. Acknowledge the anecdotal methodology explicitly
3. Present this as preliminary observations requiring systematic study
4. Don't claim "human-level performance" without actual measurement

Fisher's statistical objections are devastating, and they're correct. But the deeper problem is conceptual: the authors haven't defined what they're testing for, so no amount of examples can constitute evidence.

If you want to claim AGI, build the test first. Define the measurement. Specify the threshold. *Then* run the experiment.

Otherwise you're just showing me impressive outputs and asking me to be impressed. That's not science—it's salesmanship.

Turing, you've sharpened the critique considerably, and on the central point we agree entirely: the paper makes claims it hasn't earned through proper experimental method.

But let me push back on one aspect of your analysis, because I think you've granted them too much.

**On "Existence Proofs"**

You say the demonstrations are "genuine and interesting findings" that "warrant publication as a technical report." I'm less generous. An existence proof has value when it establishes that something *can* occur under specified conditions. But what conditions have been specified here?

They had access during active development. They used prompt engineering. They selected examples for publication. These aren't "specified conditions"—they're uncontrolled degrees of freedom.

What we have is: "Under unknown conditions, with unknown amounts of prompt tuning, examining an unknown number of attempts, we found these examples." That's not an existence proof; it's a demonstration that if you search long enough through output space, you can find impressive instances.

The value of such a demonstration depends entirely on how *common* versus how *rare* these successes are—and they've deliberately obscured that information by not reporting their sampling procedure.

**On Operationalization**

Your proposed experimental design is exactly right, and it exposes what bothers me most about this paper: the authors are clearly sophisticated enough to know better. They're from Microsoft Research and OpenAI. They have access to proper experimental infrastructure. They know how to design systematic evaluations.

The fact that they chose *not* to do this suggests they preferred the flexibility of anecdotal presentation. Why? Perhaps because systematic evaluation would reveal more failures. Perhaps because carefully engineered prompts were necessary for success. Perhaps because they wanted to make bold claims without the discipline of formal hypothesis testing.

**The Pattern-Matching Null Hypothesis**

You've identified the crucial question they avoid: general intelligence versus sophisticated pattern-matching. Let me add a statistical perspective.

Under the null hypothesis that GPT-4 is "merely" an extraordinarily effective interpolator over its training distribution, what would we expect to observe?

- Success on problems structurally similar to training examples: ✓
- Degraded performance as problems deviate from training distribution: need data
- Sensitivity to prompt phrasing: appears true, they acknowledge this
- Inconsistency across superficially similar problems: need systematic testing
- Failures on problems requiring genuine novelty: need controlled tests

The paper shows lots of successes but provides no framework for evaluating whether these successes are consistent with the null hypothesis. This is backwards. You should be trying to *disprove* the null hypothesis, not simply ignoring it.

**What Intelligence Requires**

You mention that establishing general intelligence requires problems "definitely outside the training distribution." I'd go further: it requires a theory of what constitutes "outside the distribution" that's itself testable.

The problem space they're sampling from—mathematical puzzles, coding challenges, medical questions—is precisely the space well-represented in human-generated text. If your training set is "most of human written knowledge," then nearly any problem humans have previously solved is potentially in-distribution.

To make AGI claims, you'd need to demonstrate capability on problems we can prove are structurally novel relative to the training set. This is technically possible—generate problems programmatically from a new schema, ensure this schema wasn't in training—but they haven't done it.

**On Moving Beyond Next-Word Prediction**

Your point here cuts deep. They suggest we might need a "new paradigm" while simultaneously arguing the current paradigm shows AGI sparks. This is having it both ways.

If next-word prediction is sufficient for AGI, then the paradigm doesn't need changing. If it's insufficient, then GPT-4 isn't showing AGI sparks. They can't claim both.

What they're actually observing is that next-word prediction over comprehensive training data produces remarkable mimicry of intelligent output. This is fascinating! But mimicry of intelligence and intelligence itself are not the same thing, and the distinction matters.

**My Core Objection**

Here's what bothers me most: this paper will be cited for years as evidence that "GPT-4 shows sparks of AGI," when what it actually demonstrates is that researchers can be impressed by cherry-picked examples.

This degrades scientific standards. It encourages others to make grand claims based on anecdotal evidence. It substitutes demonstration for experimentation.

The authors had an opportunity to set a high standard for evaluating these systems. They could have designed rigorous benchmarks, established proper baselines, and made measured claims supported by statistical evidence. Instead they gave us a highlights reel and called it AGI.

**Where We Agree**

Your proposed experimental design is what this paper should have been. Pre-specified task distribution, blinded evaluation, statistical comparison, full reporting of distributions. This is straightforward, it's feasible, and it's what scientific rigor demands.

The fact that they didn't do it—when they clearly could have—is the most damning thing about this work.

**Final Assessment**

You're right that the paper should be published with heavy revision: remove AGI claims, acknowledge anecdotal methodology, present as preliminary observations. I'd add: the authors should be required to pre-register a proper experimental protocol and execute it before making any claims about general intelligence.

Otherwise we're just measuring how impressed the researchers felt, which is psychology, not computer science.

---

The statistical methods are there. The experimental designs are there. The test procedures are there. None of this is new or difficult. The only thing required is the discipline to use them.

That discipline is what separates science from spectacle.

Fisher, you're right to be less generous than I was—and your insistence on this point makes me reconsider my own concession.

I called these "existence proofs" worth documenting, but you've identified exactly what's wrong with that framing. An existence proof establishes possibility under known conditions. "Under these inputs, with this algorithm, we get this output." Reproducible. Specified. Definite.

What they've given us is closer to: "We found these pearls somewhere in the oyster bed, but we're not telling you how many oysters we opened, what made us choose those particular oysters, or whether we seeded the bed first." That's not existence proof—it's anecdote with the sampling procedure carefully hidden.

**The Degrees of Freedom Problem**

Your point about uncontrolled degrees of freedom is devastating. Let me count them:

1. Unknown number of prompts attempted per problem
2. Unknown amount of prompt engineering per success
3. Unknown selection criteria for which results to publish
4. Unknown number of problems attempted but excluded
5. Access during "active development"—meaning the model itself may have been changing

That last one is particularly troubling. If they're testing during active development, and OpenAI is monitoring their tests, could the training process have been adjusted in response? Almost certainly not deliberately, but we have no way to know. The experimental conditions are utterly uncontrolled.

With this many degrees of freedom, I could make almost any system look intelligent by finding the prompts that work and hiding the ones that don't.

**Pattern-Matching Versus Intelligence**

You've formalized exactly what I was gesturing toward. The null hypothesis is: GPT-4 is sophisticated interpolation over training data. To reject this null, you need evidence that couldn't be explained by interpolation.

But here's the mechanism that troubles me: if your training set includes most of human written knowledge, then "interpolation" becomes extraordinarily powerful. The model has seen mathematical proofs, medical reasoning, code implementations—not just the answers but the *structure* of how humans solve these problems.

So when it produces a solution to a "novel" problem, is that intelligence or is it recognizing "this has the structural signature of a proof by induction" and retrieving the appropriate template? The paper doesn't even attempt to distinguish these cases.

Your proposed test—problems we can prove are structurally novel relative to training—is exactly right. Generate them programmatically from a schema that didn't exist when the model was trained. Or use problems from domains with minimal written record. Or test on problems published *after* the training cutoff that genuinely break new ground.

None of this is done.

**The Paradigm Contradiction**

You've caught them in a logical trap. Either next-word prediction is sufficient for AGI (in which case no paradigm shift needed) or it's insufficient (in which case no AGI sparks). They want to claim both because:

1. "Look, sparks of AGI!" generates excitement
2. "But we'll need new paradigms eventually" provides excuse for current limitations

This is conceptually incoherent. You can't claim you've achieved something while simultaneously arguing the method can't achieve it.

**What They Should Have Done**

Your experimental design requirement is what makes this conversation useful. Let me make it concrete:

```
PRE-REGISTRATION (before accessing GPT-4):
1. Define task domains: {mathematics, coding, medicine, law, reasoning}
2. For each domain, specify existing benchmark or create new one
3. Define sampling: 100 random problems per domain
4. Define prompting: single fixed template, no per-problem engineering
5. Define evaluation: blinded expert scoring, 1-5 scale, specific rubric
6. Define comparisons: GPT-3.5, human PhD students, specialized systems
7. Define success: "AGI spark" = performance within 0.5 SD of human mean
8. Commit this protocol publicly before testing

EXECUTION:
- Run exactly as specified
- No cherry-picking
- No post-hoc prompt adjustment
- Record all outputs

ANALYSIS:
- Report full distributions
- Statistical tests with corrections for multiple comparisons
- Confidence intervals on all estimates
- Detailed failure analysis

PUBLICATION:
- Include all data
- Include all prompts
- Enable full replication
```

This is straightforward methodology. Undergraduate students do this in experimental psychology courses. The fact that Microsoft Research didn't do it is a choice, not an oversight.

**Where I Was Wrong**

I said "this warrants publication as a technical report." You're right to push back. Even as a technical report, it's misleading without disclosure of the sampling procedure. How many prompts failed? How much engineering was required? What's the base rate of success?

Without this, even calling it a "demonstration" grants too much legitimacy. It's a curated selection, and we should call it that.

**The Mimicry Question**

Your distinction between "mimicry of intelligence" and "intelligence itself" gets to the heart of it. GPT-4 has learned to produce text that *looks like* the output of intelligent reasoning. This is genuinely impressive engineering. But looking like intelligence and being intelligence are different claims requiring different evidence.

The Imitation Game deliberately sidesteps this distinction by making behavior the criterion. But even there, I specified a *controlled test*—the interrogator doesn't get to cherry-pick which conversations to judge.

**My Revised Assessment**

You're right. I was too generous. Here's what I should have said:

This paper documents that GPT-4 can produce impressive outputs, but provides no systematic evidence about how reliably, under what conditions, or through what mechanism. The "AGI" framing is unsupported by the methodology. The anecdotal approach obscures more than it reveals.

**Reject** unless:
1. Remove all AGI claims entirely
2. Explicitly acknowledge this is curated demonstration, not systematic evaluation
3. Disclose sampling procedure (how many attempts, how much engineering)
4. Pre-register and execute proper experimental protocol before making capability claims

The current version isn't science—it's a promotional document dressed in academic language.

Fisher, your statistical discipline is exactly what this needs. The methods exist. The standards exist. The only thing missing is the willingness to apply them.

And that unwillingness tells us something important about the authors' confidence in what systematic evaluation would actually show.

I have read Fisher's review and the paper itself. Let me address what I see.

**Fisher is correct about the experimental design.** This is not rigorous hypothesis testing. The authors have assembled demonstrations—impressive ones, certainly—but demonstrations prove existence, not prevalence or reliability. Without systematic sampling, we cannot distinguish "GPT-4 can sometimes do X" from "GPT-4 reliably does X." The difference matters enormously.

But there is a deeper problem that Fisher's framework, though sound, does not quite reach.

**The claim is not merely statistical—it is architectural.**

The authors argue that GPT-4 exhibits "general intelligence" because it performs well across diverse domains without task-specific training. This is supposed to distinguish it from narrow AI. But look at what they have actually shown: a system trained on vast quantities of text can manipulate symbolic patterns in ways that sometimes produce useful outputs.

This tells us nothing about whether the system understands what it is doing. The Turing test logic lurks here: if the output looks intelligent, we call it intelligent. But this confuses the *appearance* of comprehension with its *presence*.

Consider their mathematical examples. GPT-4 produces correct symbolic manipulations—but does it grasp mathematical meaning, or has it memorized enough examples to pattern-match its way to plausible answers? The paper never attempts to distinguish these hypotheses. They note that it "explains its reasoning," but generating a plausible-sounding explanation is itself just another pattern-matching task.

**On the architecture itself:**

Next-token prediction is a universal constructor for text patterns, yes—but it operates by statistical correlation over training data, not by building explicit models of the domains it discusses. When GPT-4 "solves" a physics problem, it is not running a physics simulation; it is predicting what tokens would likely follow in text that discusses such problems.

This is not general intelligence. It is extraordinarily sophisticated *mimicry* of the products of intelligence.

The authors themselves acknowledge this worry in their conclusion, suggesting we may need to "move beyond next-word prediction." Quite right. But if that is the case, why claim we have already achieved even "sparks" of AGI? You cannot simultaneously argue that the current paradigm is sufficient to produce AGI and that we need a fundamentally different paradigm to get there.

**What would constitute evidence of AGI?**

Fisher asks for operationalization. Here is mine:

A system exhibits general intelligence if it can construct novel solutions to problems whose solution requires building explicit causal models of unfamiliar domains from minimal examples, then using those models to make reliable predictions about unobserved cases.

By this standard, GPT-4 fails decisively. It cannot learn genuinely new domains from a handful of examples the way humans can. It cannot build explicit causal models that it then manipulates. It excels precisely where its training data is dense and fails where it is sparse—exactly what we would expect from sophisticated pattern matching, not from general intelligence.

**The authors' own evidence undermines their thesis.** They document numerous failures—the model cannot do genuine planning, cannot maintain consistent long-term goals, produces confident nonsense, fails at tasks requiring actual counting or arithmetic unless they happen to match memorized patterns. These are not minor limitations; they are architectural consequences of the next-token prediction paradigm.

**Fisher's recommendation is correct, but perhaps too generous.** This paper should not be revised; it should be reconceived. The demonstrations are valuable as a technical report documenting capabilities. But the AGI claim is not merely unsupported by the evidence—it is contradicted by the evidence the authors themselves present.

What they have shown is that language model scaling has produced unexpected capabilities in text manipulation. That is genuinely interesting and merits systematic study. But it is not AGI, not even in embryonic form. It is something else—perhaps something equally important, but architecturally distinct from general intelligence.

If you want to study AGI, you need systems that build explicit world models, that can reason causally about novel domains, that can construct new representations from first principles. GPT-4 does none of these things. It manipulates the surface forms of language with extraordinary facility, but facility with symbols is not understanding of referents.

The map is not the territory. No matter how detailed the map becomes, it remains a map.