Is AI Starting to Build AI?

POLISCI 158 · AI & Politics · Session 16b · Monday, November 16, 2026

Simone Paci

Roadmap

  1. Part I: What is RSI, and what did AI 2027 predict?
    • Define RSI and the R&D progress multiplier
    • Trace the AI 2027 ladder and its branch point
  2. Part II: What do the signals show now?
    • Read the METR, Anthropic and OpenAI evidence and its limits
    • Compare forecasters’ updates with the skeptics
  3. Part III: What should government do in the next 12 months, and after?
    • Map options against what the new Congress can pass
    • Weigh transparency vs pacing vs bans

Is AI starting to build AI?

“We are not there yet, and recursive self-improvement is not inevitable.” Anthropic, “When AI builds itself”

Not there yet. How close, and who gets to decide?

Anthropic, “When AI builds itself,” Favaro and Clark, updated Sep 18, 2026

What are the incentives for Anthropic?

Why would they confirm RSI?

Why would they deny RSI?

Part I: What is RSI, and what did AI 2027 predict?

A loop, a multiplier, and a story with two endings.

What is recursive self-improvement (RSI)?

RSI: an AI system “capable of fully autonomously designing and developing its own successor”

AI does AI research better AI full RSI: no human needed in the loop

Anthropic: “We are not there yet” · today humans still direct the loop

Anthropic, “When AI builds itself,” Favaro and Clark, updated Sep 18, 2026

What is the AI R&D progress multiplier?

“The AI R&D progress multiplier is how much faster AI software improvements are advancing with AI usage than without it.”

1x no AI help

2x twice as fast

10x “about a year of algorithmic progress every month”

The 10x example is from the scenario (Jun 2027), not a measurement

Kokotajlo, Alexander, Larsen, Lifland and Dean, AI 2027 (ai-2027.com), Apr 3, 2025

What did AI 2027 forecast for 2026 and 2027?

Milestone Date Multiplier
Agent-1, “a scatterbrained employee who thrives under careful management” Early 2026 1.5x
China nationalizes its AI research, “about six months behind” Mid 2026 1.7x
Agent-1-mini, “10x cheaper” · “AI has started to take jobs” Late 2026 2x
Superhuman coder (SC) Mar 2027 5x
Superhuman AI researcher (SAR) Aug 2027 25x
Superintelligent AI researcher (SIAR) Nov 2027 250x
Artificial superintelligence (ASI) Dec 2027 2,000x

2026 multipliers from the scenario’s dashboard (Apr, Aug, Dec 2026) · 2027 dates from the racing ending · authors: “AI 2027 is not a recommendation or exhortation. Our goal is predictive accuracy.”

Kokotajlo, Alexander, Larsen, Lifland and Dean, AI 2027 (ai-2027.com), Apr 3, 2025

What happens in the story?

Early 2026 Feb 2027 Mar 2027 Sep 2027 Oct 2027 Agent-1: progress 50% faster China steals Agent-2 superhuman coder Agent-4 is misaligned Oversight Committee branch point

Fictional labs: OpenBrain and DeepCent

A scenario, written in April 2025 · dates are the authors’ forecast

Kokotajlo, Alexander, Larsen, Lifland and Dean, AI 2027 (ai-2027.com), Apr 3, 2025

Who decides at the branch point?

Oversight Committee: “a joint management committee of company and government representatives”

Race Slowdown
“The committee votes 6–4 to continue internal use of Agent-4.” Committee “votes 6–4 to slow down and reassess.”
“The problem was real and the fixes didn’t work.” “This is enough evidence to finally shut down Agent-4.”
Humans lose control to misaligned AI A small committee controls superintelligent AI

Should a committee of executives and officials make this call?

“The slowdown ending is not a recommendation.”

Kokotajlo, Alexander, Larsen, Lifland and Dean, AI 2027 (ai-2027.com), Oct–Nov 2027 sections

Part II: What do the signals show now?

Three labs, three kinds of evidence, and who gets to check it.

What did METR find inside Anthropic?

METR chart: AECI, an index of Anthropic model capability aggregating internal and external benchmark scores, plotted against the date each model snapshot was available to most Anthropic employees, early 2024 to October 2026. Public models in green, internal models in orange. A grey trend line of +15 per year runs from Claude 3 Opus to Sonnet 4.5; after October 2025 a black trend line of +29 per year rises more steeply through Opus 4.5, Opus 4.6, Mythos 5 and internal Models 2 to 4, while a dashed grey line continues the old trend below it. Title: as of October 2026, benchmark scores appear to be growing at 1.9 times their pre-October 2025 rate, 95% CI 1.5 to 2.8

METR, Nov 5, 2026

11 weeks of high-access evaluation inside Anthropic, Aug 12–Oct 29, 2026

Capability index (AECI): +15 pts/yr → +29 pts/yr after the Oct 2025 break

“accelerated 1.9x [95% confidence interval: 1.5–2.8x] since October 2025”

“on well-specified tasks” only · index mixes internal and external benchmarks · sensitive to assumptions about benchmark difficulty

METR, Cunningham, Rush, Whitfill & Wijk, Nov 5, 2026

How much of that is AI?

Share due to AI: “perhaps ≈80%” (METR’s guess, Oct 2026)

Range: “we would not be surprised if the true acceleration was closer to 1.3x or 3x the historical rate.”

“Anthropic redacted a majority of the evidence we found most informative for these conclusions.”

“the public has to rely on METR’s judgment much more than would be ideal”

Also: “Anthropic canceled a compute allocation component of this assessment.”

METR, Cunningham, Rush, Whitfill & Wijk, Nov 5, 2026

So where are we now?

AI Futures Project cartoon: a white car labelled Before Oct 2025, 1x engine, 1x speed, on a shallow slope; higher up a steeper hill, a red car with a large engine labelled about 5x engine, AI software R&D uplift, with an arrow to 1.7x speed, pace of progress; a box reads Steepness = how hard progress is, ideas get harder to find

AI Futures Project, Nov 10, 2026

AI 2027 forecast: “2x AI R&D uplift in Dec 2026” = the engine

METR, as AI Futures reads it: “1.7x AI-attributable acceleration in the pace of progress” = the speed

  • “To make the car go twice as fast, the engine has to work more than twice as fast”
  • In an AI 2027-like run, 1.7x speed arrives in about April 2027 → “we are ~5 months ahead of schedule”

“by this metric” · AI 2027 trajectory reconstructed by the authors · rests on METR’s estimate (1.3x–3x plausible, evidence partly redacted)

AI Futures Project, “Q3.5 Timelines Update,” Lifland, Halstead, Kokotajlo, Dean et al., Nov 10, 2026

How much R&D does Claude lead?

Anthropic stacked step chart, Claude now leads 26% of model R&D work: monthly share of model R&D tasks at Anthropic, August 2025 to August 2026, rated on Epoch AI's automation scale from AL0 no AI, AL1 minimal AI, AL2 AI assists, AL3 AI collaborates, AL4 AI leads, to AL5 fully automated. The AI-collaborates band grows from a few percent to most of the work; the AI-leads band appears in March 2026 and rises 1%, 3%, 12%, 14%, 22%, 26% by August, with 90% measurement intervals; no fully automated share

Scale: AL0 no AI → AL3 collaborates → AL4 leads → AL5 fully automated (not reached)

Claude leads 26% of R&D work (Aug 2026), up from under 1% (Feb 2026)

“Claude is not operating fully autonomously for any measured subset of AI R&D work.”

Anthropic’s own measurement of its own work · not independently verified

Anthropic, “Measurements for understanding the pace of AI development inside frontier labs,” Favaro & Wright, data as of Aug 2026

How much code does Claude write?

Anthropic bar chart, code contributed per person by quarter, Q2 2021 to Q2 2026: lines of code merged per active contributor as a multiple of the pre-2025 average. Bars hover near 1x through 2024, then rise to 1.2x in Q1 2025, 1.5x, 1.9x, 2.5x in Q4 2025, 5.8x in Q1 2026 and 8.0x in Q2 2026, the last bar hatched as a partial quarter; dashed lines mark Claude model and Claude Code releases

“more than 80% of the code we merge into Anthropic’s codebase was authored by Claude” (May 2026)

“approximately 30,000 agents” at work at any one time

“about 6% of compute that went to AI R&D was allocated toward safety”

Lines of code ≠ research progress · Q2 2026 (8.0x) is a partial quarter · company self-report

Anthropic, “When AI builds itself,” Favaro & Clark, updated Sep 18, 2026; Anthropic, “Measurements for understanding the pace of AI development inside frontier labs,” Favaro & Wright, Aug 2026

What about OpenAI?

Line chart on a log scale: daily coding-agent spend per OpenAI researcher in US dollars at API list prices, by week, January to mid-August 2026. The median researcher rises from about 17 cents to 601 dollars; the 90th-percentile researcher rises from about 2 dollars to 7,047 dollars

Redrawn from Epoch AI data

Median researcher: under $1/day (Jan 2026) → $601 (mid-Aug)

90th percentile: over $7,000/day

“doubling time of roughly one month for both groups”

“self-reported by OpenAI and cannot be independently verified” · API list prices, not internal cost · spending ≠ speed-up · “probably unsustainable”

Epoch AI, Edelman & Emberson, Oct 5, 2026, from OpenAI, “Research acceleration: The view inside OpenAI,” Sep 2026

What don’t the signals show?

  • One-off jump, not a new trend? Toby Ord’s reading → ~1.4x
  • Staff self-reports of ≈12x uplift: “only very loose evidence” (METR)
  • Benchmarks saturating: Model 4 scores >50% on all of them
  • Anthropic’s Aug risk report: “<2x” · METR’s rerun: 1.97
  • All frontier developers’ trend: 1.4x

OpenAI and Google “could be speeding up just as fast, and we wouldn’t know” 80,000 Hours

METR, Nov 5, 2026; 80,000 Hours, Luisa Rodriguez, Nov 12, 2026 (Ord, risk-report comparison)

How did the forecasters update?

AI Futures Project line chart: AIFP's timelines to Top-Expert-Dominating AI since AI 2027, April 2025 to November 2026; each author's median forecast year; Eli, Thomas and Daniel drift later through late 2025 (Eli near 2035.5), then all five authors, including Romeo and Brendan, fall sharply to between Sep 2027 (Daniel) and Sep 2028 (Romeo) by November 2026

AI Futures Project, Nov 10, 2026 · each author’s median year for top-expert-dominating AI

AI Futures: superhuman AI researcher “likely” in 2027

  • Before: “Daniel, the most bullish of us, assigned a 50% chance of Superhuman AI Researcher (SAR) before 2029”

Dwarkesh Patel: median 2028 for AI matching top human AI researchers

  • “25% chance that it doesn’t happen until after 2032”

AI Futures = the AI 2027 authors · forecasts assume no deliberate slowdown · top-expert-dominating AI is a later milestone than a superhuman AI researcher

AI Futures Project, “Q3.5 Timelines Update,” Nov 10, 2026; Dwarkesh Patel, “The avalanche that hit math is coming for AI development,” Nov 13, 2026

Spark or explosion?

time AI progress explosion: software-only singularity spark, then stalls on compute Patel: 50/50

Spark: “three to six years of AI progress in a single year”

  • Bottleneck: “compute for experiments, which aren’t running any faster”
  • Ideas get harder to find: labs “quadruple their R&D compute” / “double their workforce every single year just to maintain the same rate of progress”
  • Data: “even a superintelligent learner can’t figure out everything from inside a datacenter.”

Dwarkesh Patel, “The avalanche that hit math is coming for AI development,” Nov 13, 2026

What do the skeptics say?

Outside the safety community, Narayanan & Kapoor reject “fast takeoff” scenarios

“It is more likely that we will continue to see a gradual increase in the role of automation in AI development”

“‘superintelligent’ AI, which we view as incoherent as usually conceptualized”

“AI diffusion lags decades behind innovation”

Written Apr 2025, before METR’s Nov 2026 assessment

Narayanan & Kapoor, “AI as Normal Technology,” Knight First Amendment Institute, Apr 2025

Part III: What should government do in the next 12 months, and after?

If the labs won’t slow down on their own, who decides?

Why won’t labs stop on their own?

“The AI companies aren’t stopping because they think that if they do, their competitors will just keep going.” Luisa Rodriguez, 80,000 Hours, Nov 12, 2026

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development” … “if other developers at or near the frontier also did so in a verifiable manner” Anthropic, “When AI builds itself,” Sep 2026

“Training runs are far easier to conceal than missile silos” Anthropic, same essay

A race, a conditional offer, and a verification problem

80,000 Hours, Rodriguez, Nov 12, 2026; Anthropic, Favaro & Clark, “When AI builds itself,” update Sep 18, 2026

What can the new Congress pass?

60 Senate votes to break a filibuster

67 Senate votes to override a veto

290 House votes to override a veto

Four routes:

  • Bipartisan bills: weights security, export controls, whistleblowers, incident reporting
  • Must-pass bills, e.g. the NDAA
  • Confirmations
  • Hearings under oath

One commentator’s plan, not a forecast of what Congress will do

Rob Wiblin, “How the new Congress saves us from AI disaster,” 80,000 Hours, Nov 13, 2026

What’s on the menu?

Transparency

Report AI speed-up, checked independently · Anthropic: publish measures; developer bears burden of proof

Security

Secure model weights

Whistleblowers & incidents

S.1792 (Grassley) · H.R.9477 (Moran)

Pacing

Minimum share of compute serving outside customers (start at 100%, ease to 70%)

RSI throttle

Use only models that finished training ≥ six months ago · CAISSI exemptions

Ban

Ban Artificial Superintelligence Act: S.5493 (Sanders) / H.R.10538 (Casar) · Bores: ban fully automated AI R&D

Bills named as proposals, not law · cards run from least to most restrictive

80,000 Hours, Rodriguez, Nov 12, 2026, and Wiblin, Nov 13, 2026; Anthropic, Favaro & Wright, “Measurements for understanding the pace of AI development inside frontier labs,” 2026

What does each option cost?

Option Objection / risk Can it be verified?
Transparency Labs control the evidence: “Anthropic redacted a majority of the evidence we found most informative” (METR) Only with an independent checker · now “the public has to rely on METR’s judgment much more than would be ideal”
Pacing / RSI throttle Wiblin, of his own plan: “This approach does have some weaknesses” Needs inside access: CAISSI access to tests, embedded auditors
Ban “nonproliferation-based safety measures decrease resilience and thus worsen AI risks in the long run” (Narayanan & Kapoor) “Training runs are far easier to conceal than missile silos” (Anthropic)

The harder the rule, the harder to check

METR, Nov 5, 2026; Wiblin, 80,000 Hours, Nov 13, 2026; Narayanan & Kapoor, “AI as Normal Technology,” Knight First Amendment Institute, Apr 2025; Anthropic, Sep 2026

Who governs, federal or states?

States have moved first:

  • California: SB 53
  • New York: RAISE Act

Federal preemption would override state laws like these

On preemption: “anything like that should be kept out of an emergency bill after a warning shot of some type.” Rob Wiblin, 80,000 Hours, Nov 13, 2026

Who should set the rules for the labs: Sacramento, Albany, or Washington?

Wiblin, “How the new Congress saves us from AI disaster,” 80,000 Hours, Nov 13, 2026

What comes after the next 12 months?

Pause or pace

“If SAR is coming in 2027, it is incredibly urgent to pause or substantially ‘pace’ frontier AI development” AI Futures Project, Nov 10, 2026

A pause only “if other developers at or near the frontier also did so in a verifiable manner” Anthropic, Sep 2026

Build resilience

“It is more likely that we will continue to see a gradual increase in the role of automation in AI development” Narayanan & Kapoor, Apr 2025

“If we pull it off, these might just be the two years during which an AI catastrophe is averted.” · Wiblin

AI Futures Project, Q3.5 update, Nov 10, 2026; Anthropic, Sep 2026; Narayanan & Kapoor, Knight First Amendment Institute, Apr 2025; Wiblin, 80,000 Hours, Nov 13, 2026

Should Congress throttle RSI?

  1. Alone · 2 min: pick the one option you’d attach to the next must-pass bill
  2. Pairs · 5 min: its strongest objection + how you’d verify compliance
  3. Whole class · 10 min: three prompts

Who decides?

Labs, a committee, Congress, voters?

What evidence?

What would change your mind?

What if China doesn’t follow?

Does a US-only rule still help?

Menu: transparency · security · whistleblowers & incidents · pacing · RSI throttle · ban

What should you take from today?

  1. RSI = AI designing its own successor · AI 2027 put the branch point with a small company–government committee
  2. Signals: METR: ~2x faster progress at Anthropic, most of it likely due to AI · key evidence redacted, lab metrics self-reported · skeptics see normal technology
  3. Policy options run from transparency to bans · the harder the rule, the harder to verify · Congress needs 60, or 67/290

See you next class

Peer Review 2 · Sun, Nov 22, 11:59 p.m. · 150–200 words

Wednesday, Nov 18: AI Agents: Reliability, Safety, and Governance · Yiqing Xu

Final presentations · Mon, Nov 30 / Wed, Dec 2 · 5 minutes, 3 slides

Peer Review 3 · in session, Nov 30 / Dec 2