Updates are ongoing. Stay tuned!

Anthropic IPO Protest: Why Jacob Coxon Wants a Ban on Recursive Self-Improvement

The Anthropic IPO protest outside the company’s San Francisco headquarters was small, but the argument it made is not. On the day the public could first buy shares in Anthropic, in an offering that valued the company at $1.7 trillion, former Anthropic researcher Jacob Coxon told Business Insider that investment decisions are beside the point: “We have to stop RSI.” That abbreviation, short for recursive self-improvement, is the real subject of the demonstration, and it is what makes this event more than another tech-sector protest.

This analysis separates what was reported from what it implies. It looks at what the protesters want, what the available evidence supports, and where the case for a ban is stronger or weaker than the rhetoric suggests.

What happened on IPO day outside Anthropic’s headquarters

According to Business Insider, Coxon, whose departure from Anthropic in September pushed AI safety concerns into mainstream conversation, joined dozens of people for a “Ban RSI” protest. Speakers included ex-OpenAI researcher Daniel Kokotajlo and YouTuber Rob Miles. Protesters chanted and carried signs. Later, a group from the organization Stop AI arrived and blocked the office entrance.

The two actions differed in kind. The Ban RSI event was a policy demand with a specific target. The Stop AI blockade was direct action against the company’s operations. Conflating them would be a mistake, because they imply different theories of change: one seeks legislation and industry coordination, the other seeks to disrupt a firm.

Anthropic did not respond to Business Insider’s request for comment, so the company’s position on the protest itself is unknown.

What is recursive self-improvement in AI?

The definition

Recursive self-improvement describes a system that improves its own capabilities, or helps build a more capable successor, which then repeats the process. In the article’s framing, RSI means using advanced AI systems to train the next generations of AI. The idea has a long intellectual lineage in discussions of the technological singularity, where each improvement cycle could shorten the next.

Why it differs from ordinary AI-assisted development

Software engineers already use tools to build better tools, so the line between routine automation and RSI is where the debate gets difficult. The distinction that matters is human control. A researcher using a model to write test code is not the same as a pipeline in which models design, train and evaluate their successors with no human review. Anthropic’s own IPO prospectus, as reported, says it does not expect that letting AI fully design its next generation without supervision would be safe.

That sentence is notable. A company is telling prospective investors that a capability it is positioned to develop would be unsafe in its fully unsupervised form. The open question is where on the spectrum between assisted and autonomous development the company will draw its limits.

The evidence: what METR and the prospectus actually show

Two pieces of evidence anchor the debate. First, an independent report from the AI safety organization METR recently found that AI use is already accelerating Anthropic’s research and development. Second, the prospectus acknowledges the supervision risk and says the company is considering policy updates “to keep humans in control of frontier model development.”

Read together, they support a narrow but important conclusion: the feedback loop is not hypothetical, and the company knows it. They do not show that a dangerous, runaway loop is imminent. Acceleration of research is not the same as unsupervised self-improvement, and the reporting does not quantify the speedup. Anyone claiming certainty in either direction is going beyond the public record.

The phrase “considering policy updates” also deserves scrutiny. It is a statement of intent, not a commitment with defined thresholds, audit mechanisms or enforcement. Coxon’s complaint that he would like to see more is a reasonable reading of that gap.

Reading the protest critically

The strongest part of the argument

The demand is specific. A ban on RSI is a bounded target compared with the broad goal of “stopping AI.” Coxon asked Anthropic to lobby the government and coordinate with other companies for a full ban, and he noted that only a few places in the world can pursue RSI in 2027. If that claim is even approximately right, regulatory leverage is concentrated in a handful of firms and jurisdictions, which makes enforcement more tractable than it would be for a widely distributed technology.

The testimony about private alarm also matters. Coxon and Kokotajlo both said AI researchers express worry about the technology’s risks in private. That is anecdotal and cannot be verified from the article, but it is consistent with the broader public debate around existential risk from artificial intelligence.

The weakest part of the argument

Three weaknesses stand out.

  • Definitional vagueness. “Ban RSI” is easy to chant and hard to legislate. Without a precise definition of what counts as self-improvement and what counts as acceptable automation, a ban could be either toothless or overbroad.
  • The coordination problem. Asking competitors to coordinate on restrictions is the standard answer to a race dynamic, but coordination among rivals can raise competition-law questions. That is my analysis, not a claim from the protesters, and it is the kind of detail a workable proposal would need to address.
  • Rhetorical overreach. “At this point, the companies are the problem” and the remark that people at Anthropic have “drunk the Kool-Aid” may energize supporters, but they are unlikely to persuade the employees who would implement any safeguards. David Krueger himself acknowledged that protests only somewhat reach people inside the company.

How going public changes the incentives

An initial public offering changes a company’s obligations and pressures at once. Oliver Habryka, who runs the company behind LessWrong, told Business Insider he is glad Anthropic is now legally required to disclose certain information. Public-company reporting, overseen in the United States by the Securities and Exchange Commission, can surface material risks that private firms might never publish. The company’s prospectus language on RSI is already an example.

The counterweight is competitive pressure. Habryka’s worry is that a company that pitched its industry lead to investors as a competitive advantage will feel compelled to use RSI to protect that lead. This is a plausible structural concern rather than evidence of intent. Public markets reward growth and speed, and safety commitments that cost speed are harder to defend quarter after quarter. Whether disclosure duties outweigh that pressure is an empirical question that will only be answered by the company’s behavior.

Congress, regulation and the policy path

The article notes that concerns have reached Congress, where representatives have begun suggesting restrictions on RSI. No specific legislation is described, so it would be premature to treat a ban as likely. Still, the shift matters: a narrow technical practice is entering the vocabulary of lawmakers, which is often the first step toward formal regulation of artificial intelligence.

Any serious proposal would likely need to settle several questions:

  1. How to define self-improvement in a way that separates supervised assistance from unsupervised autonomy.
  2. Which capability or compute thresholds trigger obligations.
  3. How independent evaluators, such as METR, get access to verify compliance.
  4. What disclosure is required when AI contributes materially to model development.
  5. How to handle jurisdictions that decline to follow.

The last point is the hardest. A ban that binds only some developers could shift activity elsewhere rather than eliminate the risk.

Why biology keeps appearing in the safety debate

Habryka singled out AI capabilities in biology as a particular worry. He said AI has grown rapidly more capable over the last few years and that with RSI, “things can go much faster than that.” The logic is straightforward: if capability growth accelerates, the time available to evaluate dual-use risks shrinks. The article does not detail specific biological capabilities, so the concern should be read as a warning about timelines rather than a documented finding.

What to watch after the IPO

For investors, employees and policymakers, a few concrete signals will say more than slogans.

  • Specificity of commitments. Does Anthropic convert “considering policy updates” into defined limits, with thresholds for human review?
  • Disclosure quality. Do future filings quantify how much AI contributes to its own development, and describe supervision practices?
  • Independent verification. Does the company continue to allow outside groups such as METR to assess acceleration effects?
  • Legislative text. Do congressional suggestions become bills with workable definitions?
  • Industry alignment. Do other frontier developers, including OpenAI, make comparable statements, or does Anthropic’s position stay isolated?

Readers who want primary material should consult the company’s filings through the SEC’s official website and Anthropic’s own public materials rather than relying on protest rhetoric or press summaries alone.

Frequently asked questions

What is recursive self-improvement in AI?

It is a process in which advanced AI systems help train or design the next generation of AI, potentially creating a loop of accelerating progress. The risk debate centers on versions where humans no longer meaningfully supervise the loop.

Who is Jacob Coxon?

He is a former Anthropic researcher whose September departure, driven by concerns about AI development, drew wide public attention to AI safety. He spoke at the Ban RSI protest on IPO day.

Does Anthropic oppose a ban on RSI?

That is not established. Anthropic did not respond to Business Insider. Its prospectus says unsupervised AI design of the next generation would not be expected to be safe and that it is considering policy updates to keep humans in control.

Is recursive self-improvement already happening?

Per METR’s report cited by Business Insider, AI use is already accelerating Anthropic’s research and development. That is not the same as fully autonomous self-improvement, which the reporting does not claim.

The question the IPO cannot answer

The most important insight from IPO day is that the argument has moved from whether AI is risky to who controls the pace of its development. Anthropic has put in writing that fully unsupervised AI-designed successors would not be safe. Its critics now want that sentence converted into a binding rule, and they want the company to help write it.

The test is whether a firm valued at $1.7 trillion and under quarterly scrutiny will accept limits that its competitors might not. Watch the next set of disclosures, any congressional text that defines RSI with precision, and whether a rival lab volunteers the same restraint. If none of those appear, the prospectus language will look less like a safeguard and more like a risk factor that was disclosed and then priced in.

Frequently Asked Questions

¿Qué diferencia hay entre la automatización normal en el desarrollo de IA y la automejora recursiva?

La clave es el control humano. Usar un modelo para escribir código de prueba es automatización rutinaria. La automejora recursiva implicaría un proceso en el que los modelos diseñan, entrenan y evalúan a sus sucesores sin revisión humana, de modo que cada ciclo podría acortar el siguiente.

¿Demuestra el informe de METR que la automejora recursiva es inminente o peligrosa?

No. METR halló que la IA ya acelera la I+D de Anthropic, lo que indica que el bucle de retroalimentación no es hipotético. Pero acelerar la investigación no equivale a automejora sin supervisión, y el reportaje no cuantifica esa aceleración. Afirmar certeza en cualquier dirección va más allá de lo público.

¿Qué dice el prospecto de Anthropic sobre este riesgo?

Según lo publicado, el prospecto reconoce que no espera que dejar a la IA diseñar por completo su siguiente generación sin supervisión sea seguro. También dice que la empresa estudia actualizar sus políticas para mantener a los humanos al mando del desarrollo de modelos de frontera.

¿Es suficiente que Anthropic diga que está "considerando actualizar sus políticas"?

Es una declaración de intención, no un compromiso verificable. No incluye umbrales definidos, mecanismos de auditoría ni sanciones. Por eso Coxon pide más: sin criterios concretos y control externo, es difícil saber dónde pondrá la empresa el límite entre desarrollo asistido y autónomo.

¿Son lo mismo la protesta "Ban RSI" y el bloqueo de Stop AI?

No. La protesta Ban RSI planteaba una exigencia política concreta, que busca legislación y coordinación del sector. El bloqueo de Stop AI fue una acción directa para interrumpir la actividad de la empresa. Responden a teorías del cambio distintas, así que confundirlas distorsiona el análisis.

¿Cuál es la postura de Anthropic sobre la protesta?

Se desconoce. Anthropic no respondió a la solicitud de comentarios de Business Insider, de modo que no hay una reacción oficial de la compañía a la manifestación ni a las demandas de Coxon. Lo único citado es lo que figura en su prospecto.

0