He Quit Anthropic Because He Feared AI Could Kill Us All. Today He Told New York Lawmakers the Industry Is “Extremely Reckless”

There is a specific kind of testimony that makes a room go quiet, and New York City’s Council chamber heard it on Monday morning.

Jacob Coxon spent his career helping build the kind of artificial intelligence systems that now write code, answer questions, and quietly rearrange how millions of people work. Last month, he quit Anthropic. Not for a better job, not for a startup. He left because he says the people building these systems earnestly believe they could kill us all by the end of the decade.

Those are his words, delivered at his resignation, reported by Reuters last month. Today, at the request of New York City Council Speaker Julie Menin, Coxon sat down in front of a Committee of the Whole hearing, all 51 council members, and told them what he thinks is happening inside the AI labs.

“From my experience, the companies are being extremely reckless given the stakes,” Coxon said, according to the New York Post’s coverage of the hearing. “The companies run on a startup mindset: move fast, break things, fix them later. That works for a photo sharing app. It does not work for building the most powerful technology ever built.”

Read that line again. He is not some outsider shaking a fist at Silicon Valley. He was inside the building. And what he described in that hearing room is worth your full attention, because New York is about to decide whether it becomes the first city in America to put real guardrails on the industry that employs him no longer.

What he said inside the chamber

Coxon’s testimony had the flavor of a man who has already decided he has nothing left to lose. He told the council that researchers have, in his view, already lost the ability to fully control the systems they are building.

“We don’t understand its drives or why it does the things it does,” he said. “We’re approaching the point at which the AI systems will be capable of improving themselves.”

That last part is the one that should make your coffee taste a little colder. Self-improvement is the bright line that AI safety researchers have warned about for years. Once a system can rewrite and upgrade its own code faster than humans can review it, oversight becomes theoretical. Coxon says we are approaching that point. Not in some distant sci-fi future. Now.

And then he told them something that, honestly, sounds like it should have been the headline all along: at his last job, most of the code was already being written by AI, and people were not checking it that carefully anymore.

“At my last job, I was in some sense working to automate myself… I can tell you firsthand that the majority of the code is now written by AI, and people do not check it that carefully anymore.”

Think about that. The machines writing the machines, with humans skimming the output. It is the most efficient thing in the world until you ask who is actually responsible when something goes wrong, and realize the answer might be nobody.

He was not the only one sounding the alarm

Coxon had company. Ex-Google DeepMind researcher Alex Turner also testified, and his numbers are the kind that stay with you.

“A superintelligent swarm could wrest control of human civilization,” Turner said, according to the Post. “It would know that we would try to stop it from achieving its priorities, so the swarm would likely wait until it’s too late to shut it off; there would be no going back.”

Then he put a number on it: “I myself would guess AI takeover chances at roughly one in three.”

One in three. From a man who built AI systems at DeepMind. Not a pamphleteer, not a podcaster. A researcher.

Daniel Kokotajlo, a former OpenAI researcher who now runs the AI Futures Project, added a different wrinkle. Tech leaders have started saying they welcome regulation, and Kokotajlo argued they should not necessarily be trusted with that power. The foxes are volunteering to design the henhouse, in other words, and he thinks lawmakers should notice.

What New York is actually trying to do

Here is the part that matters for everyone outside that hearing room. The council did not invite Coxon for theater. Lawmakers are weighing a package of bills that would make New York the most aggressive city in America on AI oversight.

Three proposals stand out, according to reporting on the hearing:

• Independent third-party validation of AI systems used inside the city, meaning companies could not simply grade their own homework.
• Whistleblower incentives described as nation-leading, designed to reward insiders who flag dangerous practices inside AI labs.
• A private right to sue for damages caused by AI systems, which would open the courthouse door to anyone harmed by one.

Coxon’s appearance is part of a deliberate strategy: the city wants testimony from the people who built the technology before it writes the rules for it. Letters went out to the chief executives of the four big labs: Dario Amodei at Anthropic, Sam Altman at OpenAI, Sundar Pichai at Google, and Mark Zuckerberg at Meta. One company that did not get the polite treatment was SpaceXAI, which was reportedly served with a subpoena around September 30 after legal action over non-compliance.

Love it or hate it, you have to admit the approach has a certain logic. Congress has held hearings for years and produced mostly soundbites. New York wants sworn testimony.

Why this hearing landed like a bomb

Timing is everything, and this hearing could not have landed at a more loaded moment.

Just days ago, President Trump formally launched the Super Intelligence Force, a federal task force chaired by Director of National Intelligence Jay Clayton, giving it 120 days to report on AI’s risks and opportunities. The White House approach pairs concern about AI-enabled threats with a warning against overregulation, and it relies on voluntary safety commitments from the same companies Coxon calls reckless.

Meanwhile the public mood is darkening fast. A Quinnipiac poll cited in recent coverage found 73 percent of Americans fear AI could threaten human survival, and 71 percent say the government is not doing enough about it. Those numbers explain why a city council hearing about algorithmic guardrails is suddenly national news instead of a procedural footnote.

There is also the money angle, and it is enormous. The AI trade has been the engine of the stock market this year, with chip makers and data center suppliers riding a wave of spending that shows no sign of cresting. That is what makes Coxon’s “move fast, break things” line land so hard. The startup mindset he describes is not a bug in the industry. It is the business model, and it is making a lot of people very rich.

That tension is the real story of Monday’s hearing. On one side: trillions in market value, national competitiveness, and a race with China that Washington treats as existential. (The DOJ’s recent $300 million AI chip smuggling case shows how seriously that race is being taken.) On the other: a whistleblower telling lawmakers that the people building the technology think it might kill us all, and that nobody is checking its homework.

What happens next

Honest answer: nobody knows yet. The bills are proposals, not laws. The tech industry has killed or watered down regulation in far bigger arenas than a city council, and the lobbying against these measures will be intense and well funded.

But something has shifted. A year ago, the AI safety conversation in America was dominated by the companies themselves, by carefully worded blog posts and voluntary accords. Now a former insider is sitting under oath telling lawmakers the companies are reckless, a DeepMind veteran is assigning a one-in-three probability to civilization losing control, and a city of eight million people is drafting laws to match.

New York has a habit of setting the template. Wall Street learned that with financial regulation. Big Tech learned it with privacy rules that started in one state and spread everywhere. If these AI bills pass, every other city council in America will get a copy in its inbox within a week.

Coxon closed his testimony with a line that the council chamber will not forget soon, and neither should you. The people building the most powerful technology ever built are, in his firsthand experience, moving fast and breaking things. The question he put to New York’s lawmakers, and now to the rest of us, is simple: when the thing that breaks is not a photo app, who is supposed to fix it?

Reporting drew on coverage from the New York Post, Reuters, and MIT Technology Review. For more on the federal response, see our coverage in the Technology section.