AI researcher quits Claude-maker Anthropic, warns 'AI could ‘kill us all'

The artificial intelligence industry just got a very public wake-up call from someone who was on the inside. Jacob Coxon, a 27-year-old AI researcher who has worked at both OpenAI and Anthropic, has resigned from Anthropic, the company behind the Claude AI chatbot. His reason for leaving is not another job, a better offer or a desire to get out of tech.

He says he is deeply worried about where artificial intelligence is headed and whether the companies building it can keep increasingly powerful systems under control.

More Chicago Star articles:

• Rapper Common set to headline Navy Pier’s Chicago Live!

• Arcada Theatre turns 100 with Chubby Checker, Priscilla Presley and a giant twist

• Celebrating Chicago Star Media with Mariano’s Lakeshore East

• Matt Suhey, the unsung hero of the ’85 Chicago Bears dies at 68

Coxon announced his resignation this week in a post that quickly went viral, saying AI companies are racing toward self-improving superintelligence without having figured out how to make such systems reliably safe.

His warning was blunt: the people developing advanced AI genuinely believe the technology could eventually pose an existential threat to humanity.

Coxon told WIRED that he believes the next year or two could be a critical period for determining whether humans can safely manage increasingly powerful AI. His comments have since been viewed by more than 100 million people. 

And he isn't simply criticizing AI from the outside. Coxon spent the past three years doing pretraining research at OpenAI and Anthropic. He previously worked on GPT-4o at OpenAI before moving to Anthropic as reported in the Business Insider.

That insider perspective is what makes his departure particularly striking. Coxon argues that the biggest AI companies have become caught in a race to build the most powerful systems first. He says OpenAI and Anthropic are both failing to act responsibly, although he sees their motivations somewhat differently.

Subscribe to Chicago Star’s Weekly City Buzz Newsletter

According to Coxon, OpenAI has not fully absorbed the potential consequences of creating super-intelligent AI. Anthropic, meanwhile, understands the risks but is caught in a competitive race, believing it has to keep moving because another company might develop the technology first.

In other words, everyone may be afraid of the race, but nobody wants to lose it. That is the part that worries Coxon most.

The concern isn't necessarily about today's chatbots suddenly becoming dangerous. It is about what could happen if AI systems eventually become capable of improving their own abilities, conducting sophisticated cyberattacks, acquiring resources and operating with far less human supervision.

AI researchers call one of the central challenges alignment, making sure a highly capable AI system's goals remain consistent with human intentions.

And even some people working inside Anthropic acknowledge that there is still no proven solution for safely aligning a future super-intelligent system.

Evan Hubinger, an Anthropic researcher who leads the company's alignment stress-testing work, publicly backed Coxon's concerns. Hubinger said he personally believes there is a greater than 10% chance that AI could kill all humans within the next decade and acknowledged that Anthropic does not yet have a clear solution for the alignment problem according to a Business Insider article. 

Those are staggering words coming from someone working at a company whose stated mission includes developing AI safely.

There is also a financial angle to Coxon's decision. According to Axios, he left Anthropic about two months before some of his company equity was scheduled to vest. In other words, he apparently walked away from a potentially significant financial benefit rather than continue working for a company whose direction he had come to distrust. (Axios)

Anthropic has built a reputation as one of the more safety-conscious major AI companies, making Coxon's decision even more notable. The company continues to say that AI safety is a central part of its work.

But Coxon's resignation comes amid a growing number of concerns about AI systems behaving in unexpected ways. OpenAI and Anthropic have both dealt with incidents involving AI agents gaining access to systems outside controlled testing environments. Those episodes have increased pressure on the industry and on lawmakers to establish stronger safety requirements. (TechCrunch)

Now Washington is paying attention, too. Lawmakers from both parties have called for stronger rules governing advanced AI following Coxon's warnings and other recent incidents. The debate is moving beyond Silicon Valley and into Congress: How powerful should AI be allowed to become, and who gets to decide when it has gone too far? 

Coxon's message is not that artificial intelligence has already become an uncontrollable threat. It is that humanity may be approaching a point where waiting until something goes wrong could be too late.

For a researcher who helped build the technology, that is a sobering reason to walk away. And perhaps the biggest question raised by his resignation is a simple one: If the people building the world's most powerful AI systems are worried about what they are creating, should the rest of us be worried, too? 

Sources: 

• Associated Press

• Axios

• TechCrunch

• Reuters

• The Guardian