A viral resignation thread, a rare public admission from an Anthropic research lead, and a growing chorus calling for independent AI oversight — here’s what’s being proposed, and why now.
Artificial intelligence is developing at an incredible speed. But as AI systems become more powerful, a growing number of researchers are asking an important question:
Are we developing AI faster than we can understand and control it?
That question has gained new attention after a former AI researcher publicly warned that leading AI companies may be moving too quickly toward increasingly powerful and potentially self-improving AI systems.
The debate has also received attention from people working inside the AI industry, including researchers who focus specifically on AI safety.
What happened?
In early September 2026, former Anthropic and OpenAI researcher Jacob Coxon resigned from Anthropic and published a series of posts on X about the future of AI.
Coxon warned that major AI companies could be moving towards what is sometimes called “superintelligence” — AI systems that could eventually become much more capable than humans in many areas.
His posts attracted tens of millions of views and pushed the discussion about AI safety into the mainstream.
What made the story even more interesting was the response from Evan Hubinger, Anthropic’s alignment science lead.
Hubinger agreed with many of Coxon’s concerns and publicly shared his own view that there could be a significant risk of extremely harmful AI outcomes within the next decade.
However, Hubinger also made an important distinction: he does not consider today’s AI systems to be highly dangerous. His concern is more about a future where AI becomes far more capable, while researchers still do not have a reliable way to control it.
Why is this important?
It is normal for people outside the technology industry to warn about the risks of AI.
But when researchers who work directly on AI safety raise similar concerns, the discussion becomes much more serious.
The central issue is simple:
AI capabilities are improving very quickly, but our ability to understand and control increasingly powerful AI may not be improving at the same speed.
This does not mean that AI is about to become dangerous or that a disaster is inevitable.
Researchers disagree strongly about how quickly AI will develop, whether true “superintelligence” is close, and even how serious some of these long-term risks are.
There are also many AI problems happening today that deserve attention, including misinformation, privacy concerns, cyberattacks, job disruption and the misuse of AI-generated content.
But the disagreement itself raises an important question:
Should companies developing the most powerful AI systems also be the only organisations responsible for checking whether those systems are safe?
AI doesn’t yet have its equivalent of the aviation “black box”
High-risk industries have spent decades creating systems designed to identify problems, investigate accidents and prevent the same mistakes from happening again.
Consider aviation.
When a serious aircraft accident happens, investigators don’t simply ask the airline involved to investigate itself. Independent organisations investigate what happened, identify the causes and recommend changes that can improve safety across the entire industry.
Other industries have similar systems.
Finance uses independent auditors and regulators.
Medicine uses independent review processes to protect patients and ensure research is conducted responsibly.
AI does not yet have an equivalent system with the same level of independence and enforcement.
AI companies have developed their own safety policies, testing procedures and risk assessments. These are important steps, but many remain voluntary.
There is currently no single global organisation that can investigate every serious AI incident, require companies to report certain failures, or stop the release of a powerful AI system if it is considered unsafe.
That is what some experts believe needs to change.
What could stronger AI oversight look like?
Several ideas are being discussed by researchers, governments and AI safety experts.
1. Mandatory reporting of serious AI incidents
AI companies could be required to report major safety incidents to an independent organisation.
These could include:
- AI systems finding ways around safety controls
- Unexpected or dangerous capabilities
- AI behaving deceptively during testing
- Serious AI-related cyber incidents
- AI systems being used to cause real-world harm
This would be similar to the way serious incidents are reported in aviation and other high-risk industries.
2. Independent testing before major AI releases
Today, AI companies generally conduct their own testing before releasing new models.
A possible future system would require independent experts to test particularly powerful AI models before they are made widely available.
This could include “red-team” testing, where experts deliberately try to find weaknesses and dangerous behaviour.
The idea is simple:
The company should not always be the only organisation deciding whether its own product is safe.
3. An international AI safety organisation
There have already been calls for something similar to an “IAEA for AI” — an international organisation focused on AI safety.
International cooperation on AI safety has grown through meetings and initiatives in places such as Bletchley Park, Seoul and Paris.
International AI safety organisations and networks are already producing research and reports, but they generally do not have the same inspection and enforcement powers associated with organisations in other high-risk industries.
Giving an international body stronger powers could become an important part of future AI regulation.
4. Rules based on AI capability
Not every AI system presents the same level of risk.
A simple chatbot used to answer customer questions should not necessarily face the same requirements as an AI system capable of independently carrying out complex cyber operations or controlling critical infrastructure.
Instead of creating one set of rules for every AI system, regulators could establish specific thresholds.
For example, stronger oversight could be triggered when an AI system reaches certain levels of:
- Computing power
- Autonomous decision-making
- Cybersecurity capabilities
- Scientific or research capabilities
- Ability to operate independently for long periods
This would allow regulation to focus on the systems that present the greatest potential risk.
5. Better protection for AI whistleblowers
The recent debate also highlights the importance of people inside AI companies being able to raise concerns.
Employees who discover serious safety problems should be able to report them without automatically putting their careers at risk.
Strong whistleblower protections could allow problems to be identified and investigated before an employee feels that the only option is to resign and speak publicly.
6. More funding for AI safety research
One of the biggest challenges with modern AI is that we don’t always fully understand why an AI model produces a particular answer or behaviour.
AI researchers have made significant progress in making models more capable, but understanding exactly how they reach their decisions remains difficult.
This area of research is often called AI interpretability.
More independent funding for this type of research could help scientists understand how advanced AI systems work and identify potential problems before those systems become significantly more powerful.
This doesn’t mean AI should be stopped
It is important not to misunderstand the argument.
The goal of stronger AI oversight is not necessarily to stop AI development.
AI is already providing enormous benefits in areas such as healthcare, education, software development, scientific research, business automation and cybersecurity.
The challenge is finding the right balance between innovation and safety.
Too much regulation could slow useful innovation.
Too little regulation could allow serious problems to develop before anyone has a chance to respond.
The goal should therefore be to build systems that allow AI to continue developing while making sure there are independent checks when the risks become significant.
The bigger question
The current debate is ultimately about more than one researcher, one AI company or one social media post.
It is about how society should manage a technology that is developing extremely quickly.
We don’t need to believe that an AI disaster is inevitable to believe that stronger safety systems are worthwhile.
The aviation industry did not create accident investigations because aircraft were guaranteed to crash.
It created them because when something goes wrong, society needs an independent way to understand what happened and prevent it from happening again.
AI may need something similar.
As AI becomes more powerful, perhaps the biggest question is no longer simply:
“What can AI do?”
It may increasingly become:
“How do we make sure we remain in control of what AI can do?”
And that is a conversation that governments, technology companies, researchers and the public will need to have together.