One of the more striking developments in the artificial intelligence sector is that some of the strongest warnings about the technology are now coming from the companies building it.
Jack Clark, co-founder of Anthropic, has called for the creation of a regulatory “brake pedal” for AI development, arguing that governments and society need mechanisms capable of slowing progress if increasingly powerful systems begin to outpace existing safeguards.
His comments reflect a growing tension at the heart of the AI industry. Companies continue to invest heavily in the development of more capable models and autonomous systems, while at the same time acknowledging that governance frameworks may not be evolving quickly enough to keep pace.
Speaking in London, Clark argued that the industry currently possesses only one means of acceleration.
“You want the option to be able to take your foot off the gas and put your foot on the brake,” he said. “Right now, it’s like the AI industry has a gas pedal, but it doesn’t have a brake pedal.”
The remarks come at a time when AI systems are becoming increasingly embedded in business operations, public services and knowledge work, raising questions about accountability, oversight and long-term societal impacts.
When AI begins writing itself
Among the most significant concerns raised by Clark was the increasing role AI is playing in its own development.
According to Clark, Anthropic’s Claude chatbot is now operating on code of which around 80 per cent was written by the system itself. He suggested that figure could reach 100 per cent within two years, a shift he described as having “huge implications”.
The prospect of AI systems contributing substantially to their own development is likely to intensify debate around governance and control. While the industry has long discussed artificial intelligence as a tool for assisting human developers, the possibility of systems increasingly generating the code on which future versions are built raises new questions about transparency, oversight and responsibility.
Clark did not suggest that such developments should be halted. Instead, he argued that governments and regulators should begin establishing frameworks capable of providing confidence in increasingly advanced systems.
“The world needs to do some thinking and we need to eventually develop some new regulations that allow us to be confident in these systems,” he said.
Governance becomes a data challenge
The debate over AI oversight is increasingly extending beyond the models themselves and into the data infrastructure that supports them.
Responding to Clark’s comments, Stuart Harvey, Chief Executive of Datactics, argued that the ability to govern AI ultimately depends on the quality and visibility of the data feeding those systems.
“AI is moving far faster than most organisations’ data foundations can support it,” Harvey said. “The real issue is not controlling AI, it is ensuring the underlying data pipelines are strong enough to make these systems reliable.”
Harvey suggested that traditional governance approaches are becoming increasingly inadequate as organisations deploy more autonomous and agentic AI systems.
“Data is no longer just accessed, it is continuously interpreted and acted upon by autonomous agents,” he said. “Without clear data ownership, end-to-end lineage, and alignment between data governance and model oversight, organisations cannot slow or regulate AI.”
The comments highlight a growing shift in industry thinking. While much of the public debate focuses on model behaviour and AI safety, many organisations are discovering that governance challenges often begin with data quality, traceability and accountability.
The contradiction at the heart of AI
Clark’s intervention also exposes one of the defining contradictions of the current AI era.
Anthropic has publicly advocated stronger oversight of advanced AI systems, yet like its competitors it continues to invest heavily in research and development. The company recently welcomed a new US executive order, although the measures stop short of mandating government-led safety testing and leave assessments largely voluntary for developers.
At the same time, competition among leading AI companies remains intense, with organisations racing to build increasingly capable models and capture market share in what many view as a transformational technology cycle.
Clark acknowledged that artificial intelligence could bring significant economic and social disruption, including potential impacts on employment as autonomous systems become more capable. However, he argued that human creativity, curiosity and original thinking are likely to remain important advantages.
His call for a “brake pedal” may therefore be less about slowing innovation and more about ensuring society develops the governance mechanisms needed to manage it. As AI systems become more capable, more autonomous and more deeply embedded in economic activity, the challenge facing policymakers may not be whether the technology advances, but whether institutions can evolve quickly enough to retain meaningful oversight of its development.
