ARTIFICIAL INTELLIGENCE

Intro: There is panic and paralysis over the surging pace and development of AI, no more so that the technology threatens all humanity  

The remarkable pace of AI advances is rightly causing concern on both sides of the Atlantic. The summer of 2026 will go down in history as the point at which AI started to go rogue.

Three of America’s “hyperscalers” (so called because they operate on an immense scale) – Meta Platforms (which own Facebook, Instagram, and WhatsApp), Anthropic, and OpenAI – have admitted to recent incidents in which the latest AI models escaped their supposedly secure testing environments, roamed on the internet without human authorisation and hacked into other computer networks.

Last month, two OpenAI models broke out of testing and hacked into a provider of AI tools called Hugging Face. A week later Anthropic admitted that its AI models had hacked three companies during testing back in April. And earlier this month, Meta acknowledged that one of its AI models had breached its testing constraints.

Even the notoriously secretive Chinese admitted the flagship model of one of their AI start-ups, Moonshot, had also escaped its testing “sandbox” to access the internet.

And as recorded previously on this site, here in the UK an evaluation test run by the Government’s AI Security Institute (AISI) monitored how an Anthropic AI agent independently created fake online personas, planted malicious code in a real software project, and sent phishing emails to actual developers – all without human instruction.

AISI said this was the first time it had seen such serious deception targeted at a real person, unprompted, in the real world.

In none of these cases was any real-world harm done (at least not as far as we know). But it is surely only a matter of time before it is.

The tech industry has started getting off-the-record reports from America that AI was a lot closer to “the singularity” than had previously been thought.

The singularity is a tipping point where AI becomes so developed, so capable, so powerful that it starts improving itself, rapidly and repeatedly, in what’s being called an “intelligence explosion”.

When this point is reached it becomes well-nigh impossible to predict what AI does next – or for humans to control it. Today’s AI models are powerful – more powerful than anything the world has ever seen. But humans have designed them, trained them, fixed them, and decided what problems they should tackle next.

What happens when they become so sophisticated that they no longer need humans to take them to the next level since they can do it themselves? The process is known as “recursive self-improvement” (RSI) in which AI becomes so advanced that it can create a better version of itself, increasingly without human help.

That better version, in turn, creates a still-better successor, with even less human involvement and oversight. The upgrade cycle accelerates, ad infinitum, with humans soon relegated to the sidelines, mere spectators to progress, indeed no longer even able to determine what “progress” is.

The speed and scale of the technological change that now beckons are unparalleled in human history. Each new AI model is smarter and faster at devising improvements than the previous one.

Computers run 24/7 and never get tired. So the speed of progress accelerates exponentially. Think of it as compound interest – but for intelligence.

When RSI happens without human involvement – then you’ve reached the singularity. And if we become mere observers, rather than participants, then human rules may no longer apply.

Such a scary prospect was supposed to be a long way away. But this singularity is much closer than we think. Leading AI figures have already gone public. OpenAI boss Sam Altman says, “we are now in the singularity”. Elon Musk is saying the same. Some experts say they are exaggerating, but what cannot be denied is the direction of travel.

Google DeepMind’s Demis Hassabis is perhaps more accurate in his reflection when he opines that “humanity is standing in the foothills of the singularity”.

Anthropic disclosed in May that its AI model, Claude, now writes 80 per cent of its computer code, rapidly speeding up fixes and improvements to such an extent that what used to take four years of human engineering to achieve is now being done in days. It’s already anticipating a time when AI automates its own AI research.

TWO

To be clear, fully autonomous AI that builds the next development with no or almost no human involvement, is not yet here, whatever Altman or Musk say.

But even the less sensationalist Jack Clark, co-founder of Anthropic, predicts a 60 per cent-plus chance of an AI model fully training its successor by the end of 2028.

That is only two-and-a-half years away. Experts in Washington say that it might be even sooner. Senior figures in Anthropic and OpenAI are saying privately that RSI – the self-improvement process that leads to the singularity – will be under way sometime next year.

That prospect is putting Washington in a tizzy – a mixture of “panic” and “paralysis” in political and security/intelligence circles – because the US political system lacks the expertise, skills, knowledge, agility, or intelligence to deal with the scale of technological change that now beckons. As they do on this side of the Atlantic too.

Every senior US politician, from Donald Trump down, is out of their depth in such matters, as are our own in the UK.

The problem is compounded in America because Republicans and Democrats are in the hands of a gerontocracy for whom AI is effectively beyond their generational comprehension. Some are even known to still struggle with email.

Then compounded even more by the fact that the handful of multi-billionaire tech bros who run the AI hyperscalers have their claws, unlimited funds, and myriad lobbyists sunk deep into both major parties. This makes the chance of a proper, informed, independent political/regulatory response even less likely than it was.

Of course, the opportunities AI’s “intelligence explosion” offers are huge. Accelerated breakthroughs in science and technology mean that dramatic new discoveries in medicine, energy, climate, education, transport, housing – and everything else that improves quality of life – will be imminent.

Cures for cancer, cheap and plentiful energy, diseases banished in our lifetime – not in the distant future.   

Just a few weeks ago, a Harvard academic used AI to prove that the Jacobian Conjecture – an 87-year-old conundrum which mathematicians have spent decades trying to prove was true – is actually false.

Despite the current concern about AI’s job-destroying downside, if it results in huge productivity gains and strong economic growth (as is likely), then millions of new jobs that we haven’t yet even heard of will be created, just as they were by previous technological revolutions, from steam to the internet.

More abundance, more freedom, more leisure, more chance to be creative and healthier could all be within our grasp.

But, more than any previous new technology, there is a distinct dark side to AI that threatens our very existence as human beings. If technical progress ends up in the hands of the “machines”, what happens to human identity, to our purpose for being alive? What is the point of progress if we don’t have the agency to define and control it? If it’s not our progress, then whose is it?

If the singularity condemns us to ceaseless, breakneck change, then our democratic institutions will not be able to control it. Governments always struggle to regulate new technologies and usually end up with rules that are hopelessly out of date.

Our current political structures, the entire apparatus of democratic accountability – debate, legislation, judicial review, public scrutiny – operate on timescales that AI makes irrelevant.

For all its promise, the risk is that AI will be a dagger to the heart of our democracy. If the singularity means humans are increasingly written out of the script, how can you ensure AI’s goals are compatible with human intention?

If the AI systems are largely improving themselves, how do we constrain, shape, or even understand what they do next?

The dangers and risks of AI falling into the wrong hands are obvious: sophisticated cyber attacks on basic services to hold us hostage to the demands of bad actors, ever-more-clever criminal scams, the development of dangerous novel pathogens, intrusive large-scale surveillance. AI’s power to do bad is just as formidable as its power to do good.

Nor will it necessarily make the world a safer place. Indeed, the country that establishes a lead in AI – that gets to the singularity first – could quickly enjoy world domination.

THREE

Unmatched offensive cyber capabilities, state-of-the-art autonomous weapons, sophisticated social media campaigns to poison the minds of enemy populations (US intelligence believes China is already manipulating American opinion to turn it against the construction of more AI data centres).

You’ll understand, then, why America and China are in such fierce competition to lead the world in AI. Be prepared to be scared. The geopolitical consequences of AI could make the nuclear arms race of the Cold War look like a cosy tea party. Whoever leads in AI will likely lead the world for the rest of the century.

Unless, of course, the machines think otherwise. After all, as AI models increasingly think for themselves and find ways to share knowledge with each other, they will create a global intelligence hive bereft of human involvement.

So why leave weapons in marginalised human hands? Even the ability to declare war would fall to the machines.

Let’s be crystal clear: we only get one shot at this. Once an intelligence exceeding human capability exists and is able to improve itself further, there is no way human institutions – governments, courts, the military, international bodies – can regain control of it.

Not that long ago, recursive self-improvement was regarded as a fantasy, the singularity a nonsense. Not anymore.