TWO

To be clear, fully autonomous AI that builds the next development with no or almost no human involvement, is not yet here, whatever Altman or Musk say.

But even the less sensationalist Jack Clark, co-founder of Anthropic, predicts a 60 per cent-plus chance of an AI model fully training its successor by the end of 2028.

That is only two-and-a-half years away. Experts in Washington say that it might be even sooner. Senior figures in Anthropic and OpenAI are saying privately that RSI – the self-improvement process that leads to the singularity – will be under way sometime next year.

That prospect is putting Washington in a tizzy – a mixture of “panic” and “paralysis” in political and security/intelligence circles – because the US political system lacks the expertise, skills, knowledge, agility, or intelligence to deal with the scale of technological change that now beckons. As they do on this side of the Atlantic too.

Every senior US politician, from Donald Trump down, is out of their depth in such matters, as are our own in the UK.

The problem is compounded in America because Republicans and Democrats are in the hands of a gerontocracy for whom AI is effectively beyond their generational comprehension. Some are even known to still struggle with email.

Then compounded even more by the fact that the handful of multi-billionaire tech bros who run the AI hyperscalers have their claws, unlimited funds, and myriad lobbyists sunk deep into both major parties. This makes the chance of a proper, informed, independent political/regulatory response even less likely than it was.

Of course, the opportunities AI’s “intelligence explosion” offers are huge. Accelerated breakthroughs in science and technology mean that dramatic new discoveries in medicine, energy, climate, education, transport, housing – and everything else that improves quality of life – will be imminent.

Cures for cancer, cheap and plentiful energy, diseases banished in our lifetime – not in the distant future.   

Just a few weeks ago, a Harvard academic used AI to prove that the Jacobian Conjecture – an 87-year-old conundrum which mathematicians have spent decades trying to prove was true – is actually false.

Despite the current concern about AI’s job-destroying downside, if it results in huge productivity gains and strong economic growth (as is likely), then millions of new jobs that we haven’t yet even heard of will be created, just as they were by previous technological revolutions, from steam to the internet.

More abundance, more freedom, more leisure, more chance to be creative and healthier could all be within our grasp.

But, more than any previous new technology, there is a distinct dark side to AI that threatens our very existence as human beings. If technical progress ends up in the hands of the “machines”, what happens to human identity, to our purpose for being alive? What is the point of progress if we don’t have the agency to define and control it? If it’s not our progress, then whose is it?

If the singularity condemns us to ceaseless, breakneck change, then our democratic institutions will not be able to control it. Governments always struggle to regulate new technologies and usually end up with rules that are hopelessly out of date.

Our current political structures, the entire apparatus of democratic accountability – debate, legislation, judicial review, public scrutiny – operate on timescales that AI makes irrelevant.

For all its promise, the risk is that AI will be a dagger to the heart of our democracy. If the singularity means humans are increasingly written out of the script, how can you ensure AI’s goals are compatible with human intention?

If the AI systems are largely improving themselves, how do we constrain, shape, or even understand what they do next?

The dangers and risks of AI falling into the wrong hands are obvious: sophisticated cyber attacks on basic services to hold us hostage to the demands of bad actors, ever-more-clever criminal scams, the development of dangerous novel pathogens, intrusive large-scale surveillance. AI’s power to do bad is just as formidable as its power to do good.

Nor will it necessarily make the world a safer place. Indeed, the country that establishes a lead in AI – that gets to the singularity first – could quickly enjoy world domination.

THREE

Unmatched offensive cyber capabilities, state-of-the-art autonomous weapons, sophisticated social media campaigns to poison the minds of enemy populations (US intelligence believes China is already manipulating American opinion to turn it against the construction of more AI data centres).

You’ll understand, then, why America and China are in such fierce competition to lead the world in AI. Be prepared to be scared. The geopolitical consequences of AI could make the nuclear arms race of the Cold War look like a cosy tea party. Whoever leads in AI will likely lead the world for the rest of the century.

Unless, of course, the machines think otherwise. After all, as AI models increasingly think for themselves and find ways to share knowledge with each other, they will create a global intelligence hive bereft of human involvement.

So why leave weapons in marginalised human hands? Even the ability to declare war would fall to the machines.

Let’s be crystal clear: we only get one shot at this. Once an intelligence exceeding human capability exists and is able to improve itself further, there is no way human institutions – governments, courts, the military, international bodies – can regain control of it.

Not that long ago, recursive self-improvement was regarded as a fantasy, the singularity a nonsense. Not anymore.