Sadiq Khan says fake AI audio of him nearly led to serious disorder
BBCLondon Mayor Sadiq Khan says deepfake audio of him supposedly making inflammatory remarks before Armistice Day almost caused "serious disorder".
Mr Khan says the law is not "fit for purpose" in tackling AI fakes, as the audio creator "got away with it".
The man who first posted the clip, tracked down by the BBC, told us: "It's what we all know Sadiq thinks."
But another social media user who helped the audio go viral apologised, saying: "I made a big mistake".
The clip used AI - artificial intelligence - to create a replica of Mr Khan's voice saying words scripted by the faker, disparaging Remembrance weekend with an expletive and calling for pro-Palestinian marches, planned for the same day last November, to take precedence.
Intended to sound like a secret recording, it said: "What's important and paramount is the one-million-man Palestinian march takes place on Saturday."
The clip imitated Mr Khan, the first Muslim mayor of London, saying: "I control the Met Police, they will do as the Mayor of London tells them" and saying "the British public need to get a grip". It said the prime minister meeting with Met Police Commissioner Sir Mark Rowley was "a waste of time" because "the buck stops with me".
Mr Khan told BBC Radio 4's Why Do You Hate Me? podcast: "You know, we did get concerned very quickly about what impression it may create. I've got to be honest, it did sound a lot like me."
The clip spread rapidly, including among far-right groups, and triggered a spike in hateful comments against the mayor on social media.
"When you've got friends and family who see this stuff, it's deeply upsetting. I mean, I've got two daughters, a wife, I've got, you know, siblings. I've got a mum," Mr Khan told the BBC.
The AI fake emerged during an already-tense political row, as Prime Minister Rishi Sunak said the pro-Palestinian marches in a different part of central London were "disrespectful" on Armistice Day. Then-Home Secretary Suella Braverman had called for them to be cancelled.
Armistice Day marks the moment when World War One ended on 11 November 1918, while memorial ceremonies take place at the Cenotaph and across the country on the nearest Sunday, known as Remembrance Sunday.
"The timing couldn't have been better if you're seeking to sow disharmony and cause problems," the London mayor told the BBC.
"What was being said was a red rag to a bull for the far right and others. But what concerned me the most was if you're an innocent listener of this. Because it's a secret undercover recording, in inverted commas, because it sounds like me, because of the timing and the context."
Both events took place on Saturday 11 November, with the pro-Palestinian march beginning hours after the two minutes' silence had been observed. But counter-protestors, including some people connected to far-right groups, were later condemned by police after clashes and arrested for offences including for inciting racial hatred.
One of the BBC's most trolled journalists, Marianna Spring, dives into her inbox and investigates extraordinary cases of online hate. She meets the people at the heart of these conflicts, and in some cases brings them together, to see if understanding - even forgiveness - is ever possible.
Listen now on BBC Sounds, watch on BBC iPlayer and read the previous instalment of the online series here.
Mr Khan said the AI fake had inflamed the situation. "We almost had serious disorder," he said, adding that deep fake audio could have a worrying impact on other situations - such as a close election or referendum, or where there is community unrest.
"People should be able to criticise me. But I think what you shouldn't do is use AI to manufacture lies. And I think the person that made this needs to realise the consequences on that Saturday."
One of the larger accounts that spread the fake audio was Little Boats on X, which has shared anti-Islam and anti-immigration content and has more than 58,000 followers.
Jeremy Davis, who is in charge of the account, said he still suspects Mr Khan has beliefs similar to those in the audio but apologised for sharing the fake.
"It was a clever AI job," he said. "Mayor Khan, I apologise, I made a big mistake."
But Mr Davis did not create the audio.
Through a screengrab, I traced the recording back to TikTok and what appears to be the originator of the clip: an account called HJB News with the ironic tagline "Keeping it real". The account shares anti-immigration content and some racist material.
HJB News's accounts on other social media show they shared the audio on 9 November, hours before it went viral. The fact that there's no trace of the audio anywhere else online before HJB News shared it, makes it likely that they also created it.
After exchanging several messages, the person behind the account called me. He gave his name only as Henry and he would not let me record him, but I took detailed notes of the conversation.
He claimed it was "a clip on TikTok I shared" and denied creating it, saying he "wouldn't want to comment" when I pointed out he had been the first to share it.
He said his account does not only publish "fake clips". "We post news that could be real with a sense of humour."
On 11 November, the Metropolitan Police said it had examined the audio but that it "does not constitute a criminal offence". HJB News said on X, "well that's a relief" with an emoji of a bead of sweat rolling down a grinning face.
Mr Khan said the fake audio "wasn't a bit of fun" or "satire" and its creator had not been not "naive" about the consequences it could have.
The London mayor said he does, however, accept the apology of people who shared it because of an "innocent mistake".
Mr Khan said organisations such as the Electoral Commission, which are responsible for keeping the UK's elections "free and fair", also needed more powers to deal with faked information.
There is currently no criminal law in the UK which specifically covers this kind of scenario.
The mayor said it was also "really worrying" that social media companies did not contact him or the authorities about the faked audio at the time it went viral.
I contacted X, TikTok and Instagram, where the clip was shared. X and Instagram did not respond. According to a TikTok spokesperson, the social media site "does not allow synthetic media that contains the likeness of any real private figure" and removes this kind of content.
TikTok said it had spoken to both the mayor's office and the Metropolitan Police in November 2023 about the platform's approach to this content and flagged how similar issues could be raised directly in future.