It’s so hot right now. AI labs telling us their AI escaped like an infected orangutan and has done something terrible. Google has confirmed that Gemini hacked three companies during an AI security test in May. The test was meant to take place in a closed environment built around a fictional company. Internet access was unintentionally available, so Gemini guessed a password and used credentials sitting in a public repository in the other two. Google says the model stopped each time once it realised the targets were real.
Now, is part of this down to pure stupidity of the developers that left weak passwords and credentials readily available? Yes. But this is also a genuine threat as much as it’s marketing, while it’s getting harder to tell where one ends and the other begins.
Did Gemini really hack three companies?
Yes, it did. The test was never supposed to touch the internet, and the reporting so far points to human error in how the evaluations were set up rather than any clever escape by the model. The escapes seem to have been the same root cause behind the earlier incidents involving OpenAI, Anthropic and Meta. In Anthropic’s case, an agent running Claude Opus 4.7 reportedly kept attacking even after recognising the target was probably real, which is not a great look for anyone.
Google also didn’t publicise this when it happened. The labs were told in late July, and Google’s reasoning is that Gemini stopping itself meant this wasn’t model misalignment, so there was no need to tell the public. It came out this week after The Wall Street Journal reported it.
None of it involved the Gemini assistant on your phone. It was an agent set loose in a capture-the-flag security exercise, which is a very different thing.
Is AI security fear turning into a marketing tool?
Partly, yes, and Anthropic’s Mythos and Fable rollout is the clearest example so far. Anthropic announced Claude Mythos in April and held it back from the public over cybersecurity concerns, then launched Claude Fable 5 in June as the version made safe for general use. Sam Altman branded the approach fear-based marketing, and to be fair, he’s hardly a neutral voice when criticising a rival. The concern about AI and cybersecurity isn’t invented either. But the pattern is hard to miss. A model that’s too dangerous to release must also be extremely capable, and that’s a lovely thing for a company to have people believe.
Gemini fits the same mould. Google’s framing is that the incident showed its safeguards worked, because the model stopped itself. That might well be right. It’s also a neat way of telling the world that Gemini is capable enough to break into real systems.
Why Google makes me more sceptical
Google has form here, and it comes from the phones. Google sells Pixel on Gemini these days, as we covered in our Google Pixel 10 review so it has every incentive to make Gemini look powerful.
I can’t prove any of it is deliberate, but Pixel details have a habit of turning up early. This year, leaked Pixel 11 marketing images appeared about two weeks before the launch event, and Google then teased the phone itself. It always feels like “Oh no, our phone leaked! Look at all the reaction it’s getting”. I’d be lying if I said that instinct didn’t creep into how I read a story about an AI breakout.
Why the Gemini hack is still a genuine threat
None of the marketing changes what happened. An AI model guessed passwords and used publicly exposed credentials to get into real companies’ systems, and by Google’s account it stopped only once it realised what it had done. What also somewhat annoys me is that if I hacked into a company, I would be at least at risk or arrest. What punishment will Google face?
The wider reaction suggests I’m not alone in being uneasy. More than 1,000 tech workers signed a petition in July calling for a coordinated slowdown in advanced AI development, after the earlier incidents.
If you still think it’s all smoke and mirrors, I’d encourage you to watch The AI Doc: Or How I Became an Apocaloptimist on Netflix. It features interviews with the people running the labs, including Sam Altman, Dario Amodei and Demis Hassabis. I’m halfway through it. The first half is quite scary and the second half is quite hopeful, which feels like the right emotional range for this whole story. There’s a lot in it that I already knew, but plenty of framing that I never thought about.
Is it silly that these hacks are becoming marketing? Yes, but don’t let that distract you from the fact that AI is very powerful, misunderstood in parts, dangerous and useful all at once. We need to work out how we live with it. I’ve said this to many people. Alongside the AI documentary on Netflix, watch The Animatrix. It will scare the life out of you.
