AI Outsmarts Human Red Teams in Phishing Tests

No longer content to beat humans at chess and Go, artificial intelligence can now beat red teams at their own phishing game, at scale.

See Also: Live Webinar | AI-Powered Defense Against AI-Driven Threats

AI has surpassed human red teams in crafting phishing attacks, at scale and with alarming success, asserts research from cybersecurity training firm Hoxhunt. The company’s proprietary AI spear phishing agent, code-named JKR for Joker, in March outperformed human counterparts by 24%, a turnaround from a 31% deficit in 2023 when Hoxhunt ran a similar test.

“It’s a Skynet moment for social engineering,” the company said in a blog post, referencing the AI villain from the Terminator franchise. “We’ve proven that AI agents can create superior spear phishing attacks at scale.”

JKR’s edge comes from its ability to fine-tune its prompts and outputs in real time. This iterative mechanism allowed the AI to adapt to user-specific contexts like role and location, generating hyper-personalized emails for millions of enterprise users.

The phishing-as-a-service market will soon shift to mass adoption of AI spear phishing agents, the company said. Once that happens, the baseline quality and effectiveness of mass phishing campaigns will rise to a level we currently equate with targeted spear phishing attacks, it said.

The non-profit Anti-Phishing Working Group in March reported an increase of global phishing emails during the second half of last year after a lull during the previous six months. During the last three months of 2024 alone, the APWG said it detected nearly a million dedicated phishing sites. U.S. authorities have repeatedly warned residents against a surge over the past year of smishing texts purporting to come from a road toll collection service – although the non-profit said the Chinese scammers behind the campaigns haven’t invested much energy in selecting their targets (see: Surge in Smishing Fueled by Lucid PhaaS Platform).

“The phone numbers that the phishers send the messages to are usually random – they are sometimes sent to people who do not use toll roads at all, or target users in the wrong state,” the APWG said.

Casey Ellis, founder of Bugcrowd, told Information Security Media Group that “Humans are uniquely creative and adaptable,” but AI attack agents “can operate at scale and never need to sleep.” When the goal is reach rather than depth, “AI will tend to perform better.”

Amit Zimerman, co-founder and chief product officer at Oasis Security, cautioned against relying solely on AI to counter AI phishing. Limitations include false positives, poor contextual judgment and potential blind spots. Zimerman said human oversight is key in interpreting results and guiding decision-making.