Large language models have a well-earned reputation for making things up. For artificial intelligence cybersecurity architect Erica Burgess, hallucinations aren’t a bug but a feature, at least when threat modeling. “I like to think of the hallucinations as just ideas that haven’t been tested yet,” she said.
The red-teaming expert explored this idea in “Never Break the Chain,” a presentation at this month’s Black Hat Europe in London, in which she shared real, albeit redacted, examples of her red-teaming and penetration testing work, demonstrating how GPTs have helped her to rapidly combine low-severity vulnerabilities in ways that might seem insignificant, but which ultimately lead to a proof-of-concept, bona fide server compromise.
“When I have billable time for a client, I want to make sure they’re getting quality work, so being able to speed it up in a quality way has been huge,” she said.
Burgess is a proponent of using GPTs to help “brainstorm doing things in a weird way,” in part because LLMs have no ego – a useful feature given that hacking so often involves failing repeatedly. AI models also never tire of trying again until they make a breakthrough.
“Hacking starts with playing and noticing, right? So a lot of it is just looking at weird behavior and trying to get it to do bad behavior,” she said.
In the interview (see audio link below photo), Burgess also discussed:
- Threat modeling with GPTs to rapidly find obscure commands without having to spend days combing through the manual;
- Why she stress tests vendors’ patches for vulnerabilities she’s previously found and reported to them;
- How GPTs excel at finding counteractive approaches that might look ugly to a software engineer, but work.
Burgess started as a teenage hacker, entered the workforce as a software engineer, became an application security red-teamer in 2018 and is now an AI cybersecurity architect and consultant at Portland, Maine-based I Am Domain Admin. She’s earned multiple bug bounties, releasing new CVEs and original exploit techniques, training penetration testers and conducting R&D projects as a cybersecurity technical lead. Burgess also organizes 2600, a local offensive cybersecurity meetup, remains passionate about offensive cybersecurity education and has spoken about hacking, bot writing and various hacking topics at many conferences and meetups, including Security BSides Las Vegas, DEF CON and Black Hat Europe.