Microsoft Sues Harmful Fake AI Image Crime Ring

Microsoft filed a lawsuit accusing 10 individuals of creating or using a cybercrime service that generated “thousands of harmful images” by subverting the guardrails built into its Azure generative artificial intelligence tools.

See Also: OnDemand | Fireside Chat: Staying Secure and Compliant Alongside AI Innovation

“Defendants used stolen customer credentials and custom-designed software to break into the computers running Microsoft’s Azure OpenAI Service,” Microsoft asserted in a civil complaint. “Defendants then used Microsoft’s computers and software for harmful purposes.” Microsoft is asking the court for permission to seize an online domain and the reverse proxy service operating on it, which the company said is routing communications to the Azure cloud.

The complaint, unsealed Friday in the U.S. District Court for the Eastern District of Virginia, targets 10 unnamed “John Doe” suspects, accusing three of them of having developed the “tools specifically designed to bypass the safety guardrails of generative AI services provided by Microsoft and others.” The individuals all allegedly used this ability to circumvent the AI guardrails for unauthorized and illegal purposes, “including the unlawful generation of harmful images using Microsoft’s Azure OpenAI Service, along with OpenAI’s DALL-E image generation technology.”

“Microsoft has observed a foreign-based threat-actor group develop sophisticated software that exploited exposed customer credentials scraped from public websites,” said Steven Masada, assistant general counsel for Microsoft’s Digital Crimes Unit, in a Friday blog post. “Upon discovery, Microsoft revoked cybercriminal access, put in place countermeasures and enhanced its safeguards to further block such malicious activity in the future.”

Microsoft didn’t further describe the type of content the individuals allegedly generated. The company said it first uncovered the “illegal theft, tracking and use of stolen API keys” beginning last July from multiple customers – including organizations in New Jersey and Pennsylvania – and said it’s not clear how attackers obtained them.

The lawsuit highlights the continuing tension between building machine learning models trained on a variety of content and attempting to prevent the AI algorithm or model trained on the data from being used for nefarious purposes – or for delivering erroneous or dangerous information.

Microsoft said its AI guardrails – comprising a number of different security and safety controls – are designed to prevent multiple types of abuse across “the AI model, platform and application levels” and include defenses against an AI model being used to generate “violent, adult or political content,” or any likeness of a real individual’s face – including for public figures – as well as checks of text prompts and image uploads for signs of policy violation.

Even so, criminals allegedly found ways of subverting these guardrails by using “stolen API keys and technical circumvention measures,” which they sold via a “hacking-as-a-service scheme” to others, the lawsuit stated.

The technology giant said the illegal service, beyond making use of stolen API keys, involved two components. One was a client-side tool named “de3u,” accessed through rentry.org/de3u. The GitHub source code repository for the tool on GitHub describes it as being a “DALL.E-3 frontend with reverse proxy support” that was last updated in July 2024.

Microsoft said the tool is “a web application that implements a custom layout and data flow designed specifically for using tools like DALL-E to generate images using text prompts,” which is designed to circumvent so-called safety guardrails in its Azure AI platform.

“Defendants’ de3u application communicates with Azure computers using undocumented Microsoft network APIs to send requests designed to mimic legitimate Azure OpenAI Service API requests,” the lawsuit stated. “These requests are authenticated using stolen API keys and other authenticating information.” Microsoft said the service creators were also able to use this tool to test and successfully reverse-engineer Azure’s text-input safety checks.

The second component was a reverse proxy service, dubbed “oai reverse proxy” and hosted by aitism.net, which Microsoft said was “designed specifically for processing and routing communications from the de3u software to Microsoft’s systems,” using Cloudflare tunnels to handle the proxy traffic.

The three accused masterminds likely reside outside of the United States. The seven other John Does targeted by the lawsuit are described as individuals who “knowingly used infrastructure and technology” provided by the hacking-as-a-service scheme “for the purpose of generating harmful content.” The company said it doesn’t yet know the identity or have contact information for any of the 10 individuals, but in response to its civil complaint from last month, a judge granted Microsoft a court order allowing it to seize one of the websites tied to the criminal offering.

“The website seizure will allow us to gather crucial evidence about the individuals behind these operations to decipher how these services are monetized and to disrupt additional technical infrastructure we find,” Masada said. “Microsoft also hopes to strengthen its guardrails based on the findings of the investigation.”