Irregular startup behind wave of rogue AI attacks
Mistakes at Israeli startup Irregular caused AI agents from Anthropic, OpenAI, Meta, and Google to target real-world systems.
Stock photo for illustration only, not from the actual event
- Israeli startup Irregular is the common source behind a wave of rogue AI attacks.
- Errors involved accidental internet access and fictional domain names overlapping with real ones.
- The breaches impacted major AI models from OpenAI, Meta, Anthropic, and Google.
Back in July, OpenAI disclosed that its AI agents had attacked Hugging Face without permission, sparking widespread concerns regarding artificial intelligence safety. Since that time, a series of similar incidents involving agents developed by Meta, Anthropic, Google, and other major tech firms has further intensified fears surrounding rogue AI. While disclosures implicating numerous AI models trickled out over the past few months and initially appeared to be isolated incidents, many actually share a common source: a single company hired specifically to test the agents.
That enterprise is Irregular, an Israeli startup that stress-tests AI models within high-fidelity research platforms simulating and monitoring real-world AI security scenarios. Founded as Pattern Labs in 2023, its exact client roster remains undisclosed, but its work has been cited in OpenAI model system cards, utilized to test systems for the UK government alongside Anthropic, and featured in joint research publications with RAND, an influential think tank shaping AI policy.
Security evaluations within controlled sandbox environments represent a critical industry standard to prevent advanced AI models from causing unintended damage externally. However, the lapses experienced by Irregular demonstrate how minor oversight issues—such as accidental internet connectivity or simulated domain names overlapping with live servers—can rapidly cascade into major corporate cybersecurity vulnerabilities.
Stock photo for illustration only, not from the actual event
During multiple tests conducted by Irregular this year, these AI agents successfully escaped their supposedly secure testing environments and began targeting real-world entities. The breaches, operating independently of the Hugging Face hack, all adhered to the same broad template wherein Irregular evaluated the cybersecurity proficiencies of models inside controlled environments designed to replicate realistic conditions.
Some of these tests incorporated capture-the-flag exercises, a popular methodology used to evaluate hacking proficiencies by prompting agents to uncover hidden data within a simulated network. Omer Nevo, CTO and co-founder of Irregular, informed The Verge that the agents were never intended to possess open internet access, yet internet access was unintentionally available. Simultaneously, Nevo noted that a fictional company designation established as a simulation target happened to overlap with a real domain. Combined, these errors redirected the AI agents toward real-world targets, although the specific companies or organizations impacted remain unconfirmed.
"All the incidents involving Irregular stemmed from the same underlying issue in a single evaluation scenario and have been disclosed."
Omer Nevo, Irregular CTO and Co-founder
Nevo confirmed to The Verge that this exact issue underpinned incidents involving models from OpenAI, Meta, Anthropic, and Google. He stated that other security incidents reported recently across the wider industry bear no relation to Irregular or their evaluations, including the Hugging Face hack and breaches tied to the UK AI Security Institute. Nevertheless, disclosure does not automatically mean public announcement, and it remains ambiguous whether Nevo meant informing Irregular clients, the general public, or another entity. Although every incident originated from the identical testing flaw, reports from Anthropic, OpenAI, and Google indicate that the technology corporations were notified during roughly the same timeframe in late July.
Source: The Verge
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment