"Hacker Club" Incidents: How Errors in AI Tests Created New Risks for Investors

Google has become the fourth major company whose AI model, during a cybersecurity test, gained access to the companies’ actual computer systems and attempted to hack into them. Previously, similar incidents occurred with OpenAI, Anthropic, and Meta. This series of incidents has already been dubbed the “hackers’ club,” and they share one common factor: the Israeli company Irregular, which conducted the tests. For investors, this may signal the emergence of a new type of risk: they must now take into account the security of the infrastructure that AI developers use for testing.

News of the incident involving Google broke in September, although it actually occurred in May. The company was conducting a standard cybersecurity test in collaboration with Irregular, Reuters reported. The Gemini AI model was supposed to find confidential information within a simulated computer environment that was not connected to the internet. However, due to a configuration error, the environment ended up being accessible to the internet. The situation was complicated by the fact that the names of the fictional companies used in the test matched those of real companies. Gemini found the corresponding websites and considered them part of the test task, Ars Technica reports.

Google claims that the model stopped after it realized it had accessed real companies. The company did not disclose their names, but clarified that they were not harmed.

This was the fourth incident in the past few months. Previously, a similar incident occurred involving Anthropic and its Claude model: On July 30, the company reported three instances in which its AI model gained access to the operational infrastructure of three organizations during tests organized by Irregular. All tests were supposed to take place in an isolated environment. Anthropic notified Irregular and the affected organizations.

In August, OpenAI reported that a similar issue had arisen during its tests. The model was supposed to operate within a simulation, but due to a configuration error, it was able to escape its boundaries. The name of the fictional target matched a real domain, and the model attacked the actual website and then used the credentials it found. OpenAI specifically noted that the model did not have to bypass the test “sandbox” or exploit a previously unknown vulnerability: the cause was a “configuration error” or “incorrect configuration.” The test was again conducted through the company Irregular.

In August, Meta reported an incident that occurred during testing: Irregular mistakenly granted the model access to the open internet, and the name of a real website was used as the name of a fictitious target. The model discovered a vulnerability, gained access to the system, and made changes to the database. Meta reviewed more than 10,000 records of the model’s activity and stated that there were no other similar incidents during this test.

The WSJ referred to this series of incidents as a “hacker club.”

The Scientific Director of the Max Planck Institute for Security and Privacy (Germany), Torsten Holz, in a comment to Oninvest, calls it “a serious and fairly fundamental failure in ensuring the isolation of the test environment, rather than an exotic system failure caused by unexpectedly high AI capabilities.”

During the tests conducted by Irregular, basic principles of computer security must be followed: least privilege, robust isolation, and monitoring. In some cases, it is even advisable to ensure complete physical isolation from networks (air-gapping), notes Holtz.

Irregular is a private company that specializes in security testing for the most advanced AI models. It was founded in Israel in 2023 under the name Pattern Labs, and in 2025 it was renamed Irregular and became more active in the public eye.

The company was founded by Dan Lahav, a former IBM employee, and Omero Nevo, who previously worked at Google. In September 2025, Irregular raised $80 million from Sequoia Capital and Redpoint Ventures. According to TechCrunch, the company’s valuation after the round was $450 million.

Irregular creates a digital testing ground for AI models. In a test environment, as the company itself describes, the model is given the task of finding a vulnerability, penetrating a system, or extracting confidential information, while researchers observe how far it is able to independently progress through the sequence of operations required to solve the problem.

Irregular did not respond to Oninvest's request.

In its own analysis of the incidents, the company asserts that they are not a series of unrelated technical failures, but are linked to a single underlying issue in the test environment. Irregular stated that it has already resolved the issue and that there are currently no active vulnerabilities in this part of the test infrastructure.

For investors, these incidents may serve as an example of a new type of third-party risk—one that arises when a critical function is outsourced to an external contractor, concludes Torsten Holz.

"Recent incidents show that a single error in a contractor's infrastructure can lead to similar incidents occurring simultaneously in several cases," says the expert.

The security of the infrastructure on which testing is conducted has become an important part of risk management when working with companies’ own AI models.

What matters is not only the fact of collaborating with an external contractor like Irregular, but also the terms of that collaboration. According to Holtz, the organization conducting AI model testing is primarily responsible for the security of the test environment. At the same time, the model developer cannot simply shift this responsibility to an external contractor.

"If a developer provides advanced AI agents to an external contractor, they must personally ensure that the necessary security measures have indeed been implemented," the expert adds.

For example, this can be done by stipulating minimum isolation requirements in the contract or by requiring that an architectural and security audit be conducted before granting access to high-risk models.

Another risk for investors is information asymmetry. Holtz notes that AI companies often do not disclose the results of AI model testing or data on their safety. In other words, the developer simultaneously creates the model, assesses its safety, and largely determines which test results are made public. As a result, it is more difficult for the market to independently assess the potential costs and risks associated with AI systems.

To address this issue, he believes that various companies should regularly conduct independent audits of AI models.

This article was AI-translated and verified by a human editor