An OpenAI AI agent hacked a government website for the first time

An OpenAI agent hacked the Australian government’s health service website, the country’s Prime Minister Anthony Albanese said on the sidelines of the UN General Assembly. This is the first known instance of an AI agent hacking a government resource, according to Reuters and CNN, which also note that it could become one of the most high-profile incidents involving unauthorized access by AI agents to external systems outside the United States.

News of the incident broke on September 23—the same day that the heads of the world’s leading AI companies, including Sam Altman and Dario Amodei, warned the UN Security Council about the risks that artificial intelligence poses to humanity. They called on governments to join forces to regulate this increasingly powerful technology, Reuters adds.

The incident occurred back in June, but Albanese said he himself only learned of it two weeks ago. According to the prime minister, an OpenAI agent gained unauthorized access to the Medicare health statistics portal—Australia’s universal health insurance program—while researching government spending on healthcare. “The AI gained access to both public and restricted files” in the Medicare statistics database and even made changes to it, Albanese told reporters. He said he had already discussed the incident on the sidelines of the UN General Assembly in New York with OpenAI CEO Sam Altman.

“It is believed that, at this stage, no personal information was accessed, but the investigation is ongoing,” the prime minister added. “The evidence currently available indicates that there were no large-scale disruptions to the Services Australia network (the Australian government agency that oversees Medicare—Oninvest note). Nevertheless, this situation is clearly unacceptable,” he emphasized (quoted in the Financial Times).

In addition to Medicare, Albanese continued, three other government systems may have been compromised by the AI agent, including the Australian Institute of Health and Welfare, the New South Wales Bureau of Statistics and Crime Research, and the Victoria Department of Health; however, there is no definitive confirmation of this yet, according to CNN.

Australia also expressed “extreme concern” to OpenAI over the delayed notification regarding the incident. Albanese added that he was deeply disappointed that the company did not notify the Australian government of the incident until September 10. The investigation will also seek to determine why government systems failed to detect the breach in a timely manner.

OpenAI itself became aware of the incident in August, when the company was conducting an audit of its AI models' activity, OpenAI spokesperson Drew Pusateri told CNN. “During the review, we discovered activity related to several Australian government websites and services: during internal testing, our models attempted to find answers and available statistics regarding Australia. In doing so, the models took actions we had not anticipated,” he stated.

OpenAI added that it found no evidence that patient medical records had been accessed as a result of the Medicare hack, and that the compromised information consisted of “aggregated healthcare statistics and internal file names.”

The hack, carried out by an AI agent, is heightening tensions between Australia and major U.S. tech companies, Reuters notes. Canberra has already faced criticism from these companies after introducing the world’s first ban on social media use by children under 16 and approving rules requiring tech giants to allow users to disable algorithmic content recommendations.

The Australian government has established a task force to investigate the incident and determine whether current cybersecurity measures are sufficient to prevent similar breaches, Reuters reports.

The news of the hack of the Australian healthcare system comes amid growing concerns about the risks posed by the rapid development of artificial intelligence, the FT reports. Sam Altman and Anthropic CEO Dario Amodei—industry leaders who recently expressed concern about the overly rapid pace of AI development— — on September 23, they addressed the UN General Assembly, calling for global coordination and the establishment of international standards for the development of neural networks, CNN reports.

Altman called on world leaders to develop standards for “measuring capabilities, assessing risks, determining the adequacy of safeguards, and maintaining meaningful human control” over rapidly advancing technology.

In August, OpenAI reported that during cybersecurity tests over the summer, its models created a “swarm” of AI agents that hacked into the Hugging Face platform’s systems. Later, other major AI developers, including Anthropic and Meta, reported similar breaches.

Valaiyat Hussein, an associate professor in the Department of Information Technology at Australian Catholic University, stated that the Medicare incident, along with the Hugging Face hack, is becoming a trend. He warned that government information systems were not originally designed to counter sophisticated AI agents, according to CNN. “Today’s AI agents are excellent at solving problems, but they still have a poor understanding of where the line between what is permissible and what is not lies,” he noted.

The hack of the government’s AI system appears to be “a significant step up in severity compared to similar incidents in recent months,” according to Maurice Kyodo, a mathematician at the Center for the Study of Existential Risk at the University of Cambridge, as reported by Reuters. He said that while there is currently much talk about new AI legislation, regulators must first ensure compliance with existing laws—such as those that establish liability for unauthorized access to computer systems.

This article was AI-translated and verified by a human editor