Nvidia has introduced a system to prevent AI agents from getting out of control

Nvidia has unveiled the Open Agent Safety Platform, a two-tier AI security system that restricts agents' access to data in real time and shuts them down if they violate the rules. According to the company, this platform could have prevented the recent hack of Hugging Face by OpenAI agents.

If leading AI labs had used Nvidia technology to evaluate their agents and AI models in the early stages, it could have prevented the attack on Hugging Face, said Justin Boitano, Nvidia’s vice president of corporate AI, during a pre-release briefing, according to Bloomberg. “To the best of our knowledge, this new security platform could have stopped the hack,” he said.

Boitano declined to comment on whether OpenAI or its main competitor—Anthropic—plans to use Nvidia’s AI safety system to monitor its agents during testing.

One of the platform’s components is the Nvidia OpenShell software product, which the company had already unveiled at its technology conference in March, Bloomberg notes. It runs on Nvidia Vera CPUs and limits the capabilities of agents (developers can set real-time restrictions using new technology during testing). In addition, Nvidia introduced the Sentry tool, which monitors agent activity and runs not on CPUs or GPUs, but on network chips, explains CNBC. The tool is designed to provide an additional layer of AI-powered monitoring that oversees the agents’ actions and intervenes to isolate those behaving suspiciously, according to the company’s press release.

Some of the software for Nvidia's new security platform is distributed as open source, which means it can be freely used and adapted for various products, Bloomberg explains.

This article is being updated

This article was AI-translated and verified by a human editor