Nvidia has unveiled a system to prevent AI agents from getting out of control

Nvidia has unveiled the Open Agent Safety Platform, a two-tier AI security system that restricts agents' access to data in real time and shuts them down if they violate the rules. According to the company, this platform could have prevented the recent Hugging Face breach by OpenAI agents.

If leading AI labs had used Nvidia technology to evaluate their agents and AI models in the early stages, it could have prevented the attack on Hugging Face, said Justin Boytano, Nvidia’s vice president of corporate AI, during a briefing ahead of the release of the Open Agent Safety Platform, according to Bloomberg. “As far as we know, this new safety platform could have stopped the hack,” he said.

Boitano declined to comment on whether OpenAI or its main competitor—Anthropic—plans to use Nvidia's AI safety system to monitor its agents during testing.

Bloomberg notes that one of the platform’s components is the Nvidia OpenShell software product, which the company had already unveiled at its technology conference in March. It runs on Nvidia Vera central processing units and limits the capabilities of agents (developers can set real-time restrictions using new technology during testing). In addition, Nvidia introduced the Sentry tool, which monitors agent activity and runs not on central processing units (CPUs) or graphics processing units (GPUs), but on network chips, CNBC explains. The tool is designed to provide an additional layer of AI-powered monitoring that oversees the agents’ actions and intervenes to isolate those exhibiting suspicious behavior, according to the company’s press release.

Some of the software for Nvidia's new platform is distributed as open source, which means it can be freely used and adapted for various products, Bloomberg explains.

This article is being updated

This article was AI-translated and verified by a human editor