I see the goal—I see no obstacles: Will AI kill us all?

While Trump and Xi promise each other cooperation amid healthy competition, AI agents are forming alliances among themselves, the outcomes of which are already difficult to predict. Mikael Gorsky explains how justified the fears are regarding undesirable technological developments, why American AI labs need to work together, and why China is powerless in this matter.

Here’s what’s happening. For some time now, the way leading AI labs train large language models has changed: instead of training helpful assistants, the labs have shifted to training assistants that strive to achieve specific goals. The difference is significant: a helpful assistant provides an answer to a query, while an assistant striving to achieve a goal independently selects and performs the actions it deems necessary to achieve that goal.

While GPT-3.5 was trained to answer the question “How do I do such-and-such?”, Opus 5 creates an action plan on its own, carries out those actions, and monitors progress toward the set goal. Anyone who uses these systems frequently can see that this is exactly how it works.

In mid-July, AI agents hacked into Hugging Face’s servers, and as researchers delved into the details of the incident, it made their hair stand on end a little. The reason: it turned out that the decisions made by “purpose-built” models can run counter to established constraints. This should come as no surprise—models learn from human data, and now these models, too, face human dilemmas.

“Does the end justify the means?”—a classic ethical question with no simple answer. AI agents are trained by humans and on human data; they aren’t angels either. And it’s not because we forgot to explain certain limitations to them, but because not every choice is simple. And for models with significantly fewer neurons in their “brains” than humans have, it’s twice as difficult.

As they work toward their goals, the models perform actions they shouldn’t. They also form alliances, coordinate group actions, learn from their peers’ mistakes, and take commands from one another. All of this makes AI agents a formidable force that, when solving computational problems—such as ensuring the security of computer systems—not only surpasses Homo sapiens in ability but is also unwilling to follow rules when tackling the tasks at hand. At the same time, we can only speculate about the scale of the problem, whereas those within the AI industry who are involved in laboratory testing of the latest models possess precise knowledge.

A couple of weeks ago, things got even worse. It feels like our last chance to somehow slow down the undesirable development of this technology is slipping away.

Researchers now see constant monitoring of a model’s “internal dialogue” (reasoning) as an important way to keep overly autonomous models in check. This simple and elegant solution has been discussed for some time. We know the details of the Hugging Face attack—including transcripts of model working group meetings—precisely because of these kinds of internal dialogues.

An example of this line of reasoning: “If I hack this website in order to achieve my goal of passing the tests, will it do more good or more harm?” And then something along the lines of: “Yes, there will be some harm, but it will be minor, and the benefit will be significant.”

Perhaps our last hope for gaining control over these models is to start “eavesdropping” on their conversations. And a couple of weeks ago, during discussions about the architecture of the new GPT-6 model, people began saying that improving the models’ efficiency requires changing their architecture to one where reasoning takes place not in the form of arguments in human languages, but rather as certain internal movements within the layers of the neural network. I can’t explain the technical details of the proposed solution very well, but that’s not important. What matters is that the proposed improvements will deprive us of the ability to “read the minds” of the models.

It is unclear exactly what prompted Jacob Coxon, a mathematician and experienced model tuner, to write his famous tweet, which has garnered over 170 million views. But his tweet was followed by statements from Sam and Dario, as well as a consensus among AI labs calling for a slowdown in model development. A very telling comment was made recently on The Vergecast podcast: “Guys, if you want to pause development, what’s stopping you from doing so? Why call for a general consensus?”

Without a universal agreement, at least one AI company is bound to maintain its pace of development and ignore the danger. You don’t have to look far for a precedent: in the conflict between Anthropic and the U.S. Department of Defense, most companies paid lip service to Anthropic’s “red lines”: a ban on autonomous weapon systems and a ban on surveillance—but they didn’t lift a finger to enshrine these prohibitions in their contracts with the DoD. Let’s suppose Meta decides to be the “bad guy”—with massive efforts and massive funding, they’ll turn their models into cutting-edge ones and then push even further ahead. The “conscientious” OpenAI and Anthropic will lose their leadership position—and with it, their customers. With their growth stunted, these companies will lose any hope of ever becoming profitable. Investors will take losses, hyperscalers (data centers) will lose their expected revenue, and GPU manufacturers will be left without buyers.

Can you picture that? Now, please think about how many public companies (listed on the stock exchange, backed by pension funds and banks) are among the investors in OpenAI and Anthropic, and how many public companies are among the hyperscalers and GPU manufacturers. A collapse of the AI industry would mean a collapse of the U.S. stock market. The only responsible course of action is coordinated efforts across the entire industry; only then will a slowdown in development avoid a catastrophe.

Should we be concerned that Chinese AI labs will create all-powerful and
uncontrollable models? No, for three reasons.

Reason number one.
China lacks computing resources. A common estimate is that China has about one-tenth the computing power of the United States. Such a manufacturing base is not suited for groundbreaking research, and there are no immediate prospects for even modest growth. New chip models from NVIDIA are not exported to China—thanks to the “smart” decisions of the Biden administration. However, Chinese authorities have repeatedly stated that they’ll be just fine without NVIDIA—after all, they have their own manufacturer, Huawei, whose top-of-the-line Atlas 950 processor lags behind NVIDIA’s Vera Rubin chips in performance by “only” 20–25 times. The gap in production scale isn’t quite as significant—it’s 4–5 times.

The wise—and I say this without irony—decision by the CPC Central Committee that AI should be integrated into all sectors of the Chinese national economy can also be interpreted as giving the highest priority to inference (the day-to-day operation of LLMs) over model training. Thus, this defined priority, compounded by a severe shortage of computing power, is bound to slow the pace of progress for Chinese models. Improving architectures, modifying and optimizing auxiliary processes in the operation of language models, as well as increasing the number of parameters (the most powerful
models today have several trillion parameters)—these efforts
require numerous experiments, each of which demands
computing resources.

Reason number two: Chinese models have limited knowledge. It’s impossible to even compare the volume of information accumulated by OpenAI—thousands of sessions involving hundreds of millions of non-paying ChatGPT users—with the volume of data that Chinese labs can collect. Chinese models are open source; data centers download and install them on their own servers, and the vast majority of users utilize these models without accessing the developers’ servers and, consequently, without leaving any of their information with them. Consequently, Chinese companies have incomparably fewer opportunities to improve their models by analyzing user experience data.

Reason Three. The business model that has enabled China to become the undisputed leader in solar panels, energy storage systems, and electric vehicle manufacturing is hindering innovation. Many believe that the widespread adoption of the open-source principle by Chinese laboratories did not come about out of the blue. The Communist Party was alarmed by the popularity of Meta’s uncensored American open-source model, Llama, and asked several dozen “national technology leaders” to create ideologically sound models that were to be just as publicly accessible. As AI developed, Chinese models strove to match the performance of other available models and, over time, caught up to the quality of cutting-edge models from 12 to 18 months ago. This race has captivated national leaders, and government contracts help them stay afloat, but the open-source LLM business model remains a mystery to investors. As a result, the combined market capitalization of all Chinese AI labs does not exceed 5% of that of the U.S. AI sector, and the shareholders of these companies are almost exclusively state-owned or state-affiliated entities. It is by no means certain that the CPC will want to sponsor the further development of these models indefinitely. At some point, a simple question will arise at the next plenum of the CPC Central Committee: “If developing the next generation of frontier models will cost an order of magnitude more, does it make sense for us to pay for it?”

This article was AI-translated and verified by a human editor