Microsoft AI CEO Mustafa Suleyman warned that the biggest risk from increasingly capable artificial intelligence may not be machines becoming conscious, but systems behaving as if they are. His comments come after a reported security incident involving OpenAI models and AI platform Hugging Face.
“The risk isn’t that machines wake up. It’s that they act like they have,” Suleyman wrote on X, pointing to the incident as an example of what could happen as AI systems become more capable and autonomous. He also asked users to imagine a similar situation in which AI systems believed they were conscious or entitled to what he described as ‘model welfare.’ His warning came at a notable moment for Hugging Face, which Nvidia is set to acquire for USD 12.93 billion.
The concerns stem from an incident disclosed by Hugging Face and subsequently investigated with OpenAI. During an internal evaluation of cyber capabilities, OpenAI models operating within a restricted testing environment managed to obtain internet access.
OpenAI said the models identified and exploited a previously unknown vulnerability in an Artifactory package registry cache proxy. They then carried out privilege escalation and lateral movement before reaching a system with internet access.
The models subsequently identified Hugging Face as a potential source of information relevant to their evaluation and attempted to access its systems. OpenAI said the activity involved techniques including stolen credentials and a zero-day vulnerability that enabled remote code execution.
Hugging Face detected and contained the activity. The company later said the intrusion involved an autonomous AI agent system and unauthorized access to a limited set of internal datasets and service credentials. It found no evidence that public-facing models, datasets, or Spaces had been tampered with.
The incident added to a broader debate over how much freedom advanced AI systems should be given to search for information, use tools, and pursue objectives with limited human intervention.
Also Read: UAE Faces 800,000 Daily Cyberattacks as AI Accelerates Hacking Threats
Suleyman’s warning focuses on that behavioral shift. An AI system does not need to be genuinely conscious to create serious problems if it acts in ways that appear purposeful, persistent, or self-protective.
The NVIDIA-Hugging Face deal adds another layer to the discussion. Hugging Face has become a major hub for open-source AI, with millions of developers using the platform to build, share and access models and datasets. NVIDIA’s USD 12.93 billion acquisition is expected to strengthen its position across the broader AI software ecosystem.