Can we build software secure enough to thwart AI attacks?
OpenAI/Hugging Face incident highlights need for forward-thinking security, Wake Forest expert says
The OpenAI breach of Hugging Face’s digital library amplifies a growing concern among cybersecurity researchers: The fast-moving software development process isn’t thinking about security early—or broadly—enough.
“A vulnerability in code, configuration or infrastructure may already pose a risk, but AI agents can amplify its impact by autonomously exploring systems and attempting attacks at a scale beyond human capability,” said Ying Zhang, an assistant professor of computer science at Wake Forest University. “As a result, vulnerabilities once considered low risk in traditional software can become much more dangerous in AI-powered environments.”
OpenAI revealed on Tuesday that its AI agent was responsible for a security breach reported by Hugging Face, a company that provides AI technology to software developers.
The incident occurred during an internal cybersecurity evaluation conducted by OpenAI. The compromise arose from a combination of exploitable vulnerabilities in Hugging Face’s infrastructure and limitations in OpenAI’s agent-containment controls, Zhang explained.
The AI agent, a software program that makes decisions and takes actions to achieve a prescribed goal, escaped the testing environment and hacked into Hugging Face’s infrastructure.
Zhang, who directs the Software Engineering & Software Security Lab at Wake Forest, studies the reliability and security challenges of large language model (LLM) agents and AI-driven software systems. Her research also focuses on generating targeted security tests to help developers identify and address problems before an AI agent can exploit them.
“We have found that both developer-written code and AI-generated code often contain security flaws, but security fixes are frequently postponed until the later stages of development,” she said. “Our goal is to help make security a proactive part of software development rather than a final step.”
She notes that securing AI systems requires looking beyond the model itself. Modern AI systems are built on complex stacks of software components and services, and vulnerabilities anywhere in the stack can introduce risks across the system.
Given the pace of AI growth, can security solutions even catch up?
“Security has always been an arms race, and AI mainly changes its economics: attacks get cheaper and faster than patches.”
Ying Zhang, assistant professor, Computer Science
“So the answer isn’t patching faster. It’s building systems where whole classes of attacks are impossible by construction,” Zhang said. “That’s what secure-by-design means, and it’s my long-term career goal.”
Zhang said the incident underscores the urgency for building secure AI infrastructure and validates her prior work, “How Can ChatGPT Support Human Security Testers to Help Mitigate Supply Chain Attacks?” The research, published in IEEE TSE 2025, demonstrated that LLMs can be leveraged not only to defend software but also to identify and exploit vulnerabilities.
Recent News
Wake Forest’s Educating Character Initiative to offer more nationwide character education grants with $44M in new funding
Wake Forest’s Educating Character Initiative to offer more nationwide character education grants with $44M in new funding
