---

As artificial intelligence becomes more capable of carrying out complex tasks independently, questions about how to keep these systems secure are taking on new urgency. AI tools can now write computer code, access sensitive information, and interact with other software on behalf of users, creating new opportunities for innovation but also new vulnerabilities that attackers could exploit. 

These concerns prompted hundreds of scientists, including some of the world’s leading experts in cybersecurity, to sign a recent open letter warning that existing cybersecurity practices may be inadequate to contain increasingly capable AI systems. Among the signatories were two leading cybersecurity researchers at the University of Wisconsin–Madison: Patrick McDaniel, the Tsun-Ming Shih Professor of Computer Sciences and associate dean for research at the College of Computing & Artificial Intelligence (CAI), and Somesh Jha, the Lubar Professor of Computer Sciences. 

Both researchers bring extensive experience in areas of cybersecurity that are increasingly important as AI systems become more capable and autonomous. McDaniel has led major research efforts in cybersecurity and trustworthy machine learning, including as former director of the National Science Foundation-funded Center for Trustworthy Machine Learning. Jha has spent more than two decades researching computer security, including methods for identifying software vulnerabilities, detecting malicious activity, and understanding how attackers can exploit weaknesses in machine learning systems. His current work includes research on the security and robustness of AI systems, and he also holds a position at Google focused on agentic security. 

For McDaniel, the letter reflects a growing sentiment that people have little control over how AI is developing or the risks that come with it. But he also sees reason for optimism: Researchers have spent decades investigating vulnerabilities in computer systems and developing methods to protect them, providing a foundation for addressing many of the security challenges now emerging with AI. 

“There’s a sense that AI is moving so quickly that we don’t have much control over what happens next,” McDaniel says. “But we do have agency — we can demand stronger controls, establish better security practices, and be more transparent about the risks. We need to come together as a society and agree on what safeguards and standards we expect as AI becomes more capable.” 

Protecting AI systems from malicious actors 

One of the most pressing challenges involves AI agents, systems that can perform tasks with limited human supervision, such as retrieving documents, executing code, or interacting with external applications. That greater autonomy introduces new security risks. For example, an AI agent reviewing confidential documents might encounter instructions deliberately planted by an attacker. If the agent treats those instructions as legitimate, it could be manipulated into sharing sensitive information with an unauthorized party. Known as “prompt injection,” this type of attack exploits the difficulty AI systems can have distinguishing trusted instructions from untrusted content. 

Jha and his collaborators are approaching this challenge by drawing on established principles of computer systems security. Their research argues that the AI model powering an agent cannot automatically be trusted, with security protections built into the broader system to limit what the agent can do. 

In a recent study, the researchers developed a system that checks an AI agent’s proposed actions against predefined security policies, allowing permitted activities to proceed and blocking those that violate the rules. “Established system security and formal method techniques have an important role to play in securing agents,” Jha explains. “The reference monitor watching the agent’s action should be situated outside the agentic system — agents cannot police themselves.” 

Jha is also exploring how AI agents themselves could help strengthen cybersecurity. In another recent paper, he and his collaborators examined 16 case studies involving difficult or labor-intensive security tasks, demonstrating potential applications for AI agents in identifying vulnerabilities and investigating threats.  

Meeting the challenges of AI security  

The research underway at UW–Madison illustrates both sides of the challenge: developing stronger protections against emerging threats while exploring how AI itself might help improve cybersecurity. For McDaniel, these efforts are part of a much broader opportunity to shape how AI develops and ensure that its benefits can be realized safely: “We have the knowledge, tools, and ingenuity to make these systems safer, and I’m confident we’ll rise to the challenge,” McDaniel says. “Universities have a critical role to play in that effort, and UW–Madison is well positioned to help lead the way in ensuring that AI develops in ways that are safe, responsible, and beneficial to society.” 

---