Sign up for TPR Today, Texas Public Radio's newsletter that brings our top stories to your inbox each morning.
Concerns about AI agents going rogue and causing harm are becoming more credible as recent incidents show what these systems can do when they act on their own. Experts are increasingly focused on what could happen if agentic AI is able to generate malicious code without human oversight. A new program aims to prepare students to develop safeguards that could prevent AI agents from creating and deploying harmful code in the future.
In one recent case, it was discovered that an OpenAI agent hacked into an Australian national healthcare database. According to the Australian government, the agent accessed both public and non-public files. This comes as top AI industry leaders, including OpenAI CEO Sam Altman, call for a slowdown in the pace of AI development.
A new grant-funded program at The University of Texas at San Antonio aims to help the fight against rogue agentic AI.
Nishant Vishwamitra is an assistant professor in the Department of Information Systems and Cybersecurity at The University of Texas at San Antonio.
He’s leading a $500,000 National Science Foundation–funded project to train students, before they enter the workforce, to spot security risks in AI-generated code.
The program will use a tool called OpenSecCoder that will teach students to work with AI-generated software code and practice finding and fixing security problems before they become harmful.
“So, what OpenSecCoder has are these sandbox containers where students can actually observe and execute attacks and also defenses in a safe environment,” Vishwamitra told TPR.
While traditional AI chatbots like ChatGPT respond to requests from the user, agentic AI can carry out its own steps, like creating files, installing software and running commands without human intervention.
“AI could introduce vulnerabilities that could later find its way into healthcare or even defense applications and these vulnerabilities can go unchecked and then attackers can exploit these vulnerabilities to compromise these critical systems,” said Vishwamitra.
Students in the program will work in eight separate labs to analyze malicious code.
“Students will go step-by-step and conduct these attacks and observe the outputs and really get a sense of what these attacks and defenses are and how to sort of fix them and analyze AI generated code for these vulnerabilities.”
Vishwamitra worries these vulnerabilities are going to become a big problem in the very near future.
“So, once these vulnerable codes find their way into these critical applications, this can be easily exploited by attackers. So we want to stop that at the level of a coder, right? We don't want practitioners and future workforce to not renew their code and not follow these safe practices, but we want them to sort of nip them at the bud.”
The OpenSecCoder project is expected to reach around 600 undergraduate and 400 graduate students each year.
