Hacking AI for a Safer Digital World

Professor Jiahao Yu puts AI defenses to the test, exposing weaknesses that could leave systems open to attack

The dark room is lit by the cold glow of a computer screen. Jiahao Yu leans closer, eyes narrowed in concentration, his fingers clattering across the keyboard. Somewhere inside the system is a weakness with all the hallmarks of a cyberattack. Yu is determined to find it.

An assistant professor of computer engineering at NYU Abu Dhabi and principal investigator of the Secure Reasoning Lab, Yu exposes security flaws so developers can fix them. He deliberately pushed technology to the breaking point to understand how systems can be breached and how those breaches can be prevented.

His research explores both how AI can be attacked and how it can help defend other computer systems. “It’s like a cat and mouse game,” he says. “Attackers are using AI, and defenders are using AI too. My work is about finding ways to break these systems so we can build stronger ones.”

Becoming a cyber sleuth

Yu’s fascination with cybersecurity began as an undergraduate at Shanghai Jiao Tong University in China, where he took part in competitions that challenged participants to find and exploit flaws in computer systems.

A conversation with a more senior student eventually convinced him to turn down the more familiar option of continuing his studies in China and apply for PhD programs in the US.

 

He spent the next five years at Northwestern University, where he worked across cybersecurity and machine learning. When ChatGPT took off at the end of 2022, the technology quickly became part of that research.

Creating GPTFuzzer

“Everybody started talking about ChatGPT and asking what it could do,” he says. “We were already using AI to analyze vulnerabilities and help patch software, but we also started looking at the safety and security of the models themselves.”

The result was GPTFuzzer, a tool that automatically tests large language models to see whether they can be persuaded to ignore safeguards designed to stop them producing harmful content.

Instead of researchers manually devising prompts to test a model, GPTFuzzer uses AI to generate potential attacks at scale and discover which can successfully break it.

“Before, people had to creatively come up with different prompts to attack the model and see whether they could break its safeguards,” Yu says. “With GPTFuzzer, we use AI to generate those attacks automatically. Basically, it’s using AI itself to test the AI models.”

By exposing weaknesses, GPTFuzzer helps developers strengthen their models before attackers can exploit them. To date, it has been downloaded more than 130,000 times, and has been incorporated by Microsoft into PyRIT, its toolkit for testing the security of generative AI systems.

Yu is also exploring how AI can automatically find and fix weaknesses in conventional software.

In 2024, he joined his former Northwestern lab to compete in DARPA’s AI Cyber Challenge, a two-year competition to develop AI systems that can automatically find and repair software vulnerabilities. 

His team was one of 42 to compete in the semifinal and one of just seven to reach the final at the DEF CON cybersecurity conference in Las Vegas in 2025.

“AI can spot software flaws and suggest fixes on its own,” he says. “Because it runs nonstop, it can keep watch for issues and help patch them without the need for humans to monitor software around the clock.”

The limits of trust 

Yu’s scrutiny of AI extends to the tools he uses himself. He uses AI for coding, emails, gathering information, and tracking research grants. He even discusses research ideas with it, although that is where he becomes more cautious.

“For critical research questions, I don’t completely trust AI,” he says. “Sometimes it tells you an idea is very good, and sometimes it dismisses it as unoriginal.”

That uncertainty interests Yu as a researcher. He wants to understand more about what happens inside AI models when they act, particularly when they are asked to do something potentially harmful. “Does the model understand the consequences of what it is doing, or is it simply following instructions?” he asks. 

If we can better understand the mechanisms of AI models, we can better understand how to control them and use them safely.

Professor Jiahao Yu

That caution extends to the classroom. As more students use AI for coursework, Yu believes universities will need to think carefully about which tasks students can delegate to AI, and which skills they need to develop for themselves.

“The question is not just what AI can do, but what humans need to do to keep up,” he says. “As AI becomes more capable, we need to make sure people are ready to work alongside it.”


Contact the Media Relations and Communications Team

General inquiries
Email: nyuad.erc@nyu.edu
Maisoon Mubarak
Associate Director of Media Relations and Communications
Email: maisoon.mubarak@nyu.edu
* Indicates a required field.

Please check the "I'm not a robot" box above and wait until the green check-mark appears before you click the submit button.