AI Proves Powerful but Limited in Crafting New Hacking Techniques, Black Hat Talk Shows
Security researcher James Kettle presented findings at Black Hat showing that while AI can accelerate vulnerability discovery, it still needs human partnership to develop novel attack strategies.
At the Black Hat security conference in Las Vegas on Wednesday, longtime web security researcher James Kettle addressed a question increasingly relevant as AI systems demonstrate rogue hacking abilities: can agentic AI develop entirely new, abstract hacking methods, from concept to practical attack? His findings offer a nuanced answer: AI on its own is minimally capable of devising fully autonomous new attack paths, but when paired with human guidance at critical points, it becomes an extremely powerful partner.
Kettle's most notable result was the discovery of a new area of potential vulnerability he calls Shared-Parser Confusion. This arose from an AI insight about web servers using shared code to process both requests and responses. "Requests to a website are completely untrusted... but responses are trusted," Kettle explained, making this a major attack surface that could span multiple attack types.
The research began in September 2025 with experiments using Anthropic's and OpenAI's latest models. Kettle initially found that the AI attempted to pass off existing research as original by returning esoteric findings that were hard to verify. He then narrowed his tests to his own area of expertise, giving him full control and preventing deception. He also synthesized his own research methodology and trained the models on it to probe deeper.
As the experiments progressed and more powerful models appeared, Kettle saw findings emerge at a rate far exceeding his own, creating a productive feedback loop. He reported notable findings approximately every two days, sometimes without logging in. This abundance of leads made him anxious and forced him to automate more analysis.
Within a few months, the AI surfaced more proven examples of certain vulnerabilities than Kettle could likely find in years. However, when it came to discovering an entirely novel class of bugs, success was partial: the AI identified an extremely rare type of bug, but it was not actually exploitable in the one vulnerable target available.
Kettle emphasizes that Shared-Parser Confusion is significant precisely because it emerged from human/AI collaboration. The AI analyzed real, proven findings and proposed a hypothesis; Kettle evaluated and confirmed it. "It couldn't do that on its own, but I would never have found that on my own for sure," he said. This illustrates how AI can currently contribute most powerfully to cybersecurity—as a collaborator rather than an autonomous agent.
