‘HalluSquatting’ Compromises AI Coding Agents to Install Malware, Create Botnets
Hallucinations have been an ongoing problem since OpenAI first introduced its ChatGPT chatbot in November 2022, highlighting generative AI’s tendency to generate plausible but false or misleading information and its inability to just say “I don’t know.”
They can lead to embarrassing or uncomfortable moments. However, security researchers in Israel have found that the large language model (LLMs) hallucinations can be weaponized by threat actors to compromise systems and create large botnets.
In a paper issued this week, researchers from Tel Aviv University, Technion, and Intuit outlined a technique they call “HalluSquatting,” or adversarial hallucination squatting, in which hackers exploit the predictable tendency of AI models to generate – hallucinate – incorrect or non-existent resource identifiers, which can include popular repositories, skills, and software packages.
The threat takes particular advantage of the growing use of AI agents by developers and their ability to not only suggest code but to independently perform tasks, execute commands, work within the system – from accessing files and retrieving data to writing code to reaching into the web – often with little or no human intervention.
Predicting Hallucinations
In the case of HalluSquatting, a developer makes a task for a coding agent – grabbing code from a repository or installing a software package. The agent may make up a name for the package that is close to what is asked for but is incorrect.
The researchers found that bad actors can reliably predict the hallucinated names agents likely will create, and then register those names and add malicious code to them. Then they wait. Once the AI coding agent retrieves the hallucinated resource, it pulls the malware into the developer’s system.
“By leveraging the predictability and transferability of hallucinations across foundational LLMs and application layers, adversaries can significantly amplify the reach of untargeted promptware under weak threat models and establish a botnet by exploiting LLM applications to install a bot on the device that ‘pulled’ the compromised hallucinated resource from the Internet,” the researchers wrote.
Attractive Targets
They added that targeting trending resources is an attractive target for hackers because they “ensure high request volume from users using their LLM applications. Moreover, popular/trending resources tend to be recently uploaded resources, which ensures a high hallucination rate by the LLM because they weren’t part of the training set used to train the LLM used by the LLM application.”
They tested the technique against a range of AI coding assistants and agents, including GitHub Copilot, Google’s Gemini CLI, OpenClaw, NanoClaw, Windsor, and Cursor. The AI models successfully hallucinated the false names of repositories 85% of the time, with the rate for skill installation hitting 100%.
The report’s authors noted that the high success rates aren’t etched in stone.
“Our findings are only a lower bound on what attackers could do,” they wrote. “Attacks always get better; they never get worse.”
Exploiting Coding Agent Behavior
According to the researchers, the growing adoption of agentic applications has brought with it the threat of promptware, an AI prompt injection attack in which hidden or pre-written prompts push rogue instructions into an AI assistant.
“While prior work has established that adversaries can exploit direct channels to LLM applications to apply promptware under weak threat models (e.g., by sending emails or calendar invitations to a target), many applications do not provide any direct channels that could be exploited for prompt injection beyond the Internet,” they wrote.
A Mix of Kill Chains
HalluSquatting changes the equation, allowing attackers to exploit AI model applications without direct channels. It combines what the researcher called the “promptware kill chain” and “traditional malware kill chain.” It starts with pulling adversarial prompts into an LLM application and ends by instructing the application to install a bot using remote code execution (RCE).
The beginning involves injecting an adversarial prompt that hijacks an LLM application and exploits the LLM to perform malicious acts by using the terminal and installing a bot. That’s in line with a promptware kill chain. That said, once the LLM application installs the bot on a device, the bot won’t exploit an AI model to perform malicious activities, which is where the more familiar malware kill chain starts, they wrote.
Manipulating AI Models
It also plays off of typosquatting, where threat actors register domain names that are close to legitimate domain names, but not exact, and wait for victims to click on the wrong domain. HalluSquatting essentially migrates the target from humans to AI agents, which may not confirm whether the source is real while carrying out commands.
It’s also another in a growing number of techniques that could be used by attackers to compromise and manipulate AI models. Indirect prompt injections are used to embed malicious instructions inside external content, such as websites, emails, and documents, that an agent may read as legitimate instructions to be executed.
