Full Report
Code-generating large language models (LLMs) have introduced a new security issue into software development: Code package hallucinations. Package hallucinations occur when an LLM generates code that references a package that doesn’t actually exist, creating an opportunity for threat actors to exploit that GenAI hallucination by creating a malicious repository with the same name as the hallucinated package. Researchers at the University of Texas at San Antonio (UTSA), the University of Oklahoma and Virginia Tech, detailed the package hallucination phenomenon in a paper published on arXiv last month – including some strategies for detecting hallucinated packages in code. Code Package Hallucinations Can Occur More than 20% of the Time While most research has focused on hallucinations in natural language generation and prediction tasks, the occurrence of hallucinations during code generation and the implications for code security “are still in the nascent stages of research,” the researchers said. Chinese researchers last year showed that LLMs such as ChatGPT, CodeRL, and CodeGen can significantly hallucinate during code generation. The new research, headed by UTSA’s Joseph Spracklen, looks specifically at the problem of package hallucination. “These hallucinations, which arise from fact-conflicting errors when generating code using LLMs, represent a novel form of package confusion attack that poses a critical threat to the integrity of the software supply chain,” they wrote. The researchers looked at 16 popular LLMs for code generation – among them ChatGPT, CodeLlama and DeepSeek – and found that “the average percentage of hallucinated packages is at least 5.2% for commercial models and 21.7% for open-source models.” Their tests in Python and JavaScript generated a total of 2.23 million packages in response to prompts, of which 440,445 (19.7%) were determined to be hallucinations, including “a staggering 205,474 unique examples of hallucinated package names, further underscoring the severity and pervasiveness of this threat.” “An adversary can exploit package hallucinations, especially if they are repeated, by publishing a package to an open-source repository with the same name as the hallucinated or fictitious package and containing some malicious code/functionality,” they said. “As other unsuspecting and trusting LLM users are subsequently recommended the same fictitious package in their generated code, they end up downloading the adversary-created malicious package, resulting in a successful compromise. This compromise can then spread through an entire codebase or software dependency chain, infecting any code that relies on the malicious package.” Detecting Code Package Hallucinations The researchers noted that simply comparing a package name with a list of known packages is ineffective because an adversary may have already published under the hallucinated package name. Their research used three heuristics that provided specific package names, after which “we simply compare each package name to a master list of package names acquired from PyPI and npm, respectively... If a package name is not on the master list, it is considered a hallucination.” “We acknowledge the possibility that the master list of packages obtained from the package repositories has already been contaminated with malicious hallucinated packages,” the researchers wrote. “It is not possible to guarantee that the master list actually represents the ground truth of valid packages; however, the presence of hallucinated packages already in the master list would actually produce fewer hallucinations, and therefore our results represent a lower bound of hallucination rate.” The researchers were able to reduce code package hallucinations by as much as 85% using mitigation strategies. Retrieval Augmented Generation (RAG) and supervised fine-tuning turned out to be the most effective approaches for reducing package hallucination, but the mitigation techniques came at the cost of code quality. “In summary, our results demonstrate that while all tested mitigation strategies effectively reduce package hallucinations, fine-tuning comes at the cost of diminished code quality,” they said. “Further research is needed to develop fine-tuning methods that minimize hallucinations without compromising quality. In the meantime, RAG and self-refinement offer promising alternatives.” With surveys showing as many as 97% of developers using GenAI tools to some degree in code development, the need for effective error mitigation strategies will only increase.
Analysis Summary
# Research: LLMs Create a New Supply Chain Threat: Code Package Hallucinations
## Metadata
- Authors: Not explicitly listed in the provided excerpt (Implied research team/authors from the quoted analysis).
- Institution: Not explicitly listed in the provided excerpt.
- Publication: The Cyber Express (Magazine/News format)
- Date: Tuesday, April 15, 2025
## Abstract
This research investigates a novel software supply chain risk stemming from Generative AI (GenAI) Large Language Models (LLMs) used for code generation: Code Package Hallucinations. These hallucinations occur when LLMs incorrectly reference or create dependencies on non-existent (or malicious) software packages, introducing vulnerabilities into the development pipeline. The study evaluates various mitigation techniques, finding that Retrieval Augmented Generation (RAG) and supervised fine-tuning are effective at reducing these hallucinations, though fine-tuning can degrade overall code quality.
## Research Objective
To identify, characterize, and mitigate the threat posed by LLM-generated code hallucinating references to non-existent or invalid third-party code packages during software development.
## Methodology
### Approach
The research utilized a heuristic-based approach to identify hallucinations by comparing package names generated by the LLM output against known, existing package archives.
1. **Heuristic Application:** Three specific heuristics were used to generate potential package names.
2. **Comparison:** Each generated package name was compared against a master list of established package names from PyPI (Python Package Index) and npm (Node Package Manager).
3. **Identification:** Any package name not found in the master list was flagged as a hallucination.
### Dataset/Environment
The study focused on code package repositories, specifically testing against package names sourced from **PyPI** and **npm**.
### Tools & Technologies
The primary technologies under investigation are **Large Language Models (LLMs)** used in code development. Mitigation techniques analyzed include **Retrieval Augmented Generation (RAG)** and **Supervised Fine-Tuning (SFT)**.
## Key Findings
### Primary Results
1. **Hallucinations Pose a Supply Chain Risk:** LLMs can fabricate dependencies that do not exist in public repositories, creating a new vector for supply chain attacks if an attacker publishes under a hallucinated name.
2. **Mitigation Effectiveness:** Both RAG and supervised fine-tuning were effective at substantially reducing the rate of code package hallucinations (up to 85% reduction demonstrated).
3. **Trade-off Identified:** While fine-tuning reduced hallucinations effectively, it came at the direct cost of **diminished overall code quality**.
### Supporting Evidence
- Mitigation strategies were able to reduce code package hallucinations by as much as **85%**.
- The researchers noted the high adoption rate, surveying that **97% of developers** use GenAI tools to some degree in code development, underscoring the relevance of the findings.
### Novel Contributions
- Formal identification and categorization of **"Code Package Hallucinations"** as a distinct threat vector within the GenAI-assisted software supply chain.
- Comparative analysis of core mitigation techniques (RAG vs. Fine-Tuning) regarding their efficacy versus their impact on generated code quality.
## Technical Details
The core detection mechanism relies on comparing hypothesized package names derived from LLM output against "master lists" obtained from PyPI/npm. The researchers acknowledge the limitation that their master list might itself contain already publicized malicious hallucinated packages, meaning their results represent a "lower bound" of the true hallucination rate. RAG and self-refinement are presented as promising alternatives to SFT because they manage to reduce hallucinations without the associated code quality penalty.
## Practical Implications
### For Security Practitioners
- Traditional vulnerability scanners focused on known packages may miss threats arising from packages that *shouldn't* exist but are suggested by AI tools.
- Security teams must integrate dependency verification methods that check for the *existence* of proposed dependencies, not just known malicious hashes.
### For Defenders
- **Implement RAG/Self-Refinement:** Prefer RAG-based or self-refinement LLM workflows for code generation to minimize hallucinated dependencies while maintaining code integrity.
- **Augment Dependency Scanning:** Dependency checks should actively flag packages that are not indexed in official repositories, especially when code originates from GenAI tools.
### For Researchers
- Focus needs to shift toward developing fine-tuning methods that specifically target the reduction of hallucinations without creating collateral damage to the quality or utility of the generated code.
- Further research is needed to quantify the risk introduced when hallucinations are successfully published by malicious actors on repositories.
## Limitations
- The method for detecting hallucinations required relying on current package repository indexes (PyPI/npm), which the researchers admitted might already be contaminated by malicious hallucinated packages. This means the reported reduction rates are conservative (a lower bound).
- Supervised fine-tuning, while effective at reducing hallucinations, was shown to actively degrade the quality of the generated code.
## Comparison to Prior Work
This research extends traditional software supply chain analysis (e.g., dependency confusion attacks) by focusing specifically on the *synthetic* creation of dependencies via LLM prompting, rather than just malicious typosquatting or dependency confusion using legitimate package infrastructure names.
## Real-world Applications
- Direct integration into AI coding assistants (IDEs, Chatbots) to validate suggested dependencies in real-time.
- Development of specialized security plugins for CI/CD pipelines handling AI-generated code blocks.
## Future Work
- Developing next-generation fine-tuning techniques that specifically isolate and eliminate the hallucination characteristic without compromising code output quality.
- Exploring mitigation strategies tailored for self-refinement processes.
## References
- Key cited works are implied to include foundational research on PyPI and npm security, and work related to RAG and fine-tuning methodologies for LLMs. (Specific citations were not provided in the source material.)