LLMs Hallucinate Packages. Attackers Notice.

What happens when an AI coding assistant recommends a package that doesn’t exist? Usually nothing — you get an install error, shrug, and search for the real one. But what if someone anticipated that hallucination and registered the package first, loaded with malicious code?

This isn’t hypothetical. It’s happening now, and the attack surface scales with every developer who copies npm install commands from ChatGPT without checking whether the package is legitimate.

Background

Supply chain attacks target the software dependencies that developers pull into their projects rather than attacking the application code directly. The classic vector is typosquatting — registering packages with names similar to popular libraries (requets instead of requests, lod-ash instead of lodash) and waiting for someone to mistype an install command.

LLMs introduced a new variant. Instead of relying on human typos, attackers can exploit the fact that language models confidently recommend packages that were never published. The model didn’t mistype anything — it invented a plausible-sounding name from whole cloth, and it will recommend that same fictional package to thousands of developers asking similar questions.

Why LLMs Hallucinate Package Names

Language models generate text by predicting the most likely next token given context. When asked “what npm package should I use for X?”, the model draws on patterns in its training data — real package names, naming conventions, common prefixes and suffixes. Sometimes it reconstructs a real package. Sometimes it constructs something that sounds right but was never published.

The problem is structural, not incidental. LLMs don’t query package registries at inference time. They don’t verify that python-flask-auth or node-html-sanitizer actually exist on PyPI or npm. They generate names that pattern-match against what packages tend to be called, and that’s enough to sound authoritative.

This creates a predictable attack surface. If you can determine which fake packages an LLM is likely to recommend, you can register those names before anyone else does.

The Attack Chain

The full attack plays out in five steps:

1. Harvest hallucinated names. An attacker queries multiple LLMs with common programming questions — “best Python library for PDF parsing,” “npm package for WebSocket authentication,” “Go module for rate limiting.” They collect every package recommendation that doesn’t map to a real registry entry.

2. Filter for consistency. Not all hallucinations are useful. The attacker keeps names that appear repeatedly across sessions and models. A package name that GPT-4, Claude, and Gemini all independently invent is far more valuable than a one-off hallucination, because multiple models recommending the same name means more developers will encounter it.

3. Register the packages. The attacker publishes packages under the hallucinated names on npm, PyPI, or other registries. The packages may contain a thin veneer of legitimate functionality — just enough to not look obviously malicious on casual inspection.

4. Embed the payload. The malicious package includes code that runs during installation (postinstall scripts on npm, setup.py execution on PyPI) or at import time. Common payloads include credential theft, reverse shells, cryptominers, or data exfiltration. Modern attacks often use obfuscation or delayed execution to evade automated scanning.

5. Wait. Every developer who asks an LLM the same question and runs the suggested install command becomes a victim. The attacker doesn’t need to phish anyone or compromise any infrastructure. The LLM does the social engineering for them.

What Makes This Different from Traditional Typosquatting

Traditional typosquatting relies on human error — fat fingers, misremembered names. The hit rate is low because most developers type carefully, and package managers increasingly warn about suspiciously similar names.

LLM-assisted attacks are different in three critical ways:

Confidence. The model doesn’t hedge. It doesn’t say “there might be a package called…” It states the name as fact, often with usage examples, API documentation, and version numbers — all fabricated. A developer seeing a confident code example with imports and function calls has little reason to suspect the package isn’t real.

Consistency. The same hallucination often appears across sessions and users. This isn’t random noise — it’s a systematic pattern in the model’s learned distribution. If the model learned that packages for X tend to be named x-utils or python-x, it will reliably generate those patterns.

Scale. Every developer using an AI coding tool is a potential target. The attack surface isn’t limited to people who mistype — it includes anyone who trusts an LLM recommendation without verification. As AI coding assistants become standard tooling, that’s a growing majority of developers.

How Often Do LLMs Hallucinate Packages?

Several research teams have quantified this problem.

In early 2023, researchers at Vulcan Cyber (led by Bar Lanyado) conducted one of the first systematic studies. They posed common programming questions to ChatGPT and tracked how often the responses recommended packages that didn’t exist on npm or PyPI. Their findings were striking: across hundreds of queries, ChatGPT recommended non-existent packages in a significant minority of responses. More concerning, many hallucinated names appeared consistently across separate sessions, making them reliable targets for pre-registration¹.

A larger-scale study published in 2024 by researchers at the University of Texas at San Antonio and others expanded on this work. They queried multiple models (GPT-3.5, GPT-4, CodeLlama, and others) with 30,000+ prompts across Python and JavaScript ecosystems. Key findings included:

• Roughly 20% of recommended packages were hallucinated (never published on the respective registry)
• 43% of hallucinated names appeared repeatedly across multiple queries
• Commercial models hallucinated less frequently than open-source alternatives, but none were immune
• GPT-4 showed lower hallucination rates than GPT-3.5, but still produced consistent phantom packages²

A follow-up study by Socket.dev and academic collaborators specifically tested the registerability of hallucinated packages — whether attackers could actually claim these names. Over 50% of the hallucinated npm package names were available for registration, and the researchers successfully registered several as proof of concept (with empty, non-malicious payloads) to demonstrate the feasibility³.

The Repeatability Problem

The most dangerous aspect isn’t the raw hallucination rate — it’s the repeatability. When researchers queried the same model with the same question across different sessions, certain hallucinated names appeared with remarkable consistency. This means an attacker doesn’t need to guess which names to register. They can systematically mine the models and get reliable predictions.

Some researchers found that certain hallucinated package names appeared in over 75% of repeated queries for the same programming task. That’s not noise. That’s a pattern stable enough to build an attack campaign around.

Real-World Incidents

This attack vector has already been exploited in the wild.

The python-binance-sdk Case

In 2023, security researchers at Checkmarx identified malicious packages on PyPI that corresponded to names frequently hallucinated by ChatGPT. The packages had been registered within days of the Vulcan Cyber research being published — suggesting that attackers were monitoring the research or independently discovering the same technique. The malicious packages contained credential-stealing code targeting AWS keys and environment variables⁴.

npm Ecosystem Attacks

The npm registry has seen multiple instances of packages registered under AI-hallucinated names. The node-hide-console-windows package — a name that ChatGPT occasionally suggests for console manipulation in Node.js — was found to contain a cryptominer. The package had accumulated over a hundred downloads before being removed, likely from developers following AI recommendations⁵.

The Hugging Face Vector

Beyond traditional package registries, LLMs also hallucinate model names, dataset names, and API endpoints for platforms like Hugging Face. Researchers at Lasso Security demonstrated that attackers could register Hugging Face organizations and model repos matching hallucinated names, potentially distributing backdoored models through the same mechanism⁶.

Detection and Prevention

The good news: this attack vector is well-understood, and multiple layers of defense exist.

For Individual Developers

Verify before you install. Before running pip install or npm install on an AI-recommended package, check the registry directly. Look for:

• Does the package exist on npmjs.com or pypi.org?
• How many downloads does it have?
• When was it first published?
• Who is the author?
• Does it have a linked source repository?

A package with 12 downloads, published last week, by a user with no other packages, recommended by your AI assistant for a common task? That’s a red flag.

# npm: check if package exists and inspect metadata
npm view <package-name> --json 2>/dev/null | jq '{name, version, description, author, homepage, time: .time.created}'

# PyPI: check package metadata
pip index versions <package-name> 2>/dev/null || echo "Package not found on PyPI"

# Or just search the registry website directly

Use lockfiles and review diffs. When package-lock.json or poetry.lock changes, review what was added. New dependencies should be expected and intentional, not surprises from an AI suggestion you didn’t scrutinize.

Pin dependencies to exact versions. Especially for packages you haven’t audited thoroughly. This limits damage if a package is later compromised via a malicious update.

For Teams and Organizations

Allowlist approved packages. Maintain a list of vetted dependencies. New additions require review. Tools like Socket.dev, Snyk, and npm audit can automate parts of this, but the allowlist is the foundation.

Monitor for hallucination patterns. If your team uses AI coding assistants heavily, periodically audit the packages being added to your projects. Are any of them newly published, low-download packages that match common AI hallucination patterns?

Private registry proxies. Use a registry proxy (Artifactory, Verdaccio, GitHub Packages) that only allows pre-approved packages. AI-recommended packages that aren’t in the proxy simply won’t install.

For the Ecosystem

Registry-level defenses. npm and PyPI both have mechanisms for reporting malicious packages, but they’re reactive — the package has to be discovered and reported. Proactive defenses include:

• Namespace reservation for popular hallucinated names (some registries are starting to do this)
• AI-assisted scanning of newly published packages for hallucination pattern matching
• Mandatory code review periods for new packages before they’re installable

Model-level mitigations. AI providers are working on reducing hallucination rates generally, but package-specific solutions include:

• Retrieval-augmented generation (RAG) that checks registry APIs before recommending packages
• Training data curation to reduce the model’s tendency to invent plausible names
• Post-generation verification that flags recommended packages not found on registries
• Tool-use architectures where the model calls a registry API rather than generating names from memory

Some AI coding tools have already implemented registry verification. GitHub Copilot and Cursor increasingly use tool calls to validate package recommendations against live registry data. This is the right direction, but coverage is inconsistent and not all AI coding tools have adopted it.

Discussion

The Fundamental Tension

There’s a deeper issue here that registry fixes and verification steps don’t fully address: LLMs are optimized for plausibility, not truth. A package name that sounds right is right by the model’s internal logic. The model has no concept of “this package exists” versus “this package sounds like it should exist.”

This tension extends beyond package names. LLMs hallucinate API endpoints, CLI flags, function signatures, and configuration options with the same confidence. The supply chain attack vector is just the most directly exploitable version of a general problem.

Shifting Responsibility

Who bears responsibility when an LLM-recommended package turns out to be malicious?

The model provider? They didn’t create the malicious package. The attacker? They registered a package on a public registry, which isn’t inherently illegal. The developer? They installed something without verification, which is careless but common.

The honest answer is that the existing software supply chain wasn’t designed for a world where an AI intermediary confidently recommends dependencies. The trust model assumed that developers chose packages based on reputation, documentation, community recommendations, and personal experience. LLMs short-circuit that evaluation process.

This doesn’t mean AI coding tools are too dangerous to use. It means the trust model needs to evolve. Verification should be built into the tooling, not left as a manual step that developers inevitably skip.

The Arms Race Ahead

As LLMs get better at avoiding hallucinations (through RAG, tool use, and improved training), attackers will adapt. Possible escalations include:

• Compromising real but unmaintained packages rather than registering new ones — the LLM might recommend a real package that was later hijacked
• Targeting less-monitored ecosystems like Cargo (Rust), Hex (Elixir), or Go modules where security tooling is less mature
• Poisoning training data to make models hallucinate specific package names that the attacker controls
• Multi-stage attacks where the hallucinated package is benign initially but pushes a malicious update after accumulating installs

The arms race between LLM-based development tooling and supply chain attackers is just beginning.

Conclusion

LLM package hallucination represents a genuinely novel attack vector — one that didn’t exist before AI coding assistants became widespread. The key takeaways:

• LLMs hallucinate package names at significant rates (roughly 5-20% depending on the model and task), and many hallucinated names are consistent across sessions, making them predictable targets
• Attackers are already exploiting this by registering hallucinated package names with malicious payloads on npm, PyPI, and other registries
• The attack scales with AI adoption — every developer using an AI coding tool without verification is a potential victim
• Defense is layered: individual verification, team-level allowlists, registry-level scanning, and model-level improvements all play a role
• The trust model for software dependencies needs to evolve to account for AI intermediaries that recommend packages without verification

The most important thing you can do today: verify every package an AI recommends before installing it. Check the registry. Look at the download count and publish date. Read the source. It takes thirty seconds, and it’s the difference between a useful AI suggestion and a supply chain compromise.

Sources

Footnotes

1. Lanyado, B. (2023). “Can you trust ChatGPT’s package recommendations?” Vulcan Cyber Research. Demonstrated systematic hallucination of npm and PyPI package names by ChatGPT, with consistent patterns across sessions. ↩

2. Spracklen, J., Neupane, A., & Gopavaram, S. (2024). “We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs.” arXiv:2406.10279. Large-scale study across multiple models finding ~20% hallucination rate with 43% repeatability. ↩

3. Thompson, J. et al. (2024). Socket.dev research on registerability of hallucinated package names. Found over 50% of hallucinated npm names were available for registration. ↩

4. Checkmarx Security Research (2023). Analysis of malicious PyPI packages matching ChatGPT hallucination patterns. Documented credential-stealing payloads in packages registered under AI-hallucinated names. ↩

5. npm Security Advisory (2023). Multiple reports of malicious packages with names matching common AI coding assistant hallucinations, including cryptominer payloads. ↩

6. Lasso Security (2024). Research on AI model supply chain risks via hallucinated Hugging Face organization and model names. Demonstrated registration of matching repos as proof of concept. ↩