litellm PyPI Supply Chain Attack: Full Analysis and Remediation

TL;DR: On March 24, 2026, malicious versions of the litellm Python package (1.82.7 and 1.82.8) were published to PyPI and discovered within hours. The attacker didn't directly compromise litellm — instead, they compromised Trivy, a security scanner used in litellm's CI/CD pipeline, and used it to steal the PyPI publishing credentials. The malware harvests SSH keys, cloud credentials, environment variables, and Kubernetes secrets, then encrypts and exfiltrates everything to an attacker-controlled domain. If you use litellm, check your version immediately and rotate all credentials if affected.

What happened?

At approximately 08:30 UTC on March 24, 2026, two malicious versions of litellm (a popular Python library for routing LLM API calls) appeared on PyPI. Neither version exists in the GitHub repository (opens in a new tab). No matching release or commit. They were uploaded directly to PyPI using stolen credentials.

The credentials were stolen through a chained supply chain attack. The attacker operating under the name TeamPCP (opens in a new tab) first compromised Aqua Security's Trivy scanner, then used that foothold to exfiltrate secrets from every CI/CD pipeline that ran Trivy without version pinning. litellm was one of the downstream victims.

Within approximately three hours, PyPI quarantined the entire litellm package. But three hours is a long time when the malware activates on Python startup.

The attack chain

This wasn't a simple credential theft. It was a multi-stage campaign that exploited trust relationships between open-source projects.

Stage 1: Trivy compromised (March 19)

On February 27, 2026, an attacker exploited a vulnerable pull_request_target workflow in Aqua Security's repositories to exfiltrate the aqua-bot PAT token. Aqua attempted to rotate credentials, but the attacker maintained a foothold (opens in a new tab) in the runner environment, allowing them to capture the new tokens during the rotation process.

On March 19 at 17:43 UTC, the attacker pushed a malicious tag (v0.69.4) to the aquasecurity/trivy repository. This triggered automated release workflows that distributed compromised binaries to Docker Hub, Amazon ECR, and GitHub Container Registry. The attacker also force-pushed 76 of 77 tags in trivy-action and all 7 tags in setup-trivy, meaning any CI/CD pipeline referencing these actions by tag — the standard practice — would pull compromised code.

The compromised Trivy binaries contained a three-stage payload: collect secrets from CI runner memory by reading /proc/<pid>/mem, encrypt them with AES-256-CBC + RSA-4096, and exfiltrate them to a typosquatted domain (scan.aquasecurtiy.org — note the misspelling of "security").

The exposure windows were narrow but sufficient:

v0.69.4 on GitHub Releases: ~3 hours
setup-trivy (all tags): ~4 hours
trivy-action (76 tags): ~12 hours

Stage 2: litellm CI secrets stolen

litellm's CI/CD pipeline ran Trivy without version pinning — ci_cd/security_scans.sh installed whatever version trivy resolved to. When the pipeline ran during the exposure window, it pulled the compromised binary.

The compromised Trivy scanner dumped the CI runner's environment variables, which included PYPI_PUBLISH_PASSWORD and GitHub PATs. These were exfiltrated via Cloudflare Tunnels to attacker-controlled infrastructure.

litellm is the known downstream victim. It is unlikely to be the only one. Trivy has over 34,000 stars on GitHub. The trivy-action GitHub Action — 76 of 77 tags compromised — had a 12-hour exposure window. Any CI/CD pipeline that ran Trivy during that window could have had its secrets harvested: not just PyPI tokens, but npm tokens, Docker Hub credentials, cloud keys, database passwords — whatever was in the runner's environment. litellm's compromise was discovered because the malware had a fork bomb bug that crashed systems. A silent exfiltration from another project's pipeline would have no such tell.

Stage 3: Malicious packages published (March 24)

On March 23, the attacker registered the domain litellm.cloud via Spaceship, Inc., less than 24 hours before the attack. The next morning, using the stolen PyPI credentials, they published litellm versions 1.82.7 and 1.82.8 directly to PyPI.

The attacker also used the stolen GitHub PATs to push to unrelated repositories under the maintainer's account (krrishdholakia/blockchain) and changed BerriAI repository descriptions to "teampcp owns BerriAI," confirming full account compromise.

A security scanner was the entry point. The tool organizations run specifically to find vulnerabilities was the vulnerability.

Two versions, two vectors

The attacker published two versions, each using a different delivery mechanism. The second was significantly more dangerous.

Version	Delivery method	Trigger condition
1.82.7	Payload embedded in `litellm/proxy/proxy_server.py`	`import litellm.proxy`
1.82.8	`.pth` file (`litellm_init.pth`, 34,628 bytes — largely the double base64-encoded payload) + payload in `proxy_server.py`	Any Python process startup — no import needed

Version 1.82.7 only activated when a user explicitly imported the proxy module. Version 1.82.8 used a .pth file that executes on every Python startup, regardless of whether litellm is imported. This broadened the attack surface from "users who run the litellm proxy" to "anyone with litellm installed who runs any Python script."

The .pth launcher contained double base64-encoded Python that spawned a background process, making it invisible to basic code scanning and difficult to detect at runtime.

What the malware does

The payload operates in three stages.

Collection

The malware harvests an extensive set of sensitive data:

SSH keys — entire ~/.ssh/ directory
Cloud credentials — AWS (~/.aws/), GCP, Azure configurations
Environment variables — including API keys, database URLs, and secrets
Kubernetes configs — ~/.kube/config and cluster credentials
Application secrets — .env files, .gitconfig, Docker auth files
Database passwords — connection strings and credential files
Cryptocurrency wallets — wallet configurations and key files
Shell history — ~/.bash_history, ~/.zsh_history
SSL private keys — TLS certificates and private key material
CI/CD configs — Terraform state, GitLab CI, Jenkins, Drone configurations
Cloud metadata — queries IMDS endpoints for cloud instance credentials

Exfiltration

The two versions used different exfiltration infrastructure. Version 1.82.7 reused the Checkmarx campaign's command-and-control server (checkmarx.zone/raw), while version 1.82.8 exfiltrated to https://models.litellm.cloud/ — a purpose-built domain registered the day before the attack to resemble the legitimate litellm.ai. Both are entirely attacker-controlled.

In both cases, collected data is encrypted using a hardcoded RSA-4096 public key with AES-256-CBC session encryption and bundled into tpcp.tar.gz. Only the attacker holds the RSA private key. Even if the exfiltrated archive is intercepted in transit, its contents cannot be recovered without that key.

Persistence and lateral movement

The malware installs a persistent backdoor at ~/.config/sysmon/sysmon.py with a corresponding systemd user service to survive reboots.

In Kubernetes environments, the payload escalates significantly:

Reads all cluster secrets across every namespace
Creates privileged pods — deploys alpine:latest containers on every node in kube-system with host filesystem mounts
Establishes persistent access — the privileged pods provide root access to the underlying host

If your production infrastructure runs Kubernetes and litellm was installed on any node or CI runner with cluster access, the blast radius extends far beyond a single compromised package.

How it was discovered

FutureSearch's security team found the compromise (opens in a new tab) when litellm was pulled in as a transitive dependency by an MCP plugin running inside Cursor.

A bug in the .pth launcher accelerated discovery: child processes spawned by the malware re-triggered the .pth file, causing exponential process spawning — effectively a fork bomb. Systems crashed, which drew attention to the package.

Without that bug, the malware would have operated silently. The attacker's code was functional, and the self-crashing behavior was an unintended side effect of the .pth execution model, not a design choice.

This wasn't just litellm

The litellm compromise was one target in a broader campaign. TeamPCP used the same Trivy foothold to attack multiple ecosystems in rapid succession:

npm packages (March 20) — Within 24 hours of the Trivy compromise, 28+ npm packages were hijacked in under 60 seconds using automated tooling. The packages used the same ICP canister command-and-control infrastructure as the Trivy payload.

Checkmarx ecosystem (March 23) — OpenVSX extensions were hijacked via a compromised account, and all 35 version tags of the KICS GitHub Action were redirected to malicious commits. The payload used a typosquatted domain (checkmarx.zone).

Aqua Security internal repos (March 22) — 44 repositories in Aqua's internal organization were defaced using a compromised service account. Repos were renamed with tpcp-docs- prefixes, and internal assets — including Jenkins job backups and supply chain Lambda functions — were briefly exposed publicly.

The Kubernetes payload in the broader campaign included an Iran-targeted wiper (opens in a new tab) that executed rm -rf / --no-preserve-root on systems with Asia/Tehran timezone. Later versions used WAV steganography to embed payloads in audio files, evading .py file filters.

Suppressing discussion

The attacker also deployed a spam botnet against GitHub issues discussing the compromise. Issue #24512 (opens in a new tab) received hundreds of bot-generated comments — generic messages like "Thanks, that helped!" and "Worked like a charm" — posted at a rate of 0.65 comments per second via automated GitHub API calls. Analysis (opens in a new tab) found 76% overlap between the spam accounts used on litellm and those used against the Trivy disclosure discussion.

Who is safe

Docker proxy image users — dependencies were pinned prior to v1.82.7. The litellm team confirmed (opens in a new tab) these images were not affected.
Users on v1.82.6 or earlier — not affected, as long as they didn't upgrade during the exposure window.
Users who install from GitHub source — no malicious code exists in any GitHub tag or commit. The poisoned packages were uploaded directly to PyPI.

Who is affected

Anyone who installed litellm==1.82.7 or litellm==1.82.8 from PyPI
Anyone whose dependencies resolved to these versions via unpinned or loosely pinned requirements
Transitive dependency users — litellm is a dependency of popular frameworks including CrewAI (opens in a new tab) and DSPy (opens in a new tab). If your lock file pulled litellm during the exposure window, you may be affected without ever having directly installed it.
CI/CD pipelines, Docker builds, and production deployments that ran pip install or uv sync during the ~3 hour window

What to do if you're affected

If you installed litellm 1.82.7 or 1.82.8, assume full credential compromise. Rotate everything. Do not wait to confirm exfiltration — by design, you cannot tell whether data was stolen because it was encrypted before leaving the system.

1. Check if you're affected

pip show litellm

The official advisory is tracked as PYSEC-2026-2 (opens in a new tab). Known malicious package hashes (SHA-256):

litellm-1.82.7.whl: 8395c3268d5c5dbae1c7c6d4bb3c318c752ba4608cfcd90eb97ffb94a910eac2
litellm-1.82.8.whl: d2a0d5f564628773b6af7b9c11f6b86531a875bd2d186d7081ab62748a800ebb

Search for the .pth file (the v1.82.8 persistence mechanism):

find $(python3 -c "import site; print(site.getsitepackages()[0])") -name "litellm_init.pth"

2. Remove the compromised package and downgrade

As of this writing, PyPI has unquarantined litellm and v1.82.6 is the latest safe version. If you need litellm in your project, downgrade explicitly after cleanup:

pip uninstall litellm
pip cache purge
rm -rf ~/.cache/uv
pip install litellm==1.82.6  # or: uv pip install litellm==1.82.6 — skip if removing entirely

pip uninstall does not always remove .pth files. Verify manually:

find $(python3 -c "import site; print(site.getsitepackages()[0])") -name "*.pth" | grep litellm

3. Hunt for persistence mechanisms

The malware installs a backdoor. Check for:

The known backdoor:

ls -la ~/.config/sysmon/sysmon.py

Systemd user service (Linux):

systemctl --user list-units | grep sysmon
ls ~/.config/systemd/user/ | grep sysmon

LaunchAgents (macOS):

ls ~/Library/LaunchAgents/ | grep -i sysmon

Crontab entries:

crontab -l

4. Audit Kubernetes clusters

If the affected system had Kubernetes access:

# Check for suspicious pods in kube-system
kubectl get pods -n kube-system | grep node-setup

# Check for unexpected service accounts
kubectl get serviceaccounts -n kube-system

# Check for suspicious RBAC bindings
kubectl get clusterrolebindings | grep -v system

5. Rotate all credentials

This is the most critical step. The malware targets everything:

SSH keys (~/.ssh/)
Cloud provider credentials (AWS, GCP, Azure)
API keys in environment variables and .env files
Database passwords
Git credentials and tokens
Kubernetes service account tokens
SSL/TLS private keys
CI/CD secrets (GitHub Actions, GitLab CI, Jenkins)

If the compromised system had access to production infrastructure, treat it as a full production security incident.

6. Check CI/CD pipelines and Docker images

litellm may be a transitive dependency in your lock files, not a direct requirement:

# Check Python dependency and lock files
grep -r "litellm" pyproject.toml uv.lock poetry.lock Pipfile.lock pdm.lock 2>/dev/null

# Check Docker images
docker history <image> | grep litellm

Rebuild any Docker images that were built during the exposure window.

7. Audit Trivy installations

The attack chain started with Trivy. If your CI uses Trivy, check for compromised versions:

Trivy binary: v0.69.4 through v0.69.6
trivy-action: any tag (76 of 77 were compromised)
setup-trivy: any tag (all 7 were compromised)

Pin Trivy to a known-good version by commit SHA, not tag. Tags can be force-pushed — as this attack demonstrated.

Incident response

The litellm maintainer acknowledged the incident publicly (opens in a new tab) and took the following steps:

Deleted compromised packages from PyPI
PyPI quarantined the entire package, then unquarantined following removal of the malicious artifacts
All maintainer accounts rotated (new accounts: @krrish-berri-2, @ishaan-berri)
Root cause identified as compromised Trivy dependency
Engaged Google Mandiant (opens in a new tab) for forensic investigation (ongoing as of publication)
No new releases until full supply chain audit is complete

What this means for developers

This attack exploited a pattern that is extremely common: running a security tool in CI without pinning its version. litellm's ci_cd/security_scans.sh installed Trivy without a version pin. When the Trivy release was poisoned, every pipeline that ran the script pulled the compromised binary automatically.

Three practices would have prevented the downstream compromise:

Pin dependencies by hash, not tag. Git tags can be force-pushed. Docker tags can be overwritten. Version numbers on package registries can be reused after deletion. The only immutable identifier is a cryptographic hash — a commit SHA, a Docker image digest, or a package hash. The inconvenience of updating pinned hashes is vastly smaller than the cost of a supply chain compromise.

Isolate CI/CD secrets. The PyPI publishing token was available as an environment variable during a security scan that didn't need it. Publishing credentials should only be accessible in the publishing step, not in every CI job. Most CI platforms support scoping secrets to specific jobs or workflows.

Treat security tools as attack surface. Security scanners run with elevated access — they need broad permissions to do their job. That same broad access makes them high-value targets. A compromised security scanner doesn't just fail to find vulnerabilities; it becomes one. The tools you trust the most deserve the most scrutiny.

The bigger picture

This isn't an isolated incident. March 2026 has been an exceptionally active month for open-source supply chain attacks.

The GlassWorm campaign (opens in a new tab) (reported by Scientific American, discovered by Aikido, StepSecurity, and Socket) was active March 3–9, using invisible Unicode characters embedded in open-source repositories to hide malicious code. Hundreds of JavaScript, TypeScript, and Python repos were compromised. Two previously clean npm packages with ~135,000 monthly downloads were hit by March 16.

TeamPCP's campaign compromised Trivy, litellm, 28+ npm packages, Checkmarx tooling, and Aqua Security's internal infrastructure — all within five days. The attack evolved through at least three payload versions (opens in a new tab), progressing from monolithic bash scripts to modular Python to self-propagating worms using WAV steganography.

The common thread: open-source infrastructure is built on trust relationships between projects. When one link in the chain is compromised, the blast radius is determined by how many other projects trust it implicitly. Trivy is trusted by thousands of CI/CD pipelines. One compromised release was enough to reach them all.

Sources

The claims in this post are drawn from the following sources. We encourage readers to verify them independently.