# Four Custodians of AI Provider Keys, Hit in Sixteen Days
> Fireworks, OpenCode, LiteLLM and Railway all hold customers' AI provider keys or environment variables, and each has now been breached or backdoored. Attribution is unconfirmed.
- **Publisher:** Optimus Labs · Civilizations
- **Published:** 2026-10-21
- **Severity:** CRITICAL · LIVE INCIDENTS · KEY CUSTODIANS
- **Types:** Credential, Inference
- **Categories:** AI — AI Asset Supply Chain Security; CH — Cyber Hygiene
- **Vendors / products affected:** Fireworks AI, OpenCode, LiteLLM, Railway, Anthropic, OpenAI, Google
- **Blast radius:** Any team that stored provider keys or environment variables with one of the four: Fireworks evaluation jobs, OpenCode Console workspaces, a LiteLLM proxy or host that ran version 1.107.0, or Railway service variables. LiteLLM [receives about 23 million downloads a week](https://www.theregister.com/security/2026/10/21/litellm-supply-chain-attack-hoovers-up-passwords-and-secrets/5304548), and its proxy keeps every upstream provider key in one database table.
- **Canonical URL:** https://optimuslabs.io/research/briefings/ai-provider-key-custodians-breached-oct-2026

> **Live incidents, as of Oct 21, 2026. LiteLLM and Railway are still investigating and this page will be updated.**

## TL;DR

Live incidents as of Oct 21, 2026. Four companies that sit between teams and AI model providers had customer secrets exposed or targeted between Oct 6 and Oct 21: Fireworks, OpenCode, LiteLLM and Railway. Only OpenCode ties its breach to distillation. Railway calls its incident part of a series aimed at AI inference providers. LiteLLM and Fireworks have named no motive. Revoke at the provider, not at the custodian.

## Summary

Live incidents, as of Oct 21, 2026. On Oct 6, [Fireworks found](https://fireworks.ai/security/bulletins/october-2026-security-incident) that someone had used internal credentials to read environment variables, including API keys, in a limited number of customer accounts. On Oct 17, [OpenCode said](https://x.com/opencode/status/2111519637049319818) a breach of its hosted Console may have exposed provider keys for 1.4% of Console users. On the morning of Oct 21, a poisoned [LiteLLM release](https://docs.litellm.ai/blog/security-incident-oct-21-2026) went out through PyPI and Docker with a credential stealer built to take the provider keys a proxy stores. The same day, [Railway said](https://blog.railway.com/p/security-disclosure-wednesday-october-21st) someone may have read decrypted customer environment variables and paused all deployments. Railway also said it believes this is part of a series of attacks on platform companies, aimed at AI inference providers. No one has published a root cause that links all four.

## Three hosted platforms and one package

- **Fireworks**, **OpenCode** and **Railway** are hosted platforms that store customer secrets, and an intruder reached each one. [Fireworks](https://fireworks.ai/security/bulletins/october-2026-security-incident) says internal credentials were used to capture evaluation-job variables and that it found no access to customer models, datasets or inference traffic. [OpenCode's Console](https://optimuslabs.io/research/briefings/opencode-console-breach-provider-key-exposure) was reached through a legacy system being retired. [Railway](https://blog.railway.com/p/security-disclosure-wednesday-october-21st) says some encrypted variables were obtained in decrypted form.
- **LiteLLM** is different. It is the open source gateway many teams run themselves, and the poisoned release ran inside their own environments. It [reads the proxy's provider-key table directly](https://www.endorlabs.com/learn/litellm-1-107-0-compromised-python-startup-hooks-steal-cloud-ai-and-proxy-credentials), takes cloud credentials, SSH keys and environment variables, and installs a backdoor that [checks in hourly](https://www.theregister.com/security/2026/10/21/litellm-supply-chain-attack-hoovers-up-passwords-and-secrets/5304548).

## How the LiteLLM release was taken

- [LiteLLM says](https://docs.litellm.ai/blog/security-incident-oct-21-2026) the attacker used three engineers' GitHub accounts, apparently through stolen GitHub CLI OAuth tokens: one changed the pipeline, one approved the release, one deleted the workflow records. Branch protection stopped the first attempt, then a token belonging to an organization owner overrode it. The build ran from a look-alike repository, Berri-AI instead of BerriAI, and the official source tag stayed clean. [JFrog notes](https://research.jfrog.com/post/litellm-hijacked-again/) that reading the source would have missed it.
- LiteLLM has not said how the tokens were stolen. Hijacked publishing access is a repeat pattern in our own briefings: in [arrayref](https://optimuslabs.io/research/briefings/arrayref-rust-crate-build-time-rce), a compromised maintainer account pushed malicious crates to crates.io.

## Why the middleman holds the prize

- Only OpenCode ties its breach to distillation, the pulling of a strong model's answers at scale to train a cheaper copy. [Anthropic says](https://www.anthropic.com/threat-intelligence-report-september-2026) distillers often use stolen API credentials from legitimate customers. Railway says its incident is part of a series aimed at AI inference providers. Fireworks and LiteLLM have named no motive, and no source links the four incidents to one actor.
- **Our read, not the vendors':** a gateway or platform holds keys for many customers in one place, so one intrusion yields a pile of paid, aged accounts with normal usage history. That is plausible, not proven. Stored tokens as the way in is not new: in [Salesloft Drift](https://optimuslabs.io/research/briefings/salesloft-drift-oauth-supply-chain), attackers stole the OAuth tokens a vendor held for customers and used them against customer CRMs.

## LiteLLM was already hit in March

- On Mar 24, versions 1.82.7 and 1.82.8 reached PyPI for about 40 minutes after attackers took LiteLLM's publishing token through a poisoned Trivy scanner ([Cloud Security Alliance](https://labs.cloudsecurityalliance.org/research/csa-research-note-teampcp-ai-infra-supply-chain-20260406-csa/)). CloudSEK later [counted more than 2,500 organizations](https://arstechnica.com/security/2026/08/terabytes-of-credentials-leaked-in-massive-supply-chain-attack/) in the captured data, which means potential exposure, not confirmed compromise. JFrog says the October release uses different payloads and infrastructure and that [no evidence links it to the March actor](https://research.jfrog.com/post/litellm-hijacked-again/). Both went after the same asset: the keys a gateway holds.

## How it maps to OWASP

- In the [Agentic Applications Top 10](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/), ASI04 Agentic Supply Chain Vulnerabilities covers LiteLLM and the hosted custodians, and ASI03 Identity & Privilege Abuse covers the stolen keys. In the [LLM Applications Top 10](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/), LLM02 Sensitive Information Disclosure names API keys as protected information, and LLM06 Unbounded Consumption covers what stolen keys are used for, including distillation theft.
- *The OWASP GenAI Security Project does not endorse any product, service or tool, including Optimus Labs. We cite its lists to classify these incidents.*

## If you hold other people's keys

- The eight practices for storing a customer's key are in [briefing #13](https://optimuslabs.io/research/briefings/opencode-console-breach-provider-key-exposure). LiteLLM adds a question for any vendor: who can change what you ship? **Our read:** keep an organization owner's token out of developer tooling, require approval from an account not reachable the same way, and copy workflow logs where an account cannot delete them. Environment variables are a favorite place to look: in the [Coder registry hijack](https://optimuslabs.io/research/briefings/coder-registry-infrastructure-hijack), poisoned modules scraped credentials from environment variables, config files and shell history, and no CVE was assigned.

## What we still don't know

- As of Oct 21, 2026, unpublished: how LiteLLM's GitHub tokens were stolen, how many Railway accounts were read and how the attacker got in, how the Fireworks intruder got the internal credentials, whether the four share an actor, and how many of the stolen keys are in use today.

## What happened

- Fireworks, OpenCode, LiteLLM and Railway each held customer provider keys or environment variables, and each had them exposed or targeted between Oct 6 and Oct 21.
- LiteLLM's release pipeline was taken through three engineers' GitHub accounts and shipped a credential stealer for about two hours.

## Why it matters

- Your model accounts now depend on a chain of custodians. A weak link anywhere exposes the keys.
- Deleting a key at the custodian does not kill a copy. Revoking at the issuing provider does.

## What to do

- Revoke and replace every provider key that touched one of the four, at the provider, and confirm the old key is dead.
- Pin LiteLLM to 1.106.2 by digest, and never pull latest on a gateway that holds keys.
- Set a hard spend cap per key and use one key per tool, so the next leak stays small.

## Remediation

### Contain now · 0-24h

- Rotate keys at the issuing provider (Anthropic, OpenAI, Google and others), not only in the custodian's dashboard, and check billing since Oct 6.
- LiteLLM: isolate any host or container that ran 1.107.0. [Do not start with pip](https://docs.litellm.ai/blog/security-incident-oct-21-2026), because any Python start re-runs the startup hooks. Delete cook.pth, cook.py and sitecustomize.py, rebuild from verified artifacts, and rotate everything the host could reach: proxy provider keys, master and virtual keys, DATABASE_URL, the IAM role and every environment secret.
- Railway and Fireworks: follow their notices. [Railway](https://blog.railway.com/p/security-disclosure-wednesday-october-21st) says to roll every environment variable and registry credential. [Fireworks](https://fireworks.ai/security/bulletins/october-2026-security-incident) says to rotate the third-party credentials named in your notice.

### Investigate · 24-72h

- Block and search for www[.]litellm[.]dev and the GitHub dead drop in the indicators below. The marker /tmp/.pc_once_72129118c59f5636c0bb proves exfiltration if present, but its absence proves nothing.
- Check cloud audit logs for the instance role being used from outside your infrastructure after install time. LiteLLM suggests starting with CloudTrail GetCallerIdentity.

### Harden the agent surface

- Pin gateways and base images by digest. Do not pull a mutable tag such as latest, and verify the exact wheel, not only the source tag.
- Keep provider keys in a secrets manager you control, one key per tool, with a hard spend cap that stops traffic.
- Ask every vendor that holds your keys how they are encrypted, how releases are approved, and how fast you will be told about a breach.

## Timeline

- Oct 6 — Fireworks finds unauthorized use of internal credentials: Environment variables in evaluation jobs were captured in a limited number of accounts. It published its bulletin on Oct 9 ([Fireworks](https://fireworks.ai/security/bulletins/october-2026-security-incident)).
- Oct 13 — LiteLLM look-alike domain registered: litellm[.]dev was created Oct 13 and updated Oct 18, per [JFrog](https://research.jfrog.com/post/litellm-hijacked-again/). The real project uses litellm.ai.
- Oct 17 — OpenCode Console keys exposed and breach disclosed: Earliest possible exposure 00:30 UTC. 1.4% of Console users, "[related to a large scale distillation attack](https://x.com/opencode/status/2111519637049319818)".
- Oct 21, 10:08 UTC — LiteLLM release protections overridden: Branch protection rejected a change to the private release workflow. Five seconds later a privileged developer credential overrode it ([LiteLLM](https://docs.litellm.ai/blog/security-incident-oct-21-2026)).
- Oct 21, 11:39 UTC — Malicious litellm 1.107.0 live: Seven PyPI wheels until 13:23 UTC. Docker tags latest, main-latest, main-stable and v1.107.0 were poisoned from 11:46 to 14:20 UTC.
- Oct 21 — Railway detects anomalous activity: Systems that store customer environment variables. Railway [notified upstream providers to revoke keys](https://blog.railway.com/p/security-disclosure-wednesday-october-21st) and paused all deployments.

## Indicators of compromise

- Malicious package: litellm 1.107.0 (PyPI, all seven wheels). Last safe version 1.106.2.
- Docker tags on Oct 21: v1.107.0, latest, main-latest, main-stable (digest sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e)
- Files in site-packages: cook.pth, cook.py, sitecustomize.py
- Exfiltration: hxxp://www[.]litellm[.]dev/healthcheck. Dead drop: raw[.]githubusercontent[.]com/letters9813/pythonclient-test/main/domain.txt
- Lock file /tmp/remote_code_loader.lock. Look-alike source in build provenance: Berri-AI/litellm (legitimate: BerriAI/litellm)

## Sources

- [Railway security disclosure, Oct 21](https://blog.railway.com/p/security-disclosure-wednesday-october-21st)
- [Railway incident status](https://status.railway.com/incident/QH8L8B3E)
- [LiteLLM incident report, Oct 21](https://docs.litellm.ai/blog/security-incident-oct-21-2026)
- [LiteLLM security issue #47826](https://github.com/BerriAI/litellm/issues/47826)
- [The Hacker News: LiteLLM compromised](https://thehackernews.com/2026/10/litellm-compromised-in-supply-chain.html)
- [The Register: LiteLLM supply chain attack](https://www.theregister.com/security/2026/10/21/litellm-supply-chain-attack-hoovers-up-passwords-and-secrets/5304548)
- [JFrog: LiteLLM hijacked again](https://research.jfrog.com/post/litellm-hijacked-again/)
- [Socket: LiteLLM compromise](https://socket.dev/blog/litellm-compromise)
- [Wiz: LiteLLM compromised, steals credentials](https://www.wiz.io/blog/litellm-compromised-steals-credentials)
- [Endor Labs: LiteLLM 1.107.0](https://www.endorlabs.com/learn/litellm-1-107-0-compromised-python-startup-hooks-steal-cloud-ai-and-proxy-credentials)
- [Fireworks October 2026 security incident](https://fireworks.ai/security/bulletins/october-2026-security-incident)
- [LAVA: Fireworks AI incident](https://lava.security/blog/fireworks-ai-security-incident)
- [OpenCode statement on X, Oct 17](https://x.com/opencode/status/2111519637049319818)
- [Optimus Labs briefing #13: OpenCode Console](https://optimuslabs.io/research/briefings/opencode-console-breach-provider-key-exposure)
- [Optimus Labs briefing: Salesloft Drift OAuth supply chain](https://optimuslabs.io/research/briefings/salesloft-drift-oauth-supply-chain)
- [Optimus Labs briefing: arrayref poisoned Rust crate](https://optimuslabs.io/research/briefings/arrayref-rust-crate-build-time-rce)
- [Optimus Labs briefing: Coder registry hijack](https://optimuslabs.io/research/briefings/coder-registry-infrastructure-hijack)
- [Cloud Security Alliance: TeamPCP and AI infrastructure](https://labs.cloudsecurityalliance.org/research/csa-research-note-teampcp-ai-infra-supply-chain-20260406-csa/)
- [Ars Technica: terabytes of credentials leaked](https://arstechnica.com/security/2026/08/terabytes-of-credentials-leaked-in-massive-supply-chain-attack/)
- [Anthropic threat report, Sep 2026](https://www.anthropic.com/threat-intelligence-report-september-2026)
- [OWASP Top 10 for Agentic Applications 2026](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/)
- [OWASP Top 10 for LLM Applications 2026](https://genai.owasp.org/resource/owasp-genai-llm-top-10-2026/)

---

Published by Optimus Labs · Civilizations, the threat research team at Optimus Labs. Optimus Labs secures the enterprise agentic AI attack surface: discovery and governance of every AI agent, MCP server, and SKILL, posture management against the OWASP Top 10 for LLM Applications, and runtime behavior anomaly detection. More briefings: https://optimuslabs.io/research/briefings