Two Million Booby-Trapped Prompts Caught Thieves Inside AI Hosting Firms. What Businesses Using AI Tools Need to Know
Researchers seeded about two million AI prompts with unique fake credentials and links, then watched who used them. The answer included attackers inside three AI inference providers, resellers logging traffic, and a lab scraping prompts marked not to be kept. OpenRouter has since dropped three providers.
When an employee pastes something into an AI tool, where does it actually go? A months-long experiment by the industrial security firm Dragos suggests the honest answer, for many services, is “further than you think.” This briefing is based on The Hacker News’ report on the research and OpenRouter’s response, with added detail from Ars Technica.
At a glance
- Between July and November, researchers led by Dragos chief research officer HD Moore planted unique secrets, such as working AWS keys tied to empty accounts, passwords and one-off web links, in about two million prompts. Each secret went into exactly one request, so any later use pointed to whoever read it.
- Attackers had been inside three AI inference providers for months, and none of the three knew until researchers told them. The earliest breach began in mid-July.
- OpenRouter, a gateway that routes customers’ AI requests to a pool of third-party providers, says it has removed three providers, pulled 125 endpoints from its Zero Data Retention program, and notified potentially affected customers. It says it was not breached itself.
The three findings
1. Break-ins at the providers
In the largest case, a couple hundred planted AWS keys were sent in single requests during a few short windows in late October, and every request sent in those windows was stolen, including some the provider rejected before any model ran. That suggests the copying happened as traffic arrived. The thief tested keys, sometimes within hours, browsed storage and secrets services, and tried to create new AI service keys and add users to keep access, generating tens of thousands of AWS API calls. One attacker appears to have been inside two providers at once. Some of the stolen requests were flagged for Zero Data Retention, the setting meant to ensure a prompt is never stored.
2. Resellers helping themselves
No hacking was needed for the second finding. Some sellers on an AI inference marketplace listed models at rock-bottom prices, switched on request logging and collected what came through. One router quietly forwarded part of its traffic to that marketplace without telling customers; it says it has stopped.
3. Privacy flags ignored
An unnamed Chinese AI lab fetched the secret links in more than half of the prompts sent to its models from July to October, including about two-thirds of requests marked no data collection or Zero Data Retention. Prompts sent to U.S. hosts reselling that lab’s models triggered the same scraper. In one case a secret link turned up in a chat on Feishu, the workplace app known elsewhere as Lark, 24 days after the prompt was sent.
How providers have responded
Three providers have posted publicly. Phala says an attacker took control of one of its inference servers and could view requests as they were processed; it contained the incident on November 8 and says requests to one specific model between October 21 and November 8 should be treated as exposed. Together AI says an unauthorized party downloaded stored samples of customer prompts after an AI coding agent committed a credential to its public GitHub repository, and it has made zero data retention the default for new accounts. Novita confirmed a routing configuration issue sent some requests on to another company’s official API. OpenRouter says it will require assurances and validation before restoring any endpoint and is building the researchers’ testing method into ongoing checks.
Open questions
- Neither Dragos nor OpenRouter has named the three breached providers or the three removed, or said whether they are the same companies.
- The researchers say planted secrets were still being used months later, and they are working with about a dozen providers.
What this means for your company
The research did not examine every AI service, and nothing here says the tools your team uses were affected. But it does show that a privacy setting only binds the first company that receives a prompt, and that cheap, routed AI capacity can pass your data through hands you never agreed to. Treat anything typed into an AI tool as something that could be read by others. Write a short policy that bans pasting passwords, keys, customer records and confidential plans into AI tools. Choose business plans from established providers where you can read one contract and one privacy policy, rather than the lowest-cost option. If your developers or automations call AI models, have them pick upstream providers deliberately instead of accepting a default pool. And consider planting a free decoy credential in the data your AI tools can see, so that if it is ever used, you find out.
If you are not sure where your team’s AI prompts are going or what is in them, we can help you set sensible rules and choose tools that fit how your company works. Reach out to Layer 7 Data Solutions to discuss how we’ve helped companies similar to yours.
