Railway Breach Timeline: How a Cloud Hosting Platform Went From Strange Activity to Telling Every Customer to Change Their Keys
Railway, a popular platform for deploying apps, says an intruder may have read customers’ stored environment variables, which often hold API keys and database passwords. Deployments were frozen for a day. Here is the timeline so far and what to rotate if your developers or vendors use it.
Developers have been passing one instruction around since Wednesday: if you have anything on Railway, rotate your secrets. Railway, a cloud platform that runs apps and databases for millions of developers, says an unauthorized party may have reached the systems that store customers’ environment variables, the settings where apps typically keep API keys and database and registry passwords.
This timeline is built from BleepingComputer’s report, coverage by The Hacker News, Railway’s security disclosure and its incident status page. Times are converted to Eastern.
The timeline
Wednesday, October 21: detection and disclosure
Railway says it detected anomalous activity that morning in systems holding customer environment variables. Its investigation found that an intruder may have accessed the encrypted variables and, for some of them, obtained the decrypted values. The company said initial analysis suggests a subset of users was affected, but it told all users to act. It also froze all deployments, which meant customers could not ship new versions of their apps while the investigation ran.
Railway said it believes the incident is part of a series of attacks on platform companies aimed at compromising AI inference providers, and that it had asked those upstream providers to revoke keys proactively.
By Thursday: Railway resets credentials itself
Railway reset the S3 credentials for every storage bucket and invalidated GitHub tokens it held for customers, so the exposed values would stop working quickly. The trade-off, according to its follow-up guidance, is that services using those buckets need a redeploy and customers have to reconnect GitHub themselves. The Hacker News also reports API tokens were invalidated.
Thursday, October 22, 4:44 a.m.: builds start returning
Builds came back first for Pro and Enterprise workspaces. Hobby followed by 6:41 p.m., and by 7:22 p.m. builds were available for Free and Trial workspaces as well. New signups stayed paused.
Thursday night: the report slips
Railway had aimed to publish a preliminary incident report Thursday night, but said at 10:19 p.m. that it needed more time and began emailing customers with a high likelihood of impact.
Friday, October 23, early morning: notifications continue
In its 4:09 a.m. update, Railway said it had emailed customers with a very high likelihood of impact and was still doing forensic work to find more groups at risk. It said it expected the preliminary report around midday Eastern. New signups remained paused.
The side stories
- Scale: BleepingComputer reports Railway serves about 4.8 million users running more than 16 million services, which explains how fast the warning spread.
- A theft claim: security researcher Taylor Monahan says the attackers drained cryptocurrency wallets whose private keys were stored on Railway, with the first known theft at least two hours before the official disclosure, as reported by The Hacker News. Railway has not confirmed that.
- A possible link, not confirmed: Symantec’s Dick O’Brien told The Hacker News there is no evidence linking the incident to the LLMjacking activity his team has been tracking, in which attackers use stolen AI keys so the victim pays for the computing. He noted that keys taken from stored environment variables could be abused the same way, though Railway has not said whether any have been.
What Railway customers are told to do
- Rotate every environment variable stored in Railway, including API keys and database credentials.
- Rotate private registry credentials.
- Redeploy services that use storage buckets so they pick up the new bucket keys, and reconnect GitHub.
- Review logs for the services those secrets protect, such as cloud accounts, databases and payment or AI providers, for unusual activity.
What this means for your company
You may never have heard of Railway, but your web developer, an app vendor or an in-house team might use it, or a platform like it. Ask the question this week. Then look at the bigger issue: secrets stored with a hosting provider are only as safe as that provider. Keep a list of where each important key lives, give each key the least access it needs, set spending and usage alerts on cloud and AI accounts so a stolen key shows up quickly, and practice rotating keys so it takes an afternoon rather than a week.
Need help inventorying where your company’s passwords and API keys live and building a plan to rotate them fast? Reach out to Layer 7 Data Solutions to discuss how we’ve helped companies similar to yours.
About The Author
