Leaked Credentials, Nearly a Full Day Inside and a Locked Console: How the Rackspace Spot Breach Put Customers’ Kubernetes Secrets at Risk
Rackspace says an intruder used leaked credentials to get into its Rackspace Spot container platform on November 17, posing as legitimate users to create workloads and read the passwords and keys stored in customer clusters. Then the console went dark for days while customers waited to rotate them.
For customers of Rackspace Spot, the trouble started on Tuesday, November 17, though they did not learn of it until later in the week. According to a notification Rackspace sent to affected customers, which BleepingComputer obtained and reported on, someone holding leaked credentials spent most of that day inside parts of the platform. What they could reach goes to the heart of how modern applications keep their secrets.
A platform built for running containers
Rackspace Spot lets organizations deploy and manage containerized applications on Kubernetes. Like most Kubernetes users, its customers keep sensitive values such as database passwords, authentication tokens and API keys in Kubernetes Secrets, so that their applications can reach the other systems they depend on.
The day of the intrusion
Rackspace puts the window at roughly 00:20 to 22:00 UTC on November 17. During that time, the intruder was able to act as a legitimate user inside some customer organizations. That meant creating pods in their Kubernetes clusters, reading their Secrets, and listing nodes and volumes. The company’s warning is blunt: any Secret stored in an affected user’s clusters could have been read, along with any Spot API tokens or refresh tokens that user held.
Rackspace did not catch it on its own. It learned of the intrusion on November 18 from a customer report, and confirmed the same day that the reporting organization was affected.
Cleaning up
Since then, Rackspace says it has revoked affected tokens and sessions, removed the unauthorized pods, rotated keys, and tightened access controls and monitoring. It also published indicators for customers to search for: the IP address 23.224.79.106, the wider 23.224.0.0/15 range, and pods or objects whose names begin with i1, glm- or probe.
The bigger job falls to customers. They have been told to reissue Spot API tokens, replace refresh tokens used in automation, download fresh Kubernetes configuration files and rotate every credential stored in their cluster Secrets. Because those Secrets unlock other systems, Rackspace also asked them to review logs for their databases, container registries, Git repositories, cloud storage and third-party services going back to November 17.
Waiting on the console
That advice ran into an obstacle. The breach led to an extended outage of the Spot administration console, which kept customers from managing their clusters, and from rotating the very credentials they were told might be compromised. Sirish Raghuram, co-founder and chief growth officer of Platform9, which built Spot together with Rackspace, said in a Reddit discussion that access was deliberately shut off while the environment was secured. He said the team was “incredibly disappointed to have let you down,” and that shutting everything down and sanitizing the environment was the best way to protect users. After several delays, Rackspace announced full restoration on November 21.
Rackspace had not told BleepingComputer how many customers were affected or whether customer data was stolen.
What this means for your company
You may not run Kubernetes, but you almost certainly rely on a cloud service that holds keys to your other systems: a hosting control panel, a backup portal, an accounting integration, a software vendor’s remote tool. This incident shows two things worth planning for. First, a breach at the provider can expose everything stored there, so the credentials you leave with a service should be scoped to the minimum it needs and rotated on a schedule, not just after an emergency. Second, you may lose access to the provider’s console exactly when you need it most. Keep a current list of which keys and passwords live where, know how to revoke them from the other end (at the database, the cloud account or the vendor), and make sure your logs are kept somewhere the provider’s outage cannot take with it.
If you do not have a clear map of where your company’s credentials are stored and how quickly you could rotate them, we can help you build one and test it. Reach out to Layer 7 Data Solutions to discuss how we’ve helped companies similar to yours.
