From One Vendor’s Stolen Credentials to Dozens of Customer AWS Accounts: A Timeline of the Qubole Trusted-Access Breach
Between October 14 and 26, an attacker used compromised credentials tied to the Qubole data platform’s AWS account to assume roles that customers had set up to trust Qubole, then harvested secrets and copied data out of S3. Here is how it unfolded and what to check.
Many companies let outside software providers into their cloud accounts through a standing trust relationship, set up once and then forgotten. A campaign uncovered by researchers at Wiz shows what can happen when the provider’s own credentials fall into the wrong hands: every customer who still trusts that provider becomes reachable, even customers who stopped using the service.
This timeline is assembled from The Hacker News report on the Qubole incident and BleepingComputer’s coverage, which includes a statement from AWS.
Background: how the access worked
Qubole, a data lake and analytics platform owned by Idera, runs workloads inside its customers’ AWS accounts. To allow that, a customer creates an IAM role whose trust policy lets Qubole’s AWS account step into it through the AssumeRole call, along with a matching value called an External ID. AWS does not treat that External ID as a secret; anyone with access to the role can see it.
How it unfolded
- October 1: The start of the window Wiz recommends searching in CloudTrail logs for any account that holds a Qubole role.
- October 8 to 11: In what researchers treat as a separate, unlinked episode, an attacker used customers’ own AWS access keys, believed to have been created for Qubole, and compromised a few cloud environments.
- October 14: The main campaign begins. Using two compromised Qubole IAM users, the attacker starts assuming roles in customer accounts that trust Qubole. Most of those roles had been sitting unused, but the trust was still in place, so every request carrying the External ID was accepted.
- October 18: The attacker begins using a second Qubole IAM user that Qubole’s legitimate traffic also relies on.
- Through the campaign: Inside each account, the attacker read Lambda environment variables, pulled values from Secrets Manager and the SSM Parameter Store, then ran what it found through the TruffleHog tool to see which AWS keys, Microsoft Entra ID client secrets and Okta client credentials still worked. It then listed and downloaded the contents of any S3 bucket the role could reach, sometimes an entire bucket in one session. Sentra, which first spotted the activity in its customers’ environments and alerted Wiz, says secrets were collected at a pace of 100 passwords and keys in seven minutes.
- October 26: The last day of the observed campaign.
- November 9: Wiz publishes its findings. Mandiant was also part of the investigation. Sentra says dozens of victims were confirmed, with millions of files taken from a single customer. How the Qubole credentials were stolen has not been disclosed.
- November 11: AWS tells BleepingComputer its services were not compromised and worked as designed, and urges customers to review cross-account roles, remove unused integrations, grant least privilege and add conditions such as source IP limits to trust policies.
The one control that would have stopped it
According to the researchers, Qubole’s genuine traffic only ever came from AWS addresses, while the attacker’s came from hosting providers and VPN services. Wiz said a source IP condition on the trust policy, on its own, would have blocked every role assumption it observed.
What affected AWS customers are being told to do
- Search CloudTrail from October 1 onward for AssumeRole requests from Qubole’s AWS account (805246085872) that did not come from an AWS address, and review everything those sessions touched.
- If a Qubole role was assumed, rotate every secret, parameter and Lambda variable it could reach and revoke its active sessions; rotating a credential does not end sessions already issued.
- Delete Qubole roles you no longer need, or remove Qubole from their trust policies.
- Narrow each vendor role’s permissions, and limit role assumption to the vendor’s documented networks.
- Turn on an organization-wide trail and S3 data events or access logs for sensitive buckets. Several victims lacked those logs and could not tell which files were read.
What this means for your company
Every software service you connect to your cloud, email or file storage is a door that stays open until someone closes it. Make a list of the third parties holding standing access to your cloud accounts and Microsoft 365 or Google tenant, remove the ones you no longer use, and cut the rest down to the smallest set of permissions they need. Keep passwords and keys in a proper secrets store with tight access, and make sure logging is detailed enough to answer the question “what exactly did they take?” before you ever need to ask it.
Unsure how many forgotten integrations still have a key to your environment? Reach out to Layer 7 Data Solutions to discuss how we’ve helped companies similar to yours.
About The Author
