Grafana Breach Caused by Missed Token Rotation After TanStack Attack

Grafana Breach Caused by Missed Token Rotation After TanStack Attack

OverviewA recent Grafana data breach has been traced to a single GitHub workflow token that slipped through the organization’s rotation process in the wake of a TanStack npm supply-chain attack. The incident underscores how even a well-coordinated incident response can be thwarted by a single overlooked credential, and it highlights the broader risks associated with automated package supply chains and credential exposure.

Incident BackgroundThe breach is linked to the ongoing Shai-Hulud malware campaign attributed to TeamPCP hackers. In this operation, numerous TanStack packages were disseminated on the npm index with embedded credential-stealing code. These compromised packages propagated into developer environments, including Grafana’s, enabling malicious code to run within Grafana’s CI/CD environment. The malicious activity allowed the attackers to exfiltrate GitHub workflow tokens from Grafana’s shared environments.

Key developments

• Discovery window: Grafana detected malicious activity from compromised TanStack packages on May 1.
• Immediate action: An incident response plan was activated, including rotating a large set of GitHub workflow tokens.
• The missed token: Despite rapid rotation, one particular token was not rotated, which later allowed attackers to access Grafana’s private repositories.
• Compromised workflow: A specific GitHub workflow that Grafana initially deemed unaffected was found to have been compromised after later analysis.
• Stolen data: Intruders exfiltrated source code and operational information; they also accessed business details such as contact names and email addresses. Grafana emphasizes that this data is related to professional interactions and not to production systems.
• Production impact: Grafana states that there was no customer production data exposure and no impact on Grafana Cloud platform operations. The codebase itself was not altered during the incident, and code users should consider downloads from the period to be safe.

Timeline and technical details

• May 1: Grafana identifies malicious activity tied to compromised TanStack packages and initiates incident response, including token rotation.
• Token rotation efforts: A broad sweep to rotate GitHub workflow tokens is conducted to curb unauthorized access.
• Missed token discovered: A token initially deemed non-critical is later found to be compromised, enabling unauthorized access to Grafana’s private repositories.
• Ongoing review: Investigations reveal that an affected workflow had been overlooked in the initial assessment, contributing to the breach’s persistence.
• Data access and exfiltration: In addition to code, attackers obtained operational information and business contact details.
• Customer impact assessment: Grafana confirms that no customer production data was accessed and no actions are required by customers based on current evidence.
• Code integrity: The company asserts that its codebase was not modified during the incident, and downloaded code remains safe.

Impact on Grafana and customers

• Source code and private repository access: Attackers gained access to Grafana’s private repositories via the exfiltrated token.
• Production systems: There is no reported compromise of customer production data or Grafana Cloud platform operations.
• Actionability for users: Grafana does not require customers to take action at this time, pending ongoing investigation outcomes.
• Ongoing communications: If new evidence alters the assessment of impact, Grafana commits to notifying affected customers directly.

Ongoing investigation and communicationsGrafana emphasizes that the investigation is still active. The company plans to provide updates as new evidence emerges. The overarching conclusion remains that customer production systems were not compromised, and product code downloads were not altered during the incident. Still, the incident illustrates how a single rotated token can become an access point if it is missed during the remediation process.

The Validation GapAutomated pentesting tools are powerful for answering a narrow question: can an attacker move laterally through a network? They are not designed to test whether controls actually block threats, whether detection rules fire, or whether cloud configurations hold under real-world conditions. This discussion highlights six critical surfaces that organizations should validate beyond basic access checks.

• Validation focus: Ensure that controls block threats and that detection rules respond appropriately.
• Six key surfaces: The guide outlines six areas to validate to close gaps not covered by automated tools.
• Practical takeaway: Rely on a combination of automated tests and manual validation to achieve comprehensive security assurance.
• Availability: The guidance is offered as a downloadable resource for organizations seeking deeper validation.

Related articles and context

• Grafana says stolen GitHub token let hackers steal codebase
• TeamPCP hackers advertise Mistral AI code repos for sale
• Bitwarden CLI npm package compromised to steal developer credentials
• Checkmarx confirms LAPSUS$ hackers leaked its stolen GitHub data
• GitHub investigates internal repositories breach claimed by TeamPCP

Impact and next stepsThis incident reinforces the importance of comprehensive credential hygiene, including meticulous rotation and verification of tokens across all CI/CD workflows. It also underscores the necessity of validating the security of automated supply chains and ensuring that all workflows are included in rotation plans. While Grafana’s current public statements indicate no customer impact and no need for action, ongoing investigations may refine these findings. As the situation evolves, the emphasis remains on preventing credential leakage, securing private repositories, and maintaining rigorous oversight of third-party dependencies in software supply chains.