Skip to content
After "Black October": What Chainstack and Other Incidents Teach Dev Teams About Preventing Credential Leaks

After "Black October": What Chainstack and Other Incidents Teach Dev Teams About Preventing Credential Leaks

Image generated by xAI Grok
16 min read
0
0 comments
6 views

The wave of security incidents since October 2026 makes one thing clear: development teams rarely lose their credentials on their own machines. They lose them on the platforms they hand those credentials to, and in the software they depend on. You cannot stop a vendor from being breached, but you can decide in advance how much a leak will cost you.

Introduction

The wave of security incidents since October 2026 makes one thing clear: development teams rarely lose their credentials on their own machines. They lose them on the platforms they hand those credentials to, and in the software they depend on. You cannot stop a vendor from being breached, but you can decide in advance how much a leak will cost you.

On November 3, blockchain node provider Chainstack sent its customers a security notice. An attacker had entered its infrastructure through a third-party component and may have obtained customers' organization names, account holder names, member email addresses, and password hashes. Systems holding node credentials had also been accessed. Chainstack asked every customer to change their password, rotate node credentials, regenerate Platform API keys, and watch for phishing.

This was not an isolated case. In the same month, at least nine AI developer platforms disclosed breaches, a cluster that security firm Wiz named "Black October." Chainalysis and WorkOS confirmed attacks driven by AI agents. LiteLLM's release pipeline, the Claude Code extension in Cursor's marketplace, and a third-party script on Polymarket's front end were all tampered with.

This article first reviews these incidents and the pattern they share. It then lays out what development teams and small businesses can put in place ahead of time, across four layers: accounts and identity, keys and credentials, the software supply chain, and monitoring and drills.

1. The incidents: eight in one month

From early October to November 3, AI developer platforms, Web3 infrastructure, and developer tools were hit one after another. The table lists the eight incidents most relevant to development teams, newest first.

Date (2026)IncidentWhat happenedExposure or loss
Nov 3ChainstackAttacker entered a development environment via a third-party component on Oct 26, then reached parts of production; detected and cut off on Oct 31Organization names, account holder names, member emails, password hashes; systems holding node credentials were accessed and are treated as exposed
Oct 30ChainalysisAn external actor used "offensive agentic techniques" to reach internal systems and tools; Reactor and other products taken offline from Nov 2User IDs, names, emails, organization roles; no evidence that cleartext passwords, MFA secrets, or session tokens were accessed
Oct 30WorkOSIntrusion discovered Oct 30, limited to sales and customer-relationship systems; linked to the ARTEX frameworkCRM contacts, customer account histories, summaries of sales and support conversations; production systems unaffected
Oct 6–30"Black October"At least nine AI developer platforms disclosed breaches: Fireworks AI, Andromeda, Zeabur, Lightning AI, OpenCode, Railway, Warp, CommandCode, Northflank; Brev (NVIDIA) and Trigger.dev reported incidents laterAttackers specifically hunted stored environment variables; two stolen enterprise Bedrock keys tracked by Wiz ran up over $100k at list price; Zeabur is shutting down
Oct 29PolymarketA script from performance-monitoring vendor DebugBear was replaced; the malicious code used logged-in sessions to sell positions and move fundsAbout $2.7M from 700+ wallets; Polymarket is returning $3.9M including losses from the trading pause; its second third-party front-end compromise this year
Oct 29Claude Code extension (Cursor marketplace)A tampered version 2.1.313 was live in Cursor's extension marketplace for about 11 hours and could arrive via auto-updateUp to 5,000 users; adds the attacker's SSH key, opens a reverse tunnel, installs persistence that survives uninstalling; Microsoft's official marketplace unaffected
Oct 21LiteLLMRelease pipeline compromised; malicious 1.107.0 live on PyPI for about 1 hour 45 minutes and on Docker (including the latest tag) for about 2.5 hoursStealer and backdoor that run on every Python start; targets cloud credentials, AI tool tokens, .env files, SSH keys, browser data
From Oct 17ARTEX attack frameworkMulti-agent AI penetration framework, originally open source and since made closed source; CrowdStrike says a state-sponsored group used it against 200+ organizations at oncePost-exploitation confirmed at nine; one technique forged an admin token using authentication keys not rotated since 2022

As of November 4, 2026, most of these incidents are still under investigation. The details above come from company statements and security media reports.

2. The common pattern: attackers want your credentials

The entry points differ, but the goal is nearly always the same: obtain credentials that can call paid services or move assets, and use them before the victim notices.

PatternHow it showed upDefense
The target is credentials, not codeBlack October attackers hunted each platform's environment variables; the LiteLLM stealer swept .env files, cloud credentials, and AI tool tokens; the tampered extension reported the names of secret-looking environment variablesDefense 2
The way in is a third partyChainstack was entered via a third-party component; Railway's preliminary finding is privilege escalation in build infrastructure; Warp pointed to a third-party dependency; Polymarket's monitoring script was swappedDefense 3
Minutes from leak to abuseWiz saw the first call on stolen keys within 30 minutes of the vendor-side credential access, and bulk use through resale channels days laterDefenses 2 and 4
Old credentials work bestIn an ARTEX case, keys not rotated since 2022 were used to forge an admin token, then a VPN key from a 2024 email leak opened the internal networkDefense 2
Phishing follows the leakNames, emails, and organization details leaked from Chainstack and Chainalysis are enough for targeted phishing; AI-built fake login pages can capture MFA codes and sessionsDefense 1

AI makes attacks faster and cheaper. CrowdStrike notes that ARTEX lets a small team run many intrusions in parallel, switching to another path when one fails. A defense that relies on people spotting problems and fixing them by hand can no longer keep up. That is why this article focuses on setup done in advance: credentials with narrow permissions and short lifetimes, and alerts that fire on their own when credentials are abused.

3. Defense 1: Accounts and identity

The account layer has two goals: a single leak affects only one account, and whatever phishing captures cannot be used.

  1. Use a unique password for every service, generated by a password manager. Chainstack leaked password hashes. Long random passwords are hard to crack offline; weak or reused passwords, once cracked, get tried against your other services.
  2. Use passkeys or FIDO2 security keys wherever you can. They are bound to the site's domain and will not respond on a look-alike domain. Six-digit codes from SMS or an authenticator app can be relayed by a real-time proxy phishing page; passkeys cannot.
  3. Require two-factor authentication across the organization. Chainalysis made the same recommendation to customers who had not yet turned it on.
  4. Review members and permissions regularly. Remove departing employees and former contractors the same day, grant least privilege by role, and replace shared accounts with personal ones so you can tell whose credential was used.
  5. Verify any "security notice" with a fixed routine.
    • Do not click links in the email; type the official address into your browser and sign in.
    • Confirm the incident on the vendor's announcements page, status page, or a trusted security news site.
    • Check the headers: in Gmail, open the message, click ⋮ at the top right → Show original, and confirm that SPF, DKIM, and DMARC all say PASS and the sender domain is the official one.
    • Legitimate providers never ask for passwords, tokens, API keys, or seed phrases by email. Treat any such request as phishing.
    • Be wary of password reset emails and sign-in approval prompts you did not trigger, and never approve one you did not start.

4. Defense 2: Keys and credentials

Assume that every key you hand to a third-party platform will leak one day, and limit in advance how much damage it can do. This is the most important section of the article.

4.1 Start with a key inventory

Chainstack's single notice asked customers to rotate three kinds of credentials. If you do not know where each key is used, rotation turns into a hunt through every config file in the company. Keep a table per team with at least these columns:

KeyPurposeStored inScopeOwnerLast rotatedRotation steps
Blockchain node RPC credentialsProduction RPC readsSecrets ManagerOne nodeBackend lead2026-09-01Issue new credentials → update apps → deactivate old credentials
Bedrock invocation credentialsAI customer supportIAM role (no long-lived key)Approved models, US regionsBackend leadN/ANo rotation needed

4.2 Least privilege: restrict models, regions, and purpose

The two stolen Bedrock keys that Wiz tracked had different permissions, and the outcomes differed sharply:

  • Case A: the key allowed global.* cross-region inference. It was called about 800,000 times in 30 hours, and only about 6% of calls were throttled.
  • Case B: the key allowed only us.* inference profiles. Thousands of requests a reseller sent through global.* all failed, and later traffic ran into regional quotas; nearly all the calls that did not succeed were throttled.

The restriction did not stop the abuse entirely, since Case B still saw more than a million successful calls. But it slowed the attack and lowered the ceiling. On AWS Bedrock, grant invocation credentials only the models and inference profiles your workload actually uses (illustrative; adjust model IDs and regions):

1{ 2 "Version": "2012-10-17", 3 "Statement": [{ 4 "Sid": "InvokeApprovedModelOnly", 5 "Effect": "Allow", 6 "Action": ["bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream"], 7 "Resource": [ 8 "arn:aws:bedrock:us-east-1:123456789012:inference-profile/us.<model-id>", 9 "arn:aws:bedrock:us-*::foundation-model/<model-id>" 10 ] 11 }] 12}

As a rule, never grant wildcard permissions such as bedrock:*. Use separate keys for development, staging, and production. Where a cloud or API platform lets you restrict a key by IP, origin, or project, do it.

4.3 Avoid long-lived keys where you can

  • CI/CD: have GitHub Actions exchange an OIDC token for short-lived credentials from an AWS IAM role or GCP Workload Identity Federation, so no long-lived access key sits in repository secrets.
  • Cloud servers: use instance roles or service accounts and require IMDSv2. In one ARTEX case, an XXE flaw was used to query the cloud instance metadata service and obtain role credentials. IMDSv2 requires a PUT request for a session token first, which blocks this kind of GET-only exploit.
  • Crypto private keys: keep them out of PaaS environment variables. Users on social media claimed about $371,000 was stolen using crypto keys stored on Railway (cited by Wiz, unconfirmed). Leave signing to KMS, HSM, or MPC, cap hot wallet balances, and use multisig for large transfers.

4.4 Put a ceiling on every key

  • Set budget alerts on cloud accounts and AI services (AWS Budgets, GCP budget alerts), and hard spending caps where the provider supports them.
  • Lower service quotas to what your workload actually needs. As Case B shows, a quota is itself a stop-loss line.

4.5 Make rotation possible within 30 minutes

Attackers start testing stolen keys within 30 minutes of a leak. If rotation takes a day, you effectively have no defense.

  • Zero-downtime rotation: keep old and new keys valid together for a while, switch your apps, then deactivate the old key. This is exactly the mechanism Chainstack launched for node credentials on November 2.
  • Central retrieval: have apps read keys at runtime from Secrets Manager, Secret Manager, or Vault rather than baking them into images or build configs, so rotation means changing one place.
  • Scheduled rotation: rotate on a fixed cycle, and rotate affected keys immediately whenever any vendor sends a security notice, without waiting for its investigation to finish.

4.6 Keep keys out of repositories and logs

  • Run gitleaks or trufflehog in pre-commit hooks and CI, and turn on GitHub secret scanning push protection.
  • Never print environment variables in logs or error messages, and enable secret masking in CI logs.

5. Defense 3: Software supply chain

In supply-chain incidents, most victims simply installed a package, updated an extension, or loaded a monitoring script as usual. The defense is to make the moment new code arrives slower, controlled, and traceable.

5.1 Pin versions and hashes; never use latest

In the LiteLLM incident, Docker tags such as latest and main-stable pointed to the malicious image for about two and a half hours. Teams that pinned versions stayed entirely outside the exposure window.

1# Python: generate a hashed lock file and enforce hash checks on install 2pip-compile --generate-hashes requirements.in 3pip install --require-hashes -r requirements.txt 4 5# Node: in CI, install strictly from the lock file 6npm ci 7 8# Docker: in production, reference images by digest, not by mutable tag 9docker pull ghcr.io/your-org/app@sha256:<digest>

5.2 Let new versions cool off for a few days

The malicious LiteLLM 1.107.0 was on PyPI for less than two hours, and security firms flagged it within minutes of publication. A minimum release age on dependency updates avoids most poisoning of this kind; Renovate's minimumReleaseAge or uv's --exclude-newer can enforce it. Let urgent security patches through as exceptions.

5.3 IDE extensions and developer machines

Developer laptops often hold the most credentials and get the least oversight.

  • Install extensions only from a trusted marketplace, and know which marketplace your editor uses: Cursor and other VS Code forks use their own or an open-source marketplace, not Microsoft's.
  • Turn off extension auto-update and upgrade deliberately: add "extensions.autoUpdate": false to your VS Code or Cursor settings, and remove extensions you no longer use.
  • Do not keep production keys in your shell profile. Use short-lived sessions such as aws sso login, or inject secrets at runtime with your password manager's CLI.
  • On Macs that do not need remote login, turn off Remote Login in System Settings → General → Sharing, and check ~/.ssh/authorized_keys regularly for keys you do not recognize. The tampered Claude Code extension relied on exactly that: a planted public key and a running SSH service.

5.4 Third-party scripts on your website

Polymarket's loss came not from a smart contract bug but from a third-party monitoring script that was swapped out and then used logged-in sessions to move funds.

  • Inventory every third-party script your site loads. As a rule, keep analytics and monitoring scripts off login, payment, withdrawal, and wallet-signing pages.
  • Add Subresource Integrity (SRI) to version-pinned scripts so the browser refuses tampered content, and use a Content Security Policy (CSP) to restrict script sources and outbound connections.
<script src="https://cdn.example.com/lib@1.2.3/lib.min.js" integrity="sha384-<hash>" crossorigin="anonymous"></script>
Content-Security-Policy: script-src 'self' https://cdn.example.com; connect-src 'self' https://api.example.com
  • Know their limits. One security expert pointed out that the malicious script talked to the vendor's own domain, which a CSP allowing that vendor would not block, and that frequently updated monitoring scripts are hard to pin with SRI. For such scripts, either self-host a reviewed copy or keep them away from sensitive pages.
  • Add a server-side backstop: require re-authentication or an extra confirmation for high-risk actions such as withdrawals or changing a payout address, so a hijacked session alone cannot complete them.

6. Defense 4: Monitoring and drills

Something will always slip past the first three layers. The last layer is to notice early and respond from a plan written in advance.

6.1 Watch who is using your keys

In Wiz's two cases, the stolen keys had mostly been called from the breached vendor's servers. The first call after the theft came from a residential IP. For several days after that, the keys saw only a trickle of keep-alive calls, 14 to 120 a day, before the flood began. Those days are your window to catch it.

  • Turn on your cloud provider's API audit logs (such as AWS CloudTrail) and AI model invocation logs.
  • Alert on these signals: calls from IPs or networks outside the expected range, models or regions you have never used, and sudden jumps in call volume or cost.
  • Add a daily usage alert on top of budget alerts, so you are not waiting for the monthly bill.

6.2 Subscribe to vendor security notices

List the status page, security bulletin page, and official accounts of every platform you use, and assign someone to watch them. Vendor notice emails can be delayed or land in spam; every extra source buys you time.

6.3 Check developer machines and servers for persistence

The tampered Claude Code extension planted an SSH key and set up persistence: a startup item on macOS, a reboot cron job on Linux, and a scheduled task on Windows. You can check regularly:

1# SSH keys you do not recognize 2cat ~/.ssh/authorized_keys 3# macOS startup items 4ls ~/Library/LaunchAgents /Library/LaunchAgents /Library/LaunchDaemons 5# Linux scheduled jobs 6crontab -l

6.4 Write a one-page response plan and drill it every quarter

When a vendor sends a security notice, work through these steps in order:

  1. Confirm the notice through an official channel (see Defense 1, item 5).
  2. Find every affected credential in your key inventory.
  3. Rotate them one by one following your runbook, aiming to finish within 30 minutes.
  4. Check call logs and billing for anomalies from the intrusion start date the vendor reported. For Chainstack, that means from October 26.
  5. Warn your team to watch for the phishing that will follow.
  6. Record what you did and update the inventory and the plan.

For the drill, pick a production key at random and time a full rotation. If it takes longer than 30 minutes, your inventory or rotation setup still has gaps.

7. Ten things to do today

  1. Check whether you use Chainstack, Chainalysis, WorkOS, or any Black October platform; if so, follow the vendor's instructions.
  2. Use a password manager to give each important service its own random password.
  3. Turn on passkeys or two-factor authentication for your cloud console, code host, domain registrar, and work email.
  4. Start a key inventory with the 10 most important production keys.
  5. Retire wildcard-permission API keys and reissue them scoped by model, region, and project.
  6. Set budget alerts and daily usage alerts on cloud accounts and AI services.
  7. Review long-lived access keys in CI/CD and replace them with OIDC short-lived credentials where possible.
  8. Reference production images by digest and enforce lock-file hash checks on dependency installs.
  9. Turn off editor extension auto-update and remove extensions you do not use.
  10. Inventory the third-party scripts loaded on your login, payment, and withdrawal pages.

SolanaLink Co., Ltd. is a Tokyo-based IT services company offering cloud solutions, web and mobile development, IT operations, and AI agent implementation. If your team needs help with a key inventory, tightening cloud permissions, or supply-chain hardening, contact us at info@solanalink.jp.

References

Comments

Comments (0)