Solana Nearly Stopped After a Hosting Provider Outage

Solana network outage with validator servers going dark inside a data center

Solana came within 4.51 percentage points of being unable to confirm a single transaction on Wednesday morning. One misconfigured route at a hosting provider pulled 28.83% of staked SOL offline for 33 minutes. Roughly 90 validators dropped at the same instant. The incident puts the concentration of the network’s infrastructure back on the table.

Key Takeaways

  • A misconfigured default route at Teraswitch pulled 28.83% of stake offline for 33 minutes.
  • The chain stops finalizing blocks at 33.34%, so it stayed 4.51 points below the line.
  • One autonomous system carried 27.34% of stake, above the Foundation’s 25% cap.

A Default Route in Miami Knocked Out 90 Validators

The network lost 28.83% of its active stake within minutes early Wednesday, and the protocol itself had nothing to do with it. The trigger was a misconfigured default route inside hosting provider Teraswitch’s Miami facility. That configuration then propagated to the same provider’s European and Asia-Pacific regions.

Edge routers started rejecting routing information that was perfectly valid. The machines kept running, their keys kept working, but they could no longer see the rest of the network. Around 90 validators went dark at once, with nothing broken on their own side.

The episode ran 33 minutes and closed out at 04:16:15 UTC. Affected validators missed 333 SOL in rewards in total, an amount their bonds will cover at the end of the current epoch. The direct financial damage is close to a rounding error at network scale.

What matters is the shape of the failure rather than its cost. No consensus bug, no attack, no transaction flood of the kind that produced the outages Solana spent years living down. A routing mistake at a third-party vendor was enough to take nearly a third of the stake off the map.

That sits awkwardly next to the reliability story Solana has been telling. The chain had largely convinced the market that its stop-and-start years were behind it, including through the stretches when capital briefly rotated back into the large altcoins. The weak point moved down a layer, out of the code and into the plumbing.


Solana
Advertisement – investing involves risk.

The Chain Sat 4.51 Points From Confirming Nothing

Solana runs on Tower BFT, a byzantine fault tolerance variant that needs roughly 66.67% of stake actively participating to finalize blocks. Once more than 33.34% of stake goes dark, the chain loses the supermajority it needs and stops confirming transactions for good.

On Wednesday the counter froze at 28.83%. That left 4.51 percentage points before the threshold, or 86% of the distance to a full finality stall.

What happens on the other side of that line is worth spelling out. The chain keeps producing blocks, but nothing gets finalized. For an exchange, a lending protocol or a cross-chain bridge, that means no transaction can be treated as settled. Deposits freeze, liquidations get impossible to process cleanly, and bridges usually cut their relays as a precaution.

Markets did not blink. The whole thing happened in the middle of the US night, it was fixed before the open, and the critical threshold was never crossed. Price barely moved, which is unsurprising for an asset already trading far below its September highs.

The silence is itself a data point. A network brushes against a halt and no one prices it, which means either the risk was already discounted or nobody is watching that closely anymore.


Also on Cryptonomic:


Teraswitch Was Already Past the Foundation’s 25% Cap

The most awkward number in the post-mortem is a single line. Autonomous system AS20326, the one operated by Teraswitch, carried 118,890,767 SOL, more than a quarter of everything staked on the network. And 94% of that went offline together.

The Solana Foundation’s delegation program caps any single autonomous system at 25% of network stake. The cap exists to prevent exactly this scenario. AS20326 was sitting at 27.34%, above the ceiling meant to protect the chain.

The rule was written, the rule was breached, and the failure arrived in precisely the form the rule was designed to rule out. The open question is whether the Foundation forcibly redistributes delegation or settles for a warning. The first option costs yield to established operators, the second leaves the exposure where it is.

For an investor the issue runs wider than one chain. Validation concentrated across a handful of commercial hosts affects most proof-of-stake networks, and the long run of monthly declines SOL has been through pushed plenty of operators to squeeze infrastructure costs. Pooling into the cheapest provider is a rational call made validator by validator, and a systemic risk once aggregated.

The next chapter gets written on the stake map, not in the codebase. As long as the real spread of validators across providers, regions and autonomous systems stays opaque to outside investors, this exposure remains invisible until the day it fires. Not much has shifted since the spring’s promised rotation into the majors.

Follow the story on Cryptonomic.

Comments

No comments yet. Why don’t you start the discussion?

    Leave a Reply

    Your email address will not be published. Required fields are marked *