The Solana network experienced a significant outage on Feb. 6 at 10:22 am UTC, with block production halted for over five hours before validators restarted the network.
This is not the first time Solana has gone down. Since January 2022, Solana has seen around half a dozen significant outages and fifteen partial or major outage days.
Engineers from across the ecosystem are investigating an outage on mainnet-beta. This thread will be updated as more information becomes available https://t.co/rfeioQ6BG9
— Solana Status (@SolanaStatus) February 6, 2024
The incidents varied in severity, ranging from partial to total network outages, and their causes included software vulnerabilities in the network and denial-of-service and resource exhaustion attacks.
The most recent outage was blamed on a bug that resulted in an infinite loop error, which caused the validators to stall on a particular block. An infinite loop error causes a program to get stuck in an endless cycle, which, in the case of Solana, prevented validators from verifying transactions on the network as they were stuck trying to process the same block.
Post-mortem
Solana released a postmortem report of the recent outage on Feb. 9 detailing its cause and how validators restarted the network. The report revealed that Solana’s Just-in-Time (JIT) compilation cache, which compiles all programs before executing a transaction, encountered the bug.
Earlier, the JIT cache was implemented using “ExecutorsCache,” however, with the v1.16 release, ExecutorsCache was replaced by a new implementation called “LoadedPrograms.” The new loader program used accounting information stored in a program’s on-chain account to look up its most recent deployment slot and use this to calculate the effective slot height.
However, this loader program encountered a legacy loader issue resulting in the JIT cache encountering a loop with older programs, causing it to recompile them repeatedly instead of efficiently accessing and executing their instructions. This glitch resulted in a network stall, as the system became bogged down in an infinite loop, unable to process transactions or move forward.
Austin Federa, head of strategy at Solana, told Cointelegraph that there was a process to replace the old loader system with a new motor system, but it was scheduled to be disabled with an updated version.
“It looks like someone intentionally called that old instruction set, which hadn’t been used in quite a long time but was still around in the codebase. The JIT compiler ran into issues where it couldn’t find what it was looking for. And that’s what created the infinite loop,”
Solana engineers quickly discovered and fixed the problem, updated the system’s processes, and identified these older programs to prevent future recompilation cycles. This solution entailed reducing the deployment of troublesome legacy loaders to ensure that all program instructions could be accessed accurately and effectively without falling into the recompilation trap.
Recent: ZK-proofs introduce security challenges for developers
Some critics have said that a significant factor contributing to Solana’s outages is the lack of client diversity, leading to potential central points of failure. Kadan Stadelmann, chief technology officer of open source decentralized exchange platform Komodo, told Cointelegraph that Solana’s developers and validators prioritize shipping code fast over maintaining perfect network uptime, adding:
“The key reason is centralization and lack of validator client options. Solana’s ambition for high throughput and low transaction times puts immense pressure on its infrastructure, making it susceptible to performance degradation and outages.”
However, Federa suggests Solana possesses much more accurate architecture and explains that only two smart contract networks worldwide have independent validator clients: Ethereum and Solana. Currently, Solana has one major validator client built on one functional codebase.
“Solana has never had an outage due to a fault in consensus or something along those lines. The problems have always been implementation details. It’s always been that there’s been a bug somewhere in the implementation layer, which is not better or worse. It’s just an important kind of accuracy,” Austin explained.
The Solana mainnet “beta” saga
The Solana blockchain was launched in March 2020, and over the next four years, the network has made significant strides in the decentralized finance (DeFi) ecosystem. Solana ranks as the fifth-largest DeFi chain in total value locked, with over $1.7 billion in capital allocated to different on-chain protocols. Its native Solana (SOL) token has a market valuation of around $49 billion, making it the fifth-largest cryptocurrency.
Despite a magnificent track record, Solana has never officially come out of its beta phase. The blockchain is still referred to as being in mainnet beta. Beta is a term that refers to nearly finished software that a select group of users is testing before an official launch.
However, the Solana blockchain is open to everyone, with hundreds of decentralized applications utilized to launch various products. The blockchain ecosystem also boasts a vibrant nonfungible token (NFT) ecosystem.
Solana underwent brief private and public beta testing before releasing v1.0 four years ago, and today, the network processes more transactions than most other chains.
Even though the code is still changing, Solana is much beyond what can be considered beta software. The persistent misperception results from a mix-up between pre-launch testing and post-launch software maturity and misconceptions around Solana’s messaging.
Recent: How can crypto firms bridge the gap with traditional finance?
The beta status has attracted much criticism of the blockchain, especially in light of its outages over the past two years. Crypto users on social media have often mocked that Solana’s status as a mainstream blockchain often changes to mainnet beta and back, depending on whether the chain is down.
Solana is a joke. Period.
— David Miller ⬡ (@CryptoDavid_) May 1, 2022
How many halts have there been in lasts months? The “it’s a beta” is not an excuse anymore. $SOL #Solana pic.twitter.com/IjvRaM58dZ
Federa claimed that most blockchain systems are in beta today and that only Bitcoin and Ethereum are probably complete mainnets and explained:
“It takes a long time for software systems to stabilize and be feature-complete as important new features are still being shipped on the Solana network. I think we’re probably nearing the end of that beta phase. From a technology standpoint, one of the significant milestones for that will be the launch of a second independent validator client.”
Many DeFi proponents believe that Solana’s downtime is a definite roadblock to its aim of becoming an Ethereum competitor. Still, the developers have worked to ensure the bugs leading to past downtimes don’t impact the network in the future. Despite its tainted history of downtime, its high speed and low cost play a key role in attracting new projects to the platform.
