Solana validators twice restarted network due to an unknown error that caused a complete halt in economic activity on the blockchain. The problem occurred while updating the node software to a more recent version.
The network stopped properly processing user transactions on February 25 at approximately 07:46 Kyiv time (08:46 Moscow time) and switched to a “degraded” mode. The latter means that all functions are disabled, except for voting.
One of the validators – Stakewiz – explained that the error occurred after the transition of most nodes to software version 1.14.16 last week. He called the update “a huge leap with an abundance of code changes.”
We know that in the 6-12 hours prior to the start of today’s event the validators running 1.14.x versions cumulatively achieved a supermajority of stake (>66.6%).
We also know that there was a «fat block» propagated around the time of the start of this event. pic.twitter.com/ZAvfbzZd1M
— Laine ❤️ stakewiz.com (@laine_sa_) February 25, 2023
According to Stakewiz, shortly before the suspension of economic activity, a block was generated on the network containing about 150,000 shreds. It caused a malfunction of the used clusters block propagation mechanism Turbinewhich entailed a simple blockchain.
A few hours after the incident, the validators restarted the network for the first time. However, the measure proved to be ineffective, so a second iteration was required.
Stakewiz noted that the initial incident caused separate chains to emerge. He also emphasized that the restart of the blockchain is de facto a hard fork, since it requires the resynchronization of validators and 80% consent in the voting.
After the second restart on the night of February 26, the developers reported that they had solved the problem. At the time of writing, the network regularly generates blocks, according to Solana Explorer. At the same time, jumps in the average transaction processing time are observed periodically in the blockchain, which negatively affects its throughput.
Recall that in June 2022, Solana crashed due to an error in the processing of “long-term non-targeted transactions”, which led to a violation of consensus.
