Peter Jang
November 21, 2024 22:28
Sui Network had a brief outage due to a congestion control bug. Quick action by engineers and verifiers restored operations in minutes, enabling effective incident response.
The Sui mainnet recently suffered a severe outage due to a technical glitch that halted all network operations for several hours. According to the Sui Foundation, the incident, which occurred between 1:15 AM and 3:45 AM on November 21, 2024, involved a crash loop that affected all validators and prevented the processing of all transactions.
understanding of events
This issue is due to a bug in the congestion control code, specifically assert! A statement that crashes when the expected execution cost is 0. This issue has been linked to: TotalGasBudgetWithCap The mode was briefly enabled in protocol version 63 and reintroduced in version 68. A bug that appears when the network receives a transaction with a mutable shared object input and 0. MoveCall Running the command will cause all validators to crash.
The role of congestion control
Congestion control on the Sui network is critical to managing transaction rates for shared objects to avoid overloading the network. The system was recently upgraded to increase shared object utilization by accurately predicting transaction complexity. However, the upgrade inadvertently introduced a bug that caused an outage.
Resolution and Response
Once the problem was identified, Sui engineers immediately designed a fix. The fix code detailed in PR #20365 has been deployed to both mainnet and testnet in versions v1.37.4 and v1.38.1 respectively. The outstanding response from the validator community facilitated rapid deployment and enabled the network to resume operations within 15 minutes of publishing the fix.
Lessons Learned and Future Improvements
This incident highlighted the effectiveness of Sui’s incident detection and response mechanisms. Automatic alerts were immediately notified to engineers who worked with the validator community to quickly resolve the issue. Going forward, Sui plans to strengthen its testing system to prevent similar bugs and streamline its build workflow to reduce incident response time.
For more information, please visit Sui Foundation.
Image source: Shutterstock