Caroline Bishop
January 16, 2025 03:30
GitHub experienced two significant service outages in December 2024, which affected user access due to server overload and third-party service outages.
GitHub, a leading platform for software development collaboration, reported two significant service outages in December 2024, according to the GitHub blog. This incident resulted in degraded performance across our services, impacting user access and functionality.
December 17 incident
The first event occurred from 14:33 UTC to 14:50 UTC on December 17, 2024. During this period, GitHub users experienced intermittent errors and timeouts, with an average error rate of 8.5% and a peak request rate of 44.3%. The outage affected several core functions, including logging in, viewing repositories, pull requests, and managing issues.
The root cause was determined to be an inadvertent outage of the live update service due to an overload on the web server due to planned maintenance. This service is very important in providing automatic updates to users who have to manually refresh the page, putting more strain on the servers. GitHub alleviated the problem by reverting maintenance changes and scaling the service to manage increased traffic from WebSocket clients.
Post-incident analysis revealed gaps in GitHub’s warning system, which delayed the assessment of the impact of the incident. The company is currently focusing on strengthening its monitoring and alerting mechanisms to prevent similar issues in the future.
December 20 incident
The second event occurred between 15:57 UTC and 16:39 UTC on December 20, 2024. This outage was caused by a partial outage by one of GitHub’s third-party service providers, rendering some marketing pages inaccessible and causing 500 errors for users trying to access those pages. However, no operational products or service areas were affected during this period.
The service provider resolved the issue at 16:39 UTC, restoring access to the affected pages. GitHub is currently exploring ways to improve error handling and ensure gradual degradation of service in the event of future outages.
GitHub continues to research strategies to improve infrastructure resiliency and service reliability. Users can track real-time service status updates on the Status page and learn more about ongoing improvements on the GitHub Engineering Blog.
Image source: Shutterstock