While we continue to make progress, August proved to be a challenging month for availability. You can read more about these incidents in a blog post we published last month. We are aggressively investing in both architectural improvements and moving to Azure, which will give us more capacity. Meanwhile, we continue to see significant growth on our platform. We are prioritizing the most impactful work while minimizing risk, but as these incidents in August show, we cannot completely eliminate risk.

Ultimately, all the work needs to be done, and incidents give us an opportunity to adjust our priorities. As repair items from these incidents, we’ve made significant improvements to our capacity monitoring and management, retry policies that led to bigger impact, and resiliency improvements to core services. We’ve also continued to make great progress across many durable work streams.

On August 11, GitHub ran a production MySQL primary from Azure for the first time. Client-observed write impact was minimal, and no customer impact occurred in the transition. We repeated the pattern with two more primaries on August 27. We have further primaries scheduled over the coming weeks, increasing in complexity as we learn from each failover.

Read traffic also reached new highs. Reads from migrated services peaked at 60.4%, while reads from GitHub’s monolith peaked at 64.3% in Azure. Git reads reached 54%.

Away from the regional migration, the 24-table authentication-core cohort moved off GitHub’s oldest shared database, mysql1, removing approximately one million queries per second from its replicas. Separate query-hygiene changes removed another 120,000 queries per second and eliminated approximately 59,000 seconds of wasted database work per hour.

GitHub Actions gained additional capacity while longer-term isolation work continues. Job-routing changes moved 33% of jobs from a constrained production cluster to spare capacity, reducing peak cache CPU utilization from 98% to 80% and adding an estimated three months of headroom. This is a near-term containment measure, not the finish line; the August outage reinforced the need for more durable capacity and isolation, which continues.

Pull request isolation work continued. In addition to unauthenticated traffic already being served, now authenticated reads for the first production cohort reached 100%.

Investments in Git overload protection served 6.4% more traffic while improving 95th-percentile duration by 24% and maximum delay by 78%. Broader load-shedding protections at the edge also made progress, enabling levers that can protect GitHub under unexpected load—in fact, these protections were used in mitigating the aforementioned incidents in August.

We also improved monitoring and telemetry. Pull request monitoring now measures merge, review, and comment failures independently, where high read volume could otherwise mask a failing write path. On August 21, automated high impact incident detection began combining customer-support signals with service telemetry. API monitoring was also recalibrated and validated over 30 days, reducing noise and improving signal quality. These changes improve detection and response.

The next month of work includes moving the next database primaries, continuing to migrate services and corresponding traffic to Azure, chipping away at database health particularly on shared-databases, adding more automation around capacity management and auto-scaling, and extending dependency-failure handling across more of the pull request experience.

This principle continues to guide us: availability, then capacity, then features.

August 06 15:22 UTC (lasting 10 hours and 42 minutes)

What happened?

Impact Summary

• Duration: approximately 9 hours
• Services impacted: GitHub Actions (GitHub-hosted and self-hosted runners), plus actions-dependent features, including Copilot coding agent, Copilot code review, GitHub Pages builds, Dependabot, and repository migrations.
• Customer impact: At least 74 organizations saw GitHub Actions workflows fail to start, fail partway through, or sit queued far longer than usual, measured against each organization’s own normal baseline. Customers during the impact window may have seen workflow runs failing or stuck in a queued state, some actions API requests returning errors, and in some cases unexpected rate limiting in your workflows. In most cases, re-running a failed workflow once the incident was over succeeded.

The incident began with a routine deployment to an internal GitHub Actions service that processes incoming events and turns them into actions jobs. The deployment’s contents were not at fault (we rolled it back to confirm this); instead, replacing pods during the rollout briefly reduced capacity in one site and pushed the remaining sites past their limit as traffic shifted to them. Impact was heaviest through the middle hours of the incident, when a large share of actions workflow runs were failing to start or complete.

What went wrong and why?

The affected actions services were running close to their capacity and concurrency limits. A routine deployment that briefly reduced the number of running pods was enough to exhaust available headroom. This caused service mesh sidecars to experience CPU throttling and out-of-memory restarts, which then cascaded into cache, DNS, and API errors across multiple clusters. The ingress service mesh for these services had limited headroom so it could not absorb the temporary loss of capacity during the deployment.

As the core services recovered, a latent bug in the job-assignment path made recovery slower: runners were handed jobs that had already been revoked, then got stuck retrying them instead of picking up valid work, which created a self-amplifying backlog.

How did we respond?

1. A routine deployment to an internal actions service briefly reduced running capacity in one data center, and within minutes the service mesh and remaining pods saturated.
2. The failures cascaded across clusters as cache, DNS, and API errors spread and actions infrastructure failures climbed. We declared a public incident, identified the triggering deployment, and rolled it back to confirm its contents were not the cause.
3. Over roughly the next two hours we expanded capacity for the saturated services and throttled incoming webhook-triggered work so the system could stabilize.
4. With the core services recovering, a large backlog of queued jobs remained. A latent bug caused runners to be assigned jobs that were no longer valid and then get stuck retrying them, holding back real work.
5. We deployed fixes so runners stopped trying to acquire invalid jobs, drained the accumulated queues, and raised the internal rate limits that were slowing recovery. Workflow success rates climbed back toward normal.
6. System-wide queues drained, and actions returned to normal operation. A smaller set of self-hosted runners stayed stuck and were recovered manually, and some events from during the incident could not be replayed automatically and had to be re-triggered.

How are we making incidents like this less likely or less impactful?

• Add headroom and enable autoscaling for the service mesh ingress and the affected actions services so a routine deployment cannot tip them into saturation.
• Make deployments safer for these services by avoiding capacity reductions during rollout.
• Strengthen monitoring for the saturation and database-proxy conditions that preceded the incident so they are caught earlier.
• Improve how the system sheds load and drains backlogged work during large actions incidents, and prevent runners from getting stuck retrying invalid jobs.
• Ship automatic recovery for self-hosted Actions Runner Controller runners affected by this failure mode in upcoming runner and ARC releases.

August 17 13:40 UTC (lasting 7 hours and 35 minutes)

What happened?

Impact summary

• Duration: approximately 6 hours 44 minutes
• Services impacted: Issues and pull requests were the most visible (pages and API returned errors and slow responses). A broad set of other services that route through the same datacenter’s load balancers—including the REST and GraphQL APIs, actions, Copilot, sign-in and authentication, and the webhooks API—also saw errors and latency
• Customer impact: At the peak, 56.07% of front-door requests to the affected services failed or ran slow at the edge. Across the window, about 29K organizations, defined as a collection of user accounts owning repositories, saw at least one failed or slow request—over roughly 4.8M failed or slow requests in total. Retrying often worked, but during the worst stretches a large share of attempts failed. Once engineers halted the overloaded load-balancer processes and reduced the retry pressure feeding them, the affected datacenter recovered and error rates returned to normal. No data was lost.

What went wrong and why?

A new peak in traffic pushed one datacenter’s load balancers past their limits. A service-mesh sidecar reached its concurrency limit and did not scale up.

As requests backed up, several of the datacenter’s load-balancer nodes exhausted their network flow limits, which degraded the shared gateway authentication path and produced widespread authentication latency and failures across the many services that route through that datacenter.

A latent client retry bug sharply amplified traffic to one internal authentication endpoint, which slowed recovery for the Copilot Token Service. The core weakness was that our service-mesh sidecar did not scale up, together with our retry behavior, and clients were not bound enough to keep a partial degradation from amplifying into a broader overload.

How did we respond?

1. A new peak in traffic pushes a datacenter’s load balancers toward their limits; a service-mesh sidecar hits its concurrency limit and fails to scale up.
2. The overload cascades as several load-balancer nodes exhaust their network flow limits and the shared authentication path degrades; issues, pull requests, the APIs, actions, Copilot, and other services begin returning errors and slow responses.
3. Automated monitoring detects the elevated errors and an incident is opened; the affected products are marked degraded on the public status page as engineers from across the affected services converge.
4. Engineers trace the failure to network saturation on the load balancers in a single datacenter and begin shifting some traffic to another datacenter and reducing gateway retries to relieve the pressure.
5. The team stops the load-balancer processes on the saturated nodes and blocks the retry-triggering requests to the most-affected internal endpoint, which produces broad and immediate recovery.
6. Remaining authentication errors driven by client retry amplification are stabilized by ramping traffic back up gradually, and after a sustained period of healthy telemetry the incident is resolved.

How are we making incidents like this less likely or less impactful?

• Correct autoscaling policies so they account for service-mesh sidecar concurrency and capacity, not just the host service.
• Audit request, concurrency, and scaling limits for the service mesh across the affected services.
• Review retry and backoff limits across gateways and clients so a partial degradation cannot be amplified into a broader overload.
• Fix the client retry behavior that amplified authentication traffic during the incident.
• Improve load-balancer capacity monitoring and strengthen regional failover safeguards.

August 20 14:43 UTC (lasting 9 hours and 54 minutes)

What happened?

Incident summary

• Duration: approximately 10 hours 40 minutes
• Services impacted: Copilot cloud agent (task status and results not updating; the agent tasks themselves continued to run and complete)
• Customer impact: At least 54 organizations saw Copilot cloud agent task status and results lag well above their normal baseline, in some cases by up to 60 to 90 minutes. The tasks kept running and finishing; only the visibility of their status and results was delayed.

During the incident window, the Copilot cloud agent task was impacted. The tasks themselves still ran to completion, so no work was lost. Once processing caught up the correct status and results appeared. Waiting a short time, or checking back a little later, would have shown the up-to-date state.

Across the incident, at least 54 organizations experienced Copilot Cloud Agent task status and results lagging well above their own normal level. Per-minute customer-facing impact peaked at 37.5% of measured task-status activity.

What went wrong and why?

Copilot cloud agent stores the status and results of each agent task in a managed cloud database. One region of that database suffered a provider-side outage, and the calls that read and write task status in the affected region began failing and running slow.

The processors that stream task-status updates into that database then fell behind as database latency climbed. Their throughput is bounded by a fixed number of processing partitions sized for normal latency plus some headroom. The latency in this incident went well past that headroom, causing the backlog of task-status updates to grow instead of clearing. A storage configuration on the database also made the affected region slow to fail over, so the first failover attempts did not take effect, and recovery took longer than expected.

How did we respond?

1. A region of the managed cloud database that stores Copilot cloud agent task status began failing and running slow after a provider-side regional outage.
2. On-call engineers were paged, opened an incident, and traced the errors to the affected database region.
3. Engineers began a regional failover, but it did not take effect—a storage configuration on the database made the region slow to move—so task-status updates kept backing up.
4. The team forced the affected region offline and shifted task-status processing to a healthy region; write latency remained elevated, and the backlog kept the status view delayed.
5. Additional streaming capacity was added and the provider’s region gradually recovered, letting the processors work through the backlog so task status and results caught up.
6. Latency returned to normal, the backlog cleared, and the incident was mitigated and resolved.

How are we making incidents like this less likely or less impactful?

• Remove the database storage configuration that made the affected region slow to fail over, so a single region’s problems can be exited quickly.
• Improve runbooks for regional failover of this database, including a vetted, ordered list of fallback regions that keep the service healthy.
• Review failover priority so the next region chosen during a failover is the next-best healthy option.
• Make task-status streaming more resilient to elevated database latency, so a latency spike does not immediately throttle throughput and build a backlog.
• Improve monitoring and escalation on the managed databases.

August 26 15:11 UTC (lasting 2 hours and 50 minutes)

What happened?

Impact summary

• Duration: approximately 2 hours 53 minutes
• Services impacted: GitHub Actions (run starts), with knock-on delays to GitHub Pages deployments and Copilot code review.
• Customer impact: At least 24 organizations saw actions runs fail to start or start much later than usual, measured against each organization’s own normal baseline, and a total of 386 organizations saw at least some run-start impact. At peak, more than one in five actions run starts in each minute were failing or badly delayed. Affected workflow runs would have sat queued well past their normal start time, and some would have failed to start outright.

Dependent services like Copilot code review and some GitHub Pages deployments that run on top of actions, were impacted during the incident window.

In most cases, delayed runs started once the backlog drained, and runs that failed to start succeeded when re-run after the incident. A small set of runs that were created during the earliest part of the incident could not be recovered by re-running and had to be started fresh.

What went wrong and why?

The quick summary is that our shared infrastructure services have not kept up with our month-over-month actions growth and peak load.

A burst of incoming events arrived on top of an already-high load and pushed the database past its tipping point. Query times climbed, and the database primary saturated.

With the database overloaded, the internal service that turns incoming events into runner assignments could not keep up, so actions runs failed to start and began queuing well past their normal start time.

Failing over the database primary helped only partially. The throttles used to relieve inbound load were initially set slightly too high to fully protect the database, so recovery had to be ramped up slowly and manually.

There was no automatic circuit breaker to throttle inbound actions load when the database showed early signs of stress, so the protective throttling had to be applied and tuned manually during the incident. This is one of the learnings from this incident.

How did we respond?

1. During a daily traffic peak, a burst of incoming events landed while a shared database that actions depends on was already running near its limit. Write and query pressure on the database primary rose sharply and began to saturate.
2. The internal service that turns incoming events into runner assignments could no longer keep up. Actions runs began failing an began incident investigation.
3. We failed the database primary over to a replica. This improved things briefly but did not fully mitigate, so runs continued to fail or start late.
4. We throttled inbound event processing to relieve pressure on the database and let it recover. Core service health returned once the throttling and service restarts took effect, though inbound work was now intentionally slowed.
5. We raised the throttles gradually, watching telemetry at each step so we did not re-overwhelm the database, until full event processing was restored and the backlog of delayed work drained. We marked the incident mitigated.
6. A subset of jobs on larger and self-hosted runners remained stuck waiting for a runner. We deployed a change to release them, and continued follow-up work to clear runs that had been left showing as queued.

How are we making incidents like this less likely or less impactful?

• Improve query efficiency of database usage by optimizing specific code paths in the client code.
• Add an automatic circuit breaker that throttles inbound actions load when the database shows signs of stress, instead of relying on manual throttling during an incident.
• Add protections around how often the service falls back to the database primary when a replica is lagging, so a fallback cannot compound database pressure.
• Improve our ability to quickly cancel or clear runs left stuck in a queued or waiting-for-runner state after an incident, so affected jobs recover sooner.
• Continue the scaling and resiliency work already in flight for this part of actions, including changes that were completing and rolling out around the time of the incident.

August 27 10:04 UTC (lasting 2 hours and 8 minutes)

What happened?

Impact summary

• Duration: approximately 2 hours 50 minutes
• Services impacted: GitHub Copilot (the Kimi K3 model)
• Customer impact: 63.3% of Kimi K3 requests failed during this impact window.

Customers who had configuration to use the Kimi K3 model were impacted by this incident. Customers who were using other models or switched to using other models were not impacted.

What went wrong and why?

Copilot offers a choice of AI models. One of them, Kimi K3, is served by an upstream model provider.

That provider had a serving degradation that caused a large share of Kimi K3 requests to fail with errors. Because the problem was with the upstream provider, requests that used other models—and requests made with the Auto setting, which routed to a different model—were not affected.

A steady share of Kimi K3 requests kept failing until the provider’s mitigation took hold. At peak, more than half of the requests using Kimi K3 were failing.

How did we respond?

1. The upstream provider for Kimi K3 experienced a degradation, resulting in elevated failure rates for Copilot requests routed to that model.
2. Within a few minutes, monitoring flagged the elevated errors and we began investigating.
3. We declared an incident, traced the failures to a degradation at the upstream provider affecting Kimi K3 specifically, and posted a public status update pointing to the provider.
4. Requests that used other models, or the Auto setting, kept working throughout, so retrying or switching models would have succeeded.
5. We opened a ticket with the provider and monitored recovery as success rates climbed back toward normal on our dashboards.
6. We kept the incident open until the provider confirmed Kimi K3 was fully restored, then resolved it.

How are we making incidents like this less likely or less impactful?

• Work with the upstream provider to improve the reliability of the Kimi K3 model and reduce the errors seen during this incident.
• Investigate backup serving capacity for Kimi K3 so a single provider degradation has a fallback.