Copilot Policy Propagation Delays

[investigating] We are investigating reports of degraded performance for Copilot

[investigating] We’ve identified an issue where Copilot policy updates are not propagating correctly for some customers. This may prevent newly enabled models from appearing when users try to access them.<br /><br />The team is actively investigating the cause and working on a resolution. We will provide updates as they become available.

[investigating] We're continuing to investigate a an issue where Copilot policy updates are not propagating correctly for all customers.<br /><br />This may prevent newly enabled models from appearing when users try to access them.

[investigating] We're continuing to investigate an issue where Copilot policy updates are not propagating correctly for a subset of enterprise users.<br /><br />This may prevent newly enabled models from appearing when users try to access them.

[investigating] We're continuing to investigate an issue where Copilot policy updates are not propagating correctly for a subset of enterprise users.<br /><br />This may prevent newly enabled models from appearing when users try to access them.<br /><br />Next update in two hours.

[investigating] We're continuing to investigate an issue where Copilot policy updates are not propagating correctly for a subset of enterprise users.<br /><br />This may prevent newly enabled models from appearing when users try to access them.<br /><br />Next update in two hours.

[investigating] We're continuing to investigate an issue where Copilot policy updates are not propagating correctly for a subset of enterprise users.<br /><br />This may prevent newly enabled models from appearing when users try to access them.<br /><br />Next update in two hours.

[investigating] We're continuing to address an issue where Copilot policy updates are not propagating correctly for a subset of enterprise users. This may prevent newly enabled models from appearing when users try to access them.<br /> <br />This issue is understand and we are working to get the mitigation applied. Next update in one hour.

[investigating] Copilot is operating normally.

[resolved] GitHub experienced degraded Copilot policy propagation from enterprise to organizations between February 3 at 21:00 UTC through February 10 at 16:00 UTC. During this period, policy changes could take up to 24 hours to apply. We mitigated the issue on February 10 at 16:00 UTC after rolling back a regression that caused the delays. The propagation queue fully caught up on the delayed items by February 11 at 10:35 UTC, and policy changes now propagate normally.<br /><br />During this incident, whenever an enterprise updated a Copilot policy (including model policies), there were significant delays before those policy changes reached their child organizations and assigned users. The delay was caused by a large backlog in the background job queue responsible for propagating Copilot policy updates.<br /><br />Our investigation determined the incident was caused by a code change shipped on February 3 that increased the number of background jobs enqueued per policy update, in order to accommodate upcoming feature work. When new Copilot models launched on February 5th and 7th, triggering policy updates across many enterprises, the higher job volume overwhelmed the shared background worker queue, resulting in prolonged propagation delays. No policy updates were lost; they were queued and processed once the backlog cleared.<br /><br />We understand these delays disrupted policy management for customers using Copilot at scale and have taken the following immediate steps:<br /><br />1. Restored the optimized propagation path and put tests in place to avoid a regression.<br />2. Ensured upcoming features are compatible with this design. <br />3. Added alerting on queue depth to detect propagation backlogs immediately.<br /><br />GitHub is critical infrastructure for your work, your teams, and your businesses. We are focused on these mitigations and continued improvements so Copilot policy changes propagate reliably and quickly.<br />