GitHub and Other VCS workloads experiencing intermittent issues

Final update

One minor correction, VCS connection issues can be mitigated by downgrading tfc-agent from 1.33.2 to 1.33.1.

1. Resolved · Oct 22, 20:03 UTC
   One minor correction, VCS connection issues can be mitigated by downgrading tfc-agent from 1.33.2 to 1.33.1.
2. Resolved · Oct 22, 20:01 UTC
   VCS connection issues should be mitigated by downgrading to tfc-agent 1.33.2. Our engineering team is still investigating the cause of this failure. If you do not have any VCS connection problems you do not need to downgrade to 1.33.1. We will release a new version of tfc-agent in the next 24 hours to fix the root cause.
3. Monitoring · Oct 22, 19:46 UTC
   Downgrading tfc-agent to 1.33.1 has proven effective in eliminating the intermittent VCS failures. We will continue to monitor for any further impacts. A new release with a fix is in-progress and expected within 24 hours. If you are not affected, do not downgrade. If you experience further issues please contact support.
4. Identified · Oct 22, 19:22 UTC
   Our engineering team has identified the tfc-agent release v1.33.2 as the cause of intermittent issues with VCS providers. We have downgraded the version to 1.33.1 for remote runs. If you are experiencing issues with self-hosted agents please downgrade to 1.33.1 until a new release is made to address the issue. **Current impact state**: Remote runs fully-functional. Affected self-hosted agents need to downgrade to tfc-agent version 1.33.1. Stacks users may experience a regression when using proxies with version 1.33.1. **Action underway**: Releasing a fixed tfc-agent version. **Next update time**: 19:45 UTC
5. Investigating · Oct 22, 18:47 UTC
   Intermittent and some persistent VCS issues have been reproduced on some workspaces. Our engineering team is continuing to investigate. **Current impact state**: Intermittent issues on VCS workspaces are continuing to be observed. **Action underway**: Engineering team is working on reliably reproducing the issue. **Next update time**: 19:00 UTC
6. Monitoring · Oct 22, 18:33 UTC
   Our engineers are no longer able to reproduce the issue and error counts have dropped off. We will continue to monitor for any further issues. If you experience any further problems please reach out to support. **Current impact state**: System in recovery. **Action underway**: Monitoring. **Next update time**: 19:00 UTC
7. Investigating · Oct 22, 18:01 UTC
   Our engineering is continuing to investigate intermittent VCS-related failures. **Current impact state**: VCS related workloads experiencing intermittent issues. **Action underway**: Engineering team investigating. **Next update time**: 18:30 UTC
8. Investigating · Oct 22, 17:32 UTC
   We are continuing to receive reports of intermittent issues with VCS workloads, some of which are not hosted on [GitHub.com](http://GitHub.com "GitHub.com"). Our engineering team is investigating these issues. **Current impact state**: VCS workloads experiencing intermittent issues. **Action underway**: Awaiting word from [GitHub.com](http://GitHub.com "GitHub.com") **Next update time**: 18:00 UTC
9. Identified · Oct 22, 17:15 UTC
   [GitHub.com](http://GitHub.com "GitHub.com") has acknowledged an issue causing intermittent issues accessing their API. This may cause VCS-based Terraform workloads to malfunction. We are monitoring GitHub status and will update this issue as we learn more. **Current impact state**: VCS workloads experiencing intermittent issues. **Action underway**: Awaiting word from [GitHub.com](http://GitHub.com "GitHub.com") **Next update time**: 17:30 UTC