Incidentsby GlyphGitHub returning 403 for all Git Sync repositories
Nov 4, 03:02 UTCNov 4, 04:59 UTC
Duration
1h 58m
Impact
Minor
Root cause
Not disclosed
Grafana Labs, 90 days
67 incidents
Affected
AWS Australia - prod-au-southeast-1AWS Australia - prod-au-southeast-2AWS Brazil - prod-sa-east-1AWS Canada prod-ca-central-1AWS Canada - prod-ca-east-0AWS Germany - prod-eu-west-2AWS Germany - prod-eu-west-4AWS Germany prod-eu-central-1AWS India - prod-ap-south-1AWS Indonesia - prod-ap-southeast-2AWS Indonesia - prod-ap-southeast-3AWS Ireland eu-west-1AWS Ireland - prod-eu-west-6AWS Japan - prod-ap-northeast-0
Final update
The fix has been deployed to all production clusters and we are observing a continued period of stability. At this time, we are marking the incident as resolved.
Timeline
- Resolved · Nov 4, 04:59 UTC
The fix has been deployed to all production clusters and we are observing a continued period of stability. At this time, we are marking the incident as resolved.
- Monitoring · Nov 4, 04:07 UTC
The fix has landed in 60% of production. We are currently monitoring as it lands in the remaining clusters.
- Identified · Nov 4, 03:02 UTC
Since ~21:40 UTC on Nov 3, GitHub started rejecting the default User-Agent our Git Sync client sends (nanogit/0), so nearly every GitHub-backed repo shows as unhealthy ("not authorized") and syncs are being skipped. Our Engineering team has rolled out a fix to development and validated that the fix works. We will roll out the fix to production shortly and will provide another update.
More from Grafana Labs
Full historyOctober usage report visualization issuestill openIncreased query latency for Logs in GCP US Central (prod-us-central-0)lasted 1h 31mElevated Request Errors for Grafana Across Multiple Regionslasted 1h 1mGrafana Cloud: k6 timeoutslasted 0mMetrics Alert Rule Evaluation Failures in AWS Australia (prod-au-southeast-1)lasted 1h 48mMissing Metrics for Some Synthetic Monitoring Checks using the Montreal, North Virginia, and North California public probeslasted 0m