Grafana.comSupport TicketsIntegrationsCost Management and BillingFleet ManagementIncident Management and Response (IRM)GCP UK - prod-gb-south-0Public SnapshotsGCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0: APIGCP UK - prod-gb-south-0: QueryingGCP UK - prod-gb-south-0: Public ProbesGCP UK - prod-gb-south-0: IngestionGCP UK - prod-gb-south-0: Rule EvaluationGCP UK - prod-gb-south-0: Alertmanager and Rules Configuration APIGCP UK - prod-gb-south-0: QueryingGCP UK - prod-gb-south-0: AlertmanagerGCP UK - prod-gb-south-0: IngestionGCP UK - prod-gb-south-0: Metrics Generator
identified
Between 06:15 and 12:15 UTC, Grafana Cloud Frontend Observability in GCP UK (prod-gb-south-0) experienced errors accepting incoming telemetry data. This has been resolved and Frontend Observability is operating normally in this region. We are continuing to investigate and monitor the remaining impacted services.
investigating
We are continuing to investigate this issue.
investigating
We are continuing to investigate this issue.
investigating
As of 06:30 UTC, we became aware of to an issue with Grafana Cloud unavailibity in the prod-gb-south-0 region.
Engineering is actively engaged and assessing the issue. We will provide updates accordingly.
We are continuing to monitor for any further issues.
monitoring
A fix has been implemented and we are monitoring the results.
investigating
We are continuing to investigate this issue.
investigating
We’re currently investigating an issue affecting metrics reads and rule evaluations in mimir-prod-66. Our team is actively working to identify the cause. Thank you for your patience.
The fix has been implemented, and we are monitoring the results.
identified
We have noticed improvements across services. We will now be applying a more permanent fix. Impacted users may experience a disruption in the auth-api service of up to 15min while we apply a DB patch that will improve performance.
identified
The issue has been identified, and we have applied mitigations on our end. We are currently observing the progress.
investigating
We are investigating increased authentication errors affecting Grafana Cloud in the prod-us-west-0 region.
Customers may experience intermittent failures for requests that require authentication.
Our engineering team has identified database performance degradation affecting the authentication service and is investigating the underlying cause.
We will provide another update as more information becomes available.
We have put some mitigations in place which seem to have helped with the errors.
We continue to monitor the results.
investigating
We are continuing to investigate this issue.
investigating
A small number of customers with stacks in prod-us-central-4 are receiving "forbidden" errors when querying Prometheus data sources.
We are investigating and will provide an update soon.
Phone call notifications to numbers in Russia and Kazakhstan are blocked
Phone call notifications to numbers in Russia and Kazakhstan are still blocked. We are working to resolve the issue.
As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative.
We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.
investigating
Phone call notifications to numbers in Russia and Kazakhstan are currently blocked. We are working to resolve the issue.
As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative.
We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.
investigating
Phone call notifications to numbers in Russia and Kazakhstan are currently blocked. We are working to resolve the issue.
As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative.
We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.
Elevated Errors Ingesting Metrics in AWS US East (VA)
We have isolated infrastructure suspected of contributing to the metrics ingestion issue in AWS US East (VA) - prod-us-east-3. We are validating recovery and completing mitigation steps. Customers may continue to experience failed or delayed metric writes while this work continues.
identified
We have isolated infrastructure suspected of contributing to the metrics ingestion issue in AWS US East (VA) - prod-us-east-3. We are validating recovery and completing mitigation steps. Customers may continue to experience failed or delayed metric writes while this work continues.
identified
We have identified the cause of an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA) - prod-us-east-3. Customers may see failed or delayed writes to Grafana Cloud Metrics, this may also impact Metrics written by Frontend Observability and/or Synthetic Monitoring. We are continuing to work on mitigations and are monitoring the impact to Grafana Cloud Metrics. Next update by 14:00 UTC.
identified
We have identified the cause of an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA) - prod-us-east-3. Customers may see failed or delayed writes to Grafana Cloud Metrics, this may also impact Metrics written by Frontend Observability and/or Synthetic Monitoring. We are continuing to work on mitigations and are monitoring the impact to Grafana Cloud Metrics. Next update by 13:00 UTC.
investigating
We are continuing to investigate an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA). Customers may see failed or delayed writes to Grafana Cloud Metrics, this may also impact Metrics written by Frontend Observability and/or Synthetic Monitoring. Next update by 12:00 UTC.
investigating
We are continuing to investigate an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA). Customers may see failed or delayed writes. Next update by 11:00 UTC.
investigating
We are investigating an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA). Customers may see failed or delayed writes. Next update by 10:15 UTC.
Degraded Performance with Increased Latency for Grafana Cloud Metrics and Traces in GCP US Central
GCP US Central - prod-us-central-0GCP US Central - prod-us-central-5GCP US Central - prod-us-central-0: IngestionGCP US Central - prod-us-central-5: IngestionGCP US Central - prod-us-central-0: Ingestion
investigating
We are investigating degraded performance and increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and Grafana Cloud Traces in prod-us-central-0.
Customers may experience intermittent errors or delays when sending metrics and traces.
We continue to investigate the issue with our cloud provider. We do not have an estimated time to resolution at this time.
investigating
We are investigating degraded performance and increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and Grafana Cloud Traces in prod-us-central-0.
Customers may experience intermittent errors or delays when sending metrics and traces.
We continue to investigate the issue with our cloud provider. We do not have an estimated time to resolution at this time.
investigating
We are investigating degraded performance and increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and Grafana Cloud Traces in prod-us-central-0.
Customers may experience intermittent errors or delays when sending metrics and traces. We have also received reports that some customers using Google Managed Prometheus in this region are experiencing issues when querying their GMP instances.
We continue to investigate the issue with our cloud provider. We do not have an estimated time to resolution at this time.
investigating
We are investigating degraded performance and increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and Grafana Cloud Traces in prod-us-central-0.
Customers may experience intermittent errors or delays when sending metrics and traces. We have also received reports that some customers using Google Managed Prometheus in this region are experiencing issues when querying their GMP instances.
We continue to investigate the issue with our cloud provider. We do not have an estimated time to resolution at this time.
investigating
We are investigating degraded performance with increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and for Grafana Cloud Traces in prod-us-central-0. Customers may experience intermittent errors or delays when sending metrics and traces. We continue to investigate this with our cloud provider. We do not have an estimated time of resolution at this time. Next update by 18:30 UTC
investigating
We continue seeing elevated error rates and latency, and keep working with our cloud provider on this issue.
Next update by 17:30 UTC.
investigating
We are investigating degraded performance with increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and for Grafana Cloud Traces in prod-us-central-0. Customers may experience intermittent errors or delays when sending metrics and traces. We are investigating this with our cloud provider. We do not have an estimated time of resolution at this time. Next update by 16:30 UTC.
investigating
We are investigating degraded performance with increased latency in the ingestion path for Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5) and for Grafana Cloud Traces in prod-us-central-0. Customers may experience intermittent errors or delays when sending metrics and traces. We are investigating this with our cloud provider. We do not have an estimated time of resolution at this time. Next update by 15:30 UTC.
investigating
We continue to investigate intermittent elevated error rates and latency affecting Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5), and periodic increased latency affecting Grafana Cloud Traces in prod-us-central-0. Intermittent errors and latency may still occur. We are working with our cloud provider as part of the investigation. We do not have an estimated time of resolution at this time. Next update by 14:30 UTC.
investigating
Update to scope: we have not confirmed any impact to Grafana Cloud Logs, and Logs has been removed from the affected components. We continue to investigate intermittent elevated error rates and latency affecting Grafana Cloud Metrics in GCP US Central (prod-us-central-0 and prod-us-central-5), and periodic increased latency affecting Grafana Cloud Traces in prod-us-central-0. Error rates have decreased, but intermittent errors and latency may still occur. We are working with our cloud provider as part of the investigation. We do not have an estimated time of resolution at this time. Next update by 14:30 UTC.
investigating
We continue to investigate intermittent elevated error rates and latency affecting Grafana Cloud Metrics and Logs in GCP US Central (prod-us-central-0 and prod-us-central-5), and periodic increased latency affecting Grafana Cloud Traces in prod-us-central-0. Error rates have decreased, but intermittent errors and latency may still occur. We are working with our cloud provider as part of the investigation. We do not have an estimated time of resolution at this time. Next update by 14:30 UTC.
investigating
Beginning around 10:20 UTC, we have been seeing intermittent elevated error rates and latency affecting Grafana Cloud Metrics and Logs in GCP US Central (prod-us-central-0 and prod-us-central-5), and periodic increased latency and occasional errors affecting Grafana Cloud Traces in prod-us-central-0. Customers may see errors or delays when sending metrics, logs and traces. We are actively investigating. Next update by 13:40 UTC.
Legacy Incident Archive
101 archived records restored. These records predate source-at-observation attestation. They are shown as historical ServiceAlert records only and do not establish a provider-reported incident, resolution, duration, or current active state.
March 2026
Prometheus writes, Logs, and Synthetic Monitoring in prod-eu-west-3 are degraded
minorSource not attested
Last archived state: resolved
View archived record details
investigating
This is also now impacting Logs and Synthetic Monitoring in prod-eu-west-3.
For Synthetic Monitoring, users might observe errors pushing check execution metrics, and this can eventually lead to missing data.
In addition, users might observe errors evaluating Synthetic Monitoring provisioned alert rule evaluations, and this can lead to missed alerts.
For Logs, there is no immediate impact on alerts, however, remote writes to Mimir is delayed which means users may see gaps in their recordin...
investigating
We are moving this back to 'Investigating' as we are now observing a substantial drop in successful ingestion and increase in write path errors, and elevated rule evaluation latency and error. Reads are mostly fine. Our Engineering team is actively investigating this and we will provide further updates as our investigation progresses.
monitoring
We have not observed any recent errors, but we will continue to monitor while we work with our CSP.
monitoring
A fix has been implemented and we are monitoring the results.
investigating
We are currently experiencing degraded writes for mimir-prod-22 in prod-eu-west-3 since 08:45Z.
Grafana Assistant Unavailable in prod-us-east-0
majorSource not attested
Last archived state: resolved
View archived record details
identified
The issue has been identified, and we are implementing a fix.
investigating
The impact extends beyond the TOS check. Assistant is completely unavailable in the impacted region.
investigating
We are continuing to investigate this issue.
investigating
We are aware of an issue currently impacting Grafana Assistant. Impacted users are met with a request to accept the TOS, however the plugin is failing upon accepting. Our engineering are currently investigating this issue.
Authentication API Database Down in prod-eu-west-2 and prod-eu-west-4
majorSource not attested
Last archived state: resolved
View archived record details
investigating
We have observed impact in prod-eu-west-4 as well.
investigating
We are currently investigating an issue impacting the main database for Authentication API's in the prod-eu-west-2 region. Writes are currently failing, but reads are operational.
Various Datasource Issues
majorSource not attested
Last archived state: resolved
View archived record details
monitoring
We are continuing to monitor for any further issues.
monitoring
We have observed recovery for the Cloudwatch Datasource.
We are now seeing failures for the following Datasources:
Aurora
Opensearch
X-Ray
Timestream
Redshift
Sitewise
A fix for the above is being rolled out now, and we will monitor progress.
We will also change the name of this incident from "Cloudwatch Datasource Issues" to "Various Datasource Issues" to more accurately reflect impact.
monitoring
We have identified the issue, and are rolling out the fix. We are already seeing improvements and will continue to monitor progress.
investigating
We are currently investigating an issue impacting the CloudWatch Datasource causing failures.
Degraded performance of Grafana Cloud k6 test runs
majorSource not attested
Last archived state: resolved
View archived record details
investigating
Some customers are seeing degraded performance and errors from certain v6 API endpoints. We are investigating the issue.
Grafana Cloud Logs - Write degradation in Azure Netherlands (eu-west-3)
minorSource not attested
Last archived state: resolved
View archived record details
investigating
We are continuing to investigate this issue with our CSP, and will provide updates as they become available.
investigating
We are seeing issues on the write path for Loki in cluster Azure Netherlands (eu-west-3). Impact will reflect in degradation of logs ingestion on that cluster. Our engineering team is already working on restoring the service.
Increased number of Aborted-by-Systems with a k6 binary building errors
majorSource not attested
Last archived state: resolved
View archived record details
monitoring
A fix has been implemented and we are monitoring the results.
identified
The issue has been identified and a fix is being implemented.
investigating
We are seeing an increased number of Aborted-by-Systems with a k6 binary building error. We are investigating the issue.
The first occurrence of this happened back on March 9, has now been identified as a blocking issue for some customers.
Rule Evaluation Outage in prod-us-west-0
majorSource not attested
Last archived state: resolved
View archived record details
monitoring
A fix has been implemented and we are monitoring the results.
investigating
We are currently investigating an issue impacting rule evaluation for a subset of customers in the prod-us-west-0 region. We will provide updates as they become available.
Grafana Cloud Logs - Write degradation in Azure Netherlands (eu-west-3)
minorSource not attested
Last archived state: resolved
View archived record details
investigating
We are also reporting impact to Faro performance in the same region. We are continuing to investigate this issue.
investigating
We are seeing issues on the write path for Loki in cluster Azure Netherlands (eu-west-3). Impact will reflect in degradation of logs ingestion on that cluster. Our engineering team is already working on restoring the service.
Complete outage in prod-me-central-1
criticalSource not attested
Last archived state: resolved
View archived record details
investigating
The TLS certificates serving prod-me-central-1 endpoints expire on May 30, 2026. Replacement certificates have been imported, but the ongoing AWS regional incident is preventing them from propagating to all load balancer nodes, so customers may see certificate errors after that date until AWS restores normal operation.
We do not have any additional updates to share at this time. Our team is actively monitoring the situation and will provide further information as it becomes available.
In th...
investigating
AWS UAE - prod-me-central-1: Public Probe checks might suffer degraded experience. We recommend migrating checks from the UAE probe to the next nearest probe suitable for your use case.
investigating
We do not have any additional updates to share at this time. Our team is actively monitoring the situation and will provide further information as it becomes available.
In the meantime, please continue to refer to the AWS Status Page for the most detailed and up-to-date information.
investigating
We are continuing to investigate this issue.
investigating
We have not received any further updates from AWS at this time. However, we are actively monitoring the outage and will provide additional information as it becomes available.
Also, please continue to refer to the AWS status page for more detailed updates.
https://health.aws.amazon.com/health/status
All the guidance previously included about stack migration is still relevant. Please reach out to our Support team if you have any questions.
investigating
We are actively monitoring the situation, but at this time there are no new updates to share. The next update will be provided once we have more information to share. Please reach out to our Support team if you have any questions.
investigating
We are continuing to investigate this issue.
investigating
Please continue to refer to the AWS status page for more detailed updates specific to AWS.
https://health.aws.amazon.com/health/status
AWS are recommending that affected customers move workloads to alternate regions, and we are recommending the same.
Customers who are impacted and who cannot wait for a restoration of service are asked to:
1. Create a Grafana Cloud stack in an alternate region
2. Update clients to send telemetry to the new region, if using Grafana Alloy then you can use Fle...
investigating
AWS are recommending that affected customers move workloads to alternate regions https://health.aws.amazon.com/health/status and we are recommending the same.
Customers who are impacted and who cannot wait for a restoration of service are asked to:
1. Create a Grafana Cloud stack in an alternate region
2. Update clients to send telemetry to the new region, if using Grafana Alloy then you can use Fleet Management https://grafana.com/docs/grafana-cloud/send-data/fleet-management/introduction/...
investigating
Customers are recommended to configure a new blank stack in an alternative Grafana Cloud region and to reconfigure their clients (such as Grafana Alloy) to send telemetry to that region, Fleet Management can be used for this purpose https://grafana.com/docs/grafana-cloud/send-data/fleet-management/introduction/
investigating
We are updating this incident to reflect a complete outage in prod-me-central-1, due to an on-going AWS UAE data center issue. We will provide further updates accordingly.
investigating
We are observing write and read outage errors across all databases (metrics, logs, traces) in prod-me-central-1, due to an on-going AWS UAE data center issue. We will provide further updates accordingly.
investigating
We are observing write and read outage errors across all databases (metrics, logs, traces) in prod-me-central-1, due to an on-going AWS UAE data center issue. We will provide further updates accordingly.
investigating
We are seeing elevated write and read path errors in prod-me-central-1, due to an on-going AWS UAE data center issue. We will provide further updates accordingly.
