Grafana.comSupport TicketsIntegrationsCost Management and BillingFleet ManagementIncident Management and Response (IRM)GCP UK - prod-gb-south-0Public SnapshotsGCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0GCP UK - prod-gb-south-0: APIGCP UK - prod-gb-south-0: QueryingGCP UK - prod-gb-south-0: Public ProbesGCP UK - prod-gb-south-0: IngestionGCP UK - prod-gb-south-0: Rule EvaluationGCP UK - prod-gb-south-0: Alertmanager and Rules Configuration APIGCP UK - prod-gb-south-0: QueryingGCP UK - prod-gb-south-0: AlertmanagerGCP UK - prod-gb-south-0: IngestionGCP UK - prod-gb-south-0: Metrics Generator
identified
We are seeing errors in Frontend o11y again. We are continuing to investigate and monitor the impacted services.
identified
Between 06:15 and 12:15 UTC, Grafana Cloud Frontend Observability in GCP UK (prod-gb-south-0) experienced errors accepting incoming telemetry data. This has been resolved and Frontend Observability is operating normally in this region. We are continuing to investigate and monitor the remaining impacted services.
investigating
We are continuing to investigate this issue.
investigating
We are continuing to investigate this issue.
investigating
As of 06:30 UTC, we became aware of to an issue with Grafana Cloud unavailibity in the prod-gb-south-0 region.
Engineering is actively engaged and assessing the issue. We will provide updates accordingly.
We are continuing to monitor for any further issues.
monitoring
A fix has been implemented and we are monitoring the results.
investigating
We are continuing to investigate this issue.
investigating
We’re currently investigating an issue affecting metrics reads and rule evaluations in mimir-prod-66. Our team is actively working to identify the cause. Thank you for your patience.
The fix has been implemented, and we are monitoring the results.
identified
We have noticed improvements across services. We will now be applying a more permanent fix. Impacted users may experience a disruption in the auth-api service of up to 15min while we apply a DB patch that will improve performance.
identified
The issue has been identified, and we have applied mitigations on our end. We are currently observing the progress.
investigating
We are investigating increased authentication errors affecting Grafana Cloud in the prod-us-west-0 region.
Customers may experience intermittent failures for requests that require authentication.
Our engineering team has identified database performance degradation affecting the authentication service and is investigating the underlying cause.
We will provide another update as more information becomes available.
We have put some mitigations in place which seem to have helped with the errors.
We continue to monitor the results.
investigating
We are continuing to investigate this issue.
investigating
A small number of customers with stacks in prod-us-central-4 are receiving "forbidden" errors when querying Prometheus data sources.
We are investigating and will provide an update soon.
Phone call notifications to numbers in Russia and Kazakhstan are blocked
Phone call notifications to numbers in Russia and Kazakhstan are still blocked. We are working to resolve the issue.
As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative.
We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.
investigating
Phone call notifications to numbers in Russia and Kazakhstan are currently blocked. We are working to resolve the issue.
As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative.
We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.
investigating
Phone call notifications to numbers in Russia and Kazakhstan are currently blocked. We are working to resolve the issue.
As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative.
We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.
Elevated Errors Ingesting Metrics in AWS US East (VA)
We have isolated infrastructure suspected of contributing to the metrics ingestion issue in AWS US East (VA) - prod-us-east-3. We are validating recovery and completing mitigation steps. Customers may continue to experience failed or delayed metric writes while this work continues.
identified
We have isolated infrastructure suspected of contributing to the metrics ingestion issue in AWS US East (VA) - prod-us-east-3. We are validating recovery and completing mitigation steps. Customers may continue to experience failed or delayed metric writes while this work continues.
identified
We have identified the cause of an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA) - prod-us-east-3. Customers may see failed or delayed writes to Grafana Cloud Metrics, this may also impact Metrics written by Frontend Observability and/or Synthetic Monitoring. We are continuing to work on mitigations and are monitoring the impact to Grafana Cloud Metrics. Next update by 14:00 UTC.
identified
We have identified the cause of an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA) - prod-us-east-3. Customers may see failed or delayed writes to Grafana Cloud Metrics, this may also impact Metrics written by Frontend Observability and/or Synthetic Monitoring. We are continuing to work on mitigations and are monitoring the impact to Grafana Cloud Metrics. Next update by 13:00 UTC.
investigating
We are continuing to investigate an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA). Customers may see failed or delayed writes to Grafana Cloud Metrics, this may also impact Metrics written by Frontend Observability and/or Synthetic Monitoring. Next update by 12:00 UTC.
investigating
We are continuing to investigate an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA). Customers may see failed or delayed writes. Next update by 11:00 UTC.
investigating
We are investigating an issue affecting metrics ingestion for Grafana Cloud Metrics customers in AWS US East (VA). Customers may see failed or delayed writes. Next update by 10:15 UTC.
Legacy Incident Archive
101 archived records restored. These records predate source-at-observation attestation. They are shown as historical ServiceAlert records only and do not establish a provider-reported incident, resolution, duration, or current active state.
July 2026
Grafana rulers crash-looping on prometheus
minorSource not attested
Last archived state: resolved
View archived record details
monitoring
We are in the process of rolling out the fix. Our engineers are monitoring the progress.
investigating
We are investigating issues with the grafana-ruler service on prod-us-east-2 and prod-us-west-0 which are causing periodic crash conditions. A code fix is currently being deployed to mitigate this
Cannot access dashboard set up as a home page
minorSource not attested
Last archived state: resolved
View archived record details
investigating
We are currently investigating an issue affecting some customers who are unable to load or access their dashboards when configured as their home page. We will update once we have more information on this.
Grafana Cloud IRM alert groups failing to produce alerts in us-east-3 region.
majorSource not attested
Last archived state: resolved
View archived record details
monitoring
The work on bringing services back to healthy state is completed and alerts should be created correctly, without a delay now. We're keeping the incident open and monitoring for any potential hiccups that might occur.
identified
Some IRM alert groups in us-east-3 region may not be able to produce alerts and could be sluggish/misbehaving in general. The issue started around 01:00 UTC on July 3rd. The issue has been identified and the cause of the issue has been fixed. Our team is actively working on getting the service back to healthy state.
Some Queries Failing
minorSource not attested
Last archived state: resolved
View archived record details
identified
We are continuing to work on rolling back a PR responsible for this behavior. Once we have more information, we will share it here.
Thank you for your patience.
identified
Queries with drop __error__, including log volume histogram queries (the queries that generate the histogram visualization in Grafana), are failing due to a bug with series limit checks.
The Root cause has been identified, and we are working on a fix.
Loki and Frontend Observability - Major Outage in prod-us-central-0 region
criticalSource not attested
Last archived state: resolved
View archived record details
investigating
The AWS Logs integration in the same region is affected as well. We will provide further updates as our investigation progresses.
investigating
We are currently investigating a major outage in Loki writes and Frontend Observability in the prod-us-central-0 region. Our Engineering team is investigating this and we will provide further updates as our investigation progresses.
Elevated Loki Query Bytes Reporting
minorSource not attested
Last archived state: resolved
View archived record details
investigating
We are investigating an issue where some customers may see higher Loki query byte usage reported than was actually consumed. This affects usage reporting only; there is no impact to query execution or service availability.
The issue began at approximately 13:20 UTC and is ongoing. We expect the issue to be resolved soon and will provide another update as more information becomes available.
June 2026
Confluent API Outage
majorSource not attested
Last archived state: resolved
View archived record details
investigating
We are investigating an issue affecting Confluent metrics ingestion across all regions. Due to an elevated error rate on the Confluent side, some metrics may not be ingested, resulting in potential data loss. We are actively investigating the issue and will provide updates as more information becomes available.
Mimir read errors and high latency in prod-eu-west-0
minorSource not attested
Last archived state: resolved
View archived record details
identified
We've identified a possible cause, and a mitigation is in place to prevent further occurrences.
investigating
The errors and latency have now recovered, we continue investigating the root cause.
investigating
The errors are recovering, and we are still looking into the root cause of this.
investigating
We are currently investigating an issue with Mimir in prod-eu-west-0 we are seeing read errors and high latency. This incident is currently ongoing. The errors are recovering but we are currently looking into the route cause of this.
Rule evaluation error on cluster prod-gb-south-0
minorSource not attested
Last archived state: resolved
View archived record details
monitoring
A fix has been applied and we are currently monitoring results.
investigating
We are currently investigating Rule Evaluation errors on the cluster prod-gb-south-0 which is leading to error codes showing within the stacks. We are looking into the issue and will update accordingly.
K6 - Test run metrics processing is delayed
minorSource not attested
Last archived state: resolved
View archived record details
investigating
We've improved the metric ingestion delay time and are working on additional fixes to bring it down to expected range. Customers can currently expect a delay of 2 to 5 minutes before their test run metrics show up (after starting a test).
investigating
Update: Changed incident title to "Test run metrics processing is delayed"
We have found the issue and are working on deploying the fix.
investigating
We are experiencing intermittent delays with secondary metrics processing for k6 Cloud test runs due to heavy load. We don't expect any data loss or impact on user runs, but results may take longer time to appear in UI.
investigating
A small update: The issue is isolated to new test runs, and users can go see the metrics of all the previous test runs
investigating
We’re currently investigating an issue causing metrics not to appear during test runs. . Our team is actively working to identify the cause. Thank you for your patience.
