We have identified the cause as an issue in our cloud provider’s US Central region. A mitigation is in place and recovery is in progress. Customers in prod-us-central-0 and prod-us-central-5 may still see errors or delays with metrics, logs, or Grafana while services come back. We will update this page as recovery continues.
investigating
We are investigating reports of errors affecting Grafana Cloud in US Central (prod-us-central-0, prod-us-central-5). Customers in this region may see timeouts or failed requests when writing or querying logs and metrics. We are looking into this and will post another update as soon as we have more information.
This incident also had impact on Frontend Observability for the same region.
A fix has been deployed, and things have stablized. We will continue to monitor on our end.
identified
We’ve identified the cause of an issue impacting Log writes. Our team is currently implementing a fix.
Some Grafana UI features may be unavailable or reverting to legacy behaviour
A fix has been applied and metrics write error rates in prod-us-central-0 have returned to normal levels. Metrics ingestion is operating as expected. We are continuing to monitor the affected systems to confirm full recovery before marking this incident resolved.
investigating
We are investigating elevated error rates affecting metrics writes in prod-us-central-0. Customers in this region may see failed or delayed metrics ingestion, gaps in recent data on dashboards and queries, and alert rules that depend on the affected series may not evaluate as expected.
Legacy Incident Archive
101 archived records restored. These records predate source-at-observation attestation. They are shown as historical ServiceAlert records only and do not establish a provider-reported incident, resolution, duration, or current active state.
July 2026
Grafana rulers crash-looping on prometheus
minorSource not attested
Last archived state: resolved
View archived record details
monitoring
We are in the process of rolling out the fix. Our engineers are monitoring the progress.
investigating
We are investigating issues with the grafana-ruler service on prod-us-east-2 and prod-us-west-0 which are causing periodic crash conditions. A code fix is currently being deployed to mitigate this
Cannot access dashboard set up as a home page
minorSource not attested
Last archived state: resolved
View archived record details
investigating
We are currently investigating an issue affecting some customers who are unable to load or access their dashboards when configured as their home page. We will update once we have more information on this.
Grafana Cloud IRM alert groups failing to produce alerts in us-east-3 region.
majorSource not attested
Last archived state: resolved
View archived record details
monitoring
The work on bringing services back to healthy state is completed and alerts should be created correctly, without a delay now. We're keeping the incident open and monitoring for any potential hiccups that might occur.
identified
Some IRM alert groups in us-east-3 region may not be able to produce alerts and could be sluggish/misbehaving in general. The issue started around 01:00 UTC on July 3rd. The issue has been identified and the cause of the issue has been fixed. Our team is actively working on getting the service back to healthy state.
Some Queries Failing
minorSource not attested
Last archived state: resolved
View archived record details
identified
We are continuing to work on rolling back a PR responsible for this behavior. Once we have more information, we will share it here.
Thank you for your patience.
identified
Queries with drop __error__, including log volume histogram queries (the queries that generate the histogram visualization in Grafana), are failing due to a bug with series limit checks.
The Root cause has been identified, and we are working on a fix.
Loki and Frontend Observability - Major Outage in prod-us-central-0 region
criticalSource not attested
Last archived state: resolved
View archived record details
investigating
The AWS Logs integration in the same region is affected as well. We will provide further updates as our investigation progresses.
investigating
We are currently investigating a major outage in Loki writes and Frontend Observability in the prod-us-central-0 region. Our Engineering team is investigating this and we will provide further updates as our investigation progresses.
Elevated Loki Query Bytes Reporting
minorSource not attested
Last archived state: resolved
View archived record details
investigating
We are investigating an issue where some customers may see higher Loki query byte usage reported than was actually consumed. This affects usage reporting only; there is no impact to query execution or service availability.
The issue began at approximately 13:20 UTC and is ongoing. We expect the issue to be resolved soon and will provide another update as more information becomes available.
June 2026
Confluent API Outage
majorSource not attested
Last archived state: resolved
View archived record details
investigating
We are investigating an issue affecting Confluent metrics ingestion across all regions. Due to an elevated error rate on the Confluent side, some metrics may not be ingested, resulting in potential data loss. We are actively investigating the issue and will provide updates as more information becomes available.
Mimir read errors and high latency in prod-eu-west-0
minorSource not attested
Last archived state: resolved
View archived record details
identified
We've identified a possible cause, and a mitigation is in place to prevent further occurrences.
investigating
The errors and latency have now recovered, we continue investigating the root cause.
investigating
The errors are recovering, and we are still looking into the root cause of this.
investigating
We are currently investigating an issue with Mimir in prod-eu-west-0 we are seeing read errors and high latency. This incident is currently ongoing. The errors are recovering but we are currently looking into the route cause of this.
Rule evaluation error on cluster prod-gb-south-0
minorSource not attested
Last archived state: resolved
View archived record details
monitoring
A fix has been applied and we are currently monitoring results.
investigating
We are currently investigating Rule Evaluation errors on the cluster prod-gb-south-0 which is leading to error codes showing within the stacks. We are looking into the issue and will update accordingly.
K6 - Test run metrics processing is delayed
minorSource not attested
Last archived state: resolved
View archived record details
investigating
We've improved the metric ingestion delay time and are working on additional fixes to bring it down to expected range. Customers can currently expect a delay of 2 to 5 minutes before their test run metrics show up (after starting a test).
investigating
Update: Changed incident title to "Test run metrics processing is delayed"
We have found the issue and are working on deploying the fix.
investigating
We are experiencing intermittent delays with secondary metrics processing for k6 Cloud test runs due to heavy load. We don't expect any data loss or impact on user runs, but results may take longer time to appear in UI.
investigating
A small update: The issue is isolated to new test runs, and users can go see the metrics of all the previous test runs
investigating
We’re currently investigating an issue causing metrics not to appear during test runs. . Our team is actively working to identify the cause. Thank you for your patience.
