Some Pruna models may not be able to scale up and are therefore either experiencing long queue times or entirely unavailable. We are working with Pruna to resolve the issue.
Some third party models as well as https://replicate.com/qwen/qwen-edit-multiangle are affected and are unable to scale out. Please use alternate models while we work with providers to publish fixed versions.
investigating
We are investigating an issue which is preventing Replicate models from failing to start new replicas.
Legacy Incident Archive
17 archived records restored. These records predate source-at-observation attestation. They are shown as historical ServiceAlert records only and do not establish a provider-reported incident, resolution, duration, or current active state.
August 2026
Delayed scaling due to node failure
minorSource not attested
Last archived state: resolved
View archived record details
monitoring
Scaling decisions were delayed by nearly 1 hour after the controller responsible for emitting queue metrics failed to schedule on a soft-failed node. We have since cordoned and drained the node, and the controller is emitting queue metrics again.
Significant degradation
Source not attested
Last archived state: resolved
View archived record details
monitoring
We have made significant progress in recovering the services, and most traffic should be flowing normal. We are monitoring to make sure all traffic is recovered before closing the incident
investigating
We are aware of a significant degradation with the service. We have identified the root cause and are taking steps to mitigate. Please hold tight for further information.
API degraded for A100s
Source not attested
Last archived state: resolved
View archived record details
identified
The API for creating predictions for A100s is currently degraded as a piece of backing infrastructure failed. We are working on getting it back online
July 2026
Degraded scale-out due to failed setups pulling from huggingface
minorSource not attested
Last archived state: resolved
View archived record details
monitoring
All affected hardware types have recovered and we are continuing to monitor and troubleshoot. Thanks for your patience!
identified
We have confirmed that turning off our caching layer for huggingface downloads allows models to successfully set up, so we are making that change for all models while we continue to troubleshoot.
identified
Models that pull from huggingface during setup have not been succeeding, which is resulting in scale-out delays and queue backups.
Hitting GPU Capacity for H100s creating large queue times for some models
minorSource not attested
Last archived state: resolved
View archived record details
monitoring
This issue is now resolved
monitoring
GPU usage is back below capacity. We're continuing to monitor.
identified
We are over provisioned on H100s currently which is causing long queue times for some models running on H100s
HuggingFace download issues
Source not attested
Last archived state: resolved
View archived record details
monitoring
Still seeing signs of improvement, most models are able to spin up with a few still waiting their turn from HuggingFace. Models should spin up after enough tries if they are failing for HuggingFace related reasons.
monitoring
We have seen most HuggingFace calls start succeeding, except for a small subset that are being downloaded through [cas-bridge.xethub.hf.co](http://cas-bridge.xethub.hf.co "cas-bridge.xethub.hf.co"). Most models are now succeeding to spin up
monitoring
HuggingFace is reporting recovery, which we are also seeing on our end. However, we still see minor degradation, which we believe is due to thundering herd once the main issue was resolved. We are still keeping an eye out, but believe we are recovering
investigating
We are aware of 504s being returned by models that reach out to HuggingFace during setup. We believe this is likely related to the disruption visible at [https://status.huggingface.co/](https://status.huggingface.co/ "https://status.huggingface.co/"). We are monitoring and will update when we can identify the HuggingFace CDN is back up/
H100 GPU shortage resulting in high queue times
minorSource not attested
Last archived state: resolved
View archived record details
monitoring
We pushed a fix and are monitoring
investigating
Long queue times, especially on BFL models
High contention on H100 hardware
minorSource not attested
Last archived state: resolved
View archived record details
monitoring
We are seeing high contention on H100 hardware which is resulting in delays on predictions and scale-out.
June 2026
Limited H100 capacity
minorSource not attested
Last archived state: resolved
View archived record details
identified
We recently received a sharp increase in demand for H100 capacity which is resulting in delayed scale-out and queue backup.
We're seeing long setup times and high contention for models on some L40S and H200 clusters.
