Started: Jun 3, 12:45 AM
monitoring
Refresh across all executions is nearing completion with no additional impact observed. Incident is fully mitigated and resolved.
Jun 16, 9:00 PM
monitoring
The last few namespaces are still refreshing. We are starting to patch the code, and are monitoring the situation.
Jun 16, 3:56 PM
monitoring
Refresh operations across both open and closed workflows are complete in most regions; we continue to make steady progress across the remaining few namespaces in the us-east-1, us-west-2, us-east4, us-west1, us-central1, asia-south1, europe-west3. No additional impact has been observed since mitigation was implemented, and we continue to closely monitor the environment. The next update will be provided once the refresh process has completed.
Jun 11, 8:25 PM
monitoring
Refresh across open and closed workflows continues to progress smoothly. Once this is done, the incident will be resolved.
Jun 8, 7:02 PM
monitoring
Refresh across open and closed workflows continues to progress smoothly. We're monitoring the situation and will post updates once complete.
Jun 5, 10:02 PM
monitoring
As refresh of closed workflows progresses, some users may see unblocked stuck parent workflows from closed child workflows; this is happening because the missed task for closed child workflows is refreshed and delivered to parent. It could also happen on other stuck workflows that are waiting for task updates which were missed due to this incident.
Jun 4, 10:35 PM
monitoring
We are continuing to monitor the ongoing refresh process for all open workflows. We are also considering options for refreshing closed workflows. We will continue to share more updates as this progresses.
Jun 4, 5:40 PM
monitoring
Workflows will continue to be refreshed overnight, and we are monitoring the situation.
Jun 4, 7:08 AM
identified
Workflows will continue to be refreshed overnight, and we are monitoring the situation.
Jun 4, 6:45 AM
identified
Refreshes are still in progress and we are monitoring the situation.
Jun 4, 6:14 AM
identified
Rollback is complete; refresh of open workflows is underway to unblock impacted workflows.
Jun 4, 4:27 AM
identified
The rollback is making progress across impacted cells, and we are continuing to monitor the situation.
Jun 4, 1:40 AM
identified
Rollback is still in progress and we are continuing to monitor the situation.
Jun 3, 11:36 PM
identified
A bug in the recent update in History pagination caused a rare edge case where an extra task could be added before the response was returned. This incorrectly advanced the read level, causing some tasks to be skipped. We’ve identified the issue, and are currently rolling back the update to stop this from happening again.
We will be refreshing all executions in cells where the update was deployed and will monitor these closely. This will progress any stuck workflows, but will not impact any...
Jun 3, 9:35 PM
investigating
We're rolling back the recent update as a possible mitigation to avoid any further occurrences of this issue. We are however still in the process of investigating root cause and will post updates as we have them.
Jun 3, 8:41 PM
investigating
We are investigating a recent update that could have caused intermittent missed task dispatches, resulting in some tasks being stuck. Impact identified as of now is limited to a handful of workflows that were caught in a rare edge case. We will continue to share more updates as investigation progresses.
Jun 3, 7:51 PM
investigating
We are continuing to investigate this issue. Please report any stuck workflows or activities to support so we can help unblock them as we work on root cause investigation and resolution.
Jun 3, 6:30 PM
investigating
We are investigating a few reports where a workflow or activity is becoming stuck and not making progress when it should. Please report any such issues you see to support.
Jun 3, 5:55 PM