Root Cause Analysis: July 2026 Service Incidents

All timestamps are in UTC.

In July 2026, Inngest experienced four incidents: one on July 10 that degraded function execution and run scheduling, two on July 16 — one that caused increased execution latency and checkpointing errors, and one that disrupted publishing of customer execution metrics — and one on July 23 that caused an outage in event processing and delayed function execution.

We know our customers depend on Inngest as critical infrastructure, and we apologize for the disruption these incidents caused. This report covers what happened in each incident, the root causes, and the corrective actions we have taken and are still working on.

Incident 1 — July 10: Degraded function execution and run scheduling

Summary

On July 10, 2026, a disruption at one of our third-party infrastructure providers caused our feature-flag system to stop returning live values. As a result, a release deployed that day fell back to default configuration values across several of our services.

These fallback values changed how our internal services routed traffic and scheduled work, which in turn degraded our internal network and disrupted function execution and run scheduling. Separately, a code regression in the same release caused a subset of retried requests to fail. We rolled back the release, which coincided with the third-party provider restoring service, and the majority of impact was resolved.

Function execution was degraded for 60 minutes (16:45–17:45 UTC) and run scheduling for 125 minutes (16:10–18:15 UTC).

What happened

Connectivity to the third-party service that supplies our feature-flag values degraded. Our services were unable to fetch flag values, both on startup and via streaming updates, so all affected feature flags reverted to their default states.

When we rolled out a new release that day, newly started containers were unable to retrieve feature-flag values and fell back to default configuration. Two consequences followed:

• A subset of internal traffic switched from its optimized gRPC transport path to a legacy internal HTTP path, which overloaded our internal network. The legacy path had been kept in place as a per-customer opt-out in case the gRPC path caused issues. Between 16:45 and 17:47 UTC, network performance was severely degraded: DNS query volume increased roughly tenfold, DNS lookup latency spiked, and connection setup times rose sharply. Traffic on a key internal path effectively stalled, breaking function execution.
• Another subsystem was configured incorrectly, leading to retries in run scheduling for a subset of customers using debounce.

In parallel, the same release contained a regression affecting certain idempotent requests on the scheduling path. During retries, this caused a subset of events to repeatedly fail until the release was rolled back.

Customer-facing impact:

• Function execution was delayed due to impaired internal connectivity. A subset of steps failed between 16:45 and 17:45 UTC. These were retried automatically, except for function runs with disabled or exhausted retries.
• Run scheduling was delayed due to the scheduling errors and the request regression on retries.
• Some outbound step requests (step.fetch()) returned empty responses, which may have disrupted dependent application logic.
• Some customers may need to replay affected events to ensure the associated functions execute as expected.

Timeline (UTC, July 10, 2026)

Time	Event
16:10	Customer impact began. A subset of events began failing on retry due to the request regression on the scheduling path.
16:19	First feature-flag initialization errors observed as containers failed to fetch flag values.
16:20	Scheduling errors and retries started for a subset of customers.
16:45	Internal network performance began degrading severely as traffic shifted to the legacy path.
17:40	Release rolled back. Retries on the affected request path stopped failing.
17:41	Internal gateway metrics began returning toward normal.
17:45	Function execution impact ended.
17:47	Internal network performance recovered.
17:57	Rollback fully propagated; status moved to Monitoring.
18:04	Last feature-flag initialization errors observed.
18:15	Scheduling returned to normal; run-scheduling impact ended.
18:44	Incident resolved; throughput confirmed back to normal.

Duration of impact:

• Function execution: ~60 minutes (16:45–17:45 UTC).
• Run scheduling: ~125 minutes (16:10–18:15 UTC).

Root cause

The primary cause was the loss of connectivity to a third-party feature-flag service. When flag values could not be fetched, our services fell back to default configuration. Two of those defaults had significant downstream effects: one rerouted internal traffic onto a legacy transport that overloaded our network, and one triggered a configuration migration that sent scheduling work to the wrong location.

A contributing factor was a code regression in the same release that caused a subset of idempotent scheduling requests to fail during retries, independent of the feature-flag issue.

Follow-ups

Already completed:

• Switched a subset of feature flags to static values, removing the dependency on the third-party provider for those critical paths.
• Added instrumentation for feature-flag related issues to detect this class of failure faster.

Scheduled and in progress:

• Expand networking alerts. Add alerting for CPU and network throughput on the internal network, elevated connection-reset rates, resource-pool saturation, and the internal network-translation layer, so rapid changes in traffic and connectivity are caught earlier.
• Improve observability. Improve run-scheduling observability to detect events that did not produce runs, implement event auditing (including when an event was intentionally debounced or rate-limited).
• Improve resilience of internal tooling and configuration. Reduce the blast radius of the internal network-translation layer, harden deployment tooling so rollbacks are not delayed during incidents, and complete planned upgrades of the internal gateway infrastructure.

Incident 2 — July 16: Increased execution latency and checkpointing errors

Summary

On July 16, 2026, Inngest experienced an incident that caused increased execution latency and elevated errors for customers.

The issue occurred while we were preparing our state store infrastructure for upcoming upgrades. Our services access the state store through a proxy layer, which routes traffic between our current state store and the infrastructure we are upgrading to. As part of that work, we had enabled an upgrade-readiness state store path in this proxy layer behind a feature flag. That path depended on an internal state store connection during startup. While the path was covered by monitoring, it was tied to a medium-priority alert and a lower SLO than was appropriate for the customer impact this failure mode could create.

When that dependency began timing out, newly restarted state store proxy instances were unable to become ready. As more instances restarted, available proxy capacity dropped until the checkpointing API could no longer serve traffic reliably. We resolved the incident by disabling the upgrade-readiness state store path and restarting the affected proxy instances.

What happened

The checkpointing API depends on the state store proxy layer to persist intermediate function progress. During upgrade preparation, an upgrade-readiness path in that proxy layer was enabled for testing. We had tested this path extensively, including proxy restarts and several state store connection failure scenarios. The path was intended to validate readiness for future state store upgrades, but it introduced an availability dependency during proxy startup.

The system initially appeared healthy: existing proxy connections continued serving traffic, and the brief background errors we observed recovered quickly. The failure mode appeared later, as state store proxy instances were evicted and restarted over time. When those instances restarted, they had to establish fresh state store connections during startup. Those startup attempts timed out, and the instances were unable to become ready before being restarted again.

This did not affect traffic immediately, but over time the available proxy capacity degraded as instances restarted and failed to rejoin service. Once capacity dropped far enough, checkpointing requests from API services and some executors began failing.

For functions using checkpointing, SDKs were unable to persist intermediate step progress through the checkpointing API. When checkpoint writes failed, SDKs fell back to returning completed step results through the normal async response path. This fallback is designed to preserve completed step output where possible, but it is less efficient and more sensitive to retries than the checkpointed path.

Customer-facing impact:

• Customers experienced increased execution latency and elevated execution errors.
• Checkpointing requests failed for affected functions during the incident window.
• SDKs fell back to the async response path to prevent data loss when checkpoint writes failed. This increased retries and latency.
• Some functions, signals, or execution progress may have been delayed or failed during the incident window.

Timeline (UTC, July 16, 2026)

Time	Event
07:11	An upgrade-readiness state store path was enabled.
09:32	Customer impact began. State store proxy instances were evicted and rescheduled, then failed to become ready.
09:35	A medium-priority alert fired for the affected state store path.
11:40	Incident formally identified and response initiated.
12:10	Fix deployed by disabling the upgrade-readiness state store path and restarting proxies.
12:20	Incident resolved; checkpointing throughput and error rates confirmed back to normal.

Duration of impact:

• Checkpointing API and execution latency: ~168 minutes (09:32–12:20 UTC).

Root cause

The primary cause was a startup dependency in an upgrade-readiness state store path. When an internal state store connection began timing out, newly restarted state store proxy instances waited on that dependency before becoming ready. As a result, those instances could not serve traffic, and checkpointing API availability degraded as proxies restarted over time.

A contributing factor was that the upgrade-readiness path was monitored with medium-priority alerting and a lower SLO than was appropriate for its customer impact. The alerting did not reflect that this path could affect the checkpointing API when proxy instances restarted. Because the alert that fired at 09:35 UTC was medium-priority, it did not trigger immediate on-call escalation, which delayed formal identification of the incident until 11:40 UTC.

Another contributing factor was that our prior testing had covered proxy restarts and several connection-failure scenarios, but had not covered this specific timeout-at-startup behavior in the upgrade path. The system handled already-established connections as expected, but newly started proxy instances treated the timeout as a startup-blocking failure instead of continuing to serve traffic on the stable state store path.

Follow-ups

We have completed the immediate mitigation by disabling the upgrade-readiness state store path and restarting the affected proxy instances. Checkpointing throughput and error rates returned to normal after traffic moved back to the stable state store path.

We have also added safeguards before continuing this upgrade work:

• Removed the startup dependency that allowed one unhealthy state store path to prevent proxy instances from serving traffic.
• Moved monitoring for checkpointing API errors, proxy retry rates, and upgrade-readiness state store paths to higher-priority alerts with SLOs that better reflect customer impact.
• Added coverage for checkpointing API failures so customer-impacting errors are detected more directly.
• Added timeout-at-startup coverage to our upgrade validation so proxy restarts are tested against the same failure mode that occurred here.

Incident 3 — July 16: Delayed execution metrics

Summary

On July 16, 2026, starting at approximately 15:45 UTC, publishing of customer execution metrics to our metrics pipeline began to fail. During the incident window, dashboards showed no function execution or throughput metrics for affected customers. Function execution itself was not affected.

The underlying issue was drift between our live infrastructure configuration and its declarative definitions. The metrics publishing path had been introduced earlier this year as part of scaling our metrics pipeline, and the associated message-broker topic and access permissions were live in production ahead of their declarative definitions being completed. During maintenance to prepare the cluster for upcoming capacity work, broker resources were reconciled from those definitions, which unintentionally removed the not-yet-declared permissions. Metrics publishing began failing with authorization errors immediately afterward.

We restored the missing permissions, completed the declarative definitions for the topic and access permissions, and restarted the affected services. Publishing recovered and error volume dropped to zero.

What happened

Customer execution metrics are published through a message broker before being ingested into our analytics store. This path was introduced earlier this year as we scaled our metrics pipeline, and the broker topic and access permissions it relies on were live in production ahead of their declarative infrastructure definitions being completed.

During maintenance in preparation for upcoming capacity work, broker resources were reconciled from the declarative definitions. Because those definitions did not yet include the metrics topic permissions, the reconciliation removed them, and services lost authorization to publish execution metrics. Because we had not yet set up alerting on this publishing path, the failure was not detected until later, when the incident was formally declared.

Customer-facing impact:

• Dashboards showed no function execution or throughput metrics during the incident window.
• Execution metrics generated during the window were dropped and will not appear in historical metrics.
• Function execution, scheduling, and run state were not affected — only the metrics observability path.

Timeline (UTC, July 16, 2026)

Time	Event
15:45	Metrics publishing errors started; customer impact began.
17:44	Incident declared and investigation started.
17:55	Topic and access permission definitions reapplied; affected services restarted.
19:05	Publishing confirmed fully recovered and error volume at zero; incident resolved.

Root cause

The primary cause was infrastructure configuration drift: the message-broker topic and access permissions for the execution metrics path were live in production ahead of their declarative infrastructure definitions being completed. When broker resources were later reconciled from those definitions during maintenance, the permissions were removed and publishing failed.

Contributing factors:

• Execution metrics had recently moved to being published through the message broker rather than written directly to the analytics store, increasing the impact of this path.
• The publishing path had insufficient monitoring and alerting, which delayed detection.

Follow-ups

• Completed the declarative definitions for the metrics topic and access permissions, making them the source of truth.
• Added CI safeguards to detect drift between live broker configuration and the declarative definitions.
• Adding alerting on metrics publishing errors so failures on this path are detected immediately.
• Strengthened our rollout process so new infrastructure resources are declared and reviewed as part of their initial rollout.

Incident 4 — July 23: Event processing outage and delayed function execution

Summary

On July 23, 2026, starting at 14:03 UTC, the service responsible for processing incoming events and scheduling function runs stopped processing. Event processing was fully unavailable for approximately 37 minutes, and function execution was delayed until backlogs were fully processed at approximately 15:12 UTC.

The cause was a capacity-allocation failure in a storage layer that had recently been introduced to back a single message-broker topic, used for a new event lifecycle publishing path. When that storage layer stopped accepting writes, the brokers backing this topic became unavailable. The event-processing service required a connection to this path during startup, so restarting instances could not become ready and event processing halted.

We resolved the incident by disabling the event lifecycle publishing path to isolate the affected topic, and restarting and scaling the affected services. Event processing resumed at 14:40 UTC for most customers and by 14:53 UTC for all customers, and the resulting backlog was fully processed by approximately 15:12 UTC.

What happened

As part of ongoing infrastructure work, we had introduced a new storage backend for one message-broker topic used by a new event lifecycle publishing path. On July 23, that storage backend stopped allocating new capacity on the affected volume, even though free space was available elsewhere in the cluster. As a result, the brokers backing this topic became unhealthy and clients timed out connecting to them.

The event-processing service treated a connection to this publishing path as a startup requirement. As instances restarted, they failed to connect and could not become ready, which led to a full halt of event processing and run scheduling.

Automated alerts fired within minutes and the on-call team responded immediately. Once the root cause was identified, we disabled the event lifecycle publishing path, which isolated the affected topic and allowed the event-processing service to start up and resume processing. We then scaled the affected services to work through the backlog, and held new releases until throughput returned to normal.

Customer-facing impact:

• New events experienced delayed processing and new function runs were not scheduled between 14:03 and 14:40 UTC (14:53 UTC for a subset of customers on dedicated infrastructure).
• Events received during the outage were processed once service was restored; function runs were delayed until the backlog cleared at approximately 15:12 UTC.
• Function execution latency was elevated while backlogs were processed.

Timeline (UTC, July 23, 2026)

Time	Event
14:03	Customer impact began. Event processing and run scheduling stopped.
14:10	Automated alerts fired; on-call team began investigating.
14:16	Incident declared with Critical severity.
14:35	Root cause identified: capacity-allocation failure in the storage layer backing the event lifecycle topic.
14:40	Event lifecycle publishing path disabled; event processing resumed for most customers.
14:53	Event processing resumed for the remaining customers on dedicated infrastructure.
15:10	Affected path fully isolated; status moved to Monitoring.
15:12	Backlog fully processed; run scheduling latency back to normal.

Duration of impact:

• Event processing outage: ~37 minutes (14:03–14:40 UTC).
• Delayed run scheduling until full recovery: ~69 minutes (14:03–15:12 UTC).

Root cause

The primary cause was a capacity-allocation failure in a newly introduced storage backend for one message-broker topic. The affected volume reported no available capacity even though the wider storage cluster had ample free space, which made the brokers backing the topic unavailable.

A contributing factor was a startup dependency in the event-processing service: the event lifecycle publishing path is an auxiliary path, but a failure to connect to it prevented the service from starting at all. This turned the loss of a single non-critical topic into a full event-processing outage.

Full monitoring and alerting for capacity and health of this new Ceph-based storage backend was not completed. This caused the failure to surface via downstream impact rather than at the source.

Follow-ups

• Disabled the event lifecycle publishing path and paused the rollout of the new storage backend until the capacity-allocation failure is fully understood and addressed. Related planned work has been reverted.
• Adding monitoring and alerting for capacity and health of the storage layer backing message-broker topics, so allocation failures are caught before they affect brokers.
• Removing the startup dependency so the event-processing service can start and serve traffic even when the auxiliary publishing path is unavailable.

What This Means For You

• July 10 (16:10–18:15 UTC): If your functions rely on events sent during this window, some events may not have scheduled runs, and some steps may have received empty responses. We recommend reviewing activity during this window and replaying affected events where needed.
• July 16 (09:32–12:20 UTC): If your functions used checkpointing during this window, you may have seen delayed execution progress, failed checkpoint requests, additional retries, or function and signal-related failures.
• July 16 (15:45–19:05 UTC): Execution metrics for this window are missing from dashboards and cannot be backfilled. Function execution itself was not affected.
• July 23 (14:03–15:12 UTC): New function runs were not scheduled during the outage window (14:03–14:40 UTC, or until 14:53 UTC for a subset of customers) and were delayed until backlogs cleared. Events received during the window were processed after recovery; if your workloads are time-sensitive, we recommend reviewing runs triggered during this window.

If you would like help reviewing impact to your workloads during any of these windows, or identifying and replaying affected events, please reach out to our support team.