Incident report for September 18, 2026 - When Deletion Cascades Exhausted Our Database Connections

All timestamps are in UTC.

Summary

On September 18, 2026, an account deletion originating from our Vercel Marketplace integration blocked our primary Postgres database for long enough to exhaust PgBouncer, the connection pooler that sits in front of it. From 16:18 to 16:42 UTC, both PgBouncer instances were rejecting new connections, and services across the platform could not reach the database. This stalled run scheduling and event acknowledgement. A second period of impact ran from 17:50 to 18:12 UTC, while we were rolling out the mitigation, and reached more services than the first.

Events we accepted during the incident were durably stored, and scheduling caught up once the blocking transactions cleared: affected work was delayed rather than lost. If you sent an event and did not receive an acknowledgement, retry it to confirm it was submitted.

What Happened

We use Postgres for account management: accounts, workspaces and the rest of our customer-facing configuration. Our services query it periodically, resolving entitlements and tier changes, determining which part of our infrastructure an account's work should run on, and picking up configuration changes as they happen. They reach it through PgBouncer, which multiplexes many application connections onto a smaller number of real database connections, and every service draws from the same shared pools.

At 15:39, our Vercel Marketplace integration received an installation deletion and began deprovisioning the corresponding Inngest account. That path was built to hard delete, rather than to soft delete the way we do everywhere else, so that an uninstalled account leaves nothing behind that would block the customer from installing again later. It deletes the user inside a transaction, then issues a real DELETE against the account row. That second statement cascades: accounts and workspaces are referenced by foreign keys from dozens of tables, most declared ON DELETE CASCADE. One account deletion fans out across a large portion of the schema in a single transaction, holding locks on everything it touches until it commits.

The retries are what turned a slow transaction into an outage. Between 15:41 and 15:56, repeated deletion attempts arrived for the same installation and account. Each tried to delete rows the first transaction already held locks on, so instead of failing fast they queued behind it.

Blocked queries do not release their connection, so each one held a client slot in PgBouncer for the entire wait. Client connections rose to roughly fourteen times their steady-state level, almost all of them doing nothing but waiting. By 16:18–16:19 PgBouncer was full, rejecting new connections and disconnecting waiting clients:

FATAL: no more connections allowed (max_client_conn) (SQLSTATE 08P01)

At that point the failure was platform-wide, including services with no relationship to the deletion. Run scheduling volume dropped, inbound events stopped being acknowledged, Connect gateway and Constraint API throughput fell, and several services crash-looped because they could not reach the database on startup.

We restarted PgBouncer at 16:26. The blocking transactions were still running, so the pools simply refilled. The first period of impact ended instead at 16:39, when we cancelled the queued user deletions along with the original account deletion. Lock counts cleared by approximately 16:42.

The recurrence during rollout

Cancelling the queries removed the symptom but not the source: deletion requests were still arriving and still being processed. We prepared a kill switch to stop the deletion path outright.

At 17:45 the same lock-contention pattern began again, and by 18:00 PgBouncer was saturated a second time. The kill switch had merged at 17:54 and was rolling out into a system that was already contending. What triggered this second round of contention is still under investigation. This wave reached more services than the first: executor, batches, pauses, new-runs, queue-proxy, debug-api and CDC crash-looped, two queue shards stopped processing, and a batches backlog built up.

Postgres lock counts returned to baseline by 18:00, but PgBouncer stayed saturated for about ten minutes longer: the queue of waiting clients had to drain before services could reconnect. Recovery completed at approximately 18:12, and by 18:15 no clients were waiting and query times were back to normal.

Timeline

• 15:39: A Vercel Marketplace installation deletion begins. The user is deleted inside a transaction, and the cascading account delete starts.
• 15:41–15:56: Multiple further deletion attempts arrive for the same installation and account. They queue behind the first transaction.
• 16:09: PgBouncer client queues begin growing.
• 16:16: Database lock counts peak well above baseline.
• 16:18–16:19: Our PgBouncer instances fill and begin rejecting connections and disconnecting clients.
• 16:21: We declare an incident and page on-call.
• 16:22: max_client_conn errors are confirmed as the immediate failure.
• 16:26: We restart the PgBouncer instances. The pools refill, because the blocking transactions are still running.
• 16:31: We confirm sessions blocked on locks, including the long-running account and user deletion queries.
• 16:39: We cancel the queued user deletions and the original account deletion.
• 16:42: Lock counts clear and services begin recovering.
• 16:51: First period of impact confirmed recovered.
• 17:45–17:50: Lock contention and long-running queries return. Waiting clients and query times climb again.
• 17:54: The Vercel Marketplace deletion kill switch merges and begins rolling out.
• 18:00–18:05: PgBouncer saturates a second time, marginally worse than the first peak.
• ~18:12: Recovery. Lock counts had returned to baseline by 18:00; the PgBouncer queue finished draining by 18:15.

Root Cause

The incident was caused by hard-deleting an account through a cascading foreign key graph while serving live production traffic, and then retrying that deletion while the first attempt was still holding locks.

Four factors combined:

1. The deletion is unbounded in scope. A single account hard delete cascades across dozens of tables in one transaction. The work done and the locks taken scale with how much data the account has, and neither is capped or checkpointed.
2. Retries amplified it instead of being absorbed. Repeated attempts targeted the same account. Because the work is not idempotent at the transaction level and there is no serialization in front of it, each retry queued behind its predecessor and extended the blocking window rather than replacing it.
3. Blocked queries consume pool capacity. A query waiting on a lock holds its PgBouncer client slot for the full wait. Lock contention on a handful of tables was therefore converted into total connection exhaustion for every service sharing the pool.
4. PgBouncer does not recover with the database. In the second event, the database returned to baseline roughly ten minutes before the platform did. PgBouncer saturation is a separate state that has to drain on its own, which means our recovery time is bounded by PgBouncer rather than by the underlying fault.

The third and fourth factors are what made this a platform-wide event with a long tail, rather than a slow background job. A shared connection pool with no isolation between workloads means one expensive write path can deny database access to everything else, and keep denying it after the original problem is gone.

What We're Doing Now

• Gating marketplace deletions. We have shipped an operational kill switch that stops Vercel Marketplace installation and resource deletion mutations. Deletion requests are still recorded as internal events with deterministic IDs, so nothing is lost and they can be reconciled once the safe path is in place.
• Moving marketplace deprovisioning to soft deletion. We are changing the marketplace path to follow the soft deletion principle we use everywhere else, retiring only the small number of unique records that would otherwise block a reinstall, rather than hard deleting the account and cascading through its data. This removes the expensive transaction entirely while preserving the reinstall behavior that motivated hard deletion in the first place.
• Making any remaining hard deletion incremental and serialized. Where a hard delete is genuinely required, we are moving it out of the request path into batched, rate-limited work, with deduplication so that repeated requests for the same account cannot stack.
• Rolling out changes into a quiet system. The kill switch landed while the database was already contending, which gave us no clean signal about whether it had taken effect. We are tightening how we sequence mitigations during an incident so that a fix and a recurrence cannot overlap the way they did here.
• Adding headroom and isolation in PgBouncer. We have adjusted PgBouncer configuration and are reviewing pool sizing so that a single saturating workload cannot consume the connection capacity that unrelated services depend on. We are also looking at how to drain a saturated pool deliberately rather than waiting for it to clear.
• Improving visibility into database contention. We are expanding our PgBouncer and Postgres dashboards and alerting so that rising lock counts and client wait times page us before the pool is exhausted, rather than after dependent services begin failing.

What This Means For You

• Run scheduling and event acknowledgement were interrupted in two windows: approximately 16:18 to 16:42, and 17:50 to 18:12.
• Events we accepted were durably stored; delivery and scheduling caught up as connections recovered.
• Runs scheduled during these windows were delayed while queued work caught up.
• If you sent an event during either window and did not receive an acknowledgement, retry it — an unacknowledged event may not have reached us.

We apologize for the disruption. An internal administrative operation should never be able to interrupt function execution for unrelated customers, and the changes above are aimed squarely at that separation.

If you have questions about this incident or need help with affected runs, please contact our support team.