Partial outage (degraded check response and elevated 500 errors)

Write-up

Incident summary:

Oct 30, 2026, 00:44–00:52 UTC. A burst of unusually expensive bulk operations on one API endpoint (entities.delete) overloaded our primary database. API requests that needed the database slowed down or timed out. Some endpoints returned 500, 503 or 504 errors, mostly between 00:44 and 00:47 UTC.

Balance checks failed open. At the peak, around 00:47 UTC, up to 40% of check calls returned our fail-open response instead of a live balance: HTTP 202 with allowed: true and balance: null. No events were lost and all balances caught up.

We’ve since added stricter rate limits to the endpoint involved and an index to optimize its queries.

Write-up

Partial outage (degraded check response and elevated 500 errors)

Partial outage

View the incident

Incident summary:

We’ve since added stricter rate limits to the endpoint involved and an index to optimize its queries.