Impact: Signed-in users could not use the console; all authenticated API requests failed. Anonymous/marketing pages were unaffected.
What happened: A database security-hardening change revoked an execute privilege the authenticated API depended on (PostgREST pre-request hook), so every logged-in request failed. The issue is fully resolved and no data was lost.
What we're doing to prevent recurrence:
• Adding end-to-end authenticated monitoring in addition to the current non-authenticated one.
• Adding backend error-rate alerting that pages us on a spike of failed requests.
• Taking the lesson from this: the outage came from a change intended to strengthen our security. We're strengthening our database-change process with additional pre-production testing and review safeguards.
We're sorry for the disruption and the delay in detection.