PostgresAI / DBLab - Authenticated app/API outage on console.postgres.ai – Incident details

Third Party: Cloudflare → Cloudflare Sites and Services → Network experiencing degraded performance

Authenticated app/API outage on console.postgres.ai

Resolved
Major outage
Started 9 days agoLasted about 10 hours

Affected

Platform/Console

Major outage from 2:25 AM to 11:59 AM

Updates
  • Resolved
    Resolved

    Impact: Signed-in users could not use the console; all authenticated API requests failed. Anonymous/marketing pages were unaffected.

    What happened: A database security-hardening change revoked an execute privilege the authenticated API depended on (PostgREST pre-request hook), so every logged-in request failed. The issue is fully resolved and no data was lost.

    What we're doing to prevent recurrence:

    • Adding end-to-end authenticated monitoring in addition to the current non-authenticated one.

    • Adding backend error-rate alerting that pages us on a spike of failed requests.

    • Taking the lesson from this: the outage came from a change intended to strengthen our security. We're strengthening our database-change process with additional pre-production testing and review safeguards.

    We're sorry for the disruption and the delay in detection.

  • Investigating
    Investigating

    Impact: Signed-in users cannot use the console; all authenticated API requests fail. Anonymous/marketing pages are unaffected.