apply_rls: one subscription whose RLS evaluation raises silently kills postgres_changes delivery for ALL subscribers

[Raundahl]

Bug report

One subscriber whose stored claims make RLS evaluation raise causes postgres_changes delivery to stop for every subscriber on the project — silently. We hit this in production on 2026-08-13 and lost all realtime delivery for ~2.5 hours, platform-wide.

Environment

• Hosted Supabase, eu-north-1 (project ref available via support ticket on request)
• Realtime 2.125.1 (slot supabase_realtime_replication_slot_2_125_1_*)
• Postgres 15, RLS enabled on all published tables

What happened

1. For ~2 hours (coinciding with a platform-side restart of our project's Auth + Realtime at ~05:04 UTC), GoTrue-issued access tokens carried the string "null" instead of JSON null in nullable app_metadata claims (e.g. app_id). Our access-token hook stamps proper JSON nulls and was byte-identical on our dev project, which was unaffected — the mangling was platform-side and stopped on its own at ~07:19 UTC. That is a separate Auth issue; it matters here only as the trigger.
2. Our RLS policies call a helper that does NULLIF(auth.jwt() -> 'app_metadata' ->> 'app_id', '')::uuid. With the claim being the string "null", this raises 22P02 invalid input syntax for type uuid: "null".
3. realtime.apply_rls() evaluates policies per subscription using the claims stored at subscribe time. When evaluation for one subscription raises, the error aborts processing of the WAL batch — and no subscriber receives anything, including service_role subscriptions and subscriptions on unrelated tables.

Observed symptoms

• WebSocket connects (101), channels reach SUBSCRIBED, rows appear in realtime.subscription with correct claims/filters — everything looks healthy.
• The replication slot stays active with confirmed_flush_lsn advancing to pg_current_wal_lsn() — WAL is consumed and acknowledged.
• Zero postgres_changes frames are delivered to any subscriber (verified with a service_role client subscribed with and without filters, on multiple tables).
• Reproduced directly: calling realtime.apply_rls() with a wal2json-shaped record while the poisoned subscriptions were present raises 22P02 with context SQL statement "execute walrus_rls_stmt"; after hardening our helper to not raise, the same call returns records and delivery resumed instantly for all subscribers.

Expected behavior

An error while evaluating RLS for one subscription should disqualify that subscription (arguably with an error surfaced to that channel), not abort delivery for every subscriber on the project. JWT claims are external input from the subscriber's perspective of the pipeline — any upstream serializer defect (as happened here, in the platform's own Auth layer) becomes a total, silent, project-wide realtime outage that no health signal exposes: sockets up, subscriptions registered, slot advancing, nothing delivered.

Workaround for anyone else hitting this

Make every function reachable from an RLS policy raise-proof against malformed claims — validate instead of casting:

CREATE OR REPLACE FUNCTION public.claim_uuid(p_claim text)
RETURNS uuid LANGUAGE sql IMMUTABLE SET search_path = '' AS $$
  SELECT CASE
    WHEN p_claim ~* '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'
    THEN p_claim::uuid
  END;
$$;

Diagnosis query that found our poisoned subscriptions, in case it helps others:

SELECT jsonb_typeof(claims->'app_metadata'->'app_id'), count(*)
FROM realtime.subscription GROUP BY 1;
-- 'string' where you expect 'null' = poisoned tokens

Happy to provide timestamps, the project ref, and full reproduction detail through a support ticket — please reference this issue.

[edgurgel]

Hi yes please let's open a support ticket!

[edgurgel]

Thanks for the report and I think you are right. We should not give up processing other subscriptions if that's what is happening.

[hamodywe]

I went looking at apply_rls after reading this and managed to reproduce it locally, so here is what I found in case it is useful.

The per-subscription check runs execute 'execute walrus_rls_stmt' with nothing around it, so when a policy raises the exception leaves apply_rls and takes the whole list_changes call with it. Reproduced on supabase/postgres:17.6.1.127 with three subscriptions on one table: two anon (one with app_metadata.app_id set to the string "null", one without the claim at all) and one authenticated whose policy is just using (true).

ERROR:  invalid input syntax for type uuid: "null"
CONTEXT:  SQL statement "execute walrus_rls_stmt"
PL/pgSQL function realtime.apply_rls(jsonb,integer) line 219 at EXECUTE
SQL function "list_changes" statement 1

The authenticated subscription cannot fail its own policy and still receives nothing, which lines up with what @Raundahl described.

One detail worth adding to the report: the batch is not retried either. confirmed_flush_lsn moves to pg_current_wal_lsn() even though the call errored, and the next poll comes back empty:

### poll 1
ERROR:  invalid input syntax for type uuid: "null"

  slot_name  | confirmed_flush_lsn | pg_current_wal_lsn
-------------+---------------------+--------------------
 verify_slot | 0/2EEB228           | 0/2EEB228

### poll 2 - is the record still there?
 subscription_ids | details
------------------+---------
                  |

So while a subscription with unevaluable claims exists, every decoded change is dropped rather than replayed, and nothing in the slot or connection state shows it.

I have a patch for this and will open a PR against it: the per-subscription check goes in its own block, the subscription that raised is disqualified on its own and gets an Error 500 row carrying only schema/table/action (the same shape as the existing 400/401 rows, so a policy that could not be evaluated does not end up disclosing the row it was meant to gate), and everyone else is unaffected. The extra subtransaction costs about 3% with 100 subscriptions on a single record, measured both ways on the same box.