Prefect 500 error on usage of malformed block documents.

[PontyPandySam]

Bug summary

I recently spent way too much time debugging what looked like random 500 errors coming from the Prefect server.

The failures were happening in a pretty consistent pattern: deployment calls and some task calls that accepted a Pydantic model as an argument. Since Prefect is an optional dependency in our codebase, we sometimes strip block_type_slug out of the JSON so it’ll validate against our Pydantic models. This has worked fine for a while, but a recent change caused that modified object to get passed back into Prefect—both during scheduling and inside a task call.

What we eventually figured out is that if you:

1. Read a block document from Prefect
2. Remove block_type_slug from it
3. Pass that object back to Prefect as a parameter

…the Prefect server blows up with a 500 and this stack trace:

RuntimeError: dictionary changed size during iteration
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "<path>/venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1139, in __call__
    await super().__call__(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/applications.py", line 107, in __call__
    await self.middleware_stack(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "<path>/venv/lib/python3.12/site-packages/prefect/server/api/server.py", line 192, in __call__
    await self.app(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
    await self.app(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "<path>/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "<path>/venv/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/routing.py", line 462, in handle
    await self.app(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1139, in __call__
    await super().__call__(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/applications.py", line 107, in __call__
    await self.middleware_stack(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__
    await responder(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__
    await super().__call__(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__
    await self.app(scope, receive, self.send_with_compression)
  File "<path>/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "<path>/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "<path>/venv/lib/python3.12/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/routing.py", line 290, in handle
    await self.app(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/fastapi/routing.py", line 120, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "<path>/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "<path>/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "<path>/venv/lib/python3.12/site-packages/fastapi/routing.py", line 106, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "<path>/venv/lib/python3.12/site-packages/fastapi/routing.py", line 430, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/venv/lib/python3.12/site-packages/fastapi/routing.py", line 316, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/venv/lib/python3.12/site-packages/prefect/server/api/ui/schemas.py", line 41, in validate_obj
    schema = preprocess_schema(json_schema)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<path>/venv/lib/python3.12/site-packages/prefect/utilities/schema_tools/validation.py", line 260, in preprocess_schema
    for definition in definitions.values():
                      ^^^^^^^^^^^^^^^^^^^^
RuntimeError: dictionary changed size during iteration

Once you understand what’s happening, it kind of makes sense—but it took a frustrating amount of time to track down and reason about.

I’m writing this for two reasons. First, it would be awesome if Prefect returned a validation error (or literally anything more descriptive) instead of a super opaque 500. Second, I’m hoping this helps anyone else who runs into the same issue and ends up desperately googling it 😄

Version info

Version:              3.6.5
API version:          0.8.4
Python version:       3.12.3
Git commit:           857c60b5
Built:                Thu, Dec 04, 2025 08:01 PM
OS/Arch:              linux/x86_64
Profile:              local
Server type:          server
Pydantic version:     2.12.5
Server:
  Database:           sqlite
  SQLite version:     3.45.1
Integrations:
  prefect-aws:        0.5.13
  prefect-docker:     0.6.6

Additional context

No response

[zzstoatzz]

thanks for the well-written issue @PontyPandySam - this should be addressed by the PR linked above