Skip to content

fix(fns): handle BinaryImage/UserContent types as attachments in extract() and classify()#1340

Open
octo-patch wants to merge 1 commit into
PrefectHQ:mainfrom
octo-patch:fix/binary-image-token-explosion-extract-classify
Open

fix(fns): handle BinaryImage/UserContent types as attachments in extract() and classify()#1340
octo-patch wants to merge 1 commit into
PrefectHQ:mainfrom
octo-patch:fix/binary-image-token-explosion-extract-classify

Conversation

@octo-patch
Copy link
Copy Markdown

Summary

Extends the UserContent attachment handling (similar to #1257 for cast()) to extract() and classify().

Problem

When passing pydantic-ai UserContent types (BinaryImage, ImageUrl, AudioUrl, DocumentUrl, VideoUrl) to marvin.extract() or marvin.classify(), the binary data is serialized into the task context as a string, causing token explosion (200K+ tokens for moderately-sized images).

Example that fails with token explosion:

from pathlib import Path
from pydantic_ai.messages import BinaryImage
import marvin

image_bytes = Path("image.png").read_bytes()
binary_image = BinaryImage(data=image_bytes, media_type="image/png")

# Token explosion: binary data serialized into context
tags = marvin.extract(binary_image, target=str, instructions="List objects in the image")
label = marvin.classify(binary_image, labels=["safe", "unsafe"])

Solution

Added detection for pydantic-ai's UserContent types in extract() and classify() — the same fix applied to cast() in #1257. Binary/URL content types are now passed as task attachments instead of being serialized into the context dictionary.

Testing

  • Added unit tests for extract() with BinaryImage and ImageUrl inputs
  • Added unit tests for classify() with BinaryImage and ImageUrl inputs
  • All 6 new tests pass

…act() and classify()

When passing pydantic-ai UserContent types (BinaryImage, ImageUrl, AudioUrl, etc.)
to marvin.extract() or marvin.classify(), the binary data was serialized into the
task context as a string, causing token explosion (200K+ tokens for moderately-sized
images). This is the same issue as PrefectHQ#1246 which was already fixed for cast().

Fixes the same class of bug in extract() and classify() by detecting UserContent
attachment types and passing them as task attachments instead of context values.
@github-actions github-actions Bot added the tests label Apr 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant