Skip to content

[FIX] [litellm_wrapper] omit empty text block for image-only vision calls - #1712

Closed
ebarkhordar wants to merge 1 commit into
kyegomez:masterfrom
ebarkhordar:fix/vision-empty-text-block
Closed

[FIX] [litellm_wrapper] omit empty text block for image-only vision calls#1712
ebarkhordar wants to merge 1 commit into
kyegomez:masterfrom
ebarkhordar:fix/vision-empty-text-block

Conversation

@ebarkhordar

Copy link
Copy Markdown

What

LiteLLM's vision message builders always prepended a text content block, even
when the task was empty or None. An image-only call such as llm.run("", img=...)
or run(None, img=...) therefore built a user message containing
{"type": "text", "text": ""} next to the image block.

Root cause

anthropic_vision_processing (both the direct-URL and base64 branches) and
openai_vision_processing unconditionally start the content list with
{"type": "text", "text": task}. When task is empty or None, that block is
empty.

Invariant

No content block in a vision message should carry empty text. The wrapper already
enforces this for system blocks in _prepare_messages, which drops empty system
blocks with the note that Anthropic rejects them ("system: text content blocks
must be non-empty"). The vision user-text block was the one spot in the same
family that was not guarded.

Fix

Add a small _vision_content(task, image_block) helper that includes the text
block only when task is a non-empty (non-whitespace) string, then route all
three construction sites through it. With no task the message carries just the
image block; with a task, behavior is unchanged. This mirrors the existing
empty-system-block normalization.

Verification

Ran in a clean Docker container (python:3.12-slim, litellm==1.76.1), offline
(--network none), against this branch and against master:

  • New test tests/utils/test_litellm_vision_empty_text.py: the 7 image-only
    cases (empty / whitespace / None task, Anthropic and OpenAI paths, plus a
    forced direct-URL case) FAIL on master (the empty {"type": "text", "text": ""}
    block is present) and PASS on this branch.
  • 2 over-deletion guards (a real task keeps its text block) pass on both sides,
    so the fix does not drop legitimate text.
docker run --rm --network none -e PYTHONPATH=/src -e WORKSPACE_DIR=/tmp \
  -v <clone>:/src -w /src <image> \
  python -m pytest tests/utils/test_litellm_vision_empty_text.py -v

The tests assert the message structure the wrapper builds. I did not make a live
Anthropic API call, so I have not personally observed the rejection; the premise
rests on Anthropic's documented "text content blocks must be non-empty"
constraint and on this repo's own existing system-block guard for it.

Black (24.2.0) and Ruff (0.2.1) are clean on both changed files.

Prepared with AI assistance; reproduced and verified as described above.

@github-actions

Copy link
Copy Markdown

Hello there, thank you for opening an PR ! 🙏🏻 The team was notified and they will get back to you asap.

…alls

The vision message builders always prepended {"type": "text", "text": task},
so an image-only call (empty or None task) produced a user message containing
an empty text content block. Anthropic rejects empty text blocks; the wrapper
already drops empty system blocks in _prepare_messages for the same reason,
and the vision user-text block was the ungated case in that family.

Add a _vision_content(task, image_block) helper that includes the text block
only when task is a non-empty string, and route the anthropic (direct-URL and
base64) and openai vision builders through it. Behavior is unchanged when a
task is present.
@ebarkhordar
ebarkhordar force-pushed the fix/vision-empty-text-block branch from d4879ba to cc9a573 Compare July 13, 2026 08:44
@ebarkhordar

Copy link
Copy Markdown
Author

No rush on this. It is a small litellm_wrapper fix (drop the empty text block for image-only vision calls, since the provider rejects a bare empty content block), and I am happy to rebase or add a test if that would help you review.

@ebarkhordar

Copy link
Copy Markdown
Author

Closing this to keep my open PR list at a size I can actually shepherd. Nothing here is
blocked on anyone and this is not a nudge.

The change still applies if it is useful later, and I am happy to reopen and rebase it
at any point. The branch stays up so reopening is a click.

@ebarkhordar ebarkhordar closed this Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant