Enhancing Multimodal Support Capabilities

[s97712]

Currently, our read_file function is limited to processing text content only. This significantly restrains the capabilities of our Agents. If our Agents could support processing various types of content, their abilities would be significantly enhanced.

Example:

Consider an Agent designed for frontend development. If it could generate and read snapshots of web pages to understand their true rendering effect—instead of just relying on code guesswork—this would greatly improve task efficiency and the quality.

Additional Notes on Implementation:

To achieve the aforementioned multimodal support, I prefer to introduce a new tool called read_media rather than directly modifying read_file.
This approach offers the benefit of allowing the Agent to assume the file type based on context and process it accordingly, without the need for complex file type recognition rules. It also helps in maintaining clearer tool responsibilities.

[Juice10]

Oh this is cool, that could allow the model to choose if it would like to read the text base contents of an SVG or the visual contents of it "view it".

Thanks for sharing @s97712!

[chrarnoldus]

Roo also has a PR for this: RooCodeInc/Roo-Code#5262

[ricable]

That would be great