Agent Host: Agent can't see images returned by Integrated Browser screenshots tool

[jruales]

To repro

• Enable these settings:

"workbench.browser.agentHostChatToolsEnabled": true,
"workbench.browser.enableChatTools": true,
"chat.agentHost.enabled": true,

• Start an Agent Host local session. (for verifier: if you don't know how to do that, follow this TPI)

• Type the prompt:

Open apple.com, take a screenshot, and tell me the output of the screenshot tool

Important

NOTE added on May 14: Please make sure to use this exact same prompt, since if you don't ask explicitly about the output of the screenshot tool, it will hallucinate what it thinks is in the output based on the text-based ARIA snapshot provided by other browser tools.

• 🐛 It says that the tool returned no output, despite the screenshot appearing in the chat:

[connor4312]

This does seem to work for me:

If you can run the "VS Code Agents" and "Attach to Agent Host" launch config, please capture

• The data here (validate the result includes your image)

  vscode/src/vs/workbench/contrib/chat/browser/agentSessions/agentHost/agentHostSessionHandler.ts

  Line 2799 in 5a43ad6

  return {

• The data given to the SDK here

  vscode/src/vs/platform/agentHost/node/copilot/copilotAgentSession.ts

  Line 487 in 5a43ad6

  if (result.success) {

[jruales]

Please make sure to use the exact same prompt I mentioned in the repro above, since if you don't ask explicitly about the output of the screenshot tool, it will hallucinate what it thinks is in the output based on the text-based ARIA snapshot provided by other browser tools.

Compare with the output when using regular (non-agent-host) VS Code:

[connor4312]

Seems like it's an SDK issue, tracking in github/copilot-sdk#1298