Add Python tool call/result content types and update connectors and samples

[Copilot]

Motivation and Context

MEAI introduced new content types for tool calls/results (code interpreter, MCP server, image generation) and enhancements to hosted file metadata; Python core needed parity so connectors can emit/consume these shapes.

Description

• Types
  • Added CodeInterpreterToolCallContent, CodeInterpreterToolResultContent, ImageGenerationToolCallContent, ImageGenerationToolResultContent, MCPServerToolCallContent, MCPServerToolResultContent; extended HostedFileContent with optional media_type and name; updated Contents union and serialization.
  • FunctionApprovalRequestContent now accepts MCPServerToolCallContent in addition to standard function calls.
• Tools
  • Added HostedImageGenerationTool with typed options/additional properties and translate those options to OpenAI Responses parameters.
• Connectors
  • Updated OpenAI Responses parsing to emit paired tool call/result contents for code interpreter, image generation, and MCP server calls (streaming and non-streaming), even when outputs are empty, ensuring consistent pairing and typing; added option translation for image generation tools and removed raw dict image-generation remapping so callers rely on the tool type; MCP outputs now use the shared content parser.
  • Updated OpenAI Assistants to surface code interpreter and MCP server tool calls.
  • Updated Anthropic connector to emit MCP server and code interpreter tool call/result contents while keeping generic tool_use blocks as FunctionCallContent for non-code-execution tools, fixed missing ChatResponse/CitationAnnotation imports and code-block handling to satisfy type checks, and ensured MCP parsing/typing is consistent.
• Samples
  • Refreshed OpenAI image generation and code interpreter samples to use the new tool call/result contents and proper tool lists.
• Tests
  • Expanded core type/tool tests, OpenAI response/assistants tests, Anthropic tests, and added coverage for MCP server tool call acceptance in function approval requests; added guards to skip optional lightning tests when agentlightning is not installed.

Example:

call = CodeInterpreterToolCallContent(call_id="c1", inputs=[TextContent(text="print('hi')")])
result = CodeInterpreterToolResultContent(call_id="c1", outputs=[TextContent(text="log"), UriContent(uri="https://img")])
img_tool = HostedImageGenerationTool(options={"response_format": "uri", "image_size": "1024x1024"})

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>.NET: Python: add *ToolCallContent and *ToolCallResultContent</issue_title>
<issue_description>MEAI added several new content types: dotnet/extensions#6964 and dotnet/extensions#6664 and dotnet/extensions#6749

Below the .Net versions are linked to make it easy to reference.

CodeInterpreterContent

• Adds new CodeInterpreterToolCallContent and CodeInterpreterToolResultContent types
• Updates the OpenAI Assistants and Responses ChatClient implementations to produce them

Update HostedFileContent

• Updates HostedFileContent with an optional MediaType and Name, matching the corresponding properties on DataContent and UriContent.

HostedMCP

• Adds content types for MCP tool calls and results (HostedMcpServerToolCallContent, HostedMcpServerToolResultContent) to work in conjuction with HostedMCPTool

Image Generation

• Adds HostedImageGenerationTool
• Adds ImageGenerationToolCallContent and ImageGenerationToolResultContent

Computer Use
TBD
</issue_description>

<agent_instructions>Reference the dotnet links to figure out the structure and then implement them in the right way in the _tools.py for tools and _types.py for types. Make sure to also add the new content types to the Contents union typevar. There should be only python changes and they should be tested, so add test cases for all the types in test_types.py and for tools in test_tools. Everything should be inside the core package, except for some handling of these new types in different connectors, like Anthropic.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes Python: Add additional content types to match MEAI #2970

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[eavanvalkenburg]

looking good so far, there are also samples with image generation and code, can you update those as well @copilot

[Copilot]

looking good so far, there are also samples with image generation and code, can you update those as well @copilot

Updated the OpenAI image generation and code interpreter samples to use the new tool call/result content shapes and proper tool lists (1079faa).

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• api.openai.com
  • Triggering command: /usr/bin/python python -m pytest packages/core/tests/core/test_types.py packages/core/tests/core/test_tools.py packages/core/tests/openai/test_openai_responses_client.py (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[eavanvalkenburg]

some additional comments, and the McpServer contents also need handling, also those names should be MCP instead of Mcp (python standard vs dotnet standard) @copilot

[eavanvalkenburg]

couple of notes, and OpenAI Assistants also supports Code Interpreter, so that needs handling. Anthropic also support HostedMCPTool and Code interpreter, so those types need handling as well. @copilot

[eavanvalkenburg]

couple of notes, and OpenAI Assistants also supports Code Interpreter, so that needs handling. Anthropic also support HostedMCPTool and Code interpreter, so those types need handling as well. @copilot

@copilot please work on this

[Copilot]

some additional comments, and the McpServer contents also need handling, also those names should be MCP instead of Mcp (python standard vs dotnet standard) @copilot

Updated MCP content naming, parsing, and connector handling; also added option translation for image generation tools as requested (3866935).

couple of notes, and OpenAI Assistants also supports Code Interpreter, so that needs handling. Anthropic also support HostedMCPTool and Code interpreter, so those types need handling as well. @copilot...

Added code interpreter and MCP handling across Responses, Assistants, and Anthropic connectors to emit the new content types (3866935).

[eavanvalkenburg]

@copilot FunctionApprovalRequestContent also needs to allow MCPServerToolCallContent as the function_call inside of it.

[Copilot]

@copilot FunctionApprovalRequestContent also needs to allow MCPServerToolCallContent as the function_call inside of it.

FunctionApprovalRequestContent now accepts MCPServerToolCallContent (mapping or instance) as the function_call and keeps it typed (3904d09).

[eavanvalkenburg]

@copilot python/packages/core/agent_framework/openai/_responses_client.py:348 is not needed, folks should use the Tool type

[Copilot]

@copilot python/packages/core/agent_framework/openai/_responses_client.py:348 is not needed, folks should use the Tool type

Removed the raw dict image_generation remapping; callers should use the tool type directly (f7c856c).

[eavanvalkenburg]

@copilot a couple of tests failed, run uv run poe test to check which ones and fix them

[Copilot]

@copilot a couple of tests failed, run uv run poe test to check which ones and fix them

Added a guard to skip lightning tests when agentlightning isn’t installed and fixed the missing FunctionCallContent import; uv run poe test now passes (6825e70).

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• api.openai.com
  • Triggering command: /home/REDACTED/work/agent-framework/agent-framework/python/.venv/bin/pytest pytest --cov=agent_framework --cov-report=term-missing:skip-covered tests (dns block)
• test.search.windows.net
  • Triggering command: /home/REDACTED/work/agent-framework/agent-framework/python/.venv/bin/pytest pytest --cov=agent_framework_azure_ai_search --cov-report=term-missing:skip-covered tests (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/anthropic/agent_framework_anthropic
_chat_client.py	314	143	54%	242, 284, 315, 360–361, 370, 372–373, 375, 392–393, 432, 447, 451–452, 468, 480, 482, 486–487, 528–530, 532, 542–543, 549–551, 555–557, 561–564, 575, 577, 598, 608, 624–631, 635–636, 643–645, 653–657, 664–665, 671–672, 678–679, 684, 691–694, 698, 705–706, 712–713, 719–720, 725, 733–737, 744–745, 762, 769–770, 787, 807, 809, 818–819, 825, 833–835, 843–846, 855–866, 869–876, 882–889, 892–902, 905–908
packages/core/agent_framework
_tools.py	724	69	90%	226, 272, 323, 325, 353, 523, 555–556, 657, 659, 679, 697, 711, 723, 728, 730, 737, 770, 824–826, 864, 887, 889–898, 907–913, 949, 959, 1143, 1479, 1557–1561, 1682, 1751, 1844, 1850, 1893–1894, 1907–1908, 2037, 2078–2079, 2107–2109, 2151–2152, 2217–2218, 2225–2226
_types.py	1038	111	89%	131, 133, 135, 137, 139, 141, 148–149, 167–168, 305, 307, 314, 333, 373, 419–420, 456, 606, 720–721, 723, 748, 755, 772–774, 859, 864–865, 867, 874–875, 877, 904, 913, 916–918, 923–924, 931, 935–937, 1093, 1182–1185, 1193–1194, 1285, 1466, 1472, 1838, 1855, 1872, 1912–1914, 1920–1921, 1982, 2234, 2239, 2243, 2247, 2425–2427, 2439, 2490–2494, 2504, 2509, 2968, 3054–3056, 3129, 3140–3141, 3315, 3319, 3331–3333, 3434–3436, 3438–3440, 3443, 3447, 3450, 3455, 3500–3501, 3508–3509, 3543–3545, 3576, 3604, 3611
packages/core/agent_framework/openai
_assistants_client.py	217	33	84%	181, 194, 197, 203–204, 207, 210, 213–214, 225, 250, 252, 254, 256, 258, 261, 264, 267, 271, 282, 367, 386–387, 390, 398, 431, 467, 499–502, 554, 571
_responses_client.py	522	106	79%	141, 144, 146–147, 156, 174, 182, 185–192, 194, 199, 205, 209, 224, 227–228, 230, 267, 297, 342–343, 365, 370, 404, 422, 425, 454, 508, 584, 589, 593–595, 615, 630–631, 635–637, 684, 704–705, 718–719, 735–736, 814–815, 824, 827–829, 835–836, 854–855, 877–878, 894, 896, 970–978, 1001–1007, 1026, 1041–1042, 1051, 1056–1058, 1064–1065, 1078, 1090, 1121–1122, 1124–1126, 1137–1138, 1141, 1143, 1154–1155, 1161, 1176
TOTAL	17021	2695	84%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
2563	148 💤	0 ❌	0 🔥	59.028s ⏱️

[Copilot]

Pull request overview

This PR adds Python parity with .NET's MEAI by introducing new content types for tool calls and results (code interpreter, MCP server, image generation), extending HostedFileContent with optional metadata, and updating connectors to emit these specialized types.

Key Changes:

• Added six new content types: CodeInterpreterToolCallContent, CodeInterpreterToolResultContent, ImageGenerationToolCallContent, ImageGenerationToolResultContent, MCPServerToolCallContent, MCPServerToolResultContent
• Added HostedImageGenerationTool with typed options for image generation parameters
• Updated OpenAI Responses and Assistants clients to emit paired tool call/result contents and surface specialized tool types
• Updated Anthropic connector to distinguish MCP server and code interpreter tool calls from generic function calls

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
python/packages/core/agent_framework/_types.py	Added six new content types for code interpreter, image generation, and MCP server tool calls/results; extended HostedFileContent with optional media_type and name; updated Contents union and _parse_content function; modified FunctionApprovalRequestContent to accept MCPServerToolCallContent
python/packages/core/agent_framework/_tools.py	Added HostedImageGenerationTool with typed options; updated _auto_invoke_function to handle MCPServerToolCallContent
python/packages/core/agent_framework/openai/_responses_client.py	Updated response parsing to emit paired call/result contents for code interpreter, image generation, and MCP server; added HostedImageGenerationTool conversion with option mapping; removed raw dict parameter remapping for image generation
python/packages/core/agent_framework/openai/_assistants_client.py	Added support for code interpreter and MCP tool calls in assistants API by detecting tool types and creating appropriate content types
python/packages/anthropic/agent_framework_anthropic/_chat_client.py	Updated to emit MCPServerToolCallContent for mcp_tool_use and CodeInterpreterToolCallContent for code execution tools; changed mcp_tool_result to emit MCPServerToolResultContent
python/packages/core/agent_framework/_workflows/_handoff.py	Updated type signatures to accept MCPServerToolCallContent in addition to FunctionCallContent for handoff resolution
python/samples/getting_started/agents/openai/openai_responses_client_with_code_interpreter.py	Updated to use new CodeInterpreterToolCallContent and CodeInterpreterToolResultContent types to extract code and outputs
python/samples/getting_started/agents/openai/openai_responses_client_image_generation.py	Updated to use HostedImageGenerationTool and ImageGenerationToolResultContent for cleaner image generation handling
python/packages/core/tests/core/test_types.py	Added tests for new content types including HostedFileContent optional fields, code interpreter, image generation, and MCP server tool content types
python/packages/core/tests/core/test_tools.py	Added tests for HostedImageGenerationTool with default and custom options
python/packages/core/tests/openai/test_openai_responses_client.py	Updated tests to verify paired call/result content emission, added HostedImageGenerationTool conversion test, updated parameter handling tests
python/packages/ollama/pyproject.toml	Fixed typo: changed "extend" to "extends" for pyright configuration
python/packages/lab/lightning/tests/test_lightning.py	Added pytest.importorskip guard to skip tests when agentlightning is not installed

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.