feat(adk): add AfterAgent method to ChatModelAgentMiddleware#735
Open
shentongmartin wants to merge 6 commits intoalpha/08from
Open
feat(adk): add AfterAgent method to ChatModelAgentMiddleware#735shentongmartin wants to merge 6 commits intoalpha/08from
shentongmartin wants to merge 6 commits intoalpha/08from
Conversation
…722) * feat: add enhanced tool support with multimodal output capabilities and improve message formatting This commit introduces enhanced tool interfaces that support structured multimodal outputs, enabling tools to return rich content beyond simple text responses. Key Changes: 1. New Enhanced Tool Interfaces: - Added EnhancedInvokableTool and EnhancedStreamableTool interfaces for multimodal tool execution - Both interfaces use ToolCallInfo as input and return ToolResult for structured output 2. ToolResult Schema: - Introduced ToolResult type to represent multimodal tool outputs - Supports multiple content types: text, image, audio, video, and file - Added ToolOutputPart with Index field for streaming chunk merging - Implemented ToMessageInputParts() for seamless model integration 3. ToolsNode Enhancements: - Extended ToolsNode to support both legacy and enhanced tool types - Added automatic conversion between invokable and streamable endpoints - Implemented middleware support for enhanced tools - Enhanced interrupt and rerun mechanism to handle ToolResult 4. React Agent Integration: - Introduce enhancedToolResultSender and enhancedStreamToolResultSender types - Support sending *schema.ToolResult with multimodal content (images, audio, video, files) - Implement EnhancedInvokable and EnhancedStreamable middleware in tool result collector 5. Message.String() Enhancement: - Add formatting support for UserInputMultiContent, AssistantGenMultiContent, and MultiContent - Implement formatInputPart, formatOutputPart, and formatChatMessagePart helper functions - Create mediaPartFormatter interface with wrapper types for unified media formatting 6. User Input Multi-Content Concatenation: - Implement concatUserMultiContent function for merging MessageInputPart slices - Support text and base64 audio merging with proper MIME type handling - Integrate into ConcatMessages function 7. Callback System: - Added CallbackInput and CallbackOutput types for tool callbacks - Implemented conversion functions for different callback input/output types 8. Comprehensive Test Coverage: - Added tests for enhanced invokable and streamable tools - Added TestMessageString with 14 test cases covering various message types Impact: - Enables tools to return rich multimodal content (images, audio, video, files) - Provides foundation for more sophisticated tool implementations - Maintains full backward compatibility with existing tool ecosystem
Add AfterAgent method that allows middleware to perform cleanup operations after the agent completes execution while RunLocalValues are still accessible. Changes: - Add AfterAgent method to ChatModelAgentMiddleware interface - Add default no-op implementation to BaseChatModelAgentMiddleware - Add hasAfterAgent field to handlerInfo for optimization - Implement AfterAgent for no-tools path in chatmodel.go - Implement AfterAgent for react path with shared afterAgentNode - Add comprehensive tests for AfterAgent functionality~ Change-Id: I2164ae4c5ae7b2ce1cee5dbbc20f087008dd5fe7
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## alpha/08 #735 +/- ##
===========================================
Coverage ? 80.75%
===========================================
Files ? 132
Lines ? 13638
Branches ? 0
===========================================
Hits ? 11013
Misses ? 1812
Partials ? 813 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Change-Id: I6fba41e947c047a0874860b4c11cd2ff3daa7fc6
8b3de42 to
b1fbde2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Middleware authors need to perform cleanup operations after the
ChatModelAgentcompletes execution. For example, during execution, a middleware might store temporary file paths inRunLocalValues. When the agent finishes, the middleware needs to delete these temporary files.The challenge:
GetRunLocalValueonly works while the graph is executing. Once the graph exits, the state context is lost andRunLocalValuesbecome inaccessible.Solution
Add
AfterAgentmethod toChatModelAgentMiddlewareinterface:This method is called before the graph exits, so
GetRunLocalValuestill works. Thestateparameter contains all messages including the final one.Graph topology for react path:
Key Insight
The
stateModelWrapperalready appends the model response toState.Messagesand writes it back viaProcessStatebefore any post-processing. This means inAfterAgent, we can simply readState.Messagesdirectly - no need to copy/concatenate the stream ourselves.Initial implementation copied and concatenated the stream to get the final message:
But this was redundant and would have duplicated the final message. The simplified implementation just reads from state:
Summary
RunLocalValuesafter agent execution because state is lost when graph exitsAfterAgentmethod that is called before graph exits, whileGetRunLocalValuestill works