OpenAIMultiFunctionsAgent KeyError: 'actions' on custom tools

[gururise]

System Info

Langchain 0.0.209

The recent commit #6518 provided an OpenAIMultiFunctionsAgent class.

This MultiFunctions agent fails often when using Custom Tools that worked fine with the OpenAIFunctionsAgent.

File "/home/gene/endpoints/app/routers/query.py", line 44, in query3
    result = await agent.acall(inputs={"input":query.query})
  File "/home/gene/endpoints/venv/lib/python3.10/site-packages/langchain/chains/base.py", line 215, in acall
    raise e
  File "/home/gene/endpoints/venv/lib/python3.10/site-packages/langchain/chains/base.py", line 209, in acall
    await self._acall(inputs, run_manager=run_manager)
  File "/home/gene/endpoints/venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 1006, in _acall
    next_step_output = await self._atake_next_step(
  File "/home/gene/endpoints/venv/lib/python3.10/site-packages/langchain/agents/agent.py", line 853, in _atake_next_step
    output = await self.agent.aplan(
  File "/home/gene/endpoints/venv/lib/python3.10/site-packages/langchain/agents/openai_functions_multi_agent/base.py", line 301, in aplan
    agent_decision = _parse_ai_message(predicted_message)
  File "/home/gene/endpoints/venv/lib/python3.10/site-packages/langchain/agents/openai_functions_multi_agent/base.py", line 110, in _parse_ai_message
    tools = json.loads(function_call["arguments"])["actions"]
KeyError: 'actions'

Example tool that FAILS:

from typing import Optional, Type
from langchain.tools import BaseTool
from pydantic import BaseModel, Field
from langchain.callbacks.manager import AsyncCallbackManagerForToolRun, CallbackManagerForToolRun

class ProductInput(BaseModel):
    prod_name: str = Field(description="Product Name or Type of Product")

class CustomProductTool(BaseTool):
    name : str = "price_lookup"
    description : str = "useful to look up pricing for a specific product or product type and shopping url of products offered by the Company's online website."
    args_schema: Type[BaseModel] = ProductInput

    def _run(self, prod_name: str, run_manager: Optional[CallbackManagerForToolRun] = None) -> dict:
         # custom code here
         products = {}
         return products

    async def _arun(self, prod_name: str,  run_manager: Optional[AsyncCallbackManagerForToolRun] = None) -> dict:
        return self._run(prod_name)

Example tool that WORKS:

from typing import Optional, Type
from langchain.tools import BaseTool
from ..src.OrderStatus import func_get_order_status, afunc_get_order_status
from langchain.callbacks.manager import AsyncCallbackManagerForToolRun, CallbackManagerForToolRun
from pydantic import BaseModel, Field

class OrderInput(BaseModel):
    order_num: str = Field(description="order number")

class CustomOrderTool(BaseTool):
    name = "order_status"
    description = "useful for when you need to look up the shipping status of an order."
    args_schema: Type[BaseModel] = OrderInput

    def _run(self, order_num: str, run_manager: Optional[CallbackManagerForToolRun] = None) -> dict:
        # Your custom logic here
        return func_get_order_status(order_num)

    async def _arun(self, order_num: str, run_manager: Optional[AsyncCallbackManagerForToolRun] = None) -> dict:
        return await afunc_get_order_status(order_num)

Who can help?

@hwchase17

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

Instantiate a OpenAIMultiFunctionAgent:
agent = initialize_agent(tools,llm,agent=AgentType.OPENAI_MULTI_FUNCTIONS, verbose=True)

Create a custom tool (example above):

    tools=[
        CustomOrderTool(return_direct=False),
        CustomQAToolSources(llm=llm,vectorstore=general_vectorstore),
        CustomProductTool(return_direct=False),
        CustomEscalateTool(return_direct=False)
    ]

Call agent:

result = await agent.acall(inputs={"input":query.query})

Expected behavior

Tools are very similar to each other, not sure why one would work and the other fails. Might have something to do with the different description lengths? As far as I can tell the structure of the args_schema are the same between the two tools. Both tools work fine on OpenAIFunctionAgent.

I expected tools would work on OpenAIMultiFunctionAgent. Instead, KeyError: 'actions' results. Somehow the transformation of langchain tools to OpenAI function schema is not working as expected for OpenAIMultiFunctionAgent.

[dev2049]

could you provide a prompt to reproduce with?

do you know if this happens consistently across prompts? wondering if maybe the model is just outputting function args that don't match the given schema

[aayushchhabra1999]

I am getting the same issue. Did you find any solution?

[aleksanderhan]

Yeah, i get this all the time. It's totally unusable...

[liming922]

langchain==0.0.259

agent = initialize_agent(tools,
llm,
agent=AgentType.OPENAI_MULTI_FUNCTIONS,
verbose=True,
max_iterations=3,
handle_parsing_errors=True,
system_message=system_message,
agent_kwargs=agent_kwargs,
)

agent.run("I was born in 1980, this year is 2023, how old am I?")

It always throw below error. but if I ask some straightforward questions, it would work without errors.

ERROR:root:'actions'
Traceback (most recent call last):
File "/app/app.py", line 24, in ask
result = agent.run(question)
File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 451, in run
return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[
File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 258, in call
raise e
File "/usr/local/lib/python3.10/site-packages/langchain/chains/base.py", line 252, in call
self._call(inputs, run_manager=run_manager)
File "/usr/local/lib/python3.10/site-packages/langchain/agents/agent.py", line 1035, in _call
next_step_output = self._take_next_step(
File "/usr/local/lib/python3.10/site-packages/langchain/agents/agent.py", line 832, in _take_next_step
output = self.agent.plan(
File "/usr/local/lib/python3.10/site-packages/langchain/agents/openai_functions_multi_agent/base.py", line 273, in plan
agent_decision = _parse_ai_message(predicted_message)
File "/usr/local/lib/python3.10/site-packages/langchain/agents/openai_functions_multi_agent/base.py", line 111, in _parse_ai_message
tools = json.loads(function_call["arguments"])["actions"]
KeyError: 'actions'

[codelahoma]

This is still happening as of v0.0.277, and appears to be related to the OpenAI function_call response. With langchain.debug=True, I see the following when it fails:

              "function_call": {
                "name": "get_model_fields_info",
                "arguments": "{\n  \"model\": \"HREmployee\"\n}"

When it should be:

             "function_call": {
                "name": "tool_selection",
                "arguments": "{\n  \"actions\": [\n    {\n      \"action_name\": \"get_model_fields_info\",\n      \"action\": {\n        \"model\": \"HREmployee\"\n      }\n    }\n  ]\n}"

[codelahoma]

Given the code where this KeyError is thrown, one approach would be to massage the response that lacks actions into its own AgentAction to return in the final_tools list.

Does that sound reasonable?

[sdjmchattie]

Unfortunately, because the code changed in the master branch, the link above highlights the wrong lines. These are updated links for v0.0.277 which was spoken about previously and also for the most recent v0.0.297

[bgokden]

I am also having this problem. It seems related to optional parameters because it happens when I have an action without any parameters (all optional parameters are empty.).

[urlmonitor]

I get the same issue but only with 3.5. Running the same code with gpt4 and it works.

[gururise]

With the new models from OpenAI capable of multi-function calling, is this class even needed anymore?

[dosubot]

Hi, @gururise,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, the recent commit #6518 introduced the OpenAIMultiFunctionsAgent class, causing failures when using custom tools that previously worked with the OpenAIFunctionsAgent. There have been discussions about potential solutions, such as massaging the response lacking 'actions' into its own 'AgentAction' to return in the 'final_tools' list, and questioning the necessity of the OpenAIMultiFunctionsAgent class with the new models from OpenAI capable of multi-function calling.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you!