expected response body when tool calling

[lucyknada]

I'm currently writing a tool calling proxy that sits between kiln and more popular constrained generation backends (that dont implement true function calling); is there an example of what payload kiln expects back from the API?

I tried just returning oAI spec:

{
      "id": "call_12345xyz",
      "type": "function",
      "function": {
        "name": "task_response",
        "arguments": "{\"subtopics\": [\"Trees\"]}"
      }
 }

as the response body, but that just threw a model_dump() error
thanks!

[lucyknada]

resolved via openAI curl

[scosman]

👍

json_object or json_schema be preferable, but we'd need a way to expose the mode. It can be configured via ml_model_list.py for built in models, but I don't have a method for custom models or openAI compatible servers yet.

[lucyknada]

thats actually how e.g. tabbyAPI handles it yeah! it uses "json_schema" as the key for constrained generation, tool-calling as a whole is kind of unsupported in the local backend side of things and if it is; only with like 5 models total

[scosman]

yeah, I'm defaulting to tool calling because it's the oldest so potentially has more support. But I don't love the API. I'll add this idea to the queue.