listRunSteps - sttp.openai.OpenAIExceptions$OpenAIException$DeserializationOpenAIException: expected string got dictionary

[dgolda]

sttop-openai version 0.3.2

While calling listRunSteps, I encountered the following exception:

sttp.openai.OpenAIExceptions$OpenAIException$DeserializationOpenAIException: expected string got dictionary at index 2990
	at sttp.openai.OpenAIExceptions$OpenAIException$DeserializationOpenAIException$.apply(OpenAIExceptions.scala:24)
	at sttp.openai.OpenAIExceptions$OpenAIException$DeserializationOpenAIException$.apply(OpenAIExceptions.scala:27)
	at sttp.openai.json.SttpUpickleApiExtension$.$anonfun$deserializeJsonSnake$1(SttpUpickleApiExtension.scala:47)
	at sttp.openai.json.SttpUpickleApiExtension$.$anonfun$deserializeRightWithMappedExceptions$1(SttpUpickleApiExtension.scala:39)
	at sttp.client4.MappedResponseAs.$anonfun$mapWithMetadata$1(ResponseAs.scala:462)
	at sttp.client4.internal.BodyFromResponseAs.$anonfun$doApply$4(BodyFromResponseAs.scala:26)
	at sttp.monad.IdentityMonad$.eval(MonadError.scala:221)
	at sttp.client4.internal.BodyFromResponseAs.$anonfun$doApply$3(BodyFromResponseAs.scala:26)
	at sttp.monad.syntax$MonadErrorOps.map(MonadError.scala:71)
	at sttp.client4.internal.BodyFromResponseAs.$anonfun$doApply$2(BodyFromResponseAs.scala:26)
	at sttp.monad.IdentityMonad$.flatMap(MonadError.scala:216)
	at sttp.monad.syntax$MonadErrorOps.flatMap(MonadError.scala:72)
	at sttp.client4.internal.BodyFromResponseAs.doApply(BodyFromResponseAs.scala:25)
	at sttp.client4.internal.BodyFromResponseAs.$anonfun$apply$1(BodyFromResponseAs.scala:16)
	at sttp.monad.syntax$MonadErrorOps.map(MonadError.scala:71)
	at sttp.client4.internal.BodyFromResponseAs.apply(BodyFromResponseAs.scala:16)
	at sttp.client4.internal.httpclient.BodyFromHttpClient.apply(BodyFromHttpClient.scala:20)
	at sttp.client4.internal.httpclient.BodyFromHttpClient.apply$(BodyFromHttpClient.scala:16)
	at sttp.client4.httpclient.HttpClientSyncBackend$$anon$3.apply(HttpClientSyncBackend.scala:125)
	at sttp.client4.httpclient.HttpClientBackend.readResponse(HttpClientBackend.scala:142)
	at sttp.client4.httpclient.HttpClientSyncBackend.sendRegular(HttpClientSyncBackend.scala:46)
	at sttp.client4.httpclient.HttpClientSyncBackend.sendRegular(HttpClientSyncBackend.scala:26)
	at sttp.client4.httpclient.HttpClientBackend.$anonfun$send$1(HttpClientBackend.scala:53)
	at sttp.monad.MonadError.handleError(MonadError.scala:26)
	at sttp.monad.MonadError.handleError$(MonadError.scala:25)
	at sttp.monad.IdentityMonad$.handleError(MonadError.scala:213)
	at sttp.client4.SttpClientException$.adjustExceptions(SttpClientException.scala:52)
	at sttp.client4.httpclient.HttpClientBackend.adjustExceptions(HttpClientBackend.scala:62)
	at sttp.client4.httpclient.HttpClientBackend.send(HttpClientBackend.scala:53)
	at sttp.client4.wrappers.FollowRedirectsBackend.sendWithCounter(FollowRedirectsBackend.scala:19)
	at sttp.client4.wrappers.FollowRedirectsBackend.send(FollowRedirectsBackend.scala:15)
	at sttp.client4.Request.send(request.scala:183)
	at sttp.openai.OpenAISyncClient.sendOrThrow(OpenAISyncClient.scala:1092)
	at sttp.openai.OpenAISyncClient.listRunSteps(OpenAISyncClient.scala:806)

I inspected the raw JSON response and found that the issue is related to the file_search field within tool_calls:

...
                "tool_calls": [
                    {
                        "id": "call_cDiplsw8ixZu2lxgbNPQN9wF",
                        "type": "file_search",
                        "file_search": {
                            "ranking_options": {
                                "ranker": "default_2024_08_21",
                                "score_threshold": 0.0
                            },
                            "results": [
                                {
                                    "file_id": "file-GANzFXVdwWUANHif1hQi5P",
                                    "file_name": "iotSchema.txt",
                                    "score": 0.4615138602222831,
                                    "attributes": {}
                                },
..

The corresponding Scala case class is defined as:

  case class FileSearchToolCall(
      id: String,
      `type`: String,
      fileSearch: Map[String, String]
  ) extends ToolCall

So the deserializer expects fileSearch to be a Map[String, String], but the API now returns a JSON object with nested structures, which causes the deserialization to fail.

Interestingly, the OpenAI documentation still says:

file_search
map
For now, this is always going to be an empty object.

However, just below that, it also documents two fields:

ranking_options
object
The ranking options for the file search.

results
array
The results of the file search.

I suspect that OpenAI recently started populating the file_search field with these values, which broke deserialization on our end.

[dgolda]

I'll try to prepare fix

[dgolda]

Fix in pull request #322