9.0.1 is erroring on Elastic Server 9.0.0

[kapso]

This update seems to be causing this issue

[400] {"error":{"root_cause":[{"type":"media_type_header_exception","reason":"Invalid media-type value on headers [Accept, Content-Type]"}],"type":"media_type_header_exception","reason":"Invalid media-type value on headers [Accept, Content-Type]","caused_by":{"type":"status_exception","reason":"A compatible version is required on both Content-Type and Accept headers if either one has requested a compatible version. Accept=application/vnd.elasticsearch+json; compatible-with=9, Content-Type=application/json"}},"status":400}

Versions:
Elastic Server 9.0.0
elasticsearch 9.0.1 (gem)
searchkick 5.5.0
Rails 8.0.2
Ruby 3.4.3

[alexdeng-mp]

Its also failing for us with the elasticsearch:8.18.0 on CI with this reason

2025-04-24 10:12:51 NZST | /bundle/ruby/3.3.0/gems/elastic-transport-8.4.0/lib/elastic/transport/transport/base.rb:228:in `__raise_transport_error': [400] {"error":{"root_cause":[{"type":"media_type_header_exception","reason":"Invalid media-type value on headers [Accept, Content-Type]"}],"type":"media_type_header_exception","reason":"Invalid media-type value on headers [Accept, Content-Type]","caused_by":{"type":"status_exception","reason":"Accept version must be either version 8 or 7, but found 9. Accept=application/vnd.elasticsearch+json; compatible-with=9"}},"status":400} (Elastic::Transport::Transport::Errors::BadRequest)
-- | --

[picandocodigo]

Thanks for reporting this. This header is necessary for Elasticsearch version 9 of the client to work properly with version 9 of Elastic Stack.

@kapso It looks like searchkick is overriding the content-type header here. The error says A compatible version is required on both Content-Type and Accept headers if either one has requested a compatible version. It can work with application/json, but the accept header also needs to be set.

@alexdeng-mp To use version 9 of the client with Elasticsearch 8.18.0, you can set custom headers to compatible-with=8:

accept: 'application/vnd.elasticsearch+json; compatible-with=8',
content_type: 'application/vnd.elasticsearch+json; compatible-with=8'

It is recommended to use a client version that matches the server version though.

I'm working on a 9.0.2 release and updates to the documentation to make this more clear. Thanks!

[picandocodigo]

Sorry I pressed on "Close Issue" by accident, but reopened it right away 😬

[Srijitha-code]

@picandocodigo I'm encountering the same issue with Elasticsearch gem version 9.0.2, specifically when shipping logs from Fluentd to Elasticsearch.
fluentd version: 1.18.0
elasticsearch: 9.0.2
elasticsearch-api: 9.0.2
elastic-transport: 8.4.0
fluend-plugin-elasticsearch: 5.4.3
and elasticsearch cluster version is 9
2025-04-28 09:17:53 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2025-04-28 09:17:55 +0000 chunk="633aeee3e06bda3cf586d204d14f472c" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"quickstart-v2-es-http\", :port=>9200, :scheme=>\"https\", :user=>\"elastic\", :password=>\"obfuscated\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Content-Type, Accept]\"}],\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Content-Type, Accept]\",\"caused_by\":{\"type\":\"status_exception\",\"reason\":\"A compatible version is required on both Content-Type and Accept headers if either one has requested a compatible version. Accept=null, Content-Type=application/vnd.elasticsearch+x-ndjson; compatible-with=9\"}},\"status\":400}" 2025-04-28 09:17:53 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:1171:in rescue in send_bulk'
2025-04-28 09:17:53 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:1133:in send_bulk' 2025-04-28 09:17:53 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:909:in block in write'
2025-04-28 09:17:53 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:908:in each' 2025-04-28 09:17:53 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:908:in write'
2025-04-28 09:17:53 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:1225:in try_flush' 2025-04-28 09:17:53 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:1540:in flush_thread_run'
2025-04-28 09:17:53 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:510:in block (2 levels) in start' 2025-04-28 09:17:53 +0000 [warn]: #0 /usr/local/lib/ruby/gems/3.2.0/gems/fluentd-1.18.0/lib/fluent/plugin_helper/thread.rb:78:in block in thread_create'`

[picandocodigo]

@Srijitha-code as the error message says, it's the same issue as was happening with Searchkick. The fluentd plugin is setting the content-type header, but not the accept header, as you can see here.

You could try using the custom_headers parameter to also set the accept header, and that should hopefully work while you wait for an updated version of the plugin.

[Srijitha-code]

@picandocodigo Thanks for the suggestion! Yes, until the updated version is released, I'll use the custom_headers parameter as a workaround.

[alex-vergara37]

@Srijitha-code did you get this to work?

[picandocodigo]

Searchkick has released version 5.5.1 which addresses the header issue with version 9.

fluent-plugin-elasticsearch has a Pull Request open to address it, so there will hopefully be a release with the fix soon.

[picandocodigo]

fluent-plugin-elasticsearch has released v6.0.0 which fixes the issue.

[picandocodigo]

Closing as 9.0.2 addresses issues with the headers and the external libraries have released versions with fixes too.