Fluentd to ES in IPv6 only environmant via hostname having A and AAAA records #1069
Description
(check apply)
- read the contribution guideline
- (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.
Problem
I have an IPv6-only network, and my Elasticsearch service is behind a reverse proxy, where I have no direct access. The hostname for Elasticsearch is configured as a CNAME pointing to a dual-stack setup with both A (IPv4) and AAAA (IPv6) records. Fluentd is running in a container (fluentd-1.18-debian), which prefers IPv4 by default, even though I’ve set gai.conf to prefer IPv6. Despite this, Fluentd fails to connect to Elasticsearch and throws a connection_write error. However, if I configure extra_hosts in my container with the hostname and its corresponding IPv6 address, Fluentd is able to connect and write to Elasticsearch.
Steps to replicate
Provide example config and message
gai.conf
# Prefer IPv6 over IPv4
precedence ::ffff:0:0/96 10
precedence ::/0
99-outputs.conf
cat config.d/99-outputs.conf
<match **>
@type copy
<store>
@type elasticsearch_data_stream
verify_es_version_at_startup false
default_elasticsearch_version 8
@log_level debug
log_es_400_reason true
data_stream_name my-data-stream
data_stream_ilm_name some-lifecycle-policy
include_tag_key true
reload_connections false
reload_on_failure false
request_timeout 20s
hosts "https://$USER:$REDACTED@$URL:$PORT"
suppress_type_name true
<buffer>
@type memory
flush_thread_count 8
flush_interval 10s
flush_mode interval
overflow_action drop_oldest_chunk
retry_timeout 15m
disable_chunk_backup true
</buffer>
# Set content_type to application/json for ES 8.x
content_type application/json
</store>
</match>
<label @FLUENT_LOG>
<match fluent.*>
@type stdout
</match>
</label>
Logs without extra_hosts for container
May 13 10:46:21 $HOSTNAME fluentd[1092482]: The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.
May 13 10:46:21 $HOSTNAME fluentd[1092482]: 2025-05-13 10:46:21 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Failed to create data stream: <my-data-stream> connect_write timeout reached"
May 13 10:46:21 $HOSTNAME podman[1092591]: 2025-05-13 10:46:21.367187306 +0000 UTC m=+0.033227174 container died 526b58d90c5acb1a287a8e4c53084094d2bf398bed8b38e3b099881e3fdd9972 (image=docker/fluentd:v1.18-debian, name=fluentd, maintainer=Fluentd developers <[email protected]>, Description=Fluentd docker image, PODMAN_SYSTEMD_UNIT=fluentd.service, Vendor=Fluent Organization, Version=1.18.0)
May 13 10:46:21 $HOSTNAME podman[1092591]: 2025-05-13 10:46:21.403289505 +0000 UTC m=+0.069329373 container remove 526b58d90c5acb1a287a8e4c53084094d2bf398bed8b38e3b099881e3fdd9972 (image=docker/fluentd:v1.18-5, name=fluentd, PODMAN_SYSTEMD_UNIT=fluentd.service, Vendor=Fluent Organization, Version=1.18.0, maintainer=Fluentd developers <[email protected]>, Description=Fluentd docker image)
<snip>
With extra_hosts for container it just works without connect_timeout reached and I can see my logs in Kibana.
(off-topic)
I also tested fluentbit where I saw in the debug logs that the output would connect to the ipv4 address from DNS. But in the Fluentbit ES output i could set net.dns.prefer_ipv6: true which worked to get logs into Kibana. Maybe it would also an idea for fluentd when you have ipv6 only and have to use hostnames with A and AAAA records.
Expected Behavior or What you need to ask
I expect Fluentd to connect to Elasticsearch using IPv6 automatically, but it fails due to the default preference for IPv4. I need help troubleshooting this issue with Fluentd's container network settings or ensuring that it respects the IPv6 preference.
Using Fluentd and ES plugin versions
- OS version: Debian-based (Fluentd container: fluentd-1.18-debian)
- Bare Metal or within Docker or Kubernetes or others? Docker container
- Fluentd v1.18
- paste result of
fluentd --versionortd-agent --version $ fluentd --version fluentd 1.18.0
- paste result of
- ES plugin 5.4.3
- paste boot log of fluentd or td-agent
- paste result of
fluent-gem list,td-agent-gem listor your Gemfile.lock -
/$ fluent-gem list *** LOCAL GEMS *** abbrev (default: 0.1.1) async (1.32.1) async-http (0.64.2) async-io (1.43.2) async-pool (0.10.3) base64 (0.2.0, default: 0.1.1) benchmark (default: 0.2.1) bigdecimal (default: 3.1.3) bundler (default: 2.4.19) cgi (default: 0.3.7) concurrent-ruby (1.3.5) console (1.30.2) cool.io (1.9.0) csv (3.3.4, default: 3.2.6) date (default: 3.3.3) debug (1.7.1) delegate (default: 0.3.0) did_you_mean (default: 1.6.3) dig_rb (1.0.1) digest (default: 3.1.1) drb (2.2.1, default: 2.1.1) elastic-transport (8.4.0) elasticsearch (8.14.0) elasticsearch-api (8.14.0) english (default: 0.7.2) erb (default: 4.0.2) error_highlight (default: 0.5.1) etc (default: 1.4.2) excon (1.2.5) faraday (2.13.1) faraday-excon (2.3.0) faraday-net_http (3.4.0) fcntl (default: 1.0.2) ffi (1.17.2 x86_64-linux-gnu) fiber-annotation (0.2.0) fiber-local (1.1.0) fiber-storage (1.0.0) fiddle (default: 1.1.1) fileutils (default: 1.7.0) find (default: 0.1.1) fluent-config-regexp-type (1.0.0) fluent-plugin-elasticsearch (5.4.3) fluent-plugin-geoip (1.3.2) fluent-plugin-http-healthcheck (0.1.0) fluent-plugin-prometheus (2.0.3) fluent-plugin-record-modifier (2.1.1) fluent-plugin-rewrite-tag-filter (2.4.0) fluent-plugin-systemd (1.0.5) fluentd (1.18.0) forwardable (default: 1.3.3) geoip-c (0.9.1) geoip2_c (0.3.4) getoptlong (default: 0.2.0) http_parser.rb (0.8.0) io-console (default: 0.6.0) io-nonblock (default: 0.2.0) io-wait (default: 0.3.0) ipaddr (default: 1.2.5) irb (default: 1.6.2) json (2.11.3, 2.7.4, default: 2.6.3) logger (1.7.0, default: 1.5.3) matrix (0.4.2) minitest (5.25.1) msgpack (1.8.0) multi_json (1.15.0) mutex_m (default: 0.1.2) net-ftp (0.2.1) net-http (0.6.0, default: 0.4.1) net-imap (0.3.8) net-pop (0.1.2) net-protocol (default: 0.2.1) net-smtp (0.3.4) nio4r (2.7.4) nkf (default: 0.1.2) observer (default: 0.1.1) oj (3.16.5) open-uri (default: 0.3.0) open3 (default: 0.1.2) openssl (default: 3.1.0) optparse (default: 0.3.1) ostruct (default: 0.5.5) pathname (default: 0.2.1) power_assert (2.0.3) pp (default: 0.4.0) prettyprint (default: 0.1.1) prime (0.1.2) prometheus-client (4.2.4) protocol-hpack (1.5.1) protocol-http (0.26.8) protocol-http1 (0.19.1) protocol-http2 (0.16.0) pstore (default: 0.1.2) psych (default: 5.0.1) racc (default: 1.6.2) rake (13.0.6) rbs (2.8.2) rdoc (default: 6.5.1.1) readline (default: 0.0.3) readline-ext (default: 0.1.5) reline (default: 0.3.2) resolv (default: 0.2.2) resolv-replace (default: 0.1.1) rexml (3.3.9) rinda (default: 0.1.1) rss (0.3.1) ruby2_keywords (default: 0.0.5) securerandom (default: 0.2.2) serverengine (2.4.0) set (default: 1.0.3) shellwords (default: 0.1.0) sigdump (0.2.5) singleton (default: 0.1.1) stringio (default: 3.0.4) strptime (0.2.5) strscan (default: 3.0.7) syntax_suggest (default: 1.1.0) syslog (default: 0.1.1) systemd-journal (1.4.2) tempfile (default: 0.1.3) test-unit (3.5.7) time (default: 0.2.2) timeout (default: 0.3.1) timers (4.4.0) tmpdir (default: 0.1.3) traces (0.15.2) tsort (default: 0.1.1) typeprof (0.21.3) tzinfo (2.0.6) tzinfo-data (1.2025.2) un (default: 0.2.1) uri (1.0.3, default: 0.12.4) weakref (default: 0.1.2) webrick (1.9.1) yajl-ruby (1.4.3) yaml (default: 0.2.1) zlib (default: 3.0.0)
* ES version (optional)
* ES template(s) (optional)