Metricbeat can't automatically connect

[widhalmt]

When running the collection with defaults (only activating Metricbeat) it fails, because the variable elasticsearch_http_security isn't set for Metricbeat hosts. This looks like it's some legacy code from before we built the collection. It's possible that Auditbeat suffers the same problem.

I'll investigate and I hope I can provide a solution.

[widhalmt]

There is a deeper lying confusion about when to enable TLS in connections to Elasticsearch. If you browse all roles in the collection you will different conditions that will enable TLS for the connection to Elasticsearch. Most only allow TLS when the full stack variable is set.

I think we should allow to have TLS with or without the full stack variable being set (e.g. using managed Elasticsearch). So my idea is to have a local variable in each role to en- or disable TLS. It should default to true/enabled and be set automatically to true when the full stack is used.

[widhalmt]

I guess, I found the culprit: https://github.com/NETWAYS/ansible-collection-elasticstack/blob/main/roles/beats/templates/metricbeat.yml.j2#L18

Here we use a variable for a password that's not set anywhere. We're using the hardcoded user elastic which is bad, security-wise (looks like something that's leftover from a testing environment) with a password that looks like it's somehow generated or read but this happens nowhere.

I hoped for a module for user management before we need to create more users the way we did with Logstash. But it looks, like we need to copy the code from logstash-security.yml and adapt it for Beats. So we can create our own beats user as a makeshift solution. Don't put too much effort into it - we'll replace it with a module anyway.

[widhalmt]

Part of the solution should also be to make sure, metricbeat setup is only run on a single host and only one that has access to Elasticsearch and Kibana