Fix envoy access logs

[mviitane]

Changes

Fix envoy access logs

• Remove http attributes that cause problems in the Collector pipelines
• Fixes Collector complains about "Could not dynamically add mapping for field [http.connection.id]" #1866

Merge Requirements

For new features contributions, please make sure you have completed the following
essential items:

• CHANGELOG.md updated to document new feature additions
• Appropriate documentation updates in the docs
• Appropriate Helm chart updates in the helm-charts

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

[rogercoll]

Some of these attributes are defined by the OpenTelemetry Semantic Conventions: https://github.com/open-telemetry/semantic-conventions/tree/main/model/http

Would it be feasible to fix the opensearch mappings instead?

[mviitane]

Some of these attributes are defined by the OpenTelemetry Semantic Conventions: https://github.com/open-telemetry/semantic-conventions/tree/main/model/http

Would it be feasible to fix the opensearch mappings instead?

I know that some of these are in the semantic conventions, but I believe Envoy doesn't send them correctly. I didn't find a good reference configuration and don't know a way to fix these in the Envoy configuration. We can add these attributes back when we get them working.

I don't see how we could fix this with the OpenSearch mapping because the batch processor also shows the same problem.

[rogercoll]

I don't see how we could fix this with the OpenSearch mapping because the batch processor also shows the same problem.

The error is shown in the batch processor because is the error propagated by the OpenSearch exporter, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/batch_processor.go#L263

[mviitane]

OK, I didn't realize this is only opensearch related. I wonder why the automatic mappings are not enough.
https://opensearch.org/docs/latest/field-types/

[rogercoll]

OK, I didn't realize this is only opensearch related. I wonder why the automatic mappings are not enough. https://opensearch.org/docs/latest/field-types/

Not sure, I am not very familiar with OpenSearch. Have you tried removing the corresponding index? (maybe the http.connection.id mapping was created with some other data)

[mviitane]

OK, I didn't realize this is only opensearch related. I wonder why the automatic mappings are not enough. https://opensearch.org/docs/latest/field-types/

Not sure, I am not very familiar with OpenSearch. Have you tried removing the corresponding index? (maybe the http.connection.id mapping was created with some other data)

I ran docker system prune -a and started a clean environment and got the problem almost right away. Are you not seeing the same problem?

[rogercoll]

OK, I didn't realize this is only opensearch related. I wonder why the automatic mappings are not enough. https://opensearch.org/docs/latest/field-types/

Not sure, I am not very familiar with OpenSearch. Have you tried removing the corresponding index? (maybe the http.connection.id mapping was created with some other data)

I ran docker system prune -a and started a clean environment and got the problem almost right away. Are you not seeing the same problem?

Not on my environment, just tested with nightly DEMO_VERSION=nightly-12898327927 and collector logs are clear.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[puckpuck]

@mviitane I suspect the issue may be related to another service emitting this same attribute but with a different data type. When you run into it, can you use the OpenSearch API to query for this attribute and determine which service is emitting it?

[mviitane]

@mviitane I suspect the issue may be related to another service emitting this same attribute but with a different data type. When you run into it, can you use the OpenSearch API to query for this attribute and determine which service is emitting it?

I grepped the debug exporter (verbose) and saw that all signals with http.connection.id are logs with
event.name: Str(proxy.access), so I think they're all from Envoy.

otel-collector | -> event.name: Str(proxy.access)
otel-collector | -> http.connection.id: Str(85)

[puckpuck]

If you update the PR to only drop the http.connection.id field do the other ones from Envoy still work fine? Maybe we do just that instead.

[mviitane]

If you update the PR to only drop the http.connection.id field do the other ones from Envoy still work fine? Maybe we do just that instead.

No, unfortunately, that wasn't enough. I tried a few of the first ones.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[puckpuck]

We should merge this PR, but I want to create an issue so we can learn more about the actual root cause. This only affects specific environments and is related to attributes that start with http..

@mviitane, can you resolve the conflicts so we can merge this?