Improvements needed for external MongoDB, OpenTelemetry metrics, and ClickHouse schema support

[jovanradin]

Hi HyperDX team,

Thanks for the great work on this project. I’ve encountered a few issues and improvement opportunities while integrating HyperDX into our production Kubernetes setup:

1. External MongoDB PVC issue
   When configuring HyperDX with an external MongoDB, the Helm chart still tries to create a PersistentVolumeClaim for MongoDB, even though it’s not used. This PVC is left unbound and causes noise in the cluster. Ideally, no PVC should be created if mongodb.enabled=false.

2. CPU utilization metrics deprecated in OpenTelemetry Collector
   In recent versions of the OpenTelemetry Collector, the following metrics are disabled by default:

k8s.pod.cpu_utilization

k8s.container.cpu_utilization

k8s.node.cpu_utilization

They’ve been replaced with cpu.usage metrics per OpenTelemetry’s deprecation notice. HyperDX should update its expectations and dashboards to reflect this change.

3. ClickHouse "Clickstack" improvements
   When using an external ClickHouse instance:

The necessary schemas and users are not automatically created. This requires manual setup and breaks automation.

There should be a migration mechanism to create all required users and schemas, especially for the HyperDX app.

More importantly, there should be support for selecting engine types, including:

ReplicatedMergeTree

SharedMergeTree

Single-node MergeTree

In my case, I manually created the schemas and used Distributed tables backed by ReplicatedMergeTree, but a supported configuration would be highly valuable for ClickHouse power users.

Happy to assist with testing improvements or provide more details.

Thanks!

[wrn14897]

Hey @jovanradin , thanks for bringing up these issues. Let me try to answer them one by one. For

1. The mongodb pvc issue should be fixed in this PR (feat: option to keep all services PVCs when uninstalling helm #98)
2. Good call on the semantic convention for k8s. We will need to update the k8s dashboard. I will file a separate ticket for it.
3. I'd love to learn more about this workflow. Can you tell me more about the "migration mechanism"? From my understanding, once you export the data using the otel collector, the schema should be created automatically. Or is your question more about the exporter implementation?

[jovanradin]

Hi @wrn14897 , thanks for yor reply.

Regarding the third issue — I’ve never been able to successfully use an external ClickHouse instance.
It seems like self-hosted ClickHouse is not fully supported right now:

• Migrations are never run
• Schema is not created automatically

I manually created the schema using the official documentation:
https://clickhouse.com/docs/use-cases/observability/clickstack/ingesting-data/schemas

The hyperdx-hdx-oss-v2-app connects to my ClickHouse just fine, and I can query all the tables I created.
However, the collector never inserts any data.
Instead, it only logs something like "collecting configuration from remote" and appears to never actually collect or send any telemetry to ClickHouse.

Proposal for better external ClickHouse support

1. Single-node external ClickHouse

• Schema creation should happen exactly the same way as with the internal ClickHouse option in the Helm chart.

2. Multi-node (sharded or replicated) ClickHouse

• Chart should support creating tables with ReplicatedMergeTree for local storage, plus Distributed tables for querying across replicas.

Example setup:

CREATE TABLE otel_logs_local ON CLUSTER my_cluster
(
  -- column definitions same as otel_logs
)
ENGINE = ReplicatedMergeTree(
    '/clickhouse/{installation}/{cluster}/tables/{shard}/default/ats_data',
    '{replica}'
)
ORDER BY some_column;

CREATE TABLE otel_logs ON CLUSTER my_cluster
AS otel_logs_local
ENGINE = Distributed(
    my_cluster,
    current_database(),
    otel_logs_local,
    rand()
);

This would allow:

• Proper schema and user creation even for external ClickHouse
• The collector to connect and insert data reliably
• Choice between MergeTree, ReplicatedMergeTree, or SharedMergeTree (for cloud only) based on deployment type.