Skip to content

refactor: remove all receiver references to bundled config host files #229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 27, 2025
Merged

refactor: remove all receiver references to bundled config host files #229

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ go 1.23.10

require (
github.com/go-viper/mapstructure/v2 v2.2.1
github.com/invopop/jsonschema v0.13.0
github.com/jarcoal/httpmock v1.3.1
github.com/mcuadros/go-defaults v1.2.0
github.com/observeinc/observe-agent/observecol v0.0.0-00010101000000-000000000000
Expand All @@ -13,6 +14,7 @@ require (
github.com/spf13/cobra v1.9.1
github.com/spf13/pflag v1.0.6
github.com/spf13/viper v1.20.0-alpha.6
github.com/stoewer/go-strcase v1.3.1
github.com/stretchr/testify v1.10.0
go.opentelemetry.io/collector/confmap v1.34.0
go.opentelemetry.io/collector/otelcol v0.128.0
Expand Down Expand Up @@ -174,7 +176,6 @@ require (
github.com/hetznercloud/hcloud-go/v2 v2.21.0 // indirect
github.com/iancoleman/strcase v0.3.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/invopop/jsonschema v0.13.0 // indirect
github.com/ionos-cloud/sdk-go/v6 v6.3.3 // indirect
github.com/jaegertracing/jaeger-idl v0.5.0 // indirect
github.com/jcmturner/aescts/v2 v2.0.0 // indirect
Expand Down Expand Up @@ -337,7 +338,6 @@ require (
github.com/sourcegraph/conc v0.3.0 // indirect
github.com/spf13/afero v1.11.0 // indirect
github.com/spf13/cast v1.6.0 // indirect
github.com/stoewer/go-strcase v1.3.1 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/thda/tds v0.1.7 // indirect
Expand Down
3 changes: 3 additions & 0 deletions internal/commands/config/config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,10 +128,13 @@ func setEnvVars(t *testing.T, packageType PackageType) {
switch packageType {
case MacOS:
assert.NoError(t, os.Setenv("FILESTORAGE_PATH", "/var/lib/observe-agent/filestorage"))
assert.NoError(t, os.Setenv("OBSERVE_AGENT_CONFIG_PATH", "/usr/local/observe-agent/observe-agent.yaml"))
case Windows:
assert.NoError(t, os.Setenv("FILESTORAGE_PATH", "C:\\ProgramData\\Observe\\observe-agent\\filestorage"))
assert.NoError(t, os.Setenv("OBSERVE_AGENT_CONFIG_PATH", "C:\\Program Files\\Observe\\observe-agent\\observe-agent.yaml"))
case Linux, Docker:
assert.NoError(t, os.Setenv("FILESTORAGE_PATH", "/var/lib/observe-agent/filestorage"))
assert.NoError(t, os.Setenv("OBSERVE_AGENT_CONFIG_PATH", "/etc/observe-agent/observe-agent.yaml"))
default:
t.Errorf("Unknown package type: %s", packageType)
}
Expand Down
22 changes: 3 additions & 19 deletions internal/commands/config/test/snap1-docker-output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,14 +131,6 @@ processors:
- truncate_all(attributes, 4095)
- truncate_all(resource.attributes, 4095)
receivers:
filelog/agent-config:
include:
- /etc/observe-agent/connections/common/base.yaml.tmpl
include_file_path: true
multiline:
line_end_pattern: ENDOFLINEPATTERN
poll_interval: 5m
start_at: beginning
filelog/host_monitoring:
exclude:
- exclude1
Expand All @@ -160,7 +152,7 @@ receivers:
storage: file_storage
filestats/agent:
collection_interval: 5m
include: /etc/observe-agent/connections/common/base.yaml.tmpl
include: /etc/observe-agent/observe-agent.yaml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should always exist right? unless the user for some reason manually deletes it after install

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the expanded env var, so the receiver isn't tied to this path specifically. There is a weird way that you can run the agent with no observe agent config file:

  1. install
  2. delete the config at the default path
  3. start the agent using env var overrides for the required fields
    but I think that's pretty unlikely and if anyone is doing that, then it's possible that their current app is broken anyway.

initial_delay: 60s
hostmetrics/host-monitoring-host:
collection_interval: 60s
Expand Down Expand Up @@ -264,17 +256,9 @@ service:
pipelines:
logs/agent-config:
exporters:
- otlphttp/observe
processors:
- memory_limiter
- transform/truncate
- resourcedetection
- resourcedetection/cloud
- attributes/observe_global_attributes
- resource/observe_global_resource_attributes
- batch
- nop
receivers:
- filelog/agent-config
- nop
logs/agent-journald:
exporters:
- otlphttp/observe
Expand Down
22 changes: 3 additions & 19 deletions internal/commands/config/test/snap1-linux-output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,14 +128,6 @@ processors:
- truncate_all(attributes, 4095)
- truncate_all(resource.attributes, 4095)
receivers:
filelog/agent-config:
include:
- /etc/observe-agent/connections/common/base.yaml.tmpl
include_file_path: true
multiline:
line_end_pattern: ENDOFLINEPATTERN
poll_interval: 5m
start_at: beginning
filelog/host_monitoring:
exclude:
- exclude1
Expand All @@ -157,7 +149,7 @@ receivers:
storage: file_storage
filestats/agent:
collection_interval: 5m
include: /etc/observe-agent/connections/common/base.yaml.tmpl
include: /etc/observe-agent/observe-agent.yaml
initial_delay: 60s
hostmetrics/host-monitoring-host:
collection_interval: 60s
Expand Down Expand Up @@ -258,17 +250,9 @@ service:
pipelines:
logs/agent-config:
exporters:
- otlphttp/observe
processors:
- memory_limiter
- transform/truncate
- resourcedetection
- resourcedetection/cloud
- attributes/observe_global_attributes
- resource/observe_global_resource_attributes
- batch
- nop
receivers:
- filelog/agent-config
- nop
logs/agent-journald:
exporters:
- otlphttp/observe
Expand Down
22 changes: 3 additions & 19 deletions internal/commands/config/test/snap1-macos-output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,14 +128,6 @@ processors:
- truncate_all(attributes, 4095)
- truncate_all(resource.attributes, 4095)
receivers:
filelog/agent-config:
include:
- /usr/local/observe-agent/connections/common/base.yaml.tmpl
include_file_path: true
multiline:
line_end_pattern: ENDOFLINEPATTERN
poll_interval: 5m
start_at: beginning
filelog/host_monitoring:
exclude:
- exclude1
Expand All @@ -157,7 +149,7 @@ receivers:
storage: file_storage
filestats/agent:
collection_interval: 5m
include: /usr/local/observe-agent/connections/common/base.yaml.tmpl
include: /usr/local/observe-agent/observe-agent.yaml
initial_delay: 60s
hostmetrics/host-monitoring-host:
collection_interval: 60s
Expand Down Expand Up @@ -245,17 +237,9 @@ service:
pipelines:
logs/agent-config:
exporters:
- otlphttp/observe
processors:
- memory_limiter
- transform/truncate
- resourcedetection
- resourcedetection/cloud
- attributes/observe_global_attributes
- resource/observe_global_resource_attributes
- batch
- nop
receivers:
- filelog/agent-config
- nop
logs/forward:
exporters:
- otlphttp/observe
Expand Down
22 changes: 3 additions & 19 deletions internal/commands/config/test/snap1-windows-output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,14 +107,6 @@ processors:
- truncate_all(attributes, 4095)
- truncate_all(resource.attributes, 4095)
receivers:
filelog/agent-config:
include:
- C:\Program Files\Observe\observe-agent\connections\common\base.yaml.tmpl
include_file_path: true
multiline:
line_end_pattern: ENDOFLINEPATTERN
poll_interval: 5m
start_at: beginning
filelog/host_monitoring:
exclude:
- exclude1
Expand All @@ -136,7 +128,7 @@ receivers:
storage: file_storage
filestats/agent:
collection_interval: 5m
include: C:\Program Files\Observe\observe-agent\connections\common\base.yaml.tmpl
include: C:\Program Files\Observe\observe-agent\observe-agent.yaml
initial_delay: 60s
hostmetrics/host-monitoring-host:
collection_interval: 60s
Expand Down Expand Up @@ -224,17 +216,9 @@ service:
pipelines:
logs/agent-config:
exporters:
- otlphttp/observe
processors:
- memory_limiter
- transform/truncate
- resourcedetection
- resourcedetection/cloud
- attributes/observe_global_attributes
- resource/observe_global_resource_attributes
- batch
- nop
receivers:
- filelog/agent-config
- nop
logs/forward:
exporters:
- otlphttp/observe
Expand Down
1 change: 1 addition & 0 deletions internal/connections/allconnectiontypes.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ var HostMonitoringConnectionType = MakeConnectionType(
[]CollectorConfigFragment{
{
enabledCheck: func(agentConfig *config.AgentConfig) bool {
// TODO remove this deprecated template
return agentConfig.HostMonitoring.Enabled
},
colConfigFilePath: "host.yaml.tmpl",
Expand Down
5 changes: 5 additions & 0 deletions internal/root/root.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,11 @@ func setEnvVars() error {
os.Setenv("OBSERVE_AUTHORIZATION_HEADER", "Bearer "+token)
os.Setenv("FILESTORAGE_PATH", getDefaultFilestoragePath())

configFile := viper.ConfigFileUsed()
if configFile != "" {
os.Setenv("OBSERVE_AGENT_CONFIG_PATH", configFile)
}

// Default TRACE_TOKEN to be the value of the configured token if it's not set. This allows for users to upgrade to
// direct write tracing with ingest tokens in kubernetes without breaking backwards compatibility in our helm chart.
// TODO: remove this once our helm chart no longer supports TRACE_TOKEN
Expand Down
3 changes: 1 addition & 2 deletions packaging/docker/observe-agent/connections/host_monitoring/host.yaml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
receivers:
# TODO deprecate this
filestats/agent:
include: '/etc/observe-agent/connections/common/base.yaml.tmpl'
include: "${env:OBSERVE_AGENT_CONFIG_PATH}"
collection_interval: 5m
initial_delay: 60s

Expand Down
26 changes: 4 additions & 22 deletions packaging/docker/observe-agent/connections/self_monitoring/logs_and_metrics.yaml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,5 @@
receivers:
filelog/agent-config: # TODO: Add observe-agent.yaml once we can obfuscate sensitive config fields
# TODO deprecate this
include: [/etc/observe-agent/connections/common/base.yaml.tmpl]
include_file_path: true
start_at: beginning
poll_interval: 5m
multiline:
line_end_pattern: ENDOFLINEPATTERN
# TODO: Add observe-agent.yaml once we can obfuscate sensitive config fields

prometheus/agent:
config:
Expand Down Expand Up @@ -60,18 +53,7 @@ service:
- batch
exporters: [otlphttp/observe, count]

# Deprecated
logs/agent-config:
receivers: [filelog/agent-config]
processors:
- memory_limiter
- transform/truncate
- resourcedetection
- resourcedetection/cloud
{{- if .HasAttributes }}
- attributes/observe_global_attributes
{{- end }}
{{- if .HasResourceAttributes }}
- resource/observe_global_resource_attributes
{{- end }}
- batch
exporters: [otlphttp/observe]
receivers: [nop]
exporters: [nop]
3 changes: 1 addition & 2 deletions packaging/linux/connections/host_monitoring/host.yaml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
receivers:
# TODO deprecate this
filestats/agent:
include: '/etc/observe-agent/connections/common/base.yaml.tmpl'
include: "${env:OBSERVE_AGENT_CONFIG_PATH}"
collection_interval: 5m
initial_delay: 60s

Expand Down
26 changes: 4 additions & 22 deletions packaging/linux/connections/self_monitoring/logs_and_metrics.yaml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,5 @@
receivers:
filelog/agent-config: # TODO: Add observe-agent.yaml once we can obfuscate sensitive config fields
# TODO deprecate this
include: [/etc/observe-agent/connections/common/base.yaml.tmpl]
include_file_path: true
start_at: beginning
poll_interval: 5m
multiline:
line_end_pattern: ENDOFLINEPATTERN
# TODO: Add observe-agent.yaml once we can obfuscate sensitive config fields

prometheus/agent:
config:
Expand Down Expand Up @@ -60,18 +53,7 @@ service:
- batch
exporters: [otlphttp/observe, count]

# Deprecated
logs/agent-config:
receivers: [filelog/agent-config]
processors:
- memory_limiter
- transform/truncate
- resourcedetection
- resourcedetection/cloud
{{- if .HasAttributes }}
- attributes/observe_global_attributes
{{- end }}
{{- if .HasResourceAttributes }}
- resource/observe_global_resource_attributes
{{- end }}
- batch
exporters: [otlphttp/observe]
receivers: [nop]
exporters: [nop]
3 changes: 1 addition & 2 deletions packaging/macos/connections/host_monitoring/host.yaml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
receivers:
# TODO deprecate this
filestats/agent:
include: '/usr/local/observe-agent/connections/common/base.yaml.tmpl'
include: "${env:OBSERVE_AGENT_CONFIG_PATH}"
collection_interval: 5m
initial_delay: 60s

Expand Down
26 changes: 4 additions & 22 deletions packaging/macos/connections/self_monitoring/logs_and_metrics.yaml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,5 @@
receivers:
filelog/agent-config: # TODO: Add observe-agent.yaml once we can obfuscate sensitive config fields
# TODO deprecate this
include: [/usr/local/observe-agent/connections/common/base.yaml.tmpl]
include_file_path: true
start_at: beginning
poll_interval: 5m
multiline:
line_end_pattern: ENDOFLINEPATTERN
# TODO: Add observe-agent.yaml once we can obfuscate sensitive config fields

prometheus/agent:
config:
Expand Down Expand Up @@ -39,18 +32,7 @@ service:
- batch
exporters: [prometheusremotewrite/observe]

# Deprecated
logs/agent-config:
receivers: [filelog/agent-config]
processors:
- memory_limiter
- transform/truncate
- resourcedetection
- resourcedetection/cloud
{{- if .HasAttributes }}
- attributes/observe_global_attributes
{{- end }}
{{- if .HasResourceAttributes }}
- resource/observe_global_resource_attributes
{{- end }}
- batch
exporters: [otlphttp/observe]
receivers: [nop]
exporters: [nop]
3 changes: 1 addition & 2 deletions packaging/windows/connections/host_monitoring/host.yaml.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
receivers:
# TODO deprecate this
filestats/agent:
include: 'C:\Program Files\Observe\observe-agent\connections\common\base.yaml.tmpl'
include: "${env:OBSERVE_AGENT_CONFIG_PATH}"
collection_interval: 5m
initial_delay: 60s

Expand Down
Loading
Loading