enhancement(splunk_hec source): Add raw_line_splitting option for NDJSON #24401
+375
−16
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
When receiving newline-delimited JSON (NDJSON) from services like CloudFlare Logpush via the
/services/collector/rawendpoint, each line should be treated as a separate event. Without this option, the entire request body is treated as a single event, which is incorrect for NDJSON payloads.This change introduces a
raw_line_splittingconfiguration option to thesplunk_hecsource. When enabled, incoming requests to the raw endpoint are split on newlines, creating a separate event for each line. Empty lines are filtered out, and trailing newlines are handled correctly.The option defaults to
falseto maintain backward compatibility with existing configurations.Includes table-driven tests covering CloudFlare-style NDJSON payloads with various edge cases (multiple events, trailing newlines, empty lines).
See also
Vector configuration
How did you test this PR?
Unit tests.
I have not tested this integration wise getting real data from CloudFlare yet.
Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
Notes
@vectordotdev/vectorto reach out to us regarding this PR.pre-pushhook, please see this template.make fmtmake check-clippy(if there are failures it's possible some of them can be fixed withmake clippy-fix)make testgit merge origin masterandgit push.Cargo.lock), pleaserun
make build-licensesto regenerate the license inventory and commit the changes (if any). More details here.