out_datadog: added support for site configuration field #10582
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Lucas Tembras <[email protected]>
|
Example Config File: [SERVICE]
Flush 1
Log_Level trace
Parsers_File parsers.conf
[INPUT]
Name dummy
Tag dummy
Dummy {"message": "Testing support for site confiruation ", "test": "AGNTLOG-206", "container_id": "abc123def456", "container_name": "/ecs-web-service", "container_image": "nginx:1.24-alpine", "ecs_cluster": "arn:aws:ecs:us-west-2:123456789012:cluster/production-cluster", "ecs_task_definition": "web-service:42", "ecs_task_arn": "arn:aws:ecs:us-west-2:123456789012:task/production-cluster/abc123def456-7890-1234-5678-901234567890"}
[OUTPUT]
Name datadog
Match *
apikey ${DD_API_KEY}
dd_hostname foobar
provider ecs
dd_source fluent-bit
dd_service test-service
dd_tags env:test,version:1.0
compress gzip
site us3.datadoghq.com``` |
|
Valgrind Output: ==1318749==
==1318749== HEAP SUMMARY:
==1318749== in use at exit: 0 bytes in 0 blocks
==1318749== total heap usage: 10,659 allocs, 10,659 frees, 27,179,822 bytes allocated
==1318749==
==1318749== All heap blocks were freed -- no leaks are possible
==1318749==
==1318749== For lists of detected and suppressed errors, rerun with: -s
==1318749== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) |
|
Debug Log Output: [2025/07/11 18:38:45] [ info] Configuration:
[2025/07/11 18:38:45] [ info] flush time | 1.000000 seconds
[2025/07/11 18:38:45] [ info] grace | 5 seconds
[2025/07/11 18:38:45] [ info] daemon | 0
[2025/07/11 18:38:45] [ info] ___________
[2025/07/11 18:38:45] [ info] inputs:
[2025/07/11 18:38:45] [ info] dummy
[2025/07/11 18:38:45] [ info] ___________
[2025/07/11 18:38:45] [ info] filters:
[2025/07/11 18:38:45] [ info] ___________
[2025/07/11 18:38:45] [ info] outputs:
[2025/07/11 18:38:45] [ info] datadog.0
[2025/07/11 18:38:45] [ info] ___________
[2025/07/11 18:38:45] [ info] collectors:
[2025/07/11 18:38:45] [ info] [fluent bit] version=4.0.5, commit=9def01d7de, pid=1326162
[2025/07/11 18:38:45] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2025/07/11 18:38:45] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/07/11 18:38:45] [ info] [simd ] disabled
[2025/07/11 18:38:45] [ info] [cmetrics] version=1.0.4
[2025/07/11 18:38:45] [ info] [ctraces ] version=0.6.6
[2025/07/11 18:38:45] [ info] [input:dummy:dummy.0] initializing
[2025/07/11 18:38:45] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2025/07/11 18:38:45] [debug] [dummy:dummy.0] created event channels: read=28 write=29
[2025/07/11 18:38:45] [debug] [datadog:datadog.0] created event channels: read=30 write=31
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] scheme: http://
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] site parameter set to: us3.datadoghq.com
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] uri: /api/v2/logs
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] using site for host construction: us3.datadoghq.com
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] created base hostname: http-intake.logs.
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] about to concatenate site: us3.datadoghq.com (length: 17)
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] after concatenation: http-intake.logs.us3.datadoghq.com
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] host constructed from site: http-intake.logs.us3.datadoghq.com
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] port: 80
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] json_date_key: timestamp
[2025/07/11 18:38:45] [debug] [output:datadog:datadog.0] compress_gzip: 1
[2025/07/11 18:38:45] [ info] [sp] stream processor started
[2025/07/11 18:38:45] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/07/11 18:38:45] [trace] [sched] 0 timer coroutines destroyed
[2025/07/11 18:38:45] [trace] [sched] 0 timer coroutines destroyed
[2025/07/11 18:38:46] [trace] [input chunk] update output instances with new chunk size diff=403, records=1, input=dummy.0
[2025/07/11 18:38:46] [trace] [sched] 0 timer coroutines destroyed
[2025/07/11 18:38:46] [trace] [sched] 0 timer coroutines destroyed
[2025/07/11 18:38:46] [trace] [sched] 0 timer coroutines destroyed
[2025/07/11 18:38:46] [trace] [sched] 0 timer coroutines destroyed
[2025/07/11 18:38:47] [trace] [task 0x5e9f010] created (id=0)
[2025/07/11 18:38:47] [debug] [task] created task=0x5e9f010 id=0 OK
[2025/07/11 18:38:47] [trace] [input chunk] update output instances with new chunk size diff=403, records=1, input=dummy.0
[2025/07/11 18:38:47] [trace] [upstream] get new connection for http-intake.logs.us3.datadoghq.com:80, net setup:
net.connect_timeout = 10 seconds
net.source_address = any
net.keepalive = enabled
net.keepalive_idle_timeout = 30 seconds
net.max_worker_connections = 0
[2025/07/11 18:38:47] [trace] [net] connection #38 in process to http-intake.logs.us3.datadoghq.com:80 |
WalkthroughA new "site" configuration parameter was introduced to the Datadog output plugin. The parameter is added to the configuration map and the plugin's context struct. The plugin now constructs the Datadog host using the "site" value if no explicit host is provided, with appropriate parsing, storage, and memory management for the new field. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant ConfigLoader
participant DatadogPlugin
User->>ConfigLoader: Provide config with optional 'site' and/or 'host'
ConfigLoader->>DatadogPlugin: Initialize plugin context
ConfigLoader->>DatadogPlugin: Set 'site' field if provided
alt Host not provided
DatadogPlugin->>DatadogPlugin: Construct host as "http-intake.logs." + site
else Host provided
DatadogPlugin->>DatadogPlugin: Use explicit host value
end
DatadogPlugin->>DatadogPlugin: Store constructed or provided host
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Poem
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. ✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Open
1 task
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
plugins/out_datadog/datadog_conf.c (1)
150-191: Host construction logic is correct but consider reducing debug verbosity.The implementation properly handles all three scenarios:
- Site-based host construction when no explicit host is provided
- Default host fallback when neither explicit host nor site is specified
- Explicit host parameter taking precedence
The error handling for memory allocation failures is appropriate, and the string concatenation logic is sound.
Consider reducing the debug logging verbosity as some statements provide redundant information:
- flb_plg_debug(ctx->ins, "created base hostname: %s", tmp_sds); - flb_plg_debug(ctx->ins, "about to concatenate site: %s (length: %zu)", ctx->site, flb_sds_len(ctx->site)); - flb_plg_debug(ctx->ins, "after concatenation: %s", tmp_sds); flb_plg_debug(ctx->ins, "host constructed from site: %s", ctx->host);
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
plugins/out_datadog/datadog.c(1 hunks)plugins/out_datadog/datadog.h(1 hunks)plugins/out_datadog/datadog_conf.c(5 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
plugins/out_datadog/datadog_conf.c (2)
src/flb_output.c (1)
flb_output_get_property(1091-1094)src/flb_sds.c (3)
flb_sds_create(78-90)flb_sds_cat(120-141)flb_sds_destroy(389-399)
🔇 Additional comments (3)
plugins/out_datadog/datadog.h (1)
64-64: LGTM: Clean struct addition for site parameter.The new
sitefield is properly typed asflb_sds_tand positioned appropriately with other configuration fields. The field integrates well with the existing struct layout.plugins/out_datadog/datadog.c (1)
584-587: LGTM: Well-defined configuration parameter.The site configuration parameter is properly defined with:
- Correct type (
FLB_CONFIG_MAP_STR) for string input- Appropriate optional flag (
FLB_FALSE) since explicit host can override it- Proper struct field mapping via
offsetof- Clear description with practical examples of Datadog site values
plugins/out_datadog/datadog_conf.c (1)
264-266: LGTM: Proper memory cleanup for site field.The cleanup logic correctly follows the established pattern with a null check before calling
flb_sds_destroy(), ensuring no memory leaks for the site parameter.
Comment on lines
+102
to
+109
| /* Parse site parameter early so it's available for host construction */ | ||
| tmp = flb_output_get_property("site", ins); | ||
| if (tmp){ | ||
| ctx->site = flb_sds_create(tmp); | ||
| flb_plg_debug(ctx->ins, "site parameter set to: %s", ctx->site); | ||
| } else { | ||
| flb_plg_debug(ctx->ins, "no site parameter found"); | ||
| } |
There was a problem hiding this comment.
Add null check after flb_sds_create().
The code should check if flb_sds_create() returns NULL to handle memory allocation failures gracefully.
Apply this diff to add proper error handling:
/* Parse site parameter early so it's available for host construction */
tmp = flb_output_get_property("site", ins);
if (tmp){
ctx->site = flb_sds_create(tmp);
+ if (!ctx->site) {
+ flb_errno();
+ flb_datadog_conf_destroy(ctx);
+ return NULL;
+ }
flb_plg_debug(ctx->ins, "site parameter set to: %s", ctx->site);
} else {
flb_plg_debug(ctx->ins, "no site parameter found");
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| /* Parse site parameter early so it's available for host construction */ | |
| tmp = flb_output_get_property("site", ins); | |
| if (tmp){ | |
| ctx->site = flb_sds_create(tmp); | |
| flb_plg_debug(ctx->ins, "site parameter set to: %s", ctx->site); | |
| } else { | |
| flb_plg_debug(ctx->ins, "no site parameter found"); | |
| } | |
| /* Parse site parameter early so it's available for host construction */ | |
| tmp = flb_output_get_property("site", ins); | |
| if (tmp){ | |
| ctx->site = flb_sds_create(tmp); | |
| if (!ctx->site) { | |
| flb_errno(); | |
| flb_datadog_conf_destroy(ctx); | |
| return NULL; | |
| } | |
| flb_plg_debug(ctx->ins, "site parameter set to: %s", ctx->site); | |
| } else { | |
| flb_plg_debug(ctx->ins, "no site parameter found"); | |
| } |
🤖 Prompt for AI Agents
In plugins/out_datadog/datadog_conf.c around lines 102 to 109, after calling
flb_sds_create(tmp) to assign ctx->site, add a null check to verify if ctx->site
is NULL. If it is NULL, handle the memory allocation failure gracefully by
logging an error message and returning an appropriate error code or cleaning up
as needed to prevent dereferencing a null pointer.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add
siteParameter to Fluent Bit Plugin for Regional Data RoutingEnter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
Bug Fixes