config: multiline: in_tail: filter_multiline: Add configurable buffer limit for multiline interface #10653
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Hiroshi Hatake <[email protected]> Co-authored-by: Eduardo Silva <[email protected]>
Signed-off-by: Hiroshi Hatake <[email protected]> Co-authored-by: Eduardo Silva <[email protected]>
Signed-off-by: Hiroshi Hatake <[email protected]> Co-authored-by: Eduardo Silva <[email protected]>
Signed-off-by: Hiroshi Hatake <[email protected]> Co-authored-by: Eduardo Silva <[email protected]>
Signed-off-by: Hiroshi Hatake <[email protected]> Co-authored-by: Eduardo Silva <[email protected]>
cosmo0920
requested review from
edsiper,
leonardo-albertovich,
fujimotos and
koleini
as code owners
Signed-off-by: Hiroshi Hatake <[email protected]>
…ed records Signed-off-by: Hiroshi Hatake <[email protected]>
Signed-off-by: Hiroshi Hatake <[email protected]>
Signed-off-by: Hiroshi Hatake <[email protected]>
1 is also indicated for FLB_TRUE. Signed-off-by: Hiroshi Hatake <[email protected]>
This commit updates the expected output for the 'container_mix' unit test. Previously, the multiline engine could incorrectly merge pending messages when the log stream switched between different parser types (e.g., from `docker` to `cri`). The test's original expectations were written to match this buggy behavior. Recent fixes have made the engine's state handling more robust and precise. It now correctly flushes a pending message when the parser context changes, preventing improper merges. This change aligns the test case with the new, correct logic. Signed-off-by: Hiroshi Hatake <[email protected]> Co-authored-by: Eduardo Silva <[email protected]>
cosmo0920
force-pushed
the
cosmo0920-add-limit-for-multiline-concatenation
branch
from
578f46a to
3a8abd9
Compare
WalkthroughThis change introduces a configurable buffer size limit for multiline message concatenation throughout the Fluent Bit codebase. It extends configuration structures, adds new status codes and logic for truncation, updates multiline processing and logging to handle buffer limits, and includes tests for truncation scenarios. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Config
participant MultilineEngine
participant StreamGroup
participant LogPlugin
User->>Config: Set multiline_buffer_limit
Config->>MultilineEngine: Pass buffer_limit on init
LogPlugin->>MultilineEngine: Append log line
MultilineEngine->>StreamGroup: Attempt to concatenate line
alt Buffer within limit
StreamGroup-->>MultilineEngine: FLB_MULTILINE_OK
MultilineEngine-->>LogPlugin: Success
else Buffer exceeds limit
StreamGroup-->>MultilineEngine: FLB_MULTILINE_TRUNCATED
MultilineEngine-->>LogPlugin: Warn: message truncated
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~40 minutes Poem
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (28)
✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tests/internal/utils.c (1)
849-849: Fix typo in test name.There's a typo in the test name: "test_size_to_bianry_bytes" should be "test_size_to_binary_bytes".
- { "test_size_to_bianry_bytes", test_size_to_binary_bytes }, + { "test_size_to_binary_bytes", test_size_to_binary_bytes },src/multiline/flb_ml.c (1)
886-892: Consider logging invalid buffer limit configurationsThe code silently falls back to the default when the buffer limit configuration is invalid. Consider adding a warning log to help users identify configuration issues.
limit = flb_utils_size_to_binary_bytes(ml->config->multiline_buffer_limit); if (limit > 0) { ml->buffer_limit = (size_t)limit; } else { + if (ml->config->multiline_buffer_limit && strlen(ml->config->multiline_buffer_limit) > 0) { + flb_warn("[multiline] invalid buffer limit '%s', using default %zu bytes", + ml->config->multiline_buffer_limit, FLB_ML_BUFFER_LIMIT_DEFAULT); + } ml->buffer_limit = FLB_ML_BUFFER_LIMIT_DEFAULT; }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (13)
include/fluent-bit/flb_config.h(2 hunks)include/fluent-bit/flb_utils.h(1 hunks)include/fluent-bit/multiline/flb_ml.h(3 hunks)plugins/filter_multiline/ml.c(4 hunks)plugins/filter_multiline/ml.h(2 hunks)plugins/in_tail/tail_config.c(1 hunks)plugins/in_tail/tail_config.h(2 hunks)plugins/in_tail/tail_file.c(2 hunks)src/flb_config.c(2 hunks)src/flb_utils.c(1 hunks)src/multiline/flb_ml.c(13 hunks)tests/internal/multiline.c(4 hunks)tests/internal/utils.c(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (8)
- plugins/filter_multiline/ml.h
- src/flb_config.c
- plugins/filter_multiline/ml.c
- plugins/in_tail/tail_config.h
- include/fluent-bit/flb_config.h
- plugins/in_tail/tail_config.c
- plugins/in_tail/tail_file.c
- include/fluent-bit/multiline/flb_ml.h
🧰 Additional context used
🧬 Code Graph Analysis (2)
tests/internal/utils.c (1)
src/flb_utils.c (1)
flb_utils_size_to_binary_bytes(610-694)
include/fluent-bit/flb_utils.h (1)
src/flb_utils.c (1)
flb_utils_size_to_binary_bytes(610-694)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (28)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
- GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
- GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
- GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
- GitHub Check: PR - fuzzing test
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
- GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
- GitHub Check: pr-compile-centos-7
🔇 Additional comments (14)
include/fluent-bit/flb_utils.h (1)
52-52: LGTM!The function declaration is correctly placed and matches the implementation signature.
tests/internal/utils.c (1)
801-830: LGTM! Good test coverage for the new function.The test cases cover various scenarios including edge cases and overflow conditions. The expected values are mathematically correct for binary units.
tests/internal/multiline.c (4)
114-115: LGTM!The updated expected output correctly reflects the new multiline concatenation behavior where Docker log entries are properly concatenated based on stream context.
395-398: Good defensive programming!Adding the NULL check prevents potential crashes when the callback is invoked without proper context data.
1463-1531: Well-structured test for buffer truncation!The test effectively validates the multiline buffer truncation feature by:
- Setting a realistic 80-byte limit for concatenated content
- Using appropriate Docker JSON format for testing
- Correctly expecting
FLB_MULTILINE_OKfor the first append andFLB_MULTILINE_TRUNCATEDwhen the limit is exceeded
1543-1543: Test properly registered!The new buffer truncation test is correctly added to the test suite.
src/multiline/flb_ml.c (8)
27-27: LGTM!Required include for the new
flb_utils_size_to_binary_bytes()function.
217-219: Proper truncation state tracking!The function correctly tracks and propagates the truncation state from rule processing, ensuring callers are aware when buffer limits are exceeded.
Also applies to: 267-270, 347-350
421-422: Correct status propagation!The function now properly returns the actual processing status including truncation, instead of masking it with a generic success code.
Also applies to: 495-495
607-613: Good clarification on error handling strategy!The comment properly explains why sub-parser failures don't halt processing - multiline rules should still get a chance to process the raw text.
723-754: Excellent improvement to non-matching line handling!The logic now correctly:
- Flushes all pending multiline data when a non-matching line is encountered
- Processes the non-matching line as a standalone message
- Properly tracks truncation status throughout
This prevents data loss and ensures proper message boundaries.
764-855: Consistent truncation handling across text and object paths!The implementation properly mirrors the text append logic, ensuring consistent behavior regardless of input type.
1346-1384: Important fix for empty buffer handling!The code now correctly handles the case where the multiline buffer is empty by packing the original map, preventing potential data loss.
1439-1444: Excellent observability enhancement!Adding the
multiline_truncatedmetadata field allows downstream consumers to identify when messages were truncated due to buffer limits. The flag is properly reset after flushing.Also applies to: 1484-1484
size_to_byte function just converts with 1000(K), 1000*K, 1000*M. But this function converts with 1024(KiB), 1024*KiB(MiB), and 1024*MiB(GiB). Signed-off-by: Hiroshi Hatake <[email protected]>
Signed-off-by: Hiroshi Hatake <[email protected]>
cosmo0920
force-pushed
the
cosmo0920-add-limit-for-multiline-concatenation
branch
from
31386ce to
20c7695
Compare
Signed-off-by: Hiroshi Hatake <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We added an interface for configurable buffer limit for multiline.
Also, we implemented robust processing for multiline concatenations.
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit
New Features
Bug Fixes
Tests