MULTILINE_PARSER cannot be used in init tag image

[yuza1018]

Describe the question/issue

I used the init tag image and specified a file with a MULTILINE_PARSER section in the parsers_file, but this time I got an error.
MULTILINE_PARSER document

Configuration

task_definition.json

[
    {
        "name": "fluent-bit-init",
        "image": "amazon/aws-for-fluent-bit:init-2.30.0",
        "essential": true,
        "firelensConfiguration": {
            "type": "fluentbit"
        },
        "logConfiguration": {
            "logDriver": "awslogs",
            "options": {
                "awslogs-region": "<my_region>",
                "awslogs-stream-prefix": "xxx",
                "awslogs-group": "xxx"
            }
        },
        "environment": [
            {
                "name": "aws_fluent_bit_init_s3_1",
                "value": "arn:aws:s3:::<my_bucket>/error-log.conf"
            },
            {
                "name": "aws_fluent_bit_init_s3_2",
                "value": "arn:aws:s3:::<my_bucket>/parsers_multiline.conf"
            }
        ]
    }
]

error-log.conf

[SERVICE]
    flush                 1
    log_level             info
    parsers_file          parsers_multiline.conf

[FILTER]
    name                  multiline
    match                 *
    multiline.key_content log
    multiline.parser      go, multiline-regex

[FILTER]
    Name   grep
    Match  *
    regex  log error-log

[OUTPUT]
    Name   cloudwatch
    Match  *
    region <my_region>
    log_group_name xxx
    log_stream_prefix xxx

parsers_multiline.conf

[MULTILINE_PARSER]
    name          multiline-regex
    type          regex
    flush_timeout 1000

    # rules |   state name  | regex pattern                  | next state
    # ------|---------------|--------------------------------------------
    rule      "start_state"   "/^datetime.*/"                  "stack_trace"
    rule      "stack_trace"   "/^Stack.*/"                     "cont"
    rule      "cont"          "/^#\d+\s.*/"                    "cont"

Fluent Bit Log Output

The following logs were output to Cloudwatch

Sections 'multiline_parser' and 'parser' are not valid in the main configuration file. It belongs to the 'parsers_file' configuration files.

Fluent Bit Version Info

bash-4.2# /fluent-bit/bin/fluent-bit -v
Fluent Bit v1.9.10
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/02/03 20:32:43] [ info] Configuration:
[2023/02/03 20:32:43] [ info]  flush time     | 1.000000 seconds
[2023/02/03 20:32:43] [ info]  grace          | 5 seconds
[2023/02/03 20:32:43] [ info]  daemon         | 0
[2023/02/03 20:32:43] [ info] ___________
[2023/02/03 20:32:43] [ info]  inputs:
[2023/02/03 20:32:43] [ info] ___________
[2023/02/03 20:32:43] [ info]  filters:
[2023/02/03 20:32:43] [ info] ___________
[2023/02/03 20:32:43] [ info]  outputs:
[2023/02/03 20:32:43] [ info] ___________
[2023/02/03 20:32:43] [ info]  collectors:
[2023/02/03 20:32:43] [ info] [fluent bit] version=1.9.10, commit=f3bcc20e44, pid=10
[2023/02/03 20:32:43] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2023/02/03 20:32:43] [ info] [storage] version=1.3.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2023/02/03 20:32:43] [ info] [cmetrics] version=0.3.7

Considerations

I am not familiar with the go language, so perhaps, but I think that in line 188 of "aws-for-fluent-bit/init/fluent_bit_init_process.go", only the PARSE section is treated as a parser configuration.
I believe the other sections are then included in the mainConfigFile.

I changed the configuration as follows and used parsers_multiline.conf for the -R option, and it worked.

error-log.conf（Delete Service Section）

[FILTER]
    name                  multiline
    match                 *
    multiline.key_content log
    multiline.parser      go, multiline-regex

[FILTER]
    Name   grep
    Match  *
    regex  log error-log

[OUTPUT]
    Name   cloudwatch
    Match  *
    region <my_region>
    log_group_name xxx
    log_stream_prefix xxx

[jwechsler10]

Would you mind posting what you put as your override command to get this to work was @yuza1018? I am facing this issue as well and would like to try that as a workaround.

[yuza1018]

@jwechsler10
I have addressed this issue by creating a custom image that includes multiline_parser settings.
Specifically, I did the following

1. Modify fluent_bit_init_entrypoint.sh as follows

./init/fluent_bit_init_process

sed -i '/^exec/s/$/ -R \/parsers_multiline.conf/' /init/invoke_fluent_bit.sh

source /init/invoke_fluent_bit.sh

Add parsers_multiline.conf to /init/invoke_fluent_bit.sh using the sed command.

2. Creating Custom Images

FROM public.ecr.aws/aws-observability/aws-for-fluent-bit:init-2.31.0

ADD . /parsers_multiline.conf /parsers_multiline.conf
ADD . /fluent_bit_init_entrypoint.sh /init/fluent_bit_init_entrypoint.sh

RUN chmod +x /init/fluent_bit_init_entrypoint.sh

CMD /init/fluent_bit_init_entrypoint.sh

The contents of parsers_multiline.conf are as described in the first issue. Please modify it accordingly.