Skip to content

[aws-for-fluent-bit] does not Filter: Kuberntes for Labels, Annotation #1164

@sav-work

Description

@sav-work

Describe the bug
I want to filter out all stdout.
I pass a parameter to the filter and it works
Exclude $stream stdout

But I want to have the opportunity to include it separately via Labels: fluentbit-logging=stdout or Annotations: in pods or namespace, this is what my config looks like now
Regex $kubernetes['labels']['fluentbit-logging'] stdout
And it doesn't work the whole stdout is filtered.

Maybe there's another way I don't know
Thank you for your help.

Other methods
I also tried different options and it didn't work for me.
option 1

    [FILTER]
        Name                grep
        Match               application.*
        Exclude             $stream stdout

    [FILTER]
        Name                grep
        Match               application.*
        Regex               $stream stdout
        Regex               $kubernetes['pod_annotations']['fluentbit-logging'] stdout
        Regex               $kubernetes['labels']['fluentbit-logging'] stdout`

And it doesn't work the whole stdout is filtered.

Expected outcome
Filter separately through the Label or Annotations for Pods, Namespace

  • Chart name: manifest aws-for-fluent
  • Chart version: aws-for-fluent-bit:2.31.11
  • Kubernetes version: 1.28
  • Using EKS (yes/no), if so version? 1.28

Additional Context:
Configmap
data:
application-log.conf: |

    [INPUT]
        Name                tail
        Tag                 application.*
        Path                /var/log/containers/fluent-bit*
        multiline.parser    docker, cri
        DB                  /var/fluent-bit/state/flb_log.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}

    [FILTER]
        Name                kubernetes
        Match               application.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_Tag_Prefix     application.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Labels              On
        Annotations         On
        Use_Kubelet         On
        Kubelet_Port        10250
        Buffer_Size         0


    [FILTER]
        Name                grep
        Match               application.*
        Exclude             $stream stdout

    [FILTER]
        Name                grep
        Match               application.*
        Regex               $kubernetes['labels']['fluentbit-logging'] stdout

    [OUTPUT]
        Name                cloudwatch_logs
        Match               application.*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/logs
        log_group_template  /aws/containerinsights/${CLUSTER_NAME}/workload/$kubernetes['namespace_name']
        log_stream_prefix   ${HOST_NAME}-
        log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
        auto_create_group   true
        extra_user_agent    container-insights
        log_retention_days  30
        Log_Level           debug

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions