Rework AL2 dockerfiles

[ShelbyZ]

Summary

• Rework dockerfiles for cleaner build
• Update Makefile targets

export DOCKER_BUILDKIT=1 was added to buildspec to enable buildkit which was required for using newer features like - COPY --chmod=0755 entrypoint.sh /entrypoint.sh

Where did my Dockerfile* go?

The following dockerfiles are new:

• build/Dockerfile.deps-al2 - provide AL2 dependencies used during build process
• build/Dockerfile.build-common - Setup golang, source configuration files, and fluent-bit source
• build/Dockerfile.compile - builds fluent-bit source, uses RELEASE/DEBUG args to control build flags
• build/Dockerfile.compile-init - builds init-process
• build/Dockerfile.golang - golang tooling for init-process/plugin builds
• runtime/Dockerfile.deps-al2 - provide AL2 dependencies used during runtime process
• runtime/Dockerfile - sources compiled binaries, plugins and uses our runtime image with default entrypoint
• runtime/Dockerfile.init - builds on the runtime image and overrides for init process
• runtime/Dockerfile.deps-debug-al2 - provide debug AL2 dependencies used during runtime process
• runtime/Dockerfile.debug-common - updates runtime image with awscliv2, core_uploader
• runtime/Dockerfile.debug - Sets up default core uploading process via s3
• runtime/Dockerfile.init-debug- updates the init image with core uploading process via s3
• runtime/Dockerfile.debug-efs - updates the debug image with core uploading process via efs
• runtime/Dockerfile.init-debug-efs - updates the debug init image with core uploading process via efs
• runtime/Dockerfile.debug-valgrind - updates the debug image to run with valgrind

The following dockerfiles were removed/merged:

• Dockerfile.build - split across build/Dockerfile.deps-al2, build/Dockerfile.build-common
• Dockerfile.build-init - renamed/moved to build/Dockerfile.compile-init
• Dockerfile.init-debug-base -duplicated, now using runtime/Dockerfile.init
• Dockerfile.init-debug-fs - renamed/moved to runtime/Dockerfile.init-debug-efs
• Dockerfile.init-debug-s3 - split across runtime/Dockerfile.debug-common and runtime/Dockerfile.init-debug
• Dockerfile.init-release - renamed/moved to runtime/Dockerfile.init
• Dockerfile.main-debug-base - split across build/Dockerfile.compile, runtime/Dockerfile.deps-debug-al2, runtime/Dockerfile.debug-common
• Dockerfile.main-debug-fs - renamed/moved to runtime/Dockerfile.debug-efs
• Dockerfile.main-debug-s3 - split across build/Dockerfile.compile, runtime/Dockerfile.deps-debug-al2, runtime/Dockerfile.debug-common, runtime/Dockerfile.debug
• Dockerfile.main-debug-valgrind - renamed/moved runtime/Dockerfile.debug-valgrind
• Dockerfile.main-release - split across build/Dockerfile.compile, runtime/Dockerfile.deps-al2, runtime/Dockerfile
• Dockerfile.plugins - renamed/moved to build/Dockerfile.compile-plugins

What Makefile changes occurred?

The following targets were introduced:

• build-common - builds build/Dockerfile.deps-al2, build/Dockerfile.build-common, build/Dockerfile.golang, build/Dockerfile.compile-init

The following targets were updated:

• build (depends on build-common) - builds build/Dockerfile.compile
• build-debug (depends on build-common) - builds build/Dockerfile.compile with RELEASE=Off and DEBUG=On
• windows-plugins/linux-plugins - move the export OS_TYPE to target
• release (depends on build, linux-plugins) - builds runtime/Dockerfile.deps-al2, runtime/Dockerfile, runtime/Dockerfile-init
• debug (depends on build-debug, linux-plugins) - builds runtime/Dockerfile.deps-debug-al2, runtime/Dockerfile, runtime/Dockerfile.debug-common, runtime/Dockerfile.debug, runtime/Dockerfile.init, runtime/Dockerfile.debug-init, runtime/Dockerfile.debug-efs, runtime/Dockerfile.init-debug-efs
• debug-valgrind (depends on debug) - builds runtime/Dockerfile.debug-valgrind
• (cloudwatch/kinesis/firehose)-dev (depends on build) - added export OS_TYPE (required) and updated to use build_plugins.sh script
• clean - Change image cleanup to use fuzzy matching to catch all images

The following targets were removed:

• build-init - moved to build-common
• main-debug, init-debug, main-debug-all, debug-fs, debug-s3, init-debug-all, init-debug-fs, init-debug-s3 - moved to debug as the rework allowed the later image build stages to just override the runtime behavior making the docker build process faster

Why this change

The current Dockerfiles had duplication of purpose and make it a bit more difficult to re-use when targeting newer OS releases. This is an attempt to split out concerns across:

• Building plugins, source, init-process
• Providing dependencies required at correct build/runtime stages
• Allowing re-use of prior stages across image builds
• Configuring runtime images for behaviors (release, debug, init)

Additionally output images are tagged with $AL_TAG or 2 to assist with multi-stage builds targeting the correct Dockerfile stage. This is required to enable building for newer OSes as we expect to override AL_TAG and need to target the specific build stage and not a generic (or rebuild the stage).

How this moves closer to AL2023

With this change in place, we can look to create new Makefile targets for AL2023 for release/debug and provide overrides for AL_TAG, FLB_VERSION, FLB_REPOSITORY like:

.PHONY: release-al2023
release-al2023: AL_TAG=2023
release-al2023: FLB_VERSION=v4.0.3
release-al2023: FLB_REPOSITORY=https://github.com/fluent/fluent-bit.git
release-al2023: release

We need to create dockerfiles in scripts/dockerfiles for build/Dockerfile.deps-al2023 (build dependencies) and runtime/Dockerfiles.deps-al2023 (runtime dependencies).

Testing

Testing was conducted in a few phases:

1. Test change Makefile targets - release, debug, debug-valgrind, (cloudwatch/kinesis/firehose)-dev, clean
2. Test dockerfiles created from Makefile targets and ensure they run
3. Test build through pipeline and ensure images produced an tagged in amazon/aws-for-fluent-bit-test

For 1, this required running each Makefile target. For image producing targets, both docker system prune -a -f and docker image prune -a -f was run to ensure a clean slate between runs. This identified any Maekfile target dependencies.

For 2, this required running the following images:

• latest-al2
• debug-al2
• debug-efs-al2
• init-latest-al2
• init-debug-al2
• init-debug-efs-al2
• debug-valgrind-al2

For 3, ran the build through a pipeline account and ensured the images built to amazon/aws-for-fluent-bit-test matching the expected as follows:

• latest (manifest list containing 2/3)
• arm64
• amd64
• init-latest (manifest list contaning 5/6)
• init-arm64
• init-amd64
• debug-arm64
• debug-amd64

Images pushed/manifests can be observed in buildspec.yml. Also testing that these images function as expected.

make debug succeeded: yes
Integ tests succeeded: yes (local/build system)
New tests cover the changes: no

Description for the changelog

Rework AL2 dockerfiles

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[danehlim]

To confirm - was running this cleanup command unnecessary to begin with? Is that why it is no longer present after these changes? Or is it because there is some incompatibility between running this command and the reworking of Dockerfiles via this pull request?

[ShelbyZ]

Per docs:

Remove all unused containers, networks, images (both dangling and unused), and optionally, volumes.

Which in practice would cleanup the intermediary build stage images. What is a bit problematic is that it also cleanup up containers, networks, images, volumes not generated by this project and felt like overreach.

If we feel strongly about thet cleanup, we could rework it to docker image prune -f and sprinkle it as the last step in each Makefile target that uses docker build or scripts that do

[danehlim]

Got it, thanks! I have a slight preference toward that, but not a blocker for me.