Support DinD in user-specified "container:" in jobs

[zxti]

What would you like added?

I'm able to run ARC with docker fine using containerMode: dind.

But I'd also like to enable my users to specify container: in Actions workflow jobs, and start docker builds etc. from there.

My understanding is that specifying container: on an Actions workflow job will cause the actions-runner container in the pod to spin up another docker container. But in here, there's no socket:

docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.

Why is this needed?

Users want to run their own containers, and performing Docker builds etc. is a common CI/CD task.

Today, devops needs to be a central bottleneck in configuring runner sets with these specific containers, and making sure they work with the containerType: dind.

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[nikola-jokic]

Hey @zxti,

Please correct me if I misunderstood, but you can provide a volume in your workflow which can mount the docker socket to your container.

[cniessigma]

I believe the runner hardcodes a specific path to the Docker sock when running a container job. As far as I can tell they don't let us easily overwrite this without writing our own container hook which is very heavy.

Is the alternative to change every one of our workflow files to manually mount the docker.dock to the the runner container (by default seems the dind container mode seems to mount it from the host into /run/docker/docker.sock in the runner container instead of /var/run/docker.sock)? For example:

container:
  image: <image>
  options: -v "/run/docker/docker.sock:/var/run/docker.sock"

To me that seems like quite the leakage of implementation details to the consumer of the scale sets. Perhaps I'm misunderstanding the suggestion?

EDIT: To be clear I am able to launch the job with a docker container just fine, which I'm not able to do is run a docker build within that container job because it can't find the docker socket

[piotrhryszko-img]

Also interested in this, as I'm facing similary issue. I use a shared github action in my workflow

    - name: Run Trivy vulnerability scanner with exclusions
      uses: aquasecurity/trivy-action@0.12.0
      with:
        image-ref: ${{ inputs.docker_image }}
        format: 'table'
        exit-code: '1'
        ignore-unfixed: true
        trivyignores: ${{ inputs.trivy_ignore_path }}
        severity: 'CRITICAL,HIGH'
        timeout: 15m

which executes the following command

/usr/bin/docker run --name b1cbc5785fc65dd52a4c82a2774efc8b669fef_1185fa --label b1cbc5 --workdir /github/workspace --rm -e "GOOS" -e "GOARCH" -e "GOPRIVATE" -e "CGO_ENABLED" -e "INPUT_IMAGE-REF" -e "INPUT_FORMAT" -e "INPUT_EXIT-CODE" -e "INPUT_IGNORE-UNFIXED" -e "INPUT_SEVERITY" -e "INPUT_TIMEOUT" -e "INPUT_SCAN-TYPE" -e "INPUT_INPUT" -e "INPUT_SCAN-REF" -e "INPUT_VULN-TYPE" -e "INPUT_TEMPLATE" -e "INPUT_OUTPUT" -e "INPUT_SKIP-DIRS" -e "INPUT_SKIP-FILES" -e "INPUT_CACHE-DIR" -e "INPUT_IGNORE-POLICY" -e "INPUT_HIDE-PROGRESS" -e "INPUT_LIST-ALL-PKGS" -e "INPUT_SECURITY-CHECKS" -e "INPUT_TRIVYIGNORES" -e "INPUT_ARTIFACT-TYPE" -e "INPUT_GITHUB-PAT" -e "INPUT_TRIVY-CONFIG" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "GITHUB_ACTION_PATH" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/_work/_temp/_github_home":"/github/home" -v "/home/runner/_work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/_work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/_work/example/example":"/github/workspace" b1cbc5:785fc65dd52a4c82a2774efc8b669fef  "-a image" "-b table" "-c " "-d 1" "-e true" "-f os,library" "-g CRITICAL,HIGH" "-h " "-i example/example/example:e67777a6d" "-j ." "-k " "-l " "-m " "-n 15m" "-o " "-p " "-q " "-r false" "-s " "-t " "-u " "-v "

and fails with

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Apologies if it's not related, I'm still trying to understand the flow of actions

[zetaab]

dockerd can be started using hooks:

AutoscalingRunnerSet envs

        env:
        - name: ACTIONS_RUNNER_HOOK_JOB_STARTED
          value: /home/runner/hooks/common-start.sh

common-start.sh:

#!/usr/bin/env bash
set -u

source /home/runner/hooks/logger.sh
source /home/runner/hooks/wait.sh

log.debug 'Starting Docker daemon'
sudo /usr/bin/dockerd &

log.debug 'Waiting for processes to be running...'
processes=(dockerd)

for process in "${processes[@]}"; do
    if ! wait_for_process "$process"; then
        log.error "$process is not running after max time"
        exit 1
    else
        log.debug "$process is running"
    fi
done

logger and wait scripts can be seen in https://github.com/actions/actions-runner-controller/blob/master/runner/wait.sh https://github.com/actions/actions-runner-controller/blob/master/runner/logger.sh

So: 1 env variable and 3 files needs to be injected. Its up to the user how that can be done. Configmap, new image(?). We are building own image which contains for instance cached actions and settings like this.

ps. Github default image is missing quite much dependencies to get docker started. At least fuse-overlayfs and iptables is needed for that.

Oh forgot one thing:

        securityContext:
          privileged: true # needed for dockerd

that is needed as well

[cniessigma]

dockerd can be started using hooks:

AutoscalingRunnerSet envs

        env:

        - name: ACTIONS_RUNNER_HOOK_JOB_STARTED

          value: /home/runner/hooks/common-start.sh

common-start.sh:

#!/usr/bin/env bash

set -u



source /home/runner/hooks/logger.sh

source /home/runner/hooks/wait.sh



log.debug 'Starting Docker daemon'

sudo /usr/bin/dockerd &



log.debug 'Waiting for processes to be running...'

processes=(dockerd)



for process in "${processes[@]}"; do

    if ! wait_for_process "$process"; then

        log.error "$process is not running after max time"

        exit 1

    else

        log.debug "$process is running"

    fi

done

logger and wait scripts can be seen in https://github.com/actions/actions-runner-controller/blob/master/runner/wait.sh https://github.com/actions/actions-runner-controller/blob/master/runner/logger.sh

So: 1 env variable and 3 files needs to be injected. Its up to the user how that can be done. Configmap, new image(?). We are building own image which contains for instance cached actions and settings like this.

ps. Github default image is missing quite much dependencies to get docker started. At least fuse-overlayfs and iptables is needed for that.

Oh forgot one thing:

        securityContext:

          privileged: true # needed for dockerd

that is needed as well

Making a hook is exactly what I needed to get around this, using a modified Github's published example, though I am still a bit curious why the upstream runner project has the Docker socket file path hard coded? The hook works but is somewhat undesirable -- when the workflow fails for any reason, it spits out a "Please contact your self-hosted administrator" and I haven't found a way to get rid of that error message.

I'd rather not use a hook if all that's needed is an upstream change to be able to override that hardcoded string with an environment variable.

[zetaab]

yeah I do not understand why docker is not made available automatically. That is available if you use like ubuntu-latest, why its not included in runner image?

[nikola-jokic]

Hey everyone,

Until this is resolved, may I suggest a workaround for this particular use-case. To have it working, please comment out the entire containerMode object (or leave containerMode.type empty), and provide a dind spec as a side-car container as described here.

The containerMode object does not influence the controller in any way. It just expands the helm template, so it is more convenient to specify a dind configuration in most cases.

[genisd]

@nikola-jokic i think your link for the spec should not refer to the sha abc0b678d323b but to a newer one (or master).
The sha referenced might be missleading since it says there

  ##         - name: DOCKER_HOST
  ##           value: tcp://localhost:2376

I would think that the current master might be better here
Link with the sha at the time of writing this comment here