This post looks at the problems of passing secrets into a Dockerfile using --build-arg and how BuildKit can be used to resolve these injections securely.

The Problem

Docker has ushered containers to the forefront of the DevOps workflow. Developers can bundle an application and all of its dependencies into a single image and distribute it to system operators without worrying about triggering dependency hell.

As developers, we can take these benefits one step further. Rather than using containers to simply bundle the application, we can use them for compiling the application itself. By using the Dockerfile to grab the compiler, import the source code, and fetch all of the needed compile-time dependencies, we can make the entire build environment portable.

For example, let’s look at an application written in Go that has its entire build process Dockerized:

# Select the base image for the build environment
FROM golang:alpine as builder

# The Go compiler uses git for fetching dependencies
RUN apk update && apk add --no-cache git

COPY . /source

# Compile the source code into an executable
RUN cd /source &&\
    go build -o myapp

# Select the base image for the application's final image
FROM scratch

# Copy in *just* the compiled application from the "compiler" stage
COPY --from=builder /source/myapp /myapp

# Set the application as the default container executable
ENTRYPOINT ./myapp

In step 4, the go build command uses git to fetch all of the project’s dependencies hosted on remote git servers. If any one of these external git servers requires authentication, our Docker build will fail:

=> ERROR [compiler 4/4] RUN cd /source    go build -o myapp m  4.3s
------
 > [compiler 4/4] RUN cd /source    go build -o myapp:
#11 go: gitea.reynholm.local/icarus/signon@v0.0.0-20201228180413-e9111a241be2 requires
#11 bitbucket.reynholm.local/legacy/auth-sso@v0.0.0-20190421284113-436c30eb8c98:
    invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*
    in /go/pkg/mod/cache/vcs/436c30eb8c98a2c4d8a3de9112c51e3f1ce5c2c1ff3ff58d4ab336b5145:
    exit status 128:
#11 fatal: could not read Username for 'https:/bitbucket.reynholm.local': terminal
    prompts disabled
------
executor failed running [/bin/sh -c cd /source    go build -o myapp]: exit code: 1

It’s tempting to quickly modify the Dockerfile to grab these sensitive credentials from build arguments:

...
COPY . /source

ARG BITBUCKET_USER
ARG BITBUCKET_PASS

RUN git config --global url."https://${BITBUCKET_USER}:${BITBUCKET_PASS}@bitbucket.reynholm.local".insteadOf "https://bitbucket.reynholm.local" &&\
    cd /source &&\
    go build -o myapp

FROM scratch
...

And pass them into the Dockerfile from the command line:

docker build --build-arg BITBUCKET_USER=MY-NAME --build-arg BITBUCKET_PASS=MY-PASSWORD .

But this hasty resolution to our build problem will expose our username and password in both image history and build logs!

Exposure A: Build Args Are in Image History

To speed up build times, Docker includes a build cache. As images are built, each Dockerfile instruction becomes a layer and is added to this cache. As the instructions are read on subsequent builds, cached layers will be reused to the point that the commands’ text hasn’t changed. Since changing the contents of a build arg would be equivalent to changing the command, the arg contents are stored in the layer history as plaintext. This history can be seen visually using the docker image history <image-name> command:

IMAGE          CREATED         CREATED BY                                                 SIZE
b5ec8453da3a   5 seconds ago   ENTRYPOINT ["/bin/sh" "-c" "./source/myapp"]               0B
<missing>      5 seconds ago   RUN |2 BITBUCKET_USER=MY-NAME BITBUCKET_PASS=MY-PASSWOR…   2.04MB
<missing>      5 seconds ago   ARG BITBUCKET_PASS                                         0B
<missing>      5 seconds ago   ARG BITBUCKET_USER                                         0B
<missing>      5 seconds ago   COPY . /source                                             1.26MB
...

Everyone that can access either the build cache or the built layers can view the plaintext username and password. Worse yet, if we pushed one of these layers to an image registry, anyone with the ability to pull it down would be able to see it as well!

There were discussions to encrypt build-args as they got appended to image history, but the Docker maintainers decided against this to prevent giving the impression they were suitable for secrets; see moby issue #13490.

Exposure B: Build Args Contents Are in the Build Output

As the Dockerfile is parsed, each instruction gets expanded into its final form. This interpolation replaces all build-args’ names with their values and displays them in the build’s output.

...
 => [compiler 3/4] COPY . /source                                              0.0s
 => [compiler 4/4] RUN ...url."https://MY-NAME:MY-PASSWORD@bitbucket.reyn...   0.6s
 => [stage-1 2/2] COPY --from=compiler /source/myapp /myapp                    0.0s
...

This exposure might not be a big deal if we only build the image locally, and we’re hyper-vigilant not to share our screens (good luck maintaining this privacy in troubleshooting mob sessions). But if the image is ever built on a shared Jenkins instance, anyone with access can see the build logs, along with our password!

Using Buildkit to Inject Build-Time Secrets

Fortunately, there are alternatives to using build-args to inject sensitive information into a Dockerfile at build-time! Since the release of Docker 18.09, an alternative build engine known as BuildKit, is available. Among the many features, it includes two methods for overcoming our problem. Secrets mounting allows files to be temporarily mounted as part of the build, and SSH forwarding allows SSH agents to be mounted directly into the build process.

Method A: File Mounting Secrets

With BuildKit we can safely mount files containing sensitive information to individual RUN instructions. These files’ content won’t be persisted in the build cache, don’t get stored in the final image, and are only available to the RUN instructions they are bound to.

In our example Dockerfile, the goal is to safely inject our username and password into the image as it’s building, allowing the Go compiler to authenticate and download all of the needed dependencies. We can this by mounting a .netrc file into the root user’s home folder.

Step 1: Add a secret mount to the relevant RUN instruction, assigning an ID it can be referenced by, and specifying where to mount the .netrc file inside the image.

RUN --mount=type=secret,id=bitbucket,dst=/root/.netrc; && \
    cd /source &&\
    go build -o myapp

# This instruction would fail because the secret file has not been mounted and
# isn't available instruction.
# RUN cat /root/.netrc

Step 2: Create the .netrc file on the local file system, inside Docker’s build context, with our credentials. Use the chmod to ensure only our account can access it.

{
    echo "Machine bitbucket.reynholm.local"
    echo -e "\tlogin <MY-USERNAME>"
    echo -e "\password <MY-PASSWORD>\n"
} > ./.netrc
chmod 600 ./.netrc

Step 3: Inject the local .netrc file

docker build --secret id=bitbucket, src=./.netrc .

When creating sensitive files inside of a git repo, don’t forget to add a .gitignore rule to keep them from being added to source control!

Method B: Using SSH Agent Authentication Forwarding

While mounting a .netrc file works, it’s a poor security practice to share usernames and passwords across boundaries. As an alternative, we can mimic the world of SSH by opening a connection to the secured git server and forward our SSH connection into the Dockerfile.

Forwarded SSH links get mounted on individual RUN instructions, allowing BuildKit to use it for connections with the outside world. As soon as the RUN instruction has been completed, no information will be left in the Docker image.

Step 1: Update the Dockerfile

FROM golang:alpine as compiler

# Added openssh-client to work with the forwarded SSH connection
RUN apk update && apk add --no-cache openssh-client git

# Added instruction to download public keys from the git server
# and add them to the image's known hosts
RUN mkdir -p -m 0600 ~/.ssh &&\
    ssh-keyscan bitbucket.reynholm.local >> ~/.ssh/known_hosts

COPY . /source

# Added mount for forwarded ssh connection
RUN --mount=type=ssh cd /source &&\
    go build -o myapp

FROM scratch

COPY --from=compiler /source/myapp /myapp

ENTRYPOINT ./myapp

Step 2: Build with Dockerfile with SSH forwarding enabled:

docker build . --ssh default .

Final Thoughts

Dockerfile build args are handy mechanisms for injecting values into a building image - but avoid using them for sensitive information such as passwords! If there is any concern that a --build-arg is being used for anything that may be sensitive, be mindful to keep build logs private and the offending layers from being pushed to remote registries.

Further Reading