Using Buildah over DinD for building container images

In our CI pipelines, we originally relied on Docker-in-Docker (DinD)–which worked, but not without its drawbacks. This article aims to clarify why DinD is not apt for container image builds and why we shifted to using Buildah instead.

Why we dropped Docker-in-Docker (DinD)

We initially used DinD for our container image build jobs within our CI pipelines. The way DinD achieves this is by running Docker containers inside another Docker container (hence the name). However, there is a drawback: DinD requires the container to run with a privileged mode. This is a considerable security concern as the container would have elevated access to the host system. You can read about this issue and more in this article.

What is Buildah and why Buildah?

To replace DinD, we considered two options: Kaniko and Buildah. Kaniko, backed by Google, was developed with the aim of building container images inside unprivileged containers–exactly what we were looking for! Alas, Google stopped maintaining it. Some further research showed us that GitLab CI dropped support for Kaniko and no longer maintains it, you can read more here.

The original developers of Kaniko have picked it up again at Chainguard. Our next option led us to Buildah. Backed by Red Hat, Buildah was developed to provide an efficient way of building container images that were OCI (Open Container Initiative) compliant. Here are some benefits Buildah provides us over DinD:

  • No privileged access or Docker Daemon requirement
  • Has GitLab CI support
  • Comes with simple commands that we are familiar with already: buildah bud, buildah push and buildah login to name a few.

What the new CI workflow looks like

Here is a practical example from one of our project’s .gitlab-ci.yml file:

build image:
  stage: build
  extends: .image_deps
  image:
    name: quay.io/buildah/stable
    entrypoint: [""]
  script:
    - buildah bud -f deployment/Dockerfile -t "uvic-arcsoft:id-${CI_COMMIT_TAG}" .
  # Only build if there is a Dockerfile
  rules:
    - exists:
      - deployment/Dockerfile

Mainly, we switched the job’s image to now use quay.io/buildah/stable and utilized a Dockerfile that was already present within the project.

Challenges

Like most cases of transitioning between tools, switching to Buildah had its fair share of hiccups. Below are three issues that we faced during implementation–and how we eventually worked around them.

  1. Confusion between buildah build vs. buildah bud

    GitLab’s official documentation on using Buildah in CI/CD (including this one and this one) both suggest using buildah build.

    Naturally, that’s what we started with–but no matter how we configured it, buildah build wouldn’t work with our setup. The job would always fail one way or another. After rounds of trial and error, we came across a blog post that pointed us to buildah bud instead.

    Once we switched to buildah bud, things started to click. We’re still not entirely sure why buildah build failed for our use case (especially since both commands are valid).

Tip: If you are getting stuck with buildah build, try to use buildah bud–especially if you have the same simple Dockerfile workflow.

  1. Inability to utilize Makefile commands

    We noticed that wrapping our buildah bud command inside a Makefile target caused unexpected behaviour. The exact same command that ran perfectly in .gitlab-ci.yml would always fail when triggered via Make.

    This made local testing awakward and added an extra layer of friction when trying to test the commands. We still don’t know exactly why this happens. The running hypothesis is that there might be an issue with how the shell environment is invoked from Make, and some required libraries might be missing due to the invocation.

Workaround: Use shell scripts or direct CLI commands instead of a Makefile for Buildah operations.

  1. The UNSHARE permissions error

    A stranger issue cropped up: If we didn’t run a benign shell command (like echo or ls -l) before calling buildah bud, the job would fail with a permissions-related UNSHARE error.

    Turns out, this is a known issue with how Buildah invokes unshare() on certain systems, especially in rootless or containerized CI environments. We found a GitHub issue discussing this problem.

Workaround: Precede buildah bud with a simple command like ls -l or an echo command in your CI job.

Final Thoughts

As a team striving to deliver robust applications (especially when production deployments are important), we want CI pipelines that are predictable, performant and safe. Builds are now leaner, safer and easier to understand–without wrestling with privileged containers or tricky implementations. Using Buildah has given us a clear platform to build on, whether down the road we tackle multi-architecture deployments or integrate with other build tools.

Resources