Using Buildah over DinD for building container images
In our CI pipelines, we originally relied on Docker-in-Docker (DinD)–which worked, but not without its drawbacks. This article aims to clarify why DinD is not apt for container image builds and why we shifted to using Buildah instead.
Why we dropped Docker-in-Docker (DinD)
We initially used DinD for our container image build jobs within our CI pipelines. The way DinD achieves this is by running Docker containers inside another Docker container (hence the name). However, there is a drawback: DinD requires the container to run with a privileged mode. This is a considerable security concern as the container would have elevated access to the host system. You can read about this issue and more in this article.
What is Buildah and why Buildah?
To replace DinD, we considered two options: Kaniko and Buildah. Kaniko, backed by Google, was developed with the aim of building container images inside unprivileged containers–exactly what we were looking for! Alas, Google stopped maintaining it. Some further research showed us that GitLab CI dropped support for Kaniko and no longer maintains it, you can read more here.
The original developers of Kaniko have picked it up again at Chainguard. Our next option led us to Buildah. Backed by Red Hat, Buildah was developed to provide an efficient way of building container images that were OCI (Open Container Initiative) compliant. Here are some benefits Buildah provides us over DinD:
- No privileged access or Docker Daemon requirement
- Has GitLab CI support
- Comes with simple commands that we are familiar with already:
buildah bud
,buildah push
andbuildah login
to name a few.
What the new CI workflow looks like
Here is a practical example from one of our project’s .gitlab-ci.yml
file:
build image:
stage: build
extends: .image_deps
image:
name: quay.io/buildah/stable
entrypoint: [""]
script:
- buildah bud -f deployment/Dockerfile -t "uvic-arcsoft:id-${CI_COMMIT_TAG}" .
# Only build if there is a Dockerfile
rules:
- exists:
- deployment/Dockerfile
Mainly, we switched the job’s image to now use quay.io/buildah/stable
and utilized a Dockerfile that was already present
within the project.
Challenges
Like most cases of transitioning between tools, switching to Buildah had its fair share of hiccups. Below are three issues that we faced during implementation–and how we eventually worked around them.
-
Confusion between
buildah build
vs.buildah bud
GitLab’s official documentation on using Buildah in CI/CD (including this one and this one) both suggest using
buildah build
.Naturally, that’s what we started with–but no matter how we configured it,
buildah build
wouldn’t work with our setup. The job would always fail one way or another. After rounds of trial and error, we came across a blog post that pointed us tobuildah bud
instead.Once we switched to
buildah bud
, things started to click. We’re still not entirely sure whybuildah build
failed for our use case (especially since both commands are valid).
Tip: If you are getting stuck with
buildah build
, try to usebuildah bud
–especially if you have the same simple Dockerfile workflow.
-
Inability to utilize Makefile commands
We noticed that wrapping our
buildah bud
command inside a Makefile target caused unexpected behaviour. The exact same command that ran perfectly in.gitlab-ci.yml
would always fail when triggered via Make.This made local testing awakward and added an extra layer of friction when trying to test the commands. We still don’t know exactly why this happens. The running hypothesis is that there might be an issue with how the shell environment is invoked from Make, and some required libraries might be missing due to the invocation.
Workaround: Use shell scripts or direct CLI commands instead of a Makefile for Buildah operations.
-
The UNSHARE permissions error
A stranger issue cropped up: If we didn’t run a benign shell command (like
echo
orls -l
) before callingbuildah bud
, the job would fail with a permissions-related UNSHARE error.Turns out, this is a known issue with how Buildah invokes
unshare()
on certain systems, especially in rootless or containerized CI environments. We found a GitHub issue discussing this problem.
Workaround: Precede
buildah bud
with a simple command likels -l
or anecho
command in your CI job.
Final Thoughts
As a team striving to deliver robust applications (especially when production deployments are important), we want CI pipelines that are predictable, performant and safe. Builds are now leaner, safer and easier to understand–without wrestling with privileged containers or tricky implementations. Using Buildah has given us a clear platform to build on, whether down the road we tackle multi-architecture deployments or integrate with other build tools.
Resources
- How to Replace Docker-in-Docker in your CI/CD Pipeline
- Docker-in-Docker
- Docker vs Buildah vs Kaniko
- Build Containers in GitLab CI with Buildah