Quick note on patching Kubernetes resources

Drew Leske

In moving to self-hosting STRAP (sort of–where we host the Strapper application on platform) we have run into an issue where we rely on a file in local storage which we can’t furnish any other way.

Web applications I’ve developed over the last number of years have not relied on local storage, but an application that runs command-line tools is likely to run up again one of those tools making assumptions about how configuration can be defined, and here we are. That tool is kubectl and it needs to be present in order to run the Helm chart that deploys the Kubernetes resources.

Huh, on that for a moment

Whoah. It just occurred to me that Kubernetes itself exposes a REST API, and Terraform is all about using REST APIs, so why are there any command-line tools involved at all? Because Helm.

We actually looked at removing Helm as a middle layer and Archie got pretty far, but the main issue we were trying to get around wasn’t solved by doing so and we ran into a couple of wrinkles that didn’t seem to be worth ironing out. But maybe we should have another look at that.

Because the Kubernetes provider for Terraform can use your local file but the credentials can be specified in the code instead or in a variable.

Interesting! This article is still relevant though because I have to patch stuff from time to time and always have to look it up.

kubectl requires credentials to talk to the Kubernetes cluster and those credentials are going to be larger than a simple username and password. So far as I can tell there is no way to specify this via environment variables or on the command line. There’s just the file.

Strapper and STRAP itself do not currently manage local storage. There’s no mechanism for copying files in or out, and I don’t want to hack one in right now. But at the same time, I want Strapper to run on STRAP, because I want to get the benefits it brings to its hosted applications–stuff I don’t want to have to implement another way, like authentication.

So, to get things going, we can (big sigh) change the deployment of the Strapper application outside of STRAP to have this information available. This is obviously not ideal. This is a temporary hack, and even though we’re going to try to make it as painless as possible, I’m fully confident having to do this is going to be annoying enough that we’ll fix it soon enough.

The plan

To get the credentials present for kubectl to use, we’re going to:

  1. Create a secret in Kubernetes containing the essential definition of a minimal ~/.kube/config file;

  2. Create a volume in the deployment based on that secret;

  3. Mount that volume in the deployment’s containers.

I’ve tested this out with kubectl edit my-deployment and it works. This would be obnoxious to have to do every time Archie or I have to redeploy Strapper, so I’ve created a patch to handle this.

How that went

Enough prose and exposition. Let’s assume our usual example application called myapp.

Created a secret from the config file, in the correct namespace:

$ kubectl -n strap-myapp create secret generic kubeconf --from-file=$HOME/.kube/config

Patches are nontrivial in Kubernetes and there are multiple types for different contexts. A starting point would be the Kubernetes documentation on API object patching but if you’re new to it you might need to look at a few different examples to get the idea. In the end our use case is pretty simple, because we’re adding things that didn’t exist before (the volumes and mounts).

Here’s the patch, saved as myapp-deploy-patch.yaml:

spec:
  template:
    spec:
      containers:
      - name: myapp
        volumeMounts:
        - mountPath: /.kube
          name: kubeconf
      volumes:
      - name: kubeconf
        secret:
          secretName: kubeconf

Here’s how we apply the patch:

$ kubectl -n strap-myapp patch deploy strapp-myapp --patch-file myapp-deploy-patch.yaml

Final words

This was supposed to be short.

I don’t use kubectl patch very often: the killer app for me to tweak something in dev/test is kubectl edit which is just brilliant. But it’s like that weird screwdriver type you have at the bottom of your toolbox that you almost never use: when you need it, there’s nothing else that quite fits.

(Is that a really good metaphor or a really bad one? I can’t tell.)