Terraforming Gitlab Projects

Up until now we’ve manually created groups and projects in GitLab through the web interface. This has been perfectly adequate, but for a while I’ve been meaning to automate this for some consistency, and the trigger for getting to this now was that our versioning and release workflow was getting out of whack on one of our projects.

We follow semver versioning (so versions are all major.minor.bugfix) and each released version that follows this should only be on the main branch, be based on need or a logical grouping of features, and have a proper changelog entry. Because I wasn’t managing it properly, the team was creating versions on a library project so they could get the version they needed in the app project.

The proper way to do this is with a dev release: if the next version is 4.4.1, create a release 4.4.1-dev1 (or dev2, etc.). These can be created on any branch and should trigger a development package, and that’s what is happening. However, when I looked we had multiple non-dev releases that don’t align the way I would want them to.

GitLab provides a mechanism, protected tags, to help enforce such policies. Protected tags are defined by a simple wildcard pattern and a role–tags that match this pattern can only be pushed by somebody matching that role. So, we have rules that only the owner/maintainer can push non-dev tags, and these are applied on some of our projects, but not all of them.

There are a few ways to achieve the consistency we want:

1. Policy, process and documentation: A good start, but only works if I follow them every time; it becomes more and more tedious as we find more things to customize, or GitLab introduces more features; it’s a slog to implement a new configuration across all existing projects; it’s easy to miss something or make mistakes over time. GitLab has a lot of features.
2. Custom project templates but these are only available if you’re paying for GitLab, and our group can’t afford it.
3. The GitLab API is powerful and easy to use, and that was originally my plan.
4. Terraform has a GitLab provider which means I can just provide my configuration and Terraform does the rest for me.

So, Terraform’s the way to go. I love the idempotency and that I can easily verify what it’s going to do before anything is created, updated, or removed.

Super basic overview of Terraform

We have an Introduction to Terraform and several other articles but here’s a quick overview of what this is:

Terraform is an infrastructure-as-code solution where infrastructure resources are defined using structured text using a syntax kind of like if JSON went through a teleporter while TOML was hiding in the back smoking a cigarette. This configuration in .tf files can be either concrete, where each resource block defines a single entity, or can be extensible with the use of variables, in which case the .tf files can be thought of as configuration templates. Variables are stored in .tfvars files. Terraform maintains state, and on every run compares the live state of the target infrastructure with the stored state and the configuration. Any differences are actionable: if a thing is in the configuration but not in the state, it must be created; if it’s in the state but not in the configuration, it must be destroyed. Before any such action is taken the user is prompted to proceed.

One of the most important things about Terraform is its idempotency. If you’ve defined a thing, and you run Terraform, it’ll create the thing. If you run it again without changing any definition, it’ll see that there is nothing to do.

For more information, consult the Terraform site and see what infrastructure it can support in its registry. Terraform is a major component of the STRAP project.

Setting it up

Defining projects

I don’t anticipate mucking about with groups much–they’re just containers for projects or other groups–so I start with projects.

I briefly consider whether I want to sort-of-modularize the projects or define them independently. If I define them independently, the best way is a separate .tf file for each–if I want to go back and reapply a configuration item to each one, and the files are almost identical, it’s simpler if I can just do goofy but necessary things like compare line counts and then move to diffs and so on. This wouldn’t be possible with a single file of projects. But then I might wind up with a somewhat unwieldy collection of files, and if I want to apply a change across all the projects, I’ll need to script that or open up all the files in my editor and embrace the tedium.

So while Terraform discourages overly abstracting your infrastructure (or at least they used to, tbh I’m not going to hunt down that reference right now), I’m going to sort-of-modularize the projects by defining a map of names to project definitions objects, and then I’ll manage the projects from that.

The danger there is the opposite: it becomes too easy to make broad changes affecting current resources. Terraform provides safeguards, chiefly that it compares current live state of the targets against its saved state and the configuration, and tells you what will be affected. Additionally, some resources support a “lifecycle” clause in which you can ignore changes to specific aspects of the target.

On balance the latter gives me the control I need, it’s easier to track changes to the project configuration templates as well as the definitions for each project, and there’s the added bonus that I can store the configuration templates in a public repository for others to use and improve while keeping the actual definitions separate.

The two pieces look a bit like this (I’ve cut them down significantly), starting with projects.tf:

variable "projects" {
  description = "Specifications for projects"
  type = map(object({

    # basics
    name                   = optional(string)
    description            = optional(string)
    topics                 = optional(list(string), [])
    default_branch         = optional(string, "main")
    initialize_with_readme = optional(bool, false)
    visibility_level       = optional(string, "private")

    # CI
    group_runners_enabled = optional(bool, true)
  ...
}

This is heavily reduced from what’s actually in there, because it’s pretty long. Basically I’ve represented in this object everything I care about in a project, from metadata to features I want or don’t want. I’ve defined defaults such that when I add a new project I only need to specify the name and description and whatever I want or don’t want that deviates from the default.

Next in the same projects.tf file is the configuration template for the project resources. This is also just a sample of what is in there, and I’ve also redacted the name, namespace ID and path because these are determined from other variables and are a bit complex. See the source to see how they’re determined.

Note that in this template I have a hardcoded setting: at this point I don’t want anybody outside the team to have access to the pipelines so builds_access_level is set to “private”. If at a later date I decide I need this for some reason I can add it to the variables with a default value matching the previous hardcoded value and existing projects won’t be affected.

resource "gitlab_project" "projects" {
    for_each = var.projects

    # basics
    name         = ...  # The name, namespace_id and path are complex and ugly
and
    namespace_id = ...  # will be confusing and/or scary for somebody new to
    path         = ...  # Terraform.  They have to be derived.  See the
source.
    description  = each.value.description
    topics       = each.value.topics

    default_branch         = each.value.default_branch
    initialize_with_readme = each.value.initialize_with_readme
    visibility_level       = each.value.visibility_level

    # CI
    builds_access_level   = "private"
    group_runners_enabled = each.value.group_runners_enabled
    ...
}

Here is such an example defined in terraform.tfvars, where most of the defaults are good but the project requires a container registry (this project is for one of our container images used for testing and CI):

projects = {
  ...
  "uvic-arcsoft/testers/tester.node" = {
      name                            = "tester.node"
      visibility_level                = "public"
      container_registry_access_level = "enabled"
    }
  ...
}

There are some details here I am glossing over–this project name is not actually a valid name, but when used to create a project resource is broken down into the group and the project name before Terraform goes to provisioning.

Defining groups

I soon discover that the GitLab provider for Terraform requires a namespace ID unless you are creating the project in your user namespace. To specify that I want the projects under the team’s group or under nested groups, of which we have several, I need the numerical group IDs. This is fairly easy to find via the web but I hate the idea of having to copy these numbers into the variables file when the information is available via the API.

If I could have just specified the group path (uvic-arcsoft or uvic-arcsoft/subgroup, etc.) instead of the namespace ID, I would just do that in the project definition and not bother with groups, which I manage less often. That does not appear to be an option, unfortunately, so I consider the options:

1. Swallow my distaste and just create a map of project paths to namespace IDs in terraform.tfvars.
2. Not do that.

Clearly #2 is the correct answer, but I just have to figure out how.

There is a chicken-and-egg problem here: to get the namespace IDs, I can use Terraform’s data sources which query information from the target infrastructure rather than create resources. These can be used in provisioning the resources. The problem is that I have decided I want to abstract out the project definitions, and since variable definitions are evaluated before configuration code, I can’t use data sources to define variable values.

To solve this I need to violate another Terraform convention, which I’ll get to in the next section. But in the meantime, I realized I may as well provision the groups in Terraform as well, rather than retrieve them as data sources. This gives me what I need to create projects but the ability to create groups as well is worth it for the small amount of extra work.

Two-stage deployment

Terraform has a mechanism for providing information about provisioned resources and can produce this output in JSON. It can also read variables and configuration expressed as JSON. So if I get the information about groups into a file, and then read that file as variable definitions when provisioning projects, I should have what I need.

A typical Terraform run reads in all variable and configuration files in the current directory, builds a dependency graph, reads state from the infrastructure, and decides what it needs to do in what order. We need to separate the group and project runs so that the output of one can be in the input of the other. To do this we can use two separate directories, or we can use targeted provisioning which is discouraged and produces a warning.

Warning: Resource targeting is in effect

You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by the current configuration.

The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically suggests to use it as part of an error message.

(Pssh, whatever, bruh.) The documentation adds that a valid use of targeting is for “working around Terraform limitations” which is what I’m doing here.

It’s simpler and easier to ignore the warning than to break this into subdirectories. Well, so long as we don’t have to enter in the Terraform commands which are now going to be longer than a simple terraform apply. We also want to make sure that the commands are run in the correct order with the correct inputs.

So enter that age-old Unix utility: make.

Using the Makefile, I set up targets for both groups and projects. The groups target fires when groups.tf is updated and creates a file groups.json from the Terraform output. The projects target depends on this and the projects.tf and runs whenever either are updated.

The basic idea looks like this:

.PHONY: projects
projects: projects.json

projects.json: projects.tf groups
  terraform apply -target=gitlab_project.projects -compact-warnings
  terraform output -json projects | tee projects.json

.PHONY: groups
groups: groups.json

groups.json: groups.tf
  terraform apply -target=gitlab_group.groups -compact-warnings
  terraform output -json groups | tee groups.json

Both non-phony targets should include the main variables file as a dependency, but currently I have both the groups I want to query and the projects I want to provision defined in there, and I want to consider whether I want to separate these out–I don’t want an added group to kick off the project target, but that doesn’t really matter, does it? Since Terraform is idempotent we can just run them every time without concern.

But what if an idiot comes along

The idiot is typically me in two months when I no longer remember the complexities I’ve just described. If I come along, add a project and a group, and run terraform apply on this, it will fail because the group namespace ID won’t be included in the groups.json that did not get recreated.

To avoid this, I had thought to create a null resource that does nothing but run another old Unix standby, false. This trivial program simply returns 1, indicating failure. In other words, this will always fail, and the only way to avoid running it is by excluding the resource. So an untargeted terraform plan or apply will trigger it, unlike the two-step process I created.

Sadly, this doesn’t work. The provisioner doesn’t need to run on plan and on apply it doesn’t necessarily run first–and even so it does not block the rest from running, so the end result is you just get an extra error, on top of any project creation failures resulting from missing groups.

The Makefile should handle this anyway, so it should be enough to just run make. I don’t see any way an operation could do harm, at least not without the user having to approve it, and if terraform apply fails due to a missing group that’ll poke me to look at the README so this is good enough.

Putting it to work

I’ve used a test project to develop this. (Part of me feels a twinge of guilt for not using a test instance of GitLab, but that’d be a lot of work and I’m not hammering the API anyway.) Once I have things working, I delete everything with terraform destroy. I’m careful to double-check that I’m not destroying any resources I forgot about creating and want to keep, of course, before entering yes to the confirmation prompt.

I create a new group and project resource for a new project we’re working on for a client, and that looks pretty good. The next step will really help prove the efficacy and convenience of the solution.

Importing existing groups and projects

Terraform offers an import facility for bringing existing infrastructure into its management. The basic idea is straightforward, and though it can be a bit tricky in practice, it generally works well, so long as the provider supports it.

Importing a resource is kind of a half-reversal of creating one: you create the configuration defining the resource you want, but then instead of applying that configuration to the infrastructure to create it, you give Terraform the ID of existing infrastructure which it scans and reads into its state. Note that the definition does not need to cover all possible aspects of the resource: you could import a thing based only on its ID if you only want Terraform to be aware of and manage its existence. As you update the configuration, though, Terraform will compare that against the current state and determine whether there are changes to apply.

If you import a resource with a more robust definition, the import should go fine, but when you next invoke plan or apply Terraform will find the differences between state and definition and plan and/or apply the necessary updates as usual.

In my case I’ve defined what I want to define about groups and projects, and it’s a simple process to import them all: I just need their unique IDs and then issue terraform import gitlab_project.projects["projectname"] id. I’ve defined the project name in the configuration, so I just need the ID. This is available from the UI–currently on the project’s main page in the dot menu on the far right, next to “Star” and “Fork”–but going through all of our projects and pulling these out would be tedious, so I’ve reused some code I had to pull data from GitLab’s GraphQL API and I’m getting the IDs that way. I won’t get into that in detail, but here’s the query I used:

[{
  "query": "query {
    group(fullPath: \"uvic-arcsoft\") {
      projects {
        nodes {
          id
          fullPath
          name
        }
      }
    }
  }"
}]

Here’s the call to query it, using my personal access token:

curl --request POST \
  --header "PRIVATE-TOKEN: $GL_PAT" \
  --header "Content-Type: application/json" \
  --url "https://gitlab.com/api/graphql" \
  --data @ids.ql

I then have an import script to do a bunch of these at once:

projects=(
  uvic-arcsoft/group1/project1  000000001
  uvic-arcsoft/group1/project2  000000002
  uvic-arcsoft/group2/project1  000000003
)

c=0
stop=${#projects[@]}
for ((c=0; c<stop; c+=2))
do
  project=${projects[c]}
  id=${projects[c+1]}
  terraform import "gitlab_project.projects[\"$project\"]" $id
done

I’ve done them in batches as I have an inconsistent group and project structure: some projects are in subgroups, some are not, and I have an instance of subgroups of subgroups.

Creating a new project

At this point I want to make this Terraform code available for others, so I’m going to create a project for it, with it. I don’t need any of the features so the defaults are fine and the definition is therefore minimal: the only deviation is I want this project to be public. I’ve done all the hard work already: what’s left here is coming up with a name and in what group it should live.

Once I’ve done that I can create a new variable definition:

projects = {
  ...
  "uvic-arcsoft/misc/groups-and-projects.tf" = {
    name             = "Terraform for groups and projects"
    visibility_level = "public"
  }
  ...
}

This goes ahead without any problem and I have a new project. Hmm, it’d be nifty if the outputs for this included the remote URL, so I could go ahead and plug that into git right away! Sounds like an enhancement for next time.

Possible enhancements

1. As previously mentioned, it would be handy if the output included the remote URL which could then be immediately set in the git repo via git remote add origin $url.

2. It might be possible to create the groups.json from within Terraform instead of from outside. For example, if we can create a local file resource, populate it with the same data we currently do, and make the project resources depend on that, then we can get rid of the targeting, the warnings, and the Makefile.

Limitations

The solution as currently implemented supports placing projects under the main team group (uvic-arcsoft) or groups one level below that (uvic-arcsoft/clientX) such as we might do for a client with several projects with us. If we had a complex structure, we would need to figure out a way to address that, and hopefully one that’s both simple and usable, but I suspect as GitLab supports groups of groups of groups of groups a solution that could completely accommodate that would require either complex code or tedious data entry. For now, at least, a nearly flat structure of uvic-arcsoft/clientX/projectY is simple and effective.

Conclusion

There are some unexpected benefits to this. First, I realized early on that it would be simple to limit the GitLab features to only those that we need. GitLab is always adding new features and they are often enabled by default. I don’t think they’re designed such that they consume resources in any serious way if they’re not in use, and I don’t see them as particularly vulnerable attack surfaces, but it’s good practice to not enable services or components that are not needed.

Second, this had a knock-on surprise benefit to limiting services for a project, is that it significantly reduces all the options in the navigation panel, which reduces the noise.

There are also the expected benefits: it’s easier to provision projects, keep them consistent, and ensure the configurations we want are in there.