Greetings…
You might seen the title as cocktail of techs and concepts. This cocktail may give you full of enjoy and some peace while managing platforms, deploying software, operate your business on them. Of course, on condition that, you mix the cocktail well.
In this story, we will examine what is GitOps and demonstration a part of it on Openstack by using Terraform as IaC tool and Gitlab as truth source VCS.
Before start… If you want to see other stories of the series, you’re welcome:
What is GitOps and it’s part “provisioning automation”? 📕
Let’s start with “provisioning” first… Provisioning means that creation of infrastructure resources on a whatever target platform and put them into use for your compute required processes like software deployment, business operations etc. If you automate this task… Here comes… “Provisioning automation”.
But why do we need automation of it? As millions of other things in universe, there are causes of effects so we have it. These causes also outcomes of IaC principles that we follow as a truthful concept:
- Reliability (your resources should be reliable, created as you wish so business can be alive)
- Reusability (code once, use repeatedly to create a new or change)
- Effectiveness (whenever you want, you should able to do it)
As for GitOps, this is not just about “provisioning automation”.
In addition to that, based on DevOps processes like important parts CI/CD and version control, it also automates “software deployment” and “configuration management” in a software architecture. It uses Git at the core and every development or change stores in Git repository. Therefore operations are traceable and reliable from the beginning of the workflow, up to the end. This operation model is mostly using by software developers and platform engineers that working on cloud environments effectively. My personal opinion is, according to your needs, you can use GitOps concept anywhere anyplace that you can build these tools and leverage the use of target platform APIs. Thanks to tools like Terraform, it keeps your state infrastructure and application state in remote states like Terraform Cloud, GitLab or storage services like AWS S3.
Demo Use Case — Provisioning Automation 🕹️
In this story, my purpose is to show you “Provisioning Automation” as part of GitOps concept.
Our target platform is Openstack. We see how to manage Openstack resources and how to configure few things on them through GitLab CI/CD pipelines. In high level, our project will be sit on this kind of architecture:
If we’d like to see what’s going on and understand better, which stages & tasks are created in CI/CD pipeline, how those are related, we can take reference “Low Level Workflow”. (it may not align some practices in workflow drawing but hope that it’s comprehensible 🙃)
I admit that in practice, coding these may not be really easy as defining concepts. Let’s cut the chase and see how we do “in practice”…
Step-1: Create Openstack project & get your authentication details 🛫
First things first, we need to have a project to work on it so create a project in Openstack. As we use API at every stage & between tools, we need project authentication details. To get API authentication parameters from Openstack, download openrc.sh file (in bash file format, project API auth parameters) and clouds.yaml(in yaml format, project API auth parameters).
You will realize in further steps that we take reference bash parameter names since mentioned in Terraform provider docs accordingly. It also makes clear the code.
We also need to define an “application credential” to have authorization on our project. As shown below, create an application credential to use later in pipeline. Your credential has at least member and heat_stack_owner roles. Make sure that you saved your secret.
Step-2: GitLab Repository Initialization 🧱
In this project, we use GitLab as VCS and CI/CD pipeline solution so we need to create our repository in GitLab and additionally CI/CD pipeline so that we can push all files that contains necessary codes and execute task through the pipeline.
You can create your repository through GitLab UI and clone it to your local. We will get CI/CD details in further steps.
In this blog, I prefer to keep source codes and CI/CD definition in Github publicly, however, you will see some details at images in GitLab form.
If you’d like to see repository of this project, check the link below:
In addition to Openstack API authentication and application authorization details, we also need to create a secret in GitLab repository to be used in later steps. This secret allows us to reach our repository via Gitlab API and enables CI/CD execution. To make it complete, you need to go “Settings — Access Tokens”. Create your access token with api, read_api, read_repository, write_repository rights and save it securely.
Step-3: Code the Infrastructure Resources with Terraform 👨🏻💻
Feeling like it’s time to back to analogy :) This is the step where we’re starting to write cocktail recipe. How much you put from which drinks, other such methods :)
In this story, our purpose will be creation of a VM. We’ll also attach a volume to this VM and execute a startup script via Terraform basically. The script that mounts a path on a new created physical disk.
Let me briefly explain, how we code our infrastructure…
# main.tf
Here, we are defining “ci_cd_test_vm” resource object from “openstack_compute_instance_v2” resource type by declaring related object values. Values are referencing from variables.tf file and it overrides in CI/CD pipeline definition. We will mention this in later steps about how we do.
Other than Openstack instance resource, I’d like to talk about “vm” resource object that create from “terraform_data” . Why we do it? “terraform_data” which formerly known as “null_resource” and deprecated is using for two scenarios.
- It allows arbitrary values to be stored . Afterward these values can be used to implement lifecycle triggers of other resources
- It trigger provisioners when there isn’t a more appropriate managed resource available. (which is our case)
For more details please check this and this.
Even it does not create a real resource in our architecture, we use it to run remote-exec commands in correct sequence. Because, our operations that we define in the bash script config-vm.bash requires new volume attached & path mounted first on the Openstack instance. Otherwise, terraform apply will fail at remote-exec part. With “remote-exec” command, we first create a folder /tmp/conf_vm/bash_scripts/ then we put all scripts which locates./conf_vm/bash_scripts to the /tmp/conf_vm/bash_scripts/ folder on instance.
At last step, with provided connection details, our script executes on the instance.
Our basic mount bash script is like below:
# variables.tf
We define all required variables here to use in resource creation process. However, later in CI/CD definition file gitlab-ci.yml, we override these values by using terraform.tfvars file which its content defining through the CI/CD task and values come from CI/CD variables that defined in GitLab.
Variable names referrencing from Openstack provider docs in Terraform knowledge base. It also corresponds Openstack env. variable names.
# volumes.tf
We create basically Openstack block storage volume and then attach into created instance. As distinguisher, instance_id is taken.
# provider.tf
In this file we define Terraform Openstack provider and authentication details to connect Openstack API. These details can be found in openrc.sh that we download at the beginning in this story.
backend “http” {} requires for GitLab managed Terraform state configuration as mentioned here.
# outputs.tf
To have some useful details about created resources once apply process is finished, we’re defining in this file. Basically, vm_ip, vm_name, vm_image_name and vm_flavor_name outputs are sufficient for now.
Step-4: Code GitLab CI/CD Pipeline & Define Variables ✍🏻
We’re so close complete our recipe, but not yet… :)
In this step, we define our CI/CD pipeline that execute the tasks automatically on target platform and give desired infrastructure resources in a state.
To define tasks, first, we need to define “stages”. We have 5 different stages: tests, validate, plan, apply, destroy and each one of them has 1 task currently. Additionally, we have default before_script/after_script section which some operations defining here and making executable for all task at their beginning. As you can see in HLA/LLW images at the beginning of the story, each task runs in a container that provides by GitLab Runner component.
Let’s take a look at those…
# default
terraform init command is essential at every stage that we work with Terraform. It’s because each time we have unique container and providers, modules must be initialized by Terraform. Otherwise, we cannot track state between different stages/tasks and can’t create correctly our resources. That’s why, we use the command with some flags here. It enables us to initialize our working repository as backend structure.
In addition to this, we have to use same variables in every stage/task and Terraform variables.tf file must be overwritten by terraform.tfvars file which creates with the help of defined CI/CD variables.
For example, default OS_VOLUME_SIZE=1 in variables.tf file. We take a CI/CD variables from GitLab which named OPENSTACK_VOLUME_SIZE and it’s value is 300. Eventually, OS_VOLUME_SIZE will be overwritten and it’ll be used with its new value — 300- in Terraform operations.
# tests
Basically, this stage includes some tests steps. test_openstack_auth is a task that we check the API authentication on Openstack. It basically checks if the Openstack project exists via Openstack CLI but more test can be added here as well. Task runs automatically in the pipeline.
As you can see in the task, we override before_script since we don’t need to initialize Terraform for the task.
# validate
This stage validates Terraform code & configuration parameters within terraform_validate task. “terraform validate” command runs checks that verify whether a configuration is syntactically valid and internally consistent, regardless of any provided variables or existing state. It does not need to reach state or backend online and it’s safe it run automatically so that’s why we define this task to run automatically in the pipeline. It depends on success of test_openstack_auth.
# plan
terraform_plan task is creating a Terraform plan that plan & explains which resources will be created. Ideally, it validates lastly terraform.tfvars file before create the plan.
In next step, below command creates the plan by using respective terraform.tfvars file. $INFRA_CREATION_PLAN_NAME value comes from pipeline defined variable.
terraform plan -out $INFRA_CREATION_PLAN_NAME -var-file terraform.tfvars
We can also download the plan as artifact from Gitlab CI/CD. The stage depends on terraform_validate task. Since, I’ve planned to have more control, I choose execute the task manually (with one click to trigger). If you’d like you can configure as when: on_success.
# apply
Now, we’re ready to apply & create our resources at this stage. terraform_apply task is creating resources by using the plan we create in terraform_plan at the plan stage. It’s again checking terraform.tfvars in the task and runs manually to have more control. It depends on terraform_plan task. Creation of resources may take time depends on target platform response.
Once resources created, also Terraform state of the infrastructure will be created in your GitLab. We will get there and see clearly in the CI/CD execution step.
# destroy
It’s also possible to destroy resources via CI/CD pipeline. destroy stage terraform_destroy task will destroy resource with manual trigger. As the terraform destroy command runs with --auto-approve and lock=false flag, we need to be careful what we delete, which state we use as target state in execution. Because deletion command will be auto approved and our state will be unlocked.
The task also deletes the state where we store in GitLab automatically. TF_HTTP_PASSWORD corresponds GitLab access token and TF_HTTP_ADDRESS creates with multiple pre-defined values: $CI_API_V4_URL/projects/$CI_PROJECT_ID/terraform/state/$TF_STATE_NAME
curl — header “Private-Token:$TF_HTTP_PASSWORD” — request DELETE “$TF_HTTP_ADDRESS”
This task has dependency on terraform_apply as expected.
In addition to CI/CD pipeline definition, we need set our CI/CD variables in GitLab UI. These values are default values that pipeline uses each time you send commit & trigger pipeline. According to whishes or while working in different branch of repository, you overwrite these values accordingly.
You can define your key and values as shown below in GitLab UI. Go to Settings — CI/CD — Variables and set the variables. Full list are located below. You can also check variables with few details about in Github repository.
Step-5: Trigger & Execute CI/CD Pipeline 🚀
So, you have drinks under your hands and you know which drinks will be used how much, by using which mixing method etc. Your recipe is ready. Now it’s time to apply your recipe and mix them to make your delicious cocktail ready!
There are few ways to trigger CI/CD pipeline.
First option is commit to repository as working mentality of CI/CD. When you commit to repository, test and validate stages will trigger and execute automatically. (If you revert other stages to run automatically, others will work as well.) Then, you can trigger & execute other stages(plan,apply,destroy) that you run manually. It might be a good idea to keep some stages manual while developing your code.
Second option, you can run a pipeline directly from GitLab UI. In this option you need to go & click CI/CD — Pipelines — Run Pipeline. You can select different branch/tag and overwrite your default CI/CD variables here. Otherwise, your pipeline will run with default ones.
🚨 There is an important point to mention 🚨
While creating/destroying & managing your resources with Terraform & “GitLab Managed Terraform State” you need to make sure that TF_STATE_NAME, OPENSTACK_IMAGE_NAME, OPENSTACK_FLAVOR_NAME, OPENSTACK_VM_NAME variables has been changed, unless you want to work on same state. Otherwise, if your intention is not to work on same state, you may broke/change the current state accidentally. Locking of the current working state or working on different branch/environment also could be other good ideas to keep secure the state you want.
Now, let’s see a basic example of pipeline execution:
If we’d like to destroy resources, we can execute terraform_destroy job as well…
Well, that’s all that I want to cover in this story.
Thanks for your reading. See you in the next one.
Cheers!
