Terratest Azure Samples and Tips
Terratest is Amazing tool for testing infrastructure written by go. It is not only support terraform but also kubernetes, helm, Docker, Packer, ssh, scp, and major cloud providers. The first tutorial is cloud agnostic, so Azure users can use it, However, Kubernetes and helm samples seem to be created for AWS. I create a sample terratest repo for testing AKS for help Azure people getting started and enjoy terratest. I also introduce some tips for terratest on Azure.
Terratest Azure Samples
You can find the terratest on Azure samples here:
The repo introduces four samples working on AKS.
- AKS deployment by terraform with terratest
- Kubernetes Basic
- Kubernetes RBAC
- Kubernetes Ingress integration testing on RBAC cluster
Kubernets Basic and Kubernetes RBAC samples are copied and modified for AKS. Kubernetes Ingress integration testting on RBAC cluster is real world scenario. I automated an official tutorial Create an ingress controller in Azure Kubernetes Service (AKS) to see what happens on the real world scenario on Azure with terratest.
Getting Started terratest
Read the terratest readme. You can find a good getting started with cloud agnostic sample.
I recommend to watch this video to understand terratest and background ideas.
Next you might want to try some Azure sample, especially kubernetes and helm . However, these ones are written for AWS. You can refer my Azure sample repo for enjoy AKS with terratest. The official repo has an Azure sample. However, it is only for VM related testing.
Terratest on Azure tips
I noticed several tips for terratest on Azure. I’d like to share what I found so far. It includes non-Azure related topics. However, you might want to know what is going to happen if you try it. Especially someone who is not familiar with go language development.
You can find general Testing best practices in here:
I notice 8 tips for developing terratest on Azure. However, I’m a beginner of terratest. If you find better way and other recommendation, please leave comment.
- Use Azure CLI as an authentication method on development
- Write a custom helper method and contribute it
- Browse module directory
- Wait for External IP Address allocation instead of Service creation
- Configure Go Development Enviornment on VSCode
- Configure Debugging with VSCode for a single test case
- Use Windows as development environment not for WSL for Windows users
1. Use Azure CLI as an authentication method on development
We can use several authentication methods for Azure with Terraform.
However, under the heavy development of terratest, I recommend to use Azure CLI method. The configration is simple. Also, it is very helpful to switch between terraform and Azure CLI with the same authentication especially for debugging. Also, terraform can execute local command. In case, you are missing some resources on terraform Azure provider, You can use the local-exec with using latest Azure CLI. It is very flexible.
2. Write a custom helper method and contribute it
terratest support Azure, however, we have room to improvement of support of the Azure Provider. It looks only compute (VM) releated resource is supported.
For example, If you deploy AKS Cluster, how you can find it is already deployed and up and running? I wrote a custom test_helper.go to support GetManagedCluster() to fetch the current status of AKS Cluster. Then you can use it on your terratest.
You can use Azure SDK for Go to write the custom helper.
You can find my custom helper sample in here:
You can also refer cobalt project for some terratest extension on Azure.
There is an open issue for support more Azure resource. I’m contributing AKS resource. So if you want more resource, let’s contribute it! It is not difficult.
3. Browse module directory
You might think which methods can we use on terratest. Go to module directory. You can find a lot of resources and have a look at the go files. You will find soon. It is straight forward.
It also help to write your custom helper. For example, I wrote a helper method for deploy helm with external repo. In this case, I refer the helm/install.go for seeing original implementation.
4. Wait for External IP Address allocation instead of Service creation
The biggest reason why the kubernest/helm sample on terratest doesn’t work on Azure is, Azure use LoadBalancer for connect cluster from outside. Instead, AWS use NodePort. That is why, a lot of sample written against NodePort. In my sample, I change it to LoadBalancer. In the official sample, if you deploy resource(like deployment) with Service, the test will be the service is ready or not. Terratest provide helper method for kubernetes service for wait until the service is ready, however, it doesn’t useful for Load Balancer. In case of Load Balancer on Azure, AKS, will create a Load balancer and attach IPAddress. To create an IPAddress takes time. So you need to do polling until it is finished. I wrote an helper method for it.
5. Configure Go Development Enviornment on VSCode
As an OPS, you might not familiar with go lang. The terratest is not complicated and non-go user might use a lot of provider methods without deep dive into go world. However, I recommend to configure your VSCode (or other IDE for go) to make it fetch definition. In case of VSCode, all you need to do is following.
- Open VSCode on test directory that include terratests
- Ctr+shift+P then Open this window. Click Go: Install/Update Tools
Then check all tools and click OK.
- Re-Open VSCode on test directory.
Now you can use Intellisence, See the definition, and Jump to the original source. It is very convenient.
For more details:
6. Configure Debugging with VSCode for a single test case
If you use VSCode, I recommend to configure Debugging. The best practice recommend us to use defer keyword to remove the resources that we created on the test. This best practice is very important for reducing your work load of writing test. If you don’t write it, you need to manually remove these resources, and sometimes forget to remove it and cause weird behavior.
Then, how we can debug it? If you run the test, the test remove all the rest at the end of the execution. For example, If you want to see the log of Kubernetes Ingress controller’s pod when you get 503 in your test, you will type kubectl command, however, after a couple of minutes, the resource is removed by the end of the test. In this case, you can use debug feature of VSCode and set the break point. You can refer the variables, it helps us to create custom script. Also, it help to stop the execution of the test. The debugger stops at the breakpoint, then you can investigate the root cause of the issue of your test.
For configuring debugging on VS Code, I recommend to configure the debugging for execute only one test case like this.
The reason is, if you don’t specify a specific test case, it execute all tests on the same directory. If you have multiple scenario on the test directory, disaster happens.
In my case, I wrote an ingress integration test and getting very close to finish. At that time, I notice that debugger helps me to investigate. The test doesn’t require create/remove AKS cluster. Just working on an existing cluster.
I configure debugging without specifying a single test scenario. Then the debugger start execute all test scenario in Parallel. and It start de-provisioning current RBAC cluster and start to create non-RBAC cluster. I eventually, clean up the resource, and create a RBAC cluster from scratch. lol
For more details about debugging go with VSCode, please refer :
7. Use Windows as development environment not for WSL for Windows users
If you are windows user, I recommend to use Windows as development environment of Go. Instead of WSL for now. (at least WSL1)
I’m a guy who loves Linux culture. So I started with WSL. Since VSCode has a fantastic feature called Remote Connect. It enable to edit remote file as if it is in a local machine. This feature is used for WSL.
We can feel free to edit using it and read code, however, VSCode go debugger doesn’t work on Linux. So if you want to write go code, use Windows. (Mac users are no problem. )
I’m looking forward to seeing WSL2. :)
I hope this post helps.