Secret Rotation with HashiCorp Vault on AKS

HashiCorp Vault provide us a fantastic Secret Rotation architecture with Vault Agent and Consul Template. You can try it with this tutorial.

Vault Agent with Kubernetes

In this tutorial, there is no mention how to install VaultServer on AKS. I wrote this blog not to forget what I did.

Overview

HashiCorp Vault and ConsulTemplate has a feature what dynamic secret rotation with Kubernetes integration

Vault, Vault Agent, and Consul Template

1. Vault Agent accesses to the Vault Server with authenticate with Kubernetes authentication using Service Account and CulsterRoleBinding
2. Write vault volume on the volume on a pod.
3. Consul Template uses the token and fetch secrets from vault Server
4. Write secret on the Volume

Cool thing is, Consul Template runs as daemon. Once the secret expired, it ask to the Vault and update the Secret dynamically.

I just follow the tutorial however, some tips might be helpful for you when You combine it with Vault helm chart. I assume AKS is deployed with RBAC.

Install Helm

Download the helm binary and put on your path.

helm init
helm list

For the latest AKS with RBAC, you might encounter this issue.

Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "configmaps" in API group "" in the namespace "kube-system"

This is lack of access right for tiller.

Create a ClusterRole and Role binding yml file.

and apply it. Then hem init works.

$ kubectl apply -f tiller-service-account.yml
$ helm init
$ kubectl --namespace kube-system patch deploy tiller-deploy \
-p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

for more details

User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "default" ·…

Install Consul (Optional)

You can choose the storage of the Vault. The best option might be consul. Consul is a service mesh solution providing a full featured control plane with service discovery, configuration, and segmentation functionality. However, if you just want to test Vault, you don’t need this step.

$ git clone https://github.com/hashicorp/consul-helm.git
$ helm install -n consul ./consul-helm

https://www.consul.io/docs/platform/k8s/run.html

Vault Server

Install

Using Helm Chart, deploy Vault Server

$ git clone https://github.com/hashicorp/vault-helm.git
$ cd vault-helm
$ helm install --name=vault .

I haven’t tried Vault HelmChart with Consul Helm Chart. (I did it with a self-made Vault yaml file with Consul Helm Chart), However, you can do configure Vault vaules.yaml like this. Set the storage.address to the consul service name. And turn off the standalone.enabled and turn on ha.enabled

Also, ui.enabled true might be helpful.

Announcing the HashiCorp Vault Helm Chart

Helm - Kubernetes - Vault by HashiCorp

I recommend to down load vault binary and set PATH to the binary on your local machine. Then, you can find your vault and port forward it.

$ kubectl get pods 
NAME                                READY     STATUS    RESTARTS   AGE
consul-2ftq7                        1/1       Running   0          141m
consul-ht6xb                        1/1       Running   0          141m
consul-server-0                     1/1       Running   0          141m
consul-server-1                     1/1       Running   0          141m
consul-server-2                     1/1       Running   0          141m
consul-sf8r2                        1/1       Running   0          141m
vault-5cb479cf66-6gx7t              1/1       Running   0          136m
$ kubectl port-forward pod/vault-5cb479cf66-6gx7t 8200:8200

Then, configure VAULT_ADDR

$ export VAULT_ADDR=http://localhost:8200

Now you can access Vault Server from your client.

Initialize

See the status of the Vault Server

$ vault status

Initialize it.

$ vault operator init
Unseal Key 1: fXrDKdbLwL+lmrKrWCCKUt+GQu8maJ1Efv76y0roETyQ
Unseal Key 2: 8GnTaglKi5LsWLGxzU8TAEvnoVvp6wHK5DukgGGXgSfj
Unseal Key 3: YgD+Bl238aGmRWbqST61C2/JxUsjvY4Uj7w+MDTQgSBo
Unseal Key 4: UHt2X3zSaG+p7sizovViMGnD/k6Bi7FNYx8y+EwVUNLa
Unseal Key 5: TooR5MFJcGWc6ZnVYM8+jHNRpc6dMZkOXAvWWSFA+sVB
Initial Root Token: s.uIx8F9L7zl3nLqOqCDKo9L0M
Vault initialized with 5 key shares and a key threshold of 3. Please securelydistribute the key shares printed above. When the Vault is re-sealed,restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 key to reconstruct the master key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

If you want to minimize the Unseal key, you can use -nand -t option

-key-shares=<int>
      Number of key shares to split the generated master key into. This is the number of "unseal keys" to generate. This is aliased as "-n". The default is 5.
-key-threshold=<int>
      Number of key shares required to reconstruct the master key. This must be less than or equal to -key-shares. This is aliased as "-t". The default is 3.

Unseal

It requires unseal operation before starting use the vault.

$ vault operator unseal fXrDKdbLwL+lmrKrWCCKUt+GQu8maJ1Efv76y0roETyQ

Repeat three keys until the unseal status becomes false.

$ vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.2.0
Cluster Name    vault-cluster-a1b7f2b1
Cluster ID      4f50bf0c-1582-b03d-249d-e795ea18a583
HA Enabled      true
HA Cluster      https://10.244.2.33:8201
HA Mode         active

Login

Login to vault server using Intial Root Token that you see on the initialization process.

$ vault login s.uIx8F9L7zl3nLqOqCDKo9L0M

Now ready to use Vault!

Run sample

Clone the sample

$ git clone git@github.com:hashicorp/vault-guides.git
$ cd vault-guides/identity/vault-agent-k8s-demo

Create RoleBindings for Dashboard

Your AKS dashboard might look like this. Create RoleBinding for solve this.

Dashboard

$ kubectl create clusterrolebinding kubernetes-dashboard -n kube-system --clusterrole=cluster-admin -- serviceaccount=kube-system:kubernetes-dashboard  
$ az aks browse --resource-group $(terraform output resource_group_name) --name $(terraform output kubernetes_cluster_name)

Now the problem has been solved.

DashBoard (works)

Change the configuration files

Get the Kubernetes master URL. You can find the master url.

$ kubectl cluster-info
Kubernetes master is running at https://tsushikeya-keyvaultaks-abcdefg01235.hcp.westus.azmk8s.io:443
Heapster is running at

Change the setup-k8s-auth.sh K8S_HOST part and vault write auth/kubernetesconfig ‘s kubernets_host

Run the script. It create Service Account for Vault and configure Kubernetes Authenticaion and create a new user and secret has username=’appuser’ password=’suP3rsec(et!’, and ttl=’30s’. For more details please read the tutorial.

./setup-k8s-auth.sh

Deploy nginx with Vault Agent and Consul Template

Change the example-k8s.spec.yml about the VAULT_ADDR parts. then deploy.

It deploys Vault Agent and Consul Temnplate.

$ kubectl apply -f example-k8s-spec.yml

Test

In your terminal, connect to the nginx pod.

$ kubectl port-forward vault-agent-example 8090:80

Open your browser

The first try

Then update the secret then wait 30 sec.

$ vault kv put secret/myapp/config username='appuser' password='suP4rsec(et!' ttl='30s'

Works!

Works!

Conclusion

Vault with Consul Template is very cool solution for secret rotation. Also, It is useful when you want to deploy vault on on premise limited cluster.

Personal Pitfall

The helm chart and tutorial is great and cristal clear. However, I fall into a pitfall. At the first time, I deploy Vault server with my self-made YAML file, the next day, Vault Helm Chart has been released. I deploy it using helm chart.
Once I configured, it emit 403 or other very weird behaviors. I redeploy the Vault Server, clear the Consul data and so on however, nothing changed.

The reason was, My yaml file and Helm chart pod uses the same pod name. Which means Service selector cannot identify which is which, at the first time, it route to the YAML’s one, once I remove/recreate the Vault Server with YAML, it return 403 even if I specify the token. It was ridiculous mistake, however, the debugging was hard. I finally figure out that the sever I talked is different by vault statusI’d like to share for my learning. Double check which server are you talking! lol