Secret Rotation with HashiCorp Vault on AKS

Overview

HashiCorp Vault and ConsulTemplate has a feature what dynamic secret rotation with Kubernetes integration

Vault, Vault Agent, and Consul Template
  1. Vault Agent accesses to the Vault Server with authenticate with Kubernetes authentication using Service Account and CulsterRoleBinding
  2. Write vault volume on the volume on a pod.
  3. Consul Template uses the token and fetch secrets from vault Server
  4. Write secret on the Volume

Install Helm

Download the helm binary and put on your path.

helm init
helm list
Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
$ kubectl apply -f tiller-service-account.yml
$ helm init
$ kubectl --namespace kube-system patch deploy tiller-deploy \
-p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'

Install Consul (Optional)

You can choose the storage of the Vault. The best option might be consul. Consul is a service mesh solution providing a full featured control plane with service discovery, configuration, and segmentation functionality. However, if you just want to test Vault, you don’t need this step.

$ git clone https://github.com/hashicorp/consul-helm.git
$ helm install -n consul ./consul-helm

Vault Server

Install

Using Helm Chart, deploy Vault Server

$ git clone https://github.com/hashicorp/vault-helm.git
$ cd vault-helm
$ helm install --name=vault .
$ kubectl get pods 
NAME READY STATUS RESTARTS AGE
consul-2ftq7 1/1 Running 0 141m
consul-ht6xb 1/1 Running 0 141m
consul-server-0 1/1 Running 0 141m
consul-server-1 1/1 Running 0 141m
consul-server-2 1/1 Running 0 141m
consul-sf8r2 1/1 Running 0 141m
vault-5cb479cf66-6gx7t 1/1 Running 0 136m
$ kubectl port-forward pod/vault-5cb479cf66-6gx7t 8200:8200
$ export VAULT_ADDR=http://localhost:8200

Initialize

See the status of the Vault Server

$ vault status
$ vault operator initUnseal Key 1: fXrDKdbLwL+lmrKrWCCKUt+GQu8maJ1Efv76y0roETyQ
Unseal Key 2: 8GnTaglKi5LsWLGxzU8TAEvnoVvp6wHK5DukgGGXgSfj
Unseal Key 3: YgD+Bl238aGmRWbqST61C2/JxUsjvY4Uj7w+MDTQgSBo
Unseal Key 4: UHt2X3zSaG+p7sizovViMGnD/k6Bi7FNYx8y+EwVUNLa
Unseal Key 5: TooR5MFJcGWc6ZnVYM8+jHNRpc6dMZkOXAvWWSFA+sVB
Initial Root Token: s.uIx8F9L7zl3nLqOqCDKo9L0MVault initialized with 5 key shares and a key threshold of 3. Please securelydistribute the key shares printed above. When the Vault is re-sealed,restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated master key. Without at least 3 key to reconstruct the master key, Vault will remain permanently sealed!It is possible to generate new unseal keys, provided you have a quorum ofexisting unseal keys shares. See "vault operator rekey" for more information.
-key-shares=<int>
Number of key shares to split the generated master key into. This is the number of "unseal keys" to generate. This is aliased as "-n". The default is 5.
-key-threshold=<int>
Number of key shares required to reconstruct the master key. This must be less than or equal to -key-shares. This is aliased as "-t". The default is 3.

Unseal

It requires unseal operation before starting use the vault.

$ vault operator unseal fXrDKdbLwL+lmrKrWCCKUt+GQu8maJ1Efv76y0roETyQ
$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.2.0
Cluster Name vault-cluster-a1b7f2b1
Cluster ID 4f50bf0c-1582-b03d-249d-e795ea18a583
HA Enabled true
HA Cluster https://10.244.2.33:8201
HA Mode active

Login

Login to vault server using Intial Root Token that you see on the initialization process.

$ vault login s.uIx8F9L7zl3nLqOqCDKo9L0M

Run sample

Clone the sample

$ git clone git@github.com:hashicorp/vault-guides.git
$ cd vault-guides/identity/vault-agent-k8s-demo

Create RoleBindings for Dashboard

Your AKS dashboard might look like this. Create RoleBinding for solve this.

Dashboard
$ kubectl create clusterrolebinding kubernetes-dashboard -n kube-system --clusterrole=cluster-admin -- serviceaccount=kube-system:kubernetes-dashboard  $ az aks browse --resource-group $(terraform output resource_group_name) --name $(terraform output kubernetes_cluster_name)
DashBoard (works)

Change the configuration files

Get the Kubernetes master URL. You can find the master url.

$ kubectl cluster-info
Kubernetes master is running at https://tsushikeya-keyvaultaks-abcdefg01235.hcp.westus.azmk8s.io:443
Heapster is running at
./setup-k8s-auth.sh

Deploy nginx with Vault Agent and Consul Template

Change the example-k8s.spec.yml about the VAULT_ADDR parts. then deploy.

$ kubectl apply -f example-k8s-spec.yml

Test

In your terminal, connect to the nginx pod.

$ kubectl port-forward vault-agent-example 8090:80
The first try
$ vault kv put secret/myapp/config username='appuser' password='suP4rsec(et!' ttl='30s'
Works!

Conclusion

Vault with Consul Template is very cool solution for secret rotation. Also, It is useful when you want to deploy vault on on premise limited cluster.

Personal Pitfall

The helm chart and tutorial is great and cristal clear. However, I fall into a pitfall. At the first time, I deploy Vault server with my self-made YAML file, the next day, Vault Helm Chart has been released. I deploy it using helm chart.
Once I configured, it emit 403 or other very weird behaviors. I redeploy the Vault Server, clear the Consul data and so on however, nothing changed.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store