Ado Keyvault Linked Var Groups
Azure DevOps keyvault linked variable groups are not easy to automate but it can be done.
Introduction
As of late I’ve been working quite a bit with Azure Devops. It is a powerful platform with a dearth of possibilities but I quickly hit the ceiling of the platform’s capabilities. It suffers from a common automation platform deficiency, the ability to automate the platform itself (automating the automation platform as it were). This is not uncommon and I’ve seen the same issues with Jenkins (props to Jenkins Job Builder for a very clever solution to some of these restrictions).
There has been some progress with a preview az cli extension aptly named ‘devops’. If you are a little adept with Bash you can use the az cli to create and manage variable groups but you cannot create keyvault linked variable groups (or if you can, I don’t see how). Luckily, like most modern PaaS solutions, Azure DevOps is capable of being automated further than you may think if you are willing ot dive into json templates and their REST API. Oh, you will still need that extension:
Set the Stage
Automating this thing has more moving parts that I’m comfortable with but usually that’s how ‘hacks’ go right? So to automate this with some security in place you need a ton of variables and secrets including….
Firstly we need an authorized service endpoint already created within Azure Devops. This should be tied to an SPN which can access the keyvault secrets via access policies. I personally use terraform to generate these and then assign them appropriate policies to only the resources it requires. I use another automation script/hack to create these service connections in ADO.
I like to use per stage/team SPNs for service connections. This usually takes the form of STAGE_TEAM and ties back to an AKS cluster, so I can use the connection for deployment pipelines as well as grant the cluster rights to ACR, key vault policies, and so on right from terraform with the native azurerm provider.
You will also need A key vault populated with the following secrets that will need to have been granted access to ADO to create libraries (variable groups). In the script I provide I assume it is in the same key vault being linked to.
- ADOUSER (ADO Username)
- ADOPAT (Personal Access Token)
We also need some environment variables for less secret things. You will see later that I keep these in an .envrc file on a per team basis. Here are the basic ones which are required for this exercize.
Finally, we also need
- Azure CLI
- Azure CLI devops extension
- Permission to create library resources in Azure DevOps
- Fortitude, lots of that.
Finally you will need a json template file for the variable group which I’ll describe next.
The JSON Template
In order to create a keyvault linked variable group you will need to setup a template json file and submit it to the ADO API via an HTTP POST. Easy peasy. I’ve done the reverse engineering bits for you on this one. Here is a json file for a variable group called ‘SuperSecret’ that links to a keyvault called SuperSecretVault and syncs the ‘AIRFLOWFERNETKEY’ and ‘KUBESECRET’ vault secrets. One could easily construct this automatically based on an existing key vault and some python I’d think.
Shall we tokenize this? Okay,
With this at hand we can then work our magic.
The Script
I put together a script which tokenizes our json file and submits it to Azure DevOps via curl. I used old school envsubst to keep it somewhat more portable.
Conclusion
Well that kinda was a stinker ‘eh? The moment this is published I’m willing to bet the az cli extension gets updated with some new subcommand to create these things. Tis the way of our industry right?
