This site uses cookies and by using the site you are consenting to this. We utilize cookies to optimize our brand’s web presence and website experience. To learn more about cookies, click here to read our privacy statement.

Automate Data Protection Using Concourse CI & HashiCorp Vault

Author: Ben Ngo Posted In: DevOps

Your company’s secrets are your company’s, and you are tasked with protecting them. HashiCorp Vault is built just for such a purpose, providing a unified interface to any secret along with tight access control and recording a detailed audit log. But how to manage all the tasks that come with all the secrets (and tokens and passwords and certificates and API keys and…).

Let’s walk through one example process that allows you to automate your HashiCorp Vault platform by integrating Concourse CI.


Concourse CI is a simple CI/CD tool that leverages containers to manage tasks for clean execution of pipelines. Vault is an API driven secrets management platform that allows a secure centralized source of housing credentials. We’ll walk through the steps to run sample tasks. Please note, these are for local testing purposes only, not recommended for any production usage.

A black and white image showcasing a plane and a building, emphasizing the significance of data protection.

What we’ll cover

  • Generate keys and certs for Concourse CI and Vault to use as auth methods
  • Using docker-compose to deploy a local test environment
  • Initialize Vault and push secrets via cli
  • Run sample task to test access to Vault secrets in Concourse CI

Before you begin

Exercise #1: Generate keys and certificates for Concourse and Vault

Your first steps are to generate the keys needed for Concourse web and worker nodes. The following script uses the Concourse docker image to generate the keys and place them in keys/web and keys/worker directories.

$ ./generate_keys.sh

Contents of script

#!/usr/bin/env bash

set -e -u

cd $(dirname $0)

docker run --rm -v $PWD/keys/web:/keys concourse/concourse \
generate-key -t rsa -f /keys/session_signing_key

docker run --rm -v $PWD/keys/web:/keys concourse/concourse \
generate-key -t ssh -f /keys/tsa_host_key

docker run --rm -v $PWD/keys/worker:/keys concourse/concourse \
generate-key -t ssh -f /keys/worker_key


Copy pub keys from web to workers and vice-versa.

sudo cp -f ./keys/worker/worker_key.pub ./keys/web/authorized_worker_keys
sudo cp -f ./keys/web/tsa_host_key.pub ./keys/worker

Next, generate a tls-cert to be used for authentication between the Concourse web node and Vault. The following script will init a local CA authority and generate the certs needed.

$ ./generate_certs.sh
Contents of script


# Generate keys for concourse web and worker nodes

certstrap init --cn vault-ca --passphrase ""
certstrap request-cert --domain vault --ip --passphrase ""
certstrap sign vault --CA vault-ca --passphrase ""
certstrap request-cert --cn concourse --passphrase ""
certstrap sign concourse --CA vault-ca --passphrase ""
mv out vault-certs

Exercise #2: Setting up docker-compose file for Concourse CI and Vault

Use docker-compose to set-up the environment. Included in this repo is a `docker-compose.yml` file that uses the certs and keys generated to init Concourse web/workers and Vault. To run the docker-compose file, run the following;

$ docker-compose up -d

If successful, you will see the following output when running `docker container ls`

$ docker container ls
288054bf2478 concourse/concourse "dumb-init /usr/loca…" About a minute ago Up About a minute concourse_ci_vault_worker_1
9083e13ed496 concourse/concourse "dumb-init /usr/loca…" About a minute ago Up About a minute>8080/tcp concourse_ci_vault_web_1
1efd97078b52 vault "docker-entrypoint.s…" About a minute ago Up About a minute>8200/tcp concourse_ci_vault_vault_1
9b963b7e9917 postgres "docker-entrypoint.s…" About a minute ago Up About a minute 5432/tcp concourse_ci_vault_db_1

Open your browser and check `http://localhost:8080` for Concourse and `https://localhost:8200` for Vault.

Exercise #3: Using Vault cli to add credentials to Vault

Now that you have Concourse and Vault up, initialize Vault for Concourse to use.

Set the local environment variable and run the init command for Vault.

$ export VAULT_CACERT=$PWD/vault-certs/vault-ca.crt
$ vault operator init
Unseal Key 1: key1
Unseal Key 2: key2
Unseal Key 3: key3
Unseal Key 4: key4
Unseal Key 5: key5

Initial Root Token: roottoken

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Use at least 3 keys to unseal Vault and login with the root token.

$ vault operator unseal  key1
$ vault operator unseal key2
$ vault operator unseal key3
$ vault login # paste root token

Enable the cert backend to allow Concourse to authenticate to Vault. You can use the following script to enable the cert backend and secrets engine.

$ ./generate_concourse_policy.sh
Contents of script


vault policy write concourse vault-config/concourse-policy.hcl

vault auth enable cert

vault write auth/cert/certs/concourse \
policies=concourse \
certificate=@vault-certs/vault-ca.crt \

vault secrets enable -path=concourse/ kv


Add a sample secret to Vault to test the path is writable.

$ vault kv put concourse/main/demo_value TEST=HELLOWORLD

Exercise #4: Test secret using a Concourse pipeline

To test, Concourse is able to access the secret stored in Vault. Open the browser to http://localhost:8080, login with username `test` and password `test` and proceed to download the fly cli. Make sure to set it as an executable and add it to your local usr/bin.

Login to Concourse using the following commands:

$ sudo fly --target tutorial login --concourse-url -u test -p test
logging in to team 'main'

target saved
$ sudo fly --target tutorial sync
version 5.1.0 already matches; skipping

Use the sample Concourse task to test if you can access the secret and use it in a task.

$ sudo fly -t tutorial execute -c .ci/sample.yml
uploading concourse_ci_vault done
executing build 1 at
running sh -exc echo $SAMPLE

You can also browse to to see your build in the Concourse web console.

And that’s a wrap! These steps should provide you with a better understanding of both Concourse CI and Vault and their uses for managing sensitive values.