Docker Compose - Hashicorp's Vault and Consul Part C (Consul Backend)

bogotobogo.com site search:

Note

Continued from Docker Compose - Hashicorp's Vault and Consul Part B (EaaS, dynamic secrets, leases, and revocation).

So far, we've been using the Filesystem backend. This will not scale beyond a single server, so it does not take advantage of Vault's high availability (HA). Fortunately, there are a number of other Storage backends, like the Consul backend, designed for distributed systems.

In this post, we'll set up Consul. It is a distributed service mesh to connect, secure, and configure services across any runtime platform and public or private cloud.

HashiCorp's Consul has multiple components such as discovering and configuring services in our infrastructure. It provides several key features:

Service Discovery
Health Checking
KV Store
Multi Datacenter

Among the features, in this post, we'll use the Consul as KV Storage backend.

Updated docker-compose.yml

To set up Consul, we may want to update the docker-compose.yml file so that we can add the consul service:

version: '3.6'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK
    depends_on:
      - consul

  consul:
    build:
      context: ./consul
      dockerfile: Dockerfile
    ports:
      - 8500:8500
    command: agent -server -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect 1 -config-file=/consul/config/config.json
    volumes:
      - ./consul/config/consul-config.json:/consul/config/config.json
      - ./consul/data:/consul/data

consul/Dockerfile

Let's create a new directory within the project root called consul, and then add a new Dockerfile to that newly created directory:

# base image
FROM alpine:3.7

# set consul version
ENV CONSUL_VERSION 1.2.1

# create a new directory
RUN mkdir /consul

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up consul
RUN wget --quiet --output-document=/tmp/consul.zip https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip && \
    unzip /tmp/consul.zip -d /consul && \
    rm -f /tmp/consul.zip && \
    chmod +x /consul/consul

# update PATH
ENV PATH="PATH=$PATH:$PWD/consul"

# add the config file
COPY ./config/consul-config.json /consul/config/config.json

# expose ports
EXPOSE 8300 8400 8500 8600

# run consul
ENTRYPOINT ["consul"]

consul/consul-config.json:

{
  "datacenter": "localhost",
  "data_dir": "/consul/data",
  "log_level": "DEBUG",
  "server": true,
  "ui": true,
  "ports": {
    "dns": 53
  }
}

We can find more options for the configuration.

The consul directory looks like this:

../consul
├── Dockerfile
├── config
│   └── consul-config.json
└── data

Updating vault config

Exit out of the bash session and bring the container down we've been running:

$ docker-compose down
Stopping vault-consul-docker_vault_1 ... done
Removing vault-consul-docker_vault_1 ... done
Removing network vault-consul-docker_default

Then update the Vault config file (vault/vault-config):

{
  "backend": {
    "consul": {
      "address": "consul:8500",
      "path": "vault/"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

So, now we're using the Consul backend instead of the Filesystem.

We used the name of the service, consul, as part of the address. The path key defines the path in Consul's key/value store where the Vault data will be stored.

We need to clear out all files and folders within the vault/data directory to remove the Filesystem backend.

Spinning up new containers

Now we have a new Dockerfile and config files for Consul and updated config for Vault:

Let's build the new images and spin up the containers:

$ docker-compose up -d --build
...
Creating vault-consul-docker_consul_1 ... done
Creating vault-consul-docker_vault_1  ... done

$ docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED              STATUS              PORTS                                                  NAMES
0f0cc416ae2d   vault-consul-docker_vault    "vault server -confi…"   About a minute ago   Up About a minute   0.0.0.0:8200->8200/tcp                                 vault-consul-docker_vault_1
006a2fcaf6bf   vault-consul-docker_consul   "consul agent -serve…"   About a minute ago   Up About a minute   8300/tcp, 8400/tcp, 8600/tcp, 0.0.0.0:8500->8500/tcp   vault-consul-docker_consul_1

Check http://localhost:8500/ui:

Testing via CLI

As we've done before, let's create a new bash session in the Vault container:

$ docker-compose exec vault bash
bash-4.4#

The vault operator init command initializes a Vault server. Initialization is the process by which Vault's storage backend is prepared to receive data.

During initialization, Vault generates an in-memory master key and applies Shamir's secret sharing algorithm to disassemble that master key into a configuration number of key shares such that a configurable subset of those key shares must come together to regenerate the master key. These keys are often called unseal keys in Vault's documentation.

Note that this command cannot be run against already-initialized Vault cluster.

bash-4.4# vault operator init
Unseal Key 1: RDmgJrpou9uEoGhCHuul+yaUVjeVq0qqS0bjZNvSJjBd
Unseal Key 2: Typ0pcGCF8FiqReFztcoJHdpPavgQtFiyxDuvdkBw+W7
Unseal Key 3: ZH7dr8kHKlZfYdV49404UfMzSvIqAfxtWoXM8ROxMzkp
Unseal Key 4: MhaAKFPbZ20h5y1N6l/g453Izi6XdbcOVEWHK2kvMIMc
Unseal Key 5: RKm7E9q2UVyD7u7JliofmTwknvg/kRGwobUjmi86nGXd

Initial Root Token: 16336cfc-a13c-c288-ee7b-fad06f15475e

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

The vault operator unseal allows the user to provide a portion of the master key to unseal a Vault server. Vault starts in a sealed state.

bash-4.4# vault operator unseal RKm7E9q2UVyD7u7JliofmTwknvg/kRGwobUjmi86nGXd
Key                    Value
---                    -----
Seal Type              shamir
Sealed                 false
Total Shares           5
Threshold              3
Version                0.10.3
Cluster Name           vault-cluster-49bce7b8
Cluster ID             fb2f7207-6e41-bd12-abe7-ca18b256f001
HA Enabled             true
HA Cluster             n/a
HA Mode                standby
Active Node Address    

bash-4.4# vault operator unseal MhaAKFPbZ20h5y1N6l/g453Izi6XdbcOVEWHK2kvMIMc
Key             Value
---             -----
Seal Type       shamir
Sealed          false
Total Shares    5
Threshold       3
Version         0.10.3
Cluster Name    vault-cluster-49bce7b8
Cluster ID      fb2f7207-6e41-bd12-abe7-ca18b256f001
HA Enabled      true
HA Cluster      https://172.22.0.2:8201
HA Mode         active

bash-4.4# vault operator unseal ZH7dr8kHKlZfYdV49404UfMzSvIqAfxtWoXM8ROxMzkp
Key             Value
---             -----
Seal Type       shamir
Sealed          false
Total Shares    5
Threshold       3
Version         0.10.3
Cluster Name    vault-cluster-49bce7b8
Cluster ID      fb2f7207-6e41-bd12-abe7-ca18b256f001
HA Enabled      true
HA Cluster      https://172.22.0.2:8201
HA Mode         active

bash-4.4# vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                16336cfc-a13c-c288-ee7b-fad06f15475e
token_accessor       9edd90cb-0eb5-0ef5-2296-1f32a014f870
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

The vault login command authenticates users or machines to Vault using the provided arguments. A successful authentication results in a Vault token - conceptually similar to a session token on a website. By default, this token is cached on the local machine for future requests.

Let's add a new static secret:

bash-4.4# vault kv put secret/foo bar=precious
Success! Data written to: secret/foo

To read it back:

bash-4.4# vault kv get secret/foo
=== Data ===
Key    Value
---    -----
bar    precious

One of our Vault server nodes (though we have only one) successfully grabbed a lock within the data store and become an active node:

All other nodes (suppose we have three) become standby nodes.

Testing via UI

To test UI, exit out of the bash session and bring the container down we've been running.

$ docker-compose down
Stopping vault-consul-docker_vault_1  ... done
Stopping vault-consul-docker_consul_1 ... done
Removing vault-consul-docker_vault_1  ... done
Removing vault-consul-docker_consul_1 ... done
Removing network vault-consul-docker_default

We also need to undo the actions we took in the previous section with CLI: remove the folders /files in consul/data.

Let's build the new images and spin up the containers again:

$ docker-compose up -d --build
...
Creating vault-consul-docker_consul_1 ... done
Creating vault-consul-docker_vault_1  ... done

Navigate to http://localhost:8500/ui:

Click "Initialize" button:

Copy the key and paste it into our Vault in the 3rd tab:

Click "Unseal" button:

Copy the token and paste it, and click "Sign In" button:

We can see our Vault node successfully grabbed a lock within the data store and become an active node:

Let's create a secret:

Adding another Consul server

If we want to add another Consul server into the mix, we need to add a new service to docker-compose.yml:

version: '3.6'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK
    depends_on:
      - consul

  consul:
    build:
      context: ./consul
      dockerfile: Dockerfile
    ports:
      - 8500:8500
    command: agent -server -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect 1 -config-file=/consul/config/config.json
    volumes:
      - ./consul/config/consul-config.json:/consul/config/config.json
      - ./consul/data:/consul/data

  consul-worker:
    build:
      context: ./consul
      dockerfile: Dockerfile
    command: agent -server -join consul -config-file=/consul/config/config.json
    volumes:
      - ./consul/config/consul-config.json:/consul/config/config.json
    depends_on:
      - consul

Here, we used the join command to connect this agent to an existing cluster. Notice how we simply had to reference the service name - consul

With the docker-compose file, we need to take the same steps as before : tear down the containers and spin-up again:

Bring down the containers

$ docker-compose down
Stopping vault-consul-docker_vault_1  ... done
Stopping vault-consul-docker_consul_1 ... done
Removing vault-consul-docker_vault_1  ... done
Removing vault-consul-docker_consul_1 ... done
Removing network vault-consul-docker_default

Clear out the data directory in "consul/data"

Spin the containers back up and test

$ docker-compose up -d --build
Creating vault-consul-docker_consul_1 ... done
Creating vault-consul-docker_consul-worker_1 ... done
Creating vault-consul-docker_vault_1         ... done

$ docker ps
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS              PORTS                                                  NAMES
cc95cac5161d        vault-consul-docker_vault           "vault server -confi…"   18 minutes ago      Up 18 minutes       0.0.0.0:8200->8200/tcp                                 vault-consul-docker_vault_1
431a2b8735b1        vault-consul-docker_consul-worker   "consul agent -serve…"   18 minutes ago      Up 18 minutes       8300/tcp, 8400/tcp, 8500/tcp, 8600/tcp                 vault-consul-docker_consul-worker_1
93cc119b4f0d        vault-consul-docker_consul          "consul agent -serve…"   18 minutes ago      Up 18 minutes       8300/tcp, 8400/tcp, 8600/tcp, 0.0.0.0:8500->8500/tcp   vault-consul-docker_consul_1