News HashiCorp to join IBM to accelerate multi-cloud automation. Read the blog →
Boundary
Credential management

HCP credential injection with private Vault

  • 20min
  • |
  • HCP
  • Boundary
  • Vault

Boundary Community Edition supports broking credentials like SSH keys to users. HCP Boundary and Boundary Enterprise introduced credential injection: the ability to inject SSH credentials into sessions, including through Vault. Boundary 0.11 adds support for private Vault access using a self-managed worker.

In this tutorial, you will configure SSH credential injection using private Vault for an SSH server configured using Docker. You can also bring your own target to use for this tutorial.

Tutorial overview

  1. Prerequisites
  2. Background
  3. Set up an SSH target
  4. Set up Vault
  5. Set up Boundary
  6. Inject credentials into sessions

Prerequisites

  • This tutorial assumes the learner has completed the HCP Boundary Getting Started tutorials. The learner should have a working Boundary cluster and org running on HCP.

  • Docker is installed (Note: Learners may also bring their own target, instead of using Docker)

  • A Boundary binary greater than 0.12.0 in your PATH

  • A Vault binary greater than 1.7.0 in your PATH

  • Installing the Boundary Desktop App provides an optional workflow at the end of this tutorial. The 1.2.0 version or greater is required for Vault support, and installing the latest version is recommended.

This tutorial assumes basic knowledge of using Vault, including running a development server and managing policies, roles, and tokens. If you are new to using Vault, complete the Getting Started with Vault quick start tutorials before integrating Vault with Boundary.

A self-managed HCP worker is required for integrating private Vault clusters with HCP Boundary. To learn more about setting up self-managed workers, refer to the Self-Managed Worker Registration with HCP Boundary tutorial.

This tutorial also extends the workflow for credential brokering via Vault. If new to this process, we recommend completing the Vault Credential Brokering Quickstart tutorial before moving forward.

Credential injection and private Vault background

Previously Boundary supported credential brokering, where Boundary controllers check out credentials from Vault and returns them back to users and clients. In this workflow, Boundary acts as a broker of the credential.

Brokering lacks the ability to hide credentials from clients. With HCP Boundary's credential injection feature, controllers instead return credentials to Boundary workers and create a session to a target where the user/client never has access to the credential. This workflow is called credential injection because the credential is injected into a worker’s session, instead of being passed back to the client.

Integrating private Vault with HCP Boundary requires a self-managed worker to be deployed in the same network as Vault. If a Vault credential store is defined with a worker filter, Boundary’s controller will know that Vault requests should be routed to workers matching that filter.

Private Vault On-Premise

HCP Boundary can access private HCP Vault Dedicated clusters via a self-managed worker placed in a network peered with Vault Dedicated's HashiCorp Virtual Network (HVN).

Private Vault HCPV

For simplicity, this tutorial demonstrates integrating HCP Boundary with a private Vault cluster running locally in development mode with the HCP worker also running locally in a Docker container.

Set up an SSH target

This tutorial assumes the reader has installed Boundary 0.11.0 or above, and a Vault binary greater than 1.7.0.

First, set up a target which will be used to test the credential injection using Vault.

Note

Alternatively, you can bring your own target to test the credential injection. Credential injection is currently supported for user/pass and sshpass credentials. If bringing your own target, skip to the Set up Vault section.

Docker is used to deploy an openssh-server target for testing credential injection.

Create an ssh keypair that will be used to log in to the openssh container.

First, open a terminal session and create a new directory to store the keypair in. This tutorial creates the openssh directory at ~/boundary, but you can place this directory elsewhere if desired.

$ mkdir -p ~/boundary/openssh && cd ~/boundary

Next, create a new ssh keypair.

$ ssh-keygen -t ecdsa -b 521 -f openssh/openssh-key

Press enter twice when prompted for a passphrase, or supply one if desired.

The above command will create the keypair in the ~/boundary/openssh/ directory at ~/boundary/openssh/openssh-key and ~/boundary/openssh/openssh-key.pub. Update this path if you wish to store the key elsewhere. Note the path to the keypair for later.

Ensure Docker is running, and then deploy the openssh container. Supply the absolute path to the openssh/ directory created earlier after the --volume= option, for example at /Users/myusername/boundary/openssh/. This will mount the folder within the docker container, making the public key file available at /keys/.

$ docker run -d \  --name=openssh-server \  --hostname=openssh-server \  -e SUDO_ACCESS=false \  -e USER_NAME=admin \  -e PUBLIC_KEY_FILE=/keys/openssh-key.pub \  -p 2222:2222 \  --restart unless-stopped \  -v /path/to/openssh/:/keys \  lscr.io/linuxserver/openssh-server:latest

Note that the username for the openssh server is admin, and the target is available on your localhost at port 2222 (127.0.0.1:2222).

Leave this container running for the duration of the tutorial.

Set up Vault

Open a new terminal session and run Vault in development mode.

$ vault server -dev -dev-root-token-id=groot==> Vault server configuration:              Api Address: http://127.0.0.1:8200                     Cgo: disabled         Cluster Address: https://127.0.0.1:8201              Go Version: go1.16.7              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")               Log Level: info                   Mlock: supported: false, enabled: false           Recovery Mode: false                 Storage: inmem                 Version: Vault v1.8.2             Version Sha: aca76f63357041a43b49f3e8c11d67358496959f ==> Vault server started! Log data will stream in below: 2022-05-05T14:01:31.647-0500 [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""2022-05-05T14:01:31.648-0500 [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set2022-05-05T14:01:31.649-0500 [INFO]  core: security barrier not initialized2022-05-05T14:01:31.650-0500 [INFO]  core: security barrier initialized: stored=1 shares=1 threshold=12022-05-05T14:01:31.650-0500 [INFO]  core: post-unseal setup starting2022-05-05T14:01:31.657-0500 [INFO]  core: loaded wrapping token key2022-05-05T14:01:31.657-0500 [INFO]  core: successfully setup plugin catalog: plugin-directory=""2022-05-05T14:01:31.657-0500 [INFO]  core: no mounts; adding default mount table2022-05-05T14:01:31.658-0500 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/2022-05-05T14:01:31.659-0500 [INFO]  core: successfully mounted backend: type=system path=sys/2022-05-05T14:01:31.659-0500 [INFO]  core: successfully mounted backend: type=identity path=identity/2022-05-05T14:01:31.663-0500 [INFO]  core: successfully enabled credential backend: type=token path=token/2022-05-05T14:01:31.663-0500 [INFO]  rollback: starting rollback manager2022-05-05T14:01:31.663-0500 [INFO]  core: restoring leases2022-05-05T14:01:31.664-0500 [INFO]  identity: entities restored2022-05-05T14:01:31.664-0500 [INFO]  identity: groups restored2022-05-05T14:01:31.666-0500 [INFO]  core: post-unseal setup complete2022-05-05T14:01:31.667-0500 [INFO]  expiration: lease restore complete2022-05-05T14:01:31.667-0500 [INFO]  core: root token generated2022-05-05T14:01:31.667-0500 [INFO]  core: pre-seal teardown starting2022-05-05T14:01:31.667-0500 [INFO]  rollback: stopping rollback manager2022-05-05T14:01:31.667-0500 [INFO]  core: pre-seal teardown complete2022-05-05T14:01:31.668-0500 [INFO]  core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:82012022-05-05T14:01:31.668-0500 [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:82012022-05-05T14:01:31.668-0500 [INFO]  core: post-unseal setup starting2022-05-05T14:01:31.668-0500 [INFO]  core: loaded wrapping token key2022-05-05T14:01:31.668-0500 [INFO]  core: successfully setup plugin catalog: plugin-directory=""2022-05-05T14:01:31.668-0500 [INFO]  core: successfully mounted backend: type=system path=sys/2022-05-05T14:01:31.668-0500 [INFO]  core: successfully mounted backend: type=identity path=identity/2022-05-05T14:01:31.668-0500 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/2022-05-05T14:01:31.669-0500 [INFO]  core: successfully enabled credential backend: type=token path=token/2022-05-05T14:01:31.669-0500 [INFO]  rollback: starting rollback manager2022-05-05T14:01:31.669-0500 [INFO]  core: restoring leases2022-05-05T14:01:31.670-0500 [INFO]  identity: entities restored2022-05-05T14:01:31.670-0500 [INFO]  identity: groups restored2022-05-05T14:01:31.670-0500 [INFO]  expiration: lease restore complete2022-05-05T14:01:31.670-0500 [INFO]  core: post-unseal setup complete2022-05-05T14:01:31.670-0500 [INFO]  core: vault is unsealed2022-05-05T14:01:31.672-0500 [INFO]  expiration: revoked lease: lease_id=auth/token/root/h49a4c4a4eef16a18370b98f2883fff349e0917dd5913d74c02bd97d04d8edcbb2022-05-05T14:01:31.675-0500 [INFO]  core: successful mount: namespace="" path=secret/ type=kv2022-05-05T14:01:31.686-0500 [INFO]  secrets.kv.kv_c1ca00f3: collecting keys to upgrade2022-05-05T14:01:31.686-0500 [INFO]  secrets.kv.kv_c1ca00f3: done collecting keys: num_keys=12022-05-05T14:01:31.686-0500 [INFO]  secrets.kv.kv_c1ca00f3: upgrading keys finishedWARNING! dev mode is enabled! In this mode, Vault runs entirely in-memoryand starts unsealed with a single unseal key. The root token is alreadyauthenticated to the CLI, so you can immediately begin using Vault. You may need to set the following environment variable:     $ export VAULT_ADDR='http://127.0.0.1:8200' The unseal key and root token are displayed below in case you want toseal/unseal the Vault or re-authenticate. Unseal Key: Yc27V3wm7l9axQaYQTxeZUlsn4AVOtqOPPUvvQuVOng=Root Token: groot Development mode should NOT be used in production installations!

Leave Vault running in dev mode and open a new terminal window.

Deploy a self-managed worker

An HCP worker deployed on the same network as Vault is required for integrating private Vault clusters with HCP Boundary. To learn more about setting up self-managed workers, refer to the Self-Managed Worker Registration with HCP Boundary tutorial.

Download the Boundary Enterprise binary

Download the Boundary Enterprise binary to the ~/boundary/ directory.

You can manually download the latest binary for your operating system by navigating to the Boundary releases page. The example below demonstrates downloading the binary using the command line.

Note

The binary version should match the version of the HCP control plane. Check the version of the control plane in the HCP Boundary portal, and download the appropriate version using wget. The example below installs the 0.13.0 version of the Boundary Enterprise binary.

Below is an example of downloading and unzipping the Boundary Enterprise binary on Ubuntu, MacOS, and Windows.

The following command downloads the Boundary Enterprise binary and unzips it to the current directory.

$ wget -q https://releases.hashicorp.com/boundary/0.13.0+ent/boundary_0.13.0+ent_linux_amd64.zip ;\sudo apt-get update && sudo apt-get install unzip ;\unzip *.zip

Once downloaded, verify the version of the boundary binary.

$ ./boundary version Version information:  Build Date:          2023-06-07T16:41:10Z  Git Revision:        fb4ed58459d555d480e70ddc20d2639c26ad0f8f  Metadata:            ent  Version Number:      0.13.0+ent

Ensure the Version Number matches the version of the HCP Boundary control plane. They should match in order to get the latest HCP Boundary features.

Write the worker config

Next, create a new file named hcp-worker.hcl in the ~/boundary/ directory.

$ touch ~/boundary/hcp-worker.hcl

Open the file with a text editor, such as Vi.

Paste the following configuration into the worker config file:

~/boundary/hcp-worker.hcl

disable_mlock = true hcp_boundary_cluster_id = "<cluster-id>" listener "tcp" {  address = "127.0.0.1:9202"  purpose = "proxy"} worker {  auth_storage_path = "/home/myusername/boundary/worker1"  tags {    type = ["worker", "vault"]  }}

Update the cluster id in the hcp-worker.hcl file:

The <cluster-id> on line 3 can be determined from the UUID in the HCP Boundary Cluster URL. For example, if your Cluster URL is:

https://c3a7a20a-f663-40f3-a8e3-1b2f69b36254.boundary.hashicorp.cloud, then the cluster id is c3a7a20a-f663-40f3-a8e3-1b2f69b36254

The auth_storage_path should match the full path to the ~/boundary/worker1 directory, such as /home/myusername/boundary/worker1.

Save this file.

Start the worker

With the worker config defined, start the worker server. Provide the full path to the worker config file (such as /home/myusername/boundary/hcp-worker.hcl).

$ ./boundary server -config="/home/myusername/boundary/hcp-worker.hcl" ==> Boundary server configuration:                                Cgo: disabled                        Listener 1: tcp (addr: "0.0.0.0:9202", max_request_duration: "1m30s", purpose: "proxy")                         Log Level: info                             Mlock: supported: true, enabled: false                           Version: Boundary v0.13.0+ent                       Version Sha: fb4ed58459d555d480e70ddc20d2639c26ad0f8f        Worker Auth Current Key Id: preaching-favored-verbalize-widen-duchess-relish-jurist-sly  Worker Auth Registration Request: GzusqckarbczHoLGQ4UA25uSRmUHY1BGSRA6cePp8RWHQFUYSrf3hnDw4ETPswFnMrcxx6tq7BUWD5azGULzPecPicuYGD6qg3qYvaGRgHgKwvh9FLY9Gu891KSj8hAef19JjHog8d7qpo9f9KoiwrhfcV2YxGyVu1P943656iNGCFHWiBR3ofsyTatQ7fzcMV2ciKtuYYGfx4FfiRStnkAzoE98RdR2LeCk2huRkFt7ayeeWVfD7Awm8xaZfFJn4pYRJwu2LRBeNs915warEBaS8XHXSKoi3cRUYif8Qu          Worker Auth Storage Path: /home/ubuntu/boundary/worker1          Worker Public Proxy Addr: 127.0.0.1:9202 ==> Boundary server started! Log data will stream in below: {"id":"Rb7PMMBdZa","source":"https://hashicorp.com/boundary/ip-172-31-84-100/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).StartControllerConnections","data":{"msg":"Setting HCP Boundary cluster address e58fe114-7624-431c-994d-b6670e90b09f.proxy.boundary.hashicorp.cloud:9202 as upstream address"}},"datacontentype":"application/cloudevents","time":"2022-09-19T22:16:38.770232603Z"}

The worker will start and begin attempting to connect to the upstream Controller.

The worker also outputs its authorization request as the Worker Auth Registration Request token. This will also be saved to a file, auth_request_token, defined by the auth_storage_path in the worker config.

Note the Worker Auth Registration Request: value. This value can also be located in the ~/boundary/worker1/auth_request_token directory. Copy this value.

Register the worker with HCP

HCP workers can be registered using the Boundary CLI or Admin Console Web UI.

Authenticate to HCP Boundary as the admin user.

  1. Log in to the HCP portal.

  2. From the HCP Portal's Boundary page, click Open Admin UI - a new page will open.

  3. Enter the admin username and password you created when you deployed the new instance and click Authenticate.

Once logged in, navigate to the Workers page.

Notice that only HCP workers are listed.

Click New.

HCP Workers

Scroll down to the bottom of the New Worker page and paste the Worker Auth Registration Request key you copied earlier.

Click Register Worker.

HCP Workers

Click Done and notice the new worker on the Workers page.

HCP Workers

Define the controller policy

As described in the Vault Credential Brokering Quickstart tutorial, the following Vault policy must be defined for the Boundary controller to support credential brokering.

Export the required Vault environment variables to store the Vault address and token.

$ export VAULT_ADDR="http://127.0.0.1:8200"; export VAULT_TOKEN="groot"

Now write the controller policy to Vault.

$ vault policy write boundary-controller -<<EOFpath "auth/token/lookup-self" {  capabilities = ["read"]}path "auth/token/renew-self" {  capabilities = ["update"]}path "auth/token/revoke-self" {  capabilities = ["update"]}path "sys/leases/renew" {  capabilities = ["update"]}path "sys/leases/revoke" {  capabilities = ["update"]}path "sys/capabilities-self" {  capabilities = ["update"]}EOF

Define the KV policy

A policy must also be defined for the key-value secrets that will be brokered via Vault.

Create a new kv-read policy in Vault.

$ vault policy write kv-read -<<EOFpath "secret/data/my-secret" {  capabilities = ["read"]}EOF

Lastly, create a new Vault KV credential at the secret/my-secret path and provide the username and private key for the openssh-server target machine you are connecting to. Ensure the path to the private key is entered after the @ symbol.

Note

If you brought your own target to test credential brokering against, enter the target’s username after username and path to the private key after the @.

$ vault kv put secret/my-secret username=admin private_key=@./openssh/openssh-key ==== Secret Path ====secret/data/my-secret ======= Metadata =======Key                Value---                -----created_time       2022-09-28T22:37:54.496224Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1

Additionally, brokered credentials can also be attached to SSH targets. These credentials will be brokered to the end user after injection. This is useful in the case that a user needs access to application credentials on the target after the SSH credentials are successfully injected.

Create a Vault token

Now create a Vault token.

$ vault token create \  -no-default-policy=true \  -policy="boundary-controller" \  -policy="kv-read" \  -orphan=true \  -period=20m \  -renewable=true \  -field=token

Example:

$ vault token create \  -no-default-policy=true \  -policy="boundary-controller" \  -policy="kv-read" \  -orphan=true \  -period=20m \  -renewable=true \  -field=token hvs.CAESIAEzd9SuNo84gRvPRnxmck05fJBC5hcbgbMHBlDvR_EqGh4KHGh2cy5YWFp4WUQ4YUtVRDU0amJoOElzcHVkSDU

Store the returned token value in the $CRED_STORE_TOKEN variable.

$ export CRED_STORE_TOKEN=hvs.CAESIAEzd9SuNo84gRvPRnxmck05fJBC5hcbgbMHBlDvR_EqGh4KHGh2cy5YWFp4WUQ4YUtVRDU0amJoOElzcHVkSDU

Tip

The token generation and environment variable assignment can be performed in a single step with this command:

$ export CRED_STORE_TOKEN=$(vault token create \  -no-default-policy=true \  -policy="boundary-controller" \  -policy="kv-read" \  -orphan=true \  -period=20m \  -renewable=true \  -field=token)

Set up Boundary

Start by logging in to HCP Boundary within the terminal.

Export the BOUNDARY_ADDR and BOUNDARY_AUTH_METHOD_ID environment variables. These values are gathered from the HCP Boundary Admin Console, as demonstrated in the Get Started with HCP Boundary tutorials.

$ export BOUNDARY_AUTH_METHOD_ID=copied-value-from-boundary-ui; export BOUNDARY_ADDR=copied-value-from-hcp-portal

Log in to the CLI as an admin user, or a user with privileges to manage resources in the global scope. Enter your admin username for login-name.

Enter the admin login name and password when prompted.

$ boundary authenticatePlease enter the login name (it will be hidden):Please enter the password (it will be hidden):Authentication information:  Account ID:      acctpw_VOeNSFX8pQ  Auth Method ID:  ampw_ZbB6UXpW3B  Expiration Time: Mon, 13 Feb 2023 12:35:32 MST  User ID:         u_ogz79sV4sTThe token was successfully stored in the chosen keyring and is not displayed here.

You will also need an ORG_ID to create the SSH test project within. You may create a new global scope for this, or utilize an existing scope used for testing.

If creating a new org for testing, execute the following:

$ boundary scopes create -scope-id=global -name=testing-org -description="SSH testing org" Scope information:  Created Time:        Mon, 09 May 2022 12:33:12 CDT  Description:         SSH testing org  ID:                  o_YcngFcjRcP  Name:                testing-org  Updated Time:        Mon, 09 May 2022 12:33:12 CDT  Version:             1   Scope (parent):    ID:                global    Name:              global    Type:              global   Authorized Actions:    no-op    read    update    delete

Whether you create a new org or not, be sure to export the org ID.

$ export ORG_ID=global-testing-org-id

Tip

If you have the jq utility installed, this can be performed in a single step with this command:

$ export ORG_ID=$(boundary scopes create \      -name=testing-org \      -description="SSH testing org" \      -format json | jq -r '.item | .id')

Set up a new project

Create a new project scope within Boundary. You will need an org ID to create the new project within. To view your existing orgs, execute boundary scopes list -recursive.

Note

You may use an existing org scope, or create a new org scope for the ssh-project. Refer to the Manage Scopes tutorial to learn more about creating and managing scopes.

$ boundary scopes create -scope-id=$ORG_ID -name=ssh-project -description="SSH test machines" Scope information:  Created Time:        Fri, 06 May 2022 14:13:39 CDT  Description:         SSH test machines  ID:                  p_TRfAEMzkAP  Name:                ssh-project  Updated Time:        Fri, 06 May 2022 14:13:39 CDT  Version:             1   Scope (parent):    ID:                o_26VOj9vJjN    Name:              ssh-org    Parent Scope ID:   global    Type:              org   Authorized Actions:    no-op    read    update    delete

Store the returned ID value in the $PROJECT_ID variable.

$ export PROJECT_ID=p_TRfAEMzkAP

Tip

If you have the jq utility installed, this can be performed in a single step with this command:

$ export PROJECT_ID=$(boundary scopes create \      -scope-id=$ORG_ID \      -name=ssh-project \      -description="SSH test machines" \      -format json | jq -r '.item | .id')

Create a Vault credential store

Next, you will create a new credential store within Boundary using the new token.

When setting up an HCP worker, it's important to create a worker filter for the credential store. A worker filter will identify workers that should be used as proxies for the new credential store, and ensure these credentials are brokered from the private Vault.

Recall the filters defined in the hcp-worker.hcl file earlier:

worker {  auth_storage_path = "./hcp-worker1"  tags {    type = ["worker", "vault"]  }}

The Tags or Name of the worker (hcp-worker1) can be used to create a worker filter for the target.

The example below adds a worker tag filter that searches for workers with the vault tag. Boundary will consider any worker with this tag assigned to it an acceptable proxy for the new credential store.

Create a new vault credential store with a worker-filter. The vault credential store type is used to integrate with Vault, but static credential stores can also be used with credential injection.

$ boundary credential-stores create vault \  -scope-id $PROJECT_ID \  -vault-address "http://127.0.0.1:8200" \  -vault-token $CRED_STORE_TOKEN \  -worker-filter='"vault" in "/tags/type"'

Example output:

$ boundary credential-stores create vault \  -scope-id $PROJECT_ID \  -vault-address "http://127.0.0.1:8200" \  -vault-token $CRED_STORE_TOKEN \  -worker-filter='"vault" in "/tags/type"' Credential Store information:  Created Time:        Thu, 28 Jul 2022 15:08:12 MDT  ID:                  csvlt_dcpUDG2INd  Type:                vault  Updated Time:        Thu, 28 Jul 2022 15:08:12 MDT  Version:             1   Scope:    ID:                p_jd7lspegXk    Name:              ssh-project    Parent Scope ID:   o_SN2K0DAGpi    Type:              project   Authorized Actions:    no-op    read    update    delete   Authorized Actions on Credential Store's Collections:    credential-libraries:      create      list   Attributes:    Address:           http://127.0.0.1:8200    Token HMAC:        NZJWT74Jyq09gLQfP4RiK5eDWfY7NWXYHoKL4nKQFDY

Store the returned ID value in the $CRED_STORE_ID variable.

$ export CRED_STORE_ID=csvlt_dcpUDG2INd

Tip

If you have the jq utility installed, this can be performed in a single step with this command:

$ export CRED_STORE_ID=$(boundary credential-stores create vault \  -scope-id $PROJECT_ID \  -vault-address "http://127.0.0.1:8200" \  -vault-token $CRED_STORE_TOKEN \  -worker-filter='"vault" in "/tags/type"' \  -format json | jq -r '.item.id')

Create a Vault credential library

Create a new credential library of type ssh_private_key within Boundary using the credential store ID and passing the vault-path of my-secret.

$ boundary credential-libraries create vault-generic \  -credential-store-id $CRED_STORE_ID \  -vault-path "secret/data/my-secret" \  -name "vault-cred-library" \  -credential-type ssh_private_key

Example output:

$ boundary credential-libraries create vault-generic \  -credential-store-id $CRED_STORE_ID \  -vault-path "secret/data/my-secret" \  -name "vault-cred-library" \  -credential-type ssh_private_key Credential Library information:  Created Time:          Thu, 28 Jul 2022 15:09:59 MDT  Credential Store ID:   csvlt_dcpUDG2INd  ID:                    clvlt_yUr2VCd3lJ  Name:                  vault-cred-library  Type:                  vault-generic  Updated Time:          Thu, 28 Jul 2022 15:09:59 MDT  Version:               1   Scope:    ID:                  p_jd7lspegXk    Name:                ssh-project    Parent Scope ID:     o_SN2K0DAGpi    Type:                project   Authorized Actions:    no-op    read    update    delete   Attributes:    HTTP Method:         GET    Path:                secret/data/my-secret   Credential Type:    ssh_private_key

Store the returned ID value in the $CRED_LIB_ID variable.

$ export CRED_LIB_ID=clvlt_yUr2VCd3lJ

Tip

If you have the jq utility installed, this can be performed in a single step with this command:

$ export CRED_LIB_ID=$(boundary credential-libraries create vault-generic \-credential-store-id $CRED_STORE_ID \-vault-path "secret/data/my-secret" \-name "vault-cred-library" \-credential-type ssh_private_key \-format json | jq -r '.item.id')

Create an SSH target

First, create a new static host catalog.

$ boundary host-catalogs create static \  -scope-id=$PROJECT_ID \  -name=ssh-catalog \  -description="For static SSH targets"

Example output:

$ boundary host-catalogs create static \  -scope-id=$PROJECT_ID \  -name=ssh-catalog \  -description="For static SSH targets" Host Catalog information:  Created Time:        Thu, 28 Jul 2022 15:20:08 MDT  Description:         For static SSH targets  ID:                  hcst_FsS2LaNSy3  Name:                ssh-catalog  Type:                static  Updated Time:        Thu, 28 Jul 2022 15:20:08 MDT  Version:             1   Scope:    ID:                p_jd7lspegXk    Name:              ssh-project    Parent Scope ID:   o_SN2K0DAGpi    Type:              project   Authorized Actions:    no-op    read    update    delete   Authorized Actions on Host Catalog's Collections:    host-sets:      create      list    hosts:      create      list

Export the new host catalog ID as the HOST_CATALOG_ID environment variable

$ export HOST_CATALOG_ID=hcst_FsS2LaNSy3

Tip

If you have the jq utility installed, this can be performed in a single step with this command:

$ export HOST_CATALOG_ID=$(boundary host-catalogs create static \  -scope-id=$PROJECT_ID \  -name=ssh-catalog \  -description="For static SSH targets" \  -format json | jq -r '.item.id')

Now create a new static host.

Note

Learners may also bring their own target to test credential injection. If bringing your own target, enter the connection address for the -address option. The connection port is supplied when creating a target later on.

$ boundary hosts create static -name=openssh \  -description="openssh test host" \  -address="127.0.0.1" \  -host-catalog-id=$HOST_CATALOG_ID

Example output:

$ boundary hosts create static -name=openssh \  -description="openssh test host" \  -address="127.0.0.1" \  -host-catalog-id=$HOST_CATALOG_ID Host information:  Created Time:        Thu, 28 Jul 2022 15:20:41 MDT  Description:         openssh test host  Host Catalog ID:     hcst_FsS2LaNSy3  ID:                  hst_zNheyZ8wSg  Name:                openssh  Type:                static  Updated Time:        Thu, 28 Jul 2022 15:20:41 MDT  Version:             1   Scope:    ID:                p_jd7lspegXk    Name:              ssh-project    Parent Scope ID:   o_SN2K0DAGpi    Type:              project   Authorized Actions:    no-op    read    update    delete   Attributes:    address:           127.0.0.1

Export the openssh host ID as the HOST_ID environment variable.

$ export HOST_ID=hst_zNheyZ8wSg

Tip

If you have the jq utility installed, this can be performed in a single step with this command:

$ export HOST_ID=$(boundary hosts create static -name=openssh \  -description="openssh test host" \  -address="127.0.0.1" \  -host-catalog-id=$HOST_CATALOG_ID \  -format json | jq -r '.item.id')

Then create a static host set.

$ boundary host-sets create static -name="ssh-machines" \  -description="SSH host set" \  -host-catalog-id=$HOST_CATALOG_ID

Example output:

$ boundary host-sets create static -name="ssh-machines" \  -description="SSH host set" \  -host-catalog-id=$HOST_CATALOG_ID Host Set information:  Created Time:        Thu, 28 Jul 2022 15:21:44 MDT  Description:         SSH host set  Host Catalog ID:     hcst_FsS2LaNSy3  ID:                  hsst_IXcLAq4HCJ  Name:                ssh-machines  Type:                static  Updated Time:        Thu, 28 Jul 2022 15:21:44 MDT  Version:             1   Scope:    ID:                p_jd7lspegXk    Name:              ssh-project    Parent Scope ID:   o_SN2K0DAGpi    Type:              project   Authorized Actions:    no-op    read    update    delete    add-hosts    set-hosts    remove-hosts

Export the host set ID as the HOST_SET_ID environment variable.

$ export HOST_SET_ID=hsst_IXcLAq4HCJ

Tip

If you have the jq utility installed, this can be performed in a single step with this command:

$ export HOST_SET_ID=$(boundary host-sets create static -name="ssh-machines" \  -description="SSH host set" \  -host-catalog-id=$HOST_CATALOG_ID \  -format json | jq -r '.item.id')

Add the new host to the host set.

$ boundary host-sets add-hosts -id=$HOST_SET_ID -host=$HOST_ID Host Set information:  Created Time:        Thu, 28 Jul 2022 15:21:44 MDT  Description:         SSH host set  Host Catalog ID:     hcst_FsS2LaNSy3  ID:                  hsst_IXcLAq4HCJ  Name:                ssh-machines  Type:                static  Updated Time:        Thu, 28 Jul 2022 15:22:58 MDT  Version:             2   Scope:    ID:                p_jd7lspegXk    Name:              ssh-project    Parent Scope ID:   o_SN2K0DAGpi    Type:              project   Authorized Actions:    no-op    read    update    delete    add-hosts    set-hosts    remove-hosts   Host IDs:    hst_zNheyZ8wSg

With the host set up, create a new target of type ssh and set its default port to 2222. Because the target is only accesible to Boundary through the self-managed worker, also specify the -egress-worker-filter option to filter for the worker using the filter expression "worker" in "/tags/type".

Note

Learners may also bring their own target to test credential injection. If bringing your own target, enter the SSH connection port (likely 22 for SSH) for the -default-port option.

$ boundary targets create ssh -scope-id $PROJECT_ID \  -name="openssh target" \  -session-connection-limit=-1 \  -default-port 2222 \  -egress-worker-filter '"worker" in "/tags/type"'

Example output:

$ boundary targets create ssh -scope-id $PROJECT_ID \  -name="openssh target" \  -session-connection-limit=-1 \  -default-port 2222 \  -egress-worker-filter '"worker" in "/tags/type"' Target information:  Created Time:               Thu, 28 Jul 2022 15:23:42 MDT  Egress Worker Filter:       "worker" in "/tags/type"  ID:                         tssh_Lp3M0ERnoK  Name:                       openssh target  Session Connection Limit:   -1  Session Max Seconds:        28800  Type:                       ssh  Updated Time:               Thu, 28 Jul 2022 15:23:42 MDT  Version:                    1   Scope:    ID:                       p_jd7lspegXk    Name:                     ssh-project    Parent Scope ID:          o_SN2K0DAGpi    Type:                     project   Authorized Actions:    no-op    read    update    delete    add-host-sources    set-host-sources    remove-host-sources    add-credential-sources    set-credential-sources    remove-credential-sources    authorize-session   Attributes:    Default Port:             2222

Store the returned ID value in the $TOKEN_ID variable.

$ export TARGET_ID=tssh_Lp3M0ERnoK

Tip

If you have the jq utility installed, this can be performed in a single step with this command:

$ export TARGET_ID=$(boundary targets create ssh \   -name="openssh target" \   -scope-id $PROJECT_ID \   -session-connection-limit=-1 \   -default-port 2222 \   -egress-worker-filter '"worker" in "/tags/type"' \   -format json | jq -r '.item.id')

Add the host set to the target.

$ boundary targets add-host-sources -host-source=$HOST_SET_ID -id=$TARGET_ID Target information:  Created Time:               Thu, 28 Jul 2022 15:23:42 MDT  Egress Worker Filter:       "worker" in "/tags/type"  ID:                         tssh_Lp3M0ERnoK  Name:                       openssh target  Session Connection Limit:   -1  Session Max Seconds:        28800  Type:                       ssh  Updated Time:               Thu, 28 Jul 2022 15:24:56 MDT  Version:                    2   Scope:    ID:                       p_jd7lspegXk    Name:                     ssh-project    Parent Scope ID:          o_SN2K0DAGpi    Type:                     project   Authorized Actions:    no-op    read    update    delete    add-host-sources    set-host-sources    remove-host-sources    add-credential-sources    set-credential-sources    remove-credential-sources    authorize-session   Host Sources:    Host Catalog ID:          hcst_FsS2LaNSy3    ID:                       hsst_IXcLAq4HCJ   Attributes:    Default Port:             2222

Associate the SSH target with the SSH credential library for credential injection.

$ boundary targets add-credential-sources -id $TARGET_ID -injected-application-credential-source $CRED_LIB_ID Target information:  Created Time:               Thu, 28 Jul 2022 15:23:42 MDT  Egress Worker Filter:       "worker" in "/tags/type"  ID:                         tssh_Lp3M0ERnoK  Name:                       openssh target  Session Connection Limit:   -1  Session Max Seconds:        28800  Type:                       ssh  Updated Time:               Thu, 28 Jul 2022 15:27:10 MDT  Version:                    3   Scope:    ID:                       p_jd7lspegXk    Name:                     ssh-project    Parent Scope ID:          o_SN2K0DAGpi    Type:                     project   Authorized Actions:    no-op    read    update    delete    add-host-sources    set-host-sources    remove-host-sources    add-credential-sources    set-credential-sources    remove-credential-sources    authorize-session   Host Sources:    Host Catalog ID:          hcst_FsS2LaNSy3    ID:                       hsst_IXcLAq4HCJ   Injected Application Credential Sources:    Credential Store ID:      csvlt_dcpUDG2INd    ID:                       clvlt_yUr2VCd3lJ   Attributes:    Default Port:             2222

Inject credentials into sessions

Now you are ready to inject credentials directly into the shell session. You can accomplish this using the boundary connect command, which has a helper called ssh.

Enter yes when prompted to establish the connection.

$ boundary connect ssh -target-id $TARGET_ID The authenticity of host 'hst_c0ytm16ggf ([127.0.0.1]:49174)' can't be established.ECDSA key fingerprint is SHA256:+IfWTTBnnsSUSr3ueHjPU8q6jufc9GBDJQVKOIiZzQ4.Are you sure you want to continue connecting (yes/no/[fingerprint])? yesWarning: Permanently added 'hst_c0ytm16ggf' (ECDSA) to the list of known hosts.Welcome to OpenSSH Server openssh-server:~$

Troubleshooting

If the boundary connect ssh command fails with the following error message, then the Vault credential store token may have expired.

Expand the accordion below to continue troubleshooting.

$ Error from controller when performing authorize-session action against given target Error information:  Kind:                FailedPrecondition  Message:             No egress workers can handle this session, as they have all been filtered  out.  Status:              400  context:             Error from controller when performing authorize-session action against  given target

The error information states that No egress workers can handle this session, as they have all been filtered.

This implies that Boundary does not understand which worker should handle the session, and cannot route traffic through the worker it selected.

In this example the error could be due to either of the following errors:

  1. Boundary is unable to log into Vault using the provided credential store token.
  2. The openssh target's egress worker filter is unset, or incorrect

First, troubleshoot the Vault credential store by generating a new Vault token.

$ export CRED_STORE_TOKEN=$(vault token create \  -no-default-policy=true \  -policy="boundary-controller" \  -policy="kv-read" \  -orphan=true \  -period=20m \  -renewable=true \  -field=token) && echo $CRED_STORE_TOKEN

Copy this token (such as hvs.CAESIAbmG5WIi0qILYm87ucbJSGa0rv879aJAchvFR6wG0pbGh4KHGh2cy5xY1BzTGpHUW91ZEx4ZGFTcFVDTkdPMTY), and then update Boundary's Vault cred store with the new token value.

  1. Log in to the Boundary Admin UI and select the testing-org scope. Select the ssh-project, then click Credential Stores.

  2. Select the Vault credential store (such as csvlt_EsUGNcboLT). Scroll down to the bottom of the credential store's Details page, and click Edit Form.

  3. Paste the newly generated vault token into the Token field, then click Save at the bottom of the page.

With the updated token in place, attempt to connect to the SSH target again.

$ boundary connect ssh -target-id $TARGET_ID

If you are still unable to establish a connection, check that the filter has been defined correctly on the SSH target.

  1. Navigate to Targets using the sidebar and select the openssh target.

  2. Scroll down to the bottom of the target Details page. Ensure that the Egress worker filter toggle is on, and the Filter contains the following expression:

    "worker" in "/tags/type"

  3. If this is missing, click Edit Form and enable the Egress worker filter, and paste in the above filter expression. When finished, click Save.

After verifying the correct filter value, attempt to establish a session.

$ boundary connect ssh -target-id $TARGET_ID

When finished, the user can close the connection to the server using exit, or the session can be canceled directly from Boundary in a new terminal window:

$ boundary sessions list -recursive Session information:  ID:                    s_EOsEgwF3JN    Scope ID:            p_ZbpoZDE03L    Status:              active    Created Time:        Mon, 09 May 2022 12:39:02 CDT    Expiration Time:     Mon, 09 May 2022 20:39:01 CDT    Updated Time:        Mon, 09 May 2022 12:39:02 CDT    User ID:             u_1234567890    Target ID:           tssh_zIUXGVzB5L    Authorized Actions:      no-op      read      read:self      cancel      cancel:self

Copy the session ID and cancel the session:

$ boundary sessions cancel -id s_EOsEgwF3JN Session information:  Auth Token ID:       at_wXd9EASFp6  Created Time:        Mon, 09 May 2022 12:39:02 CDT  Endpoint:            ssh://127.0.0.1:2222  Expiration Time:     Mon, 09 May 2022 20:39:01 CDT  Host ID:             hst_C0YtM16ggf  Host Set ID:         hsst_DyUCqhHuiP  ID:                  s_EOsEgwF3JN  Status:              canceling  Target ID:           tssh_zIUXGVzB5L  Type:                ssh  Updated Time:        Mon, 09 May 2022 12:43:28 CDT  User ID:             u_1234567890  Version:             3   Scope:    ID:                p_ZbpoZDE03L    Name:              ssh-project    Parent Scope ID:   o_YcngFcjRcP    Type:              project   Authorized Actions:    no-op    read    read:self    cancel    cancel:self   States:    Start Time:        Mon, 09 May 2022 12:43:28 CDT    Status:            canceling     End Time:          Mon, 09 May 2022 12:43:28 CDT    Start Time:        Mon, 09 May 2022 12:39:02 CDT    Status:            active     End Time:          Mon, 09 May 2022 12:39:02 CDT    Start Time:        Mon, 09 May 2022 12:39:02 CDT    Status:            pending

You can verify the session was canceled by listing the sessions again and noticing the terminated Status:

$ boundary sessions list -recursive Session information:  ID:                    s_EOsEgwF3JN    Scope ID:            p_ZbpoZDE03L    Status:              terminated    Created Time:        Mon, 09 May 2022 12:39:02 CDT    Expiration Time:     Mon, 09 May 2022 20:39:01 CDT    Updated Time:        Mon, 09 May 2022 12:43:29 CDT    User ID:             u_1234567890    Target ID:           tssh_zIUXGVzB5L    Authorized Actions:      no-op      read      read:self      cancel      cancel:self

Note

There are cases in which targets might have multiple SSH keys or username_password credentials attached as injected application credential sources. SSH supports multiple credential sources, and Boundary will try them until it finds a usable credential when attempting a connection. This scenario is useful when rotating private keys. Both the old key and the new key can be attached to a target simultaneously, and Boundary will try both of them, stopping if one is successful. This facilitates key rotation without downtime.

Inject credentials with the Desktop App

To log into Boundary using the Desktop App, the BOUNDARY_ADDR (Boundary cluster address) and BOUNDARY_AUTH_METHOD_ID (user Auth Method ID) values must be gathered from the HCP Boundary Admin Console, as demonstrated in the HCP Boundary Getting Started tutorial.

Open the Boundary desktop app.

Enter the Boundary cluster URL (for example, https://ffee961b-5fd8-4e68-ba1d-2bbb487b576e.boundary.hashicorp.cloud) and click Submit.

Authenticate using your HCP Boundary user credentials.

Boundary Desktop Login

Under the Targets page, notice the target details for ssh-target.

Boundary Desktop Target Details

Click Connect to initiate a session.

Boundary Desktop Target Connect

The Successfully Connected page will display the target ID (Target Connection details) and Proxy URL.

To start a session, open your terminal or SSH client. A session can be started using SSH and the Proxy URL from the Boundary desktop app.

For the openssh docker target, connect on 127.0.0.1 and provide the proxy port using the -p option. Enter yes when prompted to establish a connection.

$ ssh 127.0.0.1 -p 51968 The authenticity of host '[127.0.0.1]:2222 ([127.0.0.1]:2222)' can't be established.ECDSA key fingerprint is SHA256:SzGEXZDM/6u0/9lcO8EDADHXzDsemSFaL6Q36KBrlhQ.Are you sure you want to continue connecting (yes/no/[fingerprint])? yesWarning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.Welcome to OpenSSH Server openssh-server:~$

When finished, the user can close the connection to the server by entering exit, or the session can be canceled directly from the Boundary desktop app under the Sessions view.

Cleanup and teardown

  1. Locate the terminal session used to execute the vault dev server command, and execute ctrl+c to stop Vault.

  2. Unset the environment variables used in any active terminal windows for this tutorial.

    $ unset VAULT_ADDR; unset VAULT_TOKEN

  3. Destroy the boundary-worker-hcp container created for the tutorial.

    $ docker rm -f boundary-worker-hcp

  4. Destroy the openssh-server container created for the tutorial.

    $ docker rm -f openssh-server

    Check your work by executing docker ps and ensure there are no more containers from the tutorial leftover. If unexpected containers still exist, execute docker rm -f <CONTAINER_ID> against each to remove them.

  5. Cleanup the testing-org org and associated test resources in your HCP Boundary cluster.

    Open the Orgs page and select the testing-org.

    Navigate to the Org Settings page and click the Manage dropdown.

    Click Delete Org and confirm the deletion.

    Delete Org

Help and reference

Previous

SSH certificate injection with HCP Boundary

 Next

Vault cred brokering quickstart