Secrets

Table of Contents

Overview

OpenShift CI provides its own Hashicorp Vault instance that we can use. This makes the use of secrets standard for anyone who needs them. Most of everything that we do for secrets was discovered from the OpenShift CI documentation Adding a New Secret to CI

Collections

Create Collection

Each scenario being tested will have its own collection. This allows us to prevent secret sharing between members of other collections meant to hold secrets for other scenarios.

First go to selfservice.vault.ci.openshift.org and login.

If you do not see a collection in the table for the scenario that your testing it either hasn't been created yet or you don't have access to it.

Please reach out on slack at #forum-qe-cspi-ocp-ci to ask if the collection for the scenario you are testing has already been created. If it has go to the next section

If it hasn't you can create the collection by clicking the New Collection button and adding the name. Follow the naming structure of {product short name}-qe.

Get Access to a Scenario Collection

If you do not see a collection in the table, and you have verified that it exists then you just need to be added by one of the collection owners. Please reach out on slack at #forum-qe-cspi-ocp-ci and provide details about what and why you would like to be added to a specific collection.

Get Access to the cspi-qe Collection

This collection holds team specific information holding things like AWS creds, pull secrets, ..etc. If you need access to this collection reach out on slack at #forum-qe-cspi-ocp-ci and provide details about why you would like to be added to the cspi-qe collection.

Vault

Sign in to Vault

Now that you are a member of a collection you can login to vault and access that collection.

Go to vault.ci.openshift.org and click Sign in with OIDC Provider.

You'll be redirected to your secrets homepage. cubbyhole/ is just your users personal space for any secrets you want to store. We care about kv/ this is where you'll find secret directories that correspond to the collections that you are a member of.

From here you can use the UI or Vault CLI to view, create, edit, and delete secrets.

Secrets in OpenShift CI

In order to use a secret in Vault in OpenShift CI, the secret must include two key/value pairs:

  • secretsync/target-name: The name of the secret in OpenShift CI

  • secretsync/target-namespace: The namespace of the secret in OpenShift CI

Outside of those two values, any key/value pair desired can be added to the same path for that secret making them available for use in OpenShift CI.

OpenShift CI Secret Name

The name of a secret in OpenShift CI must be unique. When deciding on a name for a secret, make sure to be descriptive. Using a descriptive name will help with readability in OpenShift CI configuration files as well as making it less-likely that a name is used twice.

Best practice in naming these secrets would be something similar to scenario_name-name_of_secret. As an example, for the MTR scenario, the FTP credentials used are named mtr-ftp-credentials. This name is unique and descriptive.

OpenShift CI Secret Namespace

The Namespace of your secret in the build clusters. Multiple namespaces can be targeted by using a comma-separated list. For the purpose of this document, test-credentials will be used as the namespace. This namespace allows for the use of a secret in the step of a job within OpenShift CI.

From the OpenShift CI Adding a New Secret to CI documentation:

The most common case is to use secrets in a step of a job. In this case, we require the user to mirror secrets to test-credentials namespace. The pod which runs the step can access the secrets defined in the credentials stanza of the step definition.

Tutorial - Create and Use Secrets in OpenShift CI

Create a New Secret

  1. Create a new collection, if one doesn't already exist.

  2. Sign into vault and navigate to kv/NAME_OF_COLLECTION.

  3. Click the "Create secret" button in the upper right-hand corner.

  4. Choose name for the secret in Vault and append that to the text already in the "Path for this secret" textbox.

    • Note: This name is not the same as the name described in the OpenShift CI Secret Name section of this document. This name can be anything and does not need to be unique.

  5. Add a key of secretsync/target-name to the secret and choose a value. This value must be unique and descriptive.

  6. Add a key of secretsync/target-namespace and set the value to test-credentials.

  7. Add the desired key/value pair(s) to the secret for use during execution.

    • Example: Add a username key with a value of SomeUser and a password key with a value of CleverPassword. These values will be available during OpenShift CI execution.

    • IMPORTANT SECURITY INFORMATION:

      • We'll be covering information from the official OpenShift CI secret docs here

      • There are 3 different practices shown in that document, depending on which case you use, your secrets may or may not get redacted when being posted to public logs.

        1. Good Practice - This is when you have a single secret value per secret key. This allows for your secret value to be redacted when printed to a public log.

        2. Risky Practice - In some cases you may need a single secret key to hold multiple secret values. In this case its possible to store multiple secret values to a single key, but this comes with a massive security risk. The secret redaction process cannot make sense of this multiple secret value syntax and it runs the risk of your secrets being posted publicly.

        3. Acceptable Practice - This is a mix of both practices you see above. Because there are cases that require us to store multiple values in a single secret key we are sometimes forced to use the Risky Practice. This is ok so long as we also include each individual secret value in its own key following the Good Practice. These individual keys holding the individual secrets from your key with many secrets will not be used in your automated CI run. Your CI run will still use the key that holds many secrets. What we are doing here though is that when it comes time to post the public log, the individual secrets that you've plucked out of the larger secret will be read by the secret redaction process and this will ensure that any secret that you've placed in the larger secret key will be redacted and never shown in a public log.

      • See the Acceptable Practice Section here for a visual representation of what your vault should look like. If you have any questions about this process please reach us at the Slack channel #forum-qe-layered-product so that we can work together to make sure that you are not at risk of leaking credentials.

  8. Click the "Save" button in the bottom left-hand side of the screen

Now that a secret has been created in Vault, it is available for use in OpenShift CI steps.

Use Secret During OpenShift CI Execution

For the following steps, this document will refer to the configuration of a ref step in OpenShift CI. For simplicity, here is the configuration that will be referred to:

some-test-ref.yaml

ref:
  as: some-test-ref
  from: cli
  commands: some-test-ref-commands.sh
  credentials:
    - namespace: test-credentials
      name: some-test-secret
      mount_path: /tmp/secrets/test
  documentation: |-
    Show how to use secrets in OpenShift CI

  1. Create the step you would like to use your new secret in, if it doesn't already exist.

  2. Add a credentials stanza to the step's configuration (.yaml) file.

  3. Under the new credentials stanza, add a new list item for your secret. Multiple secrets can be used in a step, but for the sake of simplicity, there is only one in the configuration above.

  4. In the new list object, add the following key/value pairs:

    1. namespace: The namespace specified in step 6 of the Create a New Secret section of this tutorial.

      • Note: This will usually be set to test-credentials

    2. name: The name chosen in step 5 of the Create a New Secret section of this tutorial.

    3. mount_path: This will be the directory where the secrets are stored for use during execution.

      • Note: If you add more than one secret to the credentials stanza, you must use a different mount-path value for each secret.

  5. Use the new secret in your execution.

    1. Each key/value pair added to your secret in Vault ,other than secretsync/target-namespace and secretsync/target-name, will have have a file in the mount_path directory. The file will be named the same as the key in Vault and will contain a single clear-text line with the value from Vault.

      • Example:

        • In step 7 of the Create a New Secret section of this tutorial, we created the following key/value pairs:

          • username: SomeUser

          • password: CleverPassword

        • In this example, there would be two files found in tmp/secrets/test - one file named username and one file named password. Each file would hold a single clear-text line - the value ("SomeUser" and "CleverPassword")

    2. Using these values can be as easy as setting the contents of each file to a variable: USERNAME=$(cat /tmp/secrets/test/username)

Rollback Secret to Previous Version

➜  export VAULT_ADDR=https://vault.ci.openshift.org

➜  vault login -method=oidc

# List vault secrets that you have access to
➜  vault kv list kv/selfservice
Keys
----
/3scale-test-image/
/acm-qe/
/aws-managed-cspi-qe/
/cspi-qe/
/oadp-qe/
/sbo-qe/

# Get secrets
➜  vault kv get kv/selfservice/cspi-qe/cluster-secrets

In the metadata section we see the version of the secret

➜  vault kv get kv/selfservice/cspi-qe/cluster-secrets
=============== Secret Path ===============
kv/data/selfservice/cspi-qe/cluster-secrets

====== Metadata ======
Key              Value
---              -----
created_time     2023-03-06T12:39:47.702500423Z
deletion_time    n/a
destroyed        false
version          19

This tells us that the secret version for cspi-qe/cluster-secrets is at version 19. We can get older versions of secrets like this

➜  vault kv get -version=12 kv/selfservice/cspi-qe/cluster-secrets
=============== Secret Path ===============
kv/data/selfservice/cspi-qe/cluster-secrets

====== Metadata ======
Key              Value
---              -----
created_time     2022-11-02T12:40:14.129461007Z
deletion_time    n/a
destroyed        false
version          12

At this time we believe only the most recent 10 versions are saved.

If we want to recover from an older version of a vault secret then we can do the following:

IMPORTANT

Please do not run this rollback command against any active secrets just for learning purposes (I purposely included a typo so this command should not work but just wanted to emphasize this, please do not fix this command and run just to experiment)!

There is a cubbyhole (personal space) that you are given access to in vault that you can store secrets and experiment with them there.

➜  vault kv rollback -version=10 kv/selfservice/cspi-qe/clutser-secrets
PreviousPodNextStep Registry

Last updated