Exercise 7: Pass secrets to the build pipeline

Objective

Learn how to pass confidential information to the build process

Gitlab Variables

Suppose you’re building an application that needs access to a database, and you want to test that access from your CI/CD pipeline. You need to pass the name of the database host, the account, and password, to the running pipeline.

You could save them in a file in your git repository, but that makes them visible to anyone who can view the repository. Not a good idea, to say the least!

An alternative is to use Gitlab Variables. Go to your project page, Settings tab -> CI/CD, find Variables and click on the Expand button. Here you can define variable names and values, which will be automatically passed into the gitlab pipelines, and are available as environment variables there.

There are two types of variable you can define: Variable and File, as you can see in this example:

../../../_images/gitlab-06-variables.png

DB_PASSWORD has been declared as a Variable, and given a value. This means that you can use $DB_PASSWORD directly in your pipelines as it is.

SECRET_KEY has been defined as a File. This means that the value of SECRET_KEY, in your pipeline, will be a randomly-generated filename. The contents of that file will be the value that you supply for that key.

Go ahead and add two variables like this, one a variable and one a gile. Give them arbitrary values, but do not check the Masked button for either. Click Save variables at the bottom to save them.

Then edit your .gitlab-ci.yml file, and, in the before_script section, add some lines to print those variables to the screen (i.e. to the logfile). Something like this:

before_script:
  - echo "Starting..."
  - export DOCKER_IMAGE=$RELEASE_IMAGE
  - if [ "$CI_BUILD_REF_NAME" == "master" ]; then export DOCKER_IMAGE=$LATEST_IMAGE; fi
  - echo "Build docker image $DOCKER_IMAGE"
  - echo DB_PASSWORD is $DB_PASSWORD
  - echo SECRET_KEY is $SECRET_KEY
  - echo Contents of the SECRET_KEY file are
  - cat $SECRET_KEY

Commit your file to git and push it to the server. Check the build log for the first stage and you should see how it all works, as below:

../../../_images/gitlab-07-build-log-unmasked.png

Masking the variable values

It would be nice if gitlab offered a way to prevent your variables from being printed to the logfile, in case that happens by accident. Well, it does! You can mask them.

Go back to the Settings -> CI/CD page and expand the Variables section again. Set both variables to be Masked, remembering to click Save variables. Go back to CI/CD -> Pipelines, click on the top pipeline (the one that just ran). Each stage has an icon on it to re-run that stage - two curved arrows pointing at each other in a circle. Re-run the compile stage, and examine the output.

You’ll see that this time, despite explicitly trying to print the secrets to the screen, gitlab has masked them out, preventing them from being displayed:

../../../_images/gitlab-07-build-log-masked.png

Taking it to the next level

Although this is already very secure, anyone who has access to the project dashboard can see these variables and inspect or change their values. You may not want that, you may want to restrict the visibility even further, so the secrets can only be seen at the time they’re used, and not by everyone who has access to the project.

One way to do that is to encrypt the secret variable value with OpenSSL, and store the encrypted value as the variable value. The decryption key is not stored in the project, instead it’s made available in the runtime environment of the gitlab runner, or of the environment where the final product is deployed. Then, either in your pipeline or in your production environment, you decrypt the encrypted secret at the point of use, and nowhere else.

An attacker will then have to gain access to your dashboard and the configuration of the deployment environment before they can read the secrets. Because of this, it’s safe to bake the encrypted variable into the docker image itself - it’s useless without the decryption key!

We don’t go into the details of this here, come and talk to us if you’re interested in doing this.

Conclusion

You can define variables in the gitlab user interface which are then passed on to the build pipeline. You can tell gitlab to prevent these variables from being printed to the logfiles. This means you can use them to store sensitive information, like database passwords etc.

Best Practices

  • Use variables to pass sensitive information, like passwords, account names, hostnames, license keys etc into your gitlab builds, do not under any circumstances commit their values into the code repository.
  • Ask yourself if you really need to use variables at all!
    • e.g., if you need a database, why not create one and populate it with a schema for testing as part of the CI/CD pipeline? This means you don’t need to store or pass any secrets at all.
    • Alternatively, can you store secrets in the deployment environment, so they’re only visible at runtime?
  • Make sure you mask variables unless you are certain they’re not sensitive. Even seemingly innocuous information like hostnames or port numbers can be used to successfully attack an infrastructure.
  • For extra security, encrypt the variable values, and provide the decryption key to the pipelines and deployments by other means. Kubernetes has techniques that can help here, see the sessions later today.
  • Always separate your test and production environments, don’t use the same infrastructure for both.
  • If you bake variables and their values into your docker images, think very carefully about how you do it!