The NIST glossary defines security as “a condition that results from the establishment...

Following CI/CD security best practices can ensure that security issues are fixed without markedly slowing down the pipeline. Learn what these practices are.

Best Practices to Follow for CI/CD Pipeline

CI/CD Pipeline Security Best Practices

Imagine there are multiple developers who make code changes every day. These changes need to be integrated yet not introduce errors. They need to be tested before being deployed. What can be done? The test is automated and then deployed to the development/staging environment before it goes to production. This automation is triggered with every revision. The process is called continuous integration (CI).

After the integration and testing are complete, the application needs to be deployed automatically and continuously to production. This is called continuous deployment. Alongside this is continuous delivery, wherein the application is delivered continuously to production but requires approval before deployment. Once approved, the process continues to trigger scripts or methods to deploy the application.

The CI/CD pipeline or workflow may be different for each organization. It depends on the programming language used, the organization’s policy, CI/CD security tools, the deployment target, the application type, etc.

You can see that there are some activities in the CI, such as Plan, Code, Build, and Test. Let’s explore them in brief.

Plan: Before an application is deployed, features and bug fixing need to be planned. Jira, Trello and Basecamp are some tools that can help plan features.

Code: Here, developers write code changes for features or fix bugs. Tools used include Visual Studio Code, Geany, and Visual Studio, among others. The tools depend on the programming language of the application.

Build: The build process can be automated using certain tools. Some recommendations are Github Action, CircleCI, and GitLab CI. The build result is usually as packages, binaries, zip files, etc. Note that different programming languages may have different steps and results.

Test: After the code is built, it needs to be tested. Test units are usually run here. The test process is not only a unit test, however; you can also do integration tests, performance tests, functional tests, and acceptance tests.

If the test is passed, the code is released and deployed. You can add the approval process before the deployment process as mentioned before, i.e., continuous delivery.

The next stage is monitoring. Some of the tools used are Grafana, Prometheus, Jaeger, AWS CloudWatch, and Datadog. The feedback from the monitoring process is valuable for the next planning process. Think of it as a loop.

What is CI/CD?

Security can be incorporated in the build or test step. You can add security checking in the pull request actions to avoid security problems before merging the changes from the request.

You can add static code analysis, which is a process that will analyze code and check for vulnerabilities. There are many other security practices that you can add to your workflow, such as dynamic program analysis, penetration testing, etc.

Below are a few general CI/CD security best practices. They aren’t specific to a platform.

CI/CD best practices

A mistake that some people make is to use hard-coded or plain text credentials in the CI/CD pipeline. Use encrypted credentials instead. You can use a secret manager to handle multiple secrets which are reused at many CI/CD pipelines. Credentials can include, but are not limited to, passwords, secrets tokens, and API keys.

Don’t use plain text credentials

If you create new keys or update new passwords, you will need to update multiple CI/CD pipelines. A secret manager can be used to rotate credentials. You’ll only need to update the secrets in the same.

Rotate credentials used by CI/CD pipeline

You should validate each input in the CI/CD pipeline. For example, you have a pipeline that allows a user to trigger it with an input to define the version you want to deploy. You have a script to publish by that version, but do not validate it and use the value as it is.

When you don’t validate the input, attackers or hackers can use it to inject scripts (script injection). They may do this to find out credentials, download malware, and so on. Hence, it’s crucial to validate inputs to make sure they are expected.

Validate inputs

Consider using OpenID as an authentication option in your CI/CD pipeline as it reduces the need to store credentials or secrets. You should also use it when connecting to cloud services like AWS, Azure, and GCP.

Having a sound understanding of basic cyber security is a plus so that you can implement the best practices in your CI/CD pipeline. If possible, learn more about DevOps (development and operations) and DevSecOps (development, security, and operations).

To sum up some of the important CI/CD security best practices, ensure you don’t expose your credentials, use a secret manager whenever possible, and use OpenID as an authentication method when connecting to cloud services. Implementing the principle of least privilege is an additional step you can take.

Use OpenID authentication

The Continuous Delivery Foundation (CDF) provides best practices for CI/CD. For CI, these are:

Have a complete application package in the build step that contains the binaries, version tag, and difference report.

Don’t blindly include all files; you should know what you add to the packages. This way, you can avoid including injected files that may pose a security risk.

Try to understand the build parameters. You don’t want to deliver debug symbols to production.

This reference contains CDF's best practices.

It’s important to conduct unit testing in CI to reduce the chance of delivering defective software. Unit testing validates each unit of the code to ensure it performs as expected.

You can check here for the published config pull request, and here for the sample running job. You can also check here for the published package.

Let’s implement a security practice to have source code analysis in our previous workflow.

Create a file with the name “codeql.yml” under the directory with the name “.github/workflows”.

name: "CodeQL"


on:
 push:
   branches: [ "main" ]
 pull_request:
   # The branches below must be a subset of the branches above
   branches: [ "main" ]
 schedule:
   - cron: '37 18 * * 0'


jobs:
 analyze:
   name: Analyze
   runs-on: ubuntu-latest
   permissions:
     actions: read
     contents: read
     security-events: write
   strategy:
     fail-fast: false
     matrix:
       language: [ 'csharp' ]


   steps:
   - name: Checkout repository
     uses: actions/checkout@v3


   # Initializes the CodeQL tools for scanning.
   - name: Initialize CodeQL
     uses: github/codeql-action/init@v2
     with:
       languages: ${{ matrix.language }}


   - name: Autobuild
     uses: github/codeql-action/autobuild@v2


   - name: Perform CodeQL Analysis
     uses: github/codeql-action/analyze@v2
     with:
       category: "/language:${{matrix.language}}"

You can push and pull requests to check the result.

Security practices for continuous integration

Here are some best practices for CD provided by CDF:

Have approval and approval gates. This is usually done using notification. For example, asking the product owner to approve the changes.

There are some common deployment methods, such as canary, blue/green, and rolling blue/green deployments. You need to consider which model fits your business.

The CD process should have immutable deployments. This means you should not have manual changes.

You should define policies and rules that align with the organization's needs. They will control the automation process for continuous delivery/deployment.

Security practices can be implemented after delivering the packages. For example, if you use Docker and push docker image, you can scan it to check the binaries that are delivered to it. There are various tools for scanning the docker image, such as Snyk and Trivy. Docker registry also provides scan options that help to check the image and find vulnerabilities.

It’s important to avoid unnecessary content in software packages or docker images. Choose the appropriate image when delivering the docker image. Secure the docker registry with a private docker registry as well.

Security practices for continuous delivery/deployment

Risk-based security is an important aspect of a CI/CD pipeline as it reduces the chance of data breaches and security vulnerabilities. Avoiding plain text credentials, implementing credential rotation, validating inputs, having source code analysis, and omitting unnecessary files in packages are some of the measures to take.

Conclusion

The NIST glossary defines security as “a condition that results from the establishment and maintenance of protective measures that enable an organization to perform its mission or critical functions despite risks posed by threats to its use of systems. Protective measures may involve a combination of deterrence, avoidance, prevention, recovery, and correction that should form part of the organization’s risk management approach.”

In this post, we’ll learn about the importance of security in the CI/CD pipeline as well as CI/CD security best practices and how to implement them.

A condition that results from the establishment and maintenance of protective measures that enable an organization to perform its mission or critical functions despite risks posed by threats to its use of systems.

CI stands for continuous integration, which is a practice to continuously integrate code into an application.

CD is continuous deployment/continuous delivery. Continuous deployment is a practice of continuously deploying an application automatically to production. On the other hand, continuous delivery has an approval process before it is deployed automatically.

There are many best practices that we can follow, such as ensuring credentials are not in plain text, rotating secrets, validating inputs, using a secret manager, and using OpenID authentication when connecting to cloud services.

Secrets or credentials of CI/CD should not be in plain text but should be encrypted. A secret manager should be used to manage CI/CD secrets and rotate them.

For the CI pipeline, this can be done by encrypting secrets, using a secret manager, and using source code analysis. For the CD pipeline, this includes adding only the necessary files to the docker image and protecting the docker registry.

Frequently Asked Questions

What’s up with Turing?
Get the latest news about us here.

Press

Know more about remote work.
Checkout our blog here.

Blog

Have any questions?
We’d love to hear from you.

FOR DEVELOPERS

CI/CD Pipeline Security Best Practices

Author

Bervianto Leo Pratama

10 Best Practices for Writing Clean React Code

How to Implement HashMap in Java from Scratch

How to Build Microservices With Node.JS

Typescript: How to Check Type of Objects & Variables

Math Foundations for Data Science

Frequently Asked Questions

Press

Blog

Contact

Hire remote developers