What is a Pipeline

A Deployment Pipeline (pipeline for short) is a series of steps for applying code changes to a production environment. It describes the CI/CD process and relies on a CI/CD platform (such as Jenkins, GitLab CI/CD, Buildkite, GOCD, etc.). Through a pipeline, code changes are deployed to any environment within minutes or tens of minutes via a fully automated, scripted process.

Components of a Pipeline

A pipeline is composed of a series of sequential stages. Each stage contains one or more steps.

Stage

A pipeline consists of multiple stages that run serially — if the previous stage fails, the next stage will not execute; only when the previous stage passes will the next stage run.

For example, in the diagram below, if the Test stage fails, the Acceptance test stage and all subsequent stages will not be executed.

Because stages run serially, it is important to divide them thoughtfully. Poor stage division can lead to longer execution times.

Step

A stage contains one or more steps.

A step represents a specific action to be performed, such as running unit tests. Each step should have exactly one automation script that describes what it does.

Steps within a stage run in parallel — they do not interfere with each other and execute simultaneously. Within a stage, only when all steps succeed is the stage considered successful; if any one step fails, the stage fails.

As shown below, the Test stage contains two steps: unit test and code style check. If unit test fails, the Test stage fails and the next stage will not run. The two steps — unit test and code style check — execute simultaneously. If unit test finishes before code style check, the Test stage will wait for code style check to complete regardless of the unit test result.

Build and Job

Every time a pipeline is triggered, it creates a build. Think of a pipeline as a class and a build as an instance of that class. Each build consists of a series of jobs, where each job is the concrete execution of a step defined in the pipeline.

Why Do We Need a Pipeline

• A pipeline makes the build, deployment, testing, and release process visible to everyone.

• Problems are discovered and resolved earlier.

• Any version of an application can be deployed and released to any environment through a fully automated process.

A Pipeline Example (Buildkite)

steps:
  - name: Test
    command: auto/test

  - name: Lint
    command: auto/lint

  - wait

  - name: Release
    command: auto/release
    branchs: master
  
  - wait

  - name: Deploy to staging
    command: auto/deploy-to-staging
    branchs: master
  
  - wait

  - name: Acceptance test
    command: auto/accptance-test
    branchs: master

  # This approach is continuous deployment; use the following for continuous delivery instead
  # - block: Deploy to production?
  #   branchs: master
  - wait

  - name: Deploy to production
    command: auto/deploy-to-production
    branchs: master

In Buildkite, wait is used to separate stages, and block is used for manual intervention.

Best Practices

Here are some best practices I have applied in real projects.

Use Pipeline as Code, Not Manual Configuration

CI/CD tools generally support both Pipeline as Code and manual configuration for managing a pipeline. Manual configuration is not recommended, however, because it is difficult to trace, and is prone to errors. We should never perform any manual operations on any infrastructure — everything should be Infrastructure as Code.

CI/CD-related configuration — including pipelines, automation scripts, and infrastructure — should live in the same repository as the business code. They carry the same weight as business code, and every change to them is reviewed by the team.

Use “Static” Pipelines, Avoid Dynamic Pipelines

The more dynamic and flexible a pipeline is, the more complex it becomes, and the lower its readability. A pipeline is shared within your team and may even be visible to other teams. With CI/CD, we want team members to focus on business logic — a complex pipeline places an unnecessary burden on the team.

In my view, a pipeline configuration file should be a simple YAML file (as shown in the example above), not a shell script, and certainly not Groovy.

Use Automation Scripts, Not Inline Commands

CI/CD tools typically allow you to call commands directly inside a pipeline. For example, in the example above, you could replace the script after command in the Test step with a raw command such as rspec:

steps:
  - name: Test
    command: rspec

Using inline commands directly in a pipeline is risky. CI/CD agents run inside your infrastructure and therefore have access to it — agents used for deployment often carry elevated privileges. A malicious actor could simply modify the command field to cause harm. Using automation scripts avoids this problem entirely, since the scripts must be read from the repository.

Inline commands should be prohibited in pipelines.

Use a Single Deployment Script, Not Multiple

The only difference between the staging and production environments should be configuration — for example, they may run in different VPCs or different AWS accounts. In the example above, deploying to staging and production calls two different scripts, but in practice both scripts invoke a shared auto/deploy script underneath. Using a single deployment script reduces discrepancies between deployments to different environments and gives us greater confidence that our tests are reliable.

Use Two Pipelines, Not All-in-One

In most cases, the steps for deploying to production and beyond are split into a separate pipeline. This also requires splitting repositories: for example, if you have a hello-world Git repo as your main codebase, you also need to create a hello-world-deploy Git repo for production deployments. The last step of the hello-world pipeline commits artifact information (such as a Docker image tag) to hello-world-deploy via Git, which then triggers the deployment pipeline for hello-world-deploy.

Why do this? Rollbacks are inevitable. When a production deployment has issues and must be rolled back, without a separate repo you would have to start over from scratch — rebuilding, re-testing, and so on, which takes time. With a separate repo, you can simply revert the deployment-related commit in hello-world-deploy, push the change, and trigger the deployment pipeline.

Git is the single source of truth.

For this reason, the all-in-one pipeline shown in the example above is not recommended in practice.

Run Some Simple Tests Before Pushing Code

Before committing code to the repository, run some lightweight tests — such as unit tests and code style checks — to reduce the load on the CI/CD platform.

Generate Only One Artifact per Build

If multiple artifacts (such as Docker images) are generated, we cannot guarantee that what we tested is exactly what gets deployed to production.

Use Automatic Triggers, Not Manual Triggers

Every code change committed to the repository should trigger the pipeline automatically. Without this, we cannot be certain that the relevant tests have been run and the code has been deployed, which adds burden to future development. Requiring developers to manually trigger the pipeline on the CI/CD platform for every commit also increases their workload.

Keep Code Changes as Small as Possible

Code changes should be as small as possible. This reduces the cost of team code review and lowers the risk of introducing bugs.

Never Let a Pipeline Fail Overnight

When a pipeline fails, stop what you are doing immediately and fix it. Ensure the pipeline is green before the end of the workday.

References

Continuous integration vs. continuous delivery vs. continuous deployment

Introduction to CI/CD with GitLab

Continuous integration

Delivery