CNB

Pipeline

# Pipeline Introduction

A Pipeline represents a build workflow consisting of one or more Stages that are executed sequentially. For more information, refer to the Stage Introduction.

A basic configuration for a Pipeline is as follows:

name: Pipeline Name
docker:
  image: node
  build: dev/Dockerfile
  volumes:
    - /root/.npm:copy-on-write
  # endpoints are only supported in private deployment scenarios, not in SaaS scenarios
  endpoints:
    - 80
git:
  enable: true
  submodules: true
services:
  - docker
env:
  TEST_KEY: TEST_VALUE
imports:
  - https://xxx/envs.yml
  - ./env.txt
label:
  type: MASTER
  class: MAIN
stages:
  - stage1
  - stage2
  - stage3
failStages:
  - stage1whenFail
  - stage2whenFail
ifModify:
  - a.txt
  - src/**/*
retry: 3
allowFailure: false

# name

  • type: String

Specifies the name of the pipeline. The default name is pipeline. When multiple pipelines run in parallel, the default names are pipeline, pipeline-1, pipeline-2, and so on. You can define the name to differentiate between different pipelines.

# runner

  • type: Object

Specify build machine-related parameters.

  • tags: Optional. Specifies the tags of the build machine to be used.
  • cpus: Optional. Specifies the number of CPU cores required for the build.

# tags

  • type: String | Array<String>
  • default: cnb:arch:amd64

Specifies the tags of the build machine to be used.

The following official build machine tags are available in SaaS scenarios:

  1. cnb:arch:amd64 represents an AMD64 architecture build machine.
  2. cnb:arch:arm64:v8 represents an ARM64/v8 architecture build machine.

Example:

main:
  push:
    - runner:
        tags: cnb:arch:arm64:v8
      stages:
        - name: uname
          script: uname -a

# cpus

  • type: Number

Specifies the maximum number of CPU cores required for the build (memory = CPU cores * 2G), where the CPU and memory do not exceed the actual size of the runner machine.

If not configured, the maximum available CPU cores will not exceed the actual number of CPU cores of the runner machine.

Example:

# cpus = 1, memory = 2G
main:
  push:
    - runner:
        cpus: 1
      stages:
        - name: echo
          script: echo "hello world"

# docker

  • type: Object

Specifies the parameters related to Docker. For more details, refer to build-env.

  • image: Specifies the environment image for the current Pipeline. All tasks within this Pipeline will be executed in this image environment.
  • build: Specifies a Dockerfile to build a temporary image to be used as the value for image.
  • volumes: Declares volumes for caching scenarios.

# image

  • type: Object | String

Specifies the environment image for the current Pipeline. All tasks within this Pipeline will be executed in this image environment. It supports referencing environment variables.

  • image.name: Image name, such as node:14.

  • image.dockerUser:

    • type: String

    Specifies the Docker username used to pull the specified image. If the specified image is private, this must be specified to grant permission to access the image.

  • image.dockerPassword:

    • type: String

    Specifies the Docker user password used to pull the specified image. If the specified image is private, this must be specified to grant permission to access the image.

If image is specified as a string, it is equivalent to specifying image.name.

Example:

main:
  push:
    - docker:
        # Use the node:14 image from the official Docker registry as the build container
        image: node:14
      stages:
        - name: show version
          script: node -v

main:
  push:
    - imports: https://xxx/xxx.yml
      docker:
        # Use a private image as a container for construction, and enter the docker username and password.
        image:
          name: docker.cnb.cool/images/pipeline-env:1.0
          dockerUser: $DOCKER_USER
          dockerPassword: $DOCKER_PASSWORD
      stages:
        - name: echo
          script: echo "hello world"

docker.yml

DOCKER_USER: user
DOCKER_PASSWORD: password

# build

  • type: Object | String

Specifies a Dockerfile to build a temporary image to be used as the value for image. It supports referencing environment variables.

  • build.dockerfile:

    • type: String Specifies the path to the Dockerfile.

  • build.target:

    • type: String

    Corresponds to the --target parameter in docker build. It selectively builds a specific stage in the Dockerfile instead of building the entire Dockerfile.

  • build.by:

    • type: Array<String> | String

    Declares the list of files that are dependencies during the caching build process.

    Note: Files not listed in the by list, except for the Dockerfile, are treated as non-existent during the build caching process.

    When using String type, multiple files can be separated by commas.

  • build.versionBy:

    • type: Array<String> | String

    Used for version control. When the content of the specified files changes, it is considered a new version. The calculation logic is based on the expression: sha1(dockerfile + versionBy + buildArgs).

    When using String type, multiple files can be separated by commas.

  • build.buildArgs:

    • type: Object

    Inserts additional build arguments during the build process (--build-arg $key=$value). If the value is null, only the key is added (--build-arg $key).

  • build.ignoreBuildArgsInVersion:

    • type: Boolean Specifies whether to ignore buildArgs in version calculation. See versionBy for more details.

  • build.sync:

    • type: String Specifies whether to wait for successful docker push before continuing. The default value is false.

If build is specified as a string, it is equivalent to specifying build.dockerfile.

Usage of Dockerfile:

main:
  push:
    - docker:
        # Specify the build environment using Dockerfile
        build: ./image/Dockerfile
      stages:
        - stage1
        - stage2
        - stage3

main:
  push:
    - docker:
        # Specify the build environment using Dockerfile
        build:
          dockerfile: ./image/Dockerfile
          dockerImageName: cnb.cool/project/images/pipeline-env
          dockerUser: $DOCKER_USER
          dockerPassword: $DOCKER_PASSWORD
      stages:
        - stage1
        - stage2
        - stage3

main:
  push:
    - docker:
        # Specify the build environment using Dockerfile
        build:
          dockerfile: ./image/Dockerfile
          # Build only the builder, not the entire Dockerfile
          target: builder
      stages:
        - stage1
        - stage2
        - stage3

Usage of Dockerfile versionBy:

Example: Cache pnpm in the environment image to speed up the subsequent pnpm i process.

main:
  push:
    - docker:
        build:
          dockerfile: ./Dockerfile
          versionBy:
            - package-lock.json
      stages:
        - name: pnpm i
          script: pnpm i
        - stage1
        - stage2
        - stage3

FROM node:16

# Replace with the actual source
RUN npm config set registry https://xxx.com/npm/ \
  && npm i -g pnpm \
  && pnpm config set store-dir /lib/pnpm

WORKDIR /data/orange-ci/workspace

COPY package.json package-lock.json ./

RUN pnpm i

[pnpm i] Progress: resolved 445, reused 444, downloaded 0, added 100
[pnpm i] Progress: resolved 445, reused 444, downloaded 0, added 141
[pnpm i] Progress: resolved 445, reused 444, downloaded 0, added 272
[pnpm i] Progress: resolved 445, reused 444, downloaded 0, added 444, done
[pnpm i] 
[pnpm i] dependencies:
[pnpm i] + mocha 8.4.0 (10.0.0 is available)
[pnpm i] 
[pnpm i] devDependencies:
[pnpm i] + babel-eslint 9.0.0 (10.1.0 is available)
[pnpm i] + eslint 5.16.0 (8.23.0 is available)
[pnpm i] + jest 27.5.1 (29.0.2 is available)
[pnpm i] 
[pnpm i] 
[pnpm i] Job finished, duration: 6.8s

# volumes

  • type: Array<String> | String

Declares volumes for caching scenarios. Multiple volumes can be specified as an array or separated by commas in a string. It supports referencing environment variables. The supported formats are:

  1. <group>:<path>:<type>
  2. <path>:<type>
  3. <path>

Meaning of each item:

  • group: Optional. Volume group, used to isolate volumes between different groups.
  • path: Required. Absolute path or relative path (starting with ./) to mount the volume. The path is relative to the workspace.
  • type: Optional. Volume type. The default value is copy-on-write. Supported types are:
    • read-write or rw: Read-write, concurrent write conflicts need to be handled manually. Suitable for serial build scenarios.
    • read-only or ro: Read-only, write operations will throw exceptions.
    • copy-on-write or cow: Read-write, changes (additions, modifications, deletions) are merged after a successful build. Suitable for concurrent build scenarios.
    • copy-on-write-read-only: Read-only, changes (additions, deletions, modifications) are discarded after the build.
    • data: Creates a temporary volume that will be automatically cleaned up when the pipeline ends.

# copy-on-write

copy-on-write is used in caching scenarios and supports concurrency.

copy-on-write technology allows the system to share the same copy of data before any modifications are made, enabling efficient caching replication. In a concurrent environment, this approach avoids conflicts between read and write operations on the cache because private copies of the data are only created when modifications are actually needed. As a result, only write operations incur data copying, while read operations can safely be performed in parallel without concerns about data consistency. This mechanism significantly improves performance, especially in cache scenarios where there are more reads than writes.

# data

data is used for sharing data by making a specified directory within a container available to other containers.

This is achieved by creating a data volume and then mounting it into various containers. The difference from directly mounting a directory from the build machine into a container is that if the specified directory already exists within the container, the existing content will be automatically copied to the data volume before it is mounted. This ensures that the content of the data volume is not directly overwritten by the content of the container directory.

# Volumes Example

Example 1: Mounting directories from the build node into containers to achieve local caching effect

main:
  push:
    - docker:
        image: node:14
        # Declare volumes
        volumes:
          - /data/config:read-only
          - /data/mydata:read-write
          # Use cache and update simultaneously
          - /root/.npm
          # Use main cache and update simultaneously
          - main:/root/.gradle:copy-on-write
      stages:
        - stage1
        - stage2
        - stage3
  pull_request:
    - docker:
        image: node:14
        # Declare volumes
        volumes:
          - /data/config:read-only
          - /data/mydata:read-write
          # Use copy-on-write cache
          - /root/.npm
          - node_modules
          # Use main cache for PR, but do not update
          - main:/root/.gradle:copy-on-write-read-only
      stages:
        - stage1
        - stage2
        - stage3

Example 2: Sharing files packaged within a container with other containers

# .cnb.yml
main:
  push:
    - docker:
        image: go-app-cli # Assuming there is a go application at /go-app/cli in the image
        # Declare volumes
        volumes:
          # This path exists in the go-app-cli image, so when executing the job image, the content of this path will be copied to a temporary data volume, which can be shared with other task containers
          - /go-app
      stages:
        - name: show /go-app-cli in job container
          image: alpine
          script: ls /go-app

# endpoints

  • type: Array<Number>

Endpoints are only supported in private deployment scenarios and not supported in SaaS scenarios.

Declares a list of target ports that will be automatically converted to endpoints (ip:port) accessible from outside. The converted endpoints can be referenced using environment variables.

The environment variable format is: $CNB_PIPELINE_ENDPOINT_{port}

The target ports need to be bound to the 0.0.0.0 network interface and currently only support the TCP protocol.

# endpoints Example

# .cnb.yml
main:
  push:
    - name: example-for-endpoints
      docker:
        image: nginx:latest
        endpoints:
          - 80
      stages:
        - name: echo endpoints
          script: echo "$CNB_PIPELINE_ENDPOINT_80"

In this example, the endpoints section is used to declare the target port 80 for the nginx container. The echo endpoints stage then prints the value of the environment variable $CNB_PIPELINE_ENDPOINT_80, which will contain the IP address and port number that can be used to access the container from outside.

# git

  • type: Object

Provide Git repository configuration.

# git.enable

  • type: Boolean
  • default: true

Specifies whether to fetch the code.

For branch.delete event, the default value is false. For other events, the default value is true.

# git.submodules

  • type: Object | Boolean
  • default: true

Specify whether to fetch submodules.

Support specifying specific parameters in the form of an Object, with default values for missing fields:

{
  "enable": true,
  "remote": false
}

Basic usage:

main:
  push:
    - git:
        enable: true
        submodules: true
      stages:
        - name: echo
          script: echo "hello world"
    - git:
        enable: true
        submodules:
          enable: true
          remote: true
      stages:
        - name: echo
          script: echo "hello world"

# git.submodules.enable

Specify whether to fetch submodules (submodules).

# git.submodules.remote

When executing git submodule update, you can add the --remote parameter to fetch the latest code for each submodule every time.

# services

  • type: Array<String>

To declare the services required during the build, use the format: name:[version], where version is optional.

Currently supported services include:

  • git-clone-yyds
  • docker
  • vscode

# service:git-clone-yyds

enabled by default

The git-clone-yyds (opens new window) service based on OverlayFS can complete code preparation within seconds, regardless of the size of the code repository.

Example:

main:
  push:
    - services:
        - git-clone-yyds
      docker:
        image: alpine
      stages:
        - name: ls
          script: ls -al

# service:docker

To enable the dind service,

Declare it when you need to perform operations like docker build or docker login during the build process. It will automatically inject the docker daemon and docker cli into the environment.

Example:

main:
  push:
    - services:
        - docker
      docker:
        image: alpine
      stages:
        - name: docker info
          script:
            - docker info
            - docker ps

# service:vscode

Declaration for remote development is required.

Example:

$:
  vscode:
    - services:
        - vscode
        - docker
      docker:
        image: alpine
      stages:
        - name: uname
          script: uname -a

# env

  • type: Object

Specify environment variables. You can define a set of environment variables for use during task execution. They are effective for all non-plugin tasks within the current Pipeline.

# imports

  • type: Array<String> | String

Specify the file path of a repository within the current repository or another CNB Git repository. This file can be read as a source of environment variables. Typically, a private repository is used to store credentials such as npm or docker account passwords.

Note: Ineffective for plugin tasks.

Supported file formats:

  • yaml: All property names under the root path will be exported as environment variables, parsing files with the extension .yml or .yaml.
  • json: All property names at the root level will be exported as environment variables. Files with the .json extension will be parsed.
  • plain: Each line should be in the format key=value. This format will be used for files with extensions other than the ones mentioned above. (Not recommended)

Priority of keys with the same name:

  • When the imports configuration is an array, if there are duplicate parameters, the later configurations will override the earlier ones.
  • If there are duplicates with the env parameter, the parameters in env will override the ones in the imports files.

Variable assignment:

The file paths specified in imports can read environment variables. If it is an array, the file paths below can read variables from the files above.

main:
  push:
    - imports:
        - ./env1.json
        - $FILE
        - https://xxx/xxs.yml
      stages:
        - name: echo
          script: echo $TEST_ENV

For reference on configuration file access control, please see Configuration File Reference Authorization.

Example:

team_name/project_name/* matches all repositories under a project:

key: value

allow_slugs:
  - team_name/project_name/*

Allowing all repositories to reference:

key: value

allow_slugs:
  - "**"

# label

  • type: Object

Specifies labels for the pipeline. Each label can have a string value or an array of strings. These labels can be used for filtering pipeline records and other functionalities.

Here's an example of a workflow: Merge to the main branch triggers a pre-release deployment, and after tagging, it triggers a production deployment.

main:
  push:
    - label:
        # Regular pipeline for the master branch
        type:
          - MASTER
          - PREVIEW
      stages:
        - name: install
          script: npm install
        - name: CCK-lint
          script: npm run lint
        - name: BVT-build
          script: npm run build
        - name: UT-test
          script: npm run test
        - name: pre release
          script: ./pre-release.sh

$:
  tag_push:
    - label:
        # Regular pipeline for product releases
        type: RELEASE
      stages:
        - name: install
          script: npm install
        - name: build
          script: npm run build
        - name: DELIVERY-release
          script: ./release.sh

# stages

  • type: Array<Job>

Defines a set of stage tasks that run sequentially.

# failStages

  • type: Array<Job>

Defines a set of stage tasks to be executed when the normal flow fails. These tasks will be executed sequentially.

# ifNewBranch

  • type: Boolean
  • default: false

If set to true, the pipeline will only be executed if the current branch is a new branch (i.e., CNB_IS_NEW_BRANCH is true).

When both ifNewBranch and ifModify are present, if either condition is met, the pipeline will be executed.

# ifModify

  • type: Array<String> | String

Specifies a glob file matching rule or a list of file names. The pipeline will only be executed if the files match the specified rules.

Example 1:

The pipeline will be executed if the modified file list includes a.js or b.js.

ifModify:
  - a.js
  - b.js

Example 2:

The pipeline will be executed if the modified file list includes files with the .js extension. **/*.js matches all .js files in subdirectories, while *.js matches .js files in the root directory.

ifModify:
  - "**/*.js"
  - "*.js"

Example 3:

Reverse matching: trigger the pipeline when there are changes except in the legacy directory.

ifModify:
  - "**"
  - "!(legacy/**)"

Example 4:

Reverse matching: trigger the pipeline when there are changes in the src directory, except in the src/legacy directory.

ifModify:
  - "src/**"
  - "!(src/legacy/**)"

# Supported Events

  • For the push event on non-new branches, the changes between before and after are analyzed to determine the modified files.
  • In the pipeline triggered by the push event on non-new branches, the same file change analysis is performed when the cnb:apply event is triggered.
  • For events triggered by a pull request (PR), the modified files within the pull request are analyzed.
  • In events triggered by a pull request (PR) and further triggered by the cnb:apply event, the modified files within the pull request are also analyzed.

Due to the potentially large number of modified files, the limit for file calculation is set to a maximum of 300 files.

# breakIfModify

  • type: Boolean
  • default: false

If set to true, the build will be terminated if the source branch has been updated before the job execution.

# retry

  • type: Number
  • default: 0

Specifies the number of retry attempts for failed jobs. A value of 0 means no retries will be performed.

# allowFailure

  • type: Boolean
  • default: false

Specifies whether the current pipeline is allowed to fail.

When set to true, the failure status of the pipeline will not be reported to CNB.

# lock

  • type: Object | Boolean

Sets a lock for the pipeline. The lock will be automatically released after the pipeline execution. Locks cannot be used across repositories.

Behavior: When pipeline A acquires the lock, if pipeline B requests the lock, it can either terminate pipeline A or wait for pipeline A to finish and release the lock before continuing execution.

  • key:

    • type: String

    Custom lock name. By default, it is set to branchName-pipelineName, which means the lock scope is the current pipeline.

  • expires:

    • type: Number
    • default: 3600 (1 hour)

    Lock expiration time in seconds. After expiration, the lock will be automatically released.

  • timeout:

    • type: Number
    • default: 3600 (1 hour)

    Timeout in seconds for waiting for the lock.

  • cancel-in-progress:

    • type: Boolean
    • default: false

    Whether to terminate the pipeline that currently holds the lock or is waiting for the lock, allowing the current pipeline to acquire the lock and execute.

  • wait:

    • type: Boolean
    • default: false

    Whether to wait if the lock is already acquired (without consuming pipeline resources and time). If set to false, an error will be thrown directly. Cannot be used together with cancel-in-progress.

  • cancel-in-wait:

    • type: Boolean
    • default: false

    Whether to terminate the pipeline that is waiting for the lock and add the current pipeline to the waiting lock queue. Need to be used with the wait attribute.

Example 1: Lock as a Boolean value

main:
  push:
    - lock: true
      stages:
        - name: stage1
          script: echo "stage1"

Example 2: Lock as an Object

main:
  push:
    - lock:
        key: key
        expires: 600 # 10 minutes
        wait: true
        timeout: 60 # maximum wait time of 1 minute
      stages:
        - name: stage1
          script: echo "stage1"

Example 3: Stop the previous running pipeline under a pull_request

main:
  pull_request:
    - lock:
        key: pr
        cancel-in-progress: true
      stages:
        - name: echo hello
          script: echo "stage1"

Stage →