CI/CD工具介紹

2022/10/13

名詞解釋

  • CI

    • Continuous Integration

    • ​持續整合

    • 是一種軟體工程流程,在測試驅動開發(TDD)的作法中,通常還會搭配自動單元測試。

    • 持續整合的提出主要是為解決軟體進行系統整合時面臨的各項問題。

  • CD

    • Continuous Deployment​/Delivery

    • 持續部署/交付

    • 持續部署是將開發的產品或系統以系統導向方式持續地將系統部署到指定階段環境

      • ​如開發的環境(development environment)

      • 如上線環境(production environment)

    • 持續部署是將開發的產品或是系統功能持續的改進與交付給需求方

CI工具分類

  • 由雲端主機供應商推出
     

  • 私人公司推出的CI/CD工具產品或服務

    • 自架版本

    • 雲端服務版本

由雲端主機供應商推出

  • AWS

    • AWS CodePipeline
       

  • Azure

    • Azure DevOps

  • Google Cloud Platform

    • ​Cloud Build、Cloud Deploy 以及 Cloud Run

私人公司推出的CI/CD工具產品或服務

  • Jenkins

  • GitLab CI/CD

自架版本(self-hosted)

私人公司推出的CI/CD工具產品或服務

  • Travis CI

  • GitLab.com上的GitLab CI/CD

  • CircleCI
  • GitHub Workflows

雲端服務版本(cloud)

Jenkins安裝(自架需要先安裝)

peter@devops-demo:~$ sudo apt-get install openjdk-11-jdk-headless
peter@devops-demo:~$ wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
[sudo] password for peter:
OK
peter@devops-demo:~$ sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
peter@devops-demo:~$
peter@devops-demo:~$ sudo apt-get update
Hit:2 http://mirrors.digitalocean.com/ubuntu focal InRelease
Hit:3 https://repos-droplet.digitalocean.com/apt/droplet-agent main InRelease
Hit:4 http://mirrors.digitalocean.com/ubuntu focal-updates InRelease
Hit:5 http://mirrors.digitalocean.com/ubuntu focal-backports InRelease
Ign:1 https://pkg.jenkins.io/debian-stable binary/ InRelease
Get:6 https://pkg.jenkins.io/debian-stable binary/ Release [2044 B]
Get:7 https://pkg.jenkins.io/debian-stable binary/ Release.gpg [833 B]
Hit:8 http://security.ubuntu.com/ubuntu focal-security InRelease
Get:9 https://pkg.jenkins.io/debian-stable binary/ Packages [23.2 kB]
Fetched 26.1 kB in 1s (42.5 kB/s)
Reading package lists... Done
peter@devops-demo:~$ sudo apt-get install jenkins
......
peter@devops-demo:~$ systemctl status jenkins.service
● jenkins.service - Jenkins Continuous Integration Server
     Loaded: loaded (/lib/systemd/system/jenkins.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-10-13 17:04:11 UTC; 20s ago
   Main PID: 13531 (java)
      Tasks: 51 (limit: 9508)
     Memory: 2.2G
     CGroup: /system.slice/jenkins.service
             └─13531 /usr/bin/java -Djava.awt.headless=true -jar /usr/share/java/jenkins.war --webroot=/var>
peter@devops-demo:~$

Jenkins設定並選擇安裝推薦的外掛

Jenkins設定並選擇安裝推薦的外掛

Jenkins建立第一個管理員使用者

Jenkins執行instance設定

開始使用Jenkins

開始使用Jenkins

在Jenkins上建立新的作業

在Jenkins上建立新的作業-Hello Pipelines

在Jenkins上建立新的作業-執行Hello Pipelines

在Jenkins上建立新的作業-執行Hello Pipelines

安裝Pipeline As YAML外掛

安裝Pipeline As YAML外掛

安裝Pipeline As YAML外掛

使用Pipeline As YAML外掛

執行使用Pipeline As YAML外掛的pipelines

GitLab CI/CD-安裝GitLab CE版本

root@devops-demo:/tmp# apt-get update
root@devops-demo:/tmp# apt-get install ca-certificates curl openssh-server postfix
root@devops-demo:~# cd /tmp/
root@devops-demo:/tmp# curl -LO https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6865  100  6865    0     0  30923      0 --:--:-- --:--:-- --:--:-- 30923
root@devops-demo:/tmp# bash script.deb.sh
......

The repository is setup! You can now install packages.
root@devops-demo:/tmp# apt-get install gitlab-ce
.......
Thank you for installing GitLab!
GitLab was unable to detect a valid hostname for your instance.
Please configure a URL for your GitLab instance by setting `external_url`
configuration in /etc/gitlab/gitlab.rb file.
Then, you can start your GitLab instance by running the following command:
  sudo gitlab-ctl reconfigure

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

Help us improve the installation experience, let us know how we did with a 1 minute survey:
https://gitlab.fra1.qualtrics.com/jfe/form/SV_6kVqZANThUQ1bZb?installation=omnibus&release=15-4

root@devops-demo:/tmp#

GitLab CI/CD-安裝GitLab CE版本(設定GitLab)

root@devops-demo:/tmp# vim /etc/gitlab/gitlab.rb
root@devops-demo:/tmp# cat /etc/gitlab/gitlab.rb | grep '^external_url'
external_url 'https://gitlab.peterli.website'
root@devops-demo:/tmp# cat /etc/gitlab/gitlab.rb | grep '^letsencrypt'
letsencrypt['contact_emails'] = ['peter279k@gmail.com'] # This should be an array of email addresses to add as contacts
root@devops-demo:/tmp# gitlab-ctl reconfigure
......
Running handlers:
[2022-10-13T18:18:24+00:00] INFO: Running report handlers
Running handlers complete
[2022-10-13T18:18:24+00:00] INFO: Report handlers complete
Infra Phase complete, 650/1720 resources updated in 05 minutes 14 seconds

Notes:
Default admin account has been configured with following details:
Username: root
Password: You didn't opt-in to print initial root password to STDOUT.
Password stored to /etc/gitlab/initial_root_password. This file will be cleaned up in first reconfigure run after 24 hours.

NOTE: Because these credentials might be present in your log files in plain text, it is highly recommended to reset the password following https://docs.gitlab.com/ee/security/reset_user_password.html#reset-your-root-password.

gitlab Reconfigured!
root@devops-demo:/tmp# cat /etc/gitlab/initial_root_password
# WARNING: This value is valid only in the following conditions
#          1. If provided manually (either via `GITLAB_ROOT_PASSWORD` environment variable or via `gitlab_rails['initial_root_password']` setting in `gitlab.rb`, it was provided before database was seeded for the first time (usually, the first reconfigure run).
#          2. Password hasn't been changed manually, either via UI or via command line.
#
#          If the password shown here doesn't work, you must reset the admin password following https://docs.gitlab.com/ee/security/reset_user_password.html#reset-your-root-password.

Password: l+ReQBz71nutZq7BPZ4nqD9eKpW472iWLk+30oHsfUI=

# NOTE: This file will be automatically deleted in the first reconfigure run after 24 hours.
root@devops-demo:/tmp#

GitLab CI/CD-瀏覽GitLab網址(https://gitlab.peterli.website)

GitLab CI/CD-瀏覽GitLab網址(建立專案)

GitLab CI/CD-瀏覽GitLab網址(建立Git專案)

root@devops-demo:/tmp# cd
root@devops-demo:~# mkdir hello-gitlab-ci
root@devops-demo:~# cd hello-gitlab-ci/
root@devops-demo:~/hello-gitlab-ci# vim .gitlab-ci.yml
root@devops-demo:~/hello-gitlab-ci# cat .gitlab-ci.yml
build-job:
  stage: build
  script:
    - echo "Hello, $GITLAB_USER_LOGIN!"

test-job1:
  stage: test
  script:
    - echo "This job tests something"

test-job2:
  stage: test
  script:
    - echo "This job tests something, but takes more time than test-job1."
    - echo "After the echo commands complete, it runs the sleep command for 20 seconds"
    - echo "which simulates a test that runs 20 seconds longer than test-job1"
    - sleep 20

deploy-prod:
  stage: deploy
  script:
    - echo "This job deploys something from the $CI_COMMIT_BRANCH branch."
  environment: production
root@devops-demo:~/hello-gitlab-ci# git init
Initialized empty Git repository in /root/hello-gitlab-ci/.git/
root@devops-demo:~/hello-gitlab-ci#

GitLab CI/CD-瀏覽GitLab網址(建立Git專案)

root@devops-demo:~/hello-gitlab-ci# git remote add origin https://gitlab.peterli.website/gitlab-instance-9f10964a/hello-gitlab-ci
root@devops-demo:~/hello-gitlab-ci# git config --global user.email peter279k@gmail.com
root@devops-demo:~/hello-gitlab-ci# git config --global user.name peter279k
root@devops-demo:~/hello-gitlab-ci# git add .
root@devops-demo:~/hello-gitlab-ci# git config --global core.editor vim
root@devops-demo:~/hello-gitlab-ci# git commit -a
[master (root-commit) fddb677] Initial commit
 1 file changed, 23 insertions(+)
 create mode 100644 .gitlab-ci.yml
root@devops-demo:~/hello-gitlab-ci# git push origin master
Username for 'https://gitlab.peterli.website': root
Password for 'https://root@gitlab.peterli.website':
warning: redirecting to https://gitlab.peterli.website/gitlab-instance-9f10964a/hello-gitlab-ci.git/
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 4 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 496 bytes | 496.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To https://gitlab.peterli.website/gitlab-instance-9f10964a/hello-gitlab-ci
 * [new branch]      master -> master
root@devops-demo:~/hello-gitlab-ci#

GitLab CI/CD-瀏覽GitLab網址(運行專案之GitLab CI/CD)

GitLab CI/CD-瀏覽GitLab網址(安裝Docker)

GitLab CI/CD-瀏覽GitLab網址(安裝GitLab Runner)

root@devops-demo:~# curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 48  414M   48  203M    0     0  10.6M      0  0:00:38  0:00:19  0:00:19 12.1M
 root@devops-demo:~# dpkg -i gitlab-runner_amd64.deb
Selecting previously unselected package gitlab-runner.
(Reading database ... 157514 files and directories currently installed.)
Preparing to unpack gitlab-runner_amd64.deb ...
Unpacking gitlab-runner (15.4.0) ...
......
15.4.0

Check and remove all unused containers (both dangling and unreferenced) including volumes.
------------------------------------------------------------------------------------------
Total reclaimed space: 0B
root@devops-demo:~# gitlab-runner status
Runtime platform                                    arch=amd64 os=linux pid=23296 revision=43b2dc3d version=15.4.0
gitlab-runner: Service is running

GitLab CI/CD-瀏覽GitLab網址(安裝GitLab Runner)

root@devops-demo:~/hello-gitlab-ci# gitlab-runner register
Runtime platform                                    arch=amd64 os=linux pid=24415 revision=43b2dc3d version=15.4.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.peterli.website/
Enter the registration token:
GR1348941Z1qeA8HkLxPvTaji52Zp
Enter a description for the runner:
[devops-demo]: hello-gitlab-ci
Enter tags for the runner (comma-separated):
hello-gitlab-ci
Enter optional maintenance note for the runner:

Registering runner... succeeded                     runner=GR1348941Z1qeA8Hk
Enter an executor: docker, docker-ssh, shell, docker+machine, docker-ssh+machine, kubernetes, custom, ssh, virtualbox, parallels:
docker
Enter the default Docker image (for example, ruby:2.7):
python:3.7
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"
root@devops-demo:~/hello-gitlab-ci#

GitLab CI/CD-瀏覽GitLab網址(安裝GitLab Runner)

Travis CI(雲端服務版本)

.travis.yml

language: python
python:
  - "2.7"
  - "3.4"
  - "3.5"
  - "3.6"      # current default Python on Travis CI
  - "3.7"
  - "3.8"
  - "3.9"
  - "3.9-dev"  # 3.9 development branch
  - "nightly"  # nightly build
# command to install dependencies
install:
  - pip install -r requirements.txt
# command to run tests
script:
  - pytest

Circle CI(雲端服務版本)

.circleci/config.yml

version: 2.1
orbs:
  python: circleci/python@2.0.3
  heroku: circleci/heroku@1.2.6

jobs:
  build_and_test: # this can be any name you choose
    executor: python/default
    steps:
      - checkout
      - python/install-packages:
          pkg-manager: pip
      - run:
          name: Run tests
          command: python -m pytest
      - persist_to_workspace:
          root: ~/project
          paths:
            - .

  deploy: # this can be any name you choose
    executor: python/default
    steps:
      - attach_workspace:
          at: ~/project
      - heroku/deploy-via-git:
          force: true # force push when pushing to the heroku remote, see: https://devcenter.heroku.com/articles/git

workflows:
  test_my_app:
    jobs:
      - build_and_test
      - deploy:
          requires:
            - build_and_test # only deploy if the build_and_test job has completed
          filters:
            branches:
              only: main # only deploy when on main

GitHub Workflow(雲端服務版本)

.github/workflows/your-ci-name.yml

name: Test Workflow
on: push
jobs:
  first-job:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Run Action
        # we can give directory path that contains action.yaml or repo address in username/repository_name@version format
        # in our case it would be nashmaniac/create-issue-action@v1.0 where v1.0 is the version of action
        # for now we will give directory path. The directory must contain action.yaml
        uses: ./
        # pass user input as arguments
        with:
          title: Build Failed
          token: ${{secrets.GITHUB_TOKEN}} # default token in GitHub Workflow
          assignees: ${{github.actor}} # actor who started the workflow
          labels: worflow-failed
          body: Workflow failed for commit ${{github.sha}}

總結比較

CI/CD工具或服務名稱 優點 缺點
Travis CI 快速、方便與好用 只有提供雲端服務
GitHub Workflow 與GitHub平台整合佳 只有提供雲端服務
CircleCI 方便與好用 只有提供雲端服務
GitLab CI/CD 與GitLab平台整合佳 安裝自架版本後會附加GitLab的專案託管平台
Jenknis 可自架、維護與管理 需要會寫平台自定義Pipelines語言
或是安裝外掛支援YAML檔案編寫CI/CD設定檔
有很多既有的功能是支援Java專案的
其他程式語言專案的測試環境要先自己架好
也可以使用Docker image建置測試環境

參考資料

  • https://www.digitalocean.com/community/tutorials/how-to-install-jenkins-on-ubuntu-20-04

  • https://www.jenkins.io/blog/2018/04/25/configuring-jenkins-pipeline-with-yaml-file

  • https://github.com/jenkinsci/pipeline-as-yaml-plugin/blob/master/src/test/resources/job/pipelineTestWithBranches.yml

  • https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-gitlab-on-ubuntu-18-04

  • https://docs.gitlab.com/runner/install/linux-manually.html

  • https://docs.docker.com/engine/install/ubuntu

  • https://docs.gitlab.com/ee/ci/examples

  • https://docs.gitlab.com/ee/ci/quick_start

  • https://docs.travis-ci.com/user/languages/python

  • https://circleci.com/docs/language-python

  • https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python

  • https://medium.com/@xthewiz/notify-jenkins-for-push-event-on-gitlab-18e2e4104827

  • https://www.cnblogs.com/panzi/p/10536654.html

CI/CD Tools Survey

By peter279k

CI/CD Tools Survey

CI/CD Tools Survey

  • 472