Gleb Bahmutov PRO
JavaScript ninja, image processing expert, software quality fanatic
VP of Engineering, Cypress.io
Solutions Engineer, CircleCI
C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional
(these slides)
15 people. Atlanta, Philly, Boston, LA, Chicago, NYC
Fast, easy and reliable testing for anything that runs in a browser
E2E
integration
unit
E2E
integration
unit
Really important to users
Really important to developers
$ npm install -D cypressit('adds 2 todos', () => {
  cy.visit('http://localhost:3000')
  cy.get('.new-todo')
    .type('learn testing{enter}')
    .type('be cool{enter}')
  cy.get('.todo-list li')
    .should('have.length', 2)
})$ npx cypress open$ npx cypress open$ npx cypress open$ npx cypress runMaking it easy for users is not easy
version: 2
jobs:
  test:
    docker:
      - image: cypress/base:10
    steps:
      - checkout
      # restore folders with npm dependencies and Cypress binary
      - restore_cache:
          keys:
            - cache-{{ checksum "package.json" }}
      # install npm dependencies and Cypress binary
      # if they were cached, this step is super quick
      - run:
          name: Install dependencies
          command: npm ci
      - run: npm run cy:verify
      # save npm dependencies and Cypress binary for future runs
      - save_cache:
          key: cache-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      # start server before starting tests
      - run:
          command: npm start
          background: true
      - run: npm run e2e:record
workflows:
  version: 2
  build:
    jobs:
      - testDocker image
typical CI config file
version: 2
jobs:
  test:
    docker:
      - image: cypress/base:10
    steps:
      - checkout
      # restore folders with npm dependencies and Cypress binary
      - restore_cache:
          keys:
            - cache-{{ checksum "package.json" }}
      # install npm dependencies and Cypress binary
      # if they were cached, this step is super quick
      - run:
          name: Install dependencies
          command: npm ci
      - run: npm run cy:verify
      # save npm dependencies and Cypress binary for future runs
      - save_cache:
          key: cache-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      # start server before starting tests
      - run:
          command: npm start
          background: true
      - run: npm run e2e:record
workflows:
  version: 2
  build:
    jobs:
      - testDocker image
Caching
Caching
typical CI config file
version: 2
jobs:
  test:
    docker:
      - image: cypress/base:10
    steps:
      - checkout
      # restore folders with npm dependencies and Cypress binary
      - restore_cache:
          keys:
            - cache-{{ checksum "package.json" }}
      # install npm dependencies and Cypress binary
      # if they were cached, this step is super quick
      - run:
          name: Install dependencies
          command: npm ci
      - run: npm run cy:verify
      # save npm dependencies and Cypress binary for future runs
      - save_cache:
          key: cache-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      # start server before starting tests
      - run:
          command: npm start
          background: true
      - run: npm run e2e:record
workflows:
  version: 2
  build:
    jobs:
      - testDocker image
Caching
Caching
Install
typical CI config file
version: 2
jobs:
  test:
    docker:
      - image: cypress/base:10
    steps:
      - checkout
      # restore folders with npm dependencies and Cypress binary
      - restore_cache:
          keys:
            - cache-{{ checksum "package.json" }}
      # install npm dependencies and Cypress binary
      # if they were cached, this step is super quick
      - run:
          name: Install dependencies
          command: npm ci
      - run: npm run cy:verify
      # save npm dependencies and Cypress binary for future runs
      - save_cache:
          key: cache-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      # start server before starting tests
      - run:
          command: npm start
          background: true
      - run: npm run e2e:record
workflows:
  version: 2
  build:
    jobs:
      - testDocker image
Caching
Caching
maybe start app
typical CI config file
Install
version: 2
jobs:
  test:
    docker:
      - image: cypress/base:10
    steps:
      - checkout
      # restore folders with npm dependencies and Cypress binary
      - restore_cache:
          keys:
            - cache-{{ checksum "package.json" }}
      # install npm dependencies and Cypress binary
      # if they were cached, this step is super quick
      - run:
          name: Install dependencies
          command: npm ci
      - run: npm run cy:verify
      # save npm dependencies and Cypress binary for future runs
      - save_cache:
          key: cache-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      # start server before starting tests
      - run:
          command: npm start
          background: true
      - run: npm run e2e:record
workflows:
  version: 2
  build:
    jobs:
      - testDocker image
Caching
Caching
typical CI config file
Install
run tests
maybe start app
I have 100s of tests ...
$ npx cypress run --record --parallel
Cypress v3.1.0
Spin N CI machines and
Text
| # machines | run duration | time savings | 
|---|---|---|
| 1 | 22:50 | ~ | 
| 2 | 11:47 | 48% | 
| 3 | 7:51 | 65% | 
| 4 | 5:56 | 74% | 
| 6 | 3:50 | 83% | 
| 8 | 3:00 | 87% | 
| 10 | 2:19 | 90% | 
10 machines = 10x speed up
defaults: &defaults
  working_directory: ~/app
  docker:
    - image: cypress/browsers:chrome67
version: 2
jobs:
  build:
    <<: *defaults
    steps:
      - checkout
      # find compatible cache from previous build,
      # it should have same dependencies installed from package.json checksum
      - restore_cache:
          keys:
            - cache-{{ .Branch }}-{{ checksum "package.json" }}
      - run:
          name: Install Dependencies
          command: npm ci
      # run verify and then save cache.
      # this ensures that the Cypress verified status is cached too
      - run: npm run cy:verify
      # save new cache folder if needed
      - save_cache:
          key: cache-{{ .Branch }}-{{ checksum "package.json" }}
          paths:
            - ~/.npm
            - ~/.cache
      - run: npm run types
      - run: npm run stop-only
      # all other test jobs will run AFTER this build job finishes
      # to avoid reinstalling dependencies, we persist the source folder "app"
      # and the Cypress binary to workspace, which is the fastest way
      # for Circle jobs to pass files
      - persist_to_workspace:
          root: ~/
          paths:
            - app
            - .cache/Cypress
  4x-electron:
    <<: *defaults
    # tell CircleCI to execute this job on 4 machines simultaneously
    parallelism: 4
    steps:
      - attach_workspace:
          at: ~/
      - run:
          command: npm start
          background: true
      # runs Cypress test in load balancing (parallel) mode
      # and groups them in Cypress Dashboard under name "4x-electron"
      - run: npm run e2e:record -- --parallel --group $CIRCLE_JOB
workflows:
  version: 2
  # this workflow has 4 jobs to show case Cypress --parallel and --group flags
  # "build" installs NPM dependencies so other jobs don't have to
  #   └ "1x-electron" runs all specs just like Cypress pre-3.1.0 runs them
  #   └ "4x-electron" job load balances all specs across 4 CI machines
  #   └ "2x-chrome" load balances all specs across 2 CI machines and uses Chrome browser
  build_and_test:
    jobs:
      - build
      # this group "4x-electron" will load balance all specs
      # across 4 CI machines
      - 4x-electron:
          requires:
            - buildParallel config is ... more complicated
typical CI config file
- install + run jobs
- workspace
- parallel flags
# first, install Cypress, then run all tests (in parallel)
stages:
  - build
  - test
# to cache both npm modules and Cypress binary we use environment variables
# to point at the folders we can list as paths in "cache" job settings
variables:
  npm_config_cache: "$CI_PROJECT_DIR/.npm"
  CYPRESS_CACHE_FOLDER: "$CI_PROJECT_DIR/cache/Cypress"
# cache using branch name
# https://gitlab.com/help/ci/caching/index.md
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - .npm
    - cache/Cypress
    - node_modules
# this job installs NPM dependencies and Cypress
install:
  image: cypress/base:10
  stage: build
  script:
    - npm ci
    - $(npm bin)/print-env CI
    - npm run cy:verify
# all jobs that actually run tests can use the same definition
.job_template: &job
  image: cypress/base:10
  stage: test
  script:
    # print CI environment variables for reference
    - $(npm bin)/print-env CI
    # start the server in the background
    - npm run start:ci &
    # run Cypress test in load balancing mode, pass id to tie jobs together
    - npm run e2e:record -- --parallel --ci-build-id $CI_PIPELINE_ID --group electrons
# actual job definitions
# all steps are the same, they come from the template above
electrons-1:
  <<: *job
electrons-2:
  <<: *job
electrons-3:
  <<: *job
electrons-4:
  <<: *job
electrons-5:
  <<: *jobpipeline {
  agent {
    // this image provides everything needed to run Cypress
    docker {
      image 'cypress/base:10'
    }
  }
  stages {
    // first stage installs node dependencies and Cypress binary
    stage('build') {
      steps {
        // there a few default environment variables on Jenkins
        // on local Jenkins machine (assuming port 8080) see
        // http://localhost:8080/pipeline-syntax/globals#env
        echo "Running build ${env.BUILD_ID} on ${env.JENKINS_URL}"
        sh 'npm ci'
        sh 'npm run cy:verify'
      }
    }
    stage('start local server') {
      steps {
        // start local server in the background
        // we will shut it down in "post" command block
        sh 'nohup npm start &'
      }
    }
    // this tage runs end-to-end tests, and each agent uses the workspace
    // from the previous stage
    stage('cypress parallel tests') {
      environment {
        // we will be recordint test results and video on Cypress dashboard
        // to record we need to set an environment variable
        // we can load the record key variable from credentials store
        // see https://jenkins.io/doc/book/using/using-credentials/
        CYPRESS_RECORD_KEY = credentials('cypress-example-kitchensink-record-key')
        // because parallel steps share the workspace they might race to delete
        // screenshots and videos folders. Tell Cypress not to delete these folders
        CYPRESS_trashAssetsBeforeRuns = 'false'
      }
      // https://jenkins.io/doc/book/pipeline/syntax/#parallel
      parallel {
        // start several test jobs in parallel, and they all
        // will use Cypress Dashboard to load balance any found spec files
        stage('tester A') {
          steps {
            echo "Running build ${env.BUILD_ID}"
            sh "npm run e2e:record:parallel"
          }
        }
        // second tester runs the same command
        stage('tester B') {
          steps {
            echo "Running build ${env.BUILD_ID}"
            sh "npm run e2e:record:parallel"
          }
        }
      }
    }
  }
  post {
    // shutdown the server running in the background
    always {
      echo 'Stopping local server'
      sh 'pkill -f http-server'
    }
  }
}language: node_js
node_js:
  # Node 10.3+ includes npm@6 which has good "npm ci" command
  - 10.8
cache:
  # cache both npm modules and Cypress binary
  directories:
    - ~/.npm
    - ~/.cache
  override:
    - npm ci
    - npm run cy:verify
defaults: &defaults
  script:
    #   ## print all Travis environment variables for debugging
    - $(npm bin)/print-env TRAVIS
    - npm start -- --silent &
    - npm run cy:run -- --record --parallel --group $STAGE_NAME
    # after all tests finish running we need
    # to kill all background jobs (like "npm start &")
    - kill $(jobs -p) || true
jobs:
  include:
    # we have multiple jobs to execute using just a single stage
    # but we can pass group name via environment variable to Cypress test runner
    - stage: test
      env:
        - STAGE_NAME=1x-electron
      <<: *defaults
    # run tests in parallel by including several test jobs with same name variable
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaults
    - stage: test
      env:
        - STAGE_NAME=4x-electron
      <<: *defaultsReusable CI configuration code
version: 2.1
orbs:
  cypress: cypress-io/cypress@1.1.0
workflows:
  build:
    jobs:
      - cypress/runcircle.yml
version: 2.1
orbs:
  cypress: cypress-io/cypress@1.1.0
workflows:
  build:
    jobs:
      - cypress/runVersioned namespaced package
circle.yml
version: 2.1
orbs:
  cypress: cypress-io/cypress@1.1.0
workflows:
  build:
    jobs:
      - cypress/run"run" job is defined in cypress orb
circle.yml
version: 2.1
orbs:
  cypress: cypress-io/cypress@1.1.0
workflows:
  build:
    jobs:
      - cypress/run:
          record: truecircle.yml
version: 2.1
orbs:
  cypress: cypress-io/cypress@1.1.0
workflows:
  build:
    jobs:
      - cypress/install
      - cypress/run:
          requires:
            - cypress/install
          record: true
          parallel: true
          parallelism: 10Parallel run scenario
circle.yml
version: 2.1
orbs:
  cypress: cypress-io/cypress@1.1.0
workflows:
  build:
    jobs:
      - cypress/install
      - cypress/run:
          requires:
            - cypress/install
          record: true
          parallel: true
          parallelism: 10Job parameters
circle.yml
version: 2.1
orbs:
  cypress: cypress-io/cypress@1.1.0
workflows:
  build:
    jobs:
      - cypress/install
      - cypress/run:
          requires:
            - cypress/install
          record: true
          parallel: true
          parallelism: 10Install on 1 machine
circle.yml
version: 2.1
orbs:
  cypress: cypress-io/cypress@1.1.0
workflows:
  build:
    jobs:
      - cypress/install
      - cypress/run:
          requires:
            - cypress/install
          record: true
          parallel: true
          parallelism: 10Run tests across 10 Circle machines
circle.yml
version: 2.1
orbs:
  cypress: cypress-io/cypress@1.1.0
workflows:
  build:
    jobs:
      - cypress/runcircle.yml
$ circleci config process circle.yml$ circleci config process .circleci/config.yml
# Orb 'cypress-io/cypress@1.1.0' resolved to 'cypress-io/cypress@1.1.0'
version: 2
jobs:
  cypress/run:
    docker:
    - image: cypress/base:10
    parallelism: 1
    steps:
    - checkout
    - restore_cache:
        keys:
        - cache-{{ .Branch }}-{{ checksum "package.json" }}
    - run:
        name: Npm CI
        command: npm ci
    - run:
        command: npx cypress verify
    - save_cache:
        key: cache-{{ .Branch }}-{{ checksum "package.json" }}
        paths:
        - ~/.npm
        - ~/.cache
    - persist_to_workspace:
        root: ~/
        paths:
        - project
        - .cache/Cypress
    - attach_workspace:
        at: ~/
    - run:
        name: Run Cypress tests
        command: 'npx cypress run'
workflows:
  build:
    jobs:
    - cypress/run
  version: 2Main Orbs page: circleci.com/orbs
Docs: circleci.com/docs/2.0/orb-intro
Orbs blog posts: circleci.com/blog/tag/orbs
Orbs Registry: circleci.com/orbs/registry
GitHub topic: github.com/topics/circleci-orbs
Ask the community: discuss.circleci.com/c/orbs
Open a ticket: support.circleci.com/hc/requests/new
Become a CircleCI Technology Partner: circleci.com/partners/technology
💻 github.com/cypress-io/circleci-orb
🐦 @cypress_io @bahmutov @circleci
📽 slides.com/bahmutov/circleci-cypress-orb
By Gleb Bahmutov
Cypress makes setting up, writing, running and debugging tests for web applications easy with their all-in-one testing framework, assertion library, with mocking and stubbing. With the newly released CircleCI Orbs feature, you can quickly set up Cypress on CircleCI to run all of your tests on a single machine or on several machines in parallel to cut down the testing time. Video at https://www.youtube.com/watch?time_continue=59&v=J-xbNtKgXfY
JavaScript ninja, image processing expert, software quality fanatic