Full End-to-End Testing for Your HTML Email Workflows

https://slides.com/bahmutov/email-testing

Gleb Bahmutov

Sr Director of Engineering

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end and we are close

https://www.ipcc.ch/assessment-report/ar6/

@bahmutov

survival is possible* but we need to act now

  • change your life
  • dump banks financing fossil projects
  • join an organization

@bahmutov

rebellion.global www.wedonthavetime.org 

www.sierraclub.org 350.org www.greenpeace.org 

www.sunrisemovement.org

Hello <% Name %>,

 

Today our company is happy to announce ...

Have you ever received an email like

A huge sale on all these items:

 

 

Do not miss this opportunity, it won't last!

Or like this one?

Agenda

  • An example application
  • Testing emails
    • Using local SMTP server
    • Using an external test  email account
    • Using 3rd party email service to send
  • Bonus: Visual testing

@bahmutov

🔈 Gleb Bahmutov PhD

🌎   Boston, USA

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

www.mercari.com/careers

Example App

https://www.cypress.io/blog/2021/05/11/testing-html-emails-using-cypress/

https://github.com/bahmutov/cypress-email-example

Example App

https://www.cypress.io/blog/2021/05/11/testing-html-emails-using-cypress/

https://github.com/bahmutov/cypress-email-example

Example App

https://www.cypress.io/blog/2021/05/11/testing-html-emails-using-cypress/

https://github.com/bahmutov/cypress-email-example

click on the button

Example App

https://www.cypress.io/blog/2021/05/11/testing-html-emails-using-cypress/

https://github.com/bahmutov/cypress-email-example

should take you to localhost:3000/confirm

The conference app Attendify does this!

How Does Email Get To The User?

How An Email Gets To Your Inbox

1

2

3

4

Historical tip: SMTP was born in 1981

POP3 or IMAP

 to fetch emails

Sending Emails in Node

const nodemailer = require('nodemailer')

const host = process.env.SMTP_HOST || 'localhost'
const port = Number(process.env.SMTP_PORT || 7777)

const transporter = nodemailer.createTransport({
  host,
  port,
  secure: port === 456,
})

transporter.sendMail({
  from: '"Registration system" <reg@company.co>',
  to: email,
  subject: 'Confirmation code 1️⃣2️⃣3️⃣',
  text: 'Your confirmation code is 654agc',
  html: confirmationEmailHTML,
})

Your Own SMTP Server in Node

const ms = require('smtp-tester')

// starts the SMTP server at localhost:7777
const port = 7777
const mailServer = ms.init(port)
mailServer.bind((addr, id, email) => {
  // email.body is plain text
  // email.html is rich HTML
})

https://github.com/deitch/smtp-tester#readme

For Email Testing

SMTP_HOST=localhost npm start

For Email Testing

SMTP_HOST=localhost npm start
  • Need to control the browser
  • Access the smtp-tester to get the email
  • Test the HTML email in the browser

Todo:

Let's Test The 📧

50 people. Atlanta, Philly, Boston, NYC, the world

Fast, easy and reliable testing for anything that runs in a browser

$ npm install -D cypress

Cypress Workshop Tomorrow!

Cypress with SMTP tester flow

const ms = require('smtp-tester')

module.exports = (on, config) => {
  // starts the SMTP server at localhost:7777
  const port = 7777
  const mailServer = ms.init(port)

  let lastEmail = {}

  mailServer.bind((addr, id, email) => {
    // store the email by the receiver email
    lastEmail[email.headers.to] = {
      body: email.body,
      html: email.html,
    }
  })
  
  on('task', {
    getLastEmail(userEmail) {
      return lastEmail[userEmail] || null
    },
  })
}

cypress/plugin/index.js

The test can get the last email for the given user

describe('Email confirmation', () => {
  beforeEach(() => {
    cy.task('resetEmails')
  })

  it('sends an email with code', () => {
    cy.visit('/')
    cy.get('#name').type('Joe Bravo')
    cy.get('#email').type('joe@acme.io')
    cy.get('#company_size').select('3')
    cy.get('button[type=submit]').click()
    cy.location('pathname').should('equal', '/confirm')
    // by now the SMTP server has probably received the email
    cy.task('getLastEmail', 'joe@acme.io')
      .its('body') // check the plain email text
      .then(cy.wrap)
      // Tip: modern browsers supports named groups
      .invoke('match', /code is (?<code>\w+)/)
      // the confirmation code
      .its('groups.code')
      .should('be.a', 'string')
      .then((code) => {
        cy.get('#confirmation_code').type(code)
        cy.get('button[type=submit]').click()
        cy.get('[data-cy=incorrect-code]').should('not.exist')
        cy.get('[data-cy=confirmed-code]').should('be.visible')
      })
  })
})

cypress/integration/code-spec.js

Plaintext email, local SMTP test server

describe('Email confirmation', () => {
  beforeEach(() => {
    cy.task('resetEmails')
  })

  it('sends an HTML email', () => {
    cy.visit('/')
    cy.get('#name').type('Joe Bravo')
    cy.get('#email').type('joe@acme.io')
    cy.get('#company_size').select('3')
    cy.get('button[type=submit]').click()
    cy.location('pathname').should('equal', '/confirm')
    // by now the SMTP server has probably received the email
    cy.task('getLastEmail', 'joe@acme.io')
      .its('html') // check the HTML email text
      .then((html) => {
        // load the email in the current test browser
        cy.document().invoke('write', html)
      })
    cy.contains('654agc')
      .should('be.visible')
      // I have added small wait to make sure the video shows the email
      // otherwise it passes way too quickly!
      .wait(2000)
    cy.contains('Confirm registration').click()
    cy.location('pathname').should('equal', '/confirm')
    cy.get('#confirmation_code').type('654agc')
    cy.get('button[type=submit]').click()
    cy.get('[data-cy=incorrect-code]').should('not.exist')
    cy.get('[data-cy=confirmed-code]').should('be.visible')
  })
})

cypress/integration/html-email-spec.js

HTML email, local SMTP test server

✅ Our hand-rolled email HTML is testable     Tip: use https://maizzle.com/

// by now the SMTP server has probably received the email
cy.task('getLastEmail', 'joe@acme.io')
  .its('html') // check the HTML email text
  .then((html) => {
    cy.document().invoke('write', html)
  })

cy.screenshot('2-the-email')

cypress/integration/spec.js

Tip: save screenshots at each milestone

Image saved during the test using cy.screenshot

Agenda

  • An example application
  • Testing emails
    • Using local SMTP server
    • Using an external test  email account
    • Using 3rd party email service to send
  • Bonus: Visual testing

@bahmutov

What if we are using company / commercial SMTP server?

(if we cannot point Backend API at the smtp-tester)

The App

const nodemailer = require('nodemailer')

const host = process.env.SENDGRID_HOST
const port = Number(process.env.SENDGRID_PORT)
const secure = port === 465
const auth = {
  user: process.env.SENDGRID_USER,
  pass: process.env.SENDGRID_PASSWORD,
}
const transporter = nodemailer.createTransport({
  host, port, secure, auth,
})

src/emailer.js

Ethereal email accounts

Ethereal emails

https://ethereal.email/

https://github.com/bahmutov/cypress-ethereal-email-example

https://www.cypress.io/blog/2021/05/24/full-testing-of-html-emails-using-ethereal-accounts/

// use Nodemailer to get an Ethereal email inbox
// https://nodemailer.com/about/
const nodemailer = require('nodemailer')
const testAccount = await nodemailer.createTestAccount()

cypress/plugin/index.js

https://nodemailer.com/about/

The plugin file

const makeEmailAccount = require('./email-account')

module.exports = async (on) => {
  const emailAccount = await makeEmailAccount()

  on('task', {
    getUserEmail() {
      return emailAccount.email
    },

    getLastEmail() {
      return emailAccount.getLastEmail()
    },
  })
}

cypress/plugin/index.js

The test file

it('sends confirmation code', () => {
  const userName = 'Joe Bravo'

  cy.visit('/')
  cy.get('#name').type(userName)
  cy.get('#email').type(userEmail)
  cy.get('#company_size').select('3')
  cy.get('button[type=submit]').click()

  // hmm, the email might be delayed
  cy.task('getLastEmail')
    .its('html')
    ...
})

cypress/integration/spec.js

The email does not arrive in time to the email inbox

Try fetching the email until it arrives

// https://github.com/bahmutov/cypress-recurse
const { recurse } = require('cypress-recurse')

it('sends confirmation code', () => {
  ...
  cy.get('button[type=submit]').click()
  
  // retry fetching the email
  recurse(
    () => cy.task('getLastEmail'), // Cypress commands to retry
    Cypress._.isObject, // keep retrying until the task returns an object
    {
      timeout: 60000, // retry up to 1 minute
      delay: 5000, // wait 5 seconds between attempts
    },
  )
    .its('html')
    ...
})

cypress/integration/spec.js

https://github.com/bahmutov/cypress-recurse

Retry fetching an email

Dynamic Email Templates

Sending 1000s emails that work on most devices is complicated...

Design the email using 3rd party editor

Sending an email

const sgClient = require('@sendgrid/client')
sgClient.setApiKey(process.env.SENDGRID_API_KEY)

/**
  * Sends an email by using SendGrid dynamic design template
  * @see Docs https://sendgrid.api-docs.io/v3.0/mail-send/v3-mail-send
  */
async sendTemplateEmail({ from, template_id, dynamic_template_data, to }) {
  const body = {
    from: {
      email: from || process.env.SENDGRID_FROM,
      name: 'Confirmation system',
    },
    personalizations: [
      {
        to: [{ email: to }],
        dynamic_template_data,
      },
    ],
    template_id,
  }

  const request = {
    method: 'POST',
    url: 'v3/mail/send',
    body,
  }
  const [response] = await sgClient.request(request)

  return response

src/emailer.js

Sending an email

const initEmailer = require('../../src/emailer')
const emailer = await initEmailer()
await emailer.sendTemplateEmail({
  to: email,
  // the ID of the dynamic template we have designed
  template_id: 'd-9b1e07a7d2994b14ae394026a6ccc997',
  dynamic_template_data: {
    code: confirmationCode,
    username: name,
    confirm_url: 'http://localhost:3000/confirm',
  },
})

pages/api/register.js

An email that accidentally used a wrong field in the greeting line

https://github.com/bahmutov/cypress-sendgrid-visual-testing-example

https://www.cypress.io/blog/2021/06/11/visual-testing-for-emails-sent-using-3rd-party-dynamic-templates/

The test file

// retry fetching the email
recurse(
  () => cy.task('getLastEmail'), // Cypress commands to retry
  Cypress._.isObject, // keep retrying until the task returns an object
  {
    timeout: 60000, // retry up to 1 minute
    delay: 5000, // wait 5 seconds between attempts
  },
)
  .its('html')
  .then((html) => {
    cy.document({ log: false })
      .invoke({ log: false }, 'write', html)
  })
cy.log('**email has the user name**')
cy.contains(`Dear ${userName},`).should('be.visible')

cypress/integration/spec.js

✅ SendGrid generates emails compatible with most email clients

⛔️ The HTML markup makes it hard to select particular fields in the email

cy.contains('a', 'Enter the confirmation code')
  .should('be.visible')
  .invoke('text')
  .then((text) => Cypress._.last(text.split(' ')))
  .then((code) => {
  	// click on the button
  	// enter the code
  })

Assert using text only

cypress/integration/spec.js

// https://docs.percy.io/docs/cypress
require('@percy/cypress')
describe('Email confirmation', () => {
  it('sends confirmation code', () => {
    const userName = 'Joe Bravo'

    cy.visit('/')
    cy.get('#name').type(userName)
    cy.get('#email').type(userEmail)
    cy.get('#company_size').select('3')
    // cy.screenshot('1 - registration screen')
    cy.percySnapshot('1 - registration screen')
    cy.get('button[type=submit]').click()
    ...
  })
})

💡 Visual Testing

cypress/integration/spec.js

https://on.cypress.io/visual-testing

// https://docs.percy.io/docs/cypress
require('@percy/cypress')
describe('Email confirmation', () => {
  it('sends confirmation code', () => {
    const userName = 'Joe Bravo'

    cy.visit('/')
    cy.get('#name').type(userName)
    cy.get('#email').type(userEmail)
    cy.get('#company_size').select('3')
    // cy.screenshot('1 - registration screen')
    cy.percySnapshot('1 - registration screen')
    cy.get('button[type=submit]').click()
    ...
  })
})

💡 Visual Testing

cypress/integration/spec.js

// https://docs.percy.io/docs/cypress
require('@percy/cypress')
describe('Email confirmation', () => {
  it('sends confirmation code', () => {
    const userName = 'Joe Bravo'

    cy.visit('/')
    cy.get('#name').type(userName)
    cy.get('#email').type(userEmail)
    cy.get('#company_size').select('3')
    cy.percySnapshot('1 - registration screen')
    cy.get('button[type=submit]').click()
    ...
  })
})

💡 Visual Testing

cypress/integration/spec.js

cy.task('getLastEmail')
  .its('html')
  .then((html) => {
    cy.document().invoke('write', html)
  })
cy.percySnapshot('2 - email')

💡 Visual Testing: the email

cypress/integration/spec.js

cy.task('getLastEmail')
  .its('html')
  .then((html) => {
    cy.document().invoke('write', html)
  })
cy.percySnapshot('2 - email')

💡 Visual Testing: the email

cypress/integration/spec.js

Changing confirmation code breaks the image comparison

Different data

Different HTML

Different screenshots

😞 Cannot use image diffing with dynamic data

Solution: confirm, then replace random data before taking the snapshot

cypress/integration/spec.js

.then((code) => {
  cy.log(`**confirm the code ${code} works**`)
  expect(code, 'confirmation code')
    .to.be.a('string')
    .and.have.length.gt(5)
  // replace the dynamic confirmation code with constant text
  // since we already validated the code
  cy.get('@codeLink').invoke(
    'text',
    'Enter the confirmation code abc1234',
  )
  cy.contains('strong', new RegExp('^' + code + '$'))
    .invoke('text', 'abc1234')
  cy.percySnapshot('2 - email')
})

If you need to verify the actual code: https://glebbahmutov.com/blog/verify-phone-part-one/

🎉 Consistent data generates a matching image for visual testing

If you need to verify the actual code: https://glebbahmutov.com/blog/verify-phone-part-one/

Final Thoughts

Email mistakes make you look unprofessional

Final Thoughts

Use a temporary email account to text the full email flow

Final Thoughts

Use visual testing to verify text + style

Thank You 👏

🖥  slides.com/bahmutov/email-testing

🦉  @bahmutov

Gleb Bahmutov

Sr Director of Engineering