Full End-to-End Testing for Your HTML Email Workflows

https://slides.com/bahmutov/email-testing

Gleb Bahmutov

Sr Director of Engineering

our planet is in imminent danger

https://lizkeogh.com/2019/07/02/off-the-charts/

+3 degrees Celsius will be the end and we are close

https://www.ipcc.ch/assessment-report/ar6/

@bahmutov

survival is possible* but we need to act now

  • change your life
  • dump banks financing fossil projects
  • join an organization

@bahmutov

rebellion.global www.wedonthavetime.org 

www.sierraclub.org 350.org www.greenpeace.org 

www.sunrisemovement.org

Hello <% Name %>,

 

Today our company is happy to announce ...

Have you ever received an email like

A huge sale on all these items:

 

 

Do not miss this opportunity, it won't last!

Or like this one?

Agenda

  • An example application
  • Testing emails
    • Using local SMTP server
    • Using an external test  email account
    • Using 3rd party email service to send
  • Bonus: Visual testing

@bahmutov

🔈 Gleb Bahmutov PhD

🌎   Boston, USA

@bahmutov

gleb.dev

github.com/bahmutov

glebbahmutov.com/blog

C / C++ / C# / Java / CoffeeScript / JavaScript / Node / Angular / Vue / Cycle.js / functional programming / testing

www.youtube.com/glebbahmutov

www.mercari.com/careers

Example App

https://www.cypress.io/blog/2021/05/11/testing-html-emails-using-cypress/

https://github.com/bahmutov/cypress-email-example

Example App

https://www.cypress.io/blog/2021/05/11/testing-html-emails-using-cypress/

https://github.com/bahmutov/cypress-email-example

Example App

https://www.cypress.io/blog/2021/05/11/testing-html-emails-using-cypress/

https://github.com/bahmutov/cypress-email-example

click on the button

Example App

https://www.cypress.io/blog/2021/05/11/testing-html-emails-using-cypress/

https://github.com/bahmutov/cypress-email-example

should take you to localhost:3000/confirm

The conference app Attendify does this!

How Does Email Get To The User?

How An Email Gets To Your Inbox

1

2

3

4

Historical tip: SMTP was born in 1981

POP3 or IMAP

 to fetch emails

Sending Emails in Node

const nodemailer = require('nodemailer')

const host = process.env.SMTP_HOST || 'localhost'
const port = Number(process.env.SMTP_PORT || 7777)

const transporter = nodemailer.createTransport({
  host,
  port,
  secure: port === 456,
})

transporter.sendMail({
  from: '"Registration system" <reg@company.co>',
  to: email,
  subject: 'Confirmation code 1️⃣2️⃣3️⃣',
  text: 'Your confirmation code is 654agc',
  html: confirmationEmailHTML,
})

Your Own SMTP Server in Node

const ms = require('smtp-tester')

// starts the SMTP server at localhost:7777
const port = 7777
const mailServer = ms.init(port)
mailServer.bind((addr, id, email) => {
  // email.body is plain text
  // email.html is rich HTML
})

https://github.com/deitch/smtp-tester#readme

For Email Testing

SMTP_HOST=localhost npm start

For Email Testing

SMTP_HOST=localhost npm start
  • Need to control the browser
  • Access the smtp-tester to get the email
  • Test the HTML email in the browser

Todo:

Let's Test The 📧

50 people. Atlanta, Philly, Boston, NYC, the world

Fast, easy and reliable testing for anything that runs in a browser

$ npm install -D cypress

Cypress Workshop Tomorrow!

Cypress with SMTP tester flow

const ms = require('smtp-tester')

module.exports = (on, config) => {
  // starts the SMTP server at localhost:7777
  const port = 7777
  const mailServer = ms.init(port)

  let lastEmail = {}

  mailServer.bind((addr, id, email) => {
    // store the email by the receiver email
    lastEmail[email.headers.to] = {
      body: email.body,
      html: email.html,
    }
  })
  
  on('task', {
    getLastEmail(userEmail) {
      return lastEmail[userEmail] || null
    },
  })
}

cypress/plugin/index.js

The test can get the last email for the given user

describe('Email confirmation', () => {
  beforeEach(() => {
    cy.task('resetEmails')
  })

  it('sends an email with code', () => {
    cy.visit('/')
    cy.get('#name').type('Joe Bravo')
    cy.get('#email').type('joe@acme.io')
    cy.get('#company_size').select('3')
    cy.get('button[type=submit]').click()
    cy.location('pathname').should('equal', '/confirm')
    // by now the SMTP server has probably received the email
    cy.task('getLastEmail', 'joe@acme.io')
      .its('body') // check the plain email text
      .then(cy.wrap)
      // Tip: modern browsers supports named groups
      .invoke('match', /code is (?<code>\w+)/)
      // the confirmation code
      .its('groups.code')
      .should('be.a', 'string')
      .then((code) => {
        cy.get('#confirmation_code').type(code)
        cy.get('button[type=submit]').click()
        cy.get('[data-cy=incorrect-code]').should('not.exist')
        cy.get('[data-cy=confirmed-code]').should('be.visible')
      })
  })
})

cypress/integration/code-spec.js

Plaintext email, local SMTP test server

describe('Email confirmation', () => {
  beforeEach(() => {
    cy.task('resetEmails')
  })

  it('sends an HTML email', () => {
    cy.visit('/')
    cy.get('#name').type('Joe Bravo')
    cy.get('#email').type('joe@acme.io')
    cy.get('#company_size').select('3')
    cy.get('button[type=submit]').click()
    cy.location('pathname').should('equal', '/confirm')
    // by now the SMTP server has probably received the email
    cy.task('getLastEmail', 'joe@acme.io')
      .its('html') // check the HTML email text
      .then((html) => {
        // load the email in the current test browser
        cy.document().invoke('write', html)
      })
    cy.contains('654agc')
      .should('be.visible')
      // I have added small wait to make sure the video shows the email
      // otherwise it passes way too quickly!
      .wait(2000)
    cy.contains('Confirm registration').click()
    cy.location('pathname').should('equal', '/confirm')
    cy.get('#confirmation_code').type('654agc')
    cy.get('button[type=submit]').click()
    cy.get('[data-cy=incorrect-code]').should('not.exist')
    cy.get('[data-cy=confirmed-code]').should('be.visible')
  })
})

cypress/integration/html-email-spec.js

HTML email, local SMTP test server

✅ Our hand-rolled email HTML is testable     Tip: use https://maizzle.com/

// by now the SMTP server has probably received the email
cy.task('getLastEmail', 'joe@acme.io')
  .its('html') // check the HTML email text
  .then((html) => {
    cy.document().invoke('write', html)
  })

cy.screenshot('2-the-email')

cypress/integration/spec.js

Tip: save screenshots at each milestone

Image saved during the test using cy.screenshot

Agenda

  • An example application
  • Testing emails
    • Using local SMTP server
    • Using an external test  email account
    • Using 3rd party email service to send
  • Bonus: Visual testing

@bahmutov

What if we are using company / commercial SMTP server?

(if we cannot point Backend API at the smtp-tester)

The App

const nodemailer = require('nodemailer')

const host = process.env.SENDGRID_HOST
const port = Number(process.env.SENDGRID_PORT)
const secure = port === 465
const auth = {
  user: process.env.SENDGRID_USER,
  pass: process.env.SENDGRID_PASSWORD,
}
const transporter = nodemailer.createTransport({
  host, port, secure, auth,
})

src/emailer.js

Ethereal email accounts

Ethereal emails

https://ethereal.email/

https://github.com/bahmutov/cypress-ethereal-email-example

https://www.cypress.io/blog/2021/05/24/full-testing-of-html-emails-using-ethereal-accounts/

// use Nodemailer to get an Ethereal email inbox
// https://nodemailer.com/about/
const nodemailer = require('nodemailer')
const testAccount = await nodemailer.createTestAccount()

cypress/plugin/index.js

https://nodemailer.com/about/

The plugin file

const makeEmailAccount = require('./email-account')

module.exports = async (on) => {
  const emailAccount = await makeEmailAccount()

  on('task', {
    getUserEmail() {
      return emailAccount.email
    },

    getLastEmail() {
      return emailAccount.getLastEmail()
    },
  })
}

cypress/plugin/index.js

The test file

it('sends confirmation code', () => {
  const userName = 'Joe Bravo'

  cy.visit('/')
  cy.get('#name').type(userName)
  cy.get('#email').type(userEmail)
  cy.get('#company_size').select('3')
  cy.get('button[type=submit]').click()

  // hmm, the email might be delayed
  cy.task('getLastEmail')
    .its('html')
    ...
})

cypress/integration/spec.js

The email does not arrive in time to the email inbox

Try fetching the email until it arrives

// https://github.com/bahmutov/cypress-recurse
const { recurse } = require('cypress-recurse')

it('sends confirmation code', () => {
  ...
  cy.get('button[type=submit]').click()
  
  // retry fetching the email
  recurse(
    () => cy.task('getLastEmail'), // Cypress commands to retry
    Cypress._.isObject, // keep retrying until the task returns an object
    {
      timeout: 60000, // retry up to 1 minute
      delay: 5000, // wait 5 seconds between attempts
    },
  )
    .its('html')
    ...
})

cypress/integration/spec.js

https://github.com/bahmutov/cypress-recurse

Retry fetching an email

Dynamic Email Templates

Sending 1000s emails that work on most devices is complicated...

Design the email using 3rd party editor

Sending an email

const sgClient = require('@sendgrid/client')
sgClient.setApiKey(process.env.SENDGRID_API_KEY)

/**
  * Sends an email by using SendGrid dynamic design template
  * @see Docs https://sendgrid.api-docs.io/v3.0/mail-send/v3-mail-send
  */
async sendTemplateEmail({ from, template_id, dynamic_template_data, to }) {
  const body = {
    from: {
      email: from || process.env.SENDGRID_FROM,
      name: 'Confirmation system',
    },
    personalizations: [
      {
        to: [{ email: to }],
        dynamic_template_data,
      },
    ],
    template_id,
  }

  const request = {
    method: 'POST',
    url: 'v3/mail/send',
    body,
  }
  const [response] = await sgClient.request(request)

  return response

src/emailer.js

Sending an email

const initEmailer = require('../../src/emailer')
const emailer = await initEmailer()
await emailer.sendTemplateEmail({
  to: email,
  // the ID of the dynamic template we have designed
  template_id: 'd-9b1e07a7d2994b14ae394026a6ccc997',
  dynamic_template_data: {
    code: confirmationCode,
    username: name,
    confirm_url: 'http://localhost:3000/confirm',
  },
})

pages/api/register.js

An email that accidentally used a wrong field in the greeting line

https://github.com/bahmutov/cypress-sendgrid-visual-testing-example

https://www.cypress.io/blog/2021/06/11/visual-testing-for-emails-sent-using-3rd-party-dynamic-templates/

The test file

// retry fetching the email
recurse(
  () => cy.task('getLastEmail'), // Cypress commands to retry
  Cypress._.isObject, // keep retrying until the task returns an object
  {
    timeout: 60000, // retry up to 1 minute
    delay: 5000, // wait 5 seconds between attempts
  },
)
  .its('html')
  .then((html) => {
    cy.document({ log: false })
      .invoke({ log: false }, 'write', html)
  })
cy.log('**email has the user name**')
cy.contains(`Dear ${userName},`).should('be.visible')

cypress/integration/spec.js

✅ SendGrid generates emails compatible with most email clients

⛔️ The HTML markup makes it hard to select particular fields in the email

cy.contains('a', 'Enter the confirmation code')
  .should('be.visible')
  .invoke('text')
  .then((text) => Cypress._.last(text.split(' ')))
  .then((code) => {
  	// click on the button
  	// enter the code
  })

Assert using text only

cypress/integration/spec.js

// https://docs.percy.io/docs/cypress
require('@percy/cypress')
describe('Email confirmation', () => {
  it('sends confirmation code', () => {
    const userName = 'Joe Bravo'

    cy.visit('/')
    cy.get('#name').type(userName)
    cy.get('#email').type(userEmail)
    cy.get('#company_size').select('3')
    // cy.screenshot('1 - registration screen')
    cy.percySnapshot('1 - registration screen')
    cy.get('button[type=submit]').click()
    ...
  })
})

💡 Visual Testing

cypress/integration/spec.js

https://on.cypress.io/visual-testing

// https://docs.percy.io/docs/cypress
require('@percy/cypress')
describe('Email confirmation', () => {
  it('sends confirmation code', () => {
    const userName = 'Joe Bravo'

    cy.visit('/')
    cy.get('#name').type(userName)
    cy.get('#email').type(userEmail)
    cy.get('#company_size').select('3')
    // cy.screenshot('1 - registration screen')
    cy.percySnapshot('1 - registration screen')
    cy.get('button[type=submit]').click()
    ...
  })
})

💡 Visual Testing

cypress/integration/spec.js

// https://docs.percy.io/docs/cypress
require('@percy/cypress')
describe('Email confirmation', () => {
  it('sends confirmation code', () => {
    const userName = 'Joe Bravo'

    cy.visit('/')
    cy.get('#name').type(userName)
    cy.get('#email').type(userEmail)
    cy.get('#company_size').select('3')
    cy.percySnapshot('1 - registration screen')
    cy.get('button[type=submit]').click()
    ...
  })
})

💡 Visual Testing

cypress/integration/spec.js

cy.task('getLastEmail')
  .its('html')
  .then((html) => {
    cy.document().invoke('write', html)
  })
cy.percySnapshot('2 - email')

💡 Visual Testing: the email

cypress/integration/spec.js

cy.task('getLastEmail')
  .its('html')
  .then((html) => {
    cy.document().invoke('write', html)
  })
cy.percySnapshot('2 - email')

💡 Visual Testing: the email

cypress/integration/spec.js

Changing confirmation code breaks the image comparison

Different data

Different HTML

Different screenshots

😞 Cannot use image diffing with dynamic data

Solution: confirm, then replace random data before taking the snapshot

cypress/integration/spec.js

.then((code) => {
  cy.log(`**confirm the code ${code} works**`)
  expect(code, 'confirmation code')
    .to.be.a('string')
    .and.have.length.gt(5)
  // replace the dynamic confirmation code with constant text
  // since we already validated the code
  cy.get('@codeLink').invoke(
    'text',
    'Enter the confirmation code abc1234',
  )
  cy.contains('strong', new RegExp('^' + code + '$'))
    .invoke('text', 'abc1234')
  cy.percySnapshot('2 - email')
})

If you need to verify the actual code: https://glebbahmutov.com/blog/verify-phone-part-one/

🎉 Consistent data generates a matching image for visual testing

If you need to verify the actual code: https://glebbahmutov.com/blog/verify-phone-part-one/

Final Thoughts

Email mistakes make you look unprofessional

Final Thoughts

Use a temporary email account to text the full email flow

Final Thoughts

Use visual testing to verify text + style

Thank You 👏

🖥  slides.com/bahmutov/email-testing

🦉  @bahmutov

Gleb Bahmutov

Sr Director of Engineering

Full End-to-End Testing for Your HTML Email Workflows

By Gleb Bahmutov

Full End-to-End Testing for Your HTML Email Workflows

Every time a new user registers for your service, your application probably sends a confirmation email.How does that email look in the user's browser? How does it look on a mobile screen? And most importantly: does it work? In this presentation, I will show the full end-to-end open-source testing procedure for validating HTML emails. We will test the email functionality, accessibility, and visual look and style to ensure that our users are not silently dropping out due to a broken email subsystem. Presented at InfobipShift 2021, video at https://youtu.be/igj8OQhY0Jg

  • 4,348

More from Gleb Bahmutov