E2E Test Framework

Introduction

E2E Test

  E2E Test is a software quality assurance methodology that checks if an App performs as designed on all levels and across all subsystems. The scope of E2E Test encompasses the App in its entirety, as well as its integration with external interfaces and outside Apps.

E2E Test

For example, a simplified E2E Test of an email App might involve

  • Logging in to the application
  • Accessing the inbox
  • Opening and closing the mailbox
  • Composing, forwarding or replying to email
  • Checking the sent items
  • Logging out of the application

Test Runner

A test runner is the library or tool that picks up unit tests and settings, then executes them and writes the test results to the console or log files.

Headless Browser

A headless browser is a web browser without a graphical user interface.

Advantages of Headless Browser

  • Faster
  • More stable

 

Headless browsers avoid draw operations, which handle rendering of the UI and their various pixels on the screen.

Usages of Headless Browser

  • Test automation in modern web Apps.
  • Scraping web sites for data ( Especially  for AJAX pages )
  • Automating interaction of web pages
  • Run testing in servers without screen

Some Headless Browsers

Google Chrome: support headless mode after version 59

Firefox: support headless mode since version 55 in linux, since version 56 in Windows and macOS

PhantomJS: a headless web browser using Webkit layout engine for rendering web pages and JavaScriptCore for executing scripted tests.

PC Automation Tools

NPM downloads in past 2 years

These automation tools are used to test Web App built on browser.

  • Selenium
  • PhantomJS
  • Puppeteer
  • ...

PC Web Automation Tools

PhantomJS Home Page

http://phantomjs.org/

We can divide them into two classes:

  • Selenium
  • Puppeteer
  • PhantomJS

Cross broswer support

 Focus one browser

PC Web Automation Tools

Selenium  VS  PhantomJS and Puppeteer

Selenium PhantomJS & Puppeteer
Java, Python, Javascript... Javascript
Cross browser support Only support one browser
Without browser enginee With browser enginee
Cannot forbid UI Support headless mode
Slower Faster
Large community Smaller community
PhantomJS Puppeteer
Suspended Developing
Old javascript syntax All syntax supported by Chrome
Lower performance Higher performance
Only headless mode Headless and common mode
Not real browser environment Real browser environment

PhantomJS  VS  Puppeteer

PhantomJS  VS  Puppeteer

Development status

Developer of PhantomJs noticed that

PhantomJS development is suspended until further notice.

That means:

  • PhantomJS may no longer support newest javascript syntax, for example, async/await
  • Browser may fail to test Apps with newest javascript features

PhantomJS  VS  Puppeteer

Performance

PhantomJS  VS  Puppeteer

Performance

Mobile Automation Tools

Mobile E2E Automation Tool

Macaca Appium
PC, Android, IOS, Hybrid, Electron PC, Android, IOS, Hybrid
Java, Python, Javascript, ... Java, Python, Javascript, ...
Inspector, Recorder Appium Desktop
Android API > 17 All
Detailed document for JS Detailed document
Smaller community Large community
More useful tools Less tools

Mobile E2E Automation Tool

Recorder and Inspector

Macaca Recorder&Inspector  Appium
Appium Desktop
Can only save script to Javascript Can save script to Java, Python...
Inspector cannot show coordinate of cursor Real-time coordinate display
Recoder can set assertions Cannot set assertions
Recorder supports click, select, input, swipe

Mobile E2E Automation Tool

Problem

Zoom API is broken when testing in Hybrid App

For Appium, we can use multiAction to simulate zoom operation:

action1 = TouchAction()
action2 = TouchAction()
action1.long_press(x=centerX, y=centerY)\
    .move_to(x=centerX + 50, y=centerY + 50)\
    .wait(100).release()
action2.long_press(x=centerX, y=centerY)\
    .move_to(x=centerX - 50, y=centerY - 50)\
    .wait(100).release()
m_action.add(action1, action2)
m_action.perform()

Mobile E2E Automation Tool

Problem

For Macaca, no API supports MultiAction.

This problem is still difficult to solve.

Other problem with Macaca:

  • Lack detailed documents for Python and other languages
  • Many tools only support Javascript
  • Difficult to search solutions

Finally, I choose Appium for automating.

Study Case

Background

Testing Object: Hybrid App

Goal: Automate E2E testing in Android Emulator

Environment:

    PC: Ubuntu 16.04 with python 3.5

    Emulator:

        Nexus 5X API 23

        Nexus 5X API 24

        Nexus 9 API 24

Appium

Introduction

Install Server

Install Appium Server by npm:

npm install -g appium

If download appium-chromedriver timeout,

try to use proxy to download it.

Add parameter after command:

--chromedriver_cdnurl=http://npm.taobao.org/mirrors/chromedriver

Check Denpendencies

Install appium-doctor by npm:

npm install -g appium-doctor

Run appium-doctor command, supplying the --ios or --android flags to check dependencies

After all dependencies are installed, run appium command to start server

Install Python Client

Install Python client by pip:

pip install Appium-Python-Client

import appium Python client in python file:

from appium import webdriver

Connect to Server

Start your Android emulator

Set the desired_capability

desired_capabilities = {
    'platformName': 'Android',
    'platformVersion': 'X.X',
    'deviceName': 'you-emulator-name',
    'app': 'you-app.apk'
}

Text

Connect to Server

driver = webdriver.Remote('http://localhost:PORT/wd/hub', desired_capabilities)

PORT is your Appium Server Port, default 4723

Connect to Server

For Hybrid Apps, chromedriver is necessary,

add chromedriverExecutable to desired_capabilities:

desired_capabilities = {
    'platformName': 'Android',
    'platformVersion': 'X.X',
    'deviceName': 'you-emulator-name',
    'app': 'you-app.apk',
    'chromedriverExecutable': 'your-chromedriver-path'
}

Text

Chromedriver version should match the WebView version of emulator. You can see version of WebView in chrome://inspect

Text

Appium Desktop

Download release package from

https://github.com/appium/appium-desktop/releases/tag/v1.6.3

For Ubuntu, download .AppIamge file, then make it executable and execute it:

chmod a+x appium-desktop-1.6.3-x86_64.AppImage
./appium-desktop-1.6.3-x86_64.AppImage

Text

default configure is ok

click to start server

click the button to create connection

set desired capabilities 

click to start session

touch actions

other tools

App XML source

Chrome Dev Tool ( for Hybrid App )

Open Chrome, enter chrome://inspect

WebView version

click start inspect

Javascript source

Write Testcase

Assume that testing App has XML:

<android.widget.FrameLayout>
    <android.widget.LinearLayout>
        <android.widget.FrameLayout resource-id="android:id/content">
            <android.webkit.WebView>
                <android.webkit.WebView content-desc="hybrid-base">
                    <android.view.View content-desc="Welcome">
                    <android.widget.EditText content-desc="Phone Number">
                    <android.view.View content-desc="Verify Code">
                    <android.widget.EditText content-desc="6 Digits">
                    <android.widget.Button content-desc="GET SMS">
                    <android.widget.Button content-desc="NEXT">
    <android.view.View resource-id="android:id/statusBarBackground">
    <android.view.View resource-id="android:id/navigationBarBackground">

How to locate an element

1. Use Absolute XPATH

# return element whose text is 'Welcome'
driver.find_element_by_xpath('/hierarchy/android.widget.FrameLayout/ \
android.widget.LinearLayout/android.widget.FrameLayout/android.webkit.WebView/ \
android.webkit.WebView/android.view.View[@content-desc="Welcome"]')
  • Easy
  • Absolute XPATH will be different among devices
  • Should modify after add/delete some elements
  • Too long

How to locate an element

2. Give every element an ID

# Assume element have id 'test'

driver.find_element_by_id('test')

# or
driver.find_element_by_xpath('//*[@resource-id="test"]')
  • Changing XML needs less modification
  • Can exactly locate one element
  • Need to change code of testing App

How to locate an element

3. Use Relative XPATH

# return element whose text is 'Welcome'
xpath = '(//*[@content-desc="hybrid-base"]/android.view.View)[1]'
driver.find_element_by_xpath(xpath)

# this works well here, but is not recommended
driver.find_element_by_xpath('//*[@content-desc="Welcome"]')

# notes:
# first index in W3C xpath syntax is 1
# compare these two xpaths
xpath1 = '//*[@content-desc="hybrid-base"]/android.view.View[1]'
xpath2 = '(//*[@content-desc="hybrid-base"]/android.view.View)[1]'
  • No need to add ID to every element
  • Changing XML still need modification
  • Some elements are difficult to design relative XPATH

How to locate an element

4. Switch to WebView context ( Only for Hybrid App )

# print all context
print(driver.contexts)
# switch to WebView context
driver.switch_to.context("WEBVIEW_1")
# assume the element has class 'loginHeader-mainText'
driver.find_element_by_class_name("loginHeader-mainText")
# switch back to Native APP context
driver.switch_to.context("NATIVE_APP")
  • Can use class name instead of ID
  • Can use css selector
  • CSS selector is difficult to distinguish some elements

Notes when locating elements

Remeber waiting element first

# waiting element
def wait_element_by_class_name(class_name, timeout, driver):
    # count time 
    start_time = time.monotonic()
    while True:
        # check for every 1 s
        time.sleep(1)
        try:
            driver.find_element_by_class_name(class_name)
            break
        except Exception:
            # if timeout, throw exception and exit
            if time.monotonic() - start_time > timeout:
                raise Exception("Timeout when finding element: " + class_name)
            continue
    return driver.find_element_by_class_name(class_name)

When you use find_element_by_class_name when XPATH is not exist ( page is still loading ), it will throw exception.

Waiting for other conditions

Waiting for clickable:

# waiting element to be clickable
def wait_clickable_by_class_name(class_name, timeout, driver):
    start_time = time.monotonic()
    # wait element is avaliable
    while True:
        time.sleep(1)
        try:
            driver.find_element_by_class_name(class_name)
            break
        except Exception:
            if time.monotonic() - start_time > timeout:
                raise Exception("Timeout when finding element: " + class_name)
            continue
    # wait element to be clickable
    while True:
        time.sleep(1)
        try:
            # EC.element_to_be_clickable throw exception when element is not clickable
            EC.element_to_be_clickable((By.CLASS_NAME, class_name))
            break
        except Exception:
            if time.monotonic() - start_time > timeout:
                raise Exception("Timeout when waiting element to be clickable: " + class_name)
            continue
    return driver.find_element_by_class_name(class_name)

Do not forget to wait for element first, otherwise you may catch an exception.

Waiting for other conditions

Waiting for staleness:

# waiting element to be dropped from XML
def wait_element_staleness_by_class_webview(class_name, timeout, driver):
    element = None
    # make sure that the element is in XML, otherwise stop waiting
    try:
        element = driver.find_element_by_class_name(class_name)
    except Exception as e:
        print(e)
        return
    finally:
        if element is None:
            return
        # `wait` will wait until EC.staleness_of return true
        wait = WebDriverWait(driver, timeout)
        wait.until(EC.staleness_of(element))

This is a little different from waiting for clickable. You should check if element has been dropped from XML.

Use Appium unicode keyboard

Add desired_capability:

'unicodeKeyboard': 'true'
'resetKeyboard': 'true'

Use Appium unicode keyboard can avoid exception when calling hide_keyboard after keyboard displaying.

When testing in real devices, sogou keyboard can also cause input mistakes.

Automatic permissions agreement 

Add desired_capability:

'autoGrantPermissions': 'true'

Automatic permissions agreement avoid manual clicking to agree permissions when first installing App.

Touch Actions

Zoom:

action1 = TouchAction()
action2 = TouchAction()
action1.long_press(x=centerX, y=centerY)\
    .move_to(x=centerX + 50, y=centerY + 50)\
    .wait(100).release()
action2.long_press(x=centerX, y=centerY)\
    .move_to(x=centerX - 50, y=centerY - 50)\
    .wait(100).release()
m_action = MultiAction(driver, element)
m_action.add(action1, action2)
m_action.perform()

wait(100) is used to control moving time, if it is too short, zoom operation will have no effect.

Touch Actions

Notes:

If touch actions have no effect, check source code of page and comfirm attributes of the element. These settings will forbid touch actions:

  • CSS attribute touch-action of the element is none
  • XML attribute scrollable of the element is false
  • ...

Parallel testing

Create Appium Server Instance for every emulator:

appium -p $PORT -bp $BOOTSTRAP_PORT --chromedriver-port $CHROMEDRIVER_PORT -U $UDID

Every instance should have different

  • port
  • bootstrap port
  • chromedriver port ( Hybrid App )
  • udid

Parallel testing

Different desired_capability for devices:

'udid': 'emulator-xxxx',
'systemPort': 'xxxx'

udid and systemPort is necessary. systemPort should be different among devices and values from 8200 to 8299. If testing Hybrid App, matched chromedriver is needed.

Every device should be connected to the server set to its udid

E2E Test Framework

Pytest & Python unittest

It help to execute our testcases and print result or write log to files.

Use setUp/tearDown of unittest.TestCase to set up and close connection to Appium Server.

Run command to execute testing:

pytest test
pytest test/testcase1.py

Pytest & Python unittest

Note that execution order in a python file is not the order that you define testcases.

  • Do not define testcase that depends on last defined testcase.
  • Related testing functions should be execute in one testcase.

Mocha

Mocha is a JavaScript test framework running on Node.js and in the browser, making asynchronous testing simple and fun.

Recommend working with package Power Assert

For running on Node.js, environment is different from that in browser, which is a trouble in Hybrid App testing.

Jest

Different from Mocha, you can select two environment in Jest: jsdom and node.

Node environment is a node like environment while jsdom environment providing a browser-like environment.

So when testing React Apps or Vue Apps, it is a good choice.

Extra

PhantomJS  VS  Puppeteer

Syntax Support

const puppeteer = require('puppeteer')

(async () => {
    const browser = await puppeteer.launch()
    let page = await browser.newPage()
    await page.goto('https://cn.bing.com/search?q=test', {
        waitUntil: 'domcontentloaded'
    })
    let searchElements = await page.$$eval('ol#b_results > li.b_algo > h2 > a', nodes => nodes.map(e => e.innerHTML))
    await browser.close()
})()

Codes in puppeteer:

PhantomJS  VS  Puppeteer

Syntax Support

var page = require('webpage').create()

page.open('https://cn.bing.com/search?q=test', function (status) {
    if (status === 'success') {
        var searchResults = page.evaluate(function () {
            var titles = document.querySelectorAll('ol#b_results > li.b_algo > h2 > a')
            var titleTexts = []
            for(var i = 0; i < titles.length; i += 1){
                titleTexts.push(titles[i].innerHTML)
            }
            return titleTexts
        })
    }
    phantom.exit()
})

Codes in phantomJS:

PhantomJS  VS  Puppeteer

Debug

Puppeteer can switch to common mode:

const browser = await puppeteer.launch({headless: false})

or use slow mode:

const browser = await puppeteer.launch({
    headless: false,
    slowMo: 250 // slow down by 250ms
})

PhantomJS  VS  Puppeteer

Debug

or capture console output:

page.on('console', msg => console.log('PAGE LOG:', msg.text()));

await page.evaluate(() => console.log(`url is ${location.href}`));

PhantomJS  VS  Puppeteer

Debug

PhantomJS can debug with chrome:

phantomjs --remote-debugger-port=9000 test.js

then open chrome

goto http://localhost:9000

choose link file://test.js

enter Web Inspector Interface

input in console and execute:

__run()

PhantomJS  VS  Puppeteer

Debug

or you can use autorun:

phantomjs --remote-debugger-port=9000 --remote-debugger-autorun=yes test.js

your test.js will be automatically executed

or capture console output:

var webPage = require('webpage')
var page = webPage.create()

page.onConsoleMessage = function (msg, lineNum, sourceId) {
    console.log('CONSOLE: ' + msg + ' (from line #' + lineNum + ' in "' + sourceId + '")')
}

E2E Test Framework

By Haili Zhang

Source: Fan Shi

E2E Test Framework

  • 770

More from Haili Zhang