E2E Test is a software quality assurance methodology that checks if an App performs as designed on all levels and across all subsystems. The scope of E2E Test encompasses the App in its entirety, as well as its integration with external interfaces and outside Apps.
For example, a simplified E2E Test of an email App might involve
A test runner is the library or tool that picks up unit tests and settings, then executes them and writes the test results to the console or log files.
A headless browser is a web browser without a graphical user interface.
Headless browsers avoid draw operations, which handle rendering of the UI and their various pixels on the screen.
Google Chrome: support headless mode after version 59
Firefox: support headless mode since version 55 in linux, since version 56 in Windows and macOS
PhantomJS: a headless web browser using Webkit layout engine for rendering web pages and JavaScriptCore for executing scripted tests.
These automation tools are used to test Web App built on browser.
http://phantomjs.org/
We can divide them into two classes:
Cross broswer support
Focus one browser
Selenium | PhantomJS & Puppeteer |
---|---|
Java, Python, Javascript... | Javascript |
Cross browser support | Only support one browser |
Without browser enginee | With browser enginee |
Cannot forbid UI | Support headless mode |
Slower | Faster |
Large community | Smaller community |
PhantomJS | Puppeteer |
---|---|
Suspended | Developing |
Old javascript syntax | All syntax supported by Chrome |
Lower performance | Higher performance |
Only headless mode | Headless and common mode |
Not real browser environment | Real browser environment |
Development status
Developer of PhantomJs noticed that
PhantomJS development is suspended until further notice.
That means:
Performance
Performance
Macaca | Appium |
---|---|
PC, Android, IOS, Hybrid, Electron | PC, Android, IOS, Hybrid |
Java, Python, Javascript, ... | Java, Python, Javascript, ... |
Inspector, Recorder | Appium Desktop |
Android API > 17 | All |
Detailed document for JS | Detailed document |
Smaller community | Large community |
More useful tools | Less tools |
Recorder and Inspector
Macaca Recorder&Inspector | Appium Appium Desktop |
---|---|
Can only save script to Javascript | Can save script to Java, Python... |
Inspector cannot show coordinate of cursor | Real-time coordinate display |
Recoder can set assertions | Cannot set assertions |
Recorder supports click, select, input, swipe |
Problem
Zoom API is broken when testing in Hybrid App
For Appium, we can use multiAction to simulate zoom operation:
action1 = TouchAction()
action2 = TouchAction()
action1.long_press(x=centerX, y=centerY)\
.move_to(x=centerX + 50, y=centerY + 50)\
.wait(100).release()
action2.long_press(x=centerX, y=centerY)\
.move_to(x=centerX - 50, y=centerY - 50)\
.wait(100).release()
m_action.add(action1, action2)
m_action.perform()
Problem
For Macaca, no API supports MultiAction.
This problem is still difficult to solve.
Other problem with Macaca:
Finally, I choose Appium for automating.
Testing Object: Hybrid App
Goal: Automate E2E testing in Android Emulator
Environment:
PC: Ubuntu 16.04 with python 3.5
Emulator:
Nexus 5X API 23
Nexus 5X API 24
Nexus 9 API 24
Install Appium Server by npm:
npm install -g appium
If download appium-chromedriver timeout,
try to use proxy to download it.
Add parameter after command:
--chromedriver_cdnurl=http://npm.taobao.org/mirrors/chromedriver
Install appium-doctor by npm:
npm install -g appium-doctor
Run appium-doctor command, supplying the --ios or --android flags to check dependencies
After all dependencies are installed, run appium command to start server
Install Python client by pip:
pip install Appium-Python-Client
import appium Python client in python file:
from appium import webdriver
Start your Android emulator
Set the desired_capability
desired_capabilities = {
'platformName': 'Android',
'platformVersion': 'X.X',
'deviceName': 'you-emulator-name',
'app': 'you-app.apk'
}
Text
Connect to Server
driver = webdriver.Remote('http://localhost:PORT/wd/hub', desired_capabilities)
PORT is your Appium Server Port, default 4723
For Hybrid Apps, chromedriver is necessary,
add chromedriverExecutable to desired_capabilities:
desired_capabilities = {
'platformName': 'Android',
'platformVersion': 'X.X',
'deviceName': 'you-emulator-name',
'app': 'you-app.apk',
'chromedriverExecutable': 'your-chromedriver-path'
}
Text
Chromedriver version should match the WebView version of emulator. You can see version of WebView in chrome://inspect
Text
Download release package from
https://github.com/appium/appium-desktop/releases/tag/v1.6.3
For Ubuntu, download .AppIamge file, then make it executable and execute it:
chmod a+x appium-desktop-1.6.3-x86_64.AppImage
./appium-desktop-1.6.3-x86_64.AppImage
Text
default configure is ok
click to start server
click the button to create connection
set desired capabilities
click to start session
touch actions
other tools
App XML source
Open Chrome, enter chrome://inspect
WebView version
click start inspect
Javascript source
Assume that testing App has XML:
<android.widget.FrameLayout>
<android.widget.LinearLayout>
<android.widget.FrameLayout resource-id="android:id/content">
<android.webkit.WebView>
<android.webkit.WebView content-desc="hybrid-base">
<android.view.View content-desc="Welcome">
<android.widget.EditText content-desc="Phone Number">
<android.view.View content-desc="Verify Code">
<android.widget.EditText content-desc="6 Digits">
<android.widget.Button content-desc="GET SMS">
<android.widget.Button content-desc="NEXT">
<android.view.View resource-id="android:id/statusBarBackground">
<android.view.View resource-id="android:id/navigationBarBackground">
1. Use Absolute XPATH
# return element whose text is 'Welcome'
driver.find_element_by_xpath('/hierarchy/android.widget.FrameLayout/ \
android.widget.LinearLayout/android.widget.FrameLayout/android.webkit.WebView/ \
android.webkit.WebView/android.view.View[@content-desc="Welcome"]')
2. Give every element an ID
# Assume element have id 'test'
driver.find_element_by_id('test')
# or
driver.find_element_by_xpath('//*[@resource-id="test"]')
3. Use Relative XPATH
# return element whose text is 'Welcome'
xpath = '(//*[@content-desc="hybrid-base"]/android.view.View)[1]'
driver.find_element_by_xpath(xpath)
# this works well here, but is not recommended
driver.find_element_by_xpath('//*[@content-desc="Welcome"]')
# notes:
# first index in W3C xpath syntax is 1
# compare these two xpaths
xpath1 = '//*[@content-desc="hybrid-base"]/android.view.View[1]'
xpath2 = '(//*[@content-desc="hybrid-base"]/android.view.View)[1]'
4. Switch to WebView context ( Only for Hybrid App )
# print all context
print(driver.contexts)
# switch to WebView context
driver.switch_to.context("WEBVIEW_1")
# assume the element has class 'loginHeader-mainText'
driver.find_element_by_class_name("loginHeader-mainText")
# switch back to Native APP context
driver.switch_to.context("NATIVE_APP")
Remeber waiting element first
# waiting element
def wait_element_by_class_name(class_name, timeout, driver):
# count time
start_time = time.monotonic()
while True:
# check for every 1 s
time.sleep(1)
try:
driver.find_element_by_class_name(class_name)
break
except Exception:
# if timeout, throw exception and exit
if time.monotonic() - start_time > timeout:
raise Exception("Timeout when finding element: " + class_name)
continue
return driver.find_element_by_class_name(class_name)
When you use find_element_by_class_name when XPATH is not exist ( page is still loading ), it will throw exception.
Waiting for clickable:
# waiting element to be clickable
def wait_clickable_by_class_name(class_name, timeout, driver):
start_time = time.monotonic()
# wait element is avaliable
while True:
time.sleep(1)
try:
driver.find_element_by_class_name(class_name)
break
except Exception:
if time.monotonic() - start_time > timeout:
raise Exception("Timeout when finding element: " + class_name)
continue
# wait element to be clickable
while True:
time.sleep(1)
try:
# EC.element_to_be_clickable throw exception when element is not clickable
EC.element_to_be_clickable((By.CLASS_NAME, class_name))
break
except Exception:
if time.monotonic() - start_time > timeout:
raise Exception("Timeout when waiting element to be clickable: " + class_name)
continue
return driver.find_element_by_class_name(class_name)
Do not forget to wait for element first, otherwise you may catch an exception.
Waiting for staleness:
# waiting element to be dropped from XML
def wait_element_staleness_by_class_webview(class_name, timeout, driver):
element = None
# make sure that the element is in XML, otherwise stop waiting
try:
element = driver.find_element_by_class_name(class_name)
except Exception as e:
print(e)
return
finally:
if element is None:
return
# `wait` will wait until EC.staleness_of return true
wait = WebDriverWait(driver, timeout)
wait.until(EC.staleness_of(element))
This is a little different from waiting for clickable. You should check if element has been dropped from XML.
Add desired_capability:
'unicodeKeyboard': 'true'
'resetKeyboard': 'true'
Use Appium unicode keyboard can avoid exception when calling hide_keyboard after keyboard displaying.
When testing in real devices, sogou keyboard can also cause input mistakes.
Add desired_capability:
'autoGrantPermissions': 'true'
Automatic permissions agreement avoid manual clicking to agree permissions when first installing App.
Zoom:
action1 = TouchAction()
action2 = TouchAction()
action1.long_press(x=centerX, y=centerY)\
.move_to(x=centerX + 50, y=centerY + 50)\
.wait(100).release()
action2.long_press(x=centerX, y=centerY)\
.move_to(x=centerX - 50, y=centerY - 50)\
.wait(100).release()
m_action = MultiAction(driver, element)
m_action.add(action1, action2)
m_action.perform()
wait(100) is used to control moving time, if it is too short, zoom operation will have no effect.
Notes:
If touch actions have no effect, check source code of page and comfirm attributes of the element. These settings will forbid touch actions:
Create Appium Server Instance for every emulator:
appium -p $PORT -bp $BOOTSTRAP_PORT --chromedriver-port $CHROMEDRIVER_PORT -U $UDID
Every instance should have different
Different desired_capability for devices:
'udid': 'emulator-xxxx',
'systemPort': 'xxxx'
udid and systemPort is necessary. systemPort should be different among devices and values from 8200 to 8299. If testing Hybrid App, matched chromedriver is needed.
Every device should be connected to the server set to its udid
It help to execute our testcases and print result or write log to files.
Use setUp/tearDown of unittest.TestCase to set up and close connection to Appium Server.
Run command to execute testing:
pytest test
pytest test/testcase1.py
Note that execution order in a python file is not the order that you define testcases.
Mocha is a JavaScript test framework running on Node.js and in the browser, making asynchronous testing simple and fun.
Recommend working with package Power Assert
For running on Node.js, environment is different from that in browser, which is a trouble in Hybrid App testing.
Different from Mocha, you can select two environment in Jest: jsdom and node.
Node environment is a node like environment while jsdom environment providing a browser-like environment.
So when testing React Apps or Vue Apps, it is a good choice.
Syntax Support
const puppeteer = require('puppeteer')
(async () => {
const browser = await puppeteer.launch()
let page = await browser.newPage()
await page.goto('https://cn.bing.com/search?q=test', {
waitUntil: 'domcontentloaded'
})
let searchElements = await page.$$eval('ol#b_results > li.b_algo > h2 > a', nodes => nodes.map(e => e.innerHTML))
await browser.close()
})()
Codes in puppeteer:
Syntax Support
var page = require('webpage').create()
page.open('https://cn.bing.com/search?q=test', function (status) {
if (status === 'success') {
var searchResults = page.evaluate(function () {
var titles = document.querySelectorAll('ol#b_results > li.b_algo > h2 > a')
var titleTexts = []
for(var i = 0; i < titles.length; i += 1){
titleTexts.push(titles[i].innerHTML)
}
return titleTexts
})
}
phantom.exit()
})
Codes in phantomJS:
Debug
Puppeteer can switch to common mode:
const browser = await puppeteer.launch({headless: false})
or use slow mode:
const browser = await puppeteer.launch({
headless: false,
slowMo: 250 // slow down by 250ms
})
Debug
or capture console output:
page.on('console', msg => console.log('PAGE LOG:', msg.text()));
await page.evaluate(() => console.log(`url is ${location.href}`));
Debug
PhantomJS can debug with chrome:
phantomjs --remote-debugger-port=9000 test.js
then open chrome
goto http://localhost:9000
choose link file://test.js
enter Web Inspector Interface
input in console and execute:
__run()
Debug
or you can use autorun:
phantomjs --remote-debugger-port=9000 --remote-debugger-autorun=yes test.js
your test.js will be automatically executed
or capture console output:
var webPage = require('webpage')
var page = webPage.create()
page.onConsoleMessage = function (msg, lineNum, sourceId) {
console.log('CONSOLE: ' + msg + ' (from line #' + lineNum + ' in "' + sourceId + '")')
}