Appium

TensorFlow Brain

and its

PROBLEMStatement

elementByClassname("android.v4.ViewPage")

elementByXpath(".//*[@text='cart']")

elementById("cart")

elementByClassChain(".//XCUITypeWindow//*//XCUIClassName")

Locator Strategies

Demo

Get all leaf DOM elements from the application under test, including each element’s coordinates and size.

Get a screenshot of the application.

Crop the screenshot into lots of little screenshots

Run each element’s cropped screenshot through the neural network.

How does

it WORK?

The model is a neural network that takes a 224✕224 pixel 3-channel color image as input and outputs 105 numbers that represent the confidence or probability that the input image matches each of the 105 labels that the model knows about (cart, edit, twitter, etc.

The model is MobileNet v1,2 which is a pre-trained model released by Google AI. MobileNet is a “deep”, “convolutional” neural network.

Selector Strategy
CSS/XPath	When your app’s DOM changes	Easy to understand why Easy to fix
Test.ai’s image classifier	When your app’s icons change visibly When the model changes (that is, when you upgrade the Appium plugin)	Hard to understand why Hard to fix

Appium

TensorFlow Brain

PROBLEMStatement

Locator Strategies

AI Locator

Demo

Appium AI Plugin

How does

it WORK?

MODEL WORK?

Neural Network

RECOGNIZE APP ICONS

Demo

PROS

CONS

OBJECT DETECTION TECHNIQUE

Demo

Future Improvements

Appium and AI

Appium and AI

Sai Krishna

Appium and AI

More from Sai Krishna