Building Behat Tests Like a Pro
Who Are We?
We offer trusted digital content, adaptive algorithms, and teaching support tools to make it easier for all teachers to differentiate their instruction so every child performs better.
The team
Mike
Lead developer
Dan
Senior Developer
Mathieu
Senior developer
Philippe
Developer (Main QA)
Challenges & Solutions
An SPA is Disconnected
From it's PHP Backend
- Behat works inside PHP (Process #1)
- An SPA is in a browser (Process #2)
- PHP app (backend) ignores it is in testing mode unless:
- header
- query param
- or other permanent change exists
Problems
- Injecting headers or query params (hard)
- Use a .testing file or a persistent flag somewhere to identify change of environment
Solutions
// If the app_environment is null, use the environment variable
if ($environment == null) {
if (file_exists(REAL_ROOT_DIR . '/.testing')) {
$environment = env('LB_TESTING_ENV', 'default,local,testing');
} else {
$environment = env('LB_ENV', 'default');
}
}
- .testing may not be deleted if there is a fatal error
- Easier to activate testing environment:
- touch .testing
Drawbacks & Good points
- Tracking JS errors, ajax calls and window prompts can be painful
Problems
- Inject a testing library that overrides Alerts/Confirms
Solutions
window.confirm = function (confirmText) {
var result = lb.testing.confirm.acceptConfirm;
lb.testing.confirm.active = true;
lb.testing.confirm.text = confirmText;
lb.testing.confirm.acceptConfirm = true; //Restore defaults after a call
return result;
};
window.alert = function (alertText) {
lb.testing.alert.active = true;
lb.testing.alert.text = alertText;
return true;
};
Then I should see "confirm" dialog with message
"Are you sure you want to delete the playlist: 'TESTING PLAYLIST'?"
and resolve it
- Inject a testing library that monitors javascript errors
Solutions
window.onerror = function(error, url, line) {
lb.testing.errors.push(error + ". Line: " + line);
};
$numErrors = $this->getSession()->evaluateScript('typeof(lb) != "undefined"
? lb.testing.errors.length : 0');
- Inject a testing library that monitors AJAX calls
Solutions
$(document).on('pjax:start', function () {
if (lb.testing.pjax.enabled && lb.testing.pjax.activeCount == undefined) {
lb.testing.pjax.activeCount = 0;
}
lb.testing.pjax.activeCount++;
});
$(document).on('pjax:end', function () {
if (lb.testing.pjax.enabled) {
lb.testing.pjax.activeCount--;
}
});
$(document).ajaxSend(function (event, jqXHR, ajaxOptions) {
lb.testing.ajax.activeCount++;
});
$(document).ajaxComplete(function (event, jqXHR, ajaxOptions) {
lb.testing.ajax.activeCount--;
lb.testing.ajax.latestRequest = (new Date()).getTime();
});
And I wait for ajax calls to complete
Challenges & Solutions
Handling PJAX and
Live Dom Changes
- In SPAs, page url and page load events are simulated and don't occur/change when you ask for a new page
- Mink is all about immediate DOM processing and page change handling so most mink steps fail automatically because you don't know when stuff really changes...
Problems
- Implement a completely different set of steps that use javascript to detect DOM changes
Solutions
Then I should be brought to page matching "/search\?community=1"
/**
* @Then I should be brought to page matching :regex
* @Then I should be brought to page matching :regex within :timeout seconds
*/
public function shouldBeBroughtToPageRegex($regex, $timeout = self::DEFAULT_TIMEOUT)
{
$this->getSession()->wait(
$timeout * 1000,
"new RegExp('" . $this->locatePath($regex) . "', 'i')
.test(window.location) == true"
);
\Assert\Assertion::regex($this->getSession()->getCurrentUrl(),
'|' . $this->locatePath($regex) . '|i');
}
Given I am on "/"
When X item is clicked
Then I should be on "/search?community=1"
- Create several functions
- shouldEventuallyContain
- shouldEventuallyDisapear
- etc
Solutions
$countEqFunc = function () use ($element, $numberOfItems) {
if (count($element->findAll('css', '.selectables-container button'))
== $numberOfItems) {
return true;
}
return false;
};
if (!$element->waitFor(self::DEFAULT_TIMEOUT, $countEqFunc)) {
throw new ExpectationException("message");
}
Solutions
public function waitFor($timeout, $callback)
{
$start = microtime(true);
$end = $start + $timeout;
do {
$result = call_user_func($callback, $this);
if ($result) {
break;
}
usleep(100000);
} while (microtime(true) < $end);
return $result;
}
- A lot of functions to redefine
- Naming convention gave us headache at first and became confusing
- Build slowly and think about naming
Drawbacks
Challenges & Solutions
Readability and
Change Support
- Gherkin is hard to read if you have complex scenarios
- Mink offers steps that read out well in KISS pages but not in complex apps
- Gherkin can easily become tropical jungle of CSS selectors
- Change your app's HTML/Javascript/CSS and you have to rewrite a lot of tests
Problems
- Create more steps that better describe the actions, less CSS, more words (Widget feature traits)
Solutions
When ActionsButton ".actions-widget" is clicked
Then ActionsButton ".actions-widget" should contain "Edit" action
trait ActionsButtonTrait
{
/**
* @When ActionsButton :selector is clicked
*/
public function actionsButtonClick($selector = null);
/**
* @Then ActionsButton :selector should contain :text action
*/
public function actionsButtonAssertHasAction($text, $selector = null);
/**
* @Then ActionsButton :selector should not contain :text action
*/
public function actionsButtonAssertDoesNotHaveAction($text, $selector = null);
- Reuse existing mink functions when possible inside new step
Solutions
And I am logged in as "testing+teacher@learningbird.com"
public function loginAsUser($username)
{
$this->visit('/signIn');
$this->fillField('signInEmailAddressField', $username);
$this->fillField('signInPasswordField', 'testing-password');
$this->pressButton('signInTrigger');
$this->userShouldBeLoggedIn();
}
public function userShouldBeLoggedIn()
{
try {
$this->elementShouldAppear('css', '#navbar', null, 25);
} catch (ExpectationException $ex) {
$this->assertPageNotContainsText('Could not verify your login. Please make sure you entered your details correctly.');
throw $ex;
}
}
Challenges & Solutions
No Concept of Context
(Part 2 of readability)
- A step doesn't affect the global scope
- If you want easier to read code you need to keep context
Problems
Example
Then I should see "Create Quiz"
And BaseList ".quizzes-dashboard .base-list" should
contain "1" item
And BaseList ".quizzes-dashboard .base-list" item
":nth-of-type(1)" data "numberOfAssignments" should contain "0"
And BaseList ".quizzes-dashboard .base-list" item
":nth-of-type(1)" data "numberOfQuestions" should contain "5"
And BaseList ".quizzes-dashboard .base-list" item
":nth-of-type(1)" data "commands" should contain a "ActionsButton" widget
And ActionsButton ".quizzes-dashboard .base-list:nth-of-type(1) .actions-button"
should not contain "Edit" action
And ActionsButton ".quizzes-dashboard .base-list:nth-of-type(1) .actions-button"
should not contain "Assign" action
When ActionsButton ".quizzes-dashboard .base-list:nth-of-type(1) .actions-button"
is clicked
- Create steps that preserve context
Solutions
And BaseList ".quizzes-dashboard .base-list" should contain "1" item
And BaseList item ":nth-of-type(1)" data "numberOfAssignments" should contain "0"
And BaseList item data "numberOfQuestions" should contain "5"
And BaseList item data "commands" should contain a "ActionsButton" widget
And ActionsButton should contain "Edit" action
When ActionsButton action "Edit" is clicked
//Save the current element as "BaseList"
return $this->getWidgetElement($listSelector, 'BaseList');
protected function getWidgetElement($selector, $widgetName, $inElement = null)
{
if ($selector == null) {
$element = $this->getWidgetContext($widgetName);
return $element;
} else {
$element = $this->getElement('css', $selector, $inElement);
Assertion::true(
$element->hasClass($expectedClass = $this->getWidgetClass($widgetName)),
'Element under "' . $selector . '" found but class "' . $expectedClass . '" not found'
);
$this->setWidgetContext($widgetName, $element);
return $element;
}
}
Challenges & Solutions
World Setup / Cleanup
is Hard With Behat
- Fact: Tests are always around preparing the world
- Fact: No easy way to reseed or reset database
Facts
- Gerkhin is not geared per se at fixtures or test data
- The behat language features a few constructs that can be used for that but none are perfect for the case:
- Table Nodes
- Precondition steps
Problems
- Use TableNodes
Solutions
And teacher "teacher@testing.com" has the following classes:
| className | gradeLevel | subject |
| Math class | 9 | Math |
| Science class | 10 | Science |
/**
* @Given teacher :teacherEmail has the following classes:
*/
public function createClassesForTeacher(TableNode $classData, string $teacherEmail): array //[classesRecord]
{
$teacher = $this->getOrCreateTeacher($teacherEmail);
$account = $teacher->getAccount();
if ((int)$account->getAccountTypeID() !== accountTypesModel::TEACHER) {
throw new Exception('referenced teacher must be a rogue teacher');
}
$classes = $this->createClassesFromTableNode($classData, $account);
foreach ($classes as $class) {
$this->associateTeacherToClass($teacher, $class);
}
return $classes;
}
- Creation of pre-condition steps
Solutions
Given a skeleton "teacher" with the email "teacher@testing.com" exists
Given a skeleton "teacher" with the email "teacher2@testing.com" and
password "testpwd" exists with 60 licenses
/**
* @Given a skeleton :userType with the email :emailAccount and password :password exists
* with :numLicenses licenses
*/
public function createSkeletonUserAccount($emailAccount, $userType,
$password = self::DEFAULT_PASSWORD, $numLicenses = 100)
{
switch ($userType) {
case 'teacher':
$this->createSkeletonTeacher($emailAccount, $password, $numLicenses);
break;
case 'student':
$this->createSkeletonStudent($emailAccount, $password);
break;
case 'school_admin':
$this->createSkeletonSchoolAdmin($emailAccount, $password);
break;
}
}
- Gerkhin doesn't feature steps to cleanup or prepare the world
- What it does have is annotation bindings:
- @beforeSuite
- @beforeScenario
- @beforeStep
- @afterStep
- ....
Problems
- Using hooks to handle special cases such as seeding/cleaning up
Solutions
/**
* @BeforeSuite
*
* @param \Behat\Testwork\Hook\Scope\BeforeSuiteScope $scope
*/
public static function beforeSuite(\Behat\Testwork\Hook\Scope\BeforeSuiteScope $scope)
{
$suite = $scope->getSuite();
file_put_contents(REAL_ROOT_DIR . '/.testing', \Carbon\Carbon::now()->format('Y-m-d G:i:s'));
if (!file_exists(LOGS_DIR . "/behat-screenshots/{$suite->getName()}/")) {
mkdir(LOGS_DIR . "/behat-screenshots/{$suite->getName()}/", 0755, true);
}
// Kill the screenshots
/** @var SplFileInfo $file */
foreach (new DirectoryIterator(LOGS_DIR . "/behat-screenshots/{$suite->getName()}/") as $file) {
if (substr($file->getFilename(), 0, 1) == '.') {
continue;
}
echo 'Removing ' . realpath($file->getPathName()) . PHP_EOL;
unlink($file->getPathName());
}
self::printSuiteInfo($suite);
}
- Using hooks to handle special cases such as seeding/cleaning up
Solutions
/**
* @BeforeScenario
*
* @param \Behat\Behat\Hook\Scope\BeforeScenarioScope $scope
*/
public function before(\Behat\Behat\Hook\Scope\BeforeScenarioScope $scope)
{
$this->cleanDatabase();
$this->resetWidgetContexts();
$this->getSession()->resizeWindow(1440, 900, 'current');
#Cookies cause issues when executing/evaluating scripts in
#internet explorer
$this->getSession()->getDriver()->reset();
}
- Using hooks to handle special cases such as seeding/cleaning up
Solutions
/**
* @AfterStep
*/
public function afterStepTasks(\Behat\Behat\Hook\Scope\AfterStepScope $event)
{
// Failed step
if ($event->getTestResult()->getResultCode() ==
\Behat\Testwork\Tester\Result\TestResult::FAILED) {
$this->takeScreenshotAfterFailedStep($event->getStep()->getText(),
$event->getSuite());
}
// Collect and handle javascript errors
$this->reportJavascriptErrors();
}
- Using hooks to handle special cases such as seeding/cleaning up
Solutions
/**
* @AfterScenario
*
* @param \Behat\Behat\Hook\Scope\AfterScenarioScope $scope
*/
public function after(\Behat\Behat\Hook\Scope\AfterScenarioScope $scope)
{
// Make sure you can exit without getting asked if you want to leave
$this->getSession()->executeScript('window.onbeforeunload = null;');
$this->clearMailFromMemory();
$this->cleanPlaylists();
}
- Using hooks to handle special cases such as seeding/cleaning up
Solutions
/**
* @AfterSuite
*
* @param \Behat\Testwork\Hook\Scope\AfterSuiteScope $scope
*/
public static function afterSuite(\Behat\Testwork\Hook\Scope\AfterSuiteScope $scope)
{
unlink(REAL_ROOT_DIR . '/.testing');
}
Question?
Mathieu Dumoulin
Senior Dev @ Learning Bird
Twitter: @thecrazycodr
LinkedIn: @crazycoders
GitHub: @crazycodr
Email: thecrazycodr@gmail.com
Open source contributions
Contact me
Standard-Exceptions 2.0: GitHub
Building Behat tests like a pro
By Mathieu Dumoulin
Building Behat tests like a pro
Behat is a great tool when it comes to testing pages with javascript because it gives you a great base framework mostly when you use the Mink extensions. This talk will show the different challenges we had to solve around our SPA at Learning Bird and the kind of structure we had to adopt to create beautiful and readable tests.
- 1,136