Let's Explore
Exploratory Testing
@MichaKutz
I would consider it a red flag if a team isn't doing exploratory testing at all - even if their automated testing was excellent.
Even the best automated testing is inherently scripted testing - and that alone is not good enough.
November 18th 2019
Martin Fowler
Exploratory Testing
that the software meets expectations
for additional risks
Simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next.
interesting things to vary and interesting ways in which to vary them
as soon as you think of a test
look past what you expect
to push even harder
Elisabeth Hendrickson
@testobsessed
Can a user actually pass the checkout process?
What happens when I change my basket in a different tab, right before finishing checkout?
What happens when I use app and web in parallel? Are baskets synced?
Is there a reasonable error page displayed, when the system is on overload?
What happens when I click the basket button really fast?
What happens when I change the delivery time slot in a different tab, right before finishing checkout?
Think about questions for your product…
Automation
Exploration
Load
Exploration
Exploration
Exploration
Design
Explore <target>
with <resources>
to discover <information>
Where are you exploring?
a feature
a requirement
a module
What resources will you bring with you?
a tool
a data set
a technique
a configuration
an interdependent feature
What kind of information are you hoping to find?
security
performance
reliability
capability
usability
consistency of design
violations of a standard
Charter Template
from "Explore It!"
by Elisabeth Hendrickson
My mission is to test <risk>
in the context of <coverage>.
What am I testing?
a feature
a requirement
a module
What could go wrong?
functionality is wrong
usability is bad
not accessible
inconsistency
Alternative Template
from "Tips for Writing Better Charters for Exploratory Testing Sessions"
by Michael D. Kelly
<timeframe>
How long is this supposed to take?
often implicit
Explore the behavior of the basket button
with various interaction types and speeds
to discover unintended side effects
Explore every input field in the shop
with every security tool you can find
to discover security issues
Explore the address form
with the name "Søren Anderson"
to discover if scandinavial letters are handled correctly
Too narrow
actually a test case
Too broad
you will never be finished
Explore the registration form
with common XSS injection strings
to discover XSS attack vulnerabilities
My mission is to test SQL vulnerabilities
in the context of the search form.
Find ways that a valid order modification might fail.
Experiment with invalid values when updating customer addresses.
Good Charters,
Bad Charters
Bad Charters Feel Like…
from "Tips for Writing Better Charters for Exploratory Testing Sessions"
by Michael D. Kelly
What was I supposed to focus on?!
I need more time!
Boooooooooring!
Where can I find the data needed for this?!
Didn't somebody test this already?!
Is this really important?!
We are going to offer same day delivery.
What's the latest time to order for same day?
1pm.
What happens when 1pm passes while the user is already in checkout past the time slot selection?
Can we throw the user back to the selection?
Shouldn't be any problem.
Explore checkout past time slot selection around same day cutoff
with a same day time slot
to discover problems with throw back mechanism.
Requirement Discussions
I have to think about that…
Users can reserve time slots for up to 30 minutes.
Should we prevent reservations past 12:30?
Explore reservations
with same day time slots past 12:30
to discover if reservations allow users to place same day orders past 1pm.
Requirement Discussions
Continued
Security Model
Performance Requirements
Reliability
Usability
Accessability
Scalability
Implicit Expectations
// I don't know why this works… // …but it does // DO NOT TOUCH! if (deliveryAddress != null) { …
Other sources for charters…
What happens when I change my basket in a different tab, right before finishing checkout?
What happens when I use app and web in parallel? Are baskets synced?
What happens when I click the basket button really fast?
What happens when I change the delivery time slot in a different tab, right before finishing checkout?
Think about charters for your product…
Explore the checkout process
with background changes of the basket
to discover consistency problems.
Explore the basket
with background changes of the basket via app
to discover consistency problems.
Explore delivery fee calculation after time slot selection
with changing the time slot in background
to discover surprises.
Explore the behavior of the basket button
with various click speeds
to discover surprises.
Execute
Did the installer run without errors?
Can I start the software?
Ask deeper questions
Was the add-to-basket success message displayed?
Did the product actually appear in the basket overview?
Is the logout button displayed?
Can I edit my profile?
Look past what you expect or hope to find
unexpected activity
JavaScript errors
unexpected system load
unexpected filesystem or database activity
$> watch ls
$> psql
subtle UI changes
unexpected data transfer
Forms
Browser type
Cookie values
URL & parameters
Input fields
Browser plugins
Variables
- Number of products shop
- Number of search results
- Number of users logged in
-
Uptime of the system
-
…
Countable Things
→ 0, 1, many
→ too many
→ too few
Positions
→ beginning, middle, end
Files & Storage
→ make inaccessible
→ use different then default location
Geographic Locations
→ far away, close by
→ different time zone
Formats
→ dates, decimals, binary, IP addresses
→ file formats
Size
→ huge, small
Timing, Frequency & Duration
→ at midnight
→ fast, slow
Input & Navigation
→ mouse, keyboard
→ type, paste
→ touch
Variable Types
Think about variables in your product…
- Format: try to add "a"
- Input: try pasting, typing, clicking
- Position: type/paste at beginning, middle, end
- Countable: 0, 1, (too) many
- Speed: click fast
- Input: click, press enter, press space
- Timing: add to basket and wait for September 13th
Ensemble
Evaluate Results
Who will use this software and for what purpose?
Why would someone choose this software over competitors?
Always & Never
Can you give me an elevator pitch for the software?
If nothing else is working, what has to function?
What is your greatest fear regarding the software?
What are our most important quality factors?
Accuracy? Reliability? Availability? Usability? Accessibility? Security?
Core Capabilities
"Non-Functional"
Requirements
Risks
Alternative Verification
Internal Consistency
Standards
Comparables
Plausibility
Reverse Test
Checklists
e.g. https://www.ministryoftesting.com/dojo/lessons/checklist-for-testing-web-page-functionality
Think about Alwayses and Nevers of your product…
Always bill!
Never bill twice!
Never display wrong ingredients for a product!
Always provide at least one deliver time slot within the next week!
Never display wrong data at order confirmation page!
Always allow to cancel the checkout!
Never display different prices for the same product!
Always deliver valid HTML according to W3C standards!
Always reader page withing 1 second!
Always have a reasonable alt attribute for all images!
Learn
Design
Execute
Learn
My mission is to explore the checkout process
for surprises.
My mission is to explore the checkout process
for side effects caused by browser navigation.
My mission is to explore the checkout process
for side effects caused by parallel activity in a different tab.
actual time for testing
confidence factor
on coverage of the stated risk
found bugs
possible new charters
open questions
things to be automated
things to be documented
setup steps
My mission is to explore the checkout process
for side effects caused by browser navigation.
My mission is to explore the checkout process
for side effects caused by parallel activity in a different tab.
What did you learn about your product?
Continuous Exploration
Sprint
Planning
Review/
Retro
Coding
1st story done 🎉
2nd story done 🎉
last story done 🎉
Daylies
Backlog Refinement
"Explore It!"
by Elisabeth Hendrickson
https://pragprog.com/book/ehxta/explore-it
"Tips for Writing Better Charters for Exploratory Testing Sessions"
by Michael D. Kelly
https://www.slideshare.net/EuroSTARConference/mike-kelly-euro-star-webinar
https://www.youtube.com/watch?v=dOQuzQNvaCU
"A Heuristic Approach to Test Charters"
by Adam Howard
https://solavirtusinvicta.wordpress.com/2014/12/18/a-heuristic-approach-to-test-charters/
Sources & Acknowledgments
https://youtu.be/9FKY1Is0lgs
Adding Dimensions
Nouns & Verbs
Mix randomly
Randomize your navigation
mouse only, short cuts only, search only, menu only…
Personas
how would Chuck Norris use this product?