@MichaKutz
I would consider it a red flag if a team isn't doing exploratory testing at all - even if their automated testing was excellent.
Even the best automated testing is inherently scripted testing - and that alone is not good enough.
November 18th 2019
Martin Fowler
that the software meets expectations
for additional risks
Simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next.
interesting things to vary and interesting ways in which to vary them
as soon as you think of a test
look past what you expect
to push even harder
Elisabeth Hendrickson
@testobsessed
Can a user actually pass the checkout process?
What happens when I change my basket in a different tab, right before finishing checkout?
What happens when I use app and web in parallel? Are baskets synced?
Is there a reasonable error page displayed, when the system is on overload?
What happens when I click the basket button really fast?
What happens when I change the delivery time slot in a different tab, right before finishing checkout?
Automation
Exploration
Load
Exploration
Exploration
Exploration
Explore <target>
with <resources>
to discover <information>
Where are you exploring?
a feature
a requirement
a module
What resources will you bring with you?
a tool
a data set
a technique
a configuration
an interdependent feature
What kind of information are you hoping to find?
security
performance
reliability
capability
usability
consistency of design
violations of a standard
from "Explore It!"
by Elisabeth Hendrickson
My mission is to test <risk>
in the context of <coverage>.
What am I testing?
a feature
a requirement
a module
What could go wrong?
functionality is wrong
usability is bad
not accessible
inconsistency
from "Tips for Writing Better Charters for Exploratory Testing Sessions"
by Michael D. Kelly
<timeframe>
How long is this supposed to take?
often implicit
Explore the behavior of the basket button
with various interaction types and speeds
to discover unintended side effects
Explore every input field in the shop
with every security tool you can find
to discover security issues
Explore the address form
with the name "Søren Anderson"
to discover if scandinavial letters are handled correctly
Too narrow
actually a test case
Too broad
you will never be finished
Explore the registration form
with common XSS injection strings
to discover XSS attack vulnerabilities
My mission is to test SQL vulnerabilities
in the context of the search form.
Find ways that a valid order modification might fail.
Experiment with invalid values when updating customer addresses.
from "Tips for Writing Better Charters for Exploratory Testing Sessions"
by Michael D. Kelly
What was I supposed to focus on?!
I need more time!
Boooooooooring!
Where can I find the data needed for this?!
Didn't somebody test this already?!
Is this really important?!
We are going to offer same day delivery.
What's the latest time to order for same day?
1pm.
What happens when 1pm passes while the user is already in checkout past the time slot selection?
Can we throw the user back to the selection?
Shouldn't be any problem.
Explore checkout past time slot selection around same day cutoff
with a same day time slot
to discover problems with throw back mechanism.
I have to think about that…
Users can reserve time slots for up to 30 minutes.
Should we prevent reservations past 12:30?
Explore reservations
with same day time slots past 12:30
to discover if reservations allow users to place same day orders past 1pm.
Continued
Security Model
Performance Requirements
Reliability
Usability
Accessability
Scalability
// I don't know why this works… // …but it does // DO NOT TOUCH! if (deliveryAddress != null) { …
What happens when I change my basket in a different tab, right before finishing checkout?
What happens when I use app and web in parallel? Are baskets synced?
What happens when I click the basket button really fast?
What happens when I change the delivery time slot in a different tab, right before finishing checkout?
Explore the checkout process
with background changes of the basket
to discover consistency problems.
Explore the basket
with background changes of the basket via app
to discover consistency problems.
Explore delivery fee calculation after time slot selection
with changing the time slot in background
to discover surprises.
Explore the behavior of the basket button
with various click speeds
to discover surprises.
Did the installer run without errors?
Can I start the software?
Was the add-to-basket success message displayed?
Did the product actually appear in the basket overview?
Is the logout button displayed?
Can I edit my profile?
unexpected activity
JavaScript errors
unexpected system load
unexpected filesystem or database activity
$> watch ls
$> psql
subtle UI changes
unexpected data transfer
Forms
Browser type
Cookie values
URL & parameters
Input fields
Browser plugins
Uptime of the system
…
Countable Things
→ 0, 1, many
→ too many
→ too few
Positions
→ beginning, middle, end
Files & Storage
→ make inaccessible
→ use different then default location
Geographic Locations
→ far away, close by
→ different time zone
Formats
→ dates, decimals, binary, IP addresses
→ file formats
Size
→ huge, small
Timing, Frequency & Duration
→ at midnight
→ fast, slow
Input & Navigation
→ mouse, keyboard
→ type, paste
→ touch
Who will use this software and for what purpose?
Why would someone choose this software over competitors?
Can you give me an elevator pitch for the software?
If nothing else is working, what has to function?
What is your greatest fear regarding the software?
What are our most important quality factors?
Accuracy? Reliability? Availability? Usability? Accessibility? Security?
Core Capabilities
"Non-Functional"
Requirements
Risks
Internal Consistency
Standards
Comparables
Plausibility
Reverse Test
Checklists
e.g. https://www.ministryoftesting.com/dojo/lessons/checklist-for-testing-web-page-functionality
Always bill!
Never bill twice!
Never display wrong ingredients for a product!
Always provide at least one deliver time slot within the next week!
Never display wrong data at order confirmation page!
Always allow to cancel the checkout!
Never display different prices for the same product!
Always deliver valid HTML according to W3C standards!
Always reader page withing 1 second!
Always have a reasonable alt attribute for all images!
Design
Execute
Learn
My mission is to explore the checkout process
for surprises.
My mission is to explore the checkout process
for side effects caused by browser navigation.
My mission is to explore the checkout process
for side effects caused by parallel activity in a different tab.
actual time for testing
confidence factor
on coverage of the stated risk
found bugs
possible new charters
open questions
things to be automated
things to be documented
setup steps
My mission is to explore the checkout process
for side effects caused by browser navigation.
My mission is to explore the checkout process
for side effects caused by parallel activity in a different tab.
Sprint
Planning
Review/
Retro
Coding
1st story done 🎉
2nd story done 🎉
last story done 🎉
Daylies
Backlog Refinement
"Explore It!"
by Elisabeth Hendrickson
https://pragprog.com/book/ehxta/explore-it
"Tips for Writing Better Charters for Exploratory Testing Sessions"
by Michael D. Kelly
https://www.slideshare.net/EuroSTARConference/mike-kelly-euro-star-webinar
https://www.youtube.com/watch?v=dOQuzQNvaCU
"A Heuristic Approach to Test Charters"
by Adam Howard
https://solavirtusinvicta.wordpress.com/2014/12/18/a-heuristic-approach-to-test-charters/
https://youtu.be/9FKY1Is0lgs
Nouns & Verbs
Mix randomly
Randomize your navigation
mouse only, short cuts only, search only, menu only…
Personas
how would Chuck Norris use this product?