effective Automation
by Ihor Sosnovskyi
agenda
- Automation pros & cons
- Considerations for automation
- Selecting proper tools
- Before start automating your stuff
- 6 common test automation mistakes
- Automation approaches
- Automation levels
- Testing levels
- Continuous integration and delivery
- Is manual testing still needed?
- BDD approach
- Fast feedback
Automation pros & cons?
To automate or not automate?
Pros
- Eliminates human error
- Noticeably faster
- Removes the most boring and repetitive work
- No manual alternative for testing load and performance
- Helps to generate pre condition for further manual testing
- Some test cases can not be manually tested(e.g. 3rd party responses)
- Manpower and resources involved in regression test are less
Automation pros & cons?
CONS
- Requires lot of efforts at initial stage
- Requires a higher skill level of team members
- Requires maintenance Time and Effort
- Test script debugging is of a serious concern
- Not suitable for short-term projects
Automation pros & cons?
Considerations for automation
What to automate?
- High Risk - Business Critical test cases
- Tests that need to be run against every build/release of the application(smoke test, sanity test and regression test)
- Test Cases that are very tedious or difficult to perform manually
- Test Cases which are time consuming
- Tests cases that are that are hard or impossible to execute manually(e.g. performance, load)
- Tests that execute the same workflow but use different data for its inputs for each test run e.g. data-driven
considerations for automation
What should not be automated?
- Tests that you will run only once
- User experience tests for usability
- Tests that require ad hoc/random testing based on domain knowledge/expertise – Exploratory Testing
- Test Cases for which the requirements are changing frequently
- Test cases which are executed on ad-hoc basis
- Possibly test cases that is hard to automate
- Functionally that is going to be deprecated
considerations for automation
Is manual testing still needed?
- YES!!!
- It is not feasible to automate all the test cases
- More likely to find real user issues
- Non-automatable cases
- Usability testing
- Best suited for exploratory, usability and ad-hoc testing
- Machines are great for running regression tests
- Human are great for finding new defects
considerations for automation
Selecting proper tools
main factors
- Technology Under test
- Scripting language
- Pricing Policy
- Tool support(prefer tools that are actively supported, improved)
- Versatility(supports the type of automated testing you need)
- Ease of CI/Delivery integration
- Results generation
- Platform Compatibility
- OS Compatibility
selecting proper tools
Other factors
- Ease of use
- Ease of Maintenance
- Functionality
- Flexibility
- Tool users and their skill level
- Automated tests as live documentation
- Ability to run tests in parallel or against different environment in the same time
- Ability to execute tests on CI(e.g. run UI tests in headless mode)
- Requirements traceability
selecting proper tools
Before start automating your stuff
A piece of advice
- Do not depend on platform being tested code
- Think about future requirements:
- parallelisation
- running against different environments
- results report generation
- test data
- Bigger test frameworks require more coding effort:
- organizing the code
- harder to manage
- makes testing code much harder to understand and maintain
- requires clear structure
- Do not invent a new wheel. Try to find existing solution!
- Avoid meta programming and other magic stuff
Ensure that ...
- Test scripts will be reusable
- Tests data will be reusable
- Tests won't depend on shared data(db, files etc)
- Tests won't use shared resources that other tests may use
- Testing framework will be built in a super simple way(KISS)
- Do not depend on platform being tested code
- New testing code will be properly reviewed(both code clarity/correctness and whether scripts test feature properly)
- "Quick wins" are avoided
- Coverage duplication is avoided where it is possible
6 common test automation mistakes
-
Drive testing entirely through the user interface
-
Ignore the build/test/deploy pipeline
-
Set up test data through the user interface
-
Copy/paste your test code
-
Think of test tooling as a single, large computer program
Testing levels
What to test and on which level?
Testing PYRAMID
testing levels
Three-Level Model
testing levels
Unit testing
Benefits:
- Extremely fast to execute
- Reliable
- Easy to troubleshoot
Drawbacks:
- Mostly designed by develoers
- Can be difficult to understand by QAs
- Testers should have the necessary skills
- Harder to track test cases coverage
- Monitoring and control from QA can be required
(Unit Test layer)
API/backend testing
- Faster preparation
- More sustainable
- Requires less updates
- Faster execution
- Often most suitable for covering business logic
- Third party responses can be simulated
Benefits:
Drawbacks:
- Debugging can be more complicated
- May require understanding of platform code and workflows
- A bit more complex to create and maintain
testing levels
(Service Layer non-UI)
UI testing
(The cherry on top)
- Same interface as users use
- The same test can run against different environments and browser
- Core customer usage workflows implemented on this level
- Can actually convince other stakeholders that the system does indeed work
Benefits:
Drawbacks:
- Can be technically challenging or impossible
- Time consuming to run
- Brittle
- Expensive to write
- More difficult to maintain
- Break more easily
testing levels
Continuous integration/delivery
- Key to full scale automation
- Tests are run automatically when
- Tests are added
- Code is changed
- Scheduled test runs
- Great open source solutions available
BDD Approach
“software that matters”
- Enough is enough
- Deliver stakeholder value
- It’s all behavior
- Reduced waste
- Reduced costs
- Easier and safer changes
- Faster releases
BDD APProach
BDD APProach
Disadvantages and potential challenges
- Requires high business engagement and collaboration
-
Works best in an Agile or iterative context
-
BDD doesn’t work well in a silo
-
Poorly written tests can lead to higher test-maintenance costs
BDD APProach
refactor with confidence
BDD APProach
A bit of advice
"Simplicity always wins in the long term"