effective Automation

by Ihor Sosnovskyi

agenda

• Automation pros & cons
• Considerations for automation
• Selecting proper tools
• Before start automating your stuff
• 6 common test automation mistakes
• Automation approaches
• Automation levels
• Testing levels
• Continuous integration and delivery
• Is manual testing still needed? 
• BDD approach
• Fast feedback

Automation pros & cons?

To automate or not automate?

Pros

• Eliminates human error
• Noticeably faster
• Removes the most boring and repetitive work
• No manual alternative for testing load and performance
• Helps to generate pre condition for further manual testing
• Some test cases can not be manually tested(e.g. 3rd party responses)
• Manpower and resources involved in regression test are less

Automation pros & cons?

CONS

• Requires lot of efforts at initial stage
• Requires a higher skill level of team members
• Requires maintenance Time and Effort
• Test script debugging is of a serious concern
• Not suitable for short-term projects

Automation pros & cons?

Considerations for automation

What to automate?

• High Risk - Business Critical test cases
• Tests that need to be run against every build/release of the application(smoke test, sanity test and regression test)
• Test Cases that are very tedious or difficult to perform manually
• Test Cases which are time consuming
• Tests cases that are that are hard or impossible to execute manually(e.g. performance, load)
• Tests that execute the same workflow but use different data for its inputs for each test run e.g. data-driven

considerations for automation

What should not be automated?

• Tests that you will run only once
• User experience tests for usability
• Tests that require ad hoc/random testing based on domain knowledge/expertise – Exploratory Testing
• Test Cases for which the requirements are changing frequently
• Test cases which are executed on ad-hoc basis
• Possibly test cases that is hard to automate
• Functionally that is going to be deprecated

considerations for automation

Is manual testing still needed?

• YES!!!
• It is not feasible to automate all the test cases
• More likely to find real user issues
• Non-automatable cases
• Usability testing
• Best suited for exploratory, usability and ad-hoc testing
• Machines are great for running regression tests
• Human are great for finding new defects

considerations for automation

Selecting proper tools

main factors

• Technology Under test
• Scripting language
• Pricing Policy
• Tool support(prefer tools that are actively supported, improved)
• Versatility(supports the type of automated testing you need)
• Ease of CI/Delivery integration
• Results generation
• Platform Compatibility
• OS Compatibility

selecting proper tools

Other factors

• Ease of use
• Ease of Maintenance
• Functionality
• Flexibility
• Tool users and their skill level
• Automated tests as live documentation
• Ability to run tests in parallel or against different environment in the same time
• Ability to execute tests on CI(e.g. run UI tests in headless mode)
• Requirements traceability

selecting proper tools

Before start automating your stuff

A piece of advice

• Do not depend on platform being tested code
• Think about future requirements:
  • parallelisation
  • running against different environments
  • results report generation
  • test data
• Bigger test frameworks require more coding effort:
  • organizing the code
  • harder to manage
  • makes testing code much harder to understand and maintain
  • requires clear structure
• Do not invent a new wheel. Try to find existing solution! 
• Avoid meta programming and other magic stuff

Ensure that ...

• Test scripts will be reusable 
• Tests data will be reusable
• Tests won't depend on shared data(db, files etc)
• Tests won't use shared resources that other tests may use
• Testing framework will be built  in a  super simple way(KISS)
• Do not depend on platform being tested code
• New testing code will be properly reviewed(both code clarity/correctness and whether scripts test feature properly)
• "Quick wins" are avoided
• Coverage duplication is avoided where it is possible

6 common test automation mistakes

1. Drive testing entirely through the user interface

2. Ignore the build/test/deploy pipeline

3. Set up test data through the user interface

4. Copy/paste your test code

5. Think of test tooling as a single, large computer program

Testing levels

What to test and on which level?

Testing PYRAMID

testing levels

Three-Level Model

testing levels

Unit testing

Benefits:

• Extremely fast to execute
• Reliable
• Easy to troubleshoot

Drawbacks:

• Mostly designed by develoers
• Can be difficult to understand by QAs
• Testers should have the necessary skills
• Harder to track test cases coverage
• Monitoring and control from QA can be required

(Unit Test layer)

API/backend testing

• Faster preparation
• More sustainable
• Requires less updates
• Faster execution
• Often most suitable for covering business logic
• Third party responses can be simulated

Benefits:

Drawbacks:

• Debugging can be more complicated 
• May require understanding of platform code and workflows
• A bit more complex to create and maintain

testing levels

(Service Layer non-UI)

UI testing

(The cherry on top)

• Same interface as users use
• The same test can run against different environments and browser
• Core customer usage workflows implemented on this level
• Can actually convince other stakeholders that the system does indeed work

Benefits:

Drawbacks:

• Can be technically challenging or impossible
• Time consuming to run
• Brittle
• Expensive to write
• More difficult to maintain
• Break more easily

testing levels

Continuous integration/delivery

• Key to full scale automation
• Tests are run automatically when
  • Tests are added
  • Code is changed
• Scheduled test runs
• Great open source solutions available

BDD Approach

“software that matters”

• Enough is enough
• Deliver stakeholder value
• It’s all behavior
• Reduced waste
• Reduced costs
• Easier and safer changes
• Faster releases 

BDD APProach

BDD APProach

Disadvantages and potential challenges

• Requires high business engagement and collaboration

• Works best in an Agile or iterative context

• BDD doesn’t work well in a silo

• Poorly written tests can lead to higher test-maintenance costs

BDD APProach

refactor with confidence

BDD APProach

A bit of advice

"Simplicity always wins in the long term"

Questions?

References

• Pros and Cons of Automated and Manual Testing
• AUTOMATION TESTING Tutorial: Process, Planning & Tools
  
• TestPyramid
• 6 common test automation mistakes and how to avoid them
• Introducing BDD
• TDD vs BDD