QUALITY ASSESSMENT OF
COMMERCIAL SEARCH SERVICES
Arief Karfianto
Anusha Jalahalli Chandra Murthy
David Quan
Varut Sangsa-Ard
Outline
- Overview
- Metamorphic Relations
- Test Case Generation
- Test Case Execution
- Findings
- Conclusion
Overview
- Most software testing techniques require an oracle
- Oracle is a mechanism that can be used to verify the outcomes and the correctness of test case execution.
- However, in many situations there is no oracle that can be used or the cost of using an oracle is very expensive (oracle problem).
- Metamorphic testing was proposed to overcome some of the inherent problems that are present in testing software without test oracles (Chen, Tse and Zhou 2002)
-
In this report, metamorphic testing is used to assess the quality of a number of websites:
- Car search: Carsales.com.au and Carsguide.com.au
- Job search listing: Seek.com.au and Careerone.com.au
-
Property search: Realestate.com.au and Domain.com.au
Metamorphic Relations
1. MRTransmission: tests that the number of Automatic transmission, Manual transmission and Unknown transmission cars offered on CarSales and CarsGuide matches the total amount of cars being offered (all three transmission types combined).
CarsGuide and CarSales
2. MRKilometres: tests that the number of MIN-150000KM + the number of 150000KM-MAX cars offered on CarSales and CarsGuide matches the total amount of cars being offered (MIN – MAX KMS).
Metamorphic Relations
3. MREmployment: tests that the number of Full Time + Part Time + Contract + Casual jobs matches the total amount of jobs being offered.
Seek and CareerOne
4. MRSalary150K: tests that the number of 0-150K paying jobs + the number of 150k – MAX paying jobs matches the total amount of jobs being offered (0 - MAX Salary).
Metamorphic Relations
5. MRBedrooms: tests that the number of 3, 4 and 5+ (in the case of domain) bedroom properties matches the total amount of properties with 3-5+ (Domain) and 3-4 bedrooms (RealEstate).
Domain and RealEstate
6. MRPrice: tests that the number of properties sold between 0 – 600000 + the number of properties sold between 600000 – 5000000 equals the total amount of properties sold between 0 - 5000000.
Number of Test Cases
Test Case Execution
Findings
Conclusion
- Based on MRTransmission and MRKilometres, CarSales is more accurate than CarsGuide
- Based on MREmployment and MRSalary150K, Seek is more accurate than CareerOne
- Based on MRBedrooms and MRPrice, Domain and RealEstate both display similarly high accuracy rates.
The overall conclusion that can be drawn from this research is that: