iOS Device Farm Comparison

Potential Integration Partners

• AWS Device Farm
• TestObject Device Lab
• Testdroid
• Xamarin Test Cloud

Terms - Device Time

• Billable time
• Minutes/hours of each test run you conduct on devices.
  • 100 device minute
    =
    10 device x 10 minute
                    or
    20 device x 5 minute
• Starts when app is installed
• Ends when app is uninstalled

Terms - Device Concurency

• Also: Device Slot
• Number of devices you can use simultaneously
• 5 concurrent devices
  =
  Testing 5 different device configurations
  in parallel

AWS Device Farm

by Amazon

Automated Testing

• Acces via Api
• Get results, screenshots, video, performance data
• Supported frameworks:
  • Appium
  • Calabash
  • UI Automation
  • XCTest (including KIF)
  • XCTest UI

Remote Access

• AWS displays device screen in browser
  • Internet Explorer 9, Chrome, Firefox, Safari
• Gesture, swipe, and interact with the device
• Max time: 60 minutes
  • Contact for longer
• During session
  • Logging of details
  • Video capture of the session

Pricing

Notes:

• Free trial of 250 device minutes (~4 hours)
• $0.17 per minute ~ $ 10 per hour
• 25 hour metered ~ 1 month unmetered

Important FAQs

• CI: Jenkins
• If your systems are internal, whitelist the IP range 54.244.50.32-54.244.50.63
• Simulate connection types and conditions
  • Curated network profile like "3G" or "Lossy LTE"
  • Control parameters like throughput, jitter, loss
• Built-in compatibility test suite
  • Install, uninstall, launch, and run Fuzz
• Supply latitude and longitude coordinates

TestObject Device Lab

by Sauce Labs

Automated Testing

• Access via Api
  • /devices/all, /devices/available
  • /appium/suites/123/devices
• Supported framework:
  • Appium

Remote Access

• Displays device screen in browser
• Uses VNC to transmit mouse and keyboard
• VNC client on the device
  • Relays back the graphical screen
• Remote control

Pricing

• * Fair use: Limitation on daily usage
  • 1 hour average testing time per user/day

Important FAQs

• CI: Jeckins+Gradle, Travis
• Device pool is hosted in Germany
• For iOS devices, it is currently not possible to test push notifications
• Whitelist the IP address 37.44.7.110.

Testdroid

by Bitbar

Private Cloud

• Dedicated cloud devices
• Infrastructure-as-a-Service
• Pre-selected real iOS devices hosted in Europe

Public Cloud

• Cloud-based device farm

On Premise (Enterprise)

• In-house software solution

Automated Testing

• Access via api
• Get results, screenshots, video, performance data
• Supported frameworks:
  • Appium
  • Calabash
  • Jasmine
  • XCTest (including KIF)
  • XCTest UI

Remote Access

• iOS manual testing is available on dedicated devices (Public cloud)
  • Dedicated: Reserve devices only for you
• Enterprise & Private Cloud can make devices available
• Direct VNC connection

• Customer servers  whitelist public IPs to allow connections from our devices (Public Cloud)

• VPN or proxy connection (Private Cloud)

• Customer has complete freedom on networking (Enterprise cloud)

Pricing - Public Cloud

Pricing - Private Cloud

Pricing - Enterprise Cloud

Important FAQs

• CI: Jenkins CI, Jira Integration
• Latency can be expected as the connection is live.
  • (San Jose, CA and Wroclaw, Poland.)

• In Public Cloud it is not allowed to access device settings

• Private & Enterprise environments settings can be changed via remote access

• No recorder for iOS, recommend Xcode Instruments

Xamarin Test Cloud

by Xamarin

Automated Testing

• Access via Api
  • Xamarin.UITest API
  • Calabash Ruby API for iOS

• Test any native or hybrid mobile app

  • Not just apps written with Xamarin

Xamarin Test Recorder

• Still under development
  • Start Test Recorder
  • Connect it to device
  • Use the app
• Will capture the interactions
• Outputs a Xamarin.UITest in C#

Pricing

Important FAQs

• CI: Jenkins, TFS and TeamCity
• No manual testing
• Xamarin Test Recorder requires iOS 8 or higher

COMPARISON

AWS

TestObject

TestDroid

Xamarin