iOS Device Farm Comparison

Potential Integration Partners

  • AWS Device Farm
  • TestObject Device Lab
  • Testdroid
  • Xamarin Test Cloud

Terms - Device Time

  • Billable time
  • Minutes/hours of each test run you conduct on devices.
    • 100 device minute
      =
      10 device x 10 minute
                      or
      20 device x 5 minute
  • Starts when app is installed
  • Ends when app is uninstalled

Terms - Device Concurency

  • Also: Device Slot
  • Number of devices you can use simultaneously
  • 5 concurrent devices
    =
    Testing 5 different device configurations
    in parallel

AWS Device Farm

by Amazon

Automated Testing

  • Acces via Api
  • Get results, screenshots, video, performance data
  • Supported frameworks:
    • Appium
    • Calabash
    • UI Automation
    • XCTest (including KIF)
    • XCTest UI

Remote Access

  • AWS displays device screen in browser
    • Internet Explorer 9, Chrome, Firefox, Safari
  • Gesture, swipe, and interact with the device
  • Max time: 60 minutes
    • Contact for longer
  • During session
    • Logging of details
    • Video capture of the session

Pricing

Metered - Pay as you go Unmetered - Unlimited
$0.17 per device minute $250 per device slot/month

Notes:

  • Free trial of 250 device minutes (~4 hours)
  • $0.17 per minute ~ $ 10 per hour
  • 25 hour metered ~ 1 month unmetered

Important FAQs

  • CI: Jenkins
  • If your systems are internal, whitelist the IP range 54.244.50.32-54.244.50.63
  • Simulate connection types and conditions
    • Curated network profile like "3G" or "Lossy LTE"
    • Control parameters like throughput, jitter, loss
  • Built-in compatibility test suite
    • Install, uninstall, launch, and run Fuzz
  • Supply latitude and longitude coordinates

TestObject Device Lab

by Sauce Labs

Automated Testing

  • Access via Api
    • /devices/all, /devices/available
    • /appium/suites/123/devices
  • Supported framework:
    • Appium

Remote Access

  • Displays device screen in browser
  • Uses VNC to transmit mouse and keyboard
  • VNC client on the device
    • Relays back the graphical screen
  • Remote control

Pricing

Advanced Enterprise
Automated $449 / month
Unlimited users
10 concurrent runs		 Ask for price
Individual setting
Manual $349 / month
5 users *
3 parallel sessions		 Ask for price
​Individual setting
  • * Fair use: Limitation on daily usage
    • 1 hour average testing time per user/day

Important FAQs

  • CI: Jeckins+Gradle, Travis
  • Device pool is hosted in Germany
  • For iOS devices, it is currently not possible to test push notifications
  • Whitelist the IP address 37.44.7.110.

Testdroid

by Bitbar

Private Cloud

  • Dedicated cloud devices
  • Infrastructure-as-a-Service
  • Pre-selected real iOS devices hosted in Europe

Public Cloud

  • Cloud-based device farm

On Premise (Enterprise)

  • In-house software solution

Automated Testing

  • Access via api
  • Get results, screenshots, video, performance data
  • Supported frameworks:
    • Appium
    • Calabash
    • Jasmine
    • XCTest (including KIF)
    • XCTest UI

Remote Access

  • iOS manual testing is available on dedicated devices (Public cloud)
    • Dedicated: Reserve devices only for you
  • Enterprise & Private Cloud can make devices available
  • Direct VNC connection

  • Customer servers  whitelist public IPs to allow connections from our devices (Public Cloud)

  • VPN or proxy connection (Private Cloud)

  • Customer has complete freedom on networking (Enterprise cloud)

Pricing - Public Cloud

Team Business Custom
$399 / month $1499 / month Ask
Unlimited Users
Unlimited concurrency
Open REST Api
Test automation
Remote Manual Access
JIRA Integration		 All Team features

+

Highest priority
CS Manager		 Customize

Live support
60 Hours of device time
$0.15 per extra minute
Email support		 250 H device time
$0.13 per extra minute
24/5 support		 Fixed

Pricing - Private Cloud

Private
$349 / Device / Month
Unlimited Users
Unlimited concurrency
Secure VPN Access
Remote Manual Access
Customer Success Manager
24/5 Support

Pricing - Enterprise Cloud

Enterprise
$199 / Device / Month
Unlimited Users
Unlimited concurrency
Open REST Api
Test automation
Remote Manual Access
JIRA Integration
Customer Success Manager
24/5 Support

Important FAQs

  • CI: Jenkins CI, Jira Integration
  • Latency can be expected as the connection is live.
    • (San Jose, CA and Wroclaw, Poland.)

  • In Public Cloud it is not allowed to access device settings

  • Private & Enterprise environments settings can be changed via remote access

  • No recorder for iOS, recommend Xcode Instruments

Xamarin Test Cloud

by Xamarin

Automated Testing

  • Access via Api
    • Xamarin.UITest API
    • Calabash Ruby API for iOS

  • Test any native or hybrid mobile app

    • Not just apps written with Xamarin

Xamarin Test Recorder

  • Still under development
    • Start Test Recorder
    • Connect it to device
    • Use the app
  • Will capture the interactions
  • Outputs a Xamarin.UITest in C#

Pricing

Scale Accelerate Enterprise
$379 / month $799 / month Ask
3 concurrent devices
5 device hours/day
Email support		 5 concurrent devices
10 device hours/day
Email support		 Custom
Unlimited
Email support

Important FAQs

  • CI: Jenkins, TFS and TeamCity
  • No manual testing
  • Xamarin Test Recorder requires iOS 8 or higher

COMPARISON

AWS TESTOBJECT TESTDROID XAMARIN
Rest Api YES YES YES YES
Automated YES YES YES YES
Manual YES YES YES NO
VNC Possibly YES YES NO
In-House NO NO YES ($199) NO
Appium Sup. YES YES YES NO
Calabash Sup. YES NO YES YES
XCTest Sup. YES NO YES NO
Located - Germany Poland -
CI Jenkins Jenkins, Travis Jenkins, JIRA Jenkins, TSF, TeamCity
Automated $449/month  ∞ users - 10 concurrent
Manual $349/month  5 users - 3 concurrent
Pay-as-you-go $0.17  Per device minute
Unlimited $250  Per device slot/month
Team $399/month  ∞ users - ∞ conc. - 60H
Business $1499/month  ∞ users - ∞ conc. - 250H
Private $349/month Per device (∞ user- ∞ conc.)
On-Premise $199/month Per device (∞ user- ∞ conc.)
Scale $379/month 3 conc. - 5 H/day
Accelerate ​ $799/month 5 conc. - 10 H/day

AWS

TestObject

TestDroid

Xamarin

Made with Slides.com