[guardian-dev] Let's pick a real-world emulator for our app testing

[Abel Luck]

Cool stuff.

While Jenkins integration would be nice, I don't think its necessary,
mainly because it would get way to expensive to test all our projects on
multiple devices.

I bet we could get away with doing a round of testing before each app
release, with special runs for troubleshooting device specific bugs.

Apps we might want to test:

- Gibberbot
- StoryMaker
- Orbot
- PixelKnot

I could also see doing runs of our test suite for projects like IOCipher
to benchmark performance etc, but not automatically.

AppThwack lets you do one run across all devices for 25$. That's easily
several $100 a month assuming 1 or 2 runs for each app.

TestDroid is cool, cause you can spend $0.99 to target one device.

I could see us using AppThwack to test on the top 25 devices ($100/mo
per project), then after identifying problem devices, doing individual
device runs on TestDroid ($0.99 per device per app).

Still, not cheap :|

~abel

Mark Belinsky:
> There are so many types of Android devices out there! How do we test our
> apps on all of them? Well, there  are services where a bunch of physical
> and/or emulated devices are hosted and accessible. Question is, which one
> should we go with? Anyone have experience with which ones are good, bad and
> ugly?
> 
> Here's a break down of those that I've found with a cost and performance
> breakdown:
> 
> Real devices:
> * TestDroid
> @ https://cloud.testdroid.com/
> @ * Pay as you go: $0.99/device run. Across 200+ devices
> @ * Premium: $1500/month. Unlimited test runs on max 3 projects. Top
> priority. API into build systems.
> @ * Enterprise: waiting on response
> * Appthwack
> @ * https://appthwack.com/ (pretty site!)
> @ * Hobbyist: $29/month. Across top 10 devices. Unlimited runs. 1 user. 1
> project. (including GS3)
> @ * Indie: $99/month. Across top 25 devices. Unlimited runs. 1 user. 1
> project.
> @ * Pro: $299/month. Across top 180+ devices. Unlimited runs. 1 user. 1
> project. (150 phones / 33 tablets)
> * Perfecto
> @ * http://www.perfectomobile.com/portal/cms/pricing/
> @ * Per Project: $25/hr. Across 100 handsets and tablets
> @ * Subscription: $15-$10/hr. Across 1500 handsets and tablets
> * DeviceAnywhere (ugly)
> @ *
> http://www.keynotedeviceanywhere.com/mobile-application-testing-overview.html
> @ * Monthly: $100/month. $15-$18/hr testing.
> * TestCloud
> @ * https://www.thetestcloud.com/
> @ * Requested an offer. Integration into redmine.
> Emulator:
> * TestObject - http://www.testobject.com/continuous.html - currently in
> beta.
> 
>>From some Googling for articles, it seems that TestDroid is the most
> trusted with Appthwack being an up and comer. Given that TestDroid can
> connect directly into a Jenkins build server, this seems like an awesome
> opportunity for us. However, at the $1500/month price point, it could push
> us into Appthwack land if we don't want to take that hit.
> 
> Best,
> Mark
> 
> --*
> @mbelinsky <https://twitter.com/mbelinsky> | guardianproject.info | phone:
> +1-347-466-9327 | ostel: 1003 **| pgp:
> 0xEFBFA7278D8EFFDA<http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xEFBFA7278D8EFFDA>
> *
> 
> 
> 
> _______________________________________________
> Guardian-dev mailing list
> 
> Post: Guardian-dev at lists.mayfirst.org
> List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev
> 
> To Unsubscribe
>         Send email to:  Guardian-dev-unsubscribe at lists.mayfirst.org
>         Or visit: https://lists.mayfirst.org/mailman/options/guardian-dev/abel%40guardianproject.info
> 
> You are subscribed as: abel at guardianproject.info
>