[guardian-dev] Let's pick a real-world emulator for our app testing

Tue Apr 30 18:15:30 EDT 2013

Something else to consider is that most lot of these sites tie you to a
specific functional ui test framework, some use standard stuff like
Robotium, but it looks like Google is about to release Espresso, a new UI
resting framework that Doesn't Suck.  It might be worth waiting for it's
unveil (presumably at IO in a few weeks, but they aren't saying) and see
which of these sites jumps on to support it the quickest.  Espresso really
does look insanely better than Robotium et al:

A few talks showing it off:

Espresso for Android - a Demo
http://www.youtube.com/watch?v=qtKx1WxK7cw&feature=youtu.be

GTAC 2013: Espresso: Fresh Start to Android UI Testing
http://www.youtube.com/watch?v=T7ugmCuNxDU&feature=youtu.be

GTAC 2013: Breaking the Matrix - Android Testing at Scale
http://www.youtube.com/watch?v=uHoB0KzQGRg

-Josh

On Fri, Apr 26, 2013 at 5:21 AM, Abel Luck <abel at guardianproject.info>wrote:

> Ack! I mis-read Appthwack's plans. For $99/mo per project you get
> unlimited runs on the top 25 devices, *not* just 1 run :)
>
> So, if we wanted to test the apps I listed below, that would be $400 /
> mo for unlimited runs on 25 devices.
>
> Or $900 / mo to test them on 140+ devices.
>
> Way cheaper than TestDroid's 1500$/mo plan (which only lets you run 3
> projects).
>
> ~abel
>
> Abel Luck:
> > Cool stuff.
> >
> > While Jenkins integration would be nice, I don't think its necessary,
> > mainly because it would get way to expensive to test all our projects on
> > multiple devices.
> >
> > I bet we could get away with doing a round of testing before each app
> > release, with special runs for troubleshooting device specific bugs.
> >
> > Apps we might want to test:
> >
> > - Gibberbot
> > - StoryMaker
> > - Orbot
> > - PixelKnot
> >
> > I could also see doing runs of our test suite for projects like IOCipher
> > to benchmark performance etc, but not automatically.
> >
> > AppThwack lets you do one run across all devices for 25$. That's easily
> > several $100 a month assuming 1 or 2 runs for each app.
> >
> > TestDroid is cool, cause you can spend $0.99 to target one device.
> >
> > I could see us using AppThwack to test on the top 25 devices ($100/mo
> > per project), then after identifying problem devices, doing individual
> > device runs on TestDroid ($0.99 per device per app).
> >
> > Still, not cheap :|
> >
> > ~abel
> >
> > Mark Belinsky:
> >> There are so many types of Android devices out there! How do we test our
> >> apps on all of them? Well, there  are services where a bunch of physical
> >> and/or emulated devices are hosted and accessible. Question is, which
> one
> >> should we go with? Anyone have experience with which ones are good, bad
> and
> >> ugly?
> >>
> >> Here's a break down of those that I've found with a cost and performance
> >> breakdown:
> >>
> >> Real devices:
> >> * TestDroid
> >> @ https://cloud.testdroid.com/
> >> @ * Pay as you go: $0.99/device run. Across 200+ devices
> >> @ * Premium: $1500/month. Unlimited test runs on max 3 projects. Top
> >> priority. API into build systems.
> >> @ * Enterprise: waiting on response
> >> * Appthwack
> >> @ * https://appthwack.com/ (pretty site!)
> >> @ * Hobbyist: $29/month. Across top 10 devices. Unlimited runs. 1 user.
> 1
> >> project. (including GS3)
> >> @ * Indie: $99/month. Across top 25 devices. Unlimited runs. 1 user. 1
> >> project.
> >> @ * Pro: $299/month. Across top 180+ devices. Unlimited runs. 1 user. 1
> >> project. (150 phones / 33 tablets)
> >> * Perfecto
> >> @ * http://www.perfectomobile.com/portal/cms/pricing/
> >> @ * Per Project: $25/hr. Across 100 handsets and tablets
> >> @ * Subscription: $15-$10/hr. Across 1500 handsets and tablets
> >> * DeviceAnywhere (ugly)
> >> @ *
> >>
> http://www.keynotedeviceanywhere.com/mobile-application-testing-overview.html
> >> @ * Monthly: $100/month. $15-$18/hr testing.
> >> * TestCloud
> >> @ * https://www.thetestcloud.com/
> >> @ * Requested an offer. Integration into redmine.
> >> Emulator:
> >> * TestObject - http://www.testobject.com/continuous.html - currently in
> >> beta.
> >>
> >> >From some Googling for articles, it seems that TestDroid is the most
> >> trusted with Appthwack being an up and comer. Given that TestDroid can
> >> connect directly into a Jenkins build server, this seems like an awesome
> >> opportunity for us. However, at the $1500/month price point, it could
> push
> >> us into Appthwack land if we don't want to take that hit.
> >>
> >> Best,
> >> Mark
> >>
> >> --*
> >> @mbelinsky <https://twitter.com/mbelinsky> | guardianproject.info |
> phone:
> >> +1-347-466-9327 | ostel: 1003 **| pgp:
> >> 0xEFBFA7278D8EFFDA<
> http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xEFBFA7278D8EFFDA>
> >> *
> >>
> >>
> >>
> >> _______________________________________________
> >> Guardian-dev mailing list
> >>
> >> Post: Guardian-dev at lists.mayfirst.org
> >> List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev
> >>
> >> To Unsubscribe
> >>         Send email to:  Guardian-dev-unsubscribe at lists.mayfirst.org
> >>         Or visit:
> https://lists.mayfirst.org/mailman/options/guardian-dev/abel%40guardianproject.info
> >>
> >> You are subscribed as: abel at guardianproject.info
> >>
> >
> > _______________________________________________
> > Guardian-dev mailing list
> >
> > Post: Guardian-dev at lists.mayfirst.org
> > List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev
> >
> > To Unsubscribe
> >         Send email to:  Guardian-dev-unsubscribe at lists.mayfirst.org
> >         Or visit:
> https://lists.mayfirst.org/mailman/options/guardian-dev/abel%40guardianproject.info
> >
> > You are subscribed as: abel at guardianproject.info
> >
>
> _______________________________________________
> Guardian-dev mailing list
>
> Post: Guardian-dev at lists.mayfirst.org
> List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev
>
> To Unsubscribe
>         Send email to:  Guardian-dev-unsubscribe at lists.mayfirst.org
>         Or visit:
> https://lists.mayfirst.org/mailman/options/guardian-dev/josh%40vitriolix.com
>
> You are subscribed as: josh at vitriolix.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mayfirst.org/pipermail/guardian-dev/attachments/20130430/a00bd503/attachment.html>