[guardian-dev] IOCipher with ContentProvider openFile() Method

Tue Sep 25 15:59:09 EDT 2012

Hey Hans,

While I would love to send over the code and get a second set of eyes on
the problem, we're in the middle of an investment round and I don't think
it would go over so well with the investors if we released the code. Is
there any way we could do a screen share and some pair debugging? I hate
asking this, especially from an open source shop whose software we depend
on, but I feel it's best to be honest.

I do think that our companies have a lot in common and I am genuinely
interested in fostering a relationship, I wouldn't mind seeing ArmorText on
the "3rd party apps we recommend" page. I saw that Nathan (a.k.a. the
benevolent dictator) actually lives in NYC. Our CEO, Navroop Mitter, is
going to be in NYC next week, would it be possible to set up a meeting
between the two?

-Aaron

On Tue, Sep 25, 2012 at 1:40 PM, Hans-Christoph Steiner <
hans at guardianproject.info> wrote:

>
> So about thread-safe operation, we only know in theory that it should
> work.  sqlite3 contains locking mechanisms so that you can have multiple
> processes/threads accessing a given database without causing problems.
> Since this whole package is based on sqlite3, it uses that mechanism.
>
> There might be bugs elsewhere in the whole stack, from libsqlfs to
> IOCipher.  There is also a current limitation that only one VFS can be
> open in a given process.  That keeps the API really simple since there
> is no need to tell a File object which VFS it belongs to or create a
> special File factory.
>
> I'd be happy to take a look at this issue.  It makes it much much easier
> for us to work on issues if we have a working example, especially if its
> something we can also include as a test.
>
> .hc
>
> On 09/24/2012 06:57 PM, Aaron Huttner wrote:
> > OK, now I'm positive the file really exists. I dumped the contents of the
> > VFS directory where the images are stored when the error occurred and
> it's
> > definitely there.
> >
> > 09-24 18:51:05.921: E/Mms/image(8213): IOException caught while opening
> >>> stream
> >>
> >> 09-24 18:51:05.921: E/Mms/image(8213): java.io.FileNotFoundException:
> >>> /app_parts/PART_1348527038856: open failed: ENOENT (No such file or
> >>> directory)
> >>
> >> 09-24 18:51:05.921: E/Mms/image(8213): at
> >>> info.guardianproject.libcore.io.IoBridge.open(IoBridge.java:71)
> >>
> >> 09-24 18:51:05.921: E/Mms/image(8213): at
> >>>
> info.guardianproject.iocipher.FileInputStream.<init>(FileInputStream.java:77)
> >>
> >> 09-24 18:51:05.921: E/Mms/image(8213): at
> >>>
> com.Gryphn.providers.SecureComMmsProvider.getEncryptedFileInputStream(SecureComMmsProvider.java:789)
> >>
> >> 09-24 18:51:05.921: E/Mms/image(8213): at
> >>> com.Gryphn.mms.ui.UriImage.decodeBoundsInfo(UriImage.java:161)
> >>
> >> 09-24 18:51:05.921: E/Mms/image(8213): at
> >>> com.Gryphn.mms.ui.UriImage.<init>(UriImage.java:88)
> >>
> >>
> >>>
> >>> 09-24 18:51:05.882: D/aaron(8213): file : PART_1345582002082
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345227415477
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345582101400
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345581716950
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345581804514
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504562948
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504584604
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504907727
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504999361
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505134414
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505198340
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505429032
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505939191
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506460355
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506792261
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506814727
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506842859
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506884314
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506917184
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348521829368
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348522316909
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348522434937
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348522453095
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348524511213
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348524531937
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348526895568
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348526918403
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348526985928
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348527011324
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348527038856
> >>
> >> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348527063465
> >>
> >>
> > On Mon, Sep 24, 2012 at 6:04 PM, Aaron Huttner <aaron at gryphn.co> wrote:
> >
> >> Hey Abel,
> >>
> >> The MMS app I'm working on is heavily muti-threaded and I noticed that I
> >> get File Not Found errors at seemingly random intervals. I can see that
> the
> >> POSIX file system is creating / opening the file through the ADB so I'm
> >> reasonably sure the file does in fact exist. I was going to ask if
> IOCipher
> >> was thread safe and then I noticed this issue,
> >> https://dev.guardianproject.info/issues/234, so it seems like you're
> >> reasonably sure it is.
> >>
> >> Is there a specific way I should be accessing the VFS in other threads?
> >> For example; I mount the DB on the main thread when the user first logs
> in,
> >> whenever I need to access the DB I just make sure and use an IOCipher
> >> object. I don't specify the database location, just the file location
> >> relative to ROOT in the encrypted DB, does this seem correct? Also,
> just to
> >> be sure I have added debug statements to make sure the VFS is still
> mounted
> >> before access, and it is.
> >>
> >> I haven't had any luck with getting the file channel transfer method to
> >> work or being able to reproduce the image corruption, but I'm hoping
> >> they're all connected since all of these seem to point to file access
> >> issues.
> >>
> >>
> >> -Aaron
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Sep 21, 2012 at 7:14 PM, Aaron Huttner <aaron at gryphn.co> wrote:
> >>
> >>> I wasn't able to identify anything different but I'll keep digging
> until
> >>> I'm positive.
> >>>
> >>> I'll figure out a reliable way to reproduce the corruption and send
> over
> >>> some sample code.
> >>> On Sep 21, 2012 7:06 PM, "Abel Luck" <abel at guardianproject.info>
> wrote:
> >>>
> >>>> Aaron,
> >>>>
> >>>> I haven't encountered problems with the FileChannel method.. can you
> >>>> think of anything you're doing differently in your real app compared
> to
> >>>> the test app?
> >>>>
> >>>> There really isn't a benefit over either method, they do the same
> thing
> >>>> with a different aPI.
> >>>>
> >>>> This write and read corruption concerns me, we haven't encountered
> that
> >>>> yet. Any chance you're able to reproduce it?
> >>>>
> >>>> ~abel
> >>>>
> >>>> Aaron Huttner:
> >>>>> Abel,
> >>>>>
> >>>>> I haven't been able to get the FileChannel method to work in
> practice,
> >>>>> though it works fine in the test app I made (same one I attached to
> >>>> the bug
> >>>>> report). Is there any benefit to one method vs the other? That being
> >>>> said,
> >>>>> even using the ByteArrayOutputStream doesn't seem to be 100%
> effective.
> >>>>> There are still times when it has trouble saving and when it finally
> >>>> does
> >>>>> it seems the data is corrupt, you can see what I mean in the attached
> >>>>> image. I'm not sure if it's related to the current issue (it seems
> >>>> that way
> >>>>> given it occurs after long write times), I can file a new report if
> you
> >>>>> think it's warranted. It's also possible that there is something I'm
> >>>> doing
> >>>>> to the data that's not playing nice w/ IOCipher and I just don't
> >>>> realize it
> >>>>> yet.
> >>>>>
> >>>>> In the attached file the glitch is across the top of the image, the
> >>>> blurred
> >>>>> out face is from me in Photoshop... you can see why I needed to
> write a
> >>>>> secure texting app.
> >>>>>
> >>>>> -Aaron
> >>>>>
> >>>>> Also, I've noticed some corruption when retrieving files
> >>>>>
> >>>>> On Fri, Sep 21, 2012 at 5:11 PM, Abel Luck <
> abel at guardianproject.info
> >>>>> wrote:
> >>>>>
> >>>>>> Hey Aaron,
> >>>>>>
> >>>>>> Thanks for the bug report :)
> >>>>>>
> >>>>>> Each write call to libsqlfs (done by that Posix method pwriteBytes)
> >>>> has
> >>>>>> some significant SQL overhead. So, while I don't have the data to
> >>>>>> support it yet, my hypothesis is that the SQL overhead is causing
> poor
> >>>>>> performance.
> >>>>>>
> >>>>>> Your workaround reduces the number of calls to libsqlfs, and hence
> the
> >>>>>> SQL overhead. We've talked about buffering small writes in IOCipher,
> >>>>>> then flushing them periodically in larger writes to libsqlfs, we'll
> >>>> see..
> >>>>>>
> >>>>>> Another way to implement file copying is using FileChannels. The
> >>>> result
> >>>>>> is the same, a single large write is performed rather than a bunch
> of
> >>>>>> smaller writes.
> >>>>>> Here's an example snippet https://gist.github.com/3763841
> >>>>>>
> >>>>>> We're still in the process of profiling and optimizing IOCipher and
> >>>> its
> >>>>>> underlying components. Your feedback is quite helpful, definitely
> keep
> >>>>>> it coming.
> >>>>>>
> >>>>>> Soon, I hope to write some docs for IOCipher describing how using it
> >>>>>> differs from normal Java IO. Using larger buffers is a great example
> >>>> of
> >>>>>> the types of tips I hope to include there.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> ~abel
> >>>>>>
> >>>>>> Aaron Huttner:
> >>>>>>> Hey Abel,
> >>>>>>>
> >>>>>>> I saw you were the assignee on the Bug report so I thought I would
> >>>> update
> >>>>>>> you on a work around I'm trying out.
> >>>>>>>
> >>>>>>> I noticed (somewhat obviously) that the bigger I made the buffer
> the
> >>>>>> faster
> >>>>>>> it would copy the file. I'm not overly familiar w/ the intricacies
> of
> >>>>>> I/O,
> >>>>>>> let alone encrypted I/O, but it made sense that if the performance
> >>>>>>> increases with fewer writes I'd try to just do one big write. So I
> >>>> tried
> >>>>>>> copying the file to a ByteArrayOutputStream as an intermediary and
> >>>> then
> >>>>>>> just writing to the encrypted VFS in one call
> >>>>>>> (FileOutputStream.write(baos.toByteArray()). This significantly
> >>>> improved
> >>>>>>> write time to almost as quick as the native file system.
> >>>>>>>
> >>>>>>> This seems like a viable work around for me at the moment given the
> >>>> small
> >>>>>>> file sizes I'm dealing with. However, it's probably a bad idea for
> a
> >>>>>>> general work around.
> >>>>>>>
> >>>>>>> -Aaron
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Sep 21, 2012 at 9:29 AM, Aaron Huttner <aaron at gryphn.co>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Thanks Hans,
> >>>>>>>>
> >>>>>>>> SQLCipher made a huge difference on our ability to get this app to
> >>>> the
> >>>>>>>> market as fast as we did and I'm really excited to use IOCipher to
> >>>>>> secure
> >>>>>>>> MMS attachments at rest. Honestly, it's like you guys are making
> >>>> exactly
> >>>>>>>> what we need when we need it.
> >>>>>>>>
> >>>>>>>> I'll definitely keep you posted, we've got some cool
> >>>> features/products
> >>>>>> in
> >>>>>>>> the pipe line.
> >>>>>>>>
> >>>>>>>> -Aaron
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Thu, Sep 20, 2012 at 5:35 PM, Hans-Christoph Steiner <
> >>>>>>>> hans at guardianproject.info> wrote:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I just looked into your company.  Sounds like you're making a
> nice,
> >>>>>>>>> tight Android app for secure messaging.  This is exactly the kind
> >>>> of
> >>>>>>>>> stuff we are aiming to support when we create dev tools like
> >>>>>>>>> SQLCipher-for-Android, IOCipher, Orlib, etc.  So please keep us
> >>>> posted
> >>>>>>>>> on how its working for you so we can incorporate your experience
> >>>> as we
> >>>>>>>>> push things further.
> >>>>>>>>>
> >>>>>>>>> We are also bringing on some more people so we should have more
> >>>> time
> >>>>>>>>> real soon to spend on improving IOCipher.
> >>>>>>>>>
> >>>>>>>>> .hc
> >>>>>>>>>
> >>>>>>>>> On 09/20/2012 04:45 PM, Aaron Huttner wrote:
> >>>>>>>>>> Reported, I'll keep that link bookmarked for the future.
> >>>>>>>>>>
> >>>>>>>>>> thanks.
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Sep 20, 2012 at 4:33 PM, Nathan of Guardian <
> >>>>>>>>>> nathan at guardianproject.info> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>  On 09/21/2012 02:01 AM, Aaron Huttner wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> I'm not sure if you guys have a bug reporting email or not, my
> >>>>>>>>> apologies if
> >>>>>>>>>>> I missed it.
> >>>>>>>>>>>
> >>>>>>>>>>> Anyway, I think I found a bug in IOCipher where it takes
> forever
> >>>> (~2
> >>>>>>>>>>> minutes in reality) to copy a file from the SD card to the
> >>>> encrypted
> >>>>>>>>> VFS.
> >>>>>>>>>>> The same operation only takes about a second when I don't use
> >>>>>> IOCipher.
> >>>>>>>>>>> After performing a trace it's clear that the bulk of the time,
> >>>> ~99%,
> >>>>>> is
> >>>>>>>>>>> spent in the Posix.pwriteBytes() method of the IOCipher
> library.
> >>>> I've
> >>>>>>>>>>> attached a sample Android app (VfsTest.zip) and the trace file
> >>>>>>>>>>> (IOCipher.trace) to help illustrate the problem.
> >>>>>>>>>>>
> >>>>>>>>>>>  Thanks! You can report the bug here:
> >>>>>>>>>>>
> >>>>>>>>>>> https://dev.guardianproject.info/projects/iocipher/issues/new
> >>>>>>>>>>>
> >>>>>>>>>>> I have cc'd our core devs on the project.
> >>>>>>>>>>>
> >>>>>>>>>>> +n
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Aaron Huttner, CTO
> >>>>>>>> Gryphn - "Mobile Privacy Simplified"
> >>>>>>>> +1-419-351-9133 | Aaron at gryphn.co
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>
> >>
> >> --
> >> Aaron Huttner, CTO
> >> Gryphn - "Mobile Privacy Simplified"
> >> +1-419-351-9133 | Aaron at gryphn.co
> >>
> >
> >
> >
>

-- 
Aaron Huttner, CTO
Gryphn - "Mobile Privacy Simplified"
+1-419-351-9133 | Aaron at gryphn.co
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mayfirst.org/pipermail/guardian-dev/attachments/20120925/79e4d279/attachment-0001.htm>