[guardian-dev] IOCipher with ContentProvider openFile() Method

Aaron Huttner aaron at gryphn.co
Mon Sep 24 18:04:01 EDT 2012


Hey Abel,

The MMS app I'm working on is heavily muti-threaded and I noticed that I
get File Not Found errors at seemingly random intervals. I can see that the
POSIX file system is creating / opening the file through the ADB so I'm
reasonably sure the file does in fact exist. I was going to ask if IOCipher
was thread safe and then I noticed this issue,
https://dev.guardianproject.info/issues/234, so it seems like you're
reasonably sure it is.

Is there a specific way I should be accessing the VFS in other threads? For
example; I mount the DB on the main thread when the user first logs in,
whenever I need to access the DB I just make sure and use an IOCipher
object. I don't specify the database location, just the file location
relative to ROOT in the encrypted DB, does this seem correct? Also, just to
be sure I have added debug statements to make sure the VFS is still mounted
before access, and it is.

I haven't had any luck with getting the file channel transfer method to
work or being able to reproduce the image corruption, but I'm hoping
they're all connected since all of these seem to point to file access
issues.


-Aaron





On Fri, Sep 21, 2012 at 7:14 PM, Aaron Huttner <aaron at gryphn.co> wrote:

> I wasn't able to identify anything different but I'll keep digging until
> I'm positive.
>
> I'll figure out a reliable way to reproduce the corruption and send over
> some sample code.
> On Sep 21, 2012 7:06 PM, "Abel Luck" <abel at guardianproject.info> wrote:
>
>> Aaron,
>>
>> I haven't encountered problems with the FileChannel method.. can you
>> think of anything you're doing differently in your real app compared to
>> the test app?
>>
>> There really isn't a benefit over either method, they do the same thing
>> with a different aPI.
>>
>> This write and read corruption concerns me, we haven't encountered that
>> yet. Any chance you're able to reproduce it?
>>
>> ~abel
>>
>> Aaron Huttner:
>> > Abel,
>> >
>> > I haven't been able to get the FileChannel method to work in practice,
>> > though it works fine in the test app I made (same one I attached to the
>> bug
>> > report). Is there any benefit to one method vs the other? That being
>> said,
>> > even using the ByteArrayOutputStream doesn't seem to be 100% effective.
>> > There are still times when it has trouble saving and when it finally
>> does
>> > it seems the data is corrupt, you can see what I mean in the attached
>> > image. I'm not sure if it's related to the current issue (it seems that
>> way
>> > given it occurs after long write times), I can file a new report if you
>> > think it's warranted. It's also possible that there is something I'm
>> doing
>> > to the data that's not playing nice w/ IOCipher and I just don't
>> realize it
>> > yet.
>> >
>> > In the attached file the glitch is across the top of the image, the
>> blurred
>> > out face is from me in Photoshop... you can see why I needed to write a
>> > secure texting app.
>> >
>> > -Aaron
>> >
>> > Also, I've noticed some corruption when retrieving files
>> >
>> > On Fri, Sep 21, 2012 at 5:11 PM, Abel Luck <abel at guardianproject.info
>> >wrote:
>> >
>> >> Hey Aaron,
>> >>
>> >> Thanks for the bug report :)
>> >>
>> >> Each write call to libsqlfs (done by that Posix method pwriteBytes) has
>> >> some significant SQL overhead. So, while I don't have the data to
>> >> support it yet, my hypothesis is that the SQL overhead is causing poor
>> >> performance.
>> >>
>> >> Your workaround reduces the number of calls to libsqlfs, and hence the
>> >> SQL overhead. We've talked about buffering small writes in IOCipher,
>> >> then flushing them periodically in larger writes to libsqlfs, we'll
>> see..
>> >>
>> >> Another way to implement file copying is using FileChannels. The result
>> >> is the same, a single large write is performed rather than a bunch of
>> >> smaller writes.
>> >> Here's an example snippet https://gist.github.com/3763841
>> >>
>> >> We're still in the process of profiling and optimizing IOCipher and its
>> >> underlying components. Your feedback is quite helpful, definitely keep
>> >> it coming.
>> >>
>> >> Soon, I hope to write some docs for IOCipher describing how using it
>> >> differs from normal Java IO. Using larger buffers is a great example of
>> >> the types of tips I hope to include there.
>> >>
>> >> Cheers,
>> >>
>> >> ~abel
>> >>
>> >> Aaron Huttner:
>> >>> Hey Abel,
>> >>>
>> >>> I saw you were the assignee on the Bug report so I thought I would
>> update
>> >>> you on a work around I'm trying out.
>> >>>
>> >>> I noticed (somewhat obviously) that the bigger I made the buffer the
>> >> faster
>> >>> it would copy the file. I'm not overly familiar w/ the intricacies of
>> >> I/O,
>> >>> let alone encrypted I/O, but it made sense that if the performance
>> >>> increases with fewer writes I'd try to just do one big write. So I
>> tried
>> >>> copying the file to a ByteArrayOutputStream as an intermediary and
>> then
>> >>> just writing to the encrypted VFS in one call
>> >>> (FileOutputStream.write(baos.toByteArray()). This significantly
>> improved
>> >>> write time to almost as quick as the native file system.
>> >>>
>> >>> This seems like a viable work around for me at the moment given the
>> small
>> >>> file sizes I'm dealing with. However, it's probably a bad idea for a
>> >>> general work around.
>> >>>
>> >>> -Aaron
>> >>>
>> >>>
>> >>>
>> >>> On Fri, Sep 21, 2012 at 9:29 AM, Aaron Huttner <aaron at gryphn.co>
>> wrote:
>> >>>
>> >>>> Thanks Hans,
>> >>>>
>> >>>> SQLCipher made a huge difference on our ability to get this app to
>> the
>> >>>> market as fast as we did and I'm really excited to use IOCipher to
>> >> secure
>> >>>> MMS attachments at rest. Honestly, it's like you guys are making
>> exactly
>> >>>> what we need when we need it.
>> >>>>
>> >>>> I'll definitely keep you posted, we've got some cool
>> features/products
>> >> in
>> >>>> the pipe line.
>> >>>>
>> >>>> -Aaron
>> >>>>
>> >>>>
>> >>>> On Thu, Sep 20, 2012 at 5:35 PM, Hans-Christoph Steiner <
>> >>>> hans at guardianproject.info> wrote:
>> >>>>
>> >>>>>
>> >>>>> I just looked into your company.  Sounds like you're making a nice,
>> >>>>> tight Android app for secure messaging.  This is exactly the kind of
>> >>>>> stuff we are aiming to support when we create dev tools like
>> >>>>> SQLCipher-for-Android, IOCipher, Orlib, etc.  So please keep us
>> posted
>> >>>>> on how its working for you so we can incorporate your experience as
>> we
>> >>>>> push things further.
>> >>>>>
>> >>>>> We are also bringing on some more people so we should have more time
>> >>>>> real soon to spend on improving IOCipher.
>> >>>>>
>> >>>>> .hc
>> >>>>>
>> >>>>> On 09/20/2012 04:45 PM, Aaron Huttner wrote:
>> >>>>>> Reported, I'll keep that link bookmarked for the future.
>> >>>>>>
>> >>>>>> thanks.
>> >>>>>>
>> >>>>>> On Thu, Sep 20, 2012 at 4:33 PM, Nathan of Guardian <
>> >>>>>> nathan at guardianproject.info> wrote:
>> >>>>>>
>> >>>>>>>  On 09/21/2012 02:01 AM, Aaron Huttner wrote:
>> >>>>>>>
>> >>>>>>> I'm not sure if you guys have a bug reporting email or not, my
>> >>>>> apologies if
>> >>>>>>> I missed it.
>> >>>>>>>
>> >>>>>>> Anyway, I think I found a bug in IOCipher where it takes forever
>> (~2
>> >>>>>>> minutes in reality) to copy a file from the SD card to the
>> encrypted
>> >>>>> VFS.
>> >>>>>>> The same operation only takes about a second when I don't use
>> >> IOCipher.
>> >>>>>>> After performing a trace it's clear that the bulk of the time,
>> ~99%,
>> >> is
>> >>>>>>> spent in the Posix.pwriteBytes() method of the IOCipher library.
>> I've
>> >>>>>>> attached a sample Android app (VfsTest.zip) and the trace file
>> >>>>>>> (IOCipher.trace) to help illustrate the problem.
>> >>>>>>>
>> >>>>>>>  Thanks! You can report the bug here:
>> >>>>>>>
>> >>>>>>> https://dev.guardianproject.info/projects/iocipher/issues/new
>> >>>>>>>
>> >>>>>>> I have cc'd our core devs on the project.
>> >>>>>>>
>> >>>>>>> +n
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Aaron Huttner, CTO
>> >>>> Gryphn - "Mobile Privacy Simplified"
>> >>>> +1-419-351-9133 | Aaron at gryphn.co
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >
>> >
>>
>>
>>


-- 
Aaron Huttner, CTO
Gryphn - "Mobile Privacy Simplified"
+1-419-351-9133 | Aaron at gryphn.co
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mayfirst.org/pipermail/guardian-dev/attachments/20120924/606f7816/attachment.htm>


More information about the Guardian-dev mailing list