[guardian-dev] IOCipher with ContentProvider openFile() Method

Tue Sep 25 17:50:11 EDT 2012

Hey Aaron,

Some of us are based in NYC, including me.  Nathan used to be, but he's
moved to India.  Come on by our office!  Its at 28 West 27th Street,
room 400.  Just let us know the day before when you're coming so we can
be sure to be there.

As for checking out your code, I'm afraid we can't do that unless we are
looking at free, open source code.  If we look at or work with your
proprietary code, that will open us up to the risk of being sued based
on what we implement and include in IOCipher.

That said, we don't need to look at your whole app in order to work on
your direct uses.  I often find it useful to build up stripped down test
cases when debugging.  If you can build up super simple, standalone test
cases of the problems that you are having, then release those as free
software, then you will keep your app proprietary, and we can work
directly on the problems that you're having.

.hc

On 09/25/2012 03:59 PM, Aaron Huttner wrote:
> Hey Hans,
> 
> 
> 
> While I would love to send over the code and get a second set of eyes on
> the problem, we're in the middle of an investment round and I don't think
> it would go over so well with the investors if we released the code. Is
> there any way we could do a screen share and some pair debugging? I hate
> asking this, especially from an open source shop whose software we depend
> on, but I feel it's best to be honest.
> 
> 
> 
> I do think that our companies have a lot in common and I am genuinely
> interested in fostering a relationship, I wouldn't mind seeing ArmorText on
> the "3rd party apps we recommend" page. I saw that Nathan (a.k.a. the
> benevolent dictator) actually lives in NYC. Our CEO, Navroop Mitter, is
> going to be in NYC next week, would it be possible to set up a meeting
> between the two?
> 
> 
> 
> -Aaron
> 
> On Tue, Sep 25, 2012 at 1:40 PM, Hans-Christoph Steiner <
> hans at guardianproject.info> wrote:
> 
>>
>> So about thread-safe operation, we only know in theory that it should
>> work.  sqlite3 contains locking mechanisms so that you can have multiple
>> processes/threads accessing a given database without causing problems.
>> Since this whole package is based on sqlite3, it uses that mechanism.
>>
>> There might be bugs elsewhere in the whole stack, from libsqlfs to
>> IOCipher.  There is also a current limitation that only one VFS can be
>> open in a given process.  That keeps the API really simple since there
>> is no need to tell a File object which VFS it belongs to or create a
>> special File factory.
>>
>> I'd be happy to take a look at this issue.  It makes it much much easier
>> for us to work on issues if we have a working example, especially if its
>> something we can also include as a test.
>>
>> .hc
>>
>> On 09/24/2012 06:57 PM, Aaron Huttner wrote:
>>> OK, now I'm positive the file really exists. I dumped the contents of the
>>> VFS directory where the images are stored when the error occurred and
>> it's
>>> definitely there.
>>>
>>> 09-24 18:51:05.921: E/Mms/image(8213): IOException caught while opening
>>>>> stream
>>>>
>>>> 09-24 18:51:05.921: E/Mms/image(8213): java.io.FileNotFoundException:
>>>>> /app_parts/PART_1348527038856: open failed: ENOENT (No such file or
>>>>> directory)
>>>>
>>>> 09-24 18:51:05.921: E/Mms/image(8213): at
>>>>> info.guardianproject.libcore.io.IoBridge.open(IoBridge.java:71)
>>>>
>>>> 09-24 18:51:05.921: E/Mms/image(8213): at
>>>>>
>> info.guardianproject.iocipher.FileInputStream.<init>(FileInputStream.java:77)
>>>>
>>>> 09-24 18:51:05.921: E/Mms/image(8213): at
>>>>>
>> com.Gryphn.providers.SecureComMmsProvider.getEncryptedFileInputStream(SecureComMmsProvider.java:789)
>>>>
>>>> 09-24 18:51:05.921: E/Mms/image(8213): at
>>>>> com.Gryphn.mms.ui.UriImage.decodeBoundsInfo(UriImage.java:161)
>>>>
>>>> 09-24 18:51:05.921: E/Mms/image(8213): at
>>>>> com.Gryphn.mms.ui.UriImage.<init>(UriImage.java:88)
>>>>
>>>>
>>>>>
>>>>> 09-24 18:51:05.882: D/aaron(8213): file : PART_1345582002082
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345227415477
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345582101400
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345581716950
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345581804514
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504562948
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504584604
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504907727
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504999361
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505134414
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505198340
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505429032
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505939191
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506460355
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506792261
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506814727
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506842859
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506884314
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506917184
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348521829368
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348522316909
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348522434937
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348522453095
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348524511213
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348524531937
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348526895568
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348526918403
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348526985928
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348527011324
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348527038856
>>>>
>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348527063465
>>>>
>>>>
>>> On Mon, Sep 24, 2012 at 6:04 PM, Aaron Huttner <aaron at gryphn.co> wrote:
>>>
>>>> Hey Abel,
>>>>
>>>> The MMS app I'm working on is heavily muti-threaded and I noticed that I
>>>> get File Not Found errors at seemingly random intervals. I can see that
>> the
>>>> POSIX file system is creating / opening the file through the ADB so I'm
>>>> reasonably sure the file does in fact exist. I was going to ask if
>> IOCipher
>>>> was thread safe and then I noticed this issue,
>>>> https://dev.guardianproject.info/issues/234, so it seems like you're
>>>> reasonably sure it is.
>>>>
>>>> Is there a specific way I should be accessing the VFS in other threads?
>>>> For example; I mount the DB on the main thread when the user first logs
>> in,
>>>> whenever I need to access the DB I just make sure and use an IOCipher
>>>> object. I don't specify the database location, just the file location
>>>> relative to ROOT in the encrypted DB, does this seem correct? Also,
>> just to
>>>> be sure I have added debug statements to make sure the VFS is still
>> mounted
>>>> before access, and it is.
>>>>
>>>> I haven't had any luck with getting the file channel transfer method to
>>>> work or being able to reproduce the image corruption, but I'm hoping
>>>> they're all connected since all of these seem to point to file access
>>>> issues.
>>>>
>>>>
>>>> -Aaron
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Sep 21, 2012 at 7:14 PM, Aaron Huttner <aaron at gryphn.co> wrote:
>>>>
>>>>> I wasn't able to identify anything different but I'll keep digging
>> until
>>>>> I'm positive.
>>>>>
>>>>> I'll figure out a reliable way to reproduce the corruption and send
>> over
>>>>> some sample code.
>>>>> On Sep 21, 2012 7:06 PM, "Abel Luck" <abel at guardianproject.info>
>> wrote:
>>>>>
>>>>>> Aaron,
>>>>>>
>>>>>> I haven't encountered problems with the FileChannel method.. can you
>>>>>> think of anything you're doing differently in your real app compared
>> to
>>>>>> the test app?
>>>>>>
>>>>>> There really isn't a benefit over either method, they do the same
>> thing
>>>>>> with a different aPI.
>>>>>>
>>>>>> This write and read corruption concerns me, we haven't encountered
>> that
>>>>>> yet. Any chance you're able to reproduce it?
>>>>>>
>>>>>> ~abel
>>>>>>
>>>>>> Aaron Huttner:
>>>>>>> Abel,
>>>>>>>
>>>>>>> I haven't been able to get the FileChannel method to work in
>> practice,
>>>>>>> though it works fine in the test app I made (same one I attached to
>>>>>> the bug
>>>>>>> report). Is there any benefit to one method vs the other? That being
>>>>>> said,
>>>>>>> even using the ByteArrayOutputStream doesn't seem to be 100%
>> effective.
>>>>>>> There are still times when it has trouble saving and when it finally
>>>>>> does
>>>>>>> it seems the data is corrupt, you can see what I mean in the attached
>>>>>>> image. I'm not sure if it's related to the current issue (it seems
>>>>>> that way
>>>>>>> given it occurs after long write times), I can file a new report if
>> you
>>>>>>> think it's warranted. It's also possible that there is something I'm
>>>>>> doing
>>>>>>> to the data that's not playing nice w/ IOCipher and I just don't
>>>>>> realize it
>>>>>>> yet.
>>>>>>>
>>>>>>> In the attached file the glitch is across the top of the image, the
>>>>>> blurred
>>>>>>> out face is from me in Photoshop... you can see why I needed to
>> write a
>>>>>>> secure texting app.
>>>>>>>
>>>>>>> -Aaron
>>>>>>>
>>>>>>> Also, I've noticed some corruption when retrieving files
>>>>>>>
>>>>>>> On Fri, Sep 21, 2012 at 5:11 PM, Abel Luck <
>> abel at guardianproject.info
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey Aaron,
>>>>>>>>
>>>>>>>> Thanks for the bug report :)
>>>>>>>>
>>>>>>>> Each write call to libsqlfs (done by that Posix method pwriteBytes)
>>>>>> has
>>>>>>>> some significant SQL overhead. So, while I don't have the data to
>>>>>>>> support it yet, my hypothesis is that the SQL overhead is causing
>> poor
>>>>>>>> performance.
>>>>>>>>
>>>>>>>> Your workaround reduces the number of calls to libsqlfs, and hence
>> the
>>>>>>>> SQL overhead. We've talked about buffering small writes in IOCipher,
>>>>>>>> then flushing them periodically in larger writes to libsqlfs, we'll
>>>>>> see..
>>>>>>>>
>>>>>>>> Another way to implement file copying is using FileChannels. The
>>>>>> result
>>>>>>>> is the same, a single large write is performed rather than a bunch
>> of
>>>>>>>> smaller writes.
>>>>>>>> Here's an example snippet https://gist.github.com/3763841
>>>>>>>>
>>>>>>>> We're still in the process of profiling and optimizing IOCipher and
>>>>>> its
>>>>>>>> underlying components. Your feedback is quite helpful, definitely
>> keep
>>>>>>>> it coming.
>>>>>>>>
>>>>>>>> Soon, I hope to write some docs for IOCipher describing how using it
>>>>>>>> differs from normal Java IO. Using larger buffers is a great example
>>>>>> of
>>>>>>>> the types of tips I hope to include there.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> ~abel
>>>>>>>>
>>>>>>>> Aaron Huttner:
>>>>>>>>> Hey Abel,
>>>>>>>>>
>>>>>>>>> I saw you were the assignee on the Bug report so I thought I would
>>>>>> update
>>>>>>>>> you on a work around I'm trying out.
>>>>>>>>>
>>>>>>>>> I noticed (somewhat obviously) that the bigger I made the buffer
>> the
>>>>>>>> faster
>>>>>>>>> it would copy the file. I'm not overly familiar w/ the intricacies
>> of
>>>>>>>> I/O,
>>>>>>>>> let alone encrypted I/O, but it made sense that if the performance
>>>>>>>>> increases with fewer writes I'd try to just do one big write. So I
>>>>>> tried
>>>>>>>>> copying the file to a ByteArrayOutputStream as an intermediary and
>>>>>> then
>>>>>>>>> just writing to the encrypted VFS in one call
>>>>>>>>> (FileOutputStream.write(baos.toByteArray()). This significantly
>>>>>> improved
>>>>>>>>> write time to almost as quick as the native file system.
>>>>>>>>>
>>>>>>>>> This seems like a viable work around for me at the moment given the
>>>>>> small
>>>>>>>>> file sizes I'm dealing with. However, it's probably a bad idea for
>> a
>>>>>>>>> general work around.
>>>>>>>>>
>>>>>>>>> -Aaron
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Sep 21, 2012 at 9:29 AM, Aaron Huttner <aaron at gryphn.co>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Hans,
>>>>>>>>>>
>>>>>>>>>> SQLCipher made a huge difference on our ability to get this app to
>>>>>> the
>>>>>>>>>> market as fast as we did and I'm really excited to use IOCipher to
>>>>>>>> secure
>>>>>>>>>> MMS attachments at rest. Honestly, it's like you guys are making
>>>>>> exactly
>>>>>>>>>> what we need when we need it.
>>>>>>>>>>
>>>>>>>>>> I'll definitely keep you posted, we've got some cool
>>>>>> features/products
>>>>>>>> in
>>>>>>>>>> the pipe line.
>>>>>>>>>>
>>>>>>>>>> -Aaron
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Sep 20, 2012 at 5:35 PM, Hans-Christoph Steiner <
>>>>>>>>>> hans at guardianproject.info> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I just looked into your company.  Sounds like you're making a
>> nice,
>>>>>>>>>>> tight Android app for secure messaging.  This is exactly the kind
>>>>>> of
>>>>>>>>>>> stuff we are aiming to support when we create dev tools like
>>>>>>>>>>> SQLCipher-for-Android, IOCipher, Orlib, etc.  So please keep us
>>>>>> posted
>>>>>>>>>>> on how its working for you so we can incorporate your experience
>>>>>> as we
>>>>>>>>>>> push things further.
>>>>>>>>>>>
>>>>>>>>>>> We are also bringing on some more people so we should have more
>>>>>> time
>>>>>>>>>>> real soon to spend on improving IOCipher.
>>>>>>>>>>>
>>>>>>>>>>> .hc
>>>>>>>>>>>
>>>>>>>>>>> On 09/20/2012 04:45 PM, Aaron Huttner wrote:
>>>>>>>>>>>> Reported, I'll keep that link bookmarked for the future.
>>>>>>>>>>>>
>>>>>>>>>>>> thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Sep 20, 2012 at 4:33 PM, Nathan of Guardian <
>>>>>>>>>>>> nathan at guardianproject.info> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>  On 09/21/2012 02:01 AM, Aaron Huttner wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure if you guys have a bug reporting email or not, my
>>>>>>>>>>> apologies if
>>>>>>>>>>>>> I missed it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyway, I think I found a bug in IOCipher where it takes
>> forever
>>>>>> (~2
>>>>>>>>>>>>> minutes in reality) to copy a file from the SD card to the
>>>>>> encrypted
>>>>>>>>>>> VFS.
>>>>>>>>>>>>> The same operation only takes about a second when I don't use
>>>>>>>> IOCipher.
>>>>>>>>>>>>> After performing a trace it's clear that the bulk of the time,
>>>>>> ~99%,
>>>>>>>> is
>>>>>>>>>>>>> spent in the Posix.pwriteBytes() method of the IOCipher
>> library.
>>>>>> I've
>>>>>>>>>>>>> attached a sample Android app (VfsTest.zip) and the trace file
>>>>>>>>>>>>> (IOCipher.trace) to help illustrate the problem.
>>>>>>>>>>>>>
>>>>>>>>>>>>>  Thanks! You can report the bug here:
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://dev.guardianproject.info/projects/iocipher/issues/new
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have cc'd our core devs on the project.
>>>>>>>>>>>>>
>>>>>>>>>>>>> +n
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Aaron Huttner, CTO
>>>>>>>>>> Gryphn - "Mobile Privacy Simplified"
>>>>>>>>>> +1-419-351-9133 | Aaron at gryphn.co
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Aaron Huttner, CTO
>>>> Gryphn - "Mobile Privacy Simplified"
>>>> +1-419-351-9133 | Aaron at gryphn.co
>>>>
>>>
>>>
>>>
>>
> 
> 
> 