[guardian-dev] IOCipher with ContentProvider openFile() Method

[Hans-Christoph Steiner]

Perfect, tests on github work well for us.  Any free software license is
fine by us, but to keep it simple, I recommend using LGPLv3 since that
the license that IOCipher uses.  Then we can include your test code
straight into the IOCipherTests project without needing copyright
assignment or anything (i.e. the copyright will remain with the original
author).

.hc

On 09/26/2012 03:05 PM, Abel Luck wrote:
> Putting it on github with any opensource license will be fine. Remember,
> since you own guys the copyright to the code, the license doesn't affect
> how you can use the code internally :)
> 
> Sorry for the hoop jumping, but as an not-for-profit opensource project,
> we just need to have our bases covered.
> 
> I'm excited to help you fix these issues. This is the exact sort of real
> life testing we need to make IOCipher awesome.
> 
> By the way, if you use Jabber/XMPP you can add me at
> abel at guardianproject.info, or IRC #guardianproject on freenode.
> 
> ~abel
> 
> Aaron Huttner:
>> Hmm.. India might not be a deal breaker. The two other co-founders
>> (including Navroop) are Indian and have family over there they visit
>> periodically. India's a big place but maybe down the road they they could
>> meet up. Regardless, I'm sure he'd still like to drop by and meet you guys.
>> I'll set it up and let you know.
>>
>> I'll work on creating a stripped down test case to release. If I just throw
>> it up on a public GitHub project when I'm done will that be enough to
>> constitute free/open source software for you or should I include some text
>> with a specific license (GNU/MIT) with it?
>>
>> -Aaron
>>
>> On Tue, Sep 25, 2012 at 5:50 PM, Hans-Christoph Steiner <
>> hans at guardianproject.info> wrote:
>>
>>>
>>> Hey Aaron,
>>>
>>> Some of us are based in NYC, including me.  Nathan used to be, but he's
>>> moved to India.  Come on by our office!  Its at 28 West 27th Street,
>>> room 400.  Just let us know the day before when you're coming so we can
>>> be sure to be there.
>>>
>>> As for checking out your code, I'm afraid we can't do that unless we are
>>> looking at free, open source code.  If we look at or work with your
>>> proprietary code, that will open us up to the risk of being sued based
>>> on what we implement and include in IOCipher.
>>>
>>> That said, we don't need to look at your whole app in order to work on
>>> your direct uses.  I often find it useful to build up stripped down test
>>> cases when debugging.  If you can build up super simple, standalone test
>>> cases of the problems that you are having, then release those as free
>>> software, then you will keep your app proprietary, and we can work
>>> directly on the problems that you're having.
>>>
>>> .hc
>>>
>>>
>>> On 09/25/2012 03:59 PM, Aaron Huttner wrote:
>>>> Hey Hans,
>>>>
>>>>
>>>>
>>>> While I would love to send over the code and get a second set of eyes on
>>>> the problem, we're in the middle of an investment round and I don't think
>>>> it would go over so well with the investors if we released the code. Is
>>>> there any way we could do a screen share and some pair debugging? I hate
>>>> asking this, especially from an open source shop whose software we depend
>>>> on, but I feel it's best to be honest.
>>>>
>>>>
>>>>
>>>> I do think that our companies have a lot in common and I am genuinely
>>>> interested in fostering a relationship, I wouldn't mind seeing ArmorText
>>> on
>>>> the "3rd party apps we recommend" page. I saw that Nathan (a.k.a. the
>>>> benevolent dictator) actually lives in NYC. Our CEO, Navroop Mitter, is
>>>> going to be in NYC next week, would it be possible to set up a meeting
>>>> between the two?
>>>>
>>>>
>>>>
>>>> -Aaron
>>>>
>>>> On Tue, Sep 25, 2012 at 1:40 PM, Hans-Christoph Steiner <
>>>> hans at guardianproject.info> wrote:
>>>>
>>>>>
>>>>> So about thread-safe operation, we only know in theory that it should
>>>>> work.  sqlite3 contains locking mechanisms so that you can have multiple
>>>>> processes/threads accessing a given database without causing problems.
>>>>> Since this whole package is based on sqlite3, it uses that mechanism.
>>>>>
>>>>> There might be bugs elsewhere in the whole stack, from libsqlfs to
>>>>> IOCipher.  There is also a current limitation that only one VFS can be
>>>>> open in a given process.  That keeps the API really simple since there
>>>>> is no need to tell a File object which VFS it belongs to or create a
>>>>> special File factory.
>>>>>
>>>>> I'd be happy to take a look at this issue.  It makes it much much easier
>>>>> for us to work on issues if we have a working example, especially if its
>>>>> something we can also include as a test.
>>>>>
>>>>> .hc
>>>>>
>>>>> On 09/24/2012 06:57 PM, Aaron Huttner wrote:
>>>>>> OK, now I'm positive the file really exists. I dumped the contents of
>>> the
>>>>>> VFS directory where the images are stored when the error occurred and
>>>>> it's
>>>>>> definitely there.
>>>>>>
>>>>>> 09-24 18:51:05.921: E/Mms/image(8213): IOException caught while opening
>>>>>>>> stream
>>>>>>>
>>>>>>> 09-24 18:51:05.921: E/Mms/image(8213): java.io.FileNotFoundException:
>>>>>>>> /app_parts/PART_1348527038856: open failed: ENOENT (No such file or
>>>>>>>> directory)
>>>>>>>
>>>>>>> 09-24 18:51:05.921: E/Mms/image(8213): at
>>>>>>>> info.guardianproject.libcore.io.IoBridge.open(IoBridge.java:71)
>>>>>>>
>>>>>>> 09-24 18:51:05.921: E/Mms/image(8213): at
>>>>>>>>
>>>>>
>>> info.guardianproject.iocipher.FileInputStream.<init>(FileInputStream.java:77)
>>>>>>>
>>>>>>> 09-24 18:51:05.921: E/Mms/image(8213): at
>>>>>>>>
>>>>>
>>> com.Gryphn.providers.SecureComMmsProvider.getEncryptedFileInputStream(SecureComMmsProvider.java:789)
>>>>>>>
>>>>>>> 09-24 18:51:05.921: E/Mms/image(8213): at
>>>>>>>> com.Gryphn.mms.ui.UriImage.decodeBoundsInfo(UriImage.java:161)
>>>>>>>
>>>>>>> 09-24 18:51:05.921: E/Mms/image(8213): at
>>>>>>>> com.Gryphn.mms.ui.UriImage.<init>(UriImage.java:88)
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> 09-24 18:51:05.882: D/aaron(8213): file : PART_1345582002082
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345227415477
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345582101400
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345581716950
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1345581804514
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504562948
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504584604
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504907727
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348504999361
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505134414
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505198340
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505429032
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348505939191
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506460355
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506792261
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506814727
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506842859
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506884314
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348506917184
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348521829368
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348522316909
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348522434937
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348522453095
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348524511213
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348524531937
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348526895568
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348526918403
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348526985928
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348527011324
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348527038856
>>>>>>>
>>>>>>> 09-24 18:51:05.890: D/aaron(8213): file : PART_1348527063465
>>>>>>>
>>>>>>>
>>>>>> On Mon, Sep 24, 2012 at 6:04 PM, Aaron Huttner <aaron at gryphn.co>
>>> wrote:
>>>>>>
>>>>>>> Hey Abel,
>>>>>>>
>>>>>>> The MMS app I'm working on is heavily muti-threaded and I noticed
>>> that I
>>>>>>> get File Not Found errors at seemingly random intervals. I can see
>>> that
>>>>> the
>>>>>>> POSIX file system is creating / opening the file through the ADB so
>>> I'm
>>>>>>> reasonably sure the file does in fact exist. I was going to ask if
>>>>> IOCipher
>>>>>>> was thread safe and then I noticed this issue,
>>>>>>> https://dev.guardianproject.info/issues/234, so it seems like you're
>>>>>>> reasonably sure it is.
>>>>>>>
>>>>>>> Is there a specific way I should be accessing the VFS in other
>>> threads?
>>>>>>> For example; I mount the DB on the main thread when the user first
>>> logs
>>>>> in,
>>>>>>> whenever I need to access the DB I just make sure and use an IOCipher
>>>>>>> object. I don't specify the database location, just the file location
>>>>>>> relative to ROOT in the encrypted DB, does this seem correct? Also,
>>>>> just to
>>>>>>> be sure I have added debug statements to make sure the VFS is still
>>>>> mounted
>>>>>>> before access, and it is.
>>>>>>>
>>>>>>> I haven't had any luck with getting the file channel transfer method
>>> to
>>>>>>> work or being able to reproduce the image corruption, but I'm hoping
>>>>>>> they're all connected since all of these seem to point to file access
>>>>>>> issues.
>>>>>>>
>>>>>>>
>>>>>>> -Aaron
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 21, 2012 at 7:14 PM, Aaron Huttner <aaron at gryphn.co>
>>> wrote:
>>>>>>>
>>>>>>>> I wasn't able to identify anything different but I'll keep digging
>>>>> until
>>>>>>>> I'm positive.
>>>>>>>>
>>>>>>>> I'll figure out a reliable way to reproduce the corruption and send
>>>>> over
>>>>>>>> some sample code.
>>>>>>>> On Sep 21, 2012 7:06 PM, "Abel Luck" <abel at guardianproject.info>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> Aaron,
>>>>>>>>>
>>>>>>>>> I haven't encountered problems with the FileChannel method.. can you
>>>>>>>>> think of anything you're doing differently in your real app compared
>>>>> to
>>>>>>>>> the test app?
>>>>>>>>>
>>>>>>>>> There really isn't a benefit over either method, they do the same
>>>>> thing
>>>>>>>>> with a different aPI.
>>>>>>>>>
>>>>>>>>> This write and read corruption concerns me, we haven't encountered
>>>>> that
>>>>>>>>> yet. Any chance you're able to reproduce it?
>>>>>>>>>
>>>>>>>>> ~abel
>>>>>>>>>
>>>>>>>>> Aaron Huttner:
>>>>>>>>>> Abel,
>>>>>>>>>>
>>>>>>>>>> I haven't been able to get the FileChannel method to work in
>>>>> practice,
>>>>>>>>>> though it works fine in the test app I made (same one I attached to
>>>>>>>>> the bug
>>>>>>>>>> report). Is there any benefit to one method vs the other? That
>>> being
>>>>>>>>> said,
>>>>>>>>>> even using the ByteArrayOutputStream doesn't seem to be 100%
>>>>> effective.
>>>>>>>>>> There are still times when it has trouble saving and when it
>>> finally
>>>>>>>>> does
>>>>>>>>>> it seems the data is corrupt, you can see what I mean in the
>>> attached
>>>>>>>>>> image. I'm not sure if it's related to the current issue (it seems
>>>>>>>>> that way
>>>>>>>>>> given it occurs after long write times), I can file a new report if
>>>>> you
>>>>>>>>>> think it's warranted. It's also possible that there is something
>>> I'm
>>>>>>>>> doing
>>>>>>>>>> to the data that's not playing nice w/ IOCipher and I just don't
>>>>>>>>> realize it
>>>>>>>>>> yet.
>>>>>>>>>>
>>>>>>>>>> In the attached file the glitch is across the top of the image, the
>>>>>>>>> blurred
>>>>>>>>>> out face is from me in Photoshop... you can see why I needed to
>>>>> write a
>>>>>>>>>> secure texting app.
>>>>>>>>>>
>>>>>>>>>> -Aaron
>>>>>>>>>>
>>>>>>>>>> Also, I've noticed some corruption when retrieving files
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 21, 2012 at 5:11 PM, Abel Luck <
>>>>> abel at guardianproject.info
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hey Aaron,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for the bug report :)
>>>>>>>>>>>
>>>>>>>>>>> Each write call to libsqlfs (done by that Posix method
>>> pwriteBytes)
>>>>>>>>> has
>>>>>>>>>>> some significant SQL overhead. So, while I don't have the data to
>>>>>>>>>>> support it yet, my hypothesis is that the SQL overhead is causing
>>>>> poor
>>>>>>>>>>> performance.
>>>>>>>>>>>
>>>>>>>>>>> Your workaround reduces the number of calls to libsqlfs, and hence
>>>>> the
>>>>>>>>>>> SQL overhead. We've talked about buffering small writes in
>>> IOCipher,
>>>>>>>>>>> then flushing them periodically in larger writes to libsqlfs,
>>> we'll
>>>>>>>>> see..
>>>>>>>>>>>
>>>>>>>>>>> Another way to implement file copying is using FileChannels. The
>>>>>>>>> result
>>>>>>>>>>> is the same, a single large write is performed rather than a bunch
>>>>> of
>>>>>>>>>>> smaller writes.
>>>>>>>>>>> Here's an example snippet https://gist.github.com/3763841
>>>>>>>>>>>
>>>>>>>>>>> We're still in the process of profiling and optimizing IOCipher
>>> and
>>>>>>>>> its
>>>>>>>>>>> underlying components. Your feedback is quite helpful, definitely
>>>>> keep
>>>>>>>>>>> it coming.
>>>>>>>>>>>
>>>>>>>>>>> Soon, I hope to write some docs for IOCipher describing how using
>>> it
>>>>>>>>>>> differs from normal Java IO. Using larger buffers is a great
>>> example
>>>>>>>>> of
>>>>>>>>>>> the types of tips I hope to include there.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> ~abel
>>>>>>>>>>>
>>>>>>>>>>> Aaron Huttner:
>>>>>>>>>>>> Hey Abel,
>>>>>>>>>>>>
>>>>>>>>>>>> I saw you were the assignee on the Bug report so I thought I
>>> would
>>>>>>>>> update
>>>>>>>>>>>> you on a work around I'm trying out.
>>>>>>>>>>>>
>>>>>>>>>>>> I noticed (somewhat obviously) that the bigger I made the buffer
>>>>> the
>>>>>>>>>>> faster
>>>>>>>>>>>> it would copy the file. I'm not overly familiar w/ the
>>> intricacies
>>>>> of
>>>>>>>>>>> I/O,
>>>>>>>>>>>> let alone encrypted I/O, but it made sense that if the
>>> performance
>>>>>>>>>>>> increases with fewer writes I'd try to just do one big write. So
>>> I
>>>>>>>>> tried
>>>>>>>>>>>> copying the file to a ByteArrayOutputStream as an intermediary
>>> and
>>>>>>>>> then
>>>>>>>>>>>> just writing to the encrypted VFS in one call
>>>>>>>>>>>> (FileOutputStream.write(baos.toByteArray()). This significantly
>>>>>>>>> improved
>>>>>>>>>>>> write time to almost as quick as the native file system.
>>>>>>>>>>>>
>>>>>>>>>>>> This seems like a viable work around for me at the moment given
>>> the
>>>>>>>>> small
>>>>>>>>>>>> file sizes I'm dealing with. However, it's probably a bad idea
>>> for
>>>>> a
>>>>>>>>>>>> general work around.
>>>>>>>>>>>>
>>>>>>>>>>>> -Aaron
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 21, 2012 at 9:29 AM, Aaron Huttner <aaron at gryphn.co>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Hans,
>>>>>>>>>>>>>
>>>>>>>>>>>>> SQLCipher made a huge difference on our ability to get this app
>>> to
>>>>>>>>> the
>>>>>>>>>>>>> market as fast as we did and I'm really excited to use IOCipher
>>> to
>>>>>>>>>>> secure
>>>>>>>>>>>>> MMS attachments at rest. Honestly, it's like you guys are making
>>>>>>>>> exactly
>>>>>>>>>>>>> what we need when we need it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'll definitely keep you posted, we've got some cool
>>>>>>>>> features/products
>>>>>>>>>>> in
>>>>>>>>>>>>> the pipe line.
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Aaron
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Sep 20, 2012 at 5:35 PM, Hans-Christoph Steiner <
>>>>>>>>>>>>> hans at guardianproject.info> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I just looked into your company.  Sounds like you're making a
>>>>> nice,
>>>>>>>>>>>>>> tight Android app for secure messaging.  This is exactly the
>>> kind
>>>>>>>>> of
>>>>>>>>>>>>>> stuff we are aiming to support when we create dev tools like
>>>>>>>>>>>>>> SQLCipher-for-Android, IOCipher, Orlib, etc.  So please keep us
>>>>>>>>> posted
>>>>>>>>>>>>>> on how its working for you so we can incorporate your
>>> experience
>>>>>>>>> as we
>>>>>>>>>>>>>> push things further.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We are also bringing on some more people so we should have more
>>>>>>>>> time
>>>>>>>>>>>>>> real soon to spend on improving IOCipher.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> .hc
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 09/20/2012 04:45 PM, Aaron Huttner wrote:
>>>>>>>>>>>>>>> Reported, I'll keep that link bookmarked for the future.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Sep 20, 2012 at 4:33 PM, Nathan of Guardian <
>>>>>>>>>>>>>>> nathan at guardianproject.info> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>  On 09/21/2012 02:01 AM, Aaron Huttner wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm not sure if you guys have a bug reporting email or not,
>>> my
>>>>>>>>>>>>>> apologies if
>>>>>>>>>>>>>>>> I missed it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Anyway, I think I found a bug in IOCipher where it takes
>>>>> forever
>>>>>>>>> (~2
>>>>>>>>>>>>>>>> minutes in reality) to copy a file from the SD card to the
>>>>>>>>> encrypted
>>>>>>>>>>>>>> VFS.
>>>>>>>>>>>>>>>> The same operation only takes about a second when I don't use
>>>>>>>>>>> IOCipher.
>>>>>>>>>>>>>>>> After performing a trace it's clear that the bulk of the
>>> time,
>>>>>>>>> ~99%,
>>>>>>>>>>> is
>>>>>>>>>>>>>>>> spent in the Posix.pwriteBytes() method of the IOCipher
>>>>> library.
>>>>>>>>> I've
>>>>>>>>>>>>>>>> attached a sample Android app (VfsTest.zip) and the trace
>>> file
>>>>>>>>>>>>>>>> (IOCipher.trace) to help illustrate the problem.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>  Thanks! You can report the bug here:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>> https://dev.guardianproject.info/projects/iocipher/issues/new
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have cc'd our core devs on the project.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> +n
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Aaron Huttner, CTO
>>>>>>>>>>>>> Gryphn - "Mobile Privacy Simplified"
>>>>>>>>>>>>> +1-419-351-9133 | Aaron at gryphn.co
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Aaron Huttner, CTO
>>>>>>> Gryphn - "Mobile Privacy Simplified"
>>>>>>> +1-419-351-9133 | Aaron at gryphn.co
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
> 
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 937 bytes
Desc: OpenPGP digital signature
URL: <http://lists.mayfirst.org/pipermail/guardian-dev/attachments/20120926/68fda14d/attachment.pgp>