[guardian-dev] IOCipher Status Update

Hans-Christoph Steiner hans at guardianproject.info
Tue Jan 22 18:53:11 EST 2013



On 01/22/2013 05:20 PM, Abel Luck wrote:
> Hi Stephen and all,
> 
> This is some great stuff.
> 
> I've two questions:
> 
> 1) Can you share your method for the "enhanced tracing" you used to
> troubleshoot the corrupted fsx runs? Breakpoints? printf debugging? strace?
> 
> Knowing how to reproduce this would be quite useful.
>
> 2) What is the argument for raising or lowering the sqlite3_busy_timeout
> from 10 seconds?

>From what I understand, the idea is to guarantee that certain operations won't
fail and instead will block/wait for up to 10 seconds.  I think we might even
want to increase this more, I think that filesystems generally have long
timeouts before returning and error so that it tries as hard as possible to do
successful writes, even if it blocks for a long time.


> Finally, as a review note, I think we should go ahead and purge delay()
> from the codebase. Thoughts?

Definitely sounds like a good idea, but do we need to replace it with
anything?  I guess that the sqlite3_busy_timeout() is global, i.e. affecting
all transactions?

.hc



> 
> Stephen Lombardo:
>> Hello Everyone,
>>
>> We've made some solid progress with locking and stabilization of libsqlfs under multithreaded use. 
>>
>> We started by reproducing the problems with the library under load. This involved firing up a fuse mount and running three concurrent fsx processes against the same sqlfs file system. This test quickly produced file corruption, usually within 30 seconds. From there, enhanced tracing showed that the underlying cause had to do with the database being locked at various points during execution. Even thought the library was opening an exclusive transaction, there were still numerous opportunities for the transaction to be blocked by a reader, or multiple writers to block each other. Similar behavior could also be seen with deferred transactions (i.e. standard begin).  
>>
>> This lead us to make several change to collectively improve the stability of the library.
>>
>> First, we changed the transaction command in begin_transaction to use "begin immediate". This seeks an immediate reserved lock on the database, but does not exclusively lock it. This reduces unresolvable contention for write locks that would normally occur with deferred transactions, and is less restrictive than an exclusive lock, since it will continue to allow shared locks for reading.
>>
>> It is extremely important that we prevent write operations from failing to execute due to busy timeouts, even if another process/thread has the database locked. Even using WAL, it is still possible for a command to be blocked during attempted concurrent write operations. This causes the write operation to fail leading to corruption. While libsqlfs has some "delay()" code that provides rudimentary busy handling, it is only in use for a small number of operations leaving other critical calls unprotected. Therefore, our second change was to register SQLite's internal busy handler with a relatively high timeout (currently 10 seconds, but open for discussion) via sqlite3_busy_timeout. This provides protection for all operations in libsqlfs, reducing the likelihood that a write operation would fail outright, though it may be delayed. 
>>
>> Finally, we enabled WAL mode to speed up write operations and further improve concurrency between readers and writers. Note that WAL mode only fsync()s on checkpoint operations, so it may be possible to enable NORMAL synchronous mode with lower overhead than the standard journal mode (we didn't change this yet).
>>
>> With these changes in place, three concurrent fsx processes running in parallel on a single fuse mount produced no errors in a 24hr test run. The tests also shows improved performance on read and writes. In light of these results, we'd like to get your feedback on these changes, and request that you run your own tests in the multi-threaded Android application to see if they resolve the problems that were reported. 
>>
>> All the changes are available here:
>>
>> https://github.com/sjlombardo/libsqlfs/commit/90aee5079cb26d4f4e9046b0f8d94f6e92067049
>>
>> Please let us know if you have any questions. Thanks!
>>
>> Cheers,
>> Stephen
>>
>> On 2013-01-18, Hans-Christoph Steiner wrote:
>>> On 01/18/2013 09:24 AM, David Oliver wrote:
>>>> Thanks Stephen for this.  It seems like we're closing in on completion here
>>>> with the final step being Stephen's team looking at the lock issue and to
>>>> advise Guardian as to fixes/changes/etc that we (Guardian) would implement.
>>>>
>>>> Stephen - can you provide a timeframe for your locking review?
>>
> 


More information about the Guardian-dev mailing list