Rockbox

Tasklist

FS#12216 - Dircache shuts down when closing a file that was opened prior to reloading the dircache

Attached to Project: Rockbox
Opened by sideral (sideral) - Wednesday, 03 August 2011, 21:17 GMT
Task Type Bugs
Category Operating System/Drivers
Status New
Assigned To No-one
Operating System All players
Severity Low
Priority Normal
Reported Version Daily build (which?)
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 0
Private No

Details

The dircache shuts down (resets dircache_initialized to false) when it encounters files it does not know about, assuming a dircache inconsistency. One of the reasons for shutting down the dircache is when it encounters a file handle it doesn't know about. To prevent this from happening, the dircache maintains a mapping between file handles and file names, which is maintained even when the dircache is not fully operating yet during the boot phase (by way of a small temporary cache array).

The problem is that this scheme has a few holes that allow files to be open without the dircache knowing about it:
1. When dircache_initializing is not set, the temporary cache is not maintained and newly opened files will slip through.
2. When the dircache is disabled and later re-enabled, open-file mappings that were active during the first dircache instance are not carried over to the new instance.

Both cases occur frequently when reloading the dircache after updating the database, which defaults to disabling the dircache and using the dircache buffer for its commit phase, and then reloading the dircache.
The most common case I see is when playback is active during the automatic DB update (Resume Playback is my startup screen). When I wait for the DB update and dircache reload to complete and then start a new playlist from the database, the playlist-control file (I believe) is closed. The new dircache instance does not know that this file was open before and shuts down (in update_filesize). (With current SVN, this later causes the DB browser to crash, but that's a different story.)

I'm wondering what the best way to keep the dircache online would be. Some ideas:

* When the dircache shuts down, save the current list of bindings to the temporary cache, to be reloaded when the new instance starts up. Also set dircache_initializing = true, and reset it only when the temporary cache overflows). This would require a few more entries in the temporary cache than the current two.

* Recover the bindings from the FAT filesystem layer, or maybe even generally store them there. This layer already has a list of open files, but currently does not store a file-descriptor binding. Unfortunately, this solution does not work for hosted platforms.

* Make the dircache resilient against unknown file-descriptor bindings. It's somewhat unclear to me how that could work, as each handle-based update that we ignore and just pass through to the filesystem creates a dircache inconsistency.

Thoughts?
This task depends upon

Comment by amaury pouly (pamaury) - Wednesday, 03 August 2011, 21:27 GMT
There are two different matters: file and directories (dircache handles both).
For files, I believe it is indeed possible to rebuild the open file list in dircache.c from the open list file in file.c with information from the FAT layer. However, this is not exactly a free operation. The main advantage is that then you don't need to manage a queue.
I don't think making dircache resilient against unknown fd is a good idea, it will only hide the problem.
Comment by sideral (sideral) - Wednesday, 17 August 2011, 23:23 GMT
I don't think open directories are much of a problem. Directories are not kept open for extended periods of time, so the dircache just has to wait for directory activity to quiesce when it turns on or off. It already does this when shutting down, and we should add a similar wait to the startup path.

According to an IRC conversation with Slasheri, another related problem with dircache startup is that dircache entries may become stale due to races with file-modifying code executed during the startup phase (after the dircache entry has been created, but before the dircache goes online). A possible solution could be to delay write operations while dircache_initializing is true, either with a delay loop or with a semaphore.

Following up on how to prevent dircache from shutting down when it finds an unknown file handle:

One idea could be to replace the temporary cache with a complete mapping of file names to open file handles, kept up to date at all times regardless of whether dircache is online or not. This mapping could be stored for all open files in open() or in dircache_bind(), or we can rely on the FAT layer (on native platforms).

This information could be inspected when initializing the dircache. Alternatively, it could be used to recover the filename for an unknown handle late, when we would normally shut down the dircache; once the filename is known, the dircache entry can be updated from the real directory entry.
Comment by Dave Hooper (stripwax) - Wednesday, 04 January 2012, 23:32 GMT
This (or something with the exact same symptom - i.e. dircache turning itself off) seems to bite me whenever I go to "database update" shortly after powering on my device (ipod video - current rockbox build). I'm pretty sure database builds didn't always turn off dircache (in other words, I'm pretty sure this all used to work as expected and dircache remains enabled while/after updating database).

Loading...