• Status Closed
  • Percent Complete
  • Task Type Bugs
  • Category Operating System/Drivers
  • Assigned To No-one
  • Operating System All players
  • Severity Low
  • Priority Very Low
  • Reported Version Daily build (which?)
  • Due in Version Undecided
  • Due Date Undecided
  • Votes
  • Private
Attached to Project: Rockbox
Opened by sideral - 2011-08-03
Last edited by MikeS - 2017-10-20

FS#12216 - Dircache shuts down when closing a file that was opened prior to reloading the dircache

The dircache shuts down (resets dircache_initialized to false) when it encounters files it does not know about, assuming a dircache inconsistency. One of the reasons for shutting down the dircache is when it encounters a file handle it doesn’t know about. To prevent this from happening, the dircache maintains a mapping between file handles and file names, which is maintained even when the dircache is not fully operating yet during the boot phase (by way of a small temporary cache array).

The problem is that this scheme has a few holes that allow files to be open without the dircache knowing about it:
1. When dircache_initializing is not set, the temporary cache is not maintained and newly opened files will slip through.
2. When the dircache is disabled and later re-enabled, open-file mappings that were active during the first dircache instance are not carried over to the new instance.

Both cases occur frequently when reloading the dircache after updating the database, which defaults to disabling the dircache and using the dircache buffer for its commit phase, and then reloading the dircache.
The most common case I see is when playback is active during the automatic DB update (Resume Playback is my startup screen). When I wait for the DB update and dircache reload to complete and then start a new playlist from the database, the playlist-control file (I believe) is closed. The new dircache instance does not know that this file was open before and shuts down (in update_filesize). (With current SVN, this later causes the DB browser to crash, but that’s a different story.)

I’m wondering what the best way to keep the dircache online would be. Some ideas:

* When the dircache shuts down, save the current list of bindings to the temporary cache, to be reloaded when the new instance starts up. Also set dircache_initializing = true, and reset it only when the temporary cache overflows). This would require a few more entries in the temporary cache than the current two.

* Recover the bindings from the FAT filesystem layer, or maybe even generally store them there. This layer already has a list of open files, but currently does not store a file-descriptor binding. Unfortunately, this solution does not work for hosted platforms.

* Make the dircache resilient against unknown file-descriptor bindings. It’s somewhat unclear to me how that could work, as each handle-based update that we ignore and just pass through to the filesystem creates a dircache inconsistency.


Closed by  MikeS
2017-10-20 15:30
Reason for closing:  Out of Date
Additional comments about closing:   Warning: Undefined array key "typography" in /home/rockbox/flyspray/plugins/dokuwiki/inc/parserutils.php on line 371 Warning: Undefined array key "camelcase" in /home/rockbox/flyspray/plugins/dokuwiki/inc/parserutils.php on line 407

Dircache is no longer confused by existing fds as it maintains the open file list and binds them with the entries during init. Changes to the FS during init are handled transparently. It also doesn't have to be consistent at all times in order to work.

There are two different matters: file and directories (dircache handles both).
For files, I believe it is indeed possible to rebuild the open file list in dircache.c from the open list file in file.c with information from the FAT layer. However, this is not exactly a free operation. The main advantage is that then you don’t need to manage a queue.
I don’t think making dircache resilient against unknown fd is a good idea, it will only hide the problem.

I don’t think open directories are much of a problem. Directories are not kept open for extended periods of time, so the dircache just has to wait for directory activity to quiesce when it turns on or off. It already does this when shutting down, and we should add a similar wait to the startup path.

According to an IRC conversation with Slasheri, another related problem with dircache startup is that dircache entries may become stale due to races with file-modifying code executed during the startup phase (after the dircache entry has been created, but before the dircache goes online). A possible solution could be to delay write operations while dircache_initializing is true, either with a delay loop or with a semaphore.

Following up on how to prevent dircache from shutting down when it finds an unknown file handle:

One idea could be to replace the temporary cache with a complete mapping of file names to open file handles, kept up to date at all times regardless of whether dircache is online or not. This mapping could be stored for all open files in open() or in dircache_bind(), or we can rely on the FAT layer (on native platforms).

This information could be inspected when initializing the dircache. Alternatively, it could be used to recover the filename for an unknown handle late, when we would normally shut down the dircache; once the filename is known, the dircache entry can be updated from the real directory entry.

This (or something with the exact same symptom - i.e. dircache turning itself off) seems to bite me whenever I go to “database update” shortly after powering on my device (ipod video - current rockbox build). I’m pretty sure database builds didn’t always turn off dircache (in other words, I’m pretty sure this all used to work as expected and dircache remains enabled while/after updating database).


Available keyboard shortcuts


Task Details

Task Editing