• Status New
  • Percent Complete
  • Task Type Patches
  • Category Database
  • Assigned To No-one
  • Operating System All players
  • Severity Low
  • Priority Very Low
  • Reported Version Daily build (which?)
  • Due in Version Undecided
  • Due Date Undecided
  • Votes
  • Private
Attached to Project: Rockbox
Opened by sideral - 2011-04-18

FS#12073 - Dynamically add files to database as they are played

These experimental patches dynamically add new tracks to the database when they are played. This allows users who (for various reasons [1]) prefer to not fully initialize the database to take advantage of database-based features such as autoresume (remember a resume position for each file), runtime statistics, or track ratings.

The current patch set is only a prototype. It's still somewhat inefficient because it does a full database commit and reload for each new track that is played.

The actual database-extension patch (patch 0002) works by refactoring tagcache_build to create a new interface, tagcache_add_file, that adds a single file to the database. This new function runs in the database thread and takes a dircache_entry pointer, which is provided by way of a Q_UPDATE event. The audio thread generates this event in tagtree_buffer_event when the initial DB lookup fails, and later checks the database again in tagtree_track_finish_event. If the DB commit and reload have succeeded in the meantime, this second lookup will succeed and the runtime information the user is interested in (stats, resume info) can be stored in the DB.

Patch 0001 changes the DB commit's buffer-allocation preference from using the dircache buffer to using the RAM DB's buffer. (This used to be the preference until r10003 reversed it, supposedly because the dircache buffer typically is bigger. I don't know yet how valid this argument is today.) This reversal avoids needlessly flushing and refilling the dircache and makes the Q_UPDATE event-passing mechanism more robust (because in-flight update events contain dircache_entry pointers, which would have to be invalidated when the dircache is flushed – which is not done yet).

Current limitations/caveats:
* Requires dircache and Load-DB-to-RAM enabled (and only works on targets that include these features)
* tagcache_add_file does not respect any database.[un]ignore files in the directory hierarchy above the added new file.
* DB needs to be initialized to begin with. There's no UI function for a basic DB initialization (without scanning and adding all files) yet.
* Not very thoroughly tested (mostly in simulator, with small file sets)

Possible improvements:
* Delay committing DB changes if more track-add events are in the tagcache queue
* Incremental commit: Could be optimized for add-one-track case
* Incremental load_tagcache: Avoid flushing and reloading full database
* Add function to initialize an empty DB

Comments welcome!

[1] Reasons for not using the database, or not keeping it up to date, include: metadata parser instability; prefer not keep tags up to date; prefer file browser over database browser

Random brainstorming:

Regarding the .ignore files, I think the best approach would be to refactor how they work slightly. Presently they actually exclude files from the database, moving forward I think it would make sense to have them exclude files from database scans, and also from the database GUI (but not from incremental updates). This could be implemented by:

1) Adding an "ignore" bit to the database
2) Ignoring files as usual when scanning
3) Adding all files incrementally, but setting the ignore field if a .ignore file is present
4) Having the GUI not show ignore flag files in the browser

My thinking is that someone might not want a file in their database (for example a podcast) but might still want to resume it later. It would be weird to force a user to index a podcast in their database browser just to be able to use the resume features. I could see a lot of examples where this could be annoying.

Depending on how hard it is for playback to check for .ignore files, it might also be an option to just always insert files into the database with the ignore field set, and force uses to actually initialize the database if they want it cleared. This way the database looks empty until its enabled, although it can still be used to store data. I'm not sure if this is the best option UI wise though. Do we want to force users to initialize the database?

saratoga, I think that would make the database use (significantly) more RAM, and slower to read the index files and display.

How do you figure? We've already got a flag variable with spare bits, so it shouldn't increase the size of the database at all unless someone wants to use resume info.


Available keyboard shortcuts


Task Details

Task Editing