Database

Introduction
Using the Database
tagnavi.config v2.0 Syntax
Examples
Database files file-format
FAQ
Known issues
Feature Requests

Introduction

Rockbox contains a music database system. Using the information contained in the tags (ID3v1, ID3v2, Vorbis Comments, Apev2 etc) in your audio files, Rockbox builds and maintains a database of the music files on your player and allows you to browse them by Artist, Album and Genre. It can also collect runtime data allowing for most played, unplayed and most recently played tracks to be logged, along with per-track ratings..

To enable the database it must first be Initialized (see Initialize Now below)

Using the Database

The database options can be accessed in Settings -> General Settings -> Database. The following attempts to explain the sub menus functions:

Load To Ram

Setting this to 'Yes' loads the database to the players RAM, allowing faster browsing and searching. Setting 'No' keeps the database on the disk, meaning slower browsing but it does not use extra RAM and saves some battery on boot up. However if you frequently use the database you should load to RAM as this will reduce the overall battery consumption as the disk does not need to spin on each search.

Auto Update

If "automatic update" has been set to on, it automatically does everything that "update now" does at every boot. However, without dircache (General Settings -> System -> Disk -> Directory Cache) enabled, auto update may be very slow while it performs a search for deleted files.

Initialize Now

Initialize now removes all database files (removing runtimedb data also) and then it builds the database from scratch. This takes approximately 5 to 10 minutes for most units (though quite a bit longer on Archos models except Ondios), and requires a restart to allow the files to be committed to the database. While initialize is running you can still listen to music as it runs in a background thread. You can even shut down your player and resume the database building process next time you turn it on.

Once the database has been built and committed, you can browse your music using this database from the main menu.

You may need to increase the value of the "Max files in dir browser" setting (General Settings -> System -> Limits) in order to view long lists of tracks in the ID3 database browser.

Update Now

Update now only detects new and deleted files regardless of Directory Cache settings, and updates the database without removing runtimedb data.

Gather Runtime Data

When enabled this allows for most played, unplayed and most recent played tracks to be logged and scored. To use it the database must be initialized, and gather runtime data enabled. Runtime data is stored into the database every time rebuffering happens and old track entries are unbuffered from memory.

Export Modifications

This allows for the runtime data to be exported into /.rockbox/database_changelog.txt, which backs up the runtime data in ascii format. This is needed when the database structures change, because new code can't read old database code. But all modifications exported to ascii format should be readable by all database versions.

Import Modifications

Allows the /.rockbox/database_changelog.txt backup to be conveniently loaded into the database. This is performed automatically when the database is initialized.

tagnavi.config v2.0 Syntax

The /.rockbox/tagnavi.config file and its user-editable counterpart /.rockbox/tagnavi_custom.config control the structure and layout of the ID3 File View. It is strongly recommended to modify only the /.rockbox/tagnavi_custom.config file, as the default file will be overwritten every time you install a new build.

In tagnavi.config you can define your own layout for the database navigation using powerful filters and conditionals. It is possible to logically include other files (see include below) for easier management. By default, tagnavi_custom.config is included. Make sure not to create endless include loops by including a file in itself (e.g. when using tagnavi.config as the basis for your personal tagnavi_custom.config).

The order of the different sections in the file is crucial because the file is interpreted whilst loading in the order in which it is written. The order should generally follow the list below. The important point is that for a line to display, all of its contents should have previously been defined. E.g., if a main menu item points to a sub-menu, then the sub-menu must be defined in the file prior to the main menu.

A tagnavi file can contain the following types of lines:

comment
include
format
menu start
sub menu item
filter menu item
root menu

Every comment line starts with a # followed by text (typically annotations). There is one special case: the first line in the file needs to be: #! rockbox/tagbrowser/2.0" for the file to be recognised as a valid tagnavi file.

Every include line starts with the keyword %include followed by a filename in quotes. For example: %include "/.rockbox/tagnavi_custom.config" The text from the imported file is used as if it was copied verbatim.

Every format line starts with the keyword %format followed by a fmt_identifier (in quotes), the format definition, optional modifiers and optionally a condition seperated by a ?. It is possible to use the same fmt_identifier with different definitions. The first line where the condition is true is applied, so it is suggested to use an unconditional line as a "catch all" at the end.

%format <fmt_identifier> <formatting> [<modifier>]* [? <condition>]

<fmt_identifier>

a text string in quotes; convention is to use "fmt_" as prefix

<formatting>

determines how the tracks are displayed and sorted (!)
consists of a formatting string in standard printf() notation (Wikipedia description) and the list of tags used as parameters in the formatting string: "formatting string" <tag> [<tag>]* To sort numerical tags correctly you need to specify a sufficient number of leading zeros (e.g. %02d for two-digit track numbers, %08d for length, as it is a millisecond number).

see below for supported tags

<modifier>

influences the display of the title list (last level in the browser)
- %sort = "inverse" reverses the sorting. Useful for lists where large values are more interesting (e.g. top 50 tracks)
- %limit = "<n>" limits the list to n entries. Useful for long lists (e.g. top 50 tracks)
- %strip = "<n>" removes n characters from the beginning of the display string. Useful if you want to sort the list by a tag, but not display it.

<condition>

I assume that the same rules apply as for menu item lines - needs confirmation.

Note You can only use a %format string for formatting the title of a track, it cannot be used for formatting within the album hierarchy http://forums.rockbox.org/index.php?topic=9570.0

Every menu start line starts with the keyword %menu_start followed by the name of the menu (in quotes) followed by a heading (in quotes). The heading is displayed if the Show Path option is enabled in the settings. The menu start line must be immediately be followed by a combination of one or more menu item lines that make up the menu. An empty line terminates the list.

Every root menu line starts with the keyword %root_menu followed by the name of a menu.This menu is then used as the top level selection. If there are multiple root menu line, the first one is used. This makes it possible to override the default selection in the tagnavi.config by a custom one in tagnavi_custom.config.

Every sub menu item line consists of a label, and arrow (==>) and the name of the submenu.

Every filter menu item line consists of a label, a number of filters (each terminated by a "->") with optional condition and the title with optional formatting, optional modifier and optional condition.

"label" -> [<filter> [? <condition>] ->]* title [= <fmt_identifier>] [? <condition>]

"label"

is a text string that is displayed at the top level

<filter>

is a tag (see below for supported tags)
it will introduce another level in the database tree with entries for all possible values for of the <filter> (e.g. all artists)
if there is more than one <filter> the selection will be limited by the filter above (e.g. if you pick genre "Rock", you will only find "Rock" artists in the next level)
if necessary there will also be an entry "<untagged>" for files without an entry for that tag (e.g. no artists entered)
another entry "<All Tracks>" will give a shortcut for all the titles selected so far

<condition>

limits the shown selection (e.g. album ? genre = "Ballads" & artist = "AC-DC" will only show the albums that contain ballads by AC-DC)
is a boolean expression of the form: <tag> <operator> <constant> [& <tag> <operator> <constant>]*
see below for supported tags

<tag> (see below)
<operator> is either a
- universal operator: = (equal), == (equal), != (not equal), > (greater), >= (greater or equal), < (less), <= (less or equal)
- string only operator: ~ (contains), !~ (does not contain), ^ (starts with), !^ (does not start with), $ (ends with), !$ (does not end with), @ (is one of the items in the list: "item1|item2|item3" )
- numerical only operator: (currently none)
<constant> is either a literal string enclosed in " or the empty literal "" in which case the user will be prompted to input a string. The following strings have a special meaning: "#directory#", "#title#", "#artist#", "#album#", "#genre#", "#composer#", "#albumartist#". They will be replaced by the corresponding tag from the currently playing track at runtime.
evaluation of <constant> is not case-sensitive

<fmt_identifier>

string literal as defined in a format line

Supported Tags

tag	type	origin	comment
title	string	id tag
grouping	string	id tag
filename	string	system	Does work in filter, but selecting the result immediately commences playback
albumartist	string	id tag	Read from file metadata: BAND (MP3);ALBUMARTIST, ALBUM ARTIST, ENSEMBLE (Vorbis)
artist	string	id tag
album	string	id tag
genre	string	id tag
composer	string	id tag
year	numeric	id tag
tracknum	numeric	id tag/from filename	Works, but not sorted correctly when used in filter (1,10,11,...,18,19, 2 ,20,21,...)
discnum	numeric	id tag	Read from file metadata: DISCNUMBER
length	numeric	system	Track length in milliseconds
bitrate	numeric	id tag	Works, but not sorted correctly when used in filter
playcount	numeric	runtime db	Works, but not sorted correctly when used in filter
playtime	numeric	runtime db	This tag does not work as per the others, but is used solely in the autoscore calculation
lastplayed	numeric	runtime db	Incremental number. Works, but not sorted correctly when used in filter
autoscore	numeric	virtual	100*playtime/length/playcount
entryage	numeric	runtime db	Can be used to browse last or recently added files. Last added=0, next-to-last=1 etc.
comment	string	id tag	ID3 or Vorbis comment string
rating	numeric	runtime db	Works, but not sorted correctly when used in filter
Lm	numeric	system	track length (min)
Ls	numeric	system	track length (sec)
Pm	numeric	runtime db	total playtime (min)
Ps	numeric	runtime db	total playtime (sec)

Unsupported Tags

Performer (Needs to be able to be used instead of Artist)

Examples

The following are provided to give an idea of what can be achieved using the flexibility of a custom tagnavi file. To try any of these, either download the example directly using the links (and rename to the correct name format), or copy and paste the code into a text editor and save the file as tagnavi_custom.config in the format ANSI without BOM (Byte Order Mark). UTF-8 format will not work. Podcasts have been used in the examples, but the rules apply equally to any other differentiation that the user would like to make.

Example 1 - Podcasts by Genre

It is quite likely that a user would like to differentiate podcasts from music files on their player. The following code would simply add a 'Custom...' menu item to the end of the default Database File View, selecting which would show all files genre tagged as 'podcast'.

#! rockbox/tagbrowser/2.0

%menu_start "custom"  "Podcasts"
"Podcasts" -> title = "fmt_title" ? genre = "podcast"

#EOF

> >

Note that because a standard formatting is used, a tracknumber value is shown.

http://www.rockbox.org/wiki/pub/Main/DataBase/simple_tagnavi_custom.config

Example 2 - Podcasts by Location

It is unusual for all podcasts to be tagged correctly, so the following code uses the file location to ensure that all the files are shown. For this to work, it is assumed that all podcasts are in a folder named 'podcast' or 'podcasts' somewhere on the device.

#! rockbox/tagbrowser/2.0

%format "fmt_podcast" "%s (%s)" title filename

%menu_start "custom" "Podcasts"
"All podcasts" -> title = "fmt_podcast" ? filename ~ "/podcast"

#EOF

This works because the 'filename' attribute is actually full path and filename, so the query 'title contains "/podcast"' captures all files with that in their path, regardless of tagging. The custom formatting also shows both title (from tag) and filename in case the file is untagged.

http://www.rockbox.org/wiki/pub/Main/DataBase/better_tagnavi_custom.config

Example 3 - Podcasts, Old and New

Building on the previous example, it is possible to filter information based on the runtime information (playcount and lastplayed). Using this, podcasts can be split into 'new' and 'old' (i.e., already listened to) and sorted in listening order. For this to work, the runtime database has to be enabled - [menu]>General Settings>Database>Gather Runtime Data

#! rockbox/tagbrowser/2.0

# Un-listened to files sorted by duration, shortest files at the top
%format "fmt_podcast" "%08d%s (%s)" length title filename %strip = "8"
# Listened-to files sorted by playing order, most recently played at the top
%format "fmt_podcast_old" "%06d%s (%s)" lastplayed title filename %sort = "inverse" %strip = "6"

%menu_start "custom" "Podcasts"
"New Podcasts" -> title = "fmt_podcast" ? filename ~ "/podcast" & playcount == "0"
"Old Podcasts" -> title = "fmt_podcast_old" ? filename ~ "/podcast" & playcount != "0"

#EOF

> >

http://www.rockbox.org/wiki/pub/Main/DataBase/advanced_tagnavi_custom.config

Example 4 - Full Tagnavi File

fully annotated
completely replaces the default menu
artists sorted by first initial with 'The...' artists separated out
completely customised formatting
splits single-artist and compilation albums by using a filename trick
custom searches
some smart runtime selections
uses 'one-of' operator for fancy genre grouping into 'moods'

As the full tagnavi_custom file is long, only exerpts are shown below. The best way to understand the syntax is to download the file and study it in a text editor or, even better, install it and just play with the functionality!

#The main menu is mostly calls to sub-menus with the exception of the podcast/radio selections that filter only on whether the track
# has been listened to or not (playcount), the rest of the hard work is done by the formatting

%menu_start "bascule" "Rockbox Browser"
"Album Artists" ==> "custom_artist_initial"
"All Artists" ==> "custom_artist"
"Albums" ==> "custom_album"
"Mood" ==> "mood_tree"
"New Podcasts/Radio" -> title = "f_radio" ? filename ~ "/radio/" & playcount == "0"
"Old Podcasts/Radio" -> title = "f_radio_old" ? filename ~ "/radio/" & playcount != "0"
"Tracks by usage" ==> "custom_runtime"
"Search" ==> "custom_search"
"Year" -> year -> genre -> artist -> album -> title = "f_browse"
"All tracks A-Z" -> title = "f_direct"

Note the 'Rockbox Browser' title. Using these well adds a touch of class to the look of the screen.

#Condition 'filename !~ "_cmpl"' excludes all compliation album tracks from this menu,
# by virtue of the fact that all such tracks have '_cmpl' appended to the filename i.e., 06-missing_cmpl.mp3
#'Non-alphabetic' category created as a catch-all for tracks beginning with punctuation marks etc.
#'T' artists specially formatted to exclude names beginning with 'The ' and a 'The...' category separately defined

%menu_start "custom_artist_initial" "Browse album artists"
"Non-alphabetic"   -> artist ? artist < "A" & filename !~ "_cmpl" -> album  -> title = "f_browse"
"A"   -> artist ? artist ^ "A" & filename !~ "_cmpl" -> album  -> title = "f_browse"
"B"   -> artist ? artist ^ "B" & filename !~ "_cmpl" -> album  -> title = "f_browse"
...
"T"   -> artist ? artist ^ "T" & artist !^ "The " & filename !~ "_cmpl" -> album  -> title = "f_browse"
"The..."   -> artist ? artist ^ "The " & filename !~ "_cmpl" -> album  -> title = "f_browse"
...
"Z"   -> artist ? artist ^ "Z" & filename !~ "_cmpl" -> album  -> title = "f_browse"

#A-Z of all artists, including compilation artists
#Special selection solely for artists on compilation albums

%menu_start "custom_artist" "Browse all artists"
"Artist by genre" -> genre -> artist -> album -> title = "f_browse"
"Compilations by artist" -> artist ? filename ~ "_cmpl" -> album  -> title = "f_browse"
"Non-alphabetic"   -> artist ? artist < "A" -> album  -> title = "f_browse"
"A"   -> artist ? artist ^ "A" -> album  -> title = "f_browse"
"B"   -> artist ? artist ^ "B" -> album  -> title = "f_browse"

#Most search requirements are covered here!

%menu_start "custom_search" "Search for..."
"Artist starting with..." -> artist ? artist ^ "" -> album -> title = "f_browse"
"Artist containing..." -> artist ? artist ~ "" -> album -> title = "f_browse"
"Artist between years..."       -> artist ? year >= "" & year <= "" -> album -> title = "f_chrono"
"Album starting with..." -> album ? album ^ "" -> title = "f_browse"
"Album containing..." -> album ? album ~ "" -> title = "f_browse"
"Album between years..."       -> album ? year >= "" & year <= "" -> title = "f_chrono"
"Track starting with..." -> album ? title ^ "" -> title = "f_direct" 
"Track containing..." -> album ? title ~ "" -> title = "f_direct"
"Filename starting with..." -> album ? filename ^ "" -> title = "f_direct" 
"Filename containing..." -> album ? filename ~ "" -> title = "f_direct"

#The runtime entities populated by the Database are used to derive the filters and conditions for the 'Usage database' menu
#'Most played' and 'Recently played tracks' display only tracks that have been played (playcount > "0")
#A 'favourite' is classified here as having been played more than three times and with an average of 85% of the track having been listened to.

%menu_start "custom_runtime" "Usage database"
"Most played (Plays|Score)" -> title = "f_runtime" ? playcount > "0"
"Recently played tracks" -> title = "f_lastplayed" ? playcount > "0"
"Favourite albums" -> album ? playcount > "3" & autoscore > "85" -> title = "f_browse"
"Recent favourites" -> title = "f_lastplayed" ? playcount > "3" & autoscore > "85"
"Forgotten favourites" -> title = "f_forgotten" ? playcount > "3" & autoscore > "85"
"Never played tracks" -> artist ? playcount == "0" -> album -> title = "f_browse"

> >

The runtime database information could be used creatively to construct quite complicated conditional statements if the user wished

#The following mood menus are similar in that each mood (upbeat party, downbeat rock, etc.) is purely a list of tracks
# tagged with any of the genres in the list (e.g., "acoustic|ambient|new age|easy listening")
#This can be used to quickly produce a substantial playlist of similar music when you just know what 'sort' of music you want
...
%menu_start "mood_2" "Party"
"All tracks" -> artist ? genre @ "dance|disco|electronica|pop|techno|trance|funk|soul" -> title = "f_mood"
"Upbeat" -> artist ? genre @ "dance|disco|electronica|pop|techno|trance" -> title = "f_mood"
"Downbeat" -> artist ? genre @ "funk|soul" -> title = "f_mood"
...

> >

#This ensures that, rather than being accessed via the 'Custom...' entry in the default Database view, this custom menu *is* the default menu

%root_menu "bascule"

http://www.rockbox.org/wiki/pub/Main/DataBase/tagnavi_custom.config

Database files file-format

An in depth inspection of the file-format of the files created while the database initialization can be found here: TagcacheDBFormat.

FAQ

Tracks do not sort correctly when using disc numbers with more than one digit or tracknumbers with more than 2 digits

The database uses the format string for sorting which is set to one digit for discnumber and 2 digits for tracknumber. If you want more digits to sort correctly, you can adapt the string in tagnavi.config (see http://www.rockbox.org/twiki/bin/view/Main/DataBase#tagnavi_config_v2_0_Syntax )

For example change

%format "fmt_title" "%d.%02d. %s" discnum tracknum title ? discnum > "0"

%format "fmt_title" "%02d.%03d. %s" discnum tracknum title ? discnum > "0"

for 2 digit discnumbers and 3 digit track numbers. Note that tagnavi.config will get overwritten with every update, so the better option is to create your own tagnavi_custom.config.

Known issues

The database was added to the Rockbox CVS on 26 March 2006 (then called Tag Cache). Development continues, but as of late 2006 the functionality is very stable and comprehensive.

General Problems

Add your issues here.

Model-Specific Problems

Archos Recorder: Database doen't exist because there isn't sufficient RAM. This is also the case for Directory Cache.

Feature Requests

Here you can list all features you would like to see implemented in the database in future:

Browse Options

Ability to browse by track rating and access to ratings via the WPS (in fashion similar to the old tag database). It would also be great if the database could automatically add externally created ratings stored in the RATING and TRACK_RATING tag/comment fields. (Will be supported after 3.0 release when runtime db feature is integrated in the database. - MiikaPekkarinen, but for now this functionality is partly achieved by the autoscore tag described above)

Playlists

Ability to create smart playlists on the fly. Something like songs between 1980 and 1990 by Bruce Springsteen with "People" in the title, or something based on ratings with an added random factor.
I would like to see an option to continue playback once the selected songs have finished playing. Say you go into the browser, and select Artist A and select all tracks. I would like to have the ability to choose that once all tracks by Artist A are finished, playback continues, skips down to tracks by Artist B and starts playing them. -- ChrisBerkley - 27 Aug 2006

Other

Create a log file during the database building process which lists files with no metadata and files with corrupt metadata. Could also be useful for identifying any files that cause the database building to fail.
Ability to read tags with multiple genres, so a file tagged with "Rock" and "Grunge" would appear under both "Rock" and "Grunge" when searching by genre. FS#5035
Ability to read the Cuesheet tag for WavPack and FLAC. EddyComan - cuesheet support has been added since Feb 2007, but I do not know if it includes WavPack and FLAC

Requested items already present in Flyspray as a Feature Request or Patch

FS#5923 Have the database move "The" from the beginning of artist names, so "The Red Hot Chili Peppers" become "Red Hot Chili Peppers, The" which would make it work more like WinAmp, iTunes, etc. (And require less tag modification.)

FS#7287 Support sorting tags in the database for browsing pupose. This tags are implemented for ID3v2.4, Vorbis comments and AAC tags, a unofficial support for ID3v2.3 exists. The sorting tags are applied in the database browser and let e.g. artist "The Who" shows up as "Who" in the list (so, being under W), but still as "The Who" in the WPS. The sorting tag value is "Who" in this case, while the artist is still "The Who".