This is the bug/patch tracker for Rockbox. Click here for more information.
Quick links: Bugs · Patches · Rockbox frontpage
FS#7287 - Support metadata sort tags in the Database
Attached to Project:
Rockbox
Opened by Dan Everton (safetydan) - Monday, 11 June 2007, 09:50 GMT+2
Last edited by Marc Guay (Marc_Guay) - Tuesday, 13 May 2008, 14:19 GMT+2
Opened by Dan Everton (safetydan) - Monday, 11 June 2007, 09:50 GMT+2
Last edited by Marc Guay (Marc_Guay) - Tuesday, 13 May 2008, 14:19 GMT+2
|
DetailsAttached is a patch that starts support for the sort tags specified in ID3 v2.4. It adds three new tags to the tagcache system, sortartist, sortalbum, and sorttitle which are drawn from the TSOA/TSOP/TSOT tags.
This is only the beginning and is very, very lightly tested (i.e. it compiles and doesn't seem to break anything). 
sort_tags_v1.patch
(5.1 KiB)
apps/tagcache.c | 21 ++++++++++++++++----- apps/tagcache.h | 5 +++-- firmware/export/id3.h | 5 +++++ firmware/id3.c | 5 +++++ 4 files changed, 29 insertions(+), 7 deletions(-) |
http://musicbrainz.org/doc/PicardQt/TagMapping
Another page describing the same vorbis comments mapping:
http://wiki.slimdevices.com/index.cgi?SlimServerSupportedTags
Is it possible to include ALBUMARTISTSORT in your patch? I use most of the time ALBUMARTIST instead of ARTIST when browsing so the patch wouldn't help without support for ALBUMARTISTSORT. Since in the ID3 v2.4 definition this tag is not separately defined I assume that for mp3 files TXXX:ALBUMARTISTSORT must be used instead as indicated in the above TagMapping link. For the other formats (Vorbis...) ALBUMARTISTSORT can directly be used.
I then reversed the patch and compiled again and then the simulator creates the database without problems. So it seems to be a problem with the patch to initialize the database.
...
Barbra Streisand
The Beatles
Bryan Adams
...
If I have to use ARTISTSORT instead of ARTIST in the tagnavi file does that mean that ALL my songs have to be tagged with ARTISTSORT since all songs without this tag are only shown in the <untagged> category?
I know it's a little bit off topic but I just tried to create a new patch supporting a new tag for GROUPING (or CONTENT GROUP) and now I have the same problem as I had with version 2 of your patch. After installing the patched version Rockbox fails to initialize the database. What did you have to change from version 2 to 3 that it worked (maybe I have a similar problem)?
This is no sort of recommendation, but I notice that recent versions of iTunes, which have implemented sorting tags for more fields than the 3 defined by id3v2.4, writes a TSOC tag if you add a 'Sort Composer' to a file.'
"A" -> artist ? artist ^ "A" -> yearalbum -> title = "fmt_title"
in the artist submenu with:
"A" -> sortartist ? sortartist ^ "A" -> yearalbum -> title = "fmt_title"
implement sortartist?
As I understand it, tagnavi can't display tags that are not in the database in the first place. The sort tags need to be collected when the database is built; then the tagnavi syntax can be used to display them. So it involves a rewriting of the database-building code, as well as the interpretation and display of the database.
The screen dump shows the database -> album artist view. Note that only "Die Toten Hosen" has tagged the TSOP filed (TSOP = "Toten Hosen"), the others not. Without the patch the view is sorted correctly (with "Die Toten Hosen" under D of course).
Another note: Browsing database -> A to Z .. -> Artists -> D shows "Die Toten Hosen", proving the implementation isn't working yet (I assume it's intended that it should appear under T in the A to Z view as well).
As for the MP4 metadata I based that on the documentation here http://musicbrainz.org/doc/PicardQt/TagMapping which says iTunes uses 'soaa' for the artist sort order.
Have you allready looked into tagtree.[ch]? There's most of the actual database stuff done (tagcache != tagtree). I didn't.
One question: What are your thought about the album/albumartist issue (for ID3)? Should performersortorder (TSOP) replace both, or only artist or album artist? Some people prefer browsing artist, others (like me) prefer albumartist. In both cases replacing artist and albumartist respectively with performersortorder in the list seems appropriate to me.
But on the other side: This example raises something weird. I have some compilations with like 20 or more artists. I've chosen albumartist "Various Artist" for them. If I chose performersortorder to be "Various Artists" (for whatever reason) they'll show up under V even in the artist browsing, where each track should have it's own entry, since I've tagged the artist tag properly (with each artist, and not "Various Artists"). This sounds weird.
A workaround would be of course to not tag performersortorder for compilations for ID3, but it looks more like a hack.
AAC and others shouldn't have this problem, since they support different albumartist sort and artist sort fields.
A user might intentionally using the TSOP tag for albumartist, and wonder why Rockbox doesn't read it.
I have defined an "album artist" browse menu and an "artist" browse menu in tagnavi_custom.config. I usually use the first for browsing since normally I want to browse only those artist for which I have at least one complete album. In the first browse menu the number of listed artists is thus much smaller than in the second browse menu. I use the second browse menu only when I want to browse to ALL artists, even listing artists for which I have only one song on a compilation album. Of course browsing through the second list takes much more time and the list is much more clutter with the huge number of listed artists.
How should the sorting tags be used in the tagnavi_custom.config files? If for defining a browse menu in the tagnavi_custom.config file still the strings "artist" and "albumartist" and for filtering the listed results the corresponding tags were used and only for the sorting of the resulting list the TSOP tag was used then it could be sufficient to use only one sorting tag for both tags. As far as I understood, the "album artist" browse list would then still be much short than the "artist" browse list. Of course, the user hat then to define for compilations e.g. TPE1="The Beatles", TPE2="Various Artists" but TSOP="Beatles, The".
However, if the sorting tag should be used in tagnavi_custom.config to FILTER the browse list (e.g. defining a new syntax element "artistsort" which must be used INSTEAD of "artist" for defining a browse structure in tagnavi_custom.config) then there must be two different sorting tags to be able to distinguish between artists and album artist.
These tags are only for sorting. There's absolutely no need to have it disabled in sorted lists.
Sure, there's no standard. But your idea is just like using the command tag for that stuff. It's quite random. TPEx tags are specifically designed for several performer types, which includes album artist. Surely, it's not declared as standard, but it's a de facto standard. And it makes sense, when TPE1 is used for the artist, to use the next for album artist.
I really dislike your idea. We should decide if we either use TSOP for both, artist and albumartist, or only for one of those (which I prefer -> TSOP only for artist). Album artists ("Various Artists" e.g.) may have a lower chance to include an unwanted leading article or something.
The screen dump shows the sorting working. It's in the artist view. Notes: The album artist view is still messed up. I also have tagged the TSOP tag for all those files.
Note: It shows the "Die", which is nice (as before, TSOP for it is "Toten Hosen", artist "Die Toten Hosen").
So, TODO:
- Disable the sorting for albumartist view (or enable it the same way as the artist view, I prefer disabling though, as I allready mentioned).
- Implement a fall back to artist if TSOP is untagged.
It adds the sort tags as normal tags, which is IMO definitely the cleaner, less hackish, solution.
I've managed to be able to use "artistsort" etc in the tagnavi.config. I've also implemented a fall-back to normal tags, if the sorting tag is untagged.
Play around a bit. I'm not entirely sure about the behavior with albumartistsort and mp3 (i.e. ID3-tags), I haven't really tested it.
The only remaining problem: With this patch, the sorting tag is shown in the database. E.g. in the database it reads "Toten Hosen" instead of "Die Toten Hosen". I don't really like that. I'm not sure at this point, where the proper place for this to fix is. From what I've seen, creating exceptions in tagtree isn't as easy as I thought.
Maybe Slasheri would be so kind and just jump in for this last step? :) If we've done this, we can consider completely replacing e.g. "artist" with "artistsort" (the tagnavi.config would still read "artist"), so that the sorting tags are allways taken into account.
I've also removed those #ifdef HAVE_TAGCACHE, it's pretty useless, since tagcache is only build when HAVE_TAGCACHE is defined.
I said I added it as a normal tag. So it's working like a normal tag.
What we want is that it displays artist but sorts by artistsort, like you already said.
But, I fear that this will only be possible with a "dirty hack" (see first few patches).
If anyone has compiled an iPod build with only v6 of the sort tags patch, I'd be more than happy to do some testing. All the tracks on my iPod have ID3v2.4 Artist Sort and Album Sort tags populated, and track names beginning with "The " have ID3v2.4 Track Sort tags populated.
Fix database corruption/weirdness
Edit tagnavi.config to use sort tags by default (which is fine, since we fall back to the non-sort versions if the tags empty)
It seems, titlesort works for displaying, but the translation into the file name at the end of the database browsing is messed.
It seems that using titlesort in the "# Basic format declarations" alone does the job. I haven't tested it though.
Note: You don't need to recompile, since tagnavi.config isn't compiled. Just copy the tagnavi.config in the apps/ dir to your player to update.
-"W" -> title ? title ^ "W" -> title = "fmt_title"
+"W" -> title ? title ^ "Z" -> title = "fmt_title"
I know that Sortartist will revert to artist if no TSOA tag is present. Likewise, Sortalbumartist will revert to albumartist if no TXXX:ALBUMARTIST tag is present.
However, there really needs to be two levels to this... sortalbumartist must first revert to albumartist if there is no tag... then, if there is no albumartist tag, it should check for sortartist... then if there is no sortartist, it should check for artist.
ATM if browsing by albumartist with the sort tags, a huge amount of <untagged> files show up because I only tag albumartist if the track artist is different for that track. If I browse by REGULAR albumartist, this doesn't happen... which is probably how it ought to be.
Just a thought... and thanks for picking up work on this, I assumed it was a dead patch.
Tox
At least in my opinion. I wouldn't revert albumartist to artist even without sorting tags.
But thanks for the notice. Either I didn't realize or I just have forgotten that I removed this "feature".
Can you be more precise on "browsing albumartist with and without sort tags". Do you mean without the entire patch or with a regular tagnavi.custom?
Nice to get some feedback though.
It seems that I didn't remove this (now I know why I can't remember!).
if (has_albumartist)
{
offset += add_tag(offset, &entry, tag_albumartist, &id3.albumartist);
}
else
{
offset += add_tag(offset, &entry, tag_albumartist, &id3.artist);
}
^This is the code, which implements the fall back to artist if there was no album artist. I'm a bit confused now. I'll have a look as soon I have time.
I guess what I'm saying is that one should be able to change the tagnavi from their current settings to the sort-tag equivalents and see no difference if the sort tags aren't populated.
PS I'm a tagging pragmatist, I don't put tags where I don't need them and I don't need them when there aren't any featured artists on an album.
Feel free to fix this.
PS: Having both artist and albumartist tagged (even if they're the same) doesn't hurt at all. It rather gives you more compatibility. Every fall-back from albumartist to artist is a specific implementation of the media player and NOT standard-conform.
I wonder if it has a commit chance?
The sort tags should _only_ be used internally by Rockbox when sorting the DB - never displayed (except perhaps on the metadata screen).
But looking at the code, it'll be a pain to implement. Due to the way it works, the tagtree sorts and shows exactly that what he gets by reading the tagnavi file. So if he reads the tag artist in the format, he'll lookup for artist and use this for sorting and displaying.
Just for the reference: http://www.rockbox.org/irc/log-20081212#01:21:54 Here I explain the problems I see with implementing that.
"We
"Wei
"Weird Al" Yankovic
each one carrying a diff selection of albums (some duplicate each other).
None of the "Al" has album artist in itunes. It was all build from flac using the same script. All of the artist names are identical. This could be something completely different. Just thought I would toss it out here.
The consensus though is to not use the sorting tags as seperate tags (and not showing those), but only sort by those in the database browser. Of course, that makes absolutely sense.
But IMO, it adds much complexity into code, which already nobody (except Slasheri) is really able to figure out. And it would add a good deal to RAM usage if we cached the sorting tags too besides the normal tags. Afterall, I'm not really convinced of doing it the consensus way, so I basically post-poned work on this one.
Is there any chance we could get this patch worked on for summer of code?
Gonna take a look at it, for sure :)
Would it be possible to detect at database initialisation that those tags are not used at all? Or only create the tcd entries if/when the sort tag is different from the artist? To save RAM, you know.
Other than that, good job!
I don't think so -- the sort* tags are (should be) used only when generating the indexes for the primary fields, and are not otherwise stored.
The only device I've tested this on is an iPod video. I will build the e200 sim tonight, and see if I can reproduce your error.
Big dumb grin on my end, I'm totally stoked that this is finally in the works for real! Exceptional, over-the-top props to jmillikin, way to go.
I'm dumb as hell, so I can't do it myself... but I'd be eternally grateful if someone could do a quicke patch of this to the current build for the iPod video 60/80GB so I can test it!
_jp
ramcache tries to allocate 270K here, which happens to be ~2x the value of allocation without this patch. (that's the only info I can get using logf, before the data abort, which is in load_tagcache(), btw)
EDIT: You need Load to RAM enabled for the dataabort.
> John, the sort* tags are stored in tagcache_X.tcd, where ix is 9...12. And loaded into RAM when you browse it.
Is there any way to avoid the RAM loading? With this new patch, it is necessary to keep the files around, but only for database updates. They don't need to be loaded during normal operation.
I still can't reproduce the e200 error you're seeing, in the sim. I've been using Rockbox for less than three days now, so I don't even know how to begin tracing a memory error.
> "Here is an updated patch, which allows the sorting information to be used when updating the database."
?? Can you explain this please?
> "Is there any way to avoid the RAM loading? With this new patch, it is necessary to keep the files around, but only for database updates. They don't need to be loaded during normal operation."
No. It's needed to avoid spinning the disk every time your browse for music. That's why I'm asking whether it is possible to reduce the impact, e.g. by only collecting the sort data for tracks where it is different. Hmm, but on the other hand, saving 4 int's for each song for the IDs to get the data "dynamically" is probably worse.
Edit: It could be a setting too.
I don't think it matters much... though that may be player-dependent. Using older incarnations of this patch (before they actually SORTED, but when the tag was still read and the database was still populated with it) I still had all 12 database.tcd files created and in use, and I never noticed it running appreciably slower than now when it just runs 0-9 on the SVN.
I think loading 9...12 into RAM is ok, but we should provide a user setting. Users which know they don't use special sorting tags should not suffer from doubled RAM usage, imo.
I only have 1k songs, but RAM usage already raised from 140k to 270k. Imagine this with 10k songs, or 20k. The point is not to waste RAM if the tags aren't used, not being as fast as SVN.
I dunno, I'm not a coder, just a fan of the feature, so I'm talking out of my ass here, but I just don't see a small increase in dbase size warrants having to once AGAIN go into rewrite mode on this patch to save the non-TSOing users a bit of RAM usage.
_jp
Rockbox isn't just ipod.
It doesn't double the entire DB size. But it doubles the amount of data loaded into RAM when you browse. The tagcache allocates memory as it needs at bootup, which is the maximum you can need at once (browsing by title). And the maximum is almost doubled (master index + track index w/o this patch, master index + track index + tracksort index with this patch).
OOHHHH, I think I understand. It's essentially building the db twice... once for what it displays, and once for how it sorts?
>Well, you with your 64MB ipod probably don't care. But what about the 16MB of iaudio x5
Yes, but then while it has only 1/4th of the RAM, it's also got only 1/4th of the HD space, as well (relative to the 120GB HD iPod I talked about above). That 2.2MB increase for 20,000 songs doesn't apply when 20,000 songs aren't going to fit on the player. Now you're talking about what, 5,000 songs (assuming we're comparing your 6GB e200 to a 30GB iAudio X5... I don't know for sure which ones we're talking about)? So that's only like a 500k increase in RAM consumption... 500k to a 16MB system shouldn't be much more of a hit than 2.5MB to a 64MB system, should it?
>Also, I claim that 2.2MB less RAM will also result in noticeably shorter runtime on the long run on your ipod too
Perhaps, but then isn't that how features work? Sooner or later you're going to HAVE to increase the demands on the system if you make Rockbox do more things. I don't particularly care about viewports either, I was fine with my WPS without them, but I'm betting switching rockbox to viewport-based WPSs has increased RAM usage as well. Such is the price... (Rockbox makes up for it, IMO, by adding battery optimizations, a new sleep mode feature, etc. So while some new features decrease runtime, others increase it, and it's a push in the end.)
Don't make the mistake of thinking the iPod is the only player that can have its drive replaced. Comparing an upgraded iPod (no stock 120gb iPod works with rockbox) to a stock x5 is hardly fair.
I think what we need is for rockbox only to load two tags when the sort tag is actually different from the regular tag and have it default to the regular tag otherwise. Somehow...
Okay, I can accept that. But then, you're also asking me to believe that this implementation will greatly reduce battery life on an x5 but not on an iPod... but that WITHOUT this patch, the x5's 16MB of RAM will do it just fine with a 120GB HD compared to an iPod. I hardly think it's fair to expect a 16MB system to perform as well as a 64MB system - the reality of the situation is, if you put a 120GB HD in your x5, and fill it up, you've quadrupled the size of the dbase file already.
>I think what we need is for rockbox only to load two tags when the sort tag is actually different from the regular tag and have it default to the regular tag otherwise. Somehow...
as kugel said, though, in order to do this you'd probably do more damage than you repaired by virtue of the resources required to MARK each track as "uses Sort*" or "doesn't use Sort*"
I personally think the ideal way to do it would be to only build the dbase using tags that tagcache is told to look for; so if you don't tell it to USE Sort* in your dbase tree, it won't populate the dbase with those tags to begin with. Not sure if this is doable under the current architecture, though, as I believe earlier versions of this patch built all 12 .tcd files despite none of my tags having Sort* info on them, and the tagcache never being told to use it.
Don't know how to edit a comment but...
What I'm assuming, here is that the way this patch is set to work, when you build the dbase with the tagnavi tree saying:
artist -> album -> title = "fmt_title"
It's separately creating two indexes; one for display (which is built as above), and one for sorting, which is populated with the sortartist, sortalbum, and sorttitle variants of these tags...?
I think instead, the idea should be that if you enter
artist -> album -> title = "fmt_title"
the sort tags get left alone, whereas if you enter
sortartist -> sortalbum -> sorttitle = "fmt_title"
THEN the sort tags are used... though obviously, displaying the usual "artist -> album -> title" tree.
The question remains, though, whether the dbase is configured to work in such a way that it knows which tags it needs to load and which it doesn't, or whether it just loads every tag Rockbox spec's as "usable" for tagnavi. If it's the latter, then stuck. If it's the former, switching the usage of the tags to only apply when their "sort" counterparts are entered should nicely solve the problem..?
>?? Can you explain this please?
The previous patch would only work when the database was initially generated. If tracks were then added to the device, and the database updated, the tracks would not be properly sorted.
>> "Is there any way to avoid the RAM loading? With this new patch, it is necessary to keep the files around, but only for database updates. They don't need to be loaded during normal operation."
>No. It's needed to avoid spinning the disk every time your browse for music. That's why I'm asking whether it is possible to reduce the impact, e.g. by only collecting the sort data for tracks where it is
> different. Hmm, but on the other hand, saving 4 int's for each song for the IDs to get the data "dynamically" is probably worse.
The sorting tags, if I understand correctly, are not used when browsing for music. When the index files are generated, they seem to pre-sort all tags and then cache the sorting order. I modified this process to use different strings for sorting, but the sort tags themselves are not used after that.
Hence, my question about avoiding loading to RAM. The sort tags are only needed when generating the index. They don't have to be loaded all the time.
For example, you start with the following sorted database:
Abba
The Beatles
and then add "Daft Punk". If you don't keep around sorting information for "The Beatles", your database will now be sorted as:
Abba
Daft Punk
The Beatles
1. Extract tags to temporary index file
2. (reboot)
3. Generate tag caches from index file
4. delete index file
5. (add new tracks)
6. extract tags to temporary index file
7. (reboot)
8. Load previous tag caches
9. Generate tag caches from index file
10. Write new tag caches
11. delete index file
Specifically, when I add logging to the comparison function (which is the only place where sort/not sort matters), it's *only* called when updating the caches, which apparently happens on rebooting after updating the database. Furthermore, the cached sort tags need to be available when updating, because otherwise already-populated tags will be sorted incorrectly. So the sort-tag caches *have* to be present when updating, but are not used during normal browsing. If there's any way to prevent the sort-tag caches from being loaded into RAM -- or perhaps to unload them after updating -- I do not think it would break the sorting behavior.
I assume the sort-tags caches are being loaded, despite being unneeded. Hence my question, "is there any way to avoid the RAM loading?".
On my e200 (64bit) sim, the RAM usage is higher too, btw, but not by the doubled amount.
Will test kugel's patch presently.
Please test it, I think about committing it.
Interesting conceptual problem, though, that I'd rather someone who understands the patch solve that I would attempt to by trial and error.
If I've got Artist: The Prodigy, and I add Sortartist: Prodigy, then The Prodigy should show up amongst the "Ps".
Now, suppose for one track by The Prodigy, it's Artist:The Prodigy feat. Pop Will Eat Itself. As a result, I've got that track tagged with AlbumArtist: The Prodigy.. so it all shows up A-OK, no problems.
But how do I sorttag that? Should I be using Sortartist for all of the tracks WITHOUT an albumartist tag, and Sortalbumartist for all of the tracks WITH an albumartist tag? Or, since tagnavi is being told to look at the AlbumArtist tag (and only displaying the Artist tag if the AlbumArtist tag isn't populated), should I just use the Sortalbumartist field for all in question? the question being: does the sort tag apply to whichever tag is specifically *displayed*, or does it apply to the tag that tagnavi is being told to *check*?
_jp
PS thx again for finally making this happen, SO stoked. :P
PS the tag is "artistsort", not "sortartist"..? I can never remember whether it's artistsort, sortartist, or TSOA... all seem to be used at some point or another.
The fallback order is albumartistsort -> artistsort -> albumartist -> artist.
So the actual names of the fields that the value needs to exist in are TSOA, TSOP, and TSOT?
Also, I don't know if you're right about only TSOA counting, as apparently ARTISTSORT is the preferred name for vorbis...?
Ya, sry, by bad. though doesn't rockbox need to look at both ANYWAY, considering it doesn't just use id3 tags to populate the dbase..?
Okay, well I have a bug to report then: the patch doesn't seem to DO anything.
Now, tbf this was my first actual patch-and-compile of rockbox source, so it's possible the patch didn't take for some reason and I missed it, but I doubt it, as I'm seeing the artefacts of the patch itself (e.g. showing 13/9 committed b/c the extra 4 .tcd files are being written). No crashes, no errors... but nothing is getting sorted.
TSOA tags are populated, dbase reinitialzed, and tagnavi set to browse by both artist and albumartist, but still nothing. "The Prodigy" is still showing up amongst the "Ts".
"TSOA" isnt' "Sort ARTST", it's "Sort ALBUM"! I want TSOP, as in "P"erformer!
d'oh! Excuse my stupidity, kind messrs.
Now the TSOP tags are properly populated (rather than TSOA) but still no sorting taking place, when browsing by either Artist OR Albumartist.
It works for me.
TSOP = Prodigy, Artist = The Prodigy, and the dbase still puts The Prodigy in with the "T"s.
I've tried browsing by artist vs. albumartist, getting rid of conditionals (I have a ? genre != "Spoken Word' conditional in my tagnavi) but neither works. :(
Will continue to test, let you know if I figure out the issue here...
On my MP3s with MP3Tag, it works. I used ARTIST-/ALBUM- and TITLESORTORDER.
It's id3v2, that's all the properties dialog says.
>What tagger do you use?
foobar2k
>Did you try deleting the .tcd files in the .rockbox dir before re-initializing?
I always do this when initializing (so yes)
>Also, can you try without custom tagnavi (to get rid of posible error sources)
Yes but it may take a minute, I don't have the original tagnavi on hand, will have to redl rockbox...
can you verify that these are being written to the TSOP, TSOA, and TSOT fields?
Foobar doesn't default to include sortfields, you have to specify it... I've written the tags to "Sortartist", "artistsort", and "TSOP" (and "TSOA"), but so far it's yielded nothing. I may have the id3 name for the field wrong...
Try mp3tag, it definitely supports v2.4
Have now tried with The Medic Droid (/Medic Droid) as well, to verify that it wasn't an albumartist disagreement (there are no albumartist tags in my The Medic Droid files), same result. Simply refuses to work on my end.
I'm attempting to rewrite the files with the (hopefully) appropriate field names, though frankly this patch ought to know to look for TSOP and ARTISTSORT in addition to the nonstandard ARTISTSORTORDER if that is indeed the discrepancy...
I'm telling you, this is untrue.
If it WAS true, then my "TSOP" field should have shown up with the LABEL "ARTISTSORTORDER" rather than simply "TSOP". While you may think MP3tag is using the term "Artistsortorder" as a placeholder name for the TSOP field, it's actually writing a tag *called* ArtistSortOrder. Verified this by loading the mp3s into foobar, telling it to write the sort tag with the field name "ArtistSortOrder", and loaded the files back into Mp3tag... where they showed up with Mp3tag's "ArtistSortOrder" field already filled in.
Of course, it's a moot point as this STILL hasn't fixed the problem, the Mp3tag-verified ArtistSortOrder id3v2.4 files are STILL being sorted improperly.
So what seems to have happened is when foobar wrote a tag called ArtistSortOrder, it wasn't in fact writing the correct tag... though when MP3tag LOADED the files foobar had written, it still showed an ARTISTSORTORDER tag being populated, simply because both had the same name.
This is particularly strange, but you were of course correct... foobar, though it does indeed verifiably write id3v2.4 tags, doesn't seem to understand the SORT tags. What's even STRANGER is that foobar can see all manner of nonstandard tags (if it doesn't recognize a tag it simply shows <FIELDNAME> for the tag) but for some reason these aren't showing up at all.
:S I need to go talk to HydrogenAudio, foobar is not keeping to spec like it should. Erstwhile, I believe we finally have liftoff
_jp
So now that my own idiocy has been roundly compensated for, it's time to deal with the real issues:
1) the "fallback" system seems to fail, or is at least implemented wrong. ArtistSortOrder works properly if browsing by Artist, but when browsing by albumartist it does not.
I was under the impression that the sortorder system for Albumartist went: AlbumArtistSortOrder > ArtistSortOrder > AlbumArtist > Artist... apparently, if browsing by AlbumArtist the SortArtist tag isn't even checked.
If this is intentional, that's fine (I mean, if browsing by AlbumArtist, I suppose technically it makes SENSE to have to check AlbumArtistSort), but it seems like a less-than-ideal approach when compared to checking for the four in-order specified above... especially as jmillik has said this is SUPPOSED to be the fallback order (though it's clearly not implemented).
I took a stock 3.2 source, added the v16 patch and compiled. I had it build the db up against all my oggs. All these oggs have been used with the v12 patch and worked fine in the album area showing the sort tags. With this new patch and the same files I no longer see "year album" (which is my albumsort) but instead see album and it is in alphabetical order. I am currently running this up against my mp3s with have id3v2.4 and have the sort tag set on them also for album that works fine in itunes. I will let you know how that goes. In the meantime could there be something wrong with the way it is handling oggs? or is it handling them different than v12 and previous. Also I get the db init issue that says 1/9 and ends with 13/9. That may be the result of your change to not load it into ram, it would not bother me except for the albums not sorting.
(TSOA): 1995 Alice in Chains
or on the Oggs:
albumsort: 1995 Alice in Chains
So at least on album sort there seems to be something weird.
but it ALSO doesn't use BandSortOrder, even when written from Mp3Tag or iTunes.
I assume this patch only adds support for the main three; if that's the case, I think the patch needs to be updated to use ArtistSortOrder for sorting AlbumArtist entries in addition to Artist entries.
Thomas, Sorry am I missing something? I have been reading everything but the signal to noise has been a little rough. I don't know if you are just asking me to reread all the comments, the source or explain something better.
Thanks
Oh, I guess I have some folk over at the foobar2k support forums to tell off (they seem convinced they're not yet standard).
Anyway, reading the patch I see that ALBUMARTISTSORT is the nonstandard tag being checked for use as the sort order for albumartist..? Both iTunes and Mp3tag are wanting to write "Bandsortorder" instead. No difference to me, but I'm sure this will be an area of much confusion down the road if this gets sync'd.
_jp
I also suggest, to anybody interested in testing this patch, that you either use Ogg/Vorbis files, or a well-tested tagger like MusicBrainz Picard. Trying to track down every bizarre custom tag from broken taggers will mean this patch will never be accepted.
but I will leave this saying that neither BandSortOrder (the "normal" tag for this, it would seem) NOR AlbumArtistSort (what the patch seems to say should be used, given the absence of a proper standard) are recognized. That being the case I think the patch needs to be updated so that TSOP sorts when browsing by AlbumArtist as well as by Artist, otherwise there's no way to use this patch in conjunction with browsing by AlbumArtist at all.
Is it possible this patch needs something from SVN and me sticking to the "stable" 3.2 is causing issues?
Also you can find the RAM usage under System->Debug (Keep Out)->View database info
Brian, this patch displays the normal artist etc tags, but sorts by the sorting tags.
Yes, they should... what field have you written the sort tags to, and with what program? I went through this same thing with the artist sort tags, turns out they were being written wrong.
Also, you've changed your tagnavi so it is no longer being told to DISPLAY the sortalbum field (as with earlier versions of this patch), but is instead being to display the normal Album field..?
(TSOA): 1995 Alice in Chains
or on the Oggs:
albumsort: 1995 Alice in Chains
I used mutagen in a python script to set it all up. The mp3 tags v 2.4. Also there is no older tagnavi because I applied the patch to a fresh copy of 3.2. If it was set to the old sort tags one it would show everything as blank since those tags are not loaded anymore. I get my albums. There is just no sort.
Also, a note on albumartistsort:
It seems that, when browsing by albumartist, whichever value is DISPLAYED is the one that gets sorted by. So if there is no albumartist tag, and rockbox falls back to the artist tag, it will sort those tracks by Sortartist. If there IS an albumartist tag, it will sort by SortAlbumArtist... there is no sort fallback, though (if no sortalbumartist tag is present, it sorts by albumartist, rather than by sortartist... not sure if this is the intended behaviour).
Also fixes the 13/9 problem.
-----------------
Sorting in the tagtree appears to be, for some reason, completely separate from sorting in the tagcache. I can't figure out how to re-use the tagcache indexes for tagtree sorting. I believe that with the current split sorting system, the only way to achieve correct sorting in user-specified sorting systems would require keeping sorting tags in RAM.
That is possibly correct. I'm not against doing this, however we might introduce a setting for the sort-tag support, then.
Most of the space is taken up by track titles -- a feature that I suspect would be almost entirely unused. It should be possible to allow custom artist and album sorting without much trouble.
Is a setting a problem for you?
How else would you be able to browse the albums made by a specific artist?