Rockbox

Tasklist

FS#12129 - Duplicate database entries

Attached to Project: Rockbox
Opened by bug (bug) - Wednesday, 25 May 2011, 20:45 GMT
Last edited by sideral (sideral) - Thursday, 09 June 2011, 22:07 GMT
Task Type Bugs
Category Database
Status Unconfirmed   Reopened
Assigned To No-one
Operating System Sansa AMSv1
Severity Low
Priority Normal
Reported Version Release 3.8.1
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 0
Private No

Details

I don't know whenever it happens with other versions or not, all I got is Sansa Clipv1 to test with.
My configuration is Initialize once and auto update working.

It seems like the database items got duplicated for unknown reason. I seem to have up to 7 entry's of the same item inside the database. I am not sure what causes it, but that is what I've noticed.
[At first I thought it's a shuffle bug 'till I was going to debug it only to find the database is full of clones]
This task depends upon

Comment by sideral (sideral) - Thursday, 26 May 2011, 13:51 GMT
  • Field changed: Summary (Duplicate database entry's → Duplicate database entries)
I assume you have already checked for filesystem errors (with fsck.vfat or chkdsk) and for actual file duplication (in a trash-can or similar folder; look at the file names in each duplicate item's properties, reachable from the context menu)?

Unless your database contents are terribly embarrassing ;) please attach a copy of your /.rockbox/database*.* files to this tracker item to allow a closer look at the nature of this corruption.
Comment by bug (bug) - Thursday, 26 May 2011, 14:31 GMT
Gah, I should have waited, I guess. Now I'll wait to wait for it to happen again. I already reformatted the clip with Panasonic SD formatter.
What I can say is that while working with the clip via a PC I didn't see any file system errors [as in copy into and from].
There was no trash folder nor any duplicates - I would notice if I have the file physically 7 times. Especially when I just backup the entire device with no errors at all before reformatting [Which did the files 100% fine].

File names were just fine. I would have the same item a couple of times, each with the same destination.
For example:
/MUSIC/Games/Chrono Symphonic/Chrono_Symphonic_09_Frog's_Intervention.mp3
Each of the duplicates would point to the same location. All of them would also play just fine.

Sadly, I had already deleted the backup for when moving the files back and forth from Clip & PC. Though I still don't think I would share the database*.* [Some sensitive (personally really - Nothing that would be embarrassing, just privacy) stuff].
Comment by sideral (sideral) - Thursday, 26 May 2011, 15:05 GMT
Thanks for your response, bug! You could share the database privately if you're concerned about posting it to this public website.

Unless you have a way to reproduce the problem (possibly with a minimal data set to make the database files sharable) I'm afraid I'll have to close this issue.
Comment by bug (bug) - Thursday, 09 June 2011, 22:51 GMT
I think I can re-produce it by deleting a file using the Rockbox firmware. I still need to do a bit more tests to be sure, but from the usage of the device, I'm pretty sure it's related to deleting.
Comment by bug (bug) - Thursday, 09 June 2011, 23:23 GMT
Ok, I've confirmed it. Deleting a file using the Rockbox firmware on Sansa Clip v1 does add all the files on the device to the Database without wiping it. Hence, cloning it.
This does not happen with the simulator built as of now on my Linux box.
Comment by sideral (sideral) - Friday, 10 June 2011, 07:37 GMT
Bug, thanks for the update and for you efforts to reproduce the issue with the simulator!

Perhaps the issue is configuration-dependent. Have you copied your device's /.rockbox/config.cfg to your simulator's simdisk/.rockbox/config.cfg?

We've recently developed a theory that duplication could occur in DB views when a DB query returns a track multiple times and the subsequent uniq process fails for some reason (e.g., uniq buffer to small). Hence, I'd like to know first whether the files actually are referenced multiple times in the on-disk database, or whether they just appear multiple times in the DB browser. For this, I'd still like to look at your /.rockbox/database*.* files. If you cannot share your main database and creating a minimal example with the simulator doesn't reproduce the issue, you could try generating that minimal database on device. You could use database.ignore and database.unignore files to prevent Rockbox from scanning your private music collection [1].

A few more questions / requests (please include all relevant information here, and do not rely on any information surviving former IRC conversations):

Could you please explicitly list the exact steps you take to reproduce the issue? (From your description it's still unclear to me whether you update the database after you have deleted the file, whether or not you reboot after having deleted the file, and which database view is showing the duplicate entries, for example.)

Does the duplication survive a reboot / database update / database init?

Could you please attach your /.rockbox/config.cfg file to your response?


[1] http://download.rockbox.org/daily/manual/rockbox-sansaclip/rockbox-buildch4.html#x7-450004.2.2
Comment by bug (bug) - Sunday, 12 June 2011, 08:38 GMT
__Could you please explicitly list the exact steps you take to reproduce the issue? (From your description it's still unclear to me whether you update the database after __you have deleted the file, whether or not you reboot after having deleted the file, and which database view is showing the duplicate entries, for example.)

Ok, I think I've managed to sum it into:
A) Play music on the device.
B) Delete the current played file.
##C) Reboot immediately?
D) Turn off the player.
E) Turn on the player.
F) Play some music while the database gets updated automatically.
G) When updating is done [noted by the disk activity icon], reboot the device to find database got updated.

__Does the duplication survive a reboot / database update / database init?
Reboot - Yes. [Survives]
Database update - Yes. [Survives]
Initialize - No. [Doesn't survive]

__Could you please attach your /.rockbox/config.cfg file to your response?
It's the default one with database auto updates on [I know, because I wiped my .rockbox folder]. - But just in case...
Comment by sideral (sideral) - Sunday, 12 June 2011, 22:57 GMT
Hi Bug, thanks for your efforts reproducing this bug, and for uploading your database! This is extremely helpful.

I need to do some more digging, but at first sight, your database indeed contains multiple copies of some (all?) tracks. Interesting.

It's also interesting that you trigger this issue by deleting the currently played track. I've also experienced issues with doing this, but in my case, the player typically crashed or was hung hard. Another bug related to deleting the current track is tracked as FS#10874 (No track will play after playing a deleted, semi-buffered track).

I'll report back when I find out more (no promises as to when that'll happen). In the meantime, could you please fill in some missing pieces in your instructions for reproducing the bug:

Step C – What do you mean with “Reboot immediately?” – do you have to reboot to reliably trigger the bug, or not?

Step G – What do you mean by “find database got updated” – how do you tell? By the increasing counter during DB commit before the main menu appears?

After Step G, you presumable enter the DB browser to look at the tracks somewhere – which menu entries / queries do you use?

Thanks!

PS: Uploading the DB in .lzo format was less helpful, as I had to waste half an hour finding the right archiving program to uncompress it. Please don't do that again.
Comment by bug (bug) - Saturday, 25 June 2011, 23:13 GMT
1) Step C - I think you have to reboot the player to get the bug, but I am not sure. I think it can happen without rebooting, but I had hard time reproducing it without rebooting.
Step G - Ya, automatic update thingy of the step X/9. Letting it update the duplicates.
After step G: I normally just go to Artist, All, All, pick a song and shuffle all really.
But the bug can be seen in any other menu :). Easier to see it if you try something spesific.

2) Sorry about that. I just wanted to try that compression so much that I forgot some people might have hard time with it. Would tar.gz be better next time if needed? Or would tar.xz or 7z be preffered? or whatever that isn't zip / rar / ace?
Comment by padavoine (padavoine) - Saturday, 13 August 2011, 19:52 GMT
Same issue here, DB attached. Using fresh SVN build with USB PIO patch on Fuze v2.
Comment by Kenneth Brown (GigaBrick) - Wednesday, 17 August 2011, 11:14 GMT
Okay, here is what happened when I reproduced this on a Gigabeat F

Disconnected from USB and got "Error accessing playlist control file" splash
Clicked the splash off screen, tried to "Resume play" again and got "Invalid control file" splash
I loaded a playlist, and got "Error updating playlist control file" and the player locked up hard, with disk spinning
Power would not return to main menu or power down player, so I used the battery switch
Hooked up to the PC first to check the filesystem, it came out clean
Powered player back on, checked database, and found duplicates

Comment by John Halko (jhalko4) - Monday, 22 August 2011, 21:14 GMT
So, I have managed to produce this bug for you guys and am have a hell of a time trying to fix it.

After backing up the database files, I promptly deleted them from the ./rockbox directory on my Sansa Clip+ running Rockbox 3.9 installed using the Rockbox Utility. I then Initialize Now'd and the new database still contained duplicates.

I think I triggered this bug when I first hooked up the Clip via USB to my desktop to charge. When I did this it reverted back to the Sansa UI to charge. I ejected it and rebooted to find the Rockbox UI once again. I then proceeded to reconnect via USB with the Clip powered on (in Rockbox UI), but my desktop could not detect the device. It was soon thereafter that I noticed duplicates.

Misc notes:
- I am running Linux 10.04
- I was forced to delete the aforementioned database files though the rockbox UI.
- My music is contained on an external 32GB Adata microSD card. This is also where I backed up the database files.
- I had Auto Update on, turned it off for deletion of database and ensuing Initialize Now.

Attached are the database files. I am not very savvy when it comes to this kind of thing, but I hope this helps.
   DB.tar.gz (321.9 KiB)

Loading...