Rockbox

Tasklist

FS#5852 - Running out of space corrupts recorded file

Attached to Project: Rockbox
Opened by TheHYPO (TheHYPO) - Saturday, 19 August 2006, 00:03 GMT
Last edited by Peter D'Hoye (petur) - Sunday, 16 December 2007, 00:03 GMT
Task Type Bugs
Category Recording
Status Closed
Assigned To Peter D'Hoye (petur)
Operating System SW-codec
Severity Medium
Priority High
Reported Version Daily build (which?)
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

This could be in other players but I use the H100 iriv. I took the player to a show to record a show but forgot to check the space on it. I ran out of space on the second set and the 200-plus meg file was corrupted and useless (despite almost making it the entire set, and thus losing that data). It would be very desirable for the player to know when there is no longer enough room to save the file and cutoff the recording and save correctly. I admit I'm using a somewhat old firmware so if this has been reported or even fixed, I appologize for wasting time.
This task depends upon

Closed by  Peter D'Hoye (petur)
Sunday, 16 December 2007, 00:03 GMT
Reason for closing:  Fixed
Additional comments about closing:  Can't reproduce anymore since last fix, I guess I'll hear if it's not ok ;)
Comment by Michael Sevakis (MikeS) - Thursday, 03 May 2007, 23:01 GMT
This was reported recently as well. Obviously not a problem with recording at all since this was before the first big update on August 28, 2006. A bug in the file APIs (or a layer below it) must be to blame since it shouldn't even be possible to keep writing data if the disk is full. I'm trying to link to the recent incident now but can't seem to find it now. argh.
Comment by Peter D'Hoye (petur) - Thursday, 12 July 2007, 20:45 GMT
file.c does some strange things on line 539: I'm not sure it would always return an error on disk full.
Comment by Peter D'Hoye (petur) - Friday, 13 July 2007, 17:15 GMT
actually.. it looks ok... if an error occurred but some bytes were written, it will report the number of bytes written, and not the error...
Comment by Peter D'Hoye (petur) - Saturday, 14 July 2007, 13:56 GMT
external report (does however smell like it is a hardware problem)
http://taperssection.com/index.php?topic=87758.0
Comment by Peter D'Hoye (petur) - Sunday, 30 September 2007, 12:18 GMT
first try at having it at least write the header on failure.
There is still an issue: after rebooting, the file becomes 0 bytes.
This seems to be an issue with a failing flush of data not closing the file properly (in file.c)

(partial fix, work in progress: only changes wav_enc, not mp3,...)
Comment by Peter D'Hoye (petur) - Wednesday, 03 October 2007, 23:25 GMT
work in progress: no more corrupt disk or recording, but the fat_flush code now does a panic....

EDIT: have only had the panic once -> to be tested!

Also still only implemented for WAV
Comment by Peter D'Hoye (petur) - Thursday, 04 October 2007, 20:46 GMT
almost final version. Anybody care to test?
Comment by Peter D'Hoye (petur) - Thursday, 04 October 2007, 23:34 GMT
it has given me three PANICs so far at the end of a recording with disc full

apart from the panics (which just require a reboot) the recorded file is fine and the FAT is not corrupted (except for the free cluster count)
Comment by Peter D'Hoye (petur) - Sunday, 07 October 2007, 23:00 GMT
hmmm several tests caused chkdsk to complain the file is reported too big (4096 bytes or one cluster)

we're not there yet

EDIT: not failing any more right now. Anybody else testing?
Comment by Peter D'Hoye (petur) - Monday, 08 October 2007, 23:42 GMT
adds extra checks in file.c for validness of filepointer
Comment by Davide (Davide-NYC) - Saturday, 13 October 2007, 19:22 GMT
This is (unfortunately) not fixed.
See http://forums.rockbox.org/index.php?topic=12500.msg99879#msg99879 for more detail.

Since this affects all targets and is a filesystem bug (according to some devs) I believe this is a high priority bug.
It is also the last bug left in term of core recording functionality on the iriver h1x0 targets.

Let me know if I can do anything in terms of testing to help with this.

Davide-NYC.
Comment by Peter D'Hoye (petur) - Sunday, 14 October 2007, 21:49 GMT
Have been testing again and I can't make it fail :(

EDIT: made the amount of free space a bit bigger and now it failed... Investigating...
Comment by mattias svensson (lerklompen) - Monday, 05 November 2007, 12:46 GMT
after diskspace is full it seems as if it continues recording into available RAM on my sansa c200, because it continues approximately 26 megs beyond available space on disk before it stops and reports that disk is full. which would indicate that it includes the RAM as available space?
Comment by Linus Nielsen Feltzing (linusnielsen) - Monday, 05 November 2007, 13:05 GMT
No, but it indicates that it only checks for available disk space when it tries to save the RAM contents to disk.
Comment by mattias svensson (lerklompen) - Monday, 05 November 2007, 13:17 GMT
the beginning of the file has already been written to disk, it should stop recording before continuing into RAM, shouldn't it?

to me it seems as if this is the same file, that continues inte available RAM.

if i watch the size of the file recorded, it is available diskspace + 27 megs. and the result is a corrupt file which lacks those last 27 megs...
Comment by Michael Sevakis (MikeS) - Tuesday, 06 November 2007, 04:19 GMT
Recording is done to RAM first, flushed to disk when RAM is full and continues in RAM at the same time. The data written to disk first is the oldest data in RAM and writes may fail before all of it can be flushed out. It's strictly FIFO. The first write to disk is a template header but only for files with headers which doesn't include MP3 at the moment - perhaps an INFO header will be added later. This is filled-in with the final stats just before closing the file. If recording has managed write this template header without error, it should at least be able to update it with what it managed to flush so far even if the disk ran out of space since it simply overwrites it in place. You should end up with a good file simply missing what's been left behind in memory.

If any cached sector flush must occur before seeking and filling-in the header and the disk is already full, attempting to do the fill-in may in fact fail anyway (right?). :) If that's correct, then this unflushed sector data must be discarded first so no further attempt is made to add more data to the file. Ok, I could be full of BS too. Do we even keep a good record of available disk space at all times?
Comment by mattias svensson (lerklompen) - Tuesday, 06 November 2007, 08:32 GMT
does this mean that it actually fills up RAM completely before it does any writing to disk at all?

my sansa c200 records natively in WAV, i havent tried to do recordings in MP3.

the WAV file thats is left on disk is reported as corrupt and unreadable by msOS. though after a msScandisk i could import the raw data in audacity, so it seems you are right.

-----
then there's still the problem of the system freezing - if you get the message "The disk is full. Press LEFT to...", and don't press left within a few seconds the system freezes, if the display wasn't lit it just seems as if it is "dead".

Comment by Peter D'Hoye (petur) - Tuesday, 06 November 2007, 09:05 GMT
yes. First fill up all RAM, then write it to disk. This is done like this for all targets although the reason we do it like that is only valid for HDD targets.

Yes there are still bugs with the disk full message: on H10 it shows the incorrect key and also doesn't switch on backlight on keypress.

I'm still hunting down the actual bug that causes disk corruption, it now only happens in 50% of the cases. There is some bug in the setting of the file size, and the code that keeps track of it puzzles me. Other than that, I've had limited time to look at it :/
Comment by Michael Sevakis (MikeS) - Wednesday, 07 November 2007, 04:29 GMT
I sure as heck wouldn't want to record directly to the flash all the time but then as long as you're not prerecording to flash in any way...not much different. With buffering in RAM you find out a few minutes later what you'll find out no matter what and your recording would be cut off at exactly the same point in either case.

Personally I'd never take a device out for any serious recording without it being essentially empty. I'm surprised people fill their drives often enough to notice this particular bug.

Bleh, I babble.
Comment by mattias svensson (lerklompen) - Thursday, 08 November 2007, 08:32 GMT
the sansa sound quality is much better in wav, which obviously requires a lot more diskspace (i only have 1gb), i'd say it's no surprise that people run in to this bug.
Comment by Peter D'Hoye (petur) - Thursday, 08 November 2007, 08:50 GMT
I do agree with Mike: if you're going to record, have enough space free. If the file system is fragmented (which it is most of the time) then you'll even risk having dropped packets before the disk is full because write performance degrades.

But Mike, you were babbling a bit, I will not keep my 80GB drive empty for recording, 25GB free ought to do it ;)
Comment by mattias svensson (lerklompen) - Thursday, 08 November 2007, 10:55 GMT
still, having a small flashbased recorder with approx. 1-2gb and recording to wav, makes it easy to fill up your diskspace.

say that you intend to do a long recording of some sort, and that you accidently forget to turn it off, or you just didn't check <u>exactly</u> how long you can record - it could easily be that you have a long recording rendered useless because you're out of space, instead of a long recording that gets cut off when disk runs out.

(it seems you are arguing to keep this bug, and people should blame them selves?) i think rockbox rules, but at the moment OF handles recordings better on the sansa c200.
Comment by Peter D'Hoye (petur) - Friday, 14 December 2007, 14:21 GMT
committed another bugfix, please try....

for some reason my H10 freezes solid if I leave it on the DISK FULL splash too long, don't know if that is related or part of the open issue?

if nobody reports an issue I'll close this one next week

Loading...