Rockbox

The last part of the patch was wrong (32.9.5.18) so here is a new version without that bit.

Some real progress : this patch forces the device to full speed, so we can use 64 byte packets. The send/receive model in usb_storage.c is slightly changed to ensure that only one request can come in at the same time.
The largest remaining problems are:
- no high speed
- reading more than 16k in one transfer fails, and the host does not handle this gracefully. Probably someone needs to study UMS and/or SCSI specs in some detail to find out how to solve this
- no write support yet.

Currently it is possible to mount the device in linux (provided udev doesn't try to read too much from it), but that's about it.

More progress : multi-TD transfers now work. It is now possible to mount the device in linux (read-only), and access files. It still fails at some point (I managed to copy about 60 MB of data before I got problems).

some bugfixes. This one should also work on hd-devices with 512-byte blocksize. There is also an attempt at write support, but that doesn't work yet.

Now with working write support (both in linux and windows)

Awesome work! I just did a bunch of operations with no errors:

du -csh /media/Sansa\ e260
find -name *bmp -exec md5sum "{}" ";"
cp -R "Albumname" /home/rasher/Desktop
cp -R /home/rasher/Desktop Albumname-copy
md5sum "Albumname/*"
md5sum "Albumname-copy/*"

All without flaws (and the md5sums match :-) ).

I was a bit quick there... after unmounting, it seems like filenames have been broken. When browsing around, most filenames lack their last letter, and when I attempted to reboot, I get told that rockbox.e200 is missing.

Edit: The filesystem was thoroughly hosed, and I had to format. So there's some work left still :-)

This version adds sd card support for safer write testing, and should now also compile for flsh-based targets without removable card slot

Some important bugfixes. This should be a lot more stable, but isn't fully tested. I upload anyway because the previous version has a dangerous bug.

New version with (hopefully) better block size handling

This one should detect 2048 bytes sectors correctly.

I'm using Ubuntu linux and Windows XP. When I plugged in my e280, gnome-volume-manager mounted it as /media/disk but nothing inside. In Windows XP, it recognized the device. But when I tried to access the disk, it said that the disk is corrupted and need format. Disk mode in OF works fine both under Windows and linux. Attached is the dmesg output.

It looks like the size is not returned correctly. Can you provide the dmesg output from the OF ?

This version seems to work. It avoids all known corruption-problems by using a 16 byte packetsize (which will be slow, but better than nothing).

It is good enough to install a rockbox binary over it, and then booting.

Removed some logging, and re-disabled logf. No functional changes since last time

Still have problems here. The Rockbox dmesg output is just like above. Attached is the OF dmesg output.

Looks like the size is off by exactly afactor of 1024.

Do you have a microSD card available you could plug in ? (preferably 2GB or less to make sure, but a larger one would do). I'd like to see if the same problem also happens on the SD-slot.

I'm afraid that I don't have any micro SD card. The e280's 8 GB memory is too much for me.

No problem. It was just an attempt to isolate the bug. I'm not sure why it is going wrong.

If you still want to test the rest of the patch, you can try changing line 322 in firmware/usbstack/usb_storage.c to :

capacity_data->block_count = htobe32(cinfo->numblocks * 1024 - 1);

(i.e. add the "* 1024")

Working great! Anyway it is slow. But thanks for your work. But the bootloader still boot to OF when it detected the USB, so IMO you should disable it too.

Good news! Now I just have to find out why it went wrong in the first place.

Note that you might get usb resets in your dmesg now and then. These should be harmless (but they do slow everything down even more).

Changed timeouts to be tick-based. This seems to work a lot better (a lot less resets)

Removed the actual transfer timeout since it's not needed.

tri170391 : can you try updating your svn checkout ? revision 16256 should fix the off-by-1024 issue.

This version fixes bootloader builds, and makes the bootloader load rockbox on usb-connect as well.

With the last version and the "off by 1024" fix I can read and write from and to my SansaE280.

Some observations:

I have a 6gig SDHC card (I have no "normal" micro sd card) which is recognized but it can not find a valid fat filesystem on the device. The log has:
VFS: Can't find a valid FAT filesystem on dev sdg2

With longer disk operations I always see a timeout.
usb 1-7: reset full speed USB device using ohci_hcd and address 18

It is not going into "diskmode" without forcing it to (left button). I suspect this is causing small weird problems I have been seeing before.

Attaching the OF kernel log, below with this patch.

Feb 9 23:36:45 infirit usb 1-7: new full speed USB device using ohci_hcd and address 12
Feb 9 23:36:45 infirit usb 1-7: configuration #1 chosen from 1 choice
Feb 9 23:36:45 infirit scsi26 : SCSI emulation for USB Mass Storage devices
Feb 9 23:36:45 infirit usb-storage: device found at 12
Feb 9 23:36:45 infirit usb-storage: waiting for device to settle before scanning
Feb 9 23:36:50 infirit scsi scan: INQUIRY result too short (5), using 36
Feb 9 23:36:50 infirit scsi 26:0:0:0: Direct-Access mnopqrst PQ: 0 ANSI: 0
Feb 9 23:36:50 infirit sd 26:0:0:0: [sdg] 15708160 512-byte hardware sectors (8043 MB)
Feb 9 23:36:50 infirit sd 26:0:0:0: [sdg] Write Protect is off
Feb 9 23:36:50 infirit sd 26:0:0:0: [sdg] Mode Sense: 00 00 00 00
Feb 9 23:36:50 infirit sd 26:0:0:0: [sdg] Assuming drive cache: write through
Feb 9 23:36:50 infirit sd 26:0:0:0: [sdg] 15708160 512-byte hardware sectors (8043 MB)
Feb 9 23:36:50 infirit sd 26:0:0:0: [sdg] Write Protect is off
Feb 9 23:36:50 infirit sd 26:0:0:0: [sdg] Mode Sense: 00 00 00 00
Feb 9 23:36:50 infirit sd 26:0:0:0: [sdg] Assuming drive cache: write through
Feb 9 23:36:50 infirit sdg: sdg1 sdg2
Feb 9 23:36:50 infirit sd 26:0:0:0: [sdg] Attached SCSI removable disk
Feb 9 23:36:50 infirit sd 26:0:0:0: Attached scsi generic sg7 type 0
Feb 9 23:36:50 infirit usb-storage: device scan complete

I also got:
Feb 9 23:42:59 infirit usb 1-7: reset full speed USB device using ohci_hcd and address 13
Feb 9 23:42:59 infirit sd 27:0:0:0: [sdg] Result: hostbyte=0x05 driverbyte=0x00
Feb 9 23:42:59 infirit end_request: I/O error, dev sdg, sector 10872
Feb 9 23:43:59 infirit usb 1-7: reset full speed USB device using ohci_hcd and address 13
Feb 9 23:44:05 infirit sd 27:0:0:0: [sdg] READ CAPACITY failed

Many of the below
Feb 9 23:44:05 infirit FAT: Directory bread(block 10851297) failed

Sometimes this one.
scsi 27:0:0:0: rejecting I/O to dead device

- VFS: Can't find a valid FAT filesystem on dev sdg2
That's expected. sdg2 is the sansa firmware partition, not the sd card. The sd card should be sdh, which is missing for some reason. Does the sd card work properly in rockbox ? It also doesnt't show up in the OF (the card isn't supported, but the slot should be shown)

- usb 1-7: reset full speed USB device using ohci_hcd and address 18
These are unfortunately still happening. They should be harmless, but slow things down a lot.

- I don't really understand what you mean by 'It is not going into "diskmode" without forcing it to (left button).'

- I only get the Result: hostbyte=0x05 driverbyte=0x00 and end_request: I/O error messages when I experiment with getting high speed to work. They look like USB level problems to me. Can you try checking if all cables are plugged in correctly, and maybe trying a different port on your PC (back ports vs front ports sometimes make a difference) ?

- The FAT Directory bread errors and the rejecting I/O to dead device errors are probably effects earlier I/O errors.

Re VFS, ok got it. The SDHC card is usable from rockbox but there is no new block device sdh. Maybe because SDHC is not being recognized while I thought it was (because of sdg2). The OF also does not show me a second block device with this SDHC card. I remember I do have a 2 gig standard microsd card but I need to find it :-( To be continued...

Disk mode is when the cable is connected, the music stops and a USB cable logo shows up. AFAIK the device can either be accessed as a disk and charging or just charging. The latter will allow the device to be used. With this patch I now connect the cable it will allow me to use the device and access it as a disk.

I do not see any FAT directory or any of the other errors (except the reset) when the device is in diskmode.

I changed the cable from the front connector to the back and this does not make any difference. The cables is connected correctly.

I haven't had a chance to test this yet (just found out about it, it's late, and I don't have a recent backup of everything on my Sansa) but I have both 2GB MicroSD and 4GB MicroSDHC cards, if you need me to test anything.

Frank,

I've done a little more testing on my ipod Color (60GB hard disk):

1) You probably know this already, but I'm getting a message "scsi scan: INQUIRY result too short (5), using 36" when attaching the device (others have reported this on Sansas as well).

2) I started by reformatting my device using the OF's UMS mode and installing a Rockbox build.

3) I then rebooted and connected with Rockbox's UMS mode and copied a 300MB directory of FLACs. This appeared to work fine, although I did get a single "usb reset" in the logs (earlier tests showed many more usb resets).

4) I then umounted and then remounted the device (whilst still connected in Rockbox), and the directory structure appeared to be corrupted. Rebooting into the OF's UMS mode also showed the errors. Oddly, Rockbox seemed perfectly happy with the filesystem, and played my copied album fine.


I then repeated the test (by reformatting in the OF's UMS mode), and copied the same album across, and it worked fine... I'll keep testing and let you know if I can do something repeatable.

@infirit:
- In rasher's tests (on e260), the SD card slot is always shown both from OF and from rockbox, whether or not there is a 6G card in it or not, so I don't understand what is happening on your e280.
- I see what you mean by disk mode now. Actually the USB connect screen always be should be shown on connect, but there is a bug that stops that from happening. Can you reproduce the I/O or FAT errors if you stop de music first, and don't do anything that could write to the filesystem ?

@chrisjs169:
Testing is always welcome, but right now we're still at a point where there is a small but non-zero risk involved. We also seem to have quite a few sansa testers, so we don't absolutely require more. So it's up to you...

@linuxstb:
1. Yes I know. This is because we use the SCSI-RBC command set which is recommended bu the UMS spec, but nobody else does. Someone should look up the correct INQUIRY response from the SPC-2 spec and implement that. It doesn't seem to cause any problem, so it's low priority.
4. Did you do anything in rockbox that might have caused filesystem access (or did you get the USB screen ?)

(Hopefully) fixed the usb connection screen bug. This was actually caused by a timeout expiring too soon most of the time, so I doubled it.

That gave me the USB screen indeed.

Edit: Both on Linux and Windows (XP), although my 6GB sdhc card doesn't show up in Windows.

@infirit:

Can you check if CONFIG_SCSI_MULTI_LUN is enabled in your kernel config ? You can probably find the kernel config file somewhere in /boot.

This one solves the "scsi scan: INQUIRY result too short (5), using 36". Still no sd card on windows

Some cleanups, and a fix for byte-swapping problems in the disk name

Ok, the kernel has been rebuild with CONFIG_SCSI_MULTI_LUN and the SD drive shows up. Using the v5 patch it also enters diskmode.

Standard SD works without problem but SDHC does not. It fails with error:
FAT: bogus number of reserved sectors
VFS: Can't find a valid FAT filesystem on dev sdh1.

If I have some time tomorrow I will test the v2 patch again and see if I can reproduce the errors.

There's not really a need to retest with v2.

Can you try checking the filesystem ?
fsck.vfat /dev/sdh1

fsck.vfat -r -v /dev/sdh1 gives me:
dosfsck 2.11 (12 Mar 2005)
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
Logical sector size is zero.

I also fsck'd the SDHC card with a standard sd reader and this gives me no curuption.

Any chance you could mkfs.vfat that sdhc card and try mounting it from Rockbox?

"Logical sector size is zero" doesn't sound good. Does this filesystem work in rockbox and/or with a card reader ?

Works fine in my many in one card reader and works without problem in rockbox. What it tried next,

From rockbox:
mkfs.vfat -F 32 /dev/sdcard_drive

This gives me a 2 gig fat filesystem which looks like SDHC is not working completely. It is not readable from the card reader and fsck.vfat exits with "Currently, only 1 or 2 FATs are supported, not 26". Mounting from rockbox and playing from rockbox works.

From the card reader:
mkfs.vfat -F 32 /dev/sdcard_drive

Creates 6 GB filesystem which I can mount and play music from rockbox.

It looks like the sd card before mkfs.vfat had something on it which rockbox could not handle but a card reader + linux could. Anyway it works now.

No real functional changes

The scsi device names should be a bit more meaningful now. On ipods they should match what the OF is sending. ipods also have a correct serial number.

(thanks to linuxstb for most of the changes in this one)

- Removed some custom strings after discussion
- handle START_STOP_UNIT (makes the eject not hang)
- handle READ_FORMAT_CAPACITY (makes the sd slot work in windows)

Note that we don't handle unplugging the sd card while connected properly yet

This is the same as the last one, but all new stuff is disabled by default (so it will reboot on connect, and if not rebooting use the USB_CHARGING_ONLY pseudo-driver)

defining USE_ROCKBOX_USB will enable all new stuff

FYI... I tried enabling the committed version of this patch. When plugging in my Sansa with Rockbox running, it locks up Windows Explorer (under Vista Enterprise Edition). Interestingly, even though Explorer is tongue-tied by the Rockbox USB stack as currently constituted, the device still shows up in the Safely Remove Hardware list in the lower right tray. If I click the Sansa there to remove it, that icon locks up as well. I go ahead and unplug the Sansa, and then I get the "you may safely remove your device" dialog onscreen. And even more interestingly, after unplugging the Sansa, Windows Explorer comes back to life and shows the E: and F: drives momentarily. Then they disappear and all is normal.

I guess all that means "almost working but not quite." Maybe it's just a Vista thing (which I just had forced on me and hate)?

When plugging in the Sansa turned off, it still behaves normally for me, i.e., boots the OF.

I too have had problems with Vista trying to connect to my iPod mini while using the previous v6 patch. Eventually after explorer froze and restarted the iPod did show up connected and I was able to load some songs on, although it was slow it worked. No file system damage present like previous patches. The average transfer speed was around 100 KB/s - 400 KB/s and took about 10 mins to transfer 149 MB of FLAC. Great job by the way!

New version (needs svn later than r16280):
- much less work is done in interrupt context
- rework the transfer descriptor allocation. This seems to have reduced the usb reset issue, but further testing is needed
- changed the usb_core thread shutdown as suggested by jhMikeS

Same as the previous one, but now with linuxstb's serial number code for sansa (which works for e200, but gives something different from OF on c200)

Note to testers : if you find any kind of pattern in the device resets, please let me know. Especially with ums_v8 and ums_v9 I have no idea where they are coming from.

I am getting the resets under kernel 2.6.22. But I haven't found any pattern and the system log isn't very helpful. Unfortunately, when I try to play a file on my e280 (mp3 and flac) it crashes my player with "Undefined instruction at 40004c4 (0)". It doesn't matter if I use the file browser or the database. However, the plugins still work. Not much help I know. But it does indicate that it is particular to the music playback. Let me know if there's any more info I can gather. You can e-mail me direct (jonathan.backer@gmail.com).

Thanks for your work. You're filling a huge gap.

I didn't change anything in the music playback bit, so either the USB stuff is corrupting memory or there's something wrong with your build environment. Can you play files if you don't connect USB before ?

It doesn't matter if I connect to the USB before or not. It just won't play.

When I build with the path but remove the USE_ROCKBOX_USB define from my Makefile, the firmware works without a hitch. So it's probably not the build environment.

This is the same as V9, but the serial hex letters (a-f) are lower case for Sansa, the same as the e280 OF does.

Another difference I have noticed from the OF is that Rockbox reports ANSI: 3, rather than ANSI: 0 for the OF. I was informed on IRC that this corresponds to the SCSI level. Changing it didn't make any noticable difference. Should we just report the same as the OF?

Along the same lines, the "mode sense" for the OF is 45 00 00 00, but 00 00 00 00 for Rockbox. I tried changing what Rockbox reports to be the same as the OF, but ended up getting a lot more resets and the disks never actually mounted for me on Linux.

jrbil: if USE_ROCKBOX_USB makes the difference, this is definitely caused by a bug in the usb driver. I'll try to find out more. Was this with a recent patch (v8 or v9), or with clean svn ?

barrywardell: I think we should be as standards-compliant as we can (within reason).
According to spc2r20.pdf (scsi primary commands v2), ANSI:0 means the device doesn't claim any compatibility, so the OS has to guess.
I'll try to fix the mode sense as well. The effect you are seeing is probably the OS asking for MODE SENSE data using different parameters, with 00 00 00 00 being acceptable for every one (probably interpreted as "i don't know"). and 45 00 00 00 being only valid for specific situations.

Anyway, I think we should actually start implementing what the scsi spec specifies as mandatory, instead of just doing whatever is needed to make it work (the latter will probably make it not work in some situations we didn't test yet)

Crash also occurs with clean svn.

barrywardell, the serial hex characters should be uppercase hex to conform to the mass storage spec (so the OF is violating), see document "Universal Serial Bus Mass Storage Class, Bulk-Only transport, rev. 1.0, september 31, 1999". I'll create a patch for that later today.

I agree the version and responseformat fields as reported in the INQUIRY data should correspond to actual versions. For example, if we're following the set of specifications SBC-2 (SCSI block commands - 2) and SPC-3 (SCSI primary commands - 3), those fields should be 5 and 2 respectively.

jrbil: can you check if you have filesystem errors, and if possible retest on a cleanly formatted filesystem ?

Attached patch uses the plain hex unique id inside the AS3514 chip as the USB serial number (without any "-" characters), as required by the mass storage spec.

Format and reinstall fixed the crashes. Works great now, except for USB resets.

Format and reinstall fixed the crashes. Works great now, except for USB resets.

Some reorganisations. This one should in theory be able to do mass storage and serial at the same time, but that doesn't work yet.

This one has a new and improved (really) transfer descriptor handling, which seems to eliminate the resets. I have seen it go wrong once during writing though, so if you want to keep your data safe, v7 (the one in svn) or OF is probably the better choice.

Small bugfixes since v13. The reset/disconnect/dead device bug from v13 is still there, so be careful.

It seems that the remaining problems might be triggered by control transactions not being handled in the interrupt handler any more, and thus being slower. This means that in some cases the host tries to start a new control transaction (after timing out on the previous one ?), which requires special handling by the driver software (see MCIMX31RM.pdf, 32.14.3.2).

I think this does not warrant returning everything to the interrupt handler, because (a) this problem can also occur then (although it should be much rarer), (b) this moving out significantly shortens the interrupt handling time, and (c) the new style handling allows for (IMO) much cleaner code.

What probably needs to happen is:
- implement the SETUP error-checking described in 32.14.3.2.2 (after understanding it properly...)
- find some way to make control transactions not wait on bulk. This can be done in several ways : (a) put control handling in a separate thread, and provide locking in all the right places, (b) put class drivers in separate threads, and provide the same locking, or (c) make usb_drv_send non-blocking, and adapt class drivers to handle this properly. (a) and (b) keep class drivers simple and add complexity to the controller driver, while (c) adds complexity to the class drivers while keeping the controller driver simple. I'm not sure which is best.

It appears that making usb_storage nonblocking (except for the actual ata operations) doesn't add significant complexity, so this is probably the way to go.

Both usb_serial and usb_storage are now nonblocking (except for the actual ata reads and writes). This seems to make it more stable than v12..v14, but it's still not as stable as svn.

An easy way to crash this (in linux) is to start badblocks -w on your device, and then run "while true;do lsusb -d 0781: -v;done" (change vendor id as required). This makes badblocks block after a while (watch it using strace). If you then stop the lsusb loop you'll get a device reset and badblocks will go on, but if you leave the lsusb loop running the rockbox will stop responding properly to control packets. (I recommend to only do this on a sansa on the sd card, badblocks will damage your data...)

I also managed to crash svn like this once, but it's a lot harder.

This lsusb loop is not a "normal" load, so it might be more stable in real use, but I haven't done heavy testing of that yet.

I've been doing some work on the SCSI side of things (improve SCSI error reporting, add some commands,...). It currently doesn't work well, so no patch yet. I would appreciate some comments on the thoughts below:

(a) I think RBC (reduced block commands) won't work well, because when I set the device type accordingly Windows XP doesn't find a driver, and if the device type is set to DIRECT_ACCESS_DEVICE the host OS seems to assume a standard block device.

(b) For the scsi standards compliance I would like to go for SBC-2, which might require SPC-3. The reason is that SBC-2 allows to set the maximum transfer length, which if it is honored by the host OS (to be tested...), would allow us to reduce the transfer buffer. This would require a few commands that we don't really need, but those could probably be stubs.

(c) Maybe we should use the appropriate SCSI command (probably SCSI_START_STOP_UNIT or SCSI_ALLOW_MEDIUM_REMOVAL) to trigger showing of the USB screen. This would allow to copy some files, unmount the device, and leaving it plugged in to charge while using it.

SCSI improvements. We now support MODE SENSE(10) and REPORT LUNS, and the REQUEST SENSE response has been improved

I can no longer reproduce the write corruption with 64 byte packets, so this patch uses 64 byte packets. This hasn't been tested anywhere else yet, so use at your own risk.
This change gives a speed improvement of 50% to 100%

I'm having problem to apply any of the latest patches, get a lot of Hunk #.. FAILED
are there any patch dependencies? tried with the current svn repository

ok there was something wrong with my repo, works now

Ready to do some more testing but I do not understand how to enable it :-/ Can someone explain how to define USE_ROCKBOX_USB? Thx

open the file ./firmware/export/config-[yourtarget].h
and put the following line somewhere in the file
#define USE_ROCKBOX_USB

I just tried it out and I had no problems at all, didn't crash, didn't mess up the file system.
I made a small speed test (just with the "...seconds remaining") in Windows Explorer (XP SP2) with a 21 MB file:

Rockbox: 60 seconds
OF: 25 seconds or maybe less

For me, this is more than perfect. Incredible job Frank, congratulations. Respect.

The latest path (v17) applied to the latest svn (16350) crashes my e280 hard whenever I try to connect, with both XP SP and Ubuntu. I sometimes get a blue screen od death. I've included the few error messages from my syslog below. Let me know what I can do to help diagnose the problem.

### /var/log/messages
Feb 18 20:32:45 backer-laptop kernel: [ 2054.240000] usb 3-1: new full speed USB device using uhci_hcd and address 9
Feb 18 20:32:45 backer-laptop kernel: [ 2054.408000] usb 3-1: configuration #1 chosen from 1 choice

### lshal -m
20:32:45.619: usb_device_781_7421_0000000000000000A4B4476617EA0AF4 added
20:32:45.694: usb_device_781_7421_0000000000000000A4B4476617EA0AF4_if0 added
20:32:45.697: usb_device_781_7421_0000000000000000A4B4476617EA0AF4_usbraw added
20:32:57.722: acpi_BAT1 property battery.voltage.current = 12545 (0x3101)

It worked great for me last night. As noted by kratonator, it was a bit slower than the OF but it worked fine. I saw transfer speeds as high as about 400 kb/s in spurts.

In fact, I used Rockbox's native USB support to update Rockbox itself for the first time ever.

jrbil: did you define USE_ROCKBOX_USB in your config file? building ums 16 and 17 without it defined gave me the same problems you described, but after i defined it they worked fine.

Changes:
- now passes all but one USBCV ch9 tests
- returns a SCSI error when encountering a disk/flash read/write error

To respond to earlier questions about caches.

Things not only have to be aligned to a cache line boundary but also size padded to a boundary so cacheable data isn't allocated in an overlapped manner with pieces of noncacheable data. It's best to concentrate all such data into a single structure and make sure it's padded. This helps to avoid wasting space since only one alignment/padding will be needed for all of it. If one isn't practical for all of it, of course keep the instances to a minimum.

apps/plugins/mpegplayer/video_out_rockbox.c demonstrates one (perhaps eccentric) way to do this since I'm not aware of GCC providing any sort of set-and-forget padding attribute.

Perhaps I've rambled too much :p and I hope that any questions were answered.

BTW, I think the proper alignment for PP is 16 bytes but I went cautious for now. Stick with the macros since other targets like the gigabeats actually need consideration in this manner when using DMA and they definitely are 32-byte aligned.

Thanks kfazz. I mistyped USE_ROCKBOX_USB as USB_ROCKBOX_USB. Works fine now. Used it to update the firmware.

Now uses a configurable transfer size (see BUFFER_SIZE in usb_storage.c, default 64k)

Fixed building on non-sansa

Performance improvements, by boosting the CPU when actively transfering, and using double buffering to do USB and disk I/O simultaneously

This one removes support for chained transfer requests. Their implementation was buggy, and was the cause for most remaining problems.
Removing them limits transfer buffers to 16k, but this turns out to only cause a 2% slowdown. As an added benefit, the simpler code makes simultaneous UMS and USB-serial work.

Thanks again for the great work Frank.

I made another test (same file as above) and I got just a bit over 35 seconds, which is incredible.
I'm unsure about something: when you plug in the cable while holding the center button pressed, it shouldn't appear as a drive in my computer, right? I can't notice any difference in the computer's behaviour. On the Sansa tho, if I press the center button, the USB screen doesn't appear but the connection starts anyway. I guess this can be easily fixed.

The previous comment raises a point for me. What is the preferred way to plug in a Sansa e200 for using native Rockbox USB? Turned off? Turned on and running Rockbox? Running Rockbox and holding the center button?

Some fixes:
- alignment problem on some USB string descriptors
- does not show the sansa SD card slot to the host if no card is present. This is a workaround until I know how to handle removable drives properly.
- doesn't enable serial by default

This patch still requires USE_ROCKBOX_USB to be defined, and it still connects with the select button pressed. This will be fixed somewhere in the next round of cleanups.

v23 is now commited to svn (revision 16360)

Testing with v23, I managed to transfer a 132MB file (yes comes in handy) to my Sansa in 47.4 seconds. OF took 12.6 for those wondering.

Read time (132 MB):
OF: 4.125s
Rockbox: 2m22.580s

Transferring (both to and from) a 27MB file seems to take 0.1 seconds on Rockbox though..

I'm still confused and cannot find an explicit answer to my question. Some of this has been alluded to on IRC but nowhere can I find direct answers. I hope someone with knowledge will help me. It might help others as well, or maybe I'm just thick.

What happens if I connect my e270 via USB, while running Rockbox (current SVN; native Rockbox USB support enabled), if I hold the select button while plugging in the cable?

Conversely, what happens if I *don't* hold that button while plugging in the cable?

I gather that there's some difference between these "modes" of connection, if that's the proper terminology, but I'm not understanding the difference. Is there a reason why one would hold the button or not?

major_works: There is what _should_ happen, and what actually happens.

What _should_ happen is that when holding the select button, you only charge over USB, i.e. the player doesn't present itself as a disk to the host.
What actually happens is that the player always presents itself as a disk, but the logic that makes disk connections safe is disabled by holding the button.

So in short:
- If you just want to charge, press the button while connecting. This will allow "normal" usage of the player. This is now dangerous due to a bug which allows the host to access the disk as well, which can lead to data corruption.
- If you want to transfer files, just plug in. The player will show the USB screen and disallow local disk accesses so the host has exclusive access.

I hope to fix this within the next few days

Eliminate usb core thread, fix the button-holding problem, and removes the USE_ROCKBOX_USB switch, as the code is now too integrated to easily allow that.

This has only been lightly tested, but since there are no real USB driver changes, it _should_ still be safe. Right now I am more interested in review than actual test results.

I am not able to apply the latest patch. Hunk #6 fails at 400. Reject were saved to file firmware/usbstack/usb_core.c.rej.

The content of the file is pretty big, should I upload it somewhere?

kratonator: that's because svn has evolved in the meantime. If you just want to use it, you're probably better off using the latest svn. I'll probably post a new version tonight or tomorrow.

Updated version. No real changes, except that some of it now migrated to svn.

I'll try to find a way to get USE_ROCKBOX_USB working again. It might be useful to have all code in svn for easy testing without having to enable it by default from the start.

The main issue with this version is that for some reason it doesn't see a connection if plugged in while booting.

Updates since last time:
- USE_ROCKBOX_USB is back, and needs to be enabled for UMS to work
- firewire detection is now handled separately from usb detection
- the thread priority gets increased during UMS connections

I'd like to commit this to svn, so I'd especially like testing if everything still works _without_ USE_ROCKBOX_USB. I do need to know the following:

(without USE_ROCKBOX_USB)
1.the player reboots on plugin (and firewire players reboot on firewire plugin)
2.the player doesn't reboot on plugin when the "charge only" button is held (same for firewire)
2a.the player works correctly after unplugging, and plugging in again works as expected
3.the "screenshot on plugin" setting still works as expected

(with USE_ROCKBOX_USB)
3.the player enumerates as a mass storage device on normal plugin
4.the player doesn't enumerate as a mass storage device when the "charge only" button is held
5.the "screenshot on plugin" setting still works as expected, and doesn't present a mass storage device to the host
6.the player still works after unplugging, and plugging in again works as expected
7.firewire players still reboot on firewire plugin, except when the "charge only" button is held

Also, testing if non-PortalPlayers still work correctly with this patch (mainly if hardware usb-ata bridges still work as expected) is appreciated

All works as expected on my e280 with one exception. Without USE_ROCKBOX_USB, my player crashes when plugged in instead of rebooting.

Quickly tested with USE_ROCKBOX_USB on my SanseE200 + 6Gig sdhc
3 -> works, 4 -> works, 5 -> No idea what this is, 6 -> works but will test more tomorrow, 7 -> no firewire on my sansa.

Tomorrow I will test without USE_ROCKBOX_USB if none have tested before me.

I did see a lot of data corruption with svn from yesterday (no patches) when I pushed 2 gigs of music to the sdhc card. In the logs I found a few resets and some io errors. I will try to reproduce this tomorrow with this patch.

jrbil, perhaps the device intended to reboot but crashed instead. I think there are some bugs regarding rebooting on the sansas, see:
http://www.rockbox.org/tracker/task/8272

> Also, testing if non-PortalPlayers still work correctly with this patch (mainly if hardware usb-ata bridges still work as expected) is appreciated

Tested with this patch applied on my H340. Connect, disconnect and charge only mode behave as expected. I copied over 3.1Gb and there was no corruption or errors in the logs.

Thanks bertrik. Applying that patch did indeed fix all of the problems. I'm happy to report that with the reboot patch, everything works as expected on my e280.

Well,

I hate to say it but the Mac OSX 10.5 does not like the usb storage driver.

I think this happens on alot of devices that I use :( Mac's aren't playiing very nice.

Here's the output: From dmesg

msdosfs_fat_uninit_vol: error 6 from msdosfs_fat_cache_flush
dlil_output: output error retval = 1
USBF: 337148.880 [0x3e2e400] The IOUSBFamily is having trouble enumerating a USB device that has been plugged in. It will keep retrying. (Port 1 of hub @ location: 0xfd000000)
USBF: 337148.880 [0x3e2e400] The IOUSBFamily was not able to enumerate a device.
USBF: 337362.327 AppleUSBUHCI[0x3ccc000]::Found a transaction past the completion deadline on bus 0x1d, timing out! (Addr: 3, EP: 1)

After disconnedcting and reconnecting:

USBF: 337743.369 AppleUSBUHCI[0x3e50000]::Found a transaction past the completion deadline on bus 0x3d, timing out! (Addr: 2, EP: 1)
USBF: 337774.371 AppleUSBUHCI[0x3e50000]::Found a transaction past the completion deadline on bus 0x3d, timing out! (Addr: 2, EP: 1)

Sorry more debugging noted below:

After rebooting I get this:

dlil_output: output error retval = 1
dlil_output: output error retval = 1
disk1s0: ioctl(_IOWR,'d',101,16) is unsupported.
dlil_output: output error retval = 1
dlil_output: output error retval = 1
.... continued then below
USBF: 86.370 AppleUSBUHCI[0x3e36000]::Found a transaction past the completion deadline on bus 0x3d, timing out! (Addr: 2, EP: 1)
USBF: 117.371 AppleUSBUHCI[0x3e36000]::Found a tran

I've been told that this is a problem with us just not supporting the SCSI commands that OS X expects and it should be fairly easy to add them. The line "disk1s0: ioctl(_IOWR,'d',101,16) is unsupported." looks like it supports this theory.

Other than that, on my H10 this is perfectly stable at high speed now. I've copied about a gigabyte of files back and forth several times without a single error or reset in dmesg and with the md5sums matching. What more (other than the SCSI commands) needs to be done before this can be enabled by default?

barrywardell: Once this patch is in I am happy with the overall structure of the code. After that, and the SCSI command set is more complete, we can start enabling it by default on players where it works well. (i.e. I don't think it should be enabled on the sansas yet as long as usb transfers seem to trigger write errors in the sansa ata driver)

For that scenario to work though, I really need to know that this patch doesn't harm any player with USE_ROCKBOX_USB not set. This seems to be the case for both sansa c200 and e200, and h300, but I didn't hear about others yet.

Tested on H10. 2a and 3 are fine. 1 and 2 not so much. It has the behaviour reversed for whether we reboot/recharge. Holding the right button used to reboot and let us get into disk mode, now it puts rockbox in charge mode. Not holding the right button used to put us in charge mode, but now it shows the USB screen but doesn't reboot (I'm guessing because in the usb detection there's a check for if right is held).

Admittedly, the behaviour of the H10 is different from all the others. The reason it's different is that to get into disk mode, you have to be holding down right and have the USB cable connected as it boots.

Actually, now that I look at it a bit more, the wrong behavior has been there for quite a while now, way before this patch. I think this should be dealt with separately, hopefully by just enabling UMS in Rockbox on the H10.

So, basically, this patch doesn't change the behavior on H10 when USE_ROCKBOX_USB is not set.

For me, the latest version as well as previous versions will not permit gracious disconnecting of the Sansa under Windows Vista. In other words, the "Safely Remove Hardware" method never works. I am *always* told that I cannot remove the device.

Right now, svn has the most up-to-date version, so there's no need to apply an patch. You still need USE_ROCKBOX_USB to enable mass storage, ans USE_HIGH_SPEED to enable high-speed. High speed is currently not recommended for sansa c200, since it triggers a bug in the flash driver (FS#8663), causing data corruption. This bug has also been seen a few times on e200 (but it is much rarer there). ipod and iriver H10 are not affected by this.

If you notice extreme slowness, please report the exact player, operating system, and what USB ports on your PC you used (front, back,...). If possible try different ports (back ports are expected to work better)

FYI: I experienced data corruption on my e270 using r16474M-080302. The corruption occured not in the Sansa's memory but in a 4-GB SDHC card. Several folders were corrupted to the point that they could not be deleted. It took a full format of the card to solve the problem.
For what it's worth, I'd connected the Sansa to my PC using a rear-panel USB port.

This patch provides a much cleaner separation between usb_core and the class drivers. It has no functional changes, and is mainly uploaded here for review.

There appears to be a problem with the response to SCSI_READ_FORMAT_CAPACITY. The OF returns
00 00 00 08 00 77 d7 ff 02 00 02 00 for the Sansa e260, whereas rockbox returns
00 00 00 08 00 77 d7 ff 00 00 02 02
It looks like an endian issue with SCSI_FORMAT_CAPACITY_FORMATTED_MEDIA.
Maybe these are the 2 bytes we see in the Sansa SD-driver bug?

Thanks for spotting this. The Sansa SD bug is unfortunately entirely different (it's only triggered by USB activity, not actually caused by it. We know what goes wrong, just not why or how to solve it)

I just came across another non-issue. In the inquiry data, the 3 reserved bytes contain garbage. They don't get initialized in identify2inquiry.
Actually I'm trying to find the cause of the 3 minutes lag in Windows. Windows keeps trying SCSI_READ_FORMAT_CAPACITY on the empty SD-Slot. Each time with a port reset and 20 seconds lag. It doesn't seem to like our fail CSW, but it's not the missing data_residue value...

I seem to have figured out something with the Windows lag. Windows doesn't like the way rockbox handles failed commands. If a command fails, we immediately send the csw indicating the fail. But Windows wants to see an empty command result first (on commands which expect a result). At least that's what the OF does. The OF additionally initiates a pipe reset then, but that doesn't seem to be necessary.
Since I don't know how to implement that cleanly, I did a quick&dirty patch, just to demonstrate this issue. It apparently makes Windows completely happy and let it deal properly with the empty SD slot. It's been tested with W2K, XP and Vista with high speed. I didn't dare to write much, though.

MartinR: can you try this with a blocking send (i.e. usb_drv_send()) ? Since send_csw also sends data, and the driver (currently) doesn't support more than one active transfer per endpoint, a nonblocking send there will cause problems sooner or later.