Rockbox

Tasklist

FS#12135 - IAP large packet support

Attached to Project: Rockbox
Opened by Bertrik Sikken (bertrik) - Sunday, 29 May 2011, 12:53 GMT
Task Type Patches
Category Remote
Status Requires testing
Assigned To No-one
Operating System PortalPlayer-based
Severity Low
Priority Normal
Reported Version Release 3.8.1
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 0
Private No

Details

Attached patch is an attempt to add support for IAP large packets. This is needed for some devices that (try to) send a bitmap to the player, using iap packets longer than the currently supported maximum size of 255 bytes. It uses a state machine to detect the start-of-frame, detect standard/large packet, aggregate payload data, keep track of checksum and verify the checksum, instead of the previous ad hoc logic.

This is only an attempt at large packet size support, because I don't have an iap capable player myself.

This patch can be used to see if large packets work now and to see if the old standard size iap packets still work properly. Early tests with an earlier version of this patch showed signs of corruption by overflow and problems handling devices sending a lot of packets in a short time (e.g. holding a button on a remote).

To apply use -p1 in the patch command because this patch was generated from git, e.g. "patch -p1 < iap_large_packet.patch"
This task depends upon

Comment by Bertrik Sikken (bertrik) - Tuesday, 07 June 2011, 20:33 GMT
Apparently, the patch above causes hangs for unknown reason.

Here's another attempt, features:
* basic state machine for handling both regular and large packets, storing the packet payload into serbuf
* simplify calculation of return value of iap_getc
* make variable responselen local to function iap_sendpkt
* add a locking mechanism to serbuf, to prevent the interrupt from accidentally overwriting the serbuf
Comment by AndyP (AndyP) - Monday, 13 June 2011, 18:58 GMT
Hi Bertrik,

Tried the above patch, still freezes when reading a large packet.
Have tried dumping to file from within iap_getc() but it just freezes and nothing gets written.
Is there a better way to dump what its doing within iap_getc() ? possibly to screen so I can see where its freezing


Comment by Ralf Ertzinger (Lalufu) - Friday, 09 September 2011, 17:48 GMT
The patch seems to work for me so far (on an iPod Mini 2G), the device accepts command with long and short framing.

@AndyP:
even with the patch the maximum payload size is 512 bytes, which, depending on what you're trying, may not be enough. There's no technical reason for this limit, just what the code is set to right now. In addition, there seems to be a bug in the code where IAP message handling hangs after one too long message is seen (payload > 512 bytes).

The attached patch replaces iap_large_packet_2.patch. It should fix the hang you're seeing. It also adds packet timeout handling (the spec says that at most 25ms may elapse between two packets from a device to the iPod. As long as rockbox is running with 100hz ticks 30ms will have to do).

Please test.
Comment by AndyP (AndyP) - Sunday, 11 September 2011, 16:18 GMT
Hi Ralf,

I have just tried your patch and unfortunately it does not appear to make any difference, my iPod still hangs when reading a large packet when connected to my Alpine Head Unit.
Investigation has shown that it is hanging after having received the last of the large packets. I have tried increasing the RX_BUFLEN to 1024 but it still appears to hang at the same place in the code. I find it strange that it manages to receive 5 packets to which it replies with an ack packet and it then appears to hang after returning from iap_handlepkt on the sixth and last packet. I have added the iap_logging patch, updated to suit the large_packet patch, so I can see what is the last call/command made by RockBox. The last string logged is 'exit from switch' which is written after the switch statement for mode0,2,3,4,7 towards the end of iap.c. The 6 packets are for mode 04 command 0032 and are max 507 bytes in length, including the header and checksum so the 512 should be enough.

I wonder if there is some flag that's not being reset or is being corrupted. Not only does the comms stop but the iPod hangs and only a menu/select reboot will recover.

Any thoughts ?

Anything else I can try ?

AndyP
Comment by Ralf Ertzinger (Lalufu) - Sunday, 11 September 2011, 21:49 GMT
Can you get me the complete log output?

I'm not familiar with the iap_logging patch, if it uses logf() you might want to apply the patch from FS#12264, and disable logf logging to the serial port, otherwise the logging will stomp all over the IAP communication.

Even stranger is the fact that we do not implement command 0x0032 right now, the ACK is just the default action for unknown commands in lingo 4.
Comment by AndyP (AndyP) - Monday, 12 September 2011, 20:07 GMT
Hi Ralf,

The log is attached (seriallog.txt.201109112026). The logging does not use logf(), it writes to a file in the / of the iPod. Its based on the following patch http://www.rockbox.org/tracker/task/9951?getfile=19432 but updated to work with release 3.9.1.

I have made some changes within iap.c to allow my iPod to get as far as it does. These changes are around the mode 0 commands 13/15/17 so that my iPod returns the same data in Rockbox as it does with the Apple FW.
I have attached my current version of iap.c which includes various changes including the logging, the mode 0 commands and the iap_large_packet_3 patch.

Any suggestions welcome

AndyP
Comment by Ralf Ertzinger (Lalufu) - Monday, 12 September 2011, 21:51 GMT
Unfortunately that log is a bit on the unreadable side, in my opinion. Especially the long lines are at least three commands, but I suspect there are bytes missing from the log.

Attached is a version of iap_large_packet_3.patch with a bit of packet logging. To use:

- apply the patch (it replaces iap_large_packet_3.patch)
- apply the patch from FS#12264
- increase MAX_LOGF_SIZE in firmware/export/logf.h (64k should do fine, but iPods have lots of RAM)
- do an advanced build and enable Logf (but do not enable Logf to serial)
- build, install, test.
- in the System->Debug menu you'll find "Show log file" (does what it says) and "Dump log file" (writes the log to .rockbox/logf.txt).

I'd be interested in the output.

If you're feeling enormously adventurous there's a WIP rewrite tree for IAP under http://git.camperquake.de/rockbox.git (branch iap-rework).
Comment by Ralf Ertzinger (Lalufu) - Tuesday, 13 September 2011, 18:51 GMT
Obviously it was too late yesterday, as the patch produces unreadable logs, too. Fixed version below.
Comment by AndyP (AndyP) - Tuesday, 13 September 2011, 19:13 GMT
Hi Ralf,

I will re-compile tonight with the above patch(es) and check it in the morning on my way to work. Currently the iPod hangs after the last 040032 command so I might not be able to 'Dump log file', it might be too late by them.

Not quite sure what you mean about the log being unreadable. It opens fine in nedit/gedit and other editors although there is an extraneous 0x0A on one line that upsets some viewers but most editors seem OK with it. There are some extra comments in the log that let me know what the last call was in iap.c before it hung.

As for the long lines, these are the same length as those returned when using the Apple firmware. Each long line is one received packet. Those that start 0015 are from the Alpine when it is sending the Apple authentication certificate. The first 7 (0 to 6) of these have a payload of 134 bytes each and the last has a payload of 55bytes.
Those that start 040032 are from the Alpine when it is sending a picture to be displayed on the iPod instead of the standard 'Connected' message. The first 6 of these have a payload length of 503 bytes and are in large packet format with the last having a payload length of 194 bytes in the standard packet format.

Regarding WIP rewrite tree for IAP. Is it a drop-in for the current iap.c or do I need to change some other bits as well ? I don't mind giving it a try. I did notice that I would need to change the mode0 commands 0x13 and 0x15 to suit what the Alpine expects for V2 authentication as well as adding the 0x17 request for V2 authentication. My Alpine seems to be very fussy in what it expects and will display an error if it's not what it likes.
Comment by Ralf Ertzinger (Lalufu) - Tuesday, 13 September 2011, 20:03 GMT
Hi Andy.

It seems I misread the log file, the information is indeed all there.
The device authentication seems to go smoothly, although there are some oddities to it.

The iPod sends out a GetAccessoryInfo with a challenge, but I never see an answer to that. The same goes for the GetDevAuthenticationSignature command. Strictly speaking the authentication has not finished. Rockbox does not care about this, but it's strange nonetheless.

With my rewrite you'd not even get this far, as authentication is not implemented right now. I'll try to get to the point where I can replay your log with my code.
I have to admit that I'm at a bit of a loss, especially as to the total lockup of the device.
Comment by AndyP (AndyP) - Wednesday, 14 September 2011, 20:20 GMT
Hi Ralf,

Tried the above patches but iPod still hangs so can't access the menu to dump the log. Bummer....

Have attached a capture log from running the Apple firmware. In it you can see that mode 0 commands 27, 18 and 19 appear quite a bit further down in the log. I have been trying to get Rockbox to request/answer in the same order. So far I'm OK up to where it hangs.

Will keep trying to get around the hang but not sure how. I think that a buffer is being overwritten somewhere but not sure where. It might be because of the time taken to get the 040032 commands that something else is timing out.

If you get the iap.c rewritten to include authentication I'll give it a try.


Comment by Ralf Ertzinger (Lalufu) - Sunday, 25 September 2011, 18:45 GMT
Hi Andy,

There has been some progress in my IAP work. I can promise you that it still will not work, but it should get a whole lot farther than before.

If you're feeling adventurous I'd be happy if you could test a firmware built from the new code, to see how far it gets you and if it still locks up on you. If you're willing to do that I'd need to know your iPod model number.

Alternatively I could send you the diff against the current master development tree (svn:30602/git:8718d7f41fe485aa7ff27f2da9593aa3b79605a5) and you could build the code yourself.
Comment by AndyP (AndyP) - Monday, 26 September 2011, 20:52 GMT
Hi Ralf,

My Video iPod is an 80GB 5.5Gen. If you send a test firmware I'll give it a try.

If you could also send the diff I'll be able to add some logging and see where it gets to.

Thanks
Comment by Ralf Ertzinger (Lalufu) - Tuesday, 27 September 2011, 17:27 GMT
OK, by 5.5Gen I assume you mean the Video?

If so, please find a build at https://www.skytale.net/rockbox/rockbox-iap-r30602-b2c173e-110927.zip (not a direct download link!)

Inside the ZIP is the diff against svn:30602.
Comment by AndyP (AndyP) - Wednesday, 28 September 2011, 20:09 GMT
Hi Ralf,

Yes it is the Video iPod.

Tried the included firmware. Head Unit displays 'Error-01' which relates to unsupported firmware version. Looked in the diff & iap.c code and the version sent to the Head Unit is 121, the Alpine needs 123 (it reports the same Error-01 when Apple firmware 121 is used). I will try building using the diff against SVN when I get a chance. Might not be for a couple of days as I'm away on vacation tomorrow.


Comment by Ralf Ertzinger (Lalufu) - Wednesday, 28 September 2011, 20:59 GMT
That's in response to RequestiPodSoftwareVersion, right?

Well, I'm certainly not above lying about these things. Have a nice holiday.
Comment by AndyP (AndyP) - Wednesday, 28 September 2011, 21:27 GMT
Yes, that's the one.
Comment by AndyP (AndyP) - Thursday, 20 October 2011, 19:17 GMT
Hi Ralf,

Success I think..

Have managed to get my Alpine Head Unit to talk to my iPod via Rockbox..It's not perfect but it plays OK.

Had to add dummy responses to Lingo 04 Command 0x000C as the Head Unit hangs without a reply.
Also, the Lingo 04 commands 0x0020, 0x0022, 0x0024 switch appears to cause a crash in Rockbox. If I hardcode strings as a response it will play OK.
the command 0020 responds OK but Rockbox then crashes.

I will do a cleanup and then a diff against the svn with iap_large_packet_5.patch and post the rsult.

I tried using the iap.c from camperquake but it doesn't appear to work. Will try with model as 5G and firmware 1.23

Looks like its getting there......

Comment by Ralf Ertzinger (Lalufu) - Thursday, 20 October 2011, 20:20 GMT
Hi Andy,

glad to hear it.
The iap code has evolved quite a lot since the version I sent you. There has been little to no work on Lingo 04 specifically, but the rest of the infrastructure has undergone much change.
There are some odds and ends in the other lingoes I'd like to tie up first, but reworking much of the 04 code is not far off now.
Comment by AndyP (AndyP) - Wednesday, 02 November 2011, 22:09 GMT
Hi Ralf,

Rockbox is now talking to my Alpine Head Unit most of the time.

I have had to make a few changes and have attached a patch that works against the latest git tree for iap-rework. It will support logf() if the define in iap-core.h is made.

My Alpine Head Unit will not work with a firmware of 2.03 and a 160GB Classic model.
Maybe we need a model and firmware based on the model of ipod being built. This might suit older Head Units.

Also, it does not appear to like lingo 0 version 1.08 or lingo 4 version 1.0 but does work with 1.06 and 1.12 respectively.
Maybe this could also be selectable/defined in the config somewhere.

It now displays the Artist/Track/Album and updates on track change.

There are still a few commands that will cause the Head Unit to report that there is no iPod attached and then start to re-identify.
Currently I have been just adding a cmd_ok(cmd) where possible or just returning dummy data to get around this.
I will try to get better support for the missing commands when I get a chance.

I can still get the iPod to hang/crash if I send too many iap commands too quickly.

Thanks for all your help on this, it's certainly going in the right direction.
Comment by Ralf Ertzinger (Lalufu) - Thursday, 03 November 2011, 18:41 GMT
Hi Andy.

I'm enormously grateful for all your testing and the results it gets. I'll have a look at the patch shortly.

As for the iPod model number, making that follow the model selected in configure might work (although why your head unit reacts on that setting is beyond me, as it would imply that it does not work with a genuine 160GB classic, either. Is there a firmware update for the device?)

The lingo version number is more tricky, as the IAP code actually supports the features announced by the version number (that's why Lingo 4 announces version 1.0, as my code hardly supports it at all). The downgrade to 1.06 for Lingo 0 is especially worrysome. One might get away with it without causing too much fallout, as 1.07 and 1.08 did not add any actual new functions, but merely changed the behaviour of existing ones, and I do not plan to implement 1.09 (which does add new features). If you find a second, could you test if your device accepts 1.09 as a version for Lingo 0? If all else fails we might have to add an option to lie to the device.

I'll create another branch in the GIT repo shortly, which will contain the iap rework code plus all the logging I added (which is missing in the current branch). I'd be most grateful if you could build a binary with the logging code and send me some debug logs of the version number failures.

From my tests it looks like restarting identification in the middle of communication as an error handling mechanism is not all that unusual. Are the commands that cause this implemented already, or were they just stubs?

As for the hanging problem, I'm starting to suspect that the actual cause for this might be in the serial driver itself, rather than in IAP, and I do not know how to debug that.
Comment by AndyP (AndyP) - Friday, 04 November 2011, 21:44 GMT
Hi Ralf,

Built a few variations last night from the git iap-rework tree and tried them this morning and found the following,

160GB Classic with 2.0.3 is OK. I last tried this a while ago with the svn code but there has been a lot of changes since.
Lingo 0 v1.09 is OK, I will try again with 1.08 when I get back from the states in a couple of weeks.
Lingo 4 v1.00 is NOT OK, the Alpine displays ERROR-02 and stops communicating until you remove the iPod, but 1.12 is OK. I have not tried versions between 1.00 and 1.12 to see which is the minimum. I will try variations when I get back.

The commands that cause a restart in lingo4 were not implemented and so the reply was IAP ACK BAD PARAM which causes the Aline to restart the identification. I just added some stubs to stop it complaining and restarting. Some/most of them need more work to respond properly with the correct information.

Currently it seems to get confused between pause/play when connecting as sometimes it ends up paused and I have to manually press play. Not sure why its doing it but it's not a critical issue.

The hanging is a critical issue but I don't know how to debug the serial driver either. As long as I don't try too many FF/Rewind/Pauses in a row it plays without hanging.
Comment by Ralf Ertzinger (Lalufu) - Saturday, 05 November 2011, 23:59 GMT
Hi Andy.

You've lost me there for a bit. You're saying that the 160GB classic model number with firmware 2.0.3 actually does work with your device, using the current code?

That the device declines to work with Lingo 4 version 1.00 is expected, it wants to use commands that this version does not provide (and it is the first device I've encountered that actually gives a damn about the Lingo version numbers the iPod provides, all other devices I've tested so far just use whatever they feel like).
Comment by AndyP (AndyP) - Friday, 25 November 2011, 18:39 GMT
Hi Ralf,

Just back from Memphis Tennessee (business) and trying to catch up.

With the current Git code my iPod will work with my Alpine Head Unit using the 160GB classic model and 2.0.3 firmware.

The above also works using Lingo 0 Version 1.08.

The above also works using Lingo 4 Version 1.06. It does not like Version 1.00
I will continue to find the minimum supported Lingo 4 Version.

As an aside, I had a Toyota MPV in Memphis and couldn't use Rockbox on my iPod. The Toyota's Head Unit uses a standard USB cable connection and so must use IAP over USB. Bummer.... Had to use Apple firmware. Not impressed, I don't have many tunes on it that way. We need IAP over USB support....any volunteers ?

Comment by David Hall (Soap) - Saturday, 26 November 2011, 15:13 GMT
AndyP, your problem with the rental Toyota MPV is likely related to the problem seen here:
http://forums.rockbox.org/index.php/topic,24360

Comment by Ralf Ertzinger (Lalufu) - Saturday, 26 November 2011, 15:49 GMT
For what it's worth, I think AndyP is right in his assumption. During my tests for the extended IAP support I encountered several devices that simply do not support talking to an iPod via the serial pins any more (even though all existing iPod models can do serial), but expect to use IAP over USB. Usually those devices officially only support the touch screen iPod models, all of which can do IAP via USB. I can see why, to be honest, USB is much less error prone and much faster.

Now, Rockbox could support USB via IAP, all the basic infrastructure is there. I have spent a rather unsuccessful afternoon some months ago trying to talk to an OF ipod touch via USB (trying to emulate a device) and have not much to show for it. There's a good chunk of documentation on the theory, so if someone wants to take a shot on implementing the USB transport layer it could be docked to the existing IAP code (some assembly required).

Apart from this I have had preciously little time to work on the IAP code.
Comment by AndyP (AndyP) - Saturday, 26 November 2011, 22:23 GMT
I am fairly certain that the Toyota is using IAP over USB.
I tried various Mass Storage devices ranging from USB Flash drives, USB HDD's and uSDHC cards in USB adapters, the Toyota played all of the MP3's files fine regardless of what other files were on the memory. On the iPod, it would only work with the Apple firmware. Unfortunately I forgot to take my Rockbox VM so couldn't try changing the VID/PID but I'm sure the reason is IAP over USB.

Now need to get back to IAP over Serial and unsupported Lingo 4 commands....
Comment by AndyP (AndyP) - Tuesday, 29 November 2011, 19:56 GMT
A slight update regarding the Alpine Head Unit.
Requires Lingo 4 Version 1.02 or greater. If I try 1.00 or 1.01 the Alpine displays ERROR-02.
Fussy bugg*r isn't it?

Loading...