Rockbox

This bug present at least with r29153 and with latest (current) build r29203-110203.
Fault addresses are identical to specified in initial bug-report.

I don't have building environment set up at my possession at the moment. However I was able to replicate (I believe) the bug with demo decoder available http://svn.rockbox.org/viewvc.cgi/trunk/apps/codecs/demac/libdemac/ on x86 with MinGW's gcc.
Here is how it stuck

$ gdb --args a.exe buggy.ape buggy.wav
GNU gdb (GDB) 7.1
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "mingw32".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from c:\workspace\demac/a.exe...done.
(gdb) run
Starting program: c:\workspace\demac/a.exe buggy.ape buggy.wav
[New Thread 33604.0xb04]
[New Thread 33604.0x8018]

Program received signal SIGSEGV, Segmentation fault.
0x00403222 in read_byte () at entropy.c:137
137 int ch = bytebuffer[bytebufferoffset];
(gdb) print bytebufferoffset
$1 = 3
(gdb)

(gdb) bt
#0 0x00403222 in read_byte () at entropy.c:137
#1 0x0040326e in range_dec_normalize () at entropy.c:176
#2 0x004032f5 in range_decode_culshift (shift=10) at entropy.c:195
#3 0x00403398 in range_decode_bits (n=10) at entropy.c:226
#4 0x004036c9 in entropy_decode3970 (rice=0x42af38) at entropy.c:364
#5 0x004039aa in entropy_decode (ape_ctx=0x22f540,
inbuffer=0x41cc20 "\261]\032ó%íØ\372\177\067_OÙf«Þ|\320#/c\026DDaï’)£.3k–~éÔý¶ôÏ\302uN\aÔ;œZ\030£C\005gøœ^LÉ\206\321û¢O\322pðwö:ü¯Ä÷\003\346N\177\022t\261Þ¹Z¯³µ\205¿iÚEŽwG\003hYâM6þÛ?\017]dó«Ç\nß\322»Ú\377[Þ`Æ/¶©É\037\034PªöR5ÛÆz3\f]\bØ\030/Kê}­W©¸/p\205,Q¬›á\237\260“í.dçÔCµ^È\004`º\f„?¬+H01ï»:\210ç½Çe/Y£¨R„\035\017\r\237Y"..., firstbyte=0x22f53c, bytesconsumed=0x22fee4,
decoded0=0x416154, decoded1=0x41a950, blockstodecode=2227)
at entropy.c:451
#6 0x00402e9c in decode_chunk (ape_ctx=0x22f540,
inbuffer=0x41cc20 "\261]\032ó%íØ\372\177\067_OÙf«Þ|\320#/c\026DDaï’)£.3k–~éÔý¶ôÏ\302uN\aÔ;œZ\030£C\005gøœ^LÉ\206\321û¢O\322pðwö:ü¯Ä÷\003\346N\177\022t\261Þ¹Z¯³µ\205¿iÚEŽwG\003hYâM6þÛ?\017]dó«Ç\nß\322»Ú\377[Þ`Æ/¶©É\037\034PªöR5ÛÆz3\f]\bØ\030/Kê}­W©¸/p\205,Q¬›á\237\260“í.dçÔCµ^È\004`º\f„?¬+H01ï»:\210ç½Çe/Y£¨R„\035\017\r\237Y"..., firstbyte=0x22f53c, bytesconsumed=0x22fee4,
decoded0=0x413c20, decoded1=0x418420, count=4608) at decoder.c:164
#7 0x00401634 in ape_decode (infile=0x3f2c1b "buggy.ape",
outfile=0x3f2c26 "buggy.wav") at demac.c:170
#8 0x0040197e in main (argc=3, argv=0x3f2ca8) at demac.c:269

This bug present at least with r29153 and with latest (current) build r29203-110203.
Fault addresses are identical to specified in initial bug-report.

All looks like it doesn't exit chunk decoding procedure for some reason as bytebuffer[0] is already outside of the range while I can assess previous bytes.

With slightly different setting I advanced to
0x0040323e in range_decode_culshift (shift=25) at entropy.c:197
197 return UDIV32(rc.low, rc.help);

That is likely causes that division by zero.
An expert in monkey audio encoding is needed.

mudflap shows some leaks. I attach patch for Makefile and partial (without repeatetive similar entries) log.

Perhaps it worth mentioning that file version is 3970 (3.97), compression level is 2000, ape_ctx.bps=16, channels=2, formatflags=22, samplerate=44100

For some reason cf is more than 65536 in range_get_symbol_3970() as returned by range_decode_culshift(16) thus leading to symbol>64 etc...
cf=65585, symbol=271
cf=65556, symbol=271
cf=65637, symbol=271
cf=65633, symbol=271
cf=65745, symbol=271
cf=65569, symbol=271
cf=65537, symbol=271
cf=65554, symbol=271
cf=65543, symbol=271
cf=65547, symbol=271
cf=65692, symbol=271
cf=65727, symbol=271
cf=65923, symbol=271
Floating point exception

Similar code can be found from "apt-get source ffmpeg" in libavcodec/apedec.c . This code works well for the file in question.

I suspect that read_byte() should return 0 in range_dec_normalize() when buffer is over. At least libavcodec has that protection.

Here is how to break just before all the shit. Line numbers may be off. Second entropy_decode3970() causes troubles.

Index: libdemac/entropy.c
===================================================================
--- libdemac/entropy.c (revision 29204)
+++ libdemac/entropy.c (working copy)
@@ -446,6 +487,8 @@
}
} else {
while (LIKELY(blockstodecode--)) {
+ if (inbuffer[0]==0xb1 && inbuffer[1]==0x5d && blockstodecode==2227)
+ printf("hello\n");
*(decoded0++) = entropy_decode3970(&riceY);
if (decoded1 != NULL)
*(decoded1++) = entropy_decode3970(&riceX);

Here is the difference in bit unpacking between rockbox implementation and original one from SDK. `cnt` is the number of times entropy_decode3970() was called:

$ diff -y orig_mac.log rockbox_mac.log
cnt,overflow,tmpk,x cnt,overflow,tmpk,x
23136790,21,10,22345 23136790,21,10,22345
23136791,26,10,27260 23136791,26,10,27260
23136792,35,10,36017 23136792,35,10,36017
23136793,31,10,32589 23136793,31,10,32589
23136794,0,17,107768 | 23136794,0,17,-23304
23136795,11,10,12260 23136795,11,10,12260
23136796,16,11,33813 | 23136796,16,10,16906
23136797,8,11,17262 | 23136797,6,11,14195
23136798,39,11,81441 | 23136798,0,10,178
23136799,16,11,33699 | 23136799,0,11,435
23136800,14,12,60614 | 23136800,3,10,4072
23136801,13,11,28398 | 23136801,1,11,2937
23136802,3,12,13936 | 23136802,2,10,2432

Done!

It should be
static inline unsigned short range_decode_short(void)
instead of
static inline int short range_decode_short(void)

It caused wrong sign conversion near
if (tmpk <= 16) {
x = range_decode_bits(tmpk);
} else {
x = range_decode_short();
x |= (range_decode_bits(tmpk - 16) << 16);
}
x += (overflow << tmpk);

Attached patch resolves issue for me.

"inline int short " does indeed look unintentional even if legal.