Rockbox

Even with this above patch, the recordings still clip (in the usual sense, not the funny overflow where it wraps from +limit to -limit) at -12dB. The as3214 is only a 14-bit DAC, so this might just be that the sample needs to be shifted up 2 bits to reach full scale. If so, I'm not sure where that should be done, though.

if shifting is needed it should be made in pcm-as3525.c similar to what we do on AMSv1 to fake stereo recording on (mono) microphone input

funman reported that there was no perceptible difference with this patch on a Clipv1 (but also no clipping to start with), so I investigated a bit more.

It looks like the real root cause is that the register bit the code thinks turns off the AGC does something else on AS3543; it changes the mic circuit between differential and single-ended mode. Recording pink noise with steadily-increasing volume shows very clearly that we are still in some sort of automatic gain-control mode.

There does not appear to be a register bit to turn AGC off on anymore, but noting that the AGC and the "Limiter Gain Overdrive" are the only things in the data sheet with resolution of 0.375dB makes me suspect that they are the same thing.

That would imply that the AS3543 doesn't really do "automatic gain control" like the previous ones did, it just attenuates the signal to <1Vpp (which can be used to accomplish the same thing; if it's turned up too loud, then it's automatically reduced to just loud enough).

So I think on the Clip+ (and presumably other AS3525v2), we are running the mic with too much supply voltage, and the limiter is functioning as AGC. Since that will almost always produce a signal near -0dB, it makes sense that an ADC gain that is positive will cause clipping. Do we know where the setting for AudioSet2.VMICS=0 (2.0V) came from?

It would be good to do the 14-bit to 16-bit conversion at any rate. Right now the maximum sample level is at 0.25 (I think).

BTW, Unless I crank the gain setting up to above 9dB there’s no clipping in my clipv2 recordings. And the agc doesn’t kick in /perceptibly/ unless I bang the thing on the table.

As of r30209 I've updated the recording code to scale all samples to 16 bit. During my quick testing, voice recording sounds pretty good as long as I don't put the gain too high.

That said, looking at the datasheet, it would appear that we do switch from differential to singled ended mode when recording because the AGC register was remapped. I'm not sure what the implications of that are.

For what it's worth, the Clipv2 also exhibits the overflow clipping behavior on v3.9.1.
RB recordings always max out at 12dBfs, regardless of what the gain is set to.

>For what it's worth, the Clipv2 also exhibits the overflow clipping behavior on v3.9.1.

Thanks, but that build is older then r30209, which I believe fixes the problem. If you update to something post r30209, does the problem still occur?

Hi - RB newbie here. Thanks a lot for all the work on RB.

I see exactly the same "integer overflow" problem on Clip+v2 with 3.9.1, forcing me to use recording gain of -3dB, resulting in typical recorded levels of -30dB. I've modified the Clip+ by removing the mic and bringing an external input into the high side mic connection. The input is live music (square dance), with external AGC before the Sansa, so dynamic range is moderate. As long as I keep the level there, I get excellent recordings.

With a private version I just built from r30883M (presumably including saratoga's fix) the integer overflow is very much more noticeable, even at lower levels than avoid the problem with 3.9.1.

I'm pretty motivated to get this running well. My targets (and only hardware) are Clip and Clip+; output is MP3 at 96Kb/sec 44.1KHz mono.

More homework to do - I haven't looked at the patch or diffed code to see the post r30209 change yet, and have only glanced at the datasheet so far.

I'll post back what I learn, but would be grateful for any other bits of intel about this from the guys that actually understand the code :)

Would you post a WAV file that has this problem?

Also, because of FS#12350 you should only use mono.

Sorry for the delay. 391_+3db.wav shows the "overflow clipping" or whatever we want to call it in 2 instances around 3.135, 3.157 sec, again near 12.218, with more in the last quarter of the file. They sound like clicks or pops.

I didn't see/hear any in 391_+0db.wav.

I could see the same instances in a file recorded at +3db in mp3. All files above are from the distributed 3.9.1 image - I haven't tried rebuilding that from source yet.

The local builds I've done so far are from r30883M. In r30883_-3db.wav. I think I could see the artifact ~12sec, but the real issue is that it just sounds really bad.

My next steps are to rebuild 3.9.1 from source and play with it, then start looking at the code.

Small update. I was wrong when I characterized the result of my private build as integer overflow: While it is very distorted, I DON'T see the characteristic full scale swings shown in Clipped.png in the original bug report. Sorry about that. (I do see those in 3.9.1 when recorded level is above around -30dB.)

I have rebuilt 3.9.1 from source and it behaves just like the original image I downloaded (which is a good thing). That is, at Gain -3 I get recorded level around -30dB, but with no flaws.

I downloaded dailies latest (20111110) and oldest (20111014) and both exhibit the same very garbled, maybe "underwater" sounding distortion I heard with my private r30883 build.

I also put daily 20111110 on an unmodified Clip+ and recorded a little voice thru the mic. With Gain 0dB, I got a recording at -30dB with my voice garbled/burbly just like the music I'd put into the other recorder with the mic removed. I've attached a little clip of that.

This record quality is pretty unacceptable - but it's not in a released build. Sorry - I don't know protocol - should that be raised as a separate bug, or do we just fix it with this one?

I don't think there is any point t in opening a second task.

OK - I manually applied the clipp-gain.patch to my 3.9.1 tree. It changed the gain some. Before if I was lucky I could record at Gain +0. Now if I'm lucky I can record at +12dB. But what gets recorded in those 2 cases is the same level, -26dB.

But the very interesting thing is that I looked more closely at the places where the integer overflow occurs, and (with and without the patch) it's always around absolute level of +0.249. Then in one sample it jumps to -0.250. It looks like we're clearly just 2 bits shy of full scale. If we could pick those 2 bits up, the overflow would be at FS and nobody could complain about it.

Of course there's still whatever is broken in later builds to clean up.

>If we could pick those 2 bits up, the overflow would be at FS and nobody could complain about it.

To be clear, that was the problem I fixed after 3.9.1.

Cool. And it looks like your fix did in fact fix the 0.25 record level ceiling, since I've seen recorded levels well above that with later code (both mine on r30883 and the daily builds).

So how do we find what introduced the garble distortion?

A little news. I (figured out a little about svn and) built r30209 from source. It clearly increased the record level, exactly as advertised. Unfortunately, the burble/garble distortion is there in that build as well. I've attached a little 1KHz sine wave recording with that build. Looking at the spectrum of that recording, there's a small spike a little above 21KHz that I believe is a power supply artifact. I really ought to put a bigger cap across it. But I don't think that's related. The peak at 1KHz is more spread out than I'd expect.

I looked at the waveforms of the 1KHz recording and found some distortions. All looked like perhaps missing samples. I've attached a picture of one instance of it. I scanned through the recording by eye and found 15 consecutive "missing samples" or whatever they are. All were almost exactly 23ms apart. Does that offer any insight? Is there an interrupt at ~23ms that might be spending too much time in the service routine or something?

I looked at the recorded waveforms from r30209 more carefully, this time time on a 850 Hz sawtooth with every 10th cycle silent so I could see if whole cycles were missing. The "missing samples" events occurred just as before, but now it's clear that just over 1 ms of samples are missing. In the picture you can see that the block with the glitch has one fewer cycles than the adjacent normal block. At 44.1KHz, I think that's ~50 samples. Is that right?

Since 391 meets all my needs (I think - haven't built yet for the V1 and V2 Clips I have, just for Clip+) except for the low record level, I hoped to be able to just scale the level up using that code base. I added just the static inline scalevolume() fix in pcm-as3525.c from r30209 to the 3.9.1 source. (I didn't add the #include pcm-internal.h.) While it scaled the level up just as expected, the recorded signal was again garbled/burbly. I tried shifting 1 bit instead of 2, and the signal was scaled up less (exactly as it should be), but the burbling remained.

I tried to run the same loop but do nothing, with *samples++ +=0;. Don't know if the compiler was smart enough to optimize that out, but the burbling was clearly not there. Of course level was quite low, as expected. Then I tried *samples +=1; *samples++ -=1;. That would take twice as long, but might fool the optimizer (if the optimizer was skipping the +=0.) Again no burble.

So unfortunately, it looks like there's something in the scalevolume() fix that causes the burbling to occur.

But that's awful simple (and fast) code. Any idea how it could cause trouble?

I hope some of this is helpful...

One more try. I changed scalevolume() to *samples++ +=1;. Adding 1 to every one of a bunch of 14 bit samples shouldn't change much - it's just a very small DC offset. But that was enough to somehow trigger the burble. In other tests that I'm certain were not optimized out (size of the .o increased) I've read and done arithmetic with each sample, but not written back to the sample. The burble was not triggered. I infer from that that it's not just that we're spending too much time in the callback by scalevolume() going through its loop.

I have to suspect that we're changing something besides just samples. Are we certain what's being DMA'd in is purely raw data words with no other structure? Are we certain of the size of the loop? Is there any way the starting pointer dynamically changes so we're playing with the wrong buffer?

Got this patch over here that I am willing to go ahead and commit: http://www.rockbox.org/tracker/task/11878#comment41619

Would love to try this but I have not yet figured out how to apply patches, not stuck, just lazy hehe...
I you feel confident that it works please commit it, can't be any worse than current builds^^

And thanks :)

from the checkout directiory: patch -p0 < <patch_file_name>

After that, do the normal build procedure.

I would but I want to make sure noone's too attached to the DMA recording for some reason. I'm not nor do I think it's that important in the context of what must be accomplished.

Looks fine to me. I think we just used DMA because we were using it for playback.

funman made a special point of converting to it. It was originally just an i2s interrupt implementation but didn't convert anything in the process.

Guess I'll just slam it in there and see what happens.

Went into r31180. I'll await the report on the results.

Testing this current build now, preliminary report is that it looks promising but could use further tweaking. Will post more details within 30 mins.

Big improvement over earlier implementations, the aliasing I reported as  FS#12428  is gone and the waveform extends to 0dB. However the gain setting doesn't seem to work as intended, when turned down from 0dB it simply scales down the entire waveform. Here are two recordings of the same song played on guitar, one with gain set at 0dB the other set with gain at -12dB.
http://www.box.com/s/0011b5r9jgu51uj1q8zf
http://www.box.com/s/cjmociaia1ztebe0ae5r
If you match the levels of the files in an audio editor you'll se that dynamic range is comparable, the file with lower set gain has slightly worse dynamic range due an increased DC offset.

I'll check those out. I thought I'd note first that MIC seems to have some sort of AGC setting turned on.

If it's wrong, why does amplifying the -12db file by 12 db in an editor yield identical amplitudes to the 0dB file?

>However the gain setting doesn't seem to work as intended, when turned down from 0dB it simply scales down the entire waveform.

Could you explain how it doesn't work as intended?

I'm sitting here waiting for an answer since simply scaling the waveform _is_ what should happen. :-)

>If it's wrong, why does amplifying the -12db file by 12 db in an editor yield identical amplitudes to the 0dB file?
My point is that the expected behavior is that of the -12dB wave looking different to the 0dB wave reduced by -12dB in that the -12dB file has the peaks reaching higher levels. That's how my H120 behaves and makes for higher quality recordings.

>However the gain setting doesn't seem to work as intended, when turned down from 0dB it simply scales down the entire waveform.
I realize this was a poorly worded, obviously more is going on that pure scaling. Since I don't have knowledge about how the hardware and software works you may take this with a grain of salt:
The way it is now is as if the AGC "threshold" follows the gain setting rather than remaining at 0dB. Maybe the AGC would be better left disabled to avoid dynamic range issues? Mild clipping can be removed in audio software with good results.

I don't see a problem with the gain. If I compensate for the DC offset, which is the same regardless of the recording gain setting, with a DC blocking filter and amplify the 0dB recording by -12dB, I get peaks on both files within about 1dB, which is as close as one could hope to expect playing an instrument.

I get similar waveforms regardless of whether I amplify the -12dB recording by +12dB or amplify the 0dB recording by -12dB. In either case I kill the DC. You won't get rid of it unless we add a DC blocker to recording where a hardware one isn't available since it's a hardware issue and not an uncommon one at that.

I suppose we could add an offset to compensate DC but I don't know if it's the same for every device of even the same type. That could be calulated by averaging samples a long enough recording and then saving it for later.

I don't mean the 1dB difference, I'm talking about an expected difference of max 12dB. I mean, using the gain label in this context implies a pre-amp adjustment, which is different from waveform scaling. Of course if the preamp gain is fixed in hardware it's a different story, the gain setting might as well be removed and hard coded at 0dB. Tomorrow I could post corresponding clips to the ones I posted above from my H120 to make it more clear what I believe is the expected behavior. Right now it's half past midnight and I'd drive my neighbors crazy.

I have NO idea what you mean. Amplification is the same as waveform scaling. The recording gain (amplification/scaling) is handled by the hardware before the ADC, not the software in rockbox. Amplifying the waveforms shows that the expected difference of +/- 12dB between them exists. Various power measurements (min, max, average RMS) are nearly identical as well.

I still see what I'd expect to see with without serious anomaly. So far, the problem sounds outside the scope of recording on simple hardware.

Personally, I'd stick with the H120 if you want to record anything in any seriousness. :)

>I mean, using the gain label in this context implies a pre-amp adjustment, which is different from waveform scaling.

To be clear, waveform scaling and amplification are one and the same. How were you expecting them to be different?

Here are both, processed to same levels with DC removed.

guitar strumming gain -12dB - +12 dB Gain + DC Blocker.wav:
http://jhmikes.cleansoap.org/guitar%20strumming%20gain%20-12dB%20-%20%2B12%20dB%20Gain%20%2B%20DC%20Blocker.wav

guitar strumming gain 0dB - DC Blocker.wav:
http://jhmikes.cleansoap.org/guitar%20strumming%20gain%200dB%20-%20DC%20Blocker.wav

This waveform overlay represents the difference expected between the two files after the overall volume has been matched, indicated by the colored areas: http://goodrob13.com/wp-content/uploads/2008/11/nirvana-oh-the-guilt.gif
Current behaviour has -both- files looking like the black area after volume matching. I'd argue this is not the intended behaviour.

Looks like you're hard-limiting something if you get it like the black area. The recordings look fine when amplified above using CEP.

The waveform overlay is just a figurative example. I have not processed my files, they're straight from my Sansa. My point still stands. Anyway I was hoping to save you guys some troubleshooting time, maybe I just did the opposite 'cause of my convoluted explanations XD Sorry about that. If my claimed issue is "un-fixable" due to hardware limitation or due to our differing interpretations of the meaning of 'gain' it's not the end of the world, it's already tons better now than it was before.

Though I'd love to see what a AGC user setting could do. ::)

I still don't understand what your point was. What was it that wasn't the way you expected it to be?

I expect the waveform to have expanded peaks when I turn down the gain setting and record a dynamic instrument like acoustic guitar. Now the results are the same no matter where I set the gain, with audible pumping on each strike of the strings.

I don't think I see any clipping in that waveform, so I don't think you should expect the gain control to make any difference in the dynamics.

Indeed there's no clipping, the AGC seems very efficient. Maybe a little too efficient...

From my scaling result in the processed wav's I posted, the response is clearly linear as it should be.

I agree the result should indeed be linear and that is mostly how it works now. I disagree with this only in the condition where my "0dB" wave is matched in volume to the "-12dB" wave, the "-12dB" wave should have signal in the two highest order bits, but it's missing. In the rest of the range the signal should indeed match as you say, MikeS.

I would say any remaining issues are hardware related and since the bitstream is already formed and all losses are before the bitstream is received, meaning there isn't a software solution that can reconstruct the lost information, there's nothing more to do here.

So no gain before ADC available? That's a bummer. So the gain setting works purely digitally?

The recording gain is before ADC, controlled by the software registers.

There should be an as3514 block diagram available online if you have any questions about it.

I looked for it and I found it and the AMS product page. I barely understand the function of half of the blocks though these lines caught my interest:

Microphone Inputs
-3 gain pre-setting (28dB/34dB/40dB) with AGC

Could these be controlled via software?

It's already in use to extend the mic range from -34.5dB..+12.0dB to -34.5dB..+24dB. +12.0dB and below use the 28.0dB setting, the other settings extend it further.

I see, monitoring the output on the recording screen seems to confirm what you are saying. I'll take it for now recording loud sound sources through the mic won't be as clean as on my H120 due to hardware implementation. It's a shame as my Fuze's mic seems to have a truer frequency response than the H120. In the back of my mind remains a fantasy of setting pre-gain higher and analogue gain lower before ADC to achieve higher quality mic recording. I'll keep reading up on the hardware and how to apply patches should there be a way around this that's not obvious.

If you're wondering what the heck I'm talking about the the second part of my previous comment I'm assuming the both gain stages are analogue. The gain stage just before ADC could be turned down to avoid capturing signal simmilar to what's reported in FS#11907, the pre-gain stage, assumed clean, is increased to compensate. How this reflects reality I'm obviously not clear about, only reflects my experience with stand-alone hardware.

>The gain stage just before ADC could be turned down to avoid capturing signal simmilar to what's reported in FS#11907, the pre-gain stage, assumed clean, is increased to compensate.

We already do this.

>How this reflects reality I'm obviously not clear about, only reflects my experience with stand-alone hardware.

Honestly, if you're not going to take the time to understand whats going on, you probably should not be posting here. These long circular suggestions that we implement things we implemented years ago may save you the trouble of understanding whats going on, but they're a waste of everyone else's time.

I just built r31180 for Clip+. Initial testing looks promising - good record level and sounds good at my target of 44.1K, both .wav and 96Kb/s MP3.

Then I built for my other target - ClipV1 (maybe some V2s, but haven't tried that yet). Record level is good - as expected. But the recorded signal is too slow - low pitch and slower than real time. The effect is sample rate dependent: at 48K it was ~4.4% slow; at 44.1K ~3.4% slow; at 32K ~0.7% slow; at 24K correct within measurement uncertainty. (Measurement was from the leading edge of visually identifiable cycles in specific drum beats about 15 sec apart.)

Audibly, it sounded a little slow and low pitched. At 44.1K, twice in a 20 second sample the pitch went up noticeably for a little over a second. While I'm not saying this is what was happening, those short pitch increases could be consistent with generally not keeping up, but occasionally buffer alignment being just right so it could keep up for a little bit.

The Clip sounded OK (and timing was normal) with 3.10. Level, of course, was low.

A possible interpretation is that at least at these sample rates, we can't keep up without the DMA. Do we know whether funman's DMA implementation was driven by anything more urgent than staying consistent with the playback approach?

I did look into what you said regarding "keeping up". I used sine waves as a test because any problems show up more readily to the ear than with natural sounds.

What I found was that 44.1kHz is accurate to about 0.167%. Earlier checks with test_sampr show some other hardware sample rates to be very inaccurate, with quite an audible pitch change.

What I also found was the presence duplicated samples very frequently in the waveform while recording through the clipv1's microphone a sinewave played on my computer. This isn't so much an indication that it can't keep up due to processor speed but with interrupt latency issues. One area that can disable interrupts for an extended time is the I2C driver. Possibly other interrupt callbacks spend an excessive amount of time. With DMA, the interrupts are less frequent so the overflow just happens far less often between the frames but would nonetheless still happen once in awhile.

The first thing I tried (because it was easy) was lowering the FIFO POP watermark (resulting in even MORE interrupts). Nonetheless, this got rid of the majority of the issue but not all of it. I can use FIQ for recording but this latency issue will still affect playback with an occasional 1-sample dropout between DMA frames.

Also, DMA channels should be prioritized to service audio first rather than SD.

Portal Player targets are slower (80MHz max, 30MHz normal), with a I2S input FIFO half the size of what's on AS3525 and do not suffer these problems. This AMS hardware is definitely fast enough to do it without a problem if latency is minded properly. It's fast enough that latency problems with audio should simply not exist.

Its a mistake. The patch tracker occasionally duplicates posts if someone closes or refreshes their webbrowser with the "comment posted" tab still open. So his earlier post got duplicated after yours. Feel free to delete it :)

Hmmm...I get a warning about that. I thought it was weird that it happened *after* I posted.

Well, anyway, it turns out reducing latency is not the major part of the issue. I set audio-related stuff at the highest priority (first to be serviced in DMA for I2SOUT and highest priority in VIC for I2SIN) and most importantly kept track of whether a sample was saved or discarded across interrupts, which seem to have totally solved the issue. Will have a patch shortly.

Ok, this fixes it up my clipv1.

BTW, that patch doesn't adjust anything for AMSv2.

I noticed Fuzev2 has some latency issues, with some dropped sample overflow, perhaps from the button routine which does alot of waiting. I'm thinking seriously that enabling nested IRQ to be a proper thing to do.

I retested (pre-patch) with sine waves, and the anomalies were much easier to hear that way. In particular, simultaneously playing the original tone made playback rate differences very obvious. Thanks for the tip.

I rebuilt with your patch for both ClipV1 and Clip+. Both sounded fine, both with sine waves and music.

Thanks so much for the fix!