That the LMS derivation and equation is only optimal for Guassian driving signals. If the driving noise through an unknown system has some other form of distribution (say Laplace as is the case with speech), then the best estimator of that FIR system is the simpler sign() LMS. I found this quite a pleasant surprise. Hardy

# Did you know - LMS

Started by ●July 4, 2012

Reply by ●July 4, 20122012-07-04

On 7/4/12 2:15 AM, HardySpicer wrote:> That the LMS derivation and equation is only optimal for Guassian > driving signals. If the driving noise through an unknown system has > some other form of distribution (say Laplace as is the case with > speech), then the best estimator of that FIR system is the simpler > sign() LMS. I found this quite a pleasant surprise. >what's the "sign() LMS"? i never really had the adaptive LMS in class, but the derivation, as i have seen it, doesn't make an assumption regarding the p.d.f. of the driving signals. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."

Reply by ●July 4, 20122012-07-04

On Jul 5, 2:04�am, robert bristow-johnson <r...@audioimagination.com> wrote:> On 7/4/12 2:15 AM, HardySpicer wrote: > > > That the LMS derivation and equation is only optimal for Guassian > > driving signals. If the driving noise through an unknown system has > > some other form of distribution (say Laplace as is the case with > > speech), then the best estimator of that FIR system is the simpler > > sign() LMS. I found this quite a pleasant surprise. > > what's the "sign() LMS"? > > i never really had the adaptive LMS in class, but the derivation, as i > have seen it, doesn't make an assumption regarding the p.d.f. of the > driving signals. > > -- > > r b-j � � � � � � � � �r...@audioimagination.com > > "Imagination is more important than knowledge."Same as ordinary LMS except you use sign(e(t)) instead of e(t) (essentially). Well, maximum likelihood does use the PDF of the noise and for Guassian stats you get the same as ordinary least-squares. However, with a different PDF the formula changes of course. Hardy

Reply by ●July 5, 20122012-07-05

On 7/4/12 2:49 PM, HardySpicer wrote:> On Jul 5, 2:04 am, robert bristow-johnson<r...@audioimagination.com> > wrote: >> On 7/4/12 2:15 AM, HardySpicer wrote: >> >>> That the LMS derivation and equation is only optimal for Guassian >>> driving signals. If the driving noise through an unknown system has >>> some other form of distribution (say Laplace as is the case with >>> speech), then the best estimator of that FIR system is the simpler >>> sign() LMS. I found this quite a pleasant surprise. >> >> what's the "sign() LMS"? >> >> i never really had the adaptive LMS in class, but the derivation, as i >> have seen it, doesn't make an assumption regarding the p.d.f. of the >> driving signals. >> > > Same as ordinary LMS except you use sign(e(t)) instead of e(t) > (essentially).so how does that help?> Well, maximum likelihood does use the PDF of the noise and for > Guassian stats you get the same > as ordinary least-squares. However, with a different PDF the formula > changes of course.i don't get it. you can derive the basic LMS filter by minimizing an error metric (involving the mean-square) with no assumption at all about p.d.f. there *are* assumptions about correlation and whiteness,i think. but none, as far as i understand the simple derivation, for p.d.f. -- r b-j rbj@audioimagination.com "Imagination is more important than knowledge."

Reply by ●July 5, 20122012-07-05

On Wednesday, July 4, 2012 1:15:04 AM UTC-5, HardySpicer wrote:> That the LMS derivation and equation is only optimal for Guassian > driving signals. If the driving noise through an unknown system has > some other form of distribution (say Laplace as is the case with > speech), then the best estimator of that FIR system is the simpler > sign() LMS. I found this quite a pleasant surprise. > > HardyOn what do you base this? It is my understanding that the only constraint on the LMS algorithm is that the input vector be independent of the weight vector (this in itself has hugh implications). As for as optimality is concerned, the LMS only provides optimality in the sense that the filter will estimate the Wiener optimal solution if the input is stationary (in addition to the independence requirement). LMS requires neither gaussian nor white inputs.

Reply by ●July 5, 20122012-07-05

>LMS requires neither gaussian nor white inputs. >Of course you can apply LMS to non-gaussian inputs. I believe this is done in many applications. At least I can mention a few where I know for sure that LMS is used for inputs that are non-gaussian. However, maybe one can take the viewpoint that implicitly the noise is assumed gaussian because only second order information is used. If you know that the noise is not gaussian then there should be a possibility of deriving a better performing 'LMS' filter by exploiting the information that is conveyed in the higher order moments. It will likely introduce some non-linear operation into the update. You might want to do a search on alpha-stable distributions combined with LMS filter or adaptive filtering. just my €0.02 Cheers

Reply by ●July 5, 20122012-07-05

I almost might ought to label this "OT" only because it's so far removed from earlier comments. But here goes: My simple-minded understanding of an LMS filter, just picking one variation, goes like this: Let's assume an automatic line enhancer ALE. The "model", if you will is that there are some sinusoids "S" mixed with noise "N". I don't know if the type of noise matters at all. I suspect not because: Here is the block diagram in sort of "analog" terms: Input goes into Long delay line which goes to the Summing Junction and goes into Adaptive Filter which also goes to the Summing Junction The adaptation objective is to minimize the output of the summing junction. And, the implementation uses the output of the Adaptive Filter as the system output (not the Summing Junction). The job of the delay line is to decorrelate the noises arriving at the summing junction. So, for that to happen I think all one need to say about the noise is that it's random. Take this case: Apply *only* noise to the input. The Adaptive filter "turns off" at all frequencies because random noise1 plus uncorrelated random noise2 is always larger than noise1. How could the nature of the noise matter then? OK. Now put the sinusoids back in. The Adaptive filter still "turns of" at all frequencies *except* where the sinusoids lie (where their amplitude and phase are adjusted) - in order to cancel them out at the summing junction. Anything else would not minimize the summing junction output. That particular part of the adapted filter should not be much affected by the nature of the noise - the pdf or whiteness or whatever. If the noise is narrowband noise centered where the sinusoids lie then the "noise" starts looking sinusoidal and we have a good chore for a thesis topic. Otherwise I don't see how it will matter much ... if it does. Fred

Reply by ●July 5, 20122012-07-05

On Jul 6, 2:44�am, maury <maury...@core.com> wrote:> On Wednesday, July 4, 2012 1:15:04 AM UTC-5, HardySpicer wrote: > > That the LMS derivation and equation is only optimal for Guassian > > driving signals. If the driving noise through an unknown system has > > some other form of distribution (say Laplace as is the case with > > speech), then the best estimator of that FIR system is the simpler > > sign() LMS. I found this quite a pleasant surprise. > > > Hardy > > On what do you base this? It is my understanding that the only constraint on the LMS algorithm is that the input vector be independent of the weight vector (this in itself has hugh implications). As for as optimality is concerned, the LMS only provides optimality in the sense that the filter will estimate the Wiener optimal solution if the input is stationary (in addition to the independence requirement). > > LMS requires neither gaussian nor white inputs.It's not a major difference that I can see, but it is an important observation. LMS can be derived from maximum likelihood assuming a Guassian distribution. However, if you assume a different distribution you get a different algorithm tailored to that distribution. It should give better results. From my simulations it's not that striking though. See this paper OPTIMUM ERROR NONLINEARITIES FOR LMS ADAPTATION S. C. Douglas and T. H.-Y. Meng Hardy

Reply by ●July 5, 20122012-07-05

On Jul 6, 11:53�am, HardySpicer <gyansor...@gmail.com> wrote:> On Jul 6, 2:44�am, maury <maury...@core.com> wrote: > > > On Wednesday, July 4, 2012 1:15:04 AM UTC-5, HardySpicer wrote: > > > That the LMS derivation and equation is only optimal for Guassian > > > driving signals. If the driving noise through an unknown system has > > > some other form of distribution (say Laplace as is the case with > > > speech), then the best estimator of that FIR system is the simpler > > > sign() LMS. I found this quite a pleasant surprise. > > > > Hardy > > > On what do you base this? It is my understanding that the only constraint on the LMS algorithm is that the input vector be independent of the weight vector (this in itself has hugh implications). As for as optimality is concerned, the LMS only provides optimality in the sense that the filter will estimate the Wiener optimal solution if the input is stationary (in addition to the independence requirement). > > > LMS requires neither gaussian nor white inputs. > > It's not a major difference that I can see, but it is an important > observation. > LMS can be derived from maximum likelihood assuming a Guassian > distribution. However, if you assume a different distribution > you get a different algorithm tailored to that distribution. It should > give better results. From my simulations it's not that striking > though. > See this paper > > OPTIMUM ERROR NONLINEARITIES FOR LMS ADAPTATION > S. C. Douglas and T. H.-Y. Meng > > Hardyand from that paper it says for Laplacian plant noise, we may achieve a 3dB reduction in misadjustment using a sign error nonlinearity for a given convergence rate as compared to standard LMS adaptation

Reply by ●July 8, 20122012-07-08

On Thursday, July 5, 2012 8:03:04 PM UTC-5, HardySpicer wrote:> On Jul 6, 11:53�am, HardySpicer <gyansor...@gmail.com> wrote: > > On Jul 6, 2:44�am, maury <maury...@core.com> wrote: > > > > > On Wednesday, July 4, 2012 1:15:04 AM UTC-5, HardySpicer wrote: > > > > That the LMS derivation and equation is only optimal for Guassian > > > > driving signals. If the driving noise through an unknown system has > > > > some other form of distribution (say Laplace as is the case with > > > > speech), then the best estimator of that FIR system is the simpler > > > > sign() LMS. I found this quite a pleasant surprise. > > > > > > Hardy > > > > > On what do you base this? It is my understanding that the only constraint on the LMS algorithm is that the input vector be independent of the weight vector (this in itself has hugh implications). As for as optimality is concerned, the LMS only provides optimality in the sense that the filter will estimate the Wiener optimal solution if the input is stationary (in addition to the independence requirement). > > > > > LMS requires neither gaussian nor white inputs. > > > > It's not a major difference that I can see, but it is an important > > observation. > > LMS can be derived from maximum likelihood assuming a Guassian > > distribution. However, if you assume a different distribution > > you get a different algorithm tailored to that distribution. It should > > give better results. From my simulations it's not that striking > > though. > > See this paper > > > > OPTIMUM ERROR NONLINEARITIES FOR LMS ADAPTATION > > S. C. Douglas and T. H.-Y. Meng > > > > Hardy > > and from that paper it says > > for Laplacian plant noise, we may achieve a > 3dB reduction in misadjustment using a sign error nonlinearity for a > given convergence rate as compared to standard LMS adaptationI always have a problem when people use coefficient misadjustment as a measure of the performance of the LMS algorithm. The LMS algorithm is NOT derived by determining the minimum misadjustment, nor does it try to minimize coefficient misadjustment. The ONLY parameter the derivation minimizes is the mean-square-error, nothing else.