Maximum likelihood estimation
If you cannot read the equations, or if they look like junk, you may need to install MathPlayer. Look for the icon on this page.
Hyvärinen gives the log-likelihood L in a noise-free Independent Component Analysis (ICA) model as
L=
∑
t=1
T
∑
i=1
n
log
f
i
(
w
i
T
x(t))+Tlog|
detW
|
Where does this come from? The i index the output component weights, i.e., each row of
W
. The t index time. Purcell's Maximum Likelihood Estimation Primer was especially helpful here. The t are analogous to each coin toss in the coin toss example. Essentially, the question asked by maximum likelihood is, given a set of observed data, what is the probability distribution most likely to have produced the results. In this case, the probability density for each component is represented by
f
i
. In this way we can estimate, using data on hand, the probability distribution of each independent component. Now, how does the determinant come into place here?
Forgetting about ICA for the moment, consider a set of uncorrelated random variables
y
i
each with a "known" distribution
f
i
. We then sample the set of random variables
T
times. It's clear that the probability of having a certain result
y
will be given by something like
p(y)=
∏
t=1
T
∏
i=1
n
f
i
(
y
i
(t))
Hmm, that looks almost like what we want, but there is still no way to obtain the determinant. Consider a similar argument for the mixed signals, so that
p(x)=
∏
t=1
T
∏
i=1
n
p
i
(
x
i
(t))
where
p
i
is the probability density for
x
i
. Ahh, here we are. Let's look at the entire set of inputs
p(x)=
∏
t=1
T
p
x
(x(t))
From ICA, I know that
x=Ay=
W
−1
y
To finish this off, we look to our buddy Hyvärinen (who incidentally looks quite young), who says in his tutorial that, in general, for any random vector
x
with density
p
x
and for any matrix
W
, the density of
y=Wx
is given by
p
x
(Wx)|
detW
|
Oh, wait, I'm confused again... I can see how
p(x)=
∏
t=1
T
p
x
(x(t))
=
∏
t=1
T
p
y
(
W
−1
y(t))|
det
W
−1
|
logp(x)=
∑
t=1
T
p
y
(
W
−1
y(t))
−Tlog|
detW
|
though this doesn't match L. I can also see how
p(y)=
∏
t=1
T
p
y
(y(t))
=
∏
t=1
T
p
x
(Wx(t))|
detW
|
logp(y)=
∑
t=1
T
p
x
(Wx(t))
+Tlog|
detW
|
which looks closer, but using the wrong probability density. Could it be somehow that the summation can allow us to use the "known" density? The question is whether the following statement is true:
p
x
(Wx(t))=
∑
i=1
n
f
i
(
w
i
T
x(t))
I don't off-hand see how it can be...
Update:
I figured it out with some help from Ken. It's really quite straight-forward. The hint was not quite right. For
y=Wx
MathType@MTEF@5@5@+=feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLnhiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=xfr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaGaaCyEaiabg2da9iaahEfacaWH4baaaa@39D5@
, the correct relation is
p
y
(y)=
p
x
(x)
|
detW
|
MathType@MTEF@5@5@+=feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLnhiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=xfr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaGaamiCamaaBaaaleaacaWH5baabeaakiaacIcacaWH5bGaaiykaiabg2da9maalaaabaGaamiCamaaBaaaleaacaWH4baabeaakiaacIcacaWH4bGaaiykaaqaamaaemaabaGaciizaiaacwgacaGG0bGaaC4vaaGaay5bSlaawIa7aaaaaaa@46DD@
or
p
x
(x)=
p
y
(y)|
detW
|
MathType@MTEF@5@5@+=feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLnhiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=xfr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaGaamiCamaaBaaaleaacaWH4baabeaakiaacIcacaWH4bGaaiykaiabg2da9iaadchadaWgaaWcbaGaaCyEaaqabaGccaGGOaGaaCyEaiaacMcadaabdaqaaiGacsgacaGGLbGaaiiDaiaahEfaaiaawEa7caGLiWoaaaa@46CD@
making it quite clear that
logp(x)=
∑
t=1
T
p
y
(y(t))|
detW
|
=
∑
t=1
T
p
y
(y(t))
+Tlog|
detW
|=
∑
t=1
T
∑
i=1
n
f
i
(
w
i
T
x(t))
+Tlog|
detW
|
MathType@MTEF@5@5@+=feaafeart1ev1aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLnhiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqtubsr4rNCHbGeaGqiVu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=xfr=xb9adbaqaaeGaciGaaiaabeqaamaabaabaaGcbaGaciiBaiaac+gacaGGNbGaamiCaiaacIcacaWH4bGaaiykaiabg2da9maaqahabaGaamiCamaaBaaaleaacaWH5baabeaakiaacIcacaWH5bGaaiikaiaadshacaGGPaGaaiykamaaemaabaGaciizaiaacwgacaGG0bGaaC4vaaGaay5bSlaawIa7aaWcbaGaamiDaiabg2da9iaaigdaaeaacaWGubaaniabggHiLdGccqGH9aqpdaaeWbqaaiaadchadaWgaaWcbaGaaCyEaaqabaGccaGGOaGaaCyEaiaacIcacaWG0bGaaiykaiaacMcaaSqaaiaadshacqGH9aqpcaaIXaaabaGaamivaaqdcqGHris5aOGaey4kaSIaamivaiGacYgacaGGVbGaai4zamaaemaabaGaciizaiaacwgacaGG0bGaaC4vaaGaay5bSlaawIa7aiabg2da9maaqahabaWaaabCaeaacaWGMbWaaSbaaSqaaiaadMgaaeqaaOGaaiikaiaahEhadaqhaaWcbaGaamyAaaqaaiaadsfaaaGccaWH4bGaaiikaiaadshacaGGPaGaaiykaaWcbaGaamyAaiabg2da9iaaigdaaeaacaWGUbaaniabggHiLdaaleaacaWG0bGaeyypa0JaaGymaaqaaiaadsfaa0GaeyyeIuoakiabgUcaRiaadsfaciGGSbGaai4BaiaacEgadaabdaqaaiGacsgacaGGLbGaaiiDaiaahEfaaiaawEa7caGLiWoaaaa@8B88@
Posted by torque at December 2, 2003 3:46 PM
| TrackBack