Chapter 2 Exercises

Solution to exercise 2.1

Prop n.1.

$$\sum _{x=0}^{1}p(x\mid \mu )=\mu +(1-\mu )=1$$

Prop n.2

$$\mathbb{E}[x]=\sum _{x=0}^{1}xp(x\mid \mu )=\sum _{x=0}^{1}x\cdot {\mu }^x(1-\mu {)}^{1-x}=1\cdot {\mu }^1(1-\mu {)}^{1-1}=\mu$$

Prop n.3

$$\begin{array}{rl}var[x]& =\sum _{x=0}^{1}p(x)(x-\mu {)}^2\\ & =p(0){\mu }^2+p(1)(1-\mu {)}^2\\ & =(1-\mu ){\mu }^2+\mu (1-\mu {)}^2\\ & =\mu ((1-\mu )\mu +(1-\mu {)}^2)\\ & =\mu (\mu -{\mu }^2+1-2\mu +{\mu }^2)\\ & =\mu (1-\mu )\end{array}$$

Solution to exercise 2.30

We know that

$$\mathbb{E}[z]=R^{-1}\left[\begin{array}{c}\Lambda \mu -A^{T}Lb\\ Lb\end{array}\right]$$

and that

$$cov[z]=R^{-1}=\left[\begin{array}{cc}{\Lambda }^{-1}& {\Lambda }^{-1}{A}^T\\ A{\Lambda }^{-1}& L^{-1}+A{\Lambda }^{-1}{A}^T\end{array}\right]$$

So by replacing $R^{-1}$ we get

$$\mathbb{E}[z]=\left[\begin{array}{cc}{\Lambda }^{-1}& {\Lambda }^{-1}{A}^T\\ A{\Lambda }^{-1}& L^{-1}+A{\Lambda }^{-1}{A}^T\end{array}\right]\left[\begin{array}{c}\Lambda \mu -A^{T}Lb\\ Lb\end{array}\right]$$

which results in

$$\left[\begin{array}{c}\mu -{\Lambda }^{-1}{A}^{T}Lb+{\Lambda }^{-1}{A}^{T}Lb\\ A\mu -A{\Lambda }^{-1}{A}^{T}Lb+b+A{\Lambda }^{-1}{A}^{T}Lb\end{array}\right]=\left[\begin{array}{c}\mu \\ A\mu +b\end{array}\right]$$

Solution to exercise 2.31

There are various approaches to compute the marginal distribution $p(y)$ where $y=x+z$ and $x\sim \mathcal{N}(x\mid {\mu }_x,{\Sigma }_x)$, $z\sim \mathcal{N}(z\mid {\mu }_z,{\Sigma }_z)$.

The first approach came to my mind after a video from 3Blue1Brown, that demonstrates exactly that in this case $p(y)=p(x)\ast p(z)$, where $\ast$ is the convolution operator.

$$p(y)=\int p_x(y-t)p_z(t)dt$$

In this way, we consider every way to obtain $y$ from the sum $x+z$. This solution has been adopted by Tommy Odland in his solutions.

But there is a simpler way to do this. Let's consider the conditional distribution $p(y\mid x)$. Since $x$ is fixed, and $y=x+z$, the only variability is up to $z$. We can define this as

$$p(y\mid x)=\mathcal{N}(y\mid {\mu }_z+x,{\Sigma }_z)$$

We can now compare the obtained results with expressions 2.99 and 2.100:

$$\mu ={\mu }_x{\Lambda }^{-1}={\Sigma }_{x}A=Ib={\mu }_z{L}^{-1}={\Sigma }_{z}x=x$$

And using results from 2.109 and 2.110 we know:

$$\mathbb{E}[y]=A\mu +b=I{\mu }_x+{\mu }_z={\mu }_x+{\mu }_z$$

and

$$cov[y]=L^{-1}+A{\Lambda }^{-1}{A}^T={\Sigma }_z+{\Sigma }_x$$