Exercise 2.5, Koller and Friedman (2009)

Let \(\mathbf{X}, \mathbf{Y}, \mathbf{Z}\) be three disjoint subsets of variables such that \(\mathcal{X}=\mathbf{X}\cup \mathbf{Y}\cup \mathbf{Z}\). Prove that \(P\) satisfies \(\mathbf{X}\perp \mathbf{Y}\mid \mathbf{Z}\) if and only if we can write \(P\) in the form: \[P(\mathcal{X})=\phi_1(\mathbf{X},\mathbf{Z})\phi_2(\mathbf{Y}, \mathbf{Z}).\]

[Proof] 1. Sufficiency.

\[ \begin{aligned} P(\mathbf{X},\mathbf{Y}\mid \mathbf{Z}) & = \frac{P(\mathcal{X})}{P(\mathbf{Z})}=\frac{P(\mathcal{X})}{\int P(\mathbf{x},\mathbf{y},\mathbf{Z})d\mathbf{x}d\mathbf{y}} \leftarrow (\text{by definition of conditional density})\\ & = \frac{P(\mathcal{X})}{\int \phi_1(\mathbf{x},\mathbf{Z})d\mathbf{x}\int \phi_2(\mathbf{y},\mathbf{Z})d\mathbf{y}}\leftarrow (\text{by the factorization assumption})\\ & = \frac{\phi_1(\mathbf{X},\mathbf{Z})}{\int \phi_1(\mathbf{x},\mathbf{Z})d\mathbf{x}}\cdot \frac{\phi_2(\mathbf{Y}, \mathbf{Z})}{\int \phi_2(\mathbf{y},\mathbf{Z})d\mathbf{y}}, \end{aligned} \] which means the conditional density can be factorized, hence \(\mathbf{X} \perp \mathbf{Y}\mid \mathbf{Z}\).

2. Necessity.

\[ \begin{aligned} P(\mathcal{X}) & = P(\mathbf{X}\mid \mathbf{Y},\mathbf{Z})P(\mathbf{Y},\mathbf{Z})\\ & = P(\mathbf{X}\mid \mathbf{Z})P(\mathbf{Y},\mathbf{Z}), \end{aligned} \] where the second equation holds because of the conditional independence. We then define \(\phi_1(\mathbf{X},\mathbf{Z})\) and \(\phi_2(\mathbf{Y},\mathbf{Z})\) to be the first and second terms on the right hand side, respectively.

Exercise 2.9, Koller and Friedman (2009)

Prove or disprove (by providing a counter example) each of the following properties of independence.

a) \(X\perp Y,W\mid Z\implies X\perp Y\mid Z\)

Correct.

b) \((X\perp Y\mid Z) ~and~ (X,Y\perp W\mid Z) \implies X\perp W\mid Z\)

Correct.

c) \((X\perp Y, W\mid Z) ~and~ (Y\perp W\mid Z) \implies X,W\perp Y\mid Z\)

Correct.

d) \((X\perp Y\mid Z) ~and~(X\perp Y\mid W) \implies X\perp Y\mid Z,W\)

Incorrect. Consider a DAG \(X\rightarrow Z \leftarrow U \rightarrow W \leftarrow Y\), where both \(Z\) and \(W\) are colliders with incoming arrows. Given exactly one of the two colliders will leave the path between \(X\) and \(Y\) d-separated. However, given both colliders, \(X\) and \(Y\) will no longer be d-separated.

Exercise 3.13, Koller and Friedman (2009)

Let \(\mathcal{B} = (\mathcal{G},P)\) be a Bayesian network over \(\mathcal{X}\). The Bayesian network is parameterized by a set of CPD parameters of the form \(\theta_{x\mid \mathbf{u}}\) for \(X \in \mathcal{X}\), \(\mathbf{U}=Pa_X^\mathcal{G}\), \(x \in Val(X)\), \(u \in Val(\mathbf{U})\). Consider any conditional independence statement of the form (\(\mathbf{X}\perp \mathbf{Y}\mid \mathbf{Z}\)). Show how this statement translates into a set of polynomial equalities over the set of CPD parameters \(\theta_{x\mid \mathbf{u}}\). (Note: A polynomial equality is an assertion of the form \(aθ_1^2 +bθ_1θ_2 +cθ_2^3 +d = 0\).)

[Proof] We consider discrete distributions in this problem. To specify a saturated Bayesian network, the parameters \(\theta_{x\mid \mathbf{u}}\) parameterizes child-parent conditional distributions \(P(X\mid \mathbf{U})\) for all possible values of \(X=x\) and \(\mathbf{U}=\mathbf{u}\), where \(\mathbf{U}=Pa_X^{\mathcal{G}}\). Further conditional independence constraints on the joint distribution amount to constraints on the \(\theta\) parameters, which we need to prove are in polynomial equality forms. Because \(P(\mathbf{X},\mathbf{Y},\mathbf{Z})=\sum_{V\notin \mathbf{X}\cup\mathbf{Y}\cup\mathbf{Z}}\prod_{V\in\mathcal{X}}\theta_{V\mid Pa_{V}^{\mathcal{G}}}\), we can obtain distributions not necessarily for child and parents, for example, \(P(\mathbf{Z})=\sum_{V\notin \mathbf{Z}}\prod_{V\in\mathcal{X}}\theta_{V \mid Pa_{V}^{\mathcal{G}}}\) and \(P(\mathbf{X},\mathbf{Z})=\sum_{V\notin \mathbf{X}\cup\mathbf{Z}}\prod_{V\in\mathcal{X}}\theta_{V\mid Pa_{V}^{\mathcal{G}}}\); similar for other joint distributions over a subset of nodes in \(\mathcal{B}\). We have \(P(\mathbf{X},\mathbf{Y}\mid \mathbf{Z}=\mathbf{z}) = P(\mathbf{X}\mid \mathbf{Z}=\mathbf{z}) P(\mathbf{Y}\mid \mathbf{Z}=\mathbf{z})\) being equivalent to

\[ \begin{aligned} \sum_{V\notin \mathbf{X}\cup\mathbf{Y}\cup\mathbf{Z}}\prod_{V\in\mathcal{X}}\theta_{V\mid Pa_{V}^{\mathcal{G}}} \cdot \sum_{V\notin \mathbf{Z}}\prod_{V\in\mathcal{X}}\theta_{V \mid Pa_{V}^{\mathcal{G}}} &= \sum_{V\notin \mathbf{X}\cup\mathbf{Z}}\prod_{V\in\mathcal{X}}\theta_{V\mid Pa_{V}^{\mathcal{G}}} \cdot \sum_{V\notin \mathbf{Y}\cup\mathbf{Z}}\prod_{V\in\mathcal{X}}\theta_{V\mid Pa_{V}^{\mathcal{G}}}, \end{aligned} \] a polynomial equality for \(\mathbf{z}\in \{\mathbf{z}: \sum_{V\notin \mathbf{Z}}\prod_{V\in\mathcal{X}}\theta_{V \mid Pa_{V}^{\mathcal{G}}}\neq 0\}\); If \(P(\mathbf{Z}=\mathbf{z})=0\), we have \(\sum_{V\notin \mathbf{Z}}\prod_{V\in\mathcal{X}}\theta_{V \mid Pa_{V}^{\mathcal{G}}}\mid_{\mathbf{Z}=\mathbf{z}}= 0\).