Canonical Exponential Family Parameterization
Any multivariate Gaussian distribution $\mathcal{N}(\mu, \Sigma)$ is a member of the exponential family $p(x|\eta) = h(x)\exp(\eta^T T(x) - A(\eta))$, derived via Maximum Entropy under mean and covariance moment constraints using Lagrangian multipliers.
Active Fitted Natural Parameters (η) & Sufficient Statistics
Natural parameters $\eta_1 = \Sigma^{-1}\mu$, $\eta_2 = -\frac{1}{2}\Sigma^{-1}$, sufficient statistics $T(x) = [x, xx^T]^T$, and log-partition function $A(\eta)$.
Mixture Component Markov Transition Matrix
Models a stochastic sequence transitioning between the discrete hidden mixture states $\{z_k\}_{k=1}^K$. Adjust the diagonal self-transition probability to observe convergence toward the stationary distribution $\pi = \pi P$.
Stationary Distribution vs Fitted Prior Weights ($\pi$)
Stationary probability vector $\pi^*$ derived via power iteration on the transition matrix compared to current GMM prior weights $w_k$:
$$\gamma_{ik} = \frac{\pi_k \mathcal{N}(x_i | \mu_k, \Sigma_k)}{\sum_{j=1}^K \pi_j \mathcal{N}(x_i | \mu_j, \Sigma_j)}$$
Maximization Step (M-step): Re-estimates maximum likelihood parameters under soft cluster assignments:
Effective count: $N_k = \sum_{i=1}^N \gamma_{ik}$
Updated Means: $\mu_k = \frac{1}{N_k}\sum_{i=1}^N \gamma_{ik} x_i$
Updated Covariances: $\Sigma_k = \frac{1}{N_k}\sum_{i=1}^N \gamma_{ik}(x_i - \mu_k)(x_i - \mu_k)^T + \epsilon I$ (with regularizing ridge $\epsilon=10^{-4}$)
Updated Priors: $\pi_k = \frac{N_k}{N}$
Log-Likelihood Objective: $\ln p(X|\theta) = \sum_{i=1}^N \ln \left( \sum_{k=1}^K \pi_k \mathcal{N}(x_i | \mu_k, \Sigma_k) \right)$