Machine Learning - MLE

Given the data $D = (x_1, \dots, x_n), x_i \in \mathbb{R}^d$ assume a set of distributions $\{p_\theta : \theta \in \Theta \}$ on $\mathbb{R}^d$.

Assume $D$ is a sample from $X_1, \dots , X_n \sim p_\theta$, indipendent and identically distribuited random variables for the same $\theta \in \Theta$.

Goal

We want to estimate the true value of $\theta$ the data $D$ comes from.

Because in real applications there is not a “true” value for $\theta$, we are searching for value that more probably reprensents our data $D$.

Definition

$\theta_{MLE}$ is a MLE for $\theta$ if

or, more precisely, $p(D \mid \theta_{MLE})= \underset{\theta \in \Theta}{max} \: p(D\mid \theta)$.

$p(D \mid \theta) = p(x_1, \dots, x_n \mid \theta) = \displaystyle\prod_{i=1}^{n} p(x_i \mid \theta) = \displaystyle\prod_{i=1}^{n} P(X_i = x_i \mid \theta)$.

Remark

• The MLE it might not be unique.
• The MLE may fail to exists.

Pros

• Easy to compute
• Intuitive interpretation
• Asymptotic properties
  • Consistent (as $n\rightarrow \infty$ it converges to the “true” value of $\theta$)
  • Normal (central limit theorem)
  • Efficient (is the best possible estimate of $\theta$)
• Invariant under reparametrization ( $g(\theta_{MLE})$ is a MLE for $g(\theta)$)

Cons

• Is a point estimate, so has no rapresentation of uncertainty
• Can overfit
• Existence and uniqueness are not guaranteed.

MLE for Gaussian mean and variance

Choose $\theta = (\mu, \sigma^2)$ that maximizes the probability of observed data.

From here we obtain