Maximum likelihood estimation

As an exercise, try the following problem:

• Newton-Raphson iteration
• Iteration by the Scoring Method
• Expectation-Maximization Algorithm

Asymptotic properties of the mle

Let $x=\left(\begin{array}{c}{x}_1\\ \\ x_N\end{array}\right)$ denote an IID sample of size $N$ , and each sample is distributed according to $p(, x)$ . Let $({}_N)$ denote the MLE based on a sample $x$ .

Asymptotic properties of mle

If the likelihood $(x, )=p(, x)$ satisfies certain "regularity" conditions

Since the mean of the MLE tends to the true parameter value, we say the MLE is asymptotically unbiased . Since the covariance tends to the inverse Fisher information matrix, we saythe MLE is asymptotically efficient .

In general, the rate at which the mean-squared error converges to zero is not known. It is possible that for small samplesizes, some other estimator may have a smaller MSE.The proof of consistency is an application of the weak law of largenumbers. Derivation of the asymptotic distribution relies on the central limit theorem. The theorem is also true in moregeneral settings (e.g., dependent samples). See, Kay, Vol. I, Ch. 7 for further discussion.

The mle and efficiency

In some cases, the MLE is efficient, not just asymptotically efficient. In fact, when an efficient estimator exists, itmust be the MLE, as described by the following result:

If $()$ is an efficient estimator, and the Fisher information matrix $I()$ is positive definite for all $$ , then $()$ maximizes the likelihood.

Recall the $()$ is efficient (meaning it is unbiased and achieves the Cramer-Rao lower bound) if and only if $$\frac{\partial^1\ln p(, x)}{\partial }=I()(()-)$$ for all $$ and $x$ . Since $()$ is assumed to be efficient, this equation holds, and in particular it holds when $=((x))$ . But then the derivative of the log-likelihood is zero at $=((x))$ . Thus, $()$ is a critical point of the likelihood. Since the Fisher information matrix, which is the negative ofthe matrix of second order derivatives of the log-likelihood, is positive definite, $()$ must be a maximum of the likelihood.

An important case where this happens is described in the following subsection.

Optimality of mle for linear statistical model

If the observed data $x$ are described by $$x=H+w$$ where $H$ is $N\times p$ with full rank, $$ is $p\times 1$ , and $(w, (0, C))$ , then the MLE of $$ is $$()=H^TC^{(-1)}H^{(-1)}H^TC^{(-1)}x$$ This can be established in two ways. The first is to compute the CRLB for $$ . It turns out that the condition for equality in the bound is satisfied, and $()$ can be read off from that condition.

The second way is to maximize the likelihood directly. Equivalently, we must minimize $$(x-H)^TC^{(-1)}(x-H)$$ with respect to $$ . Since $C^{(-1)}$ is positive definite, we can write $C^{(-1)}=U^TU=D^TD$ , where $D=^{\left(\frac{1}{2}\right)}U$ , where $U$ is an orthogonal matrix whose columns are eigenvectors of $C^{(-1)}$ , and $$ is a diagonal matrix with positive diagonal entries. Thus, we must minimize $$(Dx-DH)^T(Dx-DH)$$ But this is a linear least squares problem, so the solution is given by the pseudoinverse of $DH$ :

Invariance of mle

Suppose we wish to estimate the function $w=W()$ and not $$ itself. To use the maximum likelihood approach for estimating $w$ , we need an expression for the likelihood $(x, w)=p(w, x)$ . In other words, we would need to be able to parameterize thedistribution of the data by $w$ . If $W$ is not a one-to-one function, however, this may not be possible. Therefore, we define the induced likelihood $$(x, w)=\max\{ , W()=w\}(x, )$$ The MLE $(w)$ is defined to be the value of $w$ that maximizes the induced likelihood. With this definition, the following invarianceprinciple is immediate.

Let $()$ denote the MLE of $$ . Then $(w)=W(())$ is the MLE of $w=W()$ .

The proof follows directly from the definitions of $()$ and $(w)$ . As an exercise, work through the logical steps of the proof on your own.

Be aware that the MLE of a transformed parameter does not necessarily satisfy the asymptotic properties discussed earlier.

Summary of mle

The likelihood principle states that information broughtby an observation $x$ about $$ is entirely contained in the likelihood function $p(, x)$ . The maximum likelihood estimator is one effective implementation of the likelihood principle. In some cases, the MLE can be computedexactly, using calculus and linear algebra, but at other times iterative numerical algorithms are needed. The MLE has severaldesireable properties:

• It is consistent and asymptotically efficient (as $N\to$∞ we are doing as well as MVUE).
• When an efficient estimator exists, it is the MLE.
• The MLE is invariant to reparameterization.