Module 2 : Classical Unbiased Estimation and Bounds

Lecture 2 : Bound and Fisher Information

2.3.2 MVU Estimator and CRLB Attainment

In general, an MVU estimator may exist but may not attain the CRLB. To illustrate this, let us assume that there exist three unbiased estimators for estimating the unknown parameter θ in an estimation problem and their variances are shown in Figure 2.3 . As shown in Figure 2.3 (a), the estimator ˆ θ3 is the efficient as it attains the CRLB and therefore it is also MVUE. On the other hand, in Figure 2.3 (b), the estimator ˆ θ3 does not attain the CRLB so it is not an efficient. But its variance is uniformally less that other possible unbiased estimators so it is the MVUE.

Figure 2.3: Possible dependence of estimator variance with parameter θ

2.3.3 Fisher Information

As noted above, when CRLB is attained, the variance of the unbiased estimator is reciprocal of the Fisher information. The Fisher information is a way of measuring the amount of information that an observable random variable x carries about an unknown parameters θ upon which the probability of x depends. Assume that the data PDF p(x; θ) satisfies some regulairty conditions which include:

Note that the above regularity conditions are satisfied in general except when the domain of the PDF for which it is nonzero dependes on the unknown parameter (e.g., uniform distribution U(0,θ) with unknown domain parameter).

Given the PDF p(x; θ), the Fisher information I(θ) can also be expressed as

which follows directly from the “regularity” condition, E[ ] ∂lnp(x;θ) ∂θ = 0 ∀ θ , imposed on the PDF.

Proof

Proof

In the following, the Fisher information relationships are shown for the scalar parameter case p(x; θ) for the sake of simplicity,

or

The Fisher information has the essential properties of an information measure and that is obvious by noting the facts:

  1. It is non-negative, due to I(θ) = [( ) ] ∂lnp(x;θ)-2 ∂θ
  2. It is additive for independent observations

The later property leads to the result that the CRLB of N IID observations is 1∕N times that of for one observation. To verify this, note that for independent observations

This results in

In case of completely dependent identically distributed observations, the Fisher information of N observations remains same as that of for one observation and the CRLB will not decrease with increasing data record length.

In other words, if we synthetically try to increase the data length by simply repeating some of the actual observations rather than making new observation it would not result in lowering of the CRLB or better estimation performance than obtained by using the actual observations.

2.3.4 Example

2.3.5 Consistency of Estimator

Another desirable property of estimators is consistency. If we collect a large number of observations, we hope that we have a lot of information about any unknown parameter θ, and thus we can construct an estimator with a very small mean square error (MSE). An estimator is defined as consistent if

which means that as the number of observations increase the MSE of the estimator descends to zero, i.e., ˆ θ = θ.

For an example, if , then the MSE of ˆ θis 1∕n. Since limn (1∕n) = 0, x is a consistent estimator of θ or more specifically “MSE-consistant”. There are other type of consistancy definitions that look at the probability of the errors. They work better when the estimator do not have a variance.