Entropy and Variable Length Codeword assignment :

Uniform length codeword assignment is not in general optimal in terms of the required average bit rate. Suppose some message probabilities are more likely to be sent than others. Then by assigning shorter codewords to the more probable message possibilities and longer codewords to the less probable message possibilities, we may be able to reduce the average bit rate.

Codewords whose lengths are different for different message possibilities are called variable-length codewords. When the codeword is designed based on the statistical occurrence of different message probabilities, the design method is called statistical coding. To discuss the problem of designing codewords such that the average bit rate is minimized, we define an entropy H as:

	(4.8.1)

where is the probability that the message will be since it can be shown that

	(4.8.2)

The entropy H can be interpreted as the average amount of information that a message contains. Suppose . If and H is zero and is the minimum possible for . In this case the message is with probability of 1; i.e. the message contains no new information. At the other extreme, suppose . The entropy H is 1 and is the maximum possible for . In this case the two message possibilities and are equally likely. Receiving the message clearly adds new information.

Note - From information theory, the entropy H in equations (4.8.1) is theoretically the minimum possible average bit rate required in coding a message. This result is very useful. Suppose the average bit rate using the codewords we have designed is same as the entropy codewords are optimal i.e. we do not have to search any further.

Example : Suppose L is expressed as a power of 2 and each message is equally probable

So that for

Then form eqn (4.8.2) H is . Since uniform length codeword assignment results in an average bit rate of bits/message, we can conclude that it is an optimal method to design codewords in this case.

Note - The entropy also provides a standard against which the performance of a codeword design method can be measured. If average bit rate achieved by a codeword design is close to entropy the method is efficient. If we code each message separately, it is not in general possible to design codewords that result in an average bit rate given by the entropy.

Example:

Suppose .

Even, though it is not possible to design codewords that result is an average bit rate of bit/message.