<< Chapter < Page Chapter >> Page >

Let us assume that we represent a symbol x n , with probability p n , by l n bits. Then, the average number of bits spent per symbol will be

L n 1 N p n l n
We see that this equation is equal to the entropy if the code words are selected to have the lengths l n p n . Thus, if the source produces stochastically independent outcomes with probabilities p n , such that p n is an integer, then we can easily find an optimal code as we show in the next example.

Finding a minimal representation

A four-symbol alphabet produces stochastically independent outcomes with the following probabilities. x 1 1 2 x 2 1 4 x 3 1 8 x 4 1 8 and an entropy of 1.75 bits/symbol. Let's see if we can find a codebook for this four-letter alphabet that satisfies the Source CodingTheorem. The simplest code to try is known as the simple binary code : convert the symbol's index into a binary number and use the same number of bits for each symbol byincluding leading zeros where necessary.

x 1 00 x 2 01 x 3 10 x 4 11
As all symbols are represented by 2 bits, obviously the average number of bits per symbol is 2.Because the entropy equals 1.75 bits, the simple binary code is not a minimal representation according to the source coding theorem.If we chose a codebook with differing number of bits for the symbols, a smaller average number of bits can indeed be obtained. The idea is to use shorter bit sequences for the symbols that occur more often , i.e., symbols that have a higher probability. One codebook like this is
x 1 0 x 2 10 x 3 110 x 4 111
Now L 1 1 2 2 1 4 3 1 8 3 1 8 1.75 . We can reach the entropy limit! This should come as no surprise, as promised above, when p n is an integer for all n , the optimal code is easily found.

The simple binary code is, in this case, less efficient than theunequal-length code. Using the efficient code, we can transmit the symbolic-valued signal having this alphabet 12.5%faster. Furthermore, we know that no more efficient codebook can be found because of Shannon's source coding theorem.

Got questions? Get instant answers now!

Optimality of the ascii code

Let us return to the ASCII codes presented in . Is the 7-bit ASCII code optimal, i.e., is it a minimal representation? The 7-bit ASCII code assign an equal length (7-bit) to all characters it represents. Thus, it would be optimal if all of the 128 characters wereequiprobable, that is each character should have a probability of 1 128 . To find out whether the characters really are equiprobable an analysis of all English texts would be needed. Such an analysis is difficult to do. However, the letter "E" is more probable than the letter "Z", so the equiprobable assumption does not hold, and the ASCII codeis not optimal.

(A technical note: We should take into account that in English text subsequent outcomes are not stochastically independent. To see this, assume the first letter to be "b", then it is more probable that the next letter is "e", than "z". In the case where the outcomesare not stochastically independent, the formulation we have given of Shannon's source coding theorem is no longer valid, to fix this, we should replace the entropy with the entropy rate, but we will not pursue this here).

Got questions? Get instant answers now!

Generating efficient codes

From Shannon's source coding theorem we know what the minimum average rate needed to represent a source is. But other than in the case when the logarithm of the probabilities gives an integer, we do not get any indications on how to obtain that rate. It is a large area of research to getclose to the Shannon entropy bound. One clever way to do encoding is the Huffman coding scheme.

Questions & Answers

Is there any normative that regulates the use of silver nanoparticles?
Damian Reply
what king of growth are you checking .?
What fields keep nano created devices from performing or assimulating ? Magnetic fields ? Are do they assimilate ?
Stoney Reply
why we need to study biomolecules, molecular biology in nanotechnology?
Adin Reply
yes I'm doing my masters in nanotechnology, we are being studying all these domains as well..
what school?
biomolecules are e building blocks of every organics and inorganic materials.
anyone know any internet site where one can find nanotechnology papers?
Damian Reply
sciencedirect big data base
Introduction about quantum dots in nanotechnology
Praveena Reply
what does nano mean?
Anassong Reply
nano basically means 10^(-9). nanometer is a unit to measure length.
do you think it's worthwhile in the long term to study the effects and possibilities of nanotechnology on viral treatment?
Damian Reply
absolutely yes
how to know photocatalytic properties of tio2 nanoparticles...what to do now
Akash Reply
it is a goid question and i want to know the answer as well
characteristics of micro business
for teaching engĺish at school how nano technology help us
Do somebody tell me a best nano engineering book for beginners?
s. Reply
there is no specific books for beginners but there is book called principle of nanotechnology
what is fullerene does it is used to make bukky balls
Devang Reply
are you nano engineer ?
fullerene is a bucky ball aka Carbon 60 molecule. It was name by the architect Fuller. He design the geodesic dome. it resembles a soccer ball.
what is the actual application of fullerenes nowadays?
That is a great question Damian. best way to answer that question is to Google it. there are hundreds of applications for buck minister fullerenes, from medical to aerospace. you can also find plenty of research papers that will give you great detail on the potential applications of fullerenes.
what is the Synthesis, properties,and applications of carbon nano chemistry
Abhijith Reply
Mostly, they use nano carbon for electronics and for materials to be strengthened.
is Bucky paper clear?
carbon nanotubes has various application in fuel cells membrane, current research on cancer drug,and in electronics MEMS and NEMS etc
so some one know about replacing silicon atom with phosphorous in semiconductors device?
s. Reply
Yeah, it is a pain to say the least. You basically have to heat the substarte up to around 1000 degrees celcius then pass phosphene gas over top of it, which is explosive and toxic by the way, under very low pressure.
Do you know which machine is used to that process?
how to fabricate graphene ink ?
for screen printed electrodes ?
What is lattice structure?
s. Reply
of graphene you mean?
or in general
in general
Graphene has a hexagonal structure
On having this app for quite a bit time, Haven't realised there's a chat room in it.
what is biological synthesis of nanoparticles
Sanket Reply
Got questions? Join the online conversation and get instant answers!
Jobilize.com Reply

Get the best Algebra and trigonometry course in your pocket!

Source:  OpenStax, Information and signal theory. OpenStax CNX. Aug 03, 2006 Download for free at http://legacy.cnx.org/content/col10211/1.19
Google Play and the Google Play logo are trademarks of Google Inc.

Notification Switch

Would you like to follow the 'Information and signal theory' conversation and receive update notifications?