<< Chapter < Page Chapter >> Page >
The Source Coding Theorem states that the entropy of an alphabet of symbols specifies to within one bit how many bits on the average need to be used to send the alphabet.

The significance of an alphabet's entropy rests in how we can represent it with a sequence of bits . Bit sequences form the "coin of the realm" in digitalcommunications: they are the universal way of representing symbolic-valued signals. We convert back and forth betweensymbols to bit-sequences with what is known as a codebook : a table that associates symbols to bit sequences. In creating this table, we must be able to assign a unique bit sequence to each symbol so that we can go between symbol and bit sequences without error.

You may be conjuring the notion of hiding information from others when we use the name codebook for thesymbol-to-bit-sequence table. There is no relation to cryptology, which comprises mathematically provable methods ofsecuring information. The codebook terminology was developed during the beginnings of information theory just after WorldWar II.

As we shall explore in some detail elsewhere, digital communication is the transmission of symbolic-valued signals from one place toanother. When faced with the problem, for example, of sending a file across the Internet, we must first represent eachcharacter by a bit sequence. Because we want to send the file quickly, we want to use as few bits as possible. However, wedon't want to use so few bits that the receiver cannot determine what each character was from the bit sequence. Forexample, we could use one bit for every character: File transmission would be fast but useless because the codebookcreates errors. Shannon proved in his monumental work what we call today the Source Coding Theorem . Let B a k denote the number of bits used to represent the symbol a k . The average number of bits B A required to represent the entire alphabet equals k 1 K B a k a k . The Source Coding Theorem states that the average number of bits needed to accurately represent the alphabet need only to satisfy

H A B A H A 1
Thus, the alphabet's entropy specifies to within one bit how many bits on the average need to be used to send the alphabet.The smaller an alphabet's entropy, the fewer bits required for digital transmission of files expressed in that alphabet.

A four-symbol alphabet has the following probabilities. a 0 1 2 a 1 1 4 a 2 1 8 a 3 1 8 and an entropy of 1.75 bits . Let's see if we can find a codebook for this four-letter alphabet that satisfies the Source CodingTheorem. The simplest code to try is known as the simple binary code : convert the symbol's index into a binary number and use the same number of bits for each symbol byincluding leading zeros where necessary.

a 0 00 a 1 01 a 2 10 a 3 11
Whenever the number of symbols in the alphabet is a power oftwo (as in this case), the average number of bits B A equals 2 logbase --> K , which equals 2 in this case. Because the entropy equals 1.75 bits, the simple binary code indeed satisfies the Source Coding Theorem—we arewithin one bit of the entropy limit—but you might wonder if you can do better. If we choose a codebook with differingnumber of bits for the symbols, a smaller average number of bits can indeed be obtained. The idea is to use shorter bitsequences for the symbols that occur more often. One codebook like this is
a 0 0 a 1 10 a 2 110 a 3 111
Now B A 1 · 1 2 2 · 1 4 3 · 1 8 3 · 1 8 1.75 . We can reach the entropy limit! The simple binary code is, in this case, less efficient than theunequal-length code. Using the efficient code, we can transmit the symbolic-valued signal having this alphabet 12.5%faster. Furthermore, we know that no more efficient codebook can be found because of Shannon's Theorem.

Got questions? Get instant answers now!

Questions & Answers

explain and give four Example hyperbolic function
Lukman Reply
The denominator of a certain fraction is 9 more than the numerator. If 6 is added to both terms of the fraction, the value of the fraction becomes 2/3. Find the original fraction. 2. The sum of the least and greatest of 3 consecutive integers is 60. What are the valu
SABAL Reply
1. x + 6 2 -------------- = _ x + 9 + 6 3 x + 6 3 ----------- x -- (cross multiply) x + 15 2 3(x + 6) = 2(x + 15) 3x + 18 = 2x + 30 (-2x from both) x + 18 = 30 (-18 from both) x = 12 Test: 12 + 6 18 2 -------------- = --- = --- 12 + 9 + 6 27 3
Pawel
2. (x) + (x + 2) = 60 2x + 2 = 60 2x = 58 x = 29 29, 30, & 31
Pawel
ok
Ifeanyi
on number 2 question How did you got 2x +2
Ifeanyi
combine like terms. x + x + 2 is same as 2x + 2
Pawel
Mark and Don are planning to sell each of their marble collections at a garage sale. If Don has 1 more than 3 times the number of marbles Mark has, how many does each boy have to sell if the total number of marbles is 113?
mariel Reply
Mark = x,. Don = 3x + 1 x + 3x + 1 = 113 4x = 112, x = 28 Mark = 28, Don = 85, 28 + 85 = 113
Pawel
how do I set up the problem?
Harshika Reply
what is a solution set?
Harshika
find the subring of gaussian integers?
Rofiqul
hello, I am happy to help!
Shirley Reply
please can go further on polynomials quadratic
Abdullahi
hi mam
Mark
I need quadratic equation link to Alpa Beta
Abdullahi Reply
find the value of 2x=32
Felix Reply
divide by 2 on each side of the equal sign to solve for x
corri
X=16
Michael
Want to review on complex number 1.What are complex number 2.How to solve complex number problems.
Beyan
yes i wantt to review
Mark
use the y -intercept and slope to sketch the graph of the equation y=6x
Only Reply
how do we prove the quadratic formular
Seidu Reply
please help me prove quadratic formula
Darius
hello, if you have a question about Algebra 2. I may be able to help. I am an Algebra 2 Teacher
Shirley Reply
thank you help me with how to prove the quadratic equation
Seidu
may God blessed u for that. Please I want u to help me in sets.
Opoku
what is math number
Tric Reply
4
Trista
x-2y+3z=-3 2x-y+z=7 -x+3y-z=6
Sidiki Reply
can you teacch how to solve that🙏
Mark
Solve for the first variable in one of the equations, then substitute the result into the other equation. Point For: (6111,4111,−411)(6111,4111,-411) Equation Form: x=6111,y=4111,z=−411x=6111,y=4111,z=-411
Brenna
(61/11,41/11,−4/11)
Brenna
x=61/11 y=41/11 z=−4/11 x=61/11 y=41/11 z=-4/11
Brenna
Need help solving this problem (2/7)^-2
Simone Reply
x+2y-z=7
Sidiki
what is the coefficient of -4×
Mehri Reply
-1
Shedrak
the operation * is x * y =x + y/ 1+(x × y) show if the operation is commutative if x × y is not equal to -1
Alfred Reply
A soccer field is a rectangle 130 meters wide and 110 meters long. The coach asks players to run from one corner to the other corner diagonally across. What is that distance, to the nearest tenths place.
Kimberly Reply
Jeannette has $5 and $10 bills in her wallet. The number of fives is three more than six times the number of tens. Let t represent the number of tens. Write an expression for the number of fives.
August Reply
What is the expressiin for seven less than four times the number of nickels
Leonardo Reply
How do i figure this problem out.
how do you translate this in Algebraic Expressions
linda Reply
why surface tension is zero at critical temperature
Shanjida
I think if critical temperature denote high temperature then a liquid stats boils that time the water stats to evaporate so some moles of h2o to up and due to high temp the bonding break they have low density so it can be a reason
s.
Need to simplify the expresin. 3/7 (x+y)-1/7 (x-1)=
Crystal Reply
. After 3 months on a diet, Lisa had lost 12% of her original weight. She lost 21 pounds. What was Lisa's original weight?
Chris Reply
Answers please
Nikki Reply

Get Jobilize Job Search Mobile App in your pocket Now!

Get it on Google Play




Source:  OpenStax, Fundamentals of electrical engineering i. OpenStax CNX. Aug 06, 2008 Download for free at http://legacy.cnx.org/content/col10040/1.9
Google Play and the Google Play logo are trademarks of Google Inc.

Notification Switch

Would you like to follow the 'Fundamentals of electrical engineering i' conversation and receive update notifications?

Ask