-
Home
- Fundamentals of electrical
- Information communication
- Entropy
Shannon showed the power of probabilistic models for symbolic-valued signals. The dey quantity that characterizes such a signal is the entropy of its alphabet.
Communication theory has been formulated best for
symbolic-valued signals. ClaudeShannon published in 1948
The Mathematical Theory
of Communication , which became the cornerstone of digital
communication. He showed the power of
probabilistic
models for symbolic-valued signals, which allowed him to
quantify the information present in a signal. In the simplestsignal model, each symbol can occur at index
with a probability
,
. What this model says is that for each signal value a
-sided coin is flipped (note that
the coin need not be fair). For this model to make sense, theprobabilities must be numbers between zero and one and must sum
to one.
This coin-flipping model assumes that symbols occur without
regard to what preceding or succeeding symbols were, a falseassumption for typed text. Despite this probabilistic
model's over-simplicity, the ideas we develop here alsowork when more accurate, but still probabilistic, models are
used. The key quantity that characterizes a symbolic-valuedsignal is the
entropy of its alphabet.
Because we use the base-2 logarithm, entropy has units of
bits. For this definition to make sense, we must take specialnote of symbols having probability zero of occurring. A
zero-probability symbol never occurs; thus, we define
so that such symbols do not affect the entropy. The
maximum value attainable by an alphabet's entropy occurswhen the symbols are equally likely
(
). In this case, the entropy equals
. The minimum value occurs when only one symbol
occurs; it has probability one of occurring and the rest haveprobability zero.
Derive the maximum-entropy results, both the
numeric aspect (entropy equals
) and the theoretical one (equally likely symbols
maximize entropy). Derive the value of the minimum entropyalphabet.
Equally likely symbols each have a probability of
. Thus,
. To prove that this is the maximum-entropy
probability assignment, we must explicitly take into accountthat probabilities sum to one. Focus on a particular
symbol, say the first.
appears
twice in the entropy
formula: the terms
and
. The derivative with respect to this probability
(and all the others) must be zero. The derivative equals
, and all other derivatives have the same form
(just substitute your letter's index). Thus, eachprobability must equal the others, and we are done. For the
minimum entropy answer, one term is
, and the others are
, which we define to be zero also. The minimum
value of entropy is zero.
Got questions? Get instant answers now!
A four-symbol alphabet has the following probabilities.
Note that these probabilities sum to one as they should. As
,
. The
entropy of this alphabet equals
Got questions? Get instant answers now!
Questions & Answers
it is the relatively stable flow of income
what is circular flow of income
branches of macroeconomics
what is Flexible exchang rate?
is gdp a reliable measurement of wealth
introduction to econometrics
Why is unemployment rate never zero at full employment?
bcoz of existence of frictional unemployment in our economy.
Umashankar
what is flexible exchang rate?
poudel
due to existence of the pple with disabilities
Abdulraufu
the demand of a good rises, causing the demand for another good to fall
is it possible to leave every good at the same level
Joseph
I don't think so. because check it, if the demand for chicken increases, people will no longer consume fish like they used to causing a fall in the demand for fish
Anuolu
is not really possible to let the value of a goods to be same at the same time.....
Salome
Suppose the inflation rate is 6%, does it mean that all the goods you purchase will cost
6% more than previous year? Provide with reasoning.
Not necessarily. To measure the inflation rate economists normally use an averaged price index of a basket of certain goods. So if you purchase goods included in the basket, you will notice that you pay 6% more, otherwise not necessarily.
Waeth
discus major problems of macroeconomics
what is the problem of macroeconomics
Yoal
Economic growth
Stable prices
and low unemployment
Ephraim
explain inflationcause and itis degre
increase in general price levels
WEETO
Good day
How do I calculate this question: C= 100+5yd G= 2000 T= 2000 I(planned)=200.
Suppose the actual output is 3000. What is the level of planned expenditures at this level of output?
how to calculate actual output?
Chisomo
how to calculate the equilibrium income
Beshir
Criteria for determining money supply
who we can define macroeconomics in one line
Muhammad
Aggregate demand
Mohammed
C=k100 +9y and i=k50.calculate the equilibrium level of output
nawalparasi district
from belatari
Amisha
I am Camara from Guinea west Africa... happy to meet you guys here
Sekou
ahile becheclor ho
Amisha
hjr ktm bta ho
ani k kaam grnu hunxa tw
Amisha
money as unit of account means what?
Kalombe
A unit of account is something that can be used to value goods and services and make calculations
Jim
all of you please speak in English I can't understand you're language
Muhammad
I want to know how can we define macroeconomics in one line
Muhammad
it must be .9 or 0.9
no Mpc is greater than 1
Y=100+.9Y+50
Y-.9Y=150
0.1Y/0.1=150/0.1
Y=1500
Kalombe
Mercy is it clear?😋
Kalombe
hi can someone help me on this question
If a negative shocks shifts the IS curve to the left, what type of policy do you suggest so as to stabilize the level of output?
discuss your answer using appropriate graph.
if interest rate is increased this will will reduce the level of income shifting the curve to the left ◀️
Kalombe
Got questions? Join the online conversation and get instant answers!
Source:
OpenStax, Fundamentals of electrical engineering i. OpenStax CNX. Aug 06, 2008 Download for free at http://legacy.cnx.org/content/col10040/1.9
Google Play and the Google Play logo are trademarks of Google Inc.