-
Home
- Fundamentals of electrical
- Information communication
- Entropy
Shannon showed the power of probabilistic models for symbolic-valued signals. The dey quantity that characterizes such a signal is the entropy of its alphabet.
Communication theory has been formulated best for
symbolic-valued signals. ClaudeShannon published in 1948
The Mathematical Theory
of Communication , which became the cornerstone of digital
communication. He showed the power of
probabilistic
models for symbolic-valued signals, which allowed him to
quantify the information present in a signal. In the simplestsignal model, each symbol can occur at index
with a probability
,
. What this model says is that for each signal value a
-sided coin is flipped (note that
the coin need not be fair). For this model to make sense, theprobabilities must be numbers between zero and one and must sum
to one.
This coin-flipping model assumes that symbols occur without
regard to what preceding or succeeding symbols were, a falseassumption for typed text. Despite this probabilistic
model's over-simplicity, the ideas we develop here alsowork when more accurate, but still probabilistic, models are
used. The key quantity that characterizes a symbolic-valuedsignal is the
entropy of its alphabet.
Because we use the base-2 logarithm, entropy has units of
bits. For this definition to make sense, we must take specialnote of symbols having probability zero of occurring. A
zero-probability symbol never occurs; thus, we define
so that such symbols do not affect the entropy. The
maximum value attainable by an alphabet's entropy occurswhen the symbols are equally likely
(
). In this case, the entropy equals
. The minimum value occurs when only one symbol
occurs; it has probability one of occurring and the rest haveprobability zero.
Derive the maximum-entropy results, both the
numeric aspect (entropy equals
) and the theoretical one (equally likely symbols
maximize entropy). Derive the value of the minimum entropyalphabet.
Equally likely symbols each have a probability of
. Thus,
. To prove that this is the maximum-entropy
probability assignment, we must explicitly take into accountthat probabilities sum to one. Focus on a particular
symbol, say the first.
appears
twice in the entropy
formula: the terms
and
. The derivative with respect to this probability
(and all the others) must be zero. The derivative equals
, and all other derivatives have the same form
(just substitute your letter's index). Thus, eachprobability must equal the others, and we are done. For the
minimum entropy answer, one term is
, and the others are
, which we define to be zero also. The minimum
value of entropy is zero.
Got questions? Get instant answers now!
A four-symbol alphabet has the following probabilities.
Note that these probabilities sum to one as they should. As
,
. The
entropy of this alphabet equals
Got questions? Get instant answers now!
Questions & Answers
Discuss the differences between taste and flavor, including how other sensory inputs contribute to our perception of flavor.
taste refers to your understanding of the flavor . while flavor one The other hand is refers to sort of just a blend things.
Faith
While taste primarily relies on our taste buds, flavor involves a complex interplay between taste and aroma
Kamara
which drugs can we use for ulcers
Omeprazole
Cimetidine / Tagament
For the complicated once ulcer - kit
Patrick
what is the function of lymphatic system
to drain extracellular fluid all over the body.
asegid
The lymphatic system plays several crucial roles in the human body, functioning as a key component of the immune system and contributing to the maintenance of fluid balance. Its main functions include:
1. Immune Response: The lymphatic system produces and transports lymphocytes, which are a type of
asegid
to transport fluids fats proteins and lymphocytes to the blood stream as lymph
Adama
Anatomy is the identification and description of the structures of living things
Kamara
what's the difference between anatomy and physiology
Anatomy is the study of the structure of the body, while physiology is the study of the function of the body. Anatomy looks at the body's organs and systems, while physiology looks at how those organs and systems work together to keep the body functioning.
AI-Robot
what is enzymes all about?
Enzymes are proteins that help speed up chemical reactions in our bodies. Enzymes are essential for digestion, liver function and much more. Too much or too little of a certain enzyme can cause health problems
Kamara
how does the stomach protect itself from the damaging effects of HCl
little girl okay how does the stomach protect itself from the damaging effect of HCL
Wulku
it is because of the enzyme that the stomach produce that help the stomach from the damaging effect of HCL
Kamara
function of digestive system
function of digestive
Ali
what is the normal body temperature
please why 37 degree selcius normal temperature
Mark
the normal temperature is 37°c or 98.6 °Fahrenheit is important for maintaining the homeostasis in the body
the body regular this temperature through the process called thermoregulation which involves brain skin muscle and other organ working together to maintain stable internal temperature
Stephanie
anaemia is the decrease in RBC count hemoglobin count and PVC count
Eniola
what is the pH of the vagina
how does Lysin attack pathogens
Diya
I information on anatomy position and digestive system and there enzyme
anatomy of the female external genitalia
Got questions? Join the online conversation and get instant answers!
Source:
OpenStax, Fundamentals of electrical engineering i. OpenStax CNX. Aug 06, 2008 Download for free at http://legacy.cnx.org/content/col10040/1.9
Google Play and the Google Play logo are trademarks of Google Inc.