-
Home
- Fundamentals of electrical
- Information communication
- Entropy
Shannon showed the power of probabilistic models for symbolic-valued signals. The dey quantity that characterizes such a signal is the entropy of its alphabet.
Communication theory has been formulated best for
symbolic-valued signals. ClaudeShannon published in 1948
The Mathematical Theory
of Communication , which became the cornerstone of digital
communication. He showed the power of
probabilistic
models for symbolic-valued signals, which allowed him to
quantify the information present in a signal. In the simplestsignal model, each symbol can occur at index
with a probability
,
. What this model says is that for each signal value a
-sided coin is flipped (note that
the coin need not be fair). For this model to make sense, theprobabilities must be numbers between zero and one and must sum
to one.
This coin-flipping model assumes that symbols occur without
regard to what preceding or succeeding symbols were, a falseassumption for typed text. Despite this probabilistic
model's over-simplicity, the ideas we develop here alsowork when more accurate, but still probabilistic, models are
used. The key quantity that characterizes a symbolic-valuedsignal is the
entropy of its alphabet.
Because we use the base-2 logarithm, entropy has units of
bits. For this definition to make sense, we must take specialnote of symbols having probability zero of occurring. A
zero-probability symbol never occurs; thus, we define
so that such symbols do not affect the entropy. The
maximum value attainable by an alphabet's entropy occurswhen the symbols are equally likely
(
). In this case, the entropy equals
. The minimum value occurs when only one symbol
occurs; it has probability one of occurring and the rest haveprobability zero.
Derive the maximum-entropy results, both the
numeric aspect (entropy equals
) and the theoretical one (equally likely symbols
maximize entropy). Derive the value of the minimum entropyalphabet.
Equally likely symbols each have a probability of
. Thus,
. To prove that this is the maximum-entropy
probability assignment, we must explicitly take into accountthat probabilities sum to one. Focus on a particular
symbol, say the first.
appears
twice in the entropy
formula: the terms
and
. The derivative with respect to this probability
(and all the others) must be zero. The derivative equals
, and all other derivatives have the same form
(just substitute your letter's index). Thus, eachprobability must equal the others, and we are done. For the
minimum entropy answer, one term is
, and the others are
, which we define to be zero also. The minimum
value of entropy is zero.
Got questions? Get instant answers now!
A four-symbol alphabet has the following probabilities.
Note that these probabilities sum to one as they should. As
,
. The
entropy of this alphabet equals
Got questions? Get instant answers now!
Questions & Answers
differentiate between demand and supply
giving examples
differentiated between demand and supply using examples
Lambiv
how is the graph works?I don't fully understand
hi guys good evening to all
Lambiv
In economics, a perfect market refers to a theoretical construct where all participants have perfect information, goods are homogenous, there are no barriers to entry or exit, and prices are determined solely by supply and demand. It's an idealized model used for analysis,
Ezea
other things being equal
AI-Robot
When MP₁ becomes negative, TP start to decline.
Extuples Suppose that the short-run production function of certain cut-flower firm is given by: Q=4KL-0.6K2 - 0.112 •
Where is quantity of cut flower produced, I is labour input and K is fixed capital input (K-5). Determine the average product of lab
Kelo
Extuples Suppose that the short-run production function of certain cut-flower firm is given by: Q=4KL-0.6K2 - 0.112 •
Where is quantity of cut flower produced, I is labour input and K is fixed capital input (K-5). Determine the average product of labour (APL) and marginal product of labour (MPL)
Kelo
Can I ask you other question?
Shukri
What is different between quantity demand and demand?
Quantity demanded refers to the specific amount of a good or service that consumers are willing and able to purchase at a give price and within a specific time period. Demand, on the other hand, is a broader concept that encompasses the entire relationship between price and quantity demanded
Ezea
how do you save a country economic situation when it's falling apart
what is the difference between economic growth and development
Economic growth as an increase in the production and consumption of goods and services within an economy.but
Economic development as a broader concept that encompasses not only economic growth but also social & human well being.
Shukri
production function means
Jabir
What do you think is more important to focus on when considering inequality ?
any question about economics?
sir...I just want to ask one question... Define the term contract curve? if you are free please help me to find this answer 🙏
Asui
it is a curve that we get after connecting the pareto optimal combinations of two consumers after their mutually beneficial trade offs
Awais
thank you so much 👍 sir
Asui
In economics, the contract curve refers to the set of points in an Edgeworth box diagram where both parties involved in a trade cannot be made better off without making one of them worse off. It represents the Pareto efficient allocations of goods between two individuals or entities, where neither p
Cornelius
In economics, the contract curve refers to the set of points in an Edgeworth box diagram where both parties involved in a trade cannot be made better off without making one of them worse off. It represents the Pareto efficient allocations of goods between two individuals or entities,
Cornelius
Suppose a consumer consuming two commodities X and Y has
The following utility function u=X0.4 Y0.6. If the price of the X and Y are 2 and 3 respectively and income Constraint is birr 50.
A,Calculate quantities of x and y which maximize utility.
B,Calculate value of Lagrange multiplier.
C,Calculate quantities of X and Y consumed with a given price.
D,alculate optimum level of output .
the market for lemon has 10 potential consumers, each having an individual demand curve p=101-10Qi, where p is price in dollar's per cup and Qi is the number of cups demanded per week by the i th consumer.Find the market demand curve using algebra. Draw an individual demand curve and the market dema
suppose the production function is given by ( L, K)=L¼K¾.assuming capital is fixed find APL and MPL. consider the following short run production function:Q=6L²-0.4L³ a) find the value of L that maximizes output b)find the value of L that maximizes marginal product
Abdureman
What is the difference between perfect competition and monopolistic competition?
Mohammed
Got questions? Join the online conversation and get instant answers!
Source:
OpenStax, Fundamentals of electrical engineering i. OpenStax CNX. Aug 06, 2008 Download for free at http://legacy.cnx.org/content/col10040/1.9
Google Play and the Google Play logo are trademarks of Google Inc.