<< Chapter < Page Chapter >> Page >
In this module we consider differential entropy.

Consider the entropy of continuous random variables. Whereas the (normal) entropy is the entropy of a discrete random variable, the differential entropy is the entropy of a continuous random variable.

Differential entropy

Differential entropy
The differential entropy h X of a continuous random variable X with a pdf f x is defined as
h X x f x f x
Usually the logarithm is taken to be base 2, so that the unit of the differential entropy is bits/symbol. Note that is the discrete case, h X depends only on the pdf of X . Finally, we note that the differential entropy is the expected value of f x , i.e.,
h X E f x

Now, consider a calculating the differential entropy of some random variables.

Consider a uniformly distributed random variable X from c to c Δ . Then its density is 1 Δ from c to c Δ , and zero otherwise.

We can then find its differential entropy as follows,

h X x c c Δ 1 Δ 1 Δ Δ
Note that by making Δ arbitrarily small, the differential entropy can be made arbitrarily negative, while taking Δ arbitrarily large, the differential entropy becomes arbitrarily positive.

Got questions? Get instant answers now!

Consider a normal distributed random variable X , with mean m and variance σ 2 . Then its density is 1 2 σ 2 e x m 2 2 σ 2 .

We can then find its differential entropy as follows, first calculate f x :

f x 1 2 2 σ 2 e x m 2 2 σ 2
Then since E X m 2 σ 2 , we have
h X 1 2 2 σ 2 1 2 e 1 2 2 e σ 2

Got questions? Get instant answers now!

Properties of the differential entropy

In the section we list some properties of the differential entropy.

  • The differential entropy can be negative
  • h X c h X , that is translation does not change the differential entropy.
  • h a X h X a , that is scaling does change the differential entropy.
The first property is seen from both and . The two latter can be shown by using .

Questions & Answers

List and explain four factors of production
Vuyo Reply
capital labour entrepreneur natural resources
Thembi
What is supply
Ogodo Reply
when the supply decreases demand also decreases
Thembi
types of demand and the explanation
akin Reply
what is demand
akin Reply
other things remaining same if demend is increases supply is also decrease and if demend is decrease supply is also increases is called the demand
Mian
if the demand increase supply also increases
Thembi
you are wrong this is the law of demand and not the definition
Tarasum
Demand is the willingness of buy and ability to buy in a specific time period in specific place. Mian you are saying law of demand but not in proper way. you have to keep studying more. because its very basic things in Economics.
Hamza
what is commercialization?
Doris Reply
How to talk loan for bank?
Alfred Reply
what is the meaning of gpa?
Ritisha Reply
Answer: GPA stands for Grade Point Average. It is a standard way of measuring academic achievement in the U.S. Basically, it goes as follows: Each course is given a certain number of "units" or "credits", depending on the content of the course.
Yusuf
what is small and Microbuisenes
tadesse Reply
What is fiscal policy
Dansofo
Who is the funder of Economic
Dansofo
founder , that is Adam Smith
Daniel
what is model
Daniel Reply
The wealth of Nations
Yusuf Reply
the wealth of nations, is it the first?
Umar
Yes very sure it was released in 1759
Yusuf
thank you Yusuf.
Umar
then when did he died?
Umar
17 July 1790 Born: 16 June 1723, Kirkcaldy, United Kingdom Place of death: Panmure House, Edinburgh, United Kingdom
Yusuf
1790
Yusuf
that's my today questions, thank you Yusuf it's bed time see u after.
Umar
what is fiscal policy
kemigisha Reply
what's mode?
Umar Reply
mode is the highest occurring frequency in a distribution
Bola
mode is the most commonly occurring item in a set of data.
Umar
Please, what is the difference between monopoly and monopsony?
Olaleye Reply
is there monopsony word?
Umar
I have no idea though
Umar
please, in which year Adam smith was born?
Umar
monopsony is when there's only one buyer while monopoly is when there's only one producer.
Bola
who have idea on Banter
Ibrahim
like trade by barter?
Bola
Monopoly is when there's excessively one seller and there is no entry in the market while monopsony is when there is one buyer
kemigisha
Adam smith was born in 1723
Bola
 (uncountable) Good humoured, playful, typically spontaneous conversation. verb (intransitive) To engage in banter or playful conversation. (intransitive) To play or do something amusing. (transitive) To tease mildly.
Umar
which book Adam smith published first? the first book of Adam smith pls.
Umar
wealth on nation, 1776
Daniel
what is market power and how can it affect an economy?
Gab Reply
market power:- where a firm is said to be a price setter.market power benefits the powerful at the expense of others.
Umar
Market power refers to the ability of a firm (or group of firms) to raise and maintain price above the level that would prevail under competition is referred to as market or monopoly power. The exercise of market power leads to reduced output and loss of economic welfare
Kartheek
find information about the national budget
Molahlegi
three branches of economics in which tourism is likely to figure
Makgotso Reply
What are those three branches?
IlRegno
Got questions? Join the online conversation and get instant answers!
Jobilize.com Reply

Get the best Algebra and trigonometry course in your pocket!





Source:  OpenStax, Information and signal theory. OpenStax CNX. Aug 03, 2006 Download for free at http://legacy.cnx.org/content/col10211/1.19
Google Play and the Google Play logo are trademarks of Google Inc.

Notification Switch

Would you like to follow the 'Information and signal theory' conversation and receive update notifications?

Ask