Conditional expectation, regression

Conditional expectation, given a random vector, plays a fundamental role in much of modern probability theory. Various types of “conditioning” characterize some of the moreimportant random sequences and processes. The notion of conditional independence is expressed in terms of conditional expectation. Conditional independence plays an essential role inthe theory of Markov processes and in much of decision theory.

We first consider an elementary form of conditional expectation with respect to an event. Then we consider two highly intuitive special cases of conditional expectation, given a random variable.In examining these, we identify a fundamental property which provides the basis for a very general extension. We discover that conditional expectation is a random quantity. The basic property forconditional expectation and properties of ordinary expectation are used to obtain four fundamental properties which imply the “expectationlike” character of conditional expectation. An extensionof the fundamental property leads directly to the solution of the regression problem which, in turn, gives an alternate interpretation of conditional expectation.

Conditioning by an event

If a conditioning event C occurs, we modify the original probabilities by introducing the conditional probability measure $P(·|C)$ . In making the change from

we effectively do two things:

• We limit the possible outcomes to event C
• We “normalize” the probability mass by taking $P\left(C\right)$ as the new unit

It seems reasonable to make a corresponding modification of mathematical expectation when the occurrence of event C is known. The expectation $E\left[X\right]$ is the probability weighted average of the values taken on by X . Two possibilities for making the modification are suggested.

• We could replace the prior probability measure $P(·)$ with the conditional probability measure $P(·|C)$ and take the weighted average with respect to these new weights.
• We could continue to use the prior probability measure $P(·)$ and modify the averaging process as follows:
  • Consider the values $X\left(\omega \right)$ for only those $\omega \in C$ . This may be done by using the random variable $I_{C}X$ which has value $X\left(\omega \right)$ for $\omega \in C$ and zero elsewhere. The expectation $E\left[I_{C}X\right]$ is the probability weighted sum of those values taken on in C .
  • The weighted average is obtained by dividing by $P\left(C\right)$ .

These two approaches are equivalent. For a simple random variable ${\displaystyle X=\sum _{k=1}^n{t}_k{I}_{A_k}}$ in canonical form

The final sum is expectation with respect to the conditional probability measure. Arguments using basic theorems on expectation and the approximation of generalrandom variables by simple random variables allow an extension to a general random variable X . The notion of a conditional distribution, given C , and taking weighted averages with respect to the conditional probability is intuitive and natural in this case. However,this point of view is limited. In order to display a natural relationship with more the general concept of conditioning with repspect to a random vector, we adopt the following