Probability Spaces

Probability

At its core, the probability of an event is just a real number between $0$ and $1$, inclusively, which measures the likelihood of that event occurring.

Definition: Probability Space

A probability space $(\Omega ,P)$ is a sample space $\Omega$ equipped with a real-valued probability function $P:\mathcal{P}(\Omega )\to [0;1]$ defined on the Power Set of $\Omega$ with the following properties:

• $P(\varnothing )=0$ and $P(\Omega )=1$;
• For every countable collection $\mathcal{E}=\{E_1,E_2,\dots \}$ of mutually exclusive events, we have

$$P\left(\bigcup \mathcal{E}\right)=\sum _{E\in \mathcal{E}}P(E)$$

NOTATION

Some people may denote the probability space as $(\Omega ,\mathcal{P}(\Omega ),P)$.

Definition: (Absolute) Probability

Given an event $E\in \mathcal{P}(\Omega )$, we call $P(E)$ the (absolute) probability of $E$.

Properties

Theorem: Probability of Unions

If $A$ and $B$ are arbitrary events in a probability space, then

$$P(A\cup B)=P(A)+P(B)-P(A\cap B)$$

PROOF

TODO

Conditional Probability

Definition: Conditional Probability

Let $A$ and $B$ be two events in a probability space.

The probability of $A$ given $B$ is defined as

$$P(A\mid B)\stackrel{\text{def}}{=}\frac{P(A\cap B)}{P(B)}$$

Note: Prior and Posterior Probabilities

In the context of conditional probabilities, the number $P(A)$ is often called the prior probability and $P(A\mid B)$ the posterior probability of $A$.

Conditional probability is a measure of the likelihood that $A$ will occur if we know that $B$ has occurred.

Definition: Independent Events

Let $A$ and $B$ be two events in a probability space.

We say that $A$ is independent of $B$ if the conditional probability of $A$ given $B$ is the same as the absolute probability of $A$.

$$P(A\mid B)=P(A)$$

Theorem: Mutual Independence

If $A$ is independent of $B$, then $B$ is also independent of $A$.

PROOF

This follows directly from Bayes’ rule

Properties

Theorem: Bayes' Rule

If $A$ and $B$ are two events in a probability space, then their conditional probabilities are related as follows:

$$P(A\mid B)=\frac{P(A)}{P(B)}P(B\mid A)$$

NOTE

Bayes’ rules essentially allows us to switch the events around.

PROOF

By definition,

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}$$

and so

$$P(A\cap B)=P(A\mid B)P(B).$$

Similarly,

$$P(B\mid A)=\frac{P(B\cap A)}{P(A)}=\frac{P(A\cap B)}{P(A)}$$

and so

$$P(A\cap B)=P(B\mid A)P(A).$$

By combining the two equations, we obtain the result from the theorem.

$$P(A\mid B)P(B)=P(B\mid A)P(A)$$ $$P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}$$

Theorem: Law of Total Probability

Let $(\Omega ,P)$ be a probability space, let $\{B_1,\dots ,B_n\}$ be a collection of events and let $A$ be some other event.

If $\{B_1,\dots ,B_n\}$ are mutually exclusive and their union is $\Omega$, then the probability of $A$ is the sum of its conditional probability given each $B_i$ multiplied by the probability of $B_i$:

$$P(A)=\sum _{i=1}^{n}P(A\mid B_i)P(B_i)$$

PROOF

TODO