Bayes’ Theorem and Compound Probability
Abstract
In this class, two fundamental concepts in probability were discussed: conditional probability and compound probability. The difference between P(A|B) and P(B|A) was emphasized. The Compound Probability Theorem states that the probability of an event A can be expressed as the sum of the conditional probabilities P(A|B_i) multiplied by the probabilities of the events B_i. Subsequently, Bayes’ Theorem was introduced, which allows the calculation of the conditional probability P(B_k|A) using the conditional probability P(A|B_k), the probability P(B_k), and the sum of the conditional probabilities P(A|B_i) multiplied by the probabilities of the events B_i. These concepts are essential for understanding and applying conditional probability in various contexts, and Bayes’ Theorem provides a powerful tool for updating probabilities based on new information.
LEARNING OBJECTIVES:
By the end of this class, the student will be able to:
- Understand the concept of conditional probability and differentiate between P(A|B) and P(B|A).
- Calculate the probability of an event using compound probabilities.
- Demonstrate Bayes’ rule.
TABLE OF CONTENTS
Compound Probability and Conditional Probability
Bayes’ Theorem
In the previous class, we reviewed the concept of conditional probability and also clarified that one should never confuse a conditional probability of the form P(A|B) with P(B|A). Although in everyday language conditionality can be confusing, mathematically they are two very different things that, however, are related. This relationship is described by Bayes’ Theorem, which is based on the notion of compound probability for its formulation.
Compound Probability and Conditional Probability
THEOREM: If A is an event and B_1, B_2, \cdots, B_n form a set of disjoint events such that \displaystyle \bigcup_{i=1}^n B_i = \Omega, then the following holds:
\boxed{P(A) = \displaystyle \sum_{i=1}^n P(A|B_i) P(B_i)}
This way of writing the probability of A is what we call the Compound Probability of A.
PROOF:
| (1) | A is an Event | ; Premise |
| (2) | \displaystyle \bigcup_{i=1}^n B_i = \Omega | ; Premise |
| (3) | B_1, \cdots, B_n are all mutually exclusive | ; Premise |
| (4) | (A\cap B_i)\cap(A\cap B_j) = \varnothing, with i\neq j and i,j\in \{1,2,3,\cdots n\} | ; From (1,2,3) |
| (5) | \displaystyle \bigcup_{i=1}^n \left(A \cap B_i \right) = A | ; From (1,2,3) |
| (6) | \displaystyle P(A) = P\left( \bigcup_{i=1}^n \left(A \cap B_i \right) \right) = \sum_{i=1}^n P\left( A \cap B_i \right) | ; From (4,5) |
| (7) | P(A|B_i) = \dfrac{P(A\cap B_i)}{P(B_i)} | ; Definition of Conditional Probability |
| P(A\cap B_i) = P(A|B_i) P(B_i) | ||
| (8) | \boxed{\displaystyle P(A) = \sum_{i=1}^n P(A|B_i) P(B_i)} | ; From (6,7) |
Bayes’ Theorem
In the same context as the previous theorem, the following theorem holds:
THEOREM:
P(B_k|A) = \dfrac{P(A|B_k)P(B_k)}{\displaystyle\sum_{i=1}^n P(A|B_i)P(B_i)} = \dfrac{P(A|B_k)P(B_k)}{P(A)}
PROOF: If A is any event and B_1, B_2, \cdots, B_n is a collection of disjoint events such that \displaystyle \bigcup_{i=1}^n B_i = \Omega, by the previous Compound Probability Theorem, we have that:
P(A) = \displaystyle \sum_{i=1}^n P(A|B_i)P(B_i)
Now, using the fact that P(X\cap Y) = P(X|Y)P(Y), we have that if we replace Y=A and X=B_k, we will arrive at
P(A) = \dfrac{P(B_k \cap A)}{P(B_k|A)}
On the other hand, we have that
P(A|B_k) = \dfrac{P(A\cap B_k)}{P(B_k)}
From which it follows that
P(B_k \cap A) = P(A|B_k)P(B_k)
Now, if we replace the green within the blue, we will have
P(A) = \dfrac{P(A|B_k)P(B_k)}{P(B_k|A)}
Which is equivalent to saying
\boxed{P(B_k|A) = \dfrac{P(A|B_k)P(B_k)}{P(A)}= \dfrac{P(A|B_k)P(B_k)}{\displaystyle \sum_{i=1}^n P(A|B_i) P(B_i)} }
This is what we wanted to prove.