Total Probability Theorem and Bayes Theorem

Let’s talk about something easier today, basic probability theorems.

Let’s say we have some event A, and we know the probability of event A happening, given events B_{1}, B_{2}, ... B_{n}, so we know P(A|B_{i}) for all i between 1 and n. We can find the probability of event A happening by using total probability theorem,

P(A) = P(A|B_{1})P(B_{1}) + P(A|B_{2})P(B_{2}) + ... + P(A|B_{n})P(B_{n})

An example of when you could use this? Say you rolled a die and then flipped a coin as many times as the top face on the rolled die. You could find the probability of flipping a certain amount of heads by adding together the probability of getting that many heads given each specific dice roll. The problem is then broken down into 6 simpler cases, and these can be added together for the final answer.

For Bayes theorem, given an event A and B (for now, let’s say they are not independent), recall that conditional probability, probability of event A, given event B, is defined as

P(A|B) = \frac{P(A,B)}{P(B)}

we can say the same thing given we want the probability of event B, given event A,

P(B|A) = \frac{P(B,A)}{P(A)}

with these in mind, we can make it so that P(A,B) = P(A|B)P(B) and substituting into the other equation, we get Bayes theorem,

P(B|A) = \frac{P(A|B)P(A)}{P(B)}

Bayes theorem is extremely useful, and is used in Bayesian inference methods and a lot of modern statistical methods. But what practical problem is Bayes theorem useful for?

Say you are in a casino, and you think the house is sometimes using a trick coin that gets heads more often than tails (a normal coin would be called a fair coin, with 50/50 shot of getting either one). You can use Bayesian inference to make an educated guess at the probability of the casino using a biased coin, given the results you’ve observed, and the calculation would entail using probabilities of the results which, based off some assumptions and maybe prior information, should not be too hard to calculate.

Bayes theorem is used as the basis for maximum a priori estimation, or MAP estimation, Gibbs sampling, among other things, so it’s a good idea to be well acquainted with the theorem beforehand.


One thought on “Total Probability Theorem and Bayes Theorem

  1. Pingback: Feeling Lucky? – Total Probability Theory | The Mathist

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s