ML Theory: Conditional Probability 101

Venn diagrams are useful tools for visualizing different probability spaces. Consider two events C and D within some sample space S. Suppose C is the event of owning a cat and D is the event of owning a dog. Our Venn diagram might look something like the following for a sample of 450 pet owners.

Venn diagram
Venn diagram of a sample of 450 pet owners. Here, C is the event of an individual owing a cat and D is the event of an individual owing a dog.

In other words, out of a sample of 450 pet owners,

  • 400 owned a cat exclusively,
  • 36 owned a dog exclusively,
  • 9 owned both a cat and a dog, and
  • 5 owned neither a cat nor a dog.

Obviously, the probability of owning a cat,

    \[ Pr(C) = Pr(\text{owning a cat}) = \frac{400 + 9}{450} = 0.91 \]

is much higher than the probability of owning a dog,

    \[ Pr(D) = Pr(\text{owning a dog}) = \frac{36 + 9}{450} = 0.10 \]

as cats are the superior animal. We can also estimate the probability of being one of those people with neither a cat nor a dog as

    \[ Pr(\bar{C \cup D}) = \frac{5}{450} = 0.01 \]

Now, suppose we select a cat owner at random. What is the probability that they are also a dog owner? Well, since we know the individual is a cat owner, we only need to focus on the left hand bubble of our Venn diagram.

The number of individuals of interest to us is therefore 409, which we will denote n_c. Of those 409, only 9 are within the intersection (denoted C \cap D). We will refer to these 9 as n_{C \cap D}. The relevant probability can therefore be estimated as,

    \[ Pr(D|C) = \frac{n_{C \cap D}}{n_{C}} = \frac{9}{409} = 0.02 \]

Pr(D|C) represents the probability an individual owns a dog given they have a cat. This is what we refer to as a conditional probability. When calculating a conditional probability, we use additional information to reduce the size of our probability space. In this example, this meant that we could focus solely on the cat owner portion of the pictured Venn diagram.

In general, for two events A and B, the law of conditional probability states that the probability of observing A given event B can be represented as

    \[ Pr(A|B) = \frac{Pr(A \cap B)}{Pr(B)} \]

This is exactly the formula we were using for the cats and dogs example,

    \[ Pr(D|C) = \frac{Pr(D \cap C)}{Pr(C)} = \frac{9}{450} \times \frac{450}{409} = \frac{9}{409} \]

We often use conditional probability in our everyday lives without thinking about it. For example, if you tested positive for an extremely rare disease, you should be thinking that the test is likely inaccurate rather than you have the disease… or maybe not because you might be freaking out right now. However, your doctor should be! As a result, your doctor would send you for further testing.