Real study notes, generated by Neurix Reader

2 Probability: Univariate Models

2.1 Introduction

This chapter serves as a foundational introduction to the basic principles of probability theory. For more in-depth study, readers may refer to resources such as [GS97; BT08].

2.1.1 What is probability?

As Pierre Laplace famously stated, "Probability theory is nothing but common sense reduced to calculation." While we intuitively understand statements like "a fair coin has a 50% chance of landing heads," the underlying meaning of probability can be interpreted in two primary ways:

1. Frequentist interpretation: In this view, probabilities quantify the long-run relative frequencies of events that can be repeated numerous times. For instance, stating a coin has a 50% probability of heads implies that if flipped many times, approximately half the outcomes will be heads.
2. Bayesian interpretation: Here, probability is used to express our uncertainty or degree of belief about a particular outcome or state of affairs. It is intrinsically linked to information rather than requiring repeated trials [Jay03; Lin06]. Under this interpretation, the 50% probability for a fair coin means we believe it is equally likely to land heads or tails on the next toss.

A significant advantage of the Bayesian approach is its ability to model uncertainty for unique, "one-off" events that lack long-term frequencies. For example, quantifying the probability of the polar ice cap melting by a specific year (e.g., 2030 CE) is possible, even though this event will either happen once or not at all. This allows for informed decision-making under uncertainty, as further discussed in Chapter 5. This book will primarily adopt the Bayesian interpretation, though the fundamental rules of probability theory remain consistent across both viewpoints.

2.1.2 Types of uncertainty

Uncertainty in predictions typically stems from two distinct sources:

1. Epistemic uncertainty (or model uncertainty): This type of uncertainty arises from our lack of knowledge regarding the underlying causes or mechanisms that generate our data. It is related to epistemology, the study of knowledge.
2. Aleatoric uncertainty (or data uncertainty): This refers to intrinsic variability that cannot be eliminated, even with perfect knowledge or more data. The term "aleatoric" comes from the Latin word for "dice" [Hac75; KD09]. A simpler synonym is data uncertainty. For example, knowing a fair coin has a probability of heads $p=0.5$ eliminates epistemic uncertainty about $p$, but we still cannot perfectly predict the outcome of a single flip due to aleatoric uncertainty.

Distinguishing between these types of uncertainty is crucial in applications like active learning. A common strategy involves querying examples where $\mathbb{H}(p(y|x, \mathcal{D}))$ is high (where $\mathbb{H}(p)$ denotes entropy, discussed in Section 6.1). This high uncertainty could be due to either epistemic uncertainty about model parameters ($\mathbb{H}(p(\theta|\mathcal{D}))$) or inherent noise in the data itself ($\mathbb{H}(p(y|x, \theta))$) [Osb16].

2.1.3 Probability as an extension of logic

Following the framework presented in [Jay03], we can view probability as a generalization of Boolean logic.

2.1.3.1 Probability of an event

An event is defined as a state of the world that either occurs or does not occur, represented by a binary variable $A$. Examples include "it will rain tomorrow," "the label $y=1$," or "the parameter $\theta$ is between 1.5 and 2.0."

The expression $\text{Pr}(A)$ denotes the probability that event $A$ is true (or, in the frequentist view, the long-run proportion of times $A$ will occur). Probabilities must satisfy: $0 \le \text{Pr}(A) \le 1$ where $\text{Pr}(A) = 0$ signifies an impossible event, and $\text{Pr}(A) = 1$ signifies a certain event. The probability of event $A$ not happening is denoted $\text{Pr}(\bar{A})$ and is defined as: $\text{Pr}(\bar{A}) = 1 - \text{Pr}(A)$

…notes continue for the full page range you select.