Introduction to Statistical Learning

Statistical learning refers to a vast set of tools for understanding data. These tools can be classified as supervised or unsupervised.

What is Statistical Learning?

Suppose that we observe a quantitative response $Y$ and $p$ different predictors, $X_1, X_2, \dots, X_p$. We assume that there is some relationship between $Y$ and $X = (X_1, X_2, \dots, X_p)$, which can be written in the very general form:

Here $f$ is some fixed but unknown function of $X_1, \dots, X_p$, and $\epsilon$ is a random error term, which is independent of $X$ and has mean zero. In this formulation, represents the —information that provides about . Since we generally do not know , statistical learning is the set of approaches for estimating .

Why Estimate $f$?

There are two main reasons that we may wish to estimate $f$: prediction and inference.

1. Prediction: In many situations, a set of inputs $X$ are readily available, but the output $Y$ cannot be easily obtained.
2. Inference: We are often interested in understanding the way that $Y$ is affected as $X_1, \dots, X_p$ change.

The Trade-Off Between Prediction Accuracy and Model Interpretability

One might reasonably ask: Why would we ever choose to use a more restrictive method instead of a very flexible approach?

If we are mainly interested in inference, then restrictive models are much more interpretable. For instance, when inference is the goal, the linear model is clearly a good choice since it will be quite easy to understand the relationship between $Y$ and $X_1, \dots, X_p$.

In contrast, very flexible approaches, such as the support vector machines, bagging, and boosting, can lead to such complicated estimates of $f$ that it is difficult to understand how any individual predictor is associated with the response.