Chapter 8
Introduction to Linear Regression
Learning Outcomes
- Define the explanatory variable as the independent variable (predictor), and the response variable as the dependent variable (predicted).
- Plot the explanatory variable ($x$) on the x-axis and the response variable ($y$) on the y-axis, and fit a linear regression model
$y = \beta_0 + \beta_1 x$where $\beta_0$ is the intercept, and $\beta_1$ is the slope.- Note that the point estimates (estimated from observed data) for $\beta_0$ and $\beta_1$ are $b_0$ and $b_1$, respectively.
- When describing the association between two numerical variables, evaluate
- direction: positive ($x \uparrow, y \uparrow$), negative ($x \downarrow, y \uparrow$)
- form: linear or not
- strength: determined by the scatter around the underlying relationship
- Define correlation as the \emph{linear} association between two numerical variables.
- Note that a relationship that is nonlinear is simply called an association.
- Note that correlation coefficient ($r$, also called Pearson’s $r$) the following properties:
- the magnitude (absolute value) of the correlation coefficient measures the strength of the linear association between two numerical variables
- the sign of the correlation coefficient indicates the direction of association
- the correlation coefficient is always between -1 and 1, inclusive, with -1 indicating perfect negative linear association, +1 indicating perfect positive linear association, and 0 indicating no \emph{linear} relationship
- the correlation coefficient is unitless
- since the correlation coefficient is unitless, it is not affected by changes in the center or scale of either variable (such as unit conversions)
- the correlation of X with Y is the same as of Y with X
- the correlation coefficient is sensitive to outliers
- Recall that correlation does not imply causation.
- Define residual ($e$) as the difference between the observed ($y$) and predicted ($\hat{y}$) values of the response variable.
$e_i = y_i - \hat{y}_i$ - Define the least squares line as the line that minimizes the sum of the squared residuals, and list conditions necessary for fitting such line:
- linearity
- nearly normal residuals
- constant variability
- Define an indicator variable as a binary explanatory variable (with two levels).
- Calculate the estimate for the slope ($b_1$) as
$b_1 = R\frac{s_y}{s_x}$, where $r$ is the correlation coefficient, $s_y$ is the standard deviation of the response variable, and $s_x$ is the standard deviation of the explanatory variable. - Interpret the slope as
- “For each unit increase in $x$, we would expect $y$ to increase/decrease on average by $|b_1|$ units” when $x$ is numerical.
- “The average increase/decrease in the response variable when between the baseline level and the other level of the explanatory variable is $|b_1|$.” when $x$ is categorical.
- Note that whether the response variable increases or decreases is determined by the sign of $b_1$.
- Note that the least squares line always passes through the average of the response and explanatory variables ($\bar{x},\bar{y}$).
- Use the above property to calculate the estimate for the slope ($b_0$) as
$b_0 = \bar{y} - b_1 \bar{x}$, where $b_1$ is the slope, $\bar{y}$ is the average of the response variable, and $\bar{x}$ is the average of explanatory variable. - Interpret the intercept as
- “When $x = 0$, we would expect $y$ to equal, on average, $b_0$.” when $x$ is numerical.
- “The expected average value of the response variable for the reference level of the explanatory variable is $b_0$.” when $x$ is categorical.
- Predict the value of the response variable for a given value of the explanatory variable, $x^\star$, by plugging in $x^\star$ in the in the linear model:
$\hat{y} = b_0 + b_1 x^\star$- Only predict for values of $x^\star$ that are in the range of the observed data.
- Do not extrapolate beyond the range of the data, unless you are confident that the linear pattern continues.
- Define $R^2$ as the percentage of the variability in the response variable explained by the the explanatory variable.
- For a good model, we would like this number to be as close to 100% as possible.
- This value is calculated as the square of the correlation coefficient, and is between 0 and 1, inclusive.
- Define a leverage point as a point that lies away from the center of the data in the horizontal direction.
- Define an influential point as a point that influences (changes) the slope of the regression line.
- This is usually a leverage point that is away from the trajectory of the rest of the data.
- Do not remove outliers from an analysis without good reason.
- Be cautious about using a categorical explanatory variable when one of the levels has very few observations, as these may act as influential points.
- Determine whether an explanatory variable is a significant predictor for the response variable using the $t$-test and the associated p-value in the regression output.
- Set the null hypothesis testing for the significance of the predictor as
$H_0: \beta_1 = 0$, and recognize that the standard software output yields the p-value for the two-sided alternative hypothesis.- Note that $\beta_1 = 0$ means the regression line is horizontal, hence suggesting that there is no relationship between the explanatory and the response variables.
- Calculate the T score for the hypothesis test as
$T_{df}=\frac { b_{ 1 }-{ null\quad value } }{ SE_{ b_{ 1 } } }$with $df = n - 2$.- Note that the T score has $n - 2$ degrees of freedom since we lose one degree of freedom for each parameter we estimate, and in this case we estimate the intercept and the slope.
- Note that a hypothesis test for the intercept is often irrelevant since it’s usually out of the range of the data, and hence it is usually an extrapolation.
- Calculate a confidence interval for the slope as
$b_1 \pm t^\star_{df} SE_{b_1}$where $df = n - 2$ and$t^\star_{df}$is the critical score associated with the given confidence level at the desired degrees of freedom.- Note that the standard error of the slope estimate
$SE_{b_1}$can be found on the regression output.
- Note that the standard error of the slope estimate
Supplemental Readings
-
Linear regression with SAT scores - This document outlines the implementation of linear regression step-by-step emphasizing visualizations.