Model Selection in R (AIC Vs BIC)

Model Selection in R, Let’s look at a linear regression model using mtcars dataset.

First, we need to brush up on our knowledge by looking at the mtcars dataset.

head(mtcars)

M                   mpg cyl disp  hp drat    wt  qsec vs am gear carb
Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

mtcars collects data on a variety of car attributes, but let’s pretend we’re trying to develop a model to better understand the link between mpg and the other factors in the mtcars dataset.

Here we are going to start with a basic model in where we theorize that a car’s engine displacement (in cubic inches), disp, contributes to its mpg.

One Sample Analysis in R » Quick Guide »

Model Selection in R

mtcars.lm <- lm(mpg ~ disp, data=mtcars)
summary(mtcars.lm)

Call:
lm(formula = mpg ~ disp, data = mtcars)

Residuals:
    Min      1Q  Median      3Q     Max 
-4.8922 -2.2022 -0.9631  1.6272  7.2305 

Coefficients:
             Estimate Std. Error t value Pr(>|t|)    
(Intercept) 29.599855   1.229720  24.070  < 2e-16 ***
disp        -0.041215   0.004712  -8.747 9.38e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.251 on 30 degrees of freedom
Multiple R-squared:  0.7183,	Adjusted R-squared:  0.709 
F-statistic: 76.51 on 1 and 30 DF,  p-value: 9.38e-10

We can see certain metrics of model performance in our model summary, but if we want to know our model’s AIC and BIC, we can make use of the glance() function from the broom package.

Naive Bayes Classification in R » Prediction Model »

broom::glance(mtcars.lm)

r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC deviance df.residual  nobs
      <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>    <dbl>       <int> <int>
1     0.718         0.709  3.25      76.5 9.38e-10     1  -82.1  170.  175.     317.          30    32

As a result, our basic model contains AIC is170 and BIC is 175.

What if we add a third predictor to the model?

Let’s try a model in which the weight (in thousands of pounds) of an automobile, wt, is also used to explain its mpg.

mtcars.lm <- lm(mpg ~ disp + wt, data=mtcars)
broom::glance(mtcars.lm)

  r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC deviance df.residual  nobs
      <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>    <dbl>       <int> <int>
1     0.781         0.766  2.92      51.7 2.74e-10     2  -78.1  164.  170.     247.          29    32

Our new model includes AIC is164 and BIC is 170. Yes, It’s worth noting that our new model’s AIC and BIC are both lower than our previous model.

This suggests that the benefits of enhanced explanatory power outweigh the cost of increasing model complexity, according to both information criteria.

Cluster Meaning-Cluster or area sampling in a nutshell »

Let’s try to some more parameters into the model and will check how the AIC and BIC of our model change as a result.

mtcars.lm <- lm(mpg ~ disp + wt + hp, data=mtcars)
broom::glance(mtcars.lm)

 r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC deviance df.residual  nobs
      <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>    <dbl>       <int> <int>
1     0.827         0.808  2.64      44.6 8.65e-11     3  -74.3  159.  166.     195.          28    32

Yes, AIC is 159 and BIC is 166

mtcars.lm <- lm(mpg ~ disp + wt + hp + cyl, data=mtcars) 
broom::glance(mtcars.lm)

r.s  r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC deviance df.residual  nobs
      <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>    <dbl>       <int> <int>
1     0.849         0.826  2.51      37.8 1.06e-10     4  -72.2  156.  165.     170.          27    32

Wow, AIC become now 156 and BIC is 165. Cool, let’s try another model.

Principal component analysis (PCA) in R »

mtcars.lm <- lm(mpg ~ disp + wt + hp + cyl + gear, data=mtcars)
broom::glance(mtcars.lm)

  r.squared adj.r.squared sigma statistic  p.value    df logLik   AIC   BIC deviance df.residual  nobs
      <dbl>         <dbl> <dbl>     <dbl>    <dbl> <dbl>  <dbl> <dbl> <dbl>    <dbl>       <int> <int>
1     0.851         0.822  2.54      29.7 5.72e-10     5  -71.9  158.  168.     168.          26    32

Oh no, AIC is 158 and BIC is 168, both increased which indicates we’ve done too much!.

The final model’s AIC and BIC have increased while adding gears into the model. Given that our model already included disp, wt, hp, and cyl, the boost in explanatory power gained by introducing gear was not worth the increase in model complexity.

Conclusion

So we started with a simple linear regression model and gradually increased the number of parameters until the AIC and BIC stopped falling.

It’s cool, but it isn’t the end of the narrative. We haven’t given any thought to how we would determine which variables to include in our model.

Yes, we will discuss the same, fascinating story in the upcoming post!

KNN Algorithm Machine Learning » Classification & Regression »