Bias Variance Trade-Off

Underfitting: High Bias, Low Variance, High Train Error, High Test Error
Overfitting: Low Bias, High Variance, Low Train Error, High Test Error

Regularization: Choose complexity level that has both Low Bias and Low Variance. Low Training Error and Low Test Error.


High Bias + Low Variance -> High Error -> High Test Error
Low Bias + High Variance -> High Error -> High Test Error
Low Bias + Low Variance -> Low Error -> Low Test Error

| Unfortunately, we can’t do this independently, there is a trade-off


Model Error can be expressed by Test Error

Test Error + Regularization Penalty

Underfitting: High Test Error + Low Reg
Overfitting: High Test Error + High Reg