Best Subset Regression

Best subset regression fits every combination of candidate predictors, keeps the lowest residual sum of squares model of each size, and chooses among them by BIC, AIC, Mallows' Cp or adjusted R-squared.

최량 부분집합 회귀는 후보 설명변수의 모든 조합을 맞춰 크기별 잔차제곱합 최소 모형을 찾고 BIC·AIC·Mallows Cp·수정 R²로 하나를 고르는 방법입니다.

I have twelve candidate predictors; which subset gives the best linear model without trying them by hand?

후보 설명변수가 12개인데, 일일이 넣어 보지 않고 가장 좋은 변수 조합을 찾을 수 있나요?

Best subset regression evaluates all predictor combinations, so the chosen model is the true optimum of the selection criterion rather than the result of a greedy path. The size-wise table shows how fit and penalised criteria change with model size, but inference on the selected model is optimistic.

When to use Best Subset Regression

• There are up to about 20 candidate predictors and an exhaustive search is affordable.
• Model sizes need to be compared by an information criterion.
• Stepwise selection is to be checked against the global optimum.

When to use something else

• There are many more predictors or strong collinearity; use the lasso or elastic net.
• Confirmatory hypothesis tests on prespecified predictors are planned; fit the prespecified model with multiple linear regression.

Assumptions and requirements

• The full linear model holds with independent, homoskedastic errors.
• There are more observations than parameters in the full model.
• Categorical predictors enter or leave as a whole block of dummies.

What the analysis reports

• Best model of each size with RSS, R-squared, adjusted R-squared, Cp, AIC and BIC
• Chosen model under each criterion
• Coefficients and p-values of the selected model
• Residual diagnostics of the selected model

How to report it

An exhaustive best-subset search over [number] candidate predictors selected, by BIC, a [number]-predictor model ([terms]), R2 = [value], adjusted R2 = [value].

Related methods

• Stepwise Regression
• Lasso Regression
• Multiple Linear Regression

Run Best Subset Regression on Quorant