Principal Component Regression

Principal component regression regresses the outcome on the first few principal components of the standardized predictors, reducing collinearity by discarding low-variance directions.

주성분 회귀는 표준화한 설명변수의 앞쪽 주성분 몇 개로 결과를 회귀해 공선성을 줄이는 방법입니다.

My predictors are highly collinear; can I summarise them into principal components and use those in a regression?

설명변수끼리 공선성이 심한데, 주성분으로 요약해서 회귀에 써도 되나요?

Principal component regression replaces correlated predictors with their leading principal components and regresses the outcome on those components, with the number chosen by cross-validation. Because components are chosen by predictor variance alone, the leading components are not guaranteed to be the ones most related to the outcome.

When to use Principal Component Regression

• Predictors are strongly correlated and coefficient estimates are unstable
• You want a lower-dimensional summary of predictors before regression
• Prediction accuracy matters more than individual coefficients

When to use something else

• The outcome depends on low-variance directions of the predictors; use partial least squares regression instead
• You need individual coefficients with valid standard errors; use multiple linear regression instead

Assumptions and requirements

• The outcome is linearly related to the retained components
• Predictors are standardized internally, so results do not depend on their units
• Important information lies in the high-variance directions of the predictors

What the analysis reports

• Number of components (up to 10) selected by cross-validation
• Cross-validated RMSE and Q²
• Variance of the predictors explained by retained components
• Coefficients transformed back to the original predictors
• Scree of predictor eigenvalues

How to report it

A principal component regression retained [value] of [value] components (explaining [value]% of predictor variance) by 10-fold cross-validation, with cross-validated RMSE = [value] and Q² = [value].

Related methods

• Partial Least Squares Regression
• Principal Component Analysis (PCA)
• Regularised Regression (Ridge, Lasso, ElasticNet)

Run Principal Component Regression on Quorant