Bootstrap Optimism-Corrected Model Validation

Bootstrap optimism-corrected validation estimates how much a regression model's apparent performance overstates its performance on new data by refitting the model in bootstrap samples and subtracting the average optimism.

부트스트랩 낙관도 보정 검증은 부트스트랩 표본마다 모형을 다시 맞춰, 같은 자료로 잰 성능이 새 자료 성능보다 얼마나 부풀려졌는지 추정하고 빼 줍니다.

How well will my logistic prediction model perform on new patients, not just the data used to build it?

만든 로지스틱 예측모형이 개발 자료가 아닌 새 환자에서는 성능이 얼마나 될까요?

The model is refitted in many bootstrap samples, and for each one the performance in the bootstrap sample minus the performance on the original data is averaged as the optimism. Subtracting it from the apparent C statistic, R-squared, Brier score or calibration slope gives corrected values; the corrected slope is a shrinkage factor.

When to use Bootstrap Optimism-Corrected Model Validation

• Internal validation of a prediction model when a separate validation set is not available.
• Reporting optimism-corrected discrimination and calibration as recommended by TRIPOD.
• Deciding whether coefficients need shrinkage before use.

When to use something else

• Variable selection was done and cannot be repeated inside each bootstrap; the correction will be too small, so nest the selection or use external validation.
• You have a large independent test set; evaluate on it directly.

Assumptions and requirements

• The fixed predictor set is fitted the same way in each bootstrap sample.
• Observations are independent.
• Enough events per parameter for stable logistic fits.

What the analysis reports

• Apparent, optimism and corrected metrics (AUC, Somers' Dxy, Brier, Nagelkerke R2, slope, intercept or R2, MSE)
• Shrinkage factor (corrected calibration slope)
• .632+ estimate of squared prediction error
• Number of usable bootstrap replicates

How to report it

Internal validation with [value] bootstrap resamples (N = [value]) showed that the apparent C statistic of [value] was corrected to [value]; the optimism-corrected calibration slope was [value].

Related methods

• ROC Curve and AUC
• Calibration Assessment for Predicted Probabilities
• Binary Logistic Regression

Run Bootstrap Optimism-Corrected Model Validation on Quorant