Calibration Assessment for Predicted Probabilities

Calibration assessment checks whether predicted probabilities agree with observed event rates, using calibration-in-the-large, the calibration slope, a smoothed calibration curve and summary error measures.

예측확률 보정 평가는 예측한 확률이 실제 사건 발생 비율과 맞는지를 평균 보정, 보정 기울기, 평활 보정곡선과 요약 오차 지표로 확인합니다.

My model predicts a 20% risk; do about 20% of those patients actually have the event?

모형이 위험 20%라고 예측한 환자 중 실제로 약 20%에서 사건이 일어나는지 어떻게 확인하나요?

Logistic recalibration models give the calibration intercept (with the logit of the prediction as an offset) and slope, and a lowess curve of outcome against prediction gives the integrated calibration index and E50/E90. Brier score, observed/expected ratio, Spiegelhalter z and a decile reliability table complete the picture.

When to use Calibration Assessment for Predicted Probabilities

• Externally validating a published or previously developed risk model.
• Checking whether a machine-learning classifier's probabilities can be read as risks.
• Comparing recalibration by Platt scaling or isotonic regression.

When to use something else

• You only need to rank cases; use ROC and AUC.
• Predictions were fitted on the same data; apparent calibration is perfect by construction, so use bootstrap optimism correction.

Assumptions and requirements

• A binary outcome and predicted probabilities between 0 and 1.
• Predictions were made without using the evaluated outcomes.
• Enough events for a stable smoothed curve.

What the analysis reports

• Calibration-in-the-large and calibration slope with SEs
• Observed/expected ratio, ICI, E50, E90 and Emax
• Brier and scaled Brier scores, Spiegelhalter z and Hosmer-Lemeshow
• Decile reliability table and cross-validated recalibration results

How to report it

Calibration of [model] for [outcome] (N = [value]) showed a calibration-in-the-large of [value], a calibration slope of [value], an observed/expected ratio of [value], ICI = [value] and a Brier score of [value].

Related methods

• ROC Curve and AUC
• Bootstrap Optimism-Corrected Model Validation
• Decision Curve Analysis (DCA)

Run Calibration Assessment for Predicted Probabilities on Quorant