Gradient Boosting

Gradient boosting is a machine learning method that builds an ensemble of shallow decision trees sequentially, fitting each new tree to the errors of the current ensemble, for classification or regression.

그래디언트 부스팅은 얕은 의사결정나무를 하나씩 순차적으로 추가하면서 각 나무가 현재 앙상블의 오차를 학습하도록 하는 분류·회귀용 기계학습 방법이다.

How can I build an accurate prediction model for tabular data with nonlinear effects and interactions?

비선형 효과와 상호작용이 있는 표 형태 데이터로 정확한 예측 모형을 만들려면 어떻게 하나요?

Gradient boosting adds small decision trees one at a time, each correcting the remaining errors of the model built so far, which lets it capture nonlinear relationships and interactions without specifying them in advance. Its performance should be judged on held-out data through cross-validation and compared with a simple baseline, because training-set accuracy is overly optimistic.

When to use Gradient Boosting

• Predicting a binary, multiclass or continuous outcome from many tabular features
• Relationships that are likely nonlinear or involve interactions
• Prediction accuracy matters more than interpretable coefficients

When to use something else

• Estimating and testing the effect of specific variables: use multiple linear or binary logistic regression
• Very large datasets where training speed matters: use LightGBM or XGBoost

Assumptions and requirements

• Independent observations, or a cluster or time variable declared so that cross-validation respects that structure
• Enough observations for cross-validated scores to be stable
• Default settings (100 trees of depth 3, learning rate 0.1) are adequate, since no hyperparameter tuning is performed

What the analysis reports

• Cross-validated primary score (ROC AUC for binary, accuracy for multiclass, R² for regression) with mean and SD across folds
• Additional metrics such as balanced accuracy, F1, log loss, RMSE or MAE
• Comparison with a dummy baseline and a histogram-based gradient boosting baseline
• Training score for judging overfitting
• Feature importance (permutation on held-out folds; impurity-based when there are more than 60 features)

How to report it

A gradient boosting [classifier] achieved a cross-validated [ROC AUC] of [value] (SD = [value]) across [k] folds, compared with [value] for a baseline model.

Related methods

• XGBoost
• Random Forest
• LightGBM

Run Gradient Boosting on Quorant