Random Forests Explained: Build, Use & Evaluate ML Models

Hyperparameter tuning is a meta-algorithm approach to optimizing model architecture parameters that cannot be learned during training. There are two types of parameters: model parameters (learned during training) and hyperparameters (set before training). Three main tuning approaches exist: (1) Grid search exhaustively evaluates all combinations of predetermined hyperparameter values, (2) Randomized search randomly samples hyperparameter combinations and can use probability distributions, (3) Bayesian optimization incorporates prior knowledge about likely optimal values to guide the search. For random forest classifiers, key hyperparameters include criterion (split quality measurement), number of estimators (trees in ensemble), maximum depth, minimum samples for splits, minimum samples per leaf, and max features selection.

Key hyperparameters for random forest include: (1) Number of estimators (trees) - more trees generally improve accuracy but with diminishing returns; (2) Maximum depth - should be limited to prevent overfitting (typically 10-20, not unlimited); (3) Maximum features - number of features considered at each split (default is square root of total features); (4) Minimum samples - minimum samples required at a leaf node. The most commonly tuned parameters are number of estimators and maximum depth. With 200 trees and maximum depth set to 'none' (unlimited), the model can achieve 100% training accuracy but only about 64.7% test accuracy. The randomness in data sampling and feature selection causes different trees to make different errors, which average out in the final prediction.

Hyperparameter tuning finds optimal parameter combinations to improve model performance. Grid search cross-validation systematically evaluates all parameter combinations. For random forest, key hyperparameters include number_of_estimators, max_depth, min_samples_split, min_samples_leaf, and bootstrap. The total number of models trained equals the product of choices for each hyperparameter. Using 5-fold cross-validation on 48 candidates (240 total fits), optimal parameters were found: max_depth=20, min_samples_leaf=2, min_samples_split=2, n_estimators=200, bootstrap=False. Tuning slightly improved performance (accuracy 0.92, F1 0.9191).

Random Forest is an ensemble method that combines multiple decision trees. Each tree votes on the prediction, and the majority vote becomes the final prediction. It reduces overfitting compared to single decision trees and typically achieves better performance. Parameters include 'n_estimators' (number of trees, default 100) and 'random_state' for reproducibility. Hyperparameters are settings configured before training (like number of trees), while model parameters are learned from data. Use Grid Search or Randomized Search to find the best hyperparameter combinations.

This video explains how to optimize Random Forest hyperparameters using GridSearchCV and RandomizedSearchCV techniques. GridSearchCV exhaustively searches through a specified parameter grid by training and evaluating every possible combination, which is effective for small parameter spaces but computationally expensive. RandomizedSearchCV randomly samples a fixed number of parameter combinations from distributions, making it more efficient for large-scale problems with many hyperparameters. The demonstration shows that while GridSearchCV finds the optimal parameters (achieving 88% accuracy on the heart disease dataset), RandomizedSearchCV provides near-optimal results much faster, making it preferable for large datasets or complex models with numerous hyperparameters.

DALEX provides better visualization of feature importance than SHAP by converting outputs to probabilities. LIME has three critical limitations: low fidelity of surrogate models, hyperparameter dependence causing contradictory interpretations, and extreme sensitivity to input changes. Partial dependency plots and accumulated local effects reveal feature-target dependencies and identify problematic patterns. The Interpreted library creates white-box models by decomposing complex models into individual shape functions per feature, enabling inspection of each feature's contribution. This approach can identify non-smooth behavior (like erratic age predictions) and enable corrections that actually improve model performance.

SHAP (SHapley Additive exPlanations) is a feature importance method based on cooperative game theory that assigns each feature an importance value for individual predictions. It is additive (importance values can be summed), interpretable (shows whether high/low feature values correlate with predictions), and consistent (features more relied upon receive higher importance). SHAP values are computed using TreeExplainer for tree-based models and visualized with summary plots showing feature impact.

This comprehensive tutorial covers SHAP (SHapley Additive exPlanations) and Feature Importance for machine learning model interpretability. The instructor explains that SHAP helps explain how each independent feature contributes to a model's prediction, while Feature Importance provides overall contribution values across the entire dataset. Key differences include: SHAP provides per-sample analysis with values that can be positive or negative, while Feature Importance provides overall values that are always positive. The video demonstrates practical implementation using Python, including installing SHAP, creating TreeExplainer objects, and generating SHAP values using the .shap_values() method. The instructor emphasizes that SHAP works primarily with tree-based models like Random Forest Classifier and XGBoost, and is essential for understanding why models make specific predictions.

SHAP uses Shapley values from game theory to explain ML predictions by assigning each feature a contribution value. For tree-based models, TreeExplainer provides fast computation. Force plots visualize contributions: red/pink features push predictions above base value, blue features push below. Summary plots aggregate SHAP values to show feature importance. Advantages include strong theoretical foundation for legal compliance, unified local/global explanation framework, and interactive visualizations. Disadvantages include computational expense (KernelExplainer is slow), limited model support (TreeExplainer only for trees), and ignoring feature dependencies in some cases.

SHAP (SHapley Additive exPlanations) is a game-theoretic approach for explaining model predictions, assigning each feature a value representing its marginal contribution across all possible feature combinations. SHAP values satisfy three key properties: local accuracy (sum equals prediction minus average), missingness (absent features have zero value), and consistency (increased contribution yields increased SHAP value). SHAP values represent how much each feature pushes the prediction away from the average, with positive values indicating upward influence and negative values indicating downward influence. The method works for any model type, including trees, neural networks, and ensemble methods.

Out-of-bag (OOB) error estimation provides a computationally efficient way to estimate model test error without performing separate cross-validation. Since each bootstrap sample uses approximately two-thirds of the data, each observation is out-of-bag about one-third of the time across all bootstrap samples. For each observation, predictions can be generated using only the trees that were trained without that observation (approximately one-third of all trees). For regression, these predictions are averaged; for classification, majority voting is used. The resulting error estimate is virtually equivalent to leave-one-out cross-validation but requires no additional computational effort, making it particularly valuable for computationally intensive models like bagged trees.

Out-of-bag (OOB) samples provide an internal validation method: approximately one-third of observations are excluded from each bootstrap sample. To estimate test error, plug OOB samples into their respective trees and calculate mean squared error (or accuracy for classification). For prediction, input new data into all trees and average/mode results. This eliminates the need for separate cross-validation.

The out-of-bag (OOB) error is an internal validation technique unique to bagging methods that estimates model performance without requiring a separate validation set. Since bootstrap sampling randomly selects observations with replacement, approximately 37% of the training data is not used in any given tree's training process. This unused portion serves as an implicit validation set for that tree. By aggregating errors across all trees for their respective out-of-bag observations, we obtain an unbiased estimate of generalization error. The theoretical basis comes from the probability that any single observation is excluded from a bootstrap sample, which converges to approximately 1 - 1/e ≈ 0.3679 for large n. This technique provides computational efficiency by leveraging the inherent randomness of bootstrap sampling rather than requiring additional validation procedures.

Cross-validation provides reliable model evaluation by dividing data into k folds and averaging performance across iterations. For unstable classifiers like decision trees, bagging significantly improves generalization. Scikit-learn's BaggingClassifier implements bagging with parameters: estimator (base model), n_estimators (number of models, typically 10-100), and max_samples (fraction of data per model, e.g., 0.8). The OOB (Out-of-Bag) score provides internal validation by evaluating predictions on samples not used during training of each base model. This eliminates the need for separate test sets while providing unbiased generalization error estimates.

Out-of-bag (OOB) error provides a computationally free estimate of test error without requiring a separate holdout dataset. Since approximately one-third of observations are out of bag for each bootstrap sample, these observations can be used to evaluate trees trained on bootstrap samples. OOB error is similar to leave-one-out cross-validation and provides an estimate of model performance on unseen data. The OOB error is typically lower than test error when using a separate holdout dataset because the model is trained on more data (100% vs 80% with holdout).