Random Forest in R : Step by Step Tutorial

Live Online Training : Data Science with R

- Explain Advanced Algorithms in Simple English
- Live Projects
- Case Studies
- Job Placement Assistance
- Get 10% off till Oct 26, 2017
- Batch starts from October 28, 2017

This tutorial explains about random forest in simple term and how it works with examples. It includes step by step guide of running random forest in R. Also, it highlights the explanation of parameters used in random forest R package.

Background

Random forests are a way of averaging multiple deep decision trees, trained on different parts of the same training set, with the goal of overcoming over-fitting problem of individual decision tree.

In other words, random forests are an ensemble learning method for classification and regression that operate by constructing a lot of decision trees at training time and outputting the class that is the mode of the classes output by individual trees.

What is overfitting?
Explaining your training data instead of finding patterns that generalize is what overfitting is. In other words, your model learns the training data by heart instead of learning the patterns which prevent it from being able to generalized to the test data. It means your model fits well to training dataset but fails to the validation dataset.
Random Forest Explained with R
Decision Tree vs. Random Forest

Decision tree is encountered with over-fitting problem and ignorance of a variable in case of small sample size and large p-value. Whereas, random forests are a type of recursive partitioning method particularly well-suited to small sample size and large p-value problems.

Random forest comes at the expense of a some loss of interpretability, but generally greatly boosts the performance of the final model.

Popularity of Random Forest Algorithm

Random Forest is one of the most widely used machine learning algorithm for classification. It can also be used for regression model (i.e. continuous target variable) but it mainly performs well on classification model (i.e. categorical target variable). It has become a lethal weapon of modern data scientists to refine the predictive model. The best part of the algorithm is that there are a very few assumptions attached to it so data preparation is less challenging and results to time saving. It's listed as a top algorithm (with ensembling) in Kaggle Competitions.

Can Random Forest be used both for Continuous and Categorical Target Variable?

Yes, it can be used for both continuous and categorical target (dependent) variable. In random forest/decision tree, classification model refers to factor/categorical dependent variable and regression model  refers to numeric or continuous dependent variable.

How random forest works

Each tree is grown as follows:

1. Random Record Selection : Each tree is trained on roughly 2/3rd of the total training data (exactly 63.2%) . Cases are drawn at random with replacement from the original data. This sample will be the training set for growing the tree.

2. Random Variable Selection : Some predictor variables (say, m) are selected at random out of all the predictor variables and the best split on these m is used to split the node.
By default, m is square root of the total number of all predictors for classification. For regression, m is the total number of all predictors divided by 3.
The value of m is held constant during the forest growing.

Note : In a standard tree, each split is created after examining every variable and picking the best split from all the variables.

3. For each tree, using the leftover (36.8%) data, calculate the misclassification rate - out of bag (OOB) error rate. Aggregate error from all trees to determine overall OOB error rate for the classification.

4. Each tree gives a classification on leftover data (OOB), and we say the tree "votes" for that class. The forest chooses the classification having the most votes over all the trees in the forest. For a binary dependent variable, the vote will be YES or NO, count up the YES votes. This is the RF score and the percent YES votes received is the predicted probability. In regression case, it is average of dependent variable.

For example, suppose we fit 500 trees, and a case is out-of-bag in 200 of them:
 - 160 trees votes class 1
-  40 trees votes class 2

In this case, RF score is class1. Probability for that case would be 0.8 which is 160/200. Similarly, it would be an average of target variable for regression problem.

Out of Bag Predictions

In the image below, NA refers to the cases not in OOB sample while growing each tree. Whereas, non-NA values refer to values in OOB sample.

Out of Bag Predictions

Average OOB prediction for the entire forest is calculated by taking row mean of OOB prediction of trees. See the result below -
Average OOB prediction

Important Point
It is because each tree is grown on a bootstrap sample and we grow a large number of trees in a random forest, such that each observation appears in the OOB sample for a good number of trees. Hence, out of bag predictions can be provided for all cases.

What is random in 'Random Forest'?

'Random' refers to mainly two process - 1. random observations to grow each tree and 2. random variables selected for splitting at each node. See the detailed explanation in the previous section.

Important Point :
Random Forest does not require split sampling method to assess accuracy of the model. It performs internal validation as 2-3rd of available training data is used to grow each tree and the remaining one-third portion of training data always used to calculate out-of bag error to assess model performance.
Pruning

In random forest, each tree is fully grown and not pruned. In other words, it is recommended not to prune while growing trees for random forest.

Methods to find Best Split
The best split is chosen based on Gini Impurity or Information Gain methods.

Preparing Data for Random Forest 

1. Imbalance Data set

A data set is class-imbalanced if one class contains significantly more samples than the other. In other words, non-events have very large number of records than events in dependent variable.

In such cases, it is challenging to create an appropriate testing and training data sets, given that most classifiers are built with the assumption that the test data is drawn from the same distribution as the training data.

Presenting imbalanced data to a classifier will produce undesirable results such as a much lower performance on the testing than on the training data. To deal with this problem, you can do undersampling of non-events.

Undersampling
It means down-sizing the non-events by removing observations at random until the dataset is balanced.

2. Random forest is affected by multicollinearity but not by outlier problem.

3. Impute missing values within random forest as proximity matrix as a measure

Terminologies related to random forest algorithm:

1. Bagging (Bootstrap Aggregating)

Generates m new training data sets.  Each new training data set picks a sample of observations with replacement (bootstrap sample) from original data set.  By sampling with replacement, some observations may be repeated in each new training data set. The m models are fitted using the above m bootstrap samples and combined by averaging the output (for regression) or voting (for classification).

2. Out-of-Bag Error (Misclassification Rate)

Out-of-Bag is equivalent to validation or test data. In random forests, there is no need for a separate test set to validate result. It is estimated internally, during the run, as follows:

As the forest is built on training data , each tree is tested on the 1/3rd of the samples (36.8%) not used in building that tree (similar to validation data set). This is the out of bag error estimate - an internal error estimate of a random forest as it is being constructed.

3. Bootstrap Sample

It is a random with replacement sampling method.

Example : Suppose we have a bowl of 100 unique numbers from 0 to 99. We want to select a random sample of numbers from the bowl. If we put the number back in the bowl, it may be selected more than once. In this process, we are sampling randomly with replacement.

4. Proximity (Similarity)

Random Forest defines proximity between two observations :
  1. Initialize proximities to zeroes
  2. For any given tree, apply the tree to all cases
  3. If case i and case j both end up in the same node, increase proximity prox(ij) between i and j by one 
  4. Accumulate over all trees in RF and normalize by twice the number of trees in RF
It creates a proximity matrix (a square matrix with 1 on the diagonal and values between 0 and 1 in the off-diagonal positions).Observations that are “alike” will have proximities close to 1. The closer proximity to 0, the more dissimilar cases are.

Proximity matrix is used for the following cases :
  1. Missing value imputation
  2. Outlier detection

Shortcomings of Random Forest:
  1. Random Forests aren't good at generalizing to cases with completely new data. For example, if I tell you that one ice-cream costs $1, 2 ice-creams cost $2, and 3 ice-creams cost $3, how much do 10 chocolates cost? A linear regression can easily figure this out, while a Random Forest has no way of finding the answer.
  2. If a variable is a categorical variable with multiple levels, random forests are biased towards the variable having multiple levels

The forest error rate depends on two things:

1. The correlation between any two trees in the forest. Increasing the correlation increases the forest error rate.

2. The strength of each individual tree in the forest. A tree with a low error rate is a strong classifier. Increasing the strength of the individual trees decreases the forest error rate.
Reducing mtry ( Number of random variables used in each tree) reduces both the correlation and the strength. Increasing it increases both. Somewhere in between is an "optimal" range of mtry - usually quite wide. Using the oob error rate a value of mtry in the range can quickly be found. This is the only adjustable parameter to which random forests is somewhat sensitive.

How to fine tune random forest

Two parameters are important in the random forest algorithm:
  1. Number of trees used in the forest (ntree ) and 
  2. Number of random variables used in each tree (mtry ).

First set the mtry to the default value (sqrt of total number of all predictors) and search for the optimal ntree value. To find the number of trees that correspond to a stable classifier, we build random forest with different ntree values (100, 200, 300….,1,000). We build 10 RF classifiers for each ntree value, record the OOB error rate and see the number of trees where the out of bag error rate stabilizes and reach minimum.

Find the optimal mtry

There are two ways to find the optimal mtry :
  1. Apply a similar procedure such that random forest is run 10 times. The optimal number of predictors selected for split is selected for which out of bag error rate stabilizes and reach minimum.
  2. Experiment with including the (square root of total number of all predictors), (half of this square root value), and (twice of the square root value). And check which mtry returns maximum Area under curve. Thus, for 1000 predictors the number of predictors to select for each node would be 16, 32, and 64 predictors.

Important Feature : Variable Importance

Random forests can be used to rank the importance of variables in a regression or classification problem.
Interpretation : MeanDecreaseAccuracy table represents how much removing each variable reduces the accuracy of the model.

Calculation : How Variable Importance works
  1. For each tree grown in a random forest, calculate number of votes for the correct class in out-of-bag data.
  2. Now perform random permutation of  a predictor's values (let's say variable-k) in the oob data and then check the number of votes for correct class. By "random permutation of a predictor's values", it means changing the order of values (shuffling).
  3. Subtract the number of votes for the correct class in the variable-k-permuted data from the number of votes for the correct class in the original oob data. 
  4. The average of this number over all trees in the forest is the raw importance score for variable k. The score is normalized by taking the standard deviation.
  5. Variables having large values for this score are ranked as more important. It is because if building a current model without original values of a variable gives worse prediction, it means the variable is important.


Caution

For data including categorical variables with different number of levels, random forests are biased in favor of those attributes with more levels. Methods such as conditional inference trees can be used to solve the problem.


R Code : Missing Data Imputation

There are two ways to impute missing data - option “na.roughfix” and “rfImpute()” in randomForest library. The second approach gives a better result.

The "na.roughfix" option is based on the following algorithm :

1. For numerical variables, NAs are replaced with column medians.
2. For factor variables, NAs are replaced with the most frequent levels (breaking ties at random).
data(iris)
iris.na <- iris
set.seed(111)
## artificially drop some data values.
for (i in 1:4) iris.na[sample(150, sample(20)), i] <- NA
iris.roughfix <- na.roughfix(iris.na)
iris.narf <- randomForest(Species ~ ., iris.na, na.action=na.roughfix)
print(iris.narf)

The "rfImpute()" option  follows a proximity matrix explained above. It gives a better result.
data(iris)
iris.na <- iris
set.seed(111)
## artificially drop some data values.
for (i in 1:4) iris.na[sample(150, sample(20)), i] <- NA
set.seed(222)
iris.imputed <- rfImpute(Species ~ ., iris.na)
set.seed(333)
iris.rf <- randomForest(Species ~ ., iris.imputed)
print(iris.rf)

Outliers

Decision trees are less sensitive to extreme values and hence, random forest.
If you really want to detect outlier in random forest, you can use CORElearn package.

R Code : Random forest based outlier detection

#first create a random forest tree using CORElearn
dataset <- iris
md <- CoreModel(Species ~ ., dataset, model="rf", rfNoTrees=30, maxThreads=1)
outliers <- rfOutliers(md, dataset)
plot(abs(outliers))

#for a nicer display try
plot(md, dataset, graphType="outliers")
destroyModels(md) # clean up

R : Random Forest

Step I : Data Preparation
#Import data
mydata <- read.table("C:/Users/Deepanshu/Desktop/college.txt",sep=",",header=T)

#Explore data
nrow(mydata)
summary(mydata)

#Define variable types
mydata$college_flg<- as.factor(mydata$college_flg)
mydata$Degree_Flg<- as.factor(mydata$Degree_Flg)
mydata$Discipline_flg<- as.factor(mydata$Discipline_flg)
mydata$specialization<- as.numeric(mydata$specialization)
mydata$rating<- as.integer(mydata$rating)

#Install and load packages required for random forest
install.packages("party")
install.packages("randomForest")
install.packages("ROCR")
library(randomForest)
library(ROCR)

Step II : Run the following code several times with different "ntree" values

set.seed(71)
rf <-randomForest(income~.,data=mydata, ntree=200)
print(rf)

Note : If a dependent variable is a factor, classification is assumed, otherwise regression is assumed. If omitted, randomForest will run in unsupervised mode.
Random Forest R Parameters
  1. mtry : number of variables selected at each split, By default, mtry = floor of (sqrt of number of independent variables) for classification model. mtry = floor of (number of variables divided by 3) for regression model.
  2. ntree : number of trees to grow: default = 500
  3. nodesize : minimum size of terminal nodes default =1
  4. replace : To check whether sampling with/without replacement. TRUE implies with replacement. FALSE implies without replacement. TRUE is set by default.
  5. sampsize : Sample size to be drawn from the data for growing each decision tree.  By default, it takes 63.2% of the data
  6. importance: Whether importance metrics of variables to be required.

Step III : Find the number of trees where the out of bag error rate stabilizes and reach minimum.


Step IV : Find the optimal number of variables selected at each split
Select mtry value with minimum out of bag(OOB) error.
It returns the optimal number of mtry (paramter used in randomforest package). Use the number in the code below.
mtry <- tuneRF(mydata[-13],mydata$income, ntreeTry=200,
stepFactor=1.5,improve=0.01, trace=TRUE, plot=TRUE)

best.m <- mtry[mtry[, 2] == min(mtry[, 2]), 1]

print(mtry)
print(best.m)
Arguments

The specification mydata[-13] means “All but the 13th column on the set mydata”. It means all variables except 13th column are independent variables.

As dependent variable, I give the algorithm “mydata$income”, which is the income column on the set mydata.

The ntreeTry specifies the number of trees to make using this function, trying out different numbers for mtry.

The stepFactor specifies at each iteration, mtry is inflated (or deflated) by this value

The improve specifies the (relative) improvement in OOB error must be by this much for the search to continue

The trace specifies whether to print the progress of the search

The plot specifies whether to plot the OOB error as function of mtry

The doBest tells whether to run a forest using the optimal mtry found. If doBest=FALSE (default), it returns a matrix whose first column contains the mtry values searched, and the second column the corresponding OOB error.

Step V : Use the optimal number of variables selected at each split and run random forest again
set.seed(71)
rf <-randomForest(income~.,data=mydata, mtry=best.m, importance=TRUE,ntree=200)
print(rf)
Step VI : Calculate Variable Importance
#Evaluate variable importance
importance(rf)
varImpPlot(rf)
Variable Importance Plot
Higher the value of mean decrease accuracy or mean decrease gini score , higher the importance of the variable in the model. In the plot shown above, GPA is most important variable.

  1. Mean Decrease Accuracy - How much the model accuracy decreases if we drop that variable.
  2. Mean Decrease Gini - Measure of variable importance based on the Gini impurity index used for the calculation of splits in trees. 

# Plot partial dependence of each predictor
par(mfrow = c(3, 5), mar = c(2, 2, 2, 2), pty = "s");
for (i in 1:(ncol(mydata) - 1))
  {
    partialPlot(rf, mydata, names(mydata)[i], xlab = names(mydata)[i],
main = NULL);
  }

R : Calculate Predicted Probability
#Calculate predictive probabilities of training dataset.
pred1=predict(rf,type = "prob")

You can use the same code to validate a dataset (test).
pred1=predict(rf,type = "prob",test)

asd <-cbind(mydata,pred1)
write.table(asd,file="C:/Users/Deepanshu/Desktop/prob.csv",sep=",",row.names=F)

R : Model Performance
#Evaluate the performance of the random forest for classification.
pred2=predict(rf,type = "prob")

#prediction is ROCR function

perf = prediction(pred2[,2], mydata$income)

#performance in terms of true and false positive rates

1. Area under curve
auc = performance(perf, "auc")

2. True Positive and Negative Rate
pred3 = performance(perf, "tpr","fpr")

2. Plot the ROC curve
plot(pred3,main="ROC Curve for Random Forest",col=2,lwd=2)
abline(a=0,b=1,lwd=2,lty=2,col="gray")

#Plot a sample tree
cforest(income~., data=mydata, controls=cforest_control(mtry=2, mincriterion=0))

R Tutorials : 75 Free R Tutorials


Statistics Tutorials : 50 Statistics Tutorials

About Author:

Deepanshu founded ListenData with a simple objective - Make analytics easy to understand and follow. He has close to 7 years of experience in data science and predictive modeling. During his tenure, he has worked with global clients in various domains like retail and commercial banking, Telecom, HR and Automotive.


While I love having friends who agree, I only learn from those who don't.

Let's Get Connected: Email | LinkedIn

Get Free Email Updates :
*Please confirm your email address by clicking on the link sent to your Email*

Related Posts:

24 Responses to "Random Forest in R : Step by Step Tutorial"

  1. Great post - can you explain a bit about how the predicted probabilities are generated and what they represent in a more theoretical sense. I'm using randomForest but getting lots of 1.00 probabilities on my test set (bunching of probabilities) which is actually hurting me as i want to use them the filter out non relevant records in an unbiased fashion for further downstream work. I'm finding that logistic regression has a lot less of this going on. I'm combining the models to try get best of both. But as we usually think a probability of 1.00 can not exist, it's got me thinking about how to interpret probabilities from the RF model. Struggling to find a clear overview anywhere (will spend more time looking later).

    Anyway nice post - adding this blog to my list :)

    ReplyDelete
  2. Very informative - thank you. I'm having trouble going 1 step deeper and actually interpreting the output from the importance(model) command. Applied, to the iris data set, the output looks like this:
    setosa versicolor virginica MeanDecreaseAccuracy MeanDecreaseGini
    Sepal.Length 1.277324 1.632586 1.758101 1.2233029 9.173648
    Sepal.Width 1.007943 0.252736 1.014141 0.6293145 2.472105
    Petal.Length 3.685513 4.434083 4.133621 2.5139980 41.284869
    Petal.Width 3.896375 4.421567 4.385642 2.5371353 46.323415

    Can you please help me understand what the numbers in the first 3 columns are - for example, the number 1.277324 in Row 1/Col 1 (sepal.width & Setosa).

    I know Setosa is one of 3 classes and width is a feature, but, can't figure out what 1.277324 refers to.

    Thank you!

    ReplyDelete
  3. hi, can you please explain the chart which is produced by plotting a RF model using plot function

    ReplyDelete
  4. Lovely post. Everything at one place in a very simple language. I really enjoyed reading the article even late night.

    Keep writing. I am looking forward to read other your posts too.

    Thank you!

    ReplyDelete
  5. Hi, your post is very great!
    I don't understand partialPlot very well.
    Could you explain more about it?

    ReplyDelete
  6. Great post , Got everything needed .

    ReplyDelete
  7. Say My Predictor variables are a mix of Categorical and Numeric. Random forest tells me which Predictors are important. If i want to know which level under the categorical predictor is important , how can i tell ??Do i ned to use other techniques, like GLM ??

    ReplyDelete
  8. Awesome it is..thanks a lot for sharing

    ReplyDelete
  9. Great post, keep writing ...

    ReplyDelete
  10. Can you share dataset you are using here ?

    ReplyDelete
    Replies
    1. me too...the data files are local ... it would be useful to use the same data and run it

      Delete
    2. Hi Writer,

      Please provide us the data as well, that would be really helpful to understand it

      Regards

      Delete
  11. very detailed....

    ReplyDelete
  12. Excellent writing! Thanks for sharing your valuable experience!

    ReplyDelete
  13. Great post! Really appreciate your effort to write down all this. Really helped a lot! It now feels that using R software is much simpler.

    ReplyDelete
  14. Hello,

    Thanks for the tutorial. I am confused about one point. I used 10-fld cross validation and applied the RF model. to chechk the model performance, I did the same thing with you -> perf = prediction(pred2[,2], mydata$income). I got the error "Number of predictions in each run must be equal to the number of labels for each run."
    I made some search but couldnt solve the problem. If you could help me, I would appriciate.

    ReplyDelete
  15. Good Article to understand for Layman. Keep it up.

    ReplyDelete
  16. This comment has been removed by the author.

    ReplyDelete
  17. This comment has been removed by the author.

    ReplyDelete
  18. Great posting!!! very helpful.

    ReplyDelete
  19. "Random Forests aren't good at generalizing to cases with completely new data. For example, if I tell you that one ice-cream costs $1, 2 ice-creams cost $2, and 3 ice-creams cost $3, how much do 10 chocolates cost? A linear regression can easily"
    I think no algorithm can tell the chocolate cost with ice cream cost as an in put....

    ReplyDelete
  20. Really nice post...

    ReplyDelete

Next → ← Prev