Styjun

I am using sklearn GridSearch to find best parameters for random forest classification using a predefined validation set. The scores from the best estimator returned by GridSearch do not match the scores obtained by training a separate classifier with the same parameters.

The data split definition

X = pd.concat([X_train, X_devel])
y = pd.concat([y_train, y_devel])
test_fold = -X.index.str.contains('train').astype(int)
ps = PredefinedSplit(test_fold)

The GridSearch definition

n_estimators = [10]
max_depth = [4]
grid = 'n_estimators': n_estimators, 'max_depth': max_depth

rf = RandomForestClassifier(random_state=0)
rf_grid = GridSearchCV(estimator = rf, param_grid = grid, cv = ps, scoring='recall_macro')
rf_grid.fit(X, y)

The classifier definition

clf = RandomForestClassifier(n_estimators=10, max_depth=4, random_state=0)
clf.fit(X_train, y_train)

The recall was calculated explicitly using sklearn.metrics.recall_score

y_pred_train = clf.predict(X_train)
y_pred_devel = clf.predict(X_devel)

uar_train = recall_score(y_pred_train, y_train, average='macro')
uar_devel = recall_score(y_pred_devel, y_devel, average='macro')

GridSearch

uar train: 0.32189884516029466
uar devel: 0.3328299259976279

Random Forest:

uar train: 0.483040291148839
uar devel: 0.40706644557392435

What is the reason for such a mismatch?

There are multiple issues here:

1. 
   

   Your input arguments to recall_score are reversed. The actual correct order is:

   
   
   

   recall_score(y_true, y_test)
   

   
   
   

   But you are are doing:

   
   
   

   recall_score(y_pred_train, y_train, average='macro')
   

   
   
   

   Correct that to:

   
   
   

   recall_score(y_train, y_pred_train, average='macro')
   

   
   



2. You are doing rf_grid.fit(X, y) for grid-search. That means that after finding the best parameter combinations, the GridSearchCV will fit the whole data (whole X, ignoring the PredefinedSplit because that's only used during cross-validation in search of best parameters). So in essence, the estimator from GridSearchCV will have seen the whole data, so scores will be different from what you get when you do clf.fit(X_train, y_train)

It's because in your GridSearchCV you are using the scoring function as recall-macro which basically return the recall score which is macro averaged. See this link.

However, when you are returning the default score from your RandomForestClassifier it returns the mean accuracy. So, that is why the scores are different. See this link for info on the same. (Since one is recall and the other is accuracy).