1
$\begingroup$

I am trying to determine which model result is better. Both results are trying to achieve the same objective, the only difference is the exact data that is being used. I used random forest, xgboost, and elastic net for regression. Here is one of the results that has low rmse but not so good r2

model n_rows_test n_rows_train r2 rmse rf 128144 384429 0.258415240861579 8.44255341472637 xgb 128144 384429 0.103772500839367 9.28116624462333 e-net 128144 384429 0.062460300392487 9.49266713837073

The other model run has a higher r2 but not so good rmse relative to the standard deviation.

n_rows_train n_rows_test metric_col model rmse r2 37500 12500 3 year appreciation e-net 62.3613393228877 0.705221446139843 37500 12500 3 year appreciation rf 52.0034451171835 0.795011617995982 37500 12500 3 year appreciation sgd 1952637950501.17 -2.89007070463773E+020 37500 12500 3 year appreciation xgb 50.3263561914699 0.808019998691306

Which one is better?

Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
2
+50
$\begingroup$

Another way to approach the problem is to take all of the trained models and compare each of their performances on the same hold-out dataset. This is the most common way to evaluate machine learning models.

Choosing the evaluation metric to use depends on the goal of the project. Most machine learning projects care about predictive ability.

R² is not a useful metric for the predictive ability of a model.

RMSE can be a useful metric of predictive ability. However, since the errors are squared is sensitive to the properties of the data. You mention that you are using different data. Those differences in data could impact comparing RMSE across different sources. Comparing different models on the same dataset would be better when using RMSE.

Improve this answer
$\endgroup$
1
$\begingroup$

One interesting note, before proceeding:

The $RMSE$ and $R^2$ values for your problem have a strong negative correlation. Just look at this graph with the first three values:

Errors

The correlation coefficient is coming out to be -0.9999525 considering the first three models only. Now if, others are included, then also it stays the same.

Now coming to your answer, $R^2$ tells us about the % of variance that can be explained by the model. Whereas, $RMSE$ can tell us about how close our predictions were compared to the actual values. $RMSE$ often gives a better estimate about the model but it suffers badly when there are outliers in the prediction.

So, for that reason, I would suggest the following step before taking the final decision:

Compute the $MAD$ (Median absolute deviation) and $MAPE$ (Mean absolute percentage error) and see how are they coming. $MAPE$ also suffers if there are outliers in the model but we can get a better sense of the prediction as it takes the relative %.

In short: I would look at more Statistical Error Estimators before selecting the best model.

In addition, have a look at this paper, which I co-authored:

Prediction of Responses in a Sustainable Dry Turning Operation: A Comparative Analysis

Here, we have used 5 different statistical error estimators to select the best model.

Improve this answer
$\endgroup$
0
$\begingroup$

The model result with higher R2, where the R2 score move towards one, the model performance improves. You must note that R2 score is a metric that tells the performance of your model, not the loss in an absolute sense that how well your model performed.

In contrast, MAE, RMSE and MSE depend on the context whereas the R2 score is independent of context.

For these reasons second model result is better.

Improve this answer
$\endgroup$
1

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.