1
$\begingroup$

I was wondering if tuning a seed with cross-validation in order to maximize the performance of an algorithm heavily based on a randomness factor is a good idea or not. I have created an Extra Tree Classifier which performs very bad with basically every seed except the one I found by using grid search. I think this is not a problem because I really don't care about how the conditions were set as long as they classify correctly, therefore I should have the ability to try running the algorithm with different seeds until it works, in order to find the best set of casual conditions for each split. Also, note that the test is done with Leave One Out Cross Validation.

Am I right?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It's definitely an error to select an "optimal" random seed.

If performance depends a lot on the random seed, it means that the the model always overfits, i.e. the patterns used by the model depend on the specific subset used as training data and the performance on the test set is due mostly to chance.

In your scenario, the model doesn't really work better with seed X, it happens that this particular seed leads to good performance on this particular test set.

  • It wouldn't work for another model trained with the same seed on a different subset
  • It wouldn't work for a different test set.

Also I assume that you didn't apply the correct methodology for tuning hyper-parameters, otherwise you would certainly have seen the problem. When applying grid search, the performance of the best parameters (and only these) should be estimated again on a fresh test set (because parameter tuning is a kind of training, so performance on the training set is not reliable).

I think this is not a problem because I really don't care about how the conditions were set as long as they classify correctly,

This is a mistake: the classifier doesn't classify correctly, actually the performance you obtain is not reliable, it's an artifact. Testing the model on a fresh test set is the only way to obtain a reliable estimate of the performance.

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.