can we compare variable importance scores across samples?

[randomgambit]

Hello there!

I am using this pretty nice package and I wonder if variable importance score that I see (using the permutation method) are comparable across samples.

For instance, one can always compare r-square or rmse across samples. But does it make sense to say that the variable with the highest importance score in sample A does not matter because it is still way smaller than the importance score of another variable in sample B?

Does that make sense to you?

Thanks!