About Accuracy

[Zoe-Wan]

Hi, thank you for your implementation! I wonder that if you get the same accuracy on gwilliams2022 dataset as the paper said? I'm trying to re-train the model, but it is overfitting all the time! It really confuses me because I find that I have the same preprocess pipeline as the paper(and as yours).
And by the way, have you read the author's newest paper BRAIN DECODING: TOWARD REAL-TIME RECONSTRUCTION OF VISUAL PERCEPTION ? I failed to re-train that model either. :(

[SeanNobel]

Hi!
Yes, our implementation also overfits very much and is completely failing to reproduce their results.
Recently I'm working on image generation and have implemented their new work as my baseline, but that's also partially failing (they report about 11% top-5 accuracy on large test set with CLIP-Vision, but mine only 3%, where chance is 0.21%).
I will report here if I find anything that I've missed from the paper regarding reproduction.

[Zoe-Wan]

Oh, Sorry to hear that 😢 ! What about the performance of your model on small test set ? (they report 70%, but I only get 20%-25%) Have you reproduce the results?

[SeanNobel]

I haven't tested it on small test set, but I think I will get similar result as yours.
A colleague who works on a similar project also says that he only achieves 1/4 ~ 1/3 test accuracy of the paper.
Hmm, interestingly we are failing to reproduce with reproducibility 🤔

[Zoe-Wan]

So sad! I think we'll have to wait for the author to public the code.
And can I have the code you and your colleague implements about the newest paper? Maybe there are something I have not noticed.
My email is zoewan235711@gmail.com. Looking forward to hearing from you

[SeanNobel]

As the re-implementation is really integrated in the repository of my own study, I can't pass you the re-implementation part immediately but let me think of updating this repository or some other way.

[Zoe-Wan]

Thanks, you really help me a lot, and looking forward to your updates!

[Zoe-Wan]

I think I find something wrong in the preprocess pipeline. I think the author said "use Robust scaler to center the data " means using all the batched training data of each recording to fit the scaler(for each recording), and using this scaler to transform both the training data and the valid data. But you seem to fit the scaler every batch, but not every recording. I'll report the result after modifying this mistake if it works.

[SeanNobel]

Thanks, I'm almost forgetting but I think the implementation used to be like what you describe, which didn't work so I'm trying centering within batch, but anyways I'm looking forward to your results.

By the way you said you've implemented the preprocessing by yourself, but have you also implemented the model yourself? If you're using my implementation of the model, I could suspect that it's the model that might have a bug.

One more thing to add is, I used the preprocessed version of THINGS-MEG dataset (derivatives/preprocessed/preprocessed_P*-epo.fif) for reproduction as it was handy and I didn't think it would make much difference, but I should redo the preprocessing if your finding turns out to make a difference.

[Zoe-Wan]

Yes, I have implemented the model myself. But I do not find any essential difference between our implement( except some GELU functions, but I don't think it is the matter here). I'm curious about what differences there are in the implementation of our article on image generation.
I did experiments on both the THINGS-MEG‘s official preprocessed data set and the raw data set of two custom preprocessing pipelines (scaled during batch and scaled during recording). But None of them works, I think you are right, it really makes no difference.

[SeanNobel]

@Zoe-Wan Hey, how's it going with the reproduction?

I'm extracting the reproduction part of the Meta paper from my project, currently CLIP training with CLIP-Vision is working (top-5 accuracy for large test is about 3% as we talked before).
https://github.com/SeanNobel/image-decoding

Hope we'll figure out what's missing soon.