Adding supplementary LOIO figure

[gwaybio]

Summarizing LOIO results compiled in #52 into a supplementary figure. The main LOIO figure is in #47

Supplementary Figure 6. Leave-one-image-out (LOIO) analysis reveals poor performance

(A) LOIO results across three feature spaces (CellProfiler [CP], DeepProfiler [DP] and Combined [CP and DP]) compare predicted probability and prediction ranks for correct and incorrect predictions. Correct predictions generally showed higher probabilities within ranks, but many incorrect predictions had high probabilities. (B) Illumination correction (IC) and balanced models only marginally impacted LOIO across phenotypes. The bar represents how many images per phenotype were impacted by adding IC. For example, SmallIrregular phenotypes saw one more image with correct predictions after IC in both the balanced and unbalanced machine learning training approaches.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[jenna-tomkinson]

Nice PR! I left some comments on the figure for you to address prior to merging. Interesting results with IC!

[jenna-tomkinson]

Comments for this figure:

Panel A - This looks really good! I was wondering your thoughts on the Shuffled CellProfiler feature space plot (bottom left), as I find it interesting that it is the only shuffled plot that as low probability with higher rank of prediction. Is this what we expected to see in the other shuffled plots?

Panel B - Is there a reason that you are using the number of images instead of the number of single-cells correctly predicted? Are whole images associated with a phenotypic class or did you aggregate at some point?
Also, I am surprised that this result shows that across the different datasets, 16/45 classes were actually negatively impacted by IC. I am holding up my own red flag here 🚩, but it would be interesting to see if CellProfiler IC would be any different.

Overall, this plot is super clear, just raised a few questions for me.

[gwaybio]

Panel A - This looks really good! I was wondering your thoughts on the Shuffled CellProfiler feature space plot (bottom left), as I find it interesting that it is the only shuffled plot that as low probability with higher rank of prediction. Is this what we expected to see in the other shuffled plots?

Great observation! Yes, this is what we expect to see, but we don't. It is unclear why.

Panel B - Is there a reason that you are using the number of images instead of the number of single-cells correctly predicted? Are whole images associated with a phenotypic class or did you aggregate at some point?

Hmmm, this is a very good question. I think single-cells correctly predicted actually makes more sense... 🤔 I will think on this some more and then maybe make a change. Thanks for the suggestion!

Also, I am surprised that this result shows that across the different datasets, 16/45 classes were actually negatively impacted by IC. I am holding up my own red flag here 🚩, but it would be interesting to see if CellProfiler IC would be any different.

Yes, this was interesting to me as well - The numbers are so close though that my interpretation is that IC has no impact in this case. I agree that studying CP IC would be interesting in this case as well :)

[gwaybio]

I think single-cells correctly predicted actually makes more sense... 🤔 I will think on this some more and then maybe make a change. Thanks for the suggestion!

Roshan will need to confirm, but I believe that there is no way to show this at the single-cell level, since we annotate single cells with different IDs in IC vs. no-IC. The per image per phenotype is the most granular comparison we have. Let's let Roshan confirm in #54 , but I will merge this for now and then will reopen if we hear new information. Thanks again for this question!

[gwaybio]

(I will also add this detail in the supplementary figure legend)