dendrograms, correlation and marker genes filtering

[fidelram]

This PR makes more transparent the computation of the hierarchical clustering underlying the dendrograms. Now, by default, the dendrograms are computed based on the PCA using sc.tl.dendrogram.
Also, now is possible to directly plot a dendrogram without any other data:

Since for the computation of the hierarchical clustering, a correlation matrix is computed I also added a visualization for this (mostly borrowing code from https://github.com/deeptools/deepTools). The new plotting function is called sc.pl.correlation

Also I added a function to filter the results from sc.tl.rank_genes_groups based on fold change and fraction of genes that are expressing the gene within and outside the group by categories.

For example,

The first image show the case without filtering.

[falexwolf]

Sorry for the late response! This seems to have come just after I went through the issues last weekend...

It looks great! 😄

Some small notes:

• sc.pl.correlation should be sc.pl.correlation_matrix (there will be other "correlation plots", just think of the typical bivariate scatter plot...)
• sc.tl.dendrogram suggests it is a function that can be generically applied to any hierarchical clustering of observations. We could even have dendrograms of variables, right? I'm fine with putting it into the API with just that generic name, but it would be good to have a .. note:: in the docstring, which states that this does a very specific thing: computing hierarchical clustering on predefined groups using Pearson correlation as a distance metric; I know that this is super standard in the field, but we should nonetheless be very clear about it. In particular as Scanpy grows and we extend its functionality to other methods for grouping observations, structuring their relations (e.g. hierarchical clustering with another distance metric or so, or something that we don't think of at this stage), I fear that people might start to get confused. Even now, they don't know what, for instance, the relation of tl.dendrogram to PAGA is: instead of correlating cluster mediod vectors, PAGA computes the connectivity between clusters in the underlying graph. Also, it is not restricted to a tree. It would be great to have a note like that (I can also put it; also, I wanted to rewrite the PAGA docstring anyways and I'll make a link to tl.dendrogram...).

Thanks again!

[fidelram]

@falexwolf thanks for the feedback. :)

I agree with your comments on the sc.tl.dendrogram. Similar reasoning originally motivated me to separate and expose the implementation of the function. I expect that now, is easier to extend the creation of a correlation matrix to other methods and groupings as you suggest.

Currently, by default sc.tl.dendrogram uses PCA by recycling the function used by sc.tl.neighbors (tools._utils.choose_representation()). Any other embedding in .obsm can be used (as is the case by sc.tl.neighbors. Also, any group of genes can be given as parameter

What tl.dendrogram does not do is to use the underlying network to compute a distance matrix as I think seurat does and apparently you also do in PAGA.

For me, what is important is that the plotting functions get the dendrogram data from .uns and thus the generation of the hierarchical clustering is separated and can be computed by any other method.

[falexwolf]

Great! 😄

What tl.dendrogram does not do is to use the underlying network to compute a distance matrix as I think seurat does and apparently you also do in PAGA.

That is fine: if you just compute distances on the medians/medoids that's a small-scale problem that won't cost any time. If Seurat uses medians, they will not use the underlying network, because they aren't contained in there. If they use medoids, they might use the underlying network, but it will be negligible save in compute time... So don't worry about that...

[fidelram]

I will get back to this as soon as I can. Meanwhile lets keep the PR open.

[falexwolf]

Just let me know when you think this is ready to go!

[fidelram]

I am getting an error elsewhere that I want to revise before submitting a final version. Hopefully tomorrow

[fidelram]

I think it is ok to merge now.

I also updated some of the plotting functions to accept a gene_symbol column:

What is missing is sc.pl.rank_genes_groups and sc.pl.violin any volunteers?

[aopisco]

@fidelram how can I try your sc.pl.correlation_matrix?
great work by the way!

[fidelram]

@falexwolf Can we merge this branch?

[flying-sheep]

Ah, sorry for being in the way here with the unrelated logging changes. Alex is currently a bit ill I learned, which is why he probably didn’t do it yet. I didn’t have time to review the whole thing, but if y’all want I can do that too

[fidelram]

Please go ahead!

…

On Tue, Feb 12, 2019 at 6:41 PM Philipp A. ***@***.***> wrote: Ah, sorry for being in the way here with the unrelated logging changes. Alex is currently a bit ill I learned, which is why he probably didn’t do it yet. I didn’t have time to review the whole thing, but if y’all want I can do that too — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#425 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEu_1RE9LYK4sL6sLFd586y_cpEBQKxwks5vMvzRgaJpZM4Z-M3d> .

[aopisco]

@fidelram @flying-sheep reiterating my interest in trying this out 💯

[falexwolf]

Sorry about the terribly late response. I was super sick for weeks. Thank you so much, @fidelram.

Let's merge it. :)

[fidelram]

Thanks!

[wangjiawen2013]

Thanks a lot. All of these new features are what we need!

I notice that the tutorial has not been updated yet (such as sc.tl.filter_rank_genes_groups( ) and rna velocity function in https://github.com/theislab/scanpy/tree/master/scanpy/tools). I find these features occasionally. Could you add them in scanpy tutorial ?

[fidelram]

Recently, this tutorial was updated with what you need: https://scanpy-tutorials.readthedocs.io/en/latest/visualizing-marker-genes.html You can find the link to that tutorial from https://scanpy.readthedocs.io/en/latest/

…

On Fri, Mar 15, 2019 at 3:34 PM jiawen wang ***@***.***> wrote: Thanks a lot. All of these new features are what we need! I notice that the tutorial has not been updated yet (such as sc.tl.filter_rank_genes_groups( ) and rna velocity function in https://github.com/theislab/scanpy/tree/master/scanpy/tools). I find these features occasionally. Could you add them in scanpy tutorial ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#425 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEu_1S565VMCgOnXCQetV2R6_A1HONPZks5vW69qgaJpZM4Z-M3d> .