I am attempting to use DivNet and can't get divnet() to work when my tables are clustered to a higher percentage similarity. I have attempted clustering to various percentage similarities; 80% similarity results in a matrix with dim(27, 2082).
divnet_output` <- ordered_80p_samples %>%
+ divnet(tuning = "careful", base = "OTU140", ncores = nproc)
> Removing absent taxa!
| | 0%Killed
I have attempted using nproc = 1, 4, 8, 16, and 32 cores. At one and 4 cores, I tend to get the above result with "killed". At 8 through 32 cores there is usually an error message about x, y, z cores not returning data, and it will affect the final calculations. The exact values for x, y, and z will change over time, and usually represent 25%-50% of the allotted cores. I am currently running this on our computing cluster but will be testing the same objects on our workstation PC while I await your reply. The latest specs for the cluster node: 32 3.7 GHz cores, 512GB of memory (8 x 64 GB RDIMM), 10gigabit network card, 480 GB SSD.
If I cluster 20% similarity, dim(27, 63), then everything runs as expected.
To add additional context, I have to physically pick the base OTU since there is no common base. My target is a functional gene, and the samples contain different types of environments. Therefore, one of the best base OTU is present in only 56% of the samples and a mean of 74 observations per sample !=0. I had planned to run the calculations on several bases to see how it influences the results. because visual inspection of my data shows OTUs tend to be present in water or soils and not both for this gene. I have not done any filtering to remove rare OTUs either by requiring the presence in > n samples or > m total observations.
I will update you with any additional troubleshooting findings as they come in.
I am attempting to use DivNet and can't get divnet() to work when my tables are clustered to a higher percentage similarity. I have attempted clustering to various percentage similarities; 80% similarity results in a matrix with dim(27, 2082).
I have attempted using nproc = 1, 4, 8, 16, and 32 cores. At one and 4 cores, I tend to get the above result with "killed". At 8 through 32 cores there is usually an error message about x, y, z cores not returning data, and it will affect the final calculations. The exact values for x, y, and z will change over time, and usually represent 25%-50% of the allotted cores. I am currently running this on our computing cluster but will be testing the same objects on our workstation PC while I await your reply. The latest specs for the cluster node: 32 3.7 GHz cores, 512GB of memory (8 x 64 GB RDIMM), 10gigabit network card, 480 GB SSD.
If I cluster 20% similarity, dim(27, 63), then everything runs as expected.
To add additional context, I have to physically pick the base OTU since there is no common base. My target is a functional gene, and the samples contain different types of environments. Therefore, one of the best base OTU is present in only 56% of the samples and a mean of 74 observations per sample !=0. I had planned to run the calculations on several bases to see how it influences the results. because visual inspection of my data shows OTUs tend to be present in water or soils and not both for this gene. I have not done any filtering to remove rare OTUs either by requiring the presence in > n samples or > m total observations.
I will update you with any additional troubleshooting findings as they come in.