Threads and Memory options

[sejmodha]

Hi There

I am using OpenRDP pipeline on Linux for a reasonably large dataset of around 1200 sequences and noticed that all server resources (all threads, memory+ swap) are being utilised by OpenRDP after 3Seq analysis is completed. I could not find any option that allows me to limit the resources being used by the pipeline eg. the number of threads or maximum memory..

It would be great if control options for resources were provided at the top level of the pipeline such that it limits the resources for all tools.

Thanks
Sej

[ArtPoon]

Thanks Sej - sorry about hogging all threads and memory, we'll look into providing options for limiting resource use.

[ArtPoon]

Could you please copy your command line invocation to this issue? Were you only calling the 3Seq method?

[sejmodha]

Here is the abstract command, I am running the entire pipeline with -all
python3 -m openrdp {input} {output} -all

and here are the last few lines captured from standard out...

Finished GENECONV Analysis
Setting Up MaxChi Analysis
Setting Up Chimaera Analysis
Setting Up RDP Analysis
Setting Up Bootscan Analysis
Starting Scanning Phase of Bootscan/Recscan

[ArtPoon]

You mentioned that resource usage went up after the 3Seq analysis completed - do you know if that was associated with running the next method in the suite, i.e., GENECONV? Please bear with me - my former student @kwade4 did the bulk of development for this program so it's slower for me to attend to your issue.

[sejmodha]

No worries at all....any help is very much appreciated.

So far I have got 3s.log, 3s.longRec, 3s.pvalHist, 3s.rec files that are generated by 3Seq.

However, the standard out log captured in a file suggest that GENECONV analysis is finished (see above comment).
Does that log also suggest that it may be the Bootscan/Rescan analysis that may have caused the memory and threads hogging?
Moreover, I found it a bit strange that I am unable to see what step of the process is currently being executed as I can only see the original openrdp command being executed without details of what stage the pipeline is at...

Hope that helps.

Thanks once again.

[darrenpmartin]

Hi All - it is very plausible that bootscan is the resource hog - it scales terribly with sequence number (lots of n x n matrices). To handle >1000 sequences bootscan would need to have some sort of divide and conqueror based downsampling scheme. with 1K sequences amd 100K bootstrap reps at each step bootscan would generate 800Megs worth of distance matrices AT EACH STEP. If it is set up to run with 200 steps it will maybe be trying to store 200 X 800 megs = 150 Gigs worth of distance matrix data. regards darren

…

On Sat, Jan 8, 2022 at 7:47 PM sejmodha ***@***.***> wrote: No worries at all....any help is very much appreciated. So far I have got 3s.log, 3s.longRec, 3s.pvalHist, 3s.rec files that are generated by 3Seq. However, the standard out log captured in a file suggest that GENECONV analysis is finished (see above comment). Does that log also suggest that it may be the Bootscan/Rescan analysis that may have caused the memory and threads hogging? Moreover, I found it a bit strange that I am unable to see what step of the process is currently being executed as I can only see the original openrdp command being executed without details of what stage the pipeline is at... Hope that helps. Thanks once again. — Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEJ3TT7TRKMBSSQDLTROL3UVB2ELANCNFSM5LOPYGPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ArtPoon]

Thanks for the insight @darrenpmartin! Should be straight-forward for us to implement a maximum limit and downsample to deal with large inputs. I'll also check the Python script to see if there are any easy optimizations.

[ArtPoon]

I haven't forgotten about this issue, but I'm scrambling to deal with a bunch of other repositories and the main package author has moved on.

[ArtPoon]

I think first step is to run some basic tests on memory consumption for bootscan.py

[ArtPoon]

Let's start with measuring time and memory complexity for small to large data sets, and then go into optimizing bootscan.py @dpannguyen

[dpannguyen]

Real time and peak memory usage were measured for data set of 3, 10, and 50 sequences as below:

dataset (seq)	method	real time (s)	peak memory usage (MB)
3	threeseq	1.33	88.0
3	geneconv	1.35	75.9
3	maxchi	1.39	76.3
3	chimaera	1.35	76.3
3	bootscan	3.64	81.0
3	siscan	1.29	76.0
3	rdp	1.41	75.9
10	threeseq	2.38	88.5
10	geneconv	2.48	75.9
10	maxchi	12.11	76.9
10	chimaera	7.76	76.7
10	bootscan	38.12	100.1
10	siscan	5.57	76.5
10	rdp	5.62	76.7
50	threeseq	11.19	90.5
50	geneconv	26.48	76.3
50	maxchi	1635.52	78.7
50	chimaera	950.35	80.6
50	bootscan	3387.03	515
50	siscan	551.46	79.6
50	rdp	502.85	84.3

bootscan is definitely the method that takes up the most resource and it scales up significantly as the dataset grows (taking nearly an hour to process 50 sequences in this case). As @darrenpmartin mentioned, we will look into optimizing the code with a divide and conquer method.

[darrenpmartin]

Hi An - How did you manage to run a dataset of 1 sequence? the minimum should be 3. I am also surprised that the runtime and memory usage of siscan is similar to that of rdp, chimaera and maxchi. These latter methods "should" be at least as fast as geneconv. regards Darren

…

On Fri, Feb 18, 2022 at 8:33 AM An Nguyen ***@***.***> wrote: Real time and peak memory usage were measured for data set of 1, 10, and 50 sequences as below: dataset (seq) method real time (s) peak memory usage (MB) 1 threeseq 1.33 75.8 1 geneconv 1.43 75.8 1 maxchi 1.43 75.9 1 chimaera 1.36 75.8 1 bootscan *2.72* *79.9* 1 siscan 1.35 75.7 1 rdp 1.41 75.9 10 threeseq 2.38 88.5 10 geneconv 2.48 75.9 10 maxchi 12.11 76.9 10 chimaera 7.76 76.7 10 bootscan *38.12* *100.1* 10 siscan 5.57 76.5 10 rdp 5.62 76.7 50 threeseq 11.19 90.5 50 geneconv 26.48 76.3 50 maxchi 1635.52 78.7 50 chimaera 950.35 80.6 50 bootscan *3387.03* *515* 50 siscan 551.46 79.6 50 rdp 502.85 84.3 bootscan is definitely the method that takes up the most resource and it scales up significantly as the dataset grows (taking nearly an hour to process 50 sequences in this case). As @darrenpmartin <https://github.com/darrenpmartin> mentioned, we will look into optimizing the code with a divide and conquer method. — Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEJ3TUJLNOHZRL5P2P4WFLU3XR3HANCNFSM5LOPYGPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[dpannguyen]

@darrenpmartin Hi Darren, you were absolutely right. That was my bad, I wrote a short script to run and grab the runtime and memory usage of each of the methods. Since the specs still showed up even after the process got terminated by the error, it accidentally grabbed that for the dataset of 1 as well. I have updated the table above with the new values for 3 sequences.

Regarding the runtime for rdp, chimaera, and maxchi, we can probably have a look at them after bootscan optimization. Thank you!

[darrenpmartin]

Hi An - Are you shelling to geneconv and 3seq? If so then the difference may just be that between c++ and python.

…

On Fri, Feb 18, 2022 at 6:08 PM An Nguyen ***@***.***> wrote: @darrenpmartin <https://github.com/darrenpmartin> Hi Darren, you were absolutely right. That was my bad, I wrote a short script to run and grab the runtime and memory usage of each of the methods. Since the specs still showed up even after the process got terminated by the error, it accidentally grabbed that for the dataset of 1 as well. I have updated the table above with the new values for 3 sequences. Regarding the runtime for rdp, chimaera, and maxchi, we can probably have a look at them after bootscan optimization. Thank you! — Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADEJ3TQGFX57LFEPRCS7OZTU3ZVG7ANCNFSM5LOPYGPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>