How to further increase call stack depth into mpi4py?

[bcm0]

I got this warning

Reached callpath depth was 71.
Consider setting SCOREP_PROFILING_MAX_CALLPATH_DEPTH to 71.

So I set export SCOREP_PROFILING_MAX_CALLPATH_DEPTH=71

Unfortunately the call stack ends with the last user defined python function.
Inside this last function I call further MPI functions but they don't show up in cube.

Is there a way to show them? Maybe even the internals of mpi4py?

Offtopic: Do you have any advice on line-profiling with mpi4py? I'd like to even see dict lookup times.

[AndreasGocht]

Hey,

Unfortunately the call stack ends with the last user defined python function.
Inside this last function I call further MPI functions but they don't show up in cube.

This might be related to the way, mpi4py does MPI communication. It might be, that mpi4py spawns threads doing the MPI communication. This might be a different location. Can you provide me the .cubex file?

Please also consider using Vampir and tracing. This might help.

Maybe even the internals of mpi4py?

you might be able to build mpi4py using Score-P. However, I never tried it.

Offtopic: Do you have any advice on line-profiling with mpi4py? I'd like to even see dict lookup times.

If you have some specific code lines, you might want to use some user instrumentation:

def fun():
    # some code
    with scorep.user.region("my dict"):
         my_dict["key"] = my_heavy_object()

However, we do currently not support line profiling as python -m trace does (mostly because it will create an enormous overhad). However, if you do have a specific use case and a sample code, it might be worth to implement an instrumenter to do so.

Best,

Andreas

[bcm0]

For debugging it could be helpful but in the meantime I realized I could just use python's line_profiler which goes one step further although it does not track MPI communication.
Thank you.

[AndreasGocht]

Interesting ^^. However, thanks for the feedback.

[bcm0]

Let's forget mpi4py. What about numpy? Do you have some information for either of the following?

• scorep-libwrap-init for numpy
• building numpy using scorep

[AndreasGocht]

Hey,

I have never worked with scorep-libwrap-init but I think the usage is documented in the Score-P doc [perftools.pages.jsc.fz-juelich.de/cicd/scorep/docs/scorep-6.0/pdf/scorep.pdf]

I just checked for building numpy with python. It does not seem that easy. The requirements are listed here:

https://github.com/numpy/numpy/blob/v1.20.1/INSTALL.rst.txt

However, when pip tries to check the Fortran version, Score-P Issues a warning:

$ scorep-gfortran -dumpversion
[Score-P] WARNING: Found no input files
9

An due to the way, pip handles stderr link the version check fails link.

I'll have to have a closer look at this.

[AndreasGocht]

I'll track this in a new issue.

[bertwesarg]

I have never worked with scorep-libwrap-init but I think the usage is documented in the Score-P doc [perftools.pages.jsc.fz-juelich.de/cicd/scorep/docs/scorep-6.0/pdf/scorep.pdf]

just call scorep-libwrap-init --help