Proposed fix to improve pybind11 dynamic type casting

[The9Cat]

Problem

The pyEXP.coefs.setMatrix() routines fail to cast ndarray with complex type to Eigen::MatrixXd. Not all compiles demonstrated the problem. But recent compiles in the Docker container are affected.

Proposed fix

Use Eigen::Ref<Eigen::ComplexXcd> to pass arrays in setMatrix() rather than const Eigen::MatrixXcd&. Tests show that this fixed the issue.

To Do
The implementation has been verified on the failing case but more testing is needed.

[The9Cat]

Changed the binding interface to allow both copies and references. This makes implies extra copies of the setMatrix() data blocks, but probably a worthwhile trade off for ease of use.

[The9Cat]

To Do

• Confirm that these changes compiling correctly and solve the setMatrix() issues both natively and in the Docker container.
• Consider adding a regression test to CMake tests

[michael-petersen]

I've been working with this branch for two weeks, and no workflows have been affected. I wasn't demonstrating the problem originally, so as long as that is reported fixed, I'm happy.

[michael-petersen]

Suggested change

	initializtion grid
	initialization grid

[The9Cat]

Good catch, thanks!

[michael-petersen]

Suggested change

	initializtion grid
	initialization grid

[The9Cat]

Good catch, thanks!

[michael-petersen]

Suggested change

	initializtion grid
	initialization grid

[The9Cat]

Ditto!

[The9Cat]

I have now added CTest pyEXP exercises for setMatrix() and createFromArray(). It seems that this bug bit Sóley while running a test notebook from Nico. Ooops.

[The9Cat]

Summary for this final (hopefully) commit

• Fix a typo in the test dependencies for expNbodyTest (erroneously called makeNbodyTest in two locations)
• Add a Python script to read a coefficient file, use setMatrix() to zero all odd-order coefficients and confirm that the new coefficieint object has zero odd-order coefficients.
• Add a Python script to read a body file and use createFromArray() to make a coefficient structure and add it to a coefficient object.
• Checked that all tests succeed

Probably good to go now...

[michael-petersen]

Great addition of tests!

The tests pass locally for me, but why are they failing in CI? That nothing is getting printed in Halo/createCoefs.py suggests to me the failure is early on.

[michael-petersen]

I guess the failure in the CI tests must be coming from this line -- but I can't reproduce it.

[The9Cat]

That was inserted to try to bisect the error. Thought I removed that.

Not sure what to do here; if we can't reproduce the error except in the runner where we can't debug it. Not really a solution, but the test seems to work fine on native Ubuntu builds, in the Docker container, and for you with Clang, right?

I'm going to label this test "long" and merge to main unless you have another idea?

[The9Cat]

Okay, I've figured it out, and fixed the script. In short, the Python numpy array objects need to persist. Even though I got rid of the Eigen::Ref which was hosing the interface in some builds, the data is passed by reference but if the data is wiped by an assignment, the memory gets stomped on access from the C++ side. To test, I changed the script to 'hold' the instance by giving it a unique name, and the script now works in CI.

However, this could lead to Python users writing code that breaks, depending on their workflow. So now, the question is: should we eliminate pass by reference for setMatrix() and createFromAarray() and force copies? I'm sort of thinking, yes?

[michael-petersen]

I think the use case of a non-unique name is probably going to happen, so it makes sense to eliminate the pass by reference. The copies can't be that large (at least compared to RAM, I hope), so probably no harm in the copying strategy.

[The9Cat]

Summary

Linked the bug in the github CI runner to a pybind11 issue

Details

I built the act local github runner generator with a tweaked build.yml that built with with debugging symbols and included gdb. When the code died in the same way (with a seg. fault), I logged into the docker image and fired up gdb. I was able to trace the failure to a pybind11 wrapper for an eigen3 object that was pointing at a bad address. I then updated the pybind11 git submodule and that fixed the problem! Still not sure why it failed in this particular build and not others, but I feel more comfortable that the native local builds are consistent with the github CI at this point.

To Do

I suppose we should make sure that nothing else is broken but that does seem unlikely at this point. Then I'm hoping we can merge.

[The9Cat]

Another (very minor) question is: do we leave the build.yml as it is current (with the 4 builds) or scale it back? Currently, I believe that all four are identical: ubuntu-latest is ubuntu-22.04 and openmpi will use gcc for mpicc anyway, so I don't think those are probing anything new. At some point, ubuntu-latest will be ubuntu-24.04, I assume. Then that might be useful since ubuntu-22.04 images are still used in the wild.

[michael-petersen]

I think we want some sort of matrix; currently ubuntu-latest is 24.04 (at least according to this log), but I agree with you on mpicc. One option would be to replace that with clang, but I imagine that would fail unless the rest of the environment was built against clang?

[The9Cat]

The log file from the ubuntu-latest run shows 22.04. But, I assume that it will be 24.04 at some point. I'll try the clang tag in act and see what happens...I think recent clang versions should work on straight Ubuntu.

[The9Cat]

Summary of last bunch of commits

• Updated build.yml to run gcc and clang C and C++ compiler pairs using include and exclude stanzas
• Updated the top-level CMakeLists.txt to eliminate the previous default setting of the clang++ -stdlib=... flag. This should be set, if needed, by the local build configuration.
• Fixed the missing C++ override decoration to eliminate chatty warnings from clang++. Some additional non-standard (according to clang++) grammar can be fixed later. The standard says that sizes must be known at compile time but most compilers allow this. At this point, there is no clean workaround. We could use combinations of std::vector<> and auto array = std::make_unique<int[]>(size); for this where size is a variable if we really want to comply. I have a commit for this stuff, but I'm reluctant to push that here without explicit testing.

[The9Cat]

Looks like we are good to go. All merged and checked.