Exposing the Legate C++ API through Cython

[MatthiasKohl]

I'm not sure whether this is related to issue #11 but it would be good to provide an easier interface to handle existing C-level data in python, in particular for regions, but possibly other objects, too.

In particular, if we have a handle to a region in C, it's difficult to create a corresponding legate.core.legion.Region object:

1. I believe there is no way right now to get the fields of a field space (this is simply lacking from Legion's C interface AFAICT)
2. The legate.core.legion.FieldSpace class would have to support being created with a handle, like the Region and IndexSpace classes atm

Without this, I'm not sure if/how you can use the __legate_data_interface__, unless you create the underlying C objects on the python side.

[manopapad]

I have added support for initializing a FieldSpace object from an existing handle, it should be ready to use once https://gitlab.com/StanfordLegion/legion/-/merge_requests/387 and #13 are merged. Please try it out once possible and let me know if these changes are sufficient.

[manopapad]

One comment that came up during the review of the above changes is that the existing FieldSpace class assumes it's the only code that will be allocating fields in the managed field space, so passing it an externally-created field space handle is potentially unsafe. We might want to create a different class for this use-case.

How exactly are you getting the field space handle, and what do you want to do with it?

Why do you need to list the fields in the field space?

[MatthiasKohl]

Sorry for the late response, I had an issue with the mail associated with this account.

How exactly are you getting the field space handle, and what do you want to do with it?

It is created by a separate module in C++ which the python code doesn't have control over. But the python code should take ownership of that field space.

Why do you need to list the fields in the field space?

I don't need this at all actually, but ultimately, I want to be able to obtain a Region object in python (which should work with legate.numpy) from a LogicalRegion object that was created by another module in C++ and that the python code is taking ownership of.

So the only functionality I need in python is that the region object should be compatible with the current Region object and the resources should be released when deleting the object.

[lightsighter]

Sorry for the slow response on this. One dimension we should at least consider here is whether this could be improved by using Cython which is something we've been considering to improve our interfaces for calling from Python to C. It might make transitioning back and forth between C/C++ and Python in both directions easier as well.

[MatthiasKohl]

I think Cython would be useful here if it allows to directly call the Legion C++ API.
OTOH, we'd still need a python class for field space that takes ownership of an external handle, rather than assuming it was created by the class.
This might be easier to implement with Cython though.

[lightsighter]

I think Cython would be useful here if it allows to directly call the Legion C++ API.

It should definitely be able to do that.

This might be easier to implement with Cython though.

@MatthiasKohl can you do some investigation on the side to see how we might use Cython with Legate? I'm far from an expert, so it would be good to know what is and isn't possible and make sure we're not just attributing the ability to do things to Cython that it really can't handle.

[MatthiasKohl]

@MatthiasKohl can you do some investigation on the side to see how we might use Cython with Legate?

Sure, will do. I remember I already looked into this last year once, but I ran into some issues due to templates, but I didn't try for long. Let's see if I can work around that.

[lightsighter]

If we need to cut down the Legion C++ API in some way (e.g. with macros to mask off some parts) to get what we need through Cython, then we can do that. The Legion C API though should be able to go through Cython without any trouble though I would hope.

[MatthiasKohl]

After some time investigating this, I can see the following arguments against using Cython:

• You need to re-declare the whole header file to make it usable to cython (using cdef declarations). This is not needed by CFFI
• It's probably possible to either write our own tool to speed up the process above or use a tool to help with this process, like xdress, but both still take significant effort: xdress is out-dated, there aren't many other tools, see here and writing a custom tool obviously takes a lot of effort, too.
• In my opinion, the only way to go is to manually re-create the wrappers. I'm OK with doing this, but it will probably take a few weeks of work to do this along with creating the new top-level task and some low-level management like in legion_top.py currently. Plus, once this is done, everthing in legate core and the libraries needs to be switched over to the new interface to make it somewhat usable end-to-end.

[lightsighter]

I'm OK with doing this, but it will probably take a few weeks of work to do this

Definitely don't do this :) At least for now. We need to talk about whether/how to do this and we should consult some Cython experts. I suspect the answer might be we Cythonize only the C API and not the C++ API. I do think we'll need Cython eventually for performance reasons, but it's not strictly necessary at the moment.

Also, I think we'll probably need to do this for the Legate Core C++ implementation as well. We might be able to do some things there to make that interface more amenable to being Cythonized.

Plus, once this is done, everthing in legate core and the libraries needs to be switched over to the new interface to make it somewhat usable end-to-end.

I think we should be able to keep the same Python types in the core interface that we already have so the libraries don't need to change. The implementation of the types in the core will obviously be different since we're not using the CFFI anymore, but everything else will be the same.

[manopapad]

With #13 merged we now support initializing a FieldSpace object from a legion_field_space_t handle (with reduced capabilities than if the FieldSpace had been constructed in Legate), so that should be enough to allow you to create a Region object for a region allocated in C++.

Keeping the issue open to track the discussion on Cython-izing the Legate API.