Issue running on Ubuntu 20.04 using Python 3.8

[KBerstene]

I'm having an odd issue running lunatic-python-universal on the Windows Subsystem for Linux, specifically running the most recent Ubuntu 20.04 release, which runs Python 3.8.

I seem to have narrowed it down to Python 3.8 being the issue by installing a version of Python 3.6 and being able to run require "python" properly. I have also been able to install and run on another Windows machine with Ubuntu 18.04 on WSL, using Python 3.5.

I have installed using this method. It seems to work if python3 is Python 3.6, but not Python 3.8

apt install python3 python3-dev python3-pip lua5.1 liblua5.1-0-dev pkg-config
pip3 install lunatic-python-universal # I have also built this library from source

#  Link the Lua module with the name ''python.so'' into a path lua will look for it
mkdir -p /usr/local/lib/lua/5.1/
chown -R root:staff /usr/local/lib/lua
chmod -R 2775 /usr/local/lib/lua
ln -s -T /usr/local/lib/python3.8/dist-packages/lua.cpython-38-x86_64-linux-gnu.so /usr/local/lib/lua/5.1/python.so

Attempting to run inside Lua gives me this:

> require "python"
error loading module 'python' from file '/usr/local/lib/lua/5.1/python.so':
        /usr/local/lib/lua/5.1/python.so: undefined symbol: PyFloat_Type
stack traceback:
        [C]: ?
        [C]: in function 'require'
        stdin:1: in main chunk
        [C]: ?

[greatwolf]

Ok, I'm able to reproduce the issue with Python 3.8 on a Linux VM. It looks like the issue is with the python runtime naming. See https://bugs.python.org/issue36844.

Lunatic was previously assuming all python3 runtime had that m suffix. As a fix, I've changed the source to take the name of the runtime through a define PYTHON_LIBRT. CMake will set PYTHON_LIBRT to whatever python runtime it detected and is compiling with when building under Linux.

You may be wondering why Lunatic needs to know the python runtime when under Linux. The reason has to do with dlopen and RTLD_LOCAL being the default flag which causes external modules loaded later to not be able to access symbols. That causes the undefines you're seeing above. To work around it Lunatic tries to load the python runtime first chance it gets using dlopen with RTLD_GLOBAL flag. For python 3.8, lunatic tried to open it under the wrong name so dlopen is going to fail.

The build and test seems to work for me now. See if it fixes the issues for you.

[KBerstene]

Thank you for the quick turnaround and sorry for the delay in my response. I have gotten a chance to attempt to build the project and am running into an issue that may be on my end.

First, I had to install python-is-python3, since I only had python3 installed on my VM. I believe I was able to build last time by editing the makefile and changing the PYTHON=python definition to PYTHON=python3. Not a huge deal, but just a minor note.

I am receiving an error while trying to build using GNU make. Rereading your post, I attempted to run using CMake, but received a different error there. I am not well versed in building from source, so I may be building incorrectly. The errors are listed below.

GNU make (using make)

src/pythoninlua.c: In function ‘luaopen_python’:
src/pythoninlua.c:639:5: error: #error PYTHON_LIBRT must be defined when building under Linux!
  639 | #   error PYTHON_LIBRT must be defined when building under Linux!
      |     ^~~~~
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

CMake (using cmake . inside the unzipped archive of master)

CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
  (Required is at least version "2.7")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindPythonLibs.cmake:310 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:11 (find_package)

Edit: I just tried rebuilding the previous code (commit f17143a) and GNU make successfully built (though obviously with the bug this ticket was opened with), and CMake did not, resulting in the same error posted here.

[greatwolf]

I'm not sure why find_package isn't finding the python installation. You can point CMake to where to find it with these build variables: -DPYTHON_INCLUDE_DIR, -DPYTHON_LIBRARY

Usually the include headers and shared libraries on linux are in /usr/include/* and /usr/lib/* or /usr/lib64/* respectively. See if you find the shared python runtime in there for your install. You can also try locating it using:

ldconfig -p | grep python

Once you find it, try building again with:

cmake -B./build -H. -DPYTHON_INCLUDE_DIR=/usr/include/<your python install version> -DPYTHON_LIBRARY=/usr/lib/<your python installed version>
cmake --build ./build

Side note, CMake also provides -DCMAKE_PREFIX_PATH as a way to add extra paths to search in. This works ok under Windows but not so much under Linux because it dumps installed shared libraries all into one location, eg. if you have multiple python versions installed.

For example, when I was testing under LinuxLite using VirtualBox, I had multiple python versions installed and their respective shared libraries here:

/usr/lib/i386-linux-gnu/libpython2.7.so
/usr/lib/i386-linux-gnu/libpython3.8.so
/usr/lib/i386-linux-gnu/libpython3.3m.so
/usr/lib/i386-linux-gnu/libpython3.4m.so

So I was able to config and build with:

cmake -B./build -H. -DPYTHON_INCLUDE_DIR=/usr/include/python3.8 -DPYTHON_LIBRARY=/usr/lib/i386-linux-gnu/libpython3.8.so
cmake --build ./build

[KBerstene]

I believe I correctly located my python 3.8 install. As suggested, I ran:

cmake -B./build -H. -DPYTHON_INCLUDE_DIR=/usr/include/python3.8 -DPYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.8.so
cmake --build ./build

Build seemed to run successfully. After copying over the python.so, I am able to run require "python" and run python.builtins().print("Hello, world!").

Thank you so much for the help. I don't know why it doesn't like my python directory and library--they seem to be pretty default.

[greatwolf]

Glad to help!