pip install dependencies missing

[Didou09]

@rragona tried to install tofu on a Macbook using a fresh (virgin) conda virtual environment

• The conda package for Mac doesn't exist so he had to use a pip install (using the pip installed via conda)
• pip did not check the tofu dependencies and threw errors at every try

=> It appears we forgot to include a requirements.txt file that pip uses to check the dependencies
=> see this site for details on pip dependencies managing

To try and reproduce : create an empty ne conda package on a mac (with python 3.7, including just the basics like conda, pip).
Then try installing tofu using pip install tofu

=> it should complain about missing dependencies (due to missing requirements.txt)
=>and Ricardo also has an issue because he doesn't have gcc-8 (is it the only solution ? he has another compiler called LLVM)

Also, it seems we forgot openmp in the dependencies for installation

[rragona]

Thanks @Didou09 for your help and for opening this issue.
This is the error I have while trying:

pip install tofu

Collecting tofu
  Using cached https://files.pythonhosted.org/packages/5a/06/cbb85978151b0854bd2d2f022cfad1ff15a77db88593aeb236cf671bcd89/tofu-1.4.1.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /Users/rik/anaconda/envs/tofu/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/51/1mbjm0lj6098n3_rv0g7tfdr0000gn/T/pip-install-lw_tfc9_/tofu/setup.py'"'"'; __file__='"'"'/private/var/folders/51/1mbjm0lj6098n3_rv0g7tfdr0000gn/T/pip-install-lw_tfc9_/tofu/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/51/1mbjm0lj6098n3_rv0g7tfdr0000gn/T/pip-install-lw_tfc9_/tofu/pip-egg-info
         cwd: /private/var/folders/51/1mbjm0lj6098n3_rv0g7tfdr0000gn/T/pip-install-lw_tfc9_/tofu/
    Complete output (18 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/51/1mbjm0lj6098n3_rv0g7tfdr0000gn/T/pip-install-lw_tfc9_/tofu/setup.py", line 151, in <module>
        openmp_installed = not check_for_openmp(os.environ['CC'])
      File "/private/var/folders/51/1mbjm0lj6098n3_rv0g7tfdr0000gn/T/pip-install-lw_tfc9_/tofu/setup.py", line 139, in check_for_openmp
        stdout=fnull, stderr=fnull)
      File "/Users/rik/anaconda/envs/tofu/lib/python3.7/subprocess.py", line 339, in call
        with Popen(*popenargs, **kwargs) as p:
      File "/Users/rik/anaconda/envs/tofu/lib/python3.7/subprocess.py", line 800, in __init__
        restore_signals, start_new_session)
      File "/Users/rik/anaconda/envs/tofu/lib/python3.7/subprocess.py", line 1551, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'gcc-8': 'gcc-8'
    cython version = 0.29.13
    numpy  version = 1.17.3
    cython version = /Users/rik/anaconda/envs/tofu/lib/python3.7/site-packages/Cython/__init__.py
    numpy  version = /Users/rik/anaconda/envs/tofu/lib/python3.7/site-packages/numpy/__init__.py
    ................ checking if openmp installed...
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[lasofivec]

Hey @rragona
I added a requirements.txt for the release that's coming in the next few days.
Also the error regarding gcc-8 should be solved.
But if you want we can chat a bit on how to make it work in the meantime: https://gitter.im/ToFuProject/community

[flothesof]

Hi guys,

Just a thought, but as far as I understand the machinery, requirements.txt is not necessary.

Why? Because setup.py already includes dependencies.

As far as I can tell, the error is due to the heuristic that checks for the existence of the compiler and not to the dependencies.

My guess is that this will not solve any problems but adding the file will confuse us all...
Sorry for the strong opinion, but I really believe this has nothing to do with requirements.

Also: as far as I can tell, it works under Linux without requirements.txt, right? So why should Mac be different? Somethings is weird here.

[flothesof]

Okay, so after reading this again, I have to take some of what I said back. What is the current status of Mac OS as a platform? Are we not recommending to install from source in the docs? And not from pypi?

[lasofivec]

As far as I can tell, the error is due to the heuristic that checks for the existence of the compiler and not to the dependencies.

I talked directly with Didier, apparently they first tried a pip install that failed because it couldn't find the requirements.txt file. They installed this by hand and then got the error mentioned above, which does come from an error from my part in the setup.py that is now corrected in the devel branch. In short: there were 2 problems when installing with pip on Mac, the error message has nothing to do with the requirements file

Are we not recommending to install from source in the docs?

Yes, I also mentioned this to @Didou09. But it seems like installing directly from pip should also be a possibility.

I should also add the requirements.txt is quite standard and updates in github the dependencies. So I see no problem in adding it.

[flothesof]

I'm still thinking something about this is weird. In some of my other projects, I've defined dependencies in the setup.py and made wheels that registered the right dependencies. There was no need to add requirements.txt. I'm puzzled about why it is necessary here. Do you have any explanation why it works with it?

The disadvantage of adding this file I see is that there are now two places where requirements are defined. Hence when changing these, we will run into the risk of forgetting to update. It adds easy to miss complexity. Couldn't it be that it's a false positive?

Final thing I don't understand: pip install did not work at first but by adding local files it started to work? This is a weird way of installing: neither pure pip nor pure from source. For me it looks hacky / I'm thinking I'm missing something.

[flothesof]

So, for the record, I have symptoms when installing from scratch under MacOS, linked to the compiler for the Cython parts.
Following https://tofuproject.github.io/tofu/installation.html#installing-as-a-developer, I have noticed two things:

• pip install -e .[dev] does not work in my shell by pip install -e ".[dev]" does
• it hangs due to gcc throwing an error, which was first resolved by running xcode-select --install but still fails later due to this:

tofu/geom/_vignetting_tools.cpp:610:10: fatal error: 'ios' file not found
#include "ios"

So as of right now, I'm unable to install tofu on my Mac...

Luckily, there seems to be other people running into this, see cython/cython#2694

But at any rate, there seems to be no need to add requirements.txt: I think this is a false positive.

[Didou09]

So:

• What was done: empty conda environment, installing from source (pip). It failed because pip could not find the requirements.txt and dependencies were missing. So dependencies were installed by hand (conda install) and finally pip was retried, it failed this time because of the error in the setup.py, now fixed, that Laura mentioned.

• What you are saying @flothesof is that pip does not need the requirements.txt when you create a pip wheel. But this was not a pip wheel. It was installing from source using pip. We never created pip wheels. Maybe we should though ? (separate issue in that case)

• I agree storing dependencies in separate places is a source of complexity. But it is already the case. The dependencies are in the setup and also in the conda recipe (for creating the conda package).

• I feel that, by principle, installing from source with pip should always be possible and easy. What do you think ?

=> So, although it does introduce a bit of extra complexity, it does seem useful to add the requirements.txt as Laura did for non-developpers who wnat to install on Mac or windows from source using pip.
Complexity is an issue, but tofu being a difficult to install for newbies is a higher-priority issue. If requirements.txt make it easier for them I feel it is the way to go.
Furthermore, the requirements.txt being very short and simple could be automatically generated through Continuous Integration in a near future, what do you think ?

[flothesof]

Status report after discussing this with @Didou09:

• The goal is to get pip install tofu to work on Mac
• there is a distinction between build requirements (Cython, numpy which are imported by setup.py and therefore necessary to install from source) and user requirements (others already present in setup.py, later user in tofu itself like matplotlib, scipy)
• adding requirements.txt seems to solve the build requirements issue (on @lasofivec 's machine) but it's not clear why
• a better hotfix would be to upload a wheel to PyPI obtained using python setup.py bdist_wheel

As a side note, I was able to compile from source on Mac (CFLAGS='-stdlib=libc++' pip install -e ".[dev]") without requirements.txt, which I interpret to show that we don't need it...