Tensorflow 2.4 on mac

[wilsonmr]

cc: @voisey @scarlehoff @Zaharid

as mentioned in #1039 and #1047 the package for tensorflow on mac is out of date. It was suggested to try installing tensorflow through pip in the former PR and I thought I'd open an issue where I dump various bits of info whilst trying this.

After conda install nnpdf --only-deps and conda remove tensorflow-eigen I then did pip install tensorflow and got the following log

$ pip install tensorflow
Collecting tensorflow
  Downloading tensorflow-2.4.0-cp37-cp37m-macosx_10_11_x86_64.whl (175.4 MB)
     |████████████████████████████████| 175.4 MB 2.3 MB/s 
Requirement already satisfied: numpy~=1.19.2 in ./conda/envs/piptf/lib/python3.7/site-packages (from tensorflow) (1.19.2)
Requirement already satisfied: wheel~=0.35 in ./conda/envs/piptf/lib/python3.7/site-packages (from tensorflow) (0.36.2)
Requirement already satisfied: six~=1.15.0 in ./conda/envs/piptf/lib/python3.7/site-packages (from tensorflow) (1.15.0)
Collecting gast==0.3.3
  Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
Collecting absl-py~=0.10
  Downloading absl_py-0.11.0-py3-none-any.whl (127 kB)
     |████████████████████████████████| 127 kB 2.5 MB/s 
Collecting astunparse~=1.6.3
  Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting flatbuffers~=1.12.0
  Downloading flatbuffers-1.12-py2.py3-none-any.whl (15 kB)
Collecting google-pasta~=0.2
  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 3.3 MB/s 
Collecting grpcio~=1.32.0
  Downloading grpcio-1.32.0-cp37-cp37m-macosx_10_9_x86_64.whl (3.3 MB)
     |████████████████████████████████| 3.3 MB 1.4 MB/s 
Collecting h5py~=2.10.0
  Downloading h5py-2.10.0-cp37-cp37m-macosx_10_6_intel.whl (3.0 MB)
     |████████████████████████████████| 3.0 MB 2.2 MB/s 
Collecting keras-preprocessing~=1.1.2
  Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 2.3 MB/s 
Collecting opt-einsum~=3.3.0
  Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
     |████████████████████████████████| 65 kB 3.7 MB/s 
Collecting protobuf>=3.9.2
  Downloading protobuf-3.14.0-cp37-cp37m-macosx_10_9_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 3.5 MB/s 
Collecting tensorboard~=2.4
  Downloading tensorboard-2.4.0-py3-none-any.whl (10.6 MB)
     |████████████████████████████████| 10.6 MB 2.2 MB/s 
Requirement already satisfied: requests<3,>=2.21.0 in ./conda/envs/piptf/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow) (2.25.1)
Requirement already satisfied: setuptools>=41.0.0 in ./conda/envs/piptf/lib/python3.7/site-packages (from tensorboard~=2.4->tensorflow) (51.1.2.post20210110)
Collecting google-auth<2,>=1.6.3
  Downloading google_auth-1.24.0-py2.py3-none-any.whl (114 kB)
     |████████████████████████████████| 114 kB 2.2 MB/s 
Collecting cachetools<5.0,>=2.0.0
  Downloading cachetools-4.2.0-py3-none-any.whl (12 kB)
Collecting google-auth-oauthlib<0.5,>=0.4.1
  Downloading google_auth_oauthlib-0.4.2-py2.py3-none-any.whl (18 kB)
Collecting markdown>=2.6.8
  Downloading Markdown-3.3.3-py3-none-any.whl (96 kB)
     |████████████████████████████████| 96 kB 2.1 MB/s 
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
     |████████████████████████████████| 155 kB 2.2 MB/s 
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 4.1 MB/s 
Requirement already satisfied: certifi>=2017.4.17 in ./conda/envs/piptf/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow) (2020.12.5)
Requirement already satisfied: idna<3,>=2.5 in ./conda/envs/piptf/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./conda/envs/piptf/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow) (1.26.2)
Requirement already satisfied: chardet<5,>=3.0.2 in ./conda/envs/piptf/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard~=2.4->tensorflow) (4.0.0)
Collecting requests-oauthlib>=0.7.0
  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
Collecting rsa<5,>=3.1.4
  Downloading rsa-4.7-py3-none-any.whl (34 kB)
Collecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.7.0-py3-none-any.whl (779 kB)
     |████████████████████████████████| 779 kB 2.5 MB/s 
Collecting tensorflow-estimator<2.5.0,>=2.4.0rc0
  Downloading tensorflow_estimator-2.4.0-py2.py3-none-any.whl (462 kB)
     |████████████████████████████████| 462 kB 2.9 MB/s 
Collecting termcolor~=1.1.0
  Using cached termcolor-1.1.0-cp37-none-any.whl
Collecting typing-extensions~=3.7.4
  Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting werkzeug>=0.11.15
  Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
     |████████████████████████████████| 298 kB 2.8 MB/s 
Collecting wrapt~=1.12.1
  Downloading wrapt-1.12.1.tar.gz (27 kB)
Collecting importlib-metadata
  Downloading importlib_metadata-3.4.0-py3-none-any.whl (10 kB)
Collecting zipp>=0.5
  Downloading zipp-3.4.0-py3-none-any.whl (5.2 kB)
Building wheels for collected packages: wrapt
  Building wheel for wrapt (setup.py) ... done
  Created wheel for wrapt: filename=wrapt-1.12.1-cp37-cp37m-macosx_10_9_x86_64.whl size=32462 sha256=e32755e407cf3b01b687f8f92d6a6b9d56de8f9447c8b6cdafaf4560f11899c0
  Stored in directory: /Users/michael/Library/Caches/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6
Successfully built wrapt
Installing collected packages: pyasn1, zipp, typing-extensions, rsa, pyasn1-modules, oauthlib, cachetools, requests-oauthlib, importlib-metadata, google-auth, werkzeug, tensorboard-plugin-wit, protobuf, markdown, grpcio, google-auth-oauthlib, absl-py, wrapt, termcolor, tensorflow-estimator, tensorboard, opt-einsum, keras-preprocessing, h5py, google-pasta, gast, flatbuffers, astunparse, tensorflow
Successfully installed absl-py-0.11.0 astunparse-1.6.3 cachetools-4.2.0 flatbuffers-1.12 gast-0.3.3 google-auth-1.24.0 google-auth-oauthlib-0.4.2 google-pasta-0.2.0 grpcio-1.32.0 h5py-2.10.0 importlib-metadata-3.4.0 keras-preprocessing-1.1.2 markdown-3.3.3 oauthlib-3.1.0 opt-einsum-3.3.0 protobuf-3.14.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 requests-oauthlib-1.3.0 rsa-4.7 tensorboard-2.4.0 tensorboard-plugin-wit-1.7.0 tensorflow-2.4.0 tensorflow-estimator-2.4.0 termcolor-1.1.0 typing-extensions-3.7.4.3 werkzeug-1.0.1 wrapt-1.12.1 zipp-3.4.0

from what I can tell it hasn't messed with any existing dependencies, but it's not the easiest to parse.

I then did a dev install of the code and tried just running the current n3fit tests which all passed. I could try running a test fit as well and make sure there aren't any memory issues but this ostensibly seems ok, unless I missed something.

The awkward thing will of course be the conda recipe since I couldn't see an obvious way to install a package through pip automatically (and I guess in general they discourage it). CCed stakeholders for discussion.

[scarlehoff]

Another option is to do pip install tensorflow --no-deps (without uninstalling tensorflow-eigen from conda). That has the advantage of keeping the right dependencies in conda while maybe breaking tensorflow itself...

[Zaharid]

Let's see if it works but I find supporting this would indeed be complicated without some headaches. I don't think conda supports this at all and indeed works rather against it. I wonder is one can stick a pip install tensorflow in the test script and have that work...

[Zaharid]

As an aside, https://pytorch.org/get-started/locally/#linux-installation

[wilsonmr]

Another option is to do pip install tensorflow --no-deps (without uninstalling tensorflow-eigen from conda). That has the advantage of keeping the right dependencies in conda while maybe breaking tensorflow itself...

Yeah, this didn't work. It seems the newer version of tensorflow has some additional dependencies which aren't installed. Removing the --no-deps flag caused the following packages to be uninstalled:

Attempting uninstall: grpcio
    Found existing installation: grpcio 1.31.0
    Uninstalling grpcio-1.31.0:
      Successfully uninstalled grpcio-1.31.0
  Attempting uninstall: tensorflow-estimator
    Found existing installation: tensorflow-estimator 2.0.0
    Uninstalling tensorflow-estimator-2.0.0:
      Successfully uninstalled tensorflow-estimator-2.0.0
  Attempting uninstall: tensorboard
    Found existing installation: tensorboard 2.0.0
    Uninstalling tensorboard-2.0.0:
      Successfully uninstalled tensorboard-2.0.0
  Attempting uninstall: opt-einsum
    Found existing installation: opt-einsum 3.1.0
    Uninstalling opt-einsum-3.1.0:
      Successfully uninstalled opt-einsum-3.1.0
  Attempting uninstall: keras-preprocessing
    Found existing installation: Keras-Preprocessing 1.1.0
    Uninstalling Keras-Preprocessing-1.1.0:
      Successfully uninstalled Keras-Preprocessing-1.1.0
  Attempting uninstall: gast
    Found existing installation: gast 0.2.2
    Uninstalling gast-0.2.2:
      Successfully uninstalled gast-0.2.2
  Attempting uninstall: tensorflow
    Found existing installation: tensorflow 2.0.0
    Uninstalling tensorflow-2.0.0:
      Successfully uninstalled tensorflow-2.0.0

and tests still pass

[wilsonmr]

I'm starting to wonder what the end game aim with this is? If it's simply that the tests will pass with TF 2.4 features, but we are unsure if there are other issues using a combo of pip and conda then we probably wouldn't feel super comfortable relying on mac results anyway - which then raises the question why we don't just skip outputting n3fit on mac

From what I can tell we import n3fit in only a few places in validphys

• pseudodata.py which imports from reader.py but reader.py has no dependencies in n3fit so could itself be relocated. Also in the future we want to scrap this right?
• vp_rebuild_data.py which uses the N3FitConfig I think specifically for the fit definition which means you can specify a fit with a path - this could just be contained in the script.
• hyperoptplot.py which is importing something which again has no dependence on TF

[Zaharid]

I'm stating to think maybe it makes sense to split the packages (but not the repo!) on account of n3fit having more complicated dependencies...

[wilsonmr]

Yes exactly, something like https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#subpackage-requirements

[wilsonmr]

Or just two packages I guess - might be easier

[scarlehoff]

I'm starting to wonder what the end game aim with this is? If it's simply that the tests will pass with TF 2.4 features

Rather to not have to go through loops to ensure tests pass for both Linux and Mac since they are using different versions of libraries. That said, if it's more complicated than just pip install tensorflow (I was hoping the --no-deps options would solve the dependencies problem) maybe it's better to either skip tests/build on mac or just go through the necessary loops in order to ensure compatibility (it is not that difficult right now, but if nothing happens the difference between the Linux and Mac versions will be ever-growing)

pseudodata.py which imports from reader.py but reader.py has no dependencies in n3fit so could itself be relocated. Also in the future we want to scrap this right?

Yes. reader.py should be removed at some point. The hyperopt part can go in the other direction just as well (i.e., all the hyperparameters options being part of validphys and n3fit importing that).
w.r.t. the particular piece that needs N3FitConfig I think it is fine having a feature that needs n3fit to be installed? In any case, there is a "deprecate this whole script" TODO in that file...

wrt the one repo two packages policy: I think it makes sense. It would also allow you to install stuff with more granularity which is nice (for instance, I am often fine having the "official" validphys version because the only piece I'm actually changing is n3fit).
Also, I'm guessing as the M1 chip becomes more and more ubiquitous the hope for the conda packages for mac being up to date will vanish.