From 50c27b4b6aab2a1a15ed2dd5f96e9dcb256836a9 Mon Sep 17 00:00:00 2001 From: Maria Ubiali Date: Mon, 11 Mar 2024 10:07:33 +0000 Subject: [PATCH 1/7] Changed initial page, added new references and added instructions for installation on M3 --- .../source/get-started/installation.rst | 27 ++++++++++ doc/sphinx/source/index.rst | 15 ++++-- doc/sphinx/source/references.bib | 54 +++++++++++++++++++ 3 files changed, 93 insertions(+), 3 deletions(-) diff --git a/doc/sphinx/source/get-started/installation.rst b/doc/sphinx/source/get-started/installation.rst index 24db920d3e..f569e6220d 100644 --- a/doc/sphinx/source/get-started/installation.rst +++ b/doc/sphinx/source/get-started/installation.rst @@ -241,6 +241,33 @@ explained above, if the user has not already done so. .. _M1: +Installation from source on M3 Mac +---------------------------------- + +1. Setup conda environment using python 3.11 and, if you don't have + them yet, install ``lhapdf`` and ``pandoc``. + + .. code:: + + conda create -n nnpdf_env python=3.11 + conda activate nnpdf_env + conda install lhapdf pandoc + +2. Clone the repository + + .. code:: + + git clone https://github.com/NNPDF/nnpdf.git + cd nnpdf + +3. Install NNPDF packages and its dependencies + + .. code:: + + python -m pip install -e . + + + Installation from source on M1/M2 Macs -------------------------------------- diff --git a/doc/sphinx/source/index.rst b/doc/sphinx/source/index.rst index 482da3e389..80b7f76079 100644 --- a/doc/sphinx/source/index.rst +++ b/doc/sphinx/source/index.rst @@ -35,6 +35,7 @@ members: * Andrea Barontini - Università degli Studi di Milano and INFN * Alessandro Candido - Università degli Studi di Milano and INFN * Stefano Carrazza - Università degli Studi di Milano and INFN +* Mark Costantini - University of Cambridge * Juan M. Cruz-Martinez - CERN * Luigi Del Debbio - University of Edinburgh * Stefano Forte - Università degli Studi di Milano and INFN @@ -55,7 +56,7 @@ Former members of the NNPDF collaboration include * Rabah Abdul Khalek - Post-doc at Jefferson Lab, USA * Valerio Bertone - Post-doc at CEA Saclay, FR -* Francesco Cerutti +* Francesco Cerutti - Deployment of the Fly-Eye Telescope for NEO Survey - ESA/ASI * Christopher S. Deans * Alberto Guffanti - Data Scientist at PIVIGO, UK * Patrick Groth-Merrild @@ -65,12 +66,20 @@ Former members of the NNPDF collaboration include * Andrea Piccione - High School Teacher at IPIA G. Piana, IT * Luca Rottoli - Post-doc at the University of Zurich, CH * Emma Slade - Senior AI/ML Engineer at GSK, UK -* Cameron Voisey +* Cameron Voisey - Data Scientist at Privacy Hub by Datavant * Michael Wilson The NNPDF publications ====================== -* *"Evidence for intrinsic charm quarks in the proton", +* *"The Path to N3LO Parton Distributions"*, + Richard D. Ball et al. :cite:p:`NNPDF:2024nan` +* *"Determination of the theory uncertainties from missing higher orders on NNLO parton distributions with percent accuracy"*, + Richard D. Ball et al. :cite:p:`NNPDF:2024dpb` +* *"Photons in the proton: implications for the LHC"*, + Richard D. Ball et al. :cite:p:`NNPDF:2024djq` +* *"The intrinsic charm quark valence distribution of the proton"*, + Richard D. Ball et al. :cite:p:`NNPDF:2023tyk` +* *"Evidence for intrinsic charm quarks in the proton"*, Richard D. Ball et al. :cite:p:`Ball:2022qks` * *"Regularising experimental correlations in LHC data: theory and application to a global analysis of parton distributions"*, diff --git a/doc/sphinx/source/references.bib b/doc/sphinx/source/references.bib index bb646dcae4..1bc79c9bd4 100644 --- a/doc/sphinx/source/references.bib +++ b/doc/sphinx/source/references.bib @@ -1,3 +1,57 @@ +@article{NNPDF:2024nan, + author = "Ball, Richard D. and others", + collaboration = "NNPDF", + title = "{The Path to N$^3$LO Parton Distributions}", + eprint = "2402.18635", + archivePrefix = "arXiv", + primaryClass = "hep-ph", + reportNumber = "Nikhef-2023-020, TIF-UNIMI-2023-23, Edinburgh 2023/29", + month = "2", + year = "2024", + journal = "" +} + +@article{NNPDF:2024dpb, + author = "Ball, Richard D. and others", + collaboration = "NNPDF", + title = "{Determination of the theory uncertainties from missing higher orders on NNLO parton distributions with percent accuracy}", + eprint = "2401.10319", + archivePrefix = "arXiv", + primaryClass = "hep-ph", + reportNumber = "TIF-UNIMI-2023-22, Edinburgh 2023/34, CERN-TH-2024-009", + month = "1", + year = "2024", + journal = "" +} + +@article{NNPDF:2023tyk, + author = "Ball, Richard D. and Candido, Alessandro and Cruz-Martinez, Juan and Forte, Stefano and Giani, Tommaso and Hekhorn, Felix and Magni, Giacomo and Nocera, Emanuele R. and Rojo, Juan and Stegeman, Roy", + collaboration = "NNPDF", + title = "{The intrinsic charm quark valence distribution of the proton}", + eprint = "2311.00743", + archivePrefix = "arXiv", + primaryClass = "hep-ph", + reportNumber = "Edinburgh 2023/22, TIF-UNIMI-2023-28, Nikhef 2023-012, + CERN-TH-2023-196", + month = "11", + year = "2023", + journal = "" +} + +@article{NNPDF:2024djq, + author = "Ball, Richard D. and others", + collaboration = "NNPDF", + title = "{Photons in the proton: implications for the LHC}", + eprint = "2401.08749", + archivePrefix = "arXiv", + primaryClass = "hep-ph", + reportNumber = "TIF-UNIMI-2023-17, Edinburgh 2023/19, CERN-TH-2023-159", + month = "1", + year = "2024", + journal = "" +} + + @article{Ball:2022qks, author = "Ball, Richard D. and Candido, Alessandro and Cruz-Martinez, Juan and Forte, Stefano and Giani, Tommaso and Hekhorn, Felix and Kudashkin, Kirill and Magni, Giacomo and Rojo, Juan", collaboration = "NNPDF", From bb9eaa51ac6a16aa79d8c30e09b079d0234d01c5 Mon Sep 17 00:00:00 2001 From: Maria Ubiali Date: Mon, 11 Mar 2024 10:54:16 +0000 Subject: [PATCH 2/7] Changed job status --- doc/sphinx/source/index.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/sphinx/source/index.rst b/doc/sphinx/source/index.rst index 80b7f76079..27a4d77414 100644 --- a/doc/sphinx/source/index.rst +++ b/doc/sphinx/source/index.rst @@ -41,7 +41,6 @@ members: * Stefano Forte - Università degli Studi di Milano and INFN * Tommaso Giani - Vrije University Amsterdam and Nikhef * Felix Hekhorn - Università degli Studi di Milano and INFN -* Zahari Kassabov - University of Cambridge * José Ignacio Latorre - Quantum Research Centre, Technology Innovation Institute, Abu Dhabi, United Arab Emirates and Center for Quantum Technologies, National University of Singapore * Niccolò Laurenti - Università degli Studi di Milano and INFN @@ -61,7 +60,9 @@ Former members of the NNPDF collaboration include * Alberto Guffanti - Data Scientist at PIVIGO, UK * Patrick Groth-Merrild * Nathan P. Hartland - Senior Data Analyst at Dott, NL -* Shayan Iranipour - Quantitative Researcher at Tudor Investment Corporation, UK +* Shayan Iranipour - Quantitative Researcher at Tudor Investment + Corporation, UK +* Zahari Kassabov - Research Engineer, Opificio, London, UK * Rosalyn Pearson - Information Analyst at Public Health Scotland, UK * Andrea Piccione - High School Teacher at IPIA G. Piana, IT * Luca Rottoli - Post-doc at the University of Zurich, CH From f4c101d1d6fc2ce8a36e84c5b8953be9e64a4f2a Mon Sep 17 00:00:00 2001 From: Maria Ubiali Date: Mon, 11 Mar 2024 11:59:14 +0000 Subject: [PATCH 3/7] removed M1 and M2 specific installation instructions --- .../source/get-started/installation.rst | 93 +------------------ 1 file changed, 2 insertions(+), 91 deletions(-) diff --git a/doc/sphinx/source/get-started/installation.rst b/doc/sphinx/source/get-started/installation.rst index f569e6220d..23b745a582 100644 --- a/doc/sphinx/source/get-started/installation.rst +++ b/doc/sphinx/source/get-started/installation.rst @@ -241,8 +241,8 @@ explained above, if the user has not already done so. .. _M1: -Installation from source on M3 Mac ----------------------------------- +Installation from source on M1-M2-M3 Mac +---------------------------------------- 1. Setup conda environment using python 3.11 and, if you don't have them yet, install ``lhapdf`` and ``pandoc``. @@ -266,95 +266,6 @@ Installation from source on M3 Mac python -m pip install -e . - - -Installation from source on M1/M2 Macs --------------------------------------- - -Installation on M1/M2 Macs directly with cmake is not directly supported. -Make sure you have a valid installation of ``pandoc`` available in your system: - -1. Clone the repositories - - .. code:: - - mkdir nnpdfgit - cd nnpdfgit - git clone git@github.com:NNPDF/nnpdf.git - git clone git@github.com:NNPDF/binary-bootstrap.git - -2. Execute binary bootstrap to set the channels in ``.condarc`` and install miniconda. -Note: if you want to install some specific version of `miniconda `_ instead it should work just the same. - - .. code:: - - ./binary-bootstrap/bootstrap.sh - -3. Setup conda environment using python (we use in this example 3.10) and, if you don't have them yet, install ``lhapdf``, ``pandoc`` and ``sccache`` (for rust). - - .. code:: - - conda create -n nnpdf-dev python=3.10 - conda activate nnpdf-dev - conda install lhapdf pandoc sccache - - Test that everything is ok: - - .. code:: - - lhapdf install NNPDF40_nnlo_as_01180 - python -c "import lhapdf" - -4. Note for tensorflow - - At the time of writing, it is necessary to follow this extra step in order to install ``tensorflow`` which works only for python < 3.12. - Other versions of ``tensorflow-macos`` and ``tensorflow-metal`` might also work, but these are the ones we tested. - - .. code:: - - conda install -c apple tensorflow-deps - pip install tensorflow-macos==2.9.2 - pip install tensorflow-metal==0.5.0 - -5. Install NNPDF packages (``validphys``, ``n3fit`` and ``evolven3fit``) and its dependencies - - .. code:: - - pip install -e . - -6. Test - - .. code:: - - cd nnpdf/n3fit/runcards/examples - vp-setupfit Basic_runcard.yml - n3fit Basic_runcard.yml 1 - evolven3fit evolve Basic_runcard - - With these settings tensorflow will run by default on GPU which makes - the fit run very slow. To disable the GPU, type the following command: - - .. code:: - - export CUDA_VISIBLE_DEVICES=0 - - or insert the following line in the ``set_initial_state`` function in ``n3fit/src/n3fit/backends/keras_backend/internal_state.py``: - - .. code:: - - tf.config.set_visible_devices([], 'GPU') - - And to use legacy optimizers, you only need to change one line in ``n3fit/src/n3fit/backends/keras_backend/MetaModel.py``: - - .. code:: - - # from tensorflow.keras import optimizers as Kopt - import tensorflow.keras.optimizers.legacy as Kopt - - With both these tweaks, and the latest tensorflow versions, the basic runcard with 1 replica should take about 30 seconds. - -.. _docker: - Using the code with docker -------------------------- From 8a87d04b07ef99181f474dc8ed8e45c7f78956a6 Mon Sep 17 00:00:00 2001 From: RoyStegeman Date: Mon, 11 Mar 2024 18:49:30 +0000 Subject: [PATCH 4/7] update installation docs to install through pip --- .../source/get-started/installation.rst | 192 +++--------------- 1 file changed, 25 insertions(+), 167 deletions(-) diff --git a/doc/sphinx/source/get-started/installation.rst b/doc/sphinx/source/get-started/installation.rst index 23b745a582..9752d6fe04 100644 --- a/doc/sphinx/source/get-started/installation.rst +++ b/doc/sphinx/source/get-started/installation.rst @@ -61,8 +61,6 @@ directory, by default this is ``~/miniconda3``, and run the command conda activate conda install nnpdf -This will provide functioning C++ and Python executables. - **Note:** The installer will set up its own version of the LHAPDF code, with its own path for storing PDFs, which can be seen running ``lhapdf –help``. If you have an existing directory with LHAPDF grids, you may want to @@ -84,187 +82,47 @@ are transferred if you copy or move instead. Installation from source ------------------------ -If you intend to work on the NNPDF code, then building from source is -the recommended installation procedure. However, you can still use conda -to get all the dependencies and setup the validphys and C++ development -environment. Note -that the ``binary-bootstrap.sh`` should be downloaded and run as -explained above, if the user has not already done so. - -**NOTE:** For installation on M1/M2 Macs, please see the :ref:`M1` section. - -1. Create an NNPDF developer environment ``nnpdf-dev`` and install all - relevant dependencies using - - .. code:: - - conda create -n nnpdf-dev - conda activate nnpdf-dev - conda install --only-deps nnpdf - - Note that the user should be in the conda environment ``nnpdf-dev`` - whenever they wish to work on NNPDF code. The conda environment can - be exited using ``conda deactivate``. - - .. note:: - - If you are a macOS user, you will need to download the `Mac Software - Development Kit`_ or SDK for short. This is necessary to get the - correct C compiler. The `anconda documentation`_ explains in more - detail why you need this file, and why they cannot include it with - the compilers by default. - - You can check which version of SDK is currently being used by the - :ref:`CI` system by checking the ``MACOS_SDK_URL`` - and ``MACOS_SDK_FILE`` variables inside ``.travis.yml``. At the time - of writing this documentation, the version used is 10.9 but the user - is advised to check in case the documentation has become out of sync - with the CI configuration. Once you know the URL of the SDK file, you - can download it from the commandline using ``curl``, e.g.: - - .. code:: - - curl -L -O https://github.com/phracker/MacOSX-SDKs/releases/download/11.3/MacOSX10.9.sdk.tar.xz - - You can then unpack it into your root conda directory by running - - .. code:: +If you intend to work on the NNPDF code, then building from source is the +recommended installation procedure. Note that the ``binary-bootstrap.sh`` should +be downloaded and run as explained above, if the user has not already done so. - tar xfz MacOSX10.9.sdk.tar.xz -C `` by typing - ``echo $CONDA_PREFIX`` when your base conda environment is activated. You - should then export the following path - - .. code:: - - export CONDA_BUILD_SYSROOT=/MacOSX10.9.sdk - - which you may wish to write to one of your ``~/.bashrc`` or - ``~/.bash_profile`` scripts so that the SDK is easily accessible from the - shell. - -2. Install the appropriate C++ compilers using +1. Setup a conda environment with a recent Python version and, if you don't have + them yet, install ``lhapdf`` and ``pandoc`` since they are not provided + through PyPI. Any recent python version should work, but to get an exact + range of the supported Python versions see the `Github action workflow files + `_. .. code:: - conda install gxx_linux-64 - - macOS users should replace ``gxx_linux-64`` with ``clangxx_osx-64``. + conda create -n nnpdf_dev lhapdf pandoc -3. Ensure that the NNPDF repositories ``nnpdf`` and ``apfel`` are in the - ``nnpdfgit`` directory. These are required to be able to run fits and - can be obtained respectively by - - .. code:: - - git clone git@github.com:NNPDF/nnpdf.git - git clone https://github.com/scarrazza/apfel.git - -4. Obtain the dependencies of the code you want to build. Where to find - those depends on the particular code. For example, something linking - to ``libnnpdf`` will likely require ``pkg-config``. Projects based on - ``autotools`` (those that have a ``./configure`` script) will - additionally require ``automake`` and ``libtool``. Similarly projects - based on ``cmake`` will require installing the ``cmake`` package. In - the case of ``nnpdf`` itself, the build dependencies can be found in - ``/conda-recipe/meta.yaml``. We have to install the - remaining ones manually: - - .. code:: - - conda install pkg-config swig cmake - - When working on a Linux system it is `currently - `_ also needed to run +2. Clone the repository .. code:: - conda install sysroot_linux-64=2.17 + git clone https://github.com/NNPDF/nnpdf.git + cd nnpdf -5. We now need to make the installation prefix point to our - ``nnpdf-dev`` environment. Fortunately, when you activate the environment, - the location is saved to the environment variable ``CONDA_PREFIX``, e.g. +3. Install NNPDF packages and its dependencies (make sure the conda environment + is activated) .. code:: - - $ conda activate nnpdf-dev - $ echo $CONDA_PREFIX - /home/miniconda3/envs/nnpdf-dev/ - -6. Navigate to the ``nnpdf`` directory obtained from the Github - repository and create a directory. + conda activate nnpdf_dev + python -m pip install -e . .. note:: - The directory name is unimportant, - its role is to contain all of the build files, separately from the source - files - we will refer to it as the build directory. A clean - build and installation can always be achieved by - deleting the contents of the build directory (or even creating a new one) - and re-running ``cmake``. - - For this example we have created a directory called ``conda-bld`` by - - .. code:: - - nnpdf$ mkdir conda-bld - nnpdf$ cd conda-bld - - Note that it is important that for the following step to be executed - while the user is in the ``nnpdf-dev`` conda environment. The project - can be built using: - - .. code:: - - nnpdf/conda-bld$ cmake .. -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX - -7. When the user wishes to work on the NNPDF code, they should do so in, - for example, ``/nnpdfgit/nnpdf/libnnpdf``. To compile the code - navigate to the build directory created above and run - - .. code:: + Following the installation steps above will set up a development + environment that makes it possible to run and work on the nnpdf code. One + may wish to install additional, optional, dependencies. Depending on the + specific needs for an environment this can be dependencies that enable the + running of the CI test, building the documentations, performing a QED fit, + or using the code without installing ``lhapdf`` (this last option provides + somewhat limited functionality) - rm -r ./* - cmake .. -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX - make - make install + .. code:: - If you are reinstalling the code using the same build directory, it is - recommended to remove all files from the build directory as is shown - in the example above. - -.. _here: https://github.com/settings/keys -.. _Mac Software Development Kit: https://github.com/phracker/MacOSX-SDKs -.. _anconda documentation: https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html#macos-sdk - - -.. _M1: - -Installation from source on M1-M2-M3 Mac ----------------------------------------- - -1. Setup conda environment using python 3.11 and, if you don't have - them yet, install ``lhapdf`` and ``pandoc``. - - .. code:: - - conda create -n nnpdf_env python=3.11 - conda activate nnpdf_env - conda install lhapdf pandoc - -2. Clone the repository - - .. code:: - - git clone https://github.com/NNPDF/nnpdf.git - cd nnpdf - -3. Install NNPDF packages and its dependencies - - .. code:: - - python -m pip install -e . + python -m pip install -e .[tests,docs,qed,nolha] Using the code with docker -------------------------- From 6b5df4c212f5aaee0f644d0538746481433b984e Mon Sep 17 00:00:00 2001 From: RoyStegeman Date: Mon, 11 Mar 2024 18:57:35 +0000 Subject: [PATCH 5/7] don't mention nolha installation option --- doc/sphinx/source/get-started/installation.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/doc/sphinx/source/get-started/installation.rst b/doc/sphinx/source/get-started/installation.rst index 9752d6fe04..0112c682bc 100644 --- a/doc/sphinx/source/get-started/installation.rst +++ b/doc/sphinx/source/get-started/installation.rst @@ -116,13 +116,11 @@ be downloaded and run as explained above, if the user has not already done so. environment that makes it possible to run and work on the nnpdf code. One may wish to install additional, optional, dependencies. Depending on the specific needs for an environment this can be dependencies that enable the - running of the CI test, building the documentations, performing a QED fit, - or using the code without installing ``lhapdf`` (this last option provides - somewhat limited functionality) + running of the CI test, building the documentations, or performing a QED fit. .. code:: - python -m pip install -e .[tests,docs,qed,nolha] + python -m pip install -e .[tests,docs,qed] Using the code with docker -------------------------- From 4f4e57766e1cd001ad34ada6c520f28dde73e876 Mon Sep 17 00:00:00 2001 From: Roy Stegeman Date: Tue, 12 Mar 2024 09:20:33 +0000 Subject: [PATCH 6/7] Update doc/sphinx/source/get-started/installation.rst Co-authored-by: Juan M. Cruz-Martinez --- doc/sphinx/source/get-started/installation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx/source/get-started/installation.rst b/doc/sphinx/source/get-started/installation.rst index 0112c682bc..ca3c74b8be 100644 --- a/doc/sphinx/source/get-started/installation.rst +++ b/doc/sphinx/source/get-started/installation.rst @@ -82,7 +82,7 @@ are transferred if you copy or move instead. Installation from source ------------------------ -If you intend to work on the NNPDF code, then building from source is the +If you intend to work on the NNPDF code, then installing from source is the recommended installation procedure. Note that the ``binary-bootstrap.sh`` should be downloaded and run as explained above, if the user has not already done so. From 95e1333ca6acca1004ab645df1534f1c5e684e39 Mon Sep 17 00:00:00 2001 From: RoyStegeman Date: Tue, 12 Mar 2024 09:34:16 +0000 Subject: [PATCH 7/7] add linux or macos requirement to docs --- doc/sphinx/source/get-started/installation.rst | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/doc/sphinx/source/get-started/installation.rst b/doc/sphinx/source/get-started/installation.rst index ca3c74b8be..81d9d18ef1 100644 --- a/doc/sphinx/source/get-started/installation.rst +++ b/doc/sphinx/source/get-started/installation.rst @@ -1,10 +1,11 @@ -Installing the code -=================== - -There are two methods for installing the code, both of which require -conda. You can either install the code entirely with conda or install -the code from source, with the dependencies still being installed via -conda. :ref:`conda` is preferable if you simply want to run the +Installing the code on Linux or macOS +===================================== + +Installing the NNPDF code requires a system with a recent version of either a +Linux-based operating system or macOS. There are two methods for installing the +code, both of which require conda. You can either install the code entirely with +conda or install the code from source, with the dependencies still being +installed via conda. :ref:`conda` is preferable if you simply want to run the code, while the :ref:`source` is necessary if you want to develop the code. .. _conda: @@ -62,7 +63,7 @@ directory, by default this is ``~/miniconda3``, and run the command conda install nnpdf **Note:** The installer will set up its own version of the LHAPDF code, -with its own path for storing PDFs, which can be seen running ``lhapdf –help``. +with its own path for storing PDFs, which can be seen running ``lhapdf -–help``. If you have an existing directory with LHAPDF grids, you may want to either move, symlink or copy them to the new path (depending on whether you want to keep around the older installation). The command for