zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

PythonPackage: install packages with pip (#27798)

Browse Source 
* Use pip to bootstrap pip

* Bootstrap wheel from source

* Update PythonPackage to install using pip

* Update several packages

* Add wheel as base class dep

* Build phase no longer exists

* Add py-poetry package, fix py-flit-core bootstrapping

* Fix isort build

* Clean up many more packages

* Remove unused import

* Fix unit tests

* Don't directly run setup.py

* Typo fix

* Remove unused imports

* Fix issues caught by CI

* Remove custom setup.py file handling

* Use PythonPackage for installing wheels

* Remove custom phases in PythonPackages

* Remove <phase>_args methods

* Remove unused import

* Fix various packages

* Try to test Python packages directly in CI

* Actually run the pipeline

* Fix more packages

* Fix mappings, fix packages

* Fix dep version

* Work around bug in concretizer

* Various concretization fixes

* Fix gitlab yaml, packages

* Fix typo in gitlab yaml

* Skip more packages that fail to concretize

* Fix? jupyter ecosystem concretization issues

* Solve Jupyter concretization issues

* Prevent duplicate entries in PYTHONPATH

* Skip fenics-dolfinx

* Build fewer Python packages

* Fix missing npm dep

* Specify image

* More package fixes

* Add backends for every from-source package

* Fix version arg

* Remove GitLab CI stuff, add py-installer package

* Remove test deps, re-add install_options

* Function declaration syntax fix

* More build fixes

* Update spack create template

* Update PythonPackage documentation

* Fix documentation build

* Fix unit tests

* Remove pip flag added only in newer pip

* flux: add explicit dependency on jsonschema

* Update packages that have been added since this was branched off of develop

* Move Python 2 deprecation to a separate PR

* py-neurolab: add build dep on py-setuptools

* Use wheels for pip/wheel

* Allow use of pre-installed pip for external Python

* pip -> python -m pip

* Use python -m pip for all packages

* Fix py-wrapt

* Add both platlib and purelib to PYTHONPATH

* py-pyyaml: setuptools is needed for all versions

* py-pyyaml: link flags aren't needed

* Appease spack audit packages

* Some build backend is required for all versions, distutils -> setuptools

* Correctly handle different setup.py filename

* Use wheels for py-tomli to avoid circular dep on py-flit-core

* Fix busco installation procedure

* Clarify things in spack create template

* Test other Python build backends

* Undo changes to busco

* Various fixes

* Don't test other backends
This commit is contained in:
Adam J. Stewart
2022-01-14 12:37:57 -06:00
committed by GitHub
parent 0b2507053e
commit 3540f8200a
331 changed files with 1488 additions and 1577 deletions
Download Patch File Download Diff File Expand all files Collapse all files

807
lib/spack/docs/build_systems/pythonpackage.rst
View File

@@ -9,216 +9,80 @@
PythonPackage
-------------
Python packages and modules have their own special build system.
Python packages and modules have their own special build system. This
documentation covers everything you'll need to know in order to write
a Spack build recipe for a Python library.
^^^^^^
Phases
^^^^^^
^^^^^^^^^^^
Terminology
^^^^^^^^^^^
The ``PythonPackage`` base class provides the following phases that
can be overridden:
In the Python ecosystem, there are a number of terms that are
important to understand.
* ``build``
* ``build_py``
* ``build_ext``
* ``build_clib``
* ``build_scripts``
* ``install``
* ``install_lib``
* ``install_headers``
* ``install_scripts``
* ``install_data``
**PyPI**
The `Python Package Index <https://pypi.org/>`_, where most Python
libraries are hosted.
These are all standard ``setup.py`` commands and can be found by running:
**sdist**
Source distributions, distributed as tarballs (.tar.gz) and zip
files (.zip). Contain the source code of the package.
.. code-block:: console
**bdist**
Built distributions, distributed as wheels (.whl). Contain the
pre-built library.
$ python setup.py --help-commands
**wheel**
A binary distribution format common in the Python ecosystem. This
file is actually just a zip file containing specific metadata and
code. See the
`documentation <https://packaging.python.org/en/latest/specifications/binary-distribution-format/>`_
for more details.
**build frontend**
Command-line tools used to build and install wheels. Examples
include `pip <https://pip.pypa.io/>`_,
`build <https://pypa-build.readthedocs.io/>`_, and
`installer <https://installer.readthedocs.io/>`_.
By default, only the ``build`` and ``install`` phases are run:
**build backend**
Libraries used to define how to build a wheel. Examples
include `setuptools <https://setuptools.pypa.io/>`__,
`flit <https://flit.readthedocs.io/>`_, and
`poetry <https://python-poetry.org/>`_.
#. ``build`` - build everything needed to install
#. ``install`` - install everything from build directory
^^^^^^^^^^^
Downloading
^^^^^^^^^^^
If for whatever reason you need to run more phases, simply modify your
``phases`` list like so:
.. code-block:: python
phases = ['build_ext', 'install']
Each phase provides a function ``<phase>`` that runs:
.. code-block:: console
$ python -s setup.py --no-user-cfg <phase>
Each phase also has a ``<phase_args>`` function that can pass arguments to
this call. All of these functions are empty except for the ``install_args``
function, which passes ``--prefix=/path/to/installation/prefix``. There is
also some additional logic specific to setuptools and eggs.
If you need to run a phase that is not a standard ``setup.py`` command,
you'll need to define a function for it like so:
.. code-block:: python
phases = ['configure', 'build', 'install']
def configure(self, spec, prefix):
self.setup_py('configure')
^^^^^^
Wheels
^^^^^^
Some Python packages are closed-source and distributed as wheels.
Instead of using the ``PythonPackage`` base class, you should extend
the ``Package`` base class and implement the following custom installation
procedure:
.. code-block:: python
def install(self, spec, prefix):
pip = which('pip')
pip('install', self.stage.archive_file, '--prefix={0}'.format(prefix))
This will require a dependency on pip, as mentioned below.
^^^^^^^^^^^^^^^
Important files
^^^^^^^^^^^^^^^
Python packages can be identified by the presence of a ``setup.py`` file.
This file is used by package managers like ``pip`` to determine a
package's dependencies and the version of dependencies required, so if
the ``setup.py`` file is not accurate, the package will not build properly.
For this reason, the ``setup.py`` file should be fairly reliable. If the
documentation and ``setup.py`` disagree on something, the ``setup.py``
file should be considered to be the truth. As dependencies are added or
removed, the documentation is much more likely to become outdated than
the ``setup.py``.
The Python ecosystem has evolved significantly over the years. Before
setuptools became popular, most packages listed their dependencies in a
``requirements.txt`` file. Once setuptools took over, these dependencies
were listed directly in the ``setup.py``. Newer PEPs introduced additional
files, like ``setup.cfg`` and ``pyproject.toml``. You should look out for
all of these files, as they may all contain important information about
package dependencies.
Some Python packages are closed-source and are distributed as Python
wheels. For example, ``py-azureml-sdk`` downloads a ``.whl`` file. This
file is simply a zip file, and can be extracted using:
.. code-block:: console
$ unzip *.whl
The zip file will not contain a ``setup.py``, but it will contain a
``METADATA`` file which contains all the information you need to
write a ``package.py`` build recipe.
.. _pypi:
^^^^
PyPI
^^^^
The vast majority of Python packages are hosted on PyPI (The Python
Package Index), which is :ref:`preferred over GitHub <pypi-vs-github>`
for downloading packages. ``pip`` only supports packages hosted on PyPI, making
it the only option for developers who want a simple installation.
Search for "PyPI <package-name>" to find the download page. Note that
some pages are versioned, and the first result may not be the newest
version. Click on the "Latest Version" button to the top right to see
if a newer version is available. The download page is usually at::
The first step in packaging a Python library is to figure out where
to download it from. The vast majority of Python packages are hosted
on `PyPI <https://pypi.org/>`_, which is
:ref:`preferred over GitHub <pypi-vs-github>` for downloading
packages. Search for the package name on PyPI to find the project
page. The project page is usually located at::
https://pypi.org/project/<package-name>
Since PyPI is so common, the ``PythonPackage`` base class has a
``pypi`` attribute that can be set. Once set, ``pypi`` will be used
to define the ``homepage``, ``url``, and ``list_url``. For example,
the following:
.. code-block:: python
homepage = 'https://pypi.org/project/setuptools/'
url = 'https://pypi.org/packages/source/s/setuptools/setuptools-49.2.0.zip'
list_url = 'https://pypi.org/simple/setuptools/'
is equivalent to:
.. code-block:: python
pypi = 'setuptools/setuptools-49.2.0.zip'
^^^^^^^^^^^
Description
^^^^^^^^^^^
The top of the PyPI downloads page contains a description of the
package. The first line is usually a short description, while there
may be a several line "Project Description" that follows. Choose whichever
is more useful. You can also get these descriptions on the command-line
using:
.. code-block:: console
$ python setup.py --description
$ python setup.py --long-description
^^^^^^^^
Homepage
^^^^^^^^
Package developers use ``setup.py`` to upload new versions to PyPI.
The ``setup`` method often passes metadata like ``homepage`` to PyPI.
This metadata is displayed on the left side of the download page.
Search for the text "Homepage" under "Project links" to find it. You
should use this page instead of the PyPI page if they differ. You can
also get the homepage on the command-line by running:
.. code-block:: console
$ python setup.py --url
^^^
URL
^^^
If ``pypi`` is set as mentioned above, ``url`` and ``list_url`` will
be automatically set for you. If both ``.tar.gz`` and ``.zip`` versions
are available, ``.tar.gz`` is preferred. If some releases offer both
``.tar.gz`` and ``.zip`` versions, but some only offer ``.zip`` versions,
use ``.zip``.
Some Python packages are closed-source and do not ship ``.tar.gz`` or ``.zip``
files on either PyPI or GitHub. If this is the case, you can still download
and install a Python wheel. For example, ``py-azureml-sdk`` is closed source
and can be downloaded from::
On the project page, there is a "Download files" tab containing
download URLs. Whenever possible, we prefer to build Spack packages
from source. If PyPI only has wheels, check to see if the project is
hosted on GitHub and see if GitHub has source distributions. The
project page usually has a "Homepage" and/or "Source code" link for
this. If the project is closed-source, it may only have wheels
available. For example, ``py-azureml-sdk`` is closed-source and can
be downloaded from::
https://pypi.io/packages/py3/a/azureml_sdk/azureml_sdk-1.11.0-py3-none-any.whl
Once you've found a URL to download the package from, run:
You may see Python-specific or OS-specific URLs. Note that when you add a
``.whl`` URL, you should add ``expand=False`` to ensure that Spack doesn't
try to extract the wheel:
.. code-block:: console
.. code-block:: python
$ spack create <url>
version('1.11.0', sha256='d8c9d24ea90457214d798b0d922489863dad518adde3638e08ef62de28fb183a', expand=False)
to create a new package template.
.. _pypi-vs-github:
@@ -226,11 +90,13 @@ try to extract the wheel:
PyPI vs. GitHub
"""""""""""""""
Many packages are hosted on PyPI, but are developed on GitHub or another
version control systems. The tarball can be downloaded from either
location, but PyPI is preferred for the following reasons:
Many packages are hosted on PyPI, but are developed on GitHub or
another version control system hosting service. The source code can
be downloaded from either location, but PyPI is preferred for the
following reasons:
#. PyPI contains the bare minimum number of files needed to install the package.
#. PyPI contains the bare minimum number of files needed to install
the package.
You may notice that the tarball you download from PyPI does not
have the same checksum as the tarball you download from GitHub.
@@ -267,252 +133,124 @@ location, but PyPI is preferred for the following reasons:
PyPI is nice because it makes it physically impossible to
re-release the same version of a package with a different checksum.
Use the :ref:`pypi attribute <pypi>` to facilitate construction of PyPI package
references.
The only reason to use GitHub instead of PyPI is if PyPI only has
wheels or if the PyPI sdist is missing a file needed to build the
package. If this is the case, please add a comment above the ``url``
explaining this.
^^^^^^^^^^^^^^^^^^^^^^^^^
Build system dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^
PyPI
^^^^
There are a few dependencies common to the ``PythonPackage`` build system.
""""""
Python
""""""
Obviously, every ``PythonPackage`` needs Python at build-time to run
``python setup.py build && python setup.py install``. Python is also
needed at run-time if you want to import the module. Due to backwards
incompatible changes between Python 2 and 3, it is very important to
specify which versions of Python are supported. If the documentation
mentions that Python 3 is required, this can be specified as:
Since PyPI is so commonly used to host Python libraries, the
``PythonPackage`` base class has a ``pypi`` attribute that can be
set. Once set, ``pypi`` will be used to define the ``homepage``,
``url``, and ``list_url``. For example, the following:
.. code-block:: python
depends_on('python@3:', type=('build', 'run'))
homepage = 'https://pypi.org/project/setuptools/'
url = 'https://pypi.org/packages/source/s/setuptools/setuptools-49.2.0.zip'
list_url = 'https://pypi.org/simple/setuptools/'
If Python 2 is required, this would look like:
is equivalent to:
.. code-block:: python
depends_on('python@:2', type=('build', 'run'))
pypi = 'setuptools/setuptools-49.2.0.zip'
If Python 2.7 is the only version that works, you can use:
If a package has a different homepage listed on PyPI, you can
override it by setting your own ``homepage``.
^^^^^^^^^^^
Description
^^^^^^^^^^^
The top of the PyPI project page contains a short description of the
package. The "Project description" tab may also contain a longer
description of the package. Either of these can be used to populate
the package docstring.
^^^^^^^^^^^^^
Build backend
^^^^^^^^^^^^^
Once you've determined the basic metadata for a package, the next
step is to determine the build backend. ``PythonPackage`` uses
`pip <https://pip.pypa.io/>`_ to install the package, but pip
requires a backend to actually build the package.
To determine the build backend, look for a ``pyproject.toml`` file.
If there is no ``pyproject.toml`` file and only a ``setup.py`` or
``setup.cfg`` file, you can assume that the project uses
:ref:`setuptools`. If there is a ``pyproject.toml`` file, see if it
contains a ``[build-system]`` section. For example:
.. code-block:: toml
[build-system]
requires = [
"setuptools>=42",
"wheel",
]
build-backend = "setuptools.build_meta"
This section does two things: the ``requires`` key lists build
dependencies of the project, and the ``build-backend`` key defines
the build backend. All of these build dependencies should be added as
dependencies to your package:
.. code-block:: python
depends_on('python@2.7:2.8', type=('build', 'run'))
depends_on('py-setuptools@42:', type='build')
The documentation may not always specify supported Python versions.
Another place to check is in the ``setup.py`` or ``setup.cfg`` file.
Look for a line containing ``python_requires``. An example from
`py-numpy <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-numpy/package.py>`_
looks like:
Note that ``py-wheel`` is already listed as a build dependency in the
``PythonPackage`` base class, so you don't need to add it unless you
need to specify a specific version requirement or change the
dependency type.
.. code-block:: python
See `PEP 517 <https://www.python.org/dev/peps/pep-0517/>`_ and
`PEP 518 <https://www.python.org/dev/peps/pep-0518/>`_ for more
information on the design of ``pyproject.toml``.
python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*'
Depending on which build backend a project uses, there are various
places that run-time dependencies can be listed.
"""""""""
distutils
"""""""""
You may also find a version check at the top of the ``setup.py``:
Before the introduction of setuptools and other build backends,
Python packages had to rely on the built-in distutils library.
Distutils is missing many of the features that setuptools and other
build backends offer, and users are encouraged to use setuptools
instead. In fact, distutils was deprecated in Python 3.10 and will be
removed in Python 3.12. Because of this, pip actually replaces all
imports of distutils with setuptools. If a package uses distutils,
you should instead add a build dependency on setuptools. Check for a
``requirements.txt`` file that may list dependencies of the project.
.. code-block:: python
if sys.version_info[:2] < (2, 7) or (3, 0) <= sys.version_info[:2] < (3, 4):
raise RuntimeError("Python version 2.7 or >= 3.4 required.")
This can be converted to Spack's spec notation like so:
.. code-block:: python
depends_on('python@2.7:2.8,3.4:', type=('build', 'run'))
If you are writing a recipe for a package that only distributes
wheels, look for a section in the ``METADATA`` file that looks like::
Requires-Python: >=3.5,<4
This would be translated to:
.. code-block:: python
extends('python')
depends_on('python@3.5:3', type=('build', 'run'))
Many ``setup.py`` or ``setup.cfg`` files also contain information like::
Programming Language :: Python :: 2
Programming Language :: Python :: 2.6
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.3
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5
Programming Language :: Python :: 3.6
This is a list of versions of Python that the developer likely tests.
However, you should not use this to restrict the versions of Python
the package uses unless one of the two former methods (``python_requires``
or ``sys.version_info``) is used. There is no logic in setuptools
that prevents the package from building for Python versions not in
this list, and often new releases like Python 3.7 or 3.8 work just fine.
.. _setuptools:
""""""""""
setuptools
""""""""""
Originally, the Python language had a single build system called
distutils, which is built into Python. Distutils provided a common
framework for package authors to describe their project and how it
should be built. However, distutils was not without limitations.
Most notably, there was no way to list a project's dependencies
with distutils. Along came setuptools, a non-builtin build system
designed to overcome the limitations of distutils. Both projects
use a similar API, making the transition easy while adding much
needed functionality. Today, setuptools is used in around 90% of
the Python packages in Spack.
Since setuptools isn't built-in to Python, you need to add it as a
dependency. To determine whether or not a package uses setuptools,
search the file for an import statement like:
.. code-block:: python
import setuptools
or:
.. code-block:: python
from setuptools import setup
Some packages are designed to work with both setuptools and distutils,
so you may find something like:
.. code-block:: python
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
This uses setuptools if available, and falls back to distutils if not.
In this case, you would still want to add a setuptools dependency, as
it offers us more control over the installation.
Unless specified otherwise, setuptools is usually a build-only dependency.
That is, it is needed to install the software, but is not needed at
run-time. This can be specified as:
.. code-block:: python
depends_on('py-setuptools', type='build')
"""
pip
"""
Packages distributed as Python wheels will require an extra dependency
on pip:
.. code-block:: python
depends_on('py-pip', type='build')
We will use pip to install the actual wheel.
""""""
cython
""""""
Compared to compiled languages, interpreted languages like Python can
be quite a bit slower. To work around this, some Python developers
rewrite computationally demanding sections of code in C, a process
referred to as "cythonizing". In order to build these package, you
need to add a build dependency on cython:
.. code-block:: python
depends_on('py-cython', type='build')
Look for references to "cython" in the ``setup.py`` to determine
whether or not this is necessary. Cython may be optional, but
even then you should list it as a required dependency. Spack is
designed to compile software, and is meant for HPC facilities
where speed is crucial. There is no reason why someone would not
want an optimized version of a library instead of the pure-Python
version.
Note that some release tarballs come pre-cythonized, and cython is
not needed as a dependency. However, this is becoming less common
as Python continues to evolve and developers discover that cythonized
sources are no longer compatible with newer versions of Python and
need to be re-cythonized.
^^^^^^^^^^^^^^^^^^^
Python dependencies
^^^^^^^^^^^^^^^^^^^
When you install a package with ``pip``, it reads the ``setup.py``
file in order to determine the dependencies of the package.
If the dependencies are not yet installed, ``pip`` downloads them
and installs them for you. This may sound convenient, but Spack
cannot rely on this behavior for two reasons:
#. Spack needs to be able to install packages on air-gapped networks.
If there is no internet connection, ``pip`` can't download the
package dependencies. By explicitly listing every dependency in
the ``package.py``, Spack knows what to download ahead of time.
#. Duplicate installations of the same dependency may occur.
Spack supports *activation* of Python extensions, which involves
symlinking the package installation prefix to the Python installation
prefix. If your package is missing a dependency, that dependency
will be installed to the installation directory of the same package.
If you try to activate the package + dependency, it may cause a
problem if that package has already been activated.
For these reasons, you must always explicitly list all dependencies.
Although the documentation may list the package's dependencies,
often the developers assume people will use ``pip`` and won't have to
worry about it. Always check the ``setup.py`` to find the true
dependencies.
If the package relies on ``distutils``, it may not explicitly list its
dependencies. Check for statements like:
.. code-block:: python
try:
import numpy
except ImportError:
raise ImportError("numpy must be installed prior to installation")
Obviously, this means that ``py-numpy`` is a dependency.
If the package uses ``setuptools``, check for the following clues:
If the ``pyproject.toml`` lists ``setuptools.build_meta`` as a
``build-backend``, or if the package has a ``setup.py`` that imports
``setuptools``, or if the package has a ``setup.cfg`` file, then it
uses setuptools to build. Setuptools is a replacement for the
distutils library, and has almost the exact same API. Dependencies
can be listed in the ``setup.py`` or ``setup.cfg`` file. Look for the
following arguments:
* ``python_requires``
As mentioned above, this specifies which versions of Python are
required.
This specifies the version of Python that is required.
* ``setup_requires``
@@ -524,43 +262,88 @@ If the package uses ``setuptools``, check for the following clues:
These packages are required for building and installation. You can
add them with ``type=('build', 'run')``.
* ``extra_requires``
* ``extras_require``
These packages are optional dependencies that enable additional
functionality. You should add a variant that optionally adds these
dependencies. This variant should be False by default.
* ``test_requires``
* ``tests_require``
These are packages that are required to run the unit tests for the
package. These dependencies can be specified using the
``type='test'`` dependency type. However, the PyPI tarballs rarely
contain unit tests, so there is usually no reason to add these.
In the root directory of the package, you may notice a
``requirements.txt`` file. It may look like this file contains a list
of all of the package's dependencies. Don't be fooled. This file is
used by tools like Travis to install the pre-requisites for the
package... and a whole bunch of other things. It often contains
dependencies only needed for unit tests, like:
See https://setuptools.pypa.io/en/latest/userguide/dependency_management.html
for more information on how setuptools handles dependency management.
See `PEP 440 <https://www.python.org/dev/peps/pep-0440/#version-specifiers>`_
for documentation on version specifiers in setuptools.
* mock
* nose
* pytest
""""
flit
""""
It can also contain dependencies for building the documentation, like
sphinx. If you can't find any information about the package's
dependencies, you can take a look in ``requirements.txt``, but be sure
not to add test or documentation dependencies.
There are actually two possible ``build-backend`` for flit, ``flit``
and ``flit_core``. If you see these in the ``pyproject.toml``, add a
build dependency to your package. With flit, all dependencies are
listed directly in the ``pyproject.toml`` file. Older versions of
flit used to store this info in a ``flit.ini`` file, so check for
this too.
Newer PEPs have added alternative ways to specify a package's dependencies.
If you don't see any dependencies listed in the ``setup.py``, look for a
``setup.cfg`` or ``pyproject.toml``. These files can be used to store the
same ``install_requires`` information that ``setup.py`` used to use.
Either of these files may contain keys like:
If you are write a recipe for a package that only distributes wheels,
check the ``METADATA`` file for lines like::
* ``requires-python``
This specifies the version of Python that is required
* ``dependencies`` or ``requires``
These packages are required for building and installation. You can
add them with ``type=('build', 'run')``.
* ``project.optional-dependencies`` or ``requires-extra``
This section includes keys with lists of optional dependencies
needed to enable those features. You should add a variant that
optionally adds these dependencies. This variant should be False
by default.
See https://flit.readthedocs.io/en/latest/pyproject_toml.html for
more information.
""""""
poetry
""""""
Like flit, poetry also has two possible ``build-backend``, ``poetry``
and ``poetry_core``. If you see these in the ``pyproject.toml``, add
a build dependency to your package. With poetry, all dependencies are
listed directly in the ``pyproject.toml`` file. Dependencies are
listed in a ``[tool.poetry.dependencies]`` section, and use a
`custom syntax <https://python-poetry.org/docs/dependency-specification/#version-constraints>`_
for specifying the version requirements. Note that ``~=`` works
differently in poetry than in setuptools and flit for versions that
start with a zero.
""""""
wheels
""""""
Some Python packages are closed-source and are distributed as Python
wheels. For example, ``py-azureml-sdk`` downloads a ``.whl`` file. This
file is simply a zip file, and can be extracted using:
.. code-block:: console
$ unzip *.whl
The zip file will not contain a ``setup.py``, but it will contain a
``METADATA`` file which contains all the information you need to
write a ``package.py`` build recipe. Check for lines like::
Requires-Python: >=3.5,<4
Requires-Dist: azureml-core (~=1.11.0)
Requires-Dist: azureml-dataset-runtime[fuse] (~=1.11.0)
Requires-Dist: azureml-train (~=1.11.0)
@@ -572,62 +355,58 @@ check the ``METADATA`` file for lines like::
Requires-Dist: azureml-train-automl (~=1.11.0); extra == 'automl'
Lines that use ``Requires-Dist`` are similar to ``install_requires``.
Lines that use ``Provides-Extra`` are similar to ``extra_requires``,
and you can add a variant for those dependencies. The ``~=1.11.0``
syntax is equivalent to ``1.11.0:1.11``.
""""""""""
setuptools
""""""""""
Setuptools is a bit of a special case. If a package requires setuptools
at run-time, how do they express this? They could add it to
``install_requires``, but setuptools is imported long before this and is
needed to read this line. And since you can't install the package
without setuptools, the developers assume that setuptools will already
be there, so they never mention when it is required. We don't want to
add run-time dependencies if they aren't needed, so you need to
determine whether or not setuptools is needed. Grep the installation
directory for any files containing a reference to ``setuptools`` or
``pkg_resources``. Both modules come from ``py-setuptools``.
``pkg_resources`` is particularly common in scripts found in
``prefix/bin``.
``Requires-Python`` is equivalent to ``python_requires`` and
``Requires-Dist`` is equivalent to ``install_requires``.
``Provides-Extra`` is used to name optional features (variants) and
a ``Requires-Dist`` with ``extra == 'foo'`` will list any
dependencies needed for that feature.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Passing arguments to setup.py
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The default build and install phases should be sufficient to install
most packages. However, you may want to pass additional flags to
either phase.
The default install phase should be sufficient to install most
packages. However, the installation instructions for a package may
suggest passing certain flags to the ``setup.py`` call. The
``PythonPackage`` class has two techniques for doing this.
You can view the available options for a particular phase with:
""""""""""""""
Global options
""""""""""""""
.. code-block:: console
$ python setup.py <phase> --help
Each phase provides a ``<phase_args>`` function that can be used to
pass arguments to that phase. For example,
`py-numpy <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-numpy/package.py>`_
adds:
These flags are added directly after ``setup.py`` when pip runs
``python setup.py install``. For example, the ``py-pyyaml`` package
has an optional dependency on ``libyaml`` that can be enabled like so:
.. code-block:: python
def build_args(self, spec, prefix):
args = []
def global_options(self, spec, prefix):
options = []
if '+libyaml' in spec:
options.append('--with-libyaml')
else:
options.append('--without-libyaml')
return options
# From NumPy 1.10.0 on it's possible to do a parallel build.
if self.version >= Version('1.10.0'):
# But Parallel build in Python 3.5+ is broken. See:
# https://github.com/spack/spack/issues/7927
# https://github.com/scipy/scipy/issues/7112
if spec['python'].version < Version('3.5'):
args = ['-j', str(make_jobs)]
return args
"""""""""""""""
Install options
"""""""""""""""
These flags are added directly after ``install`` when pip runs
``python setup.py install``. For example, the ``py-pyyaml`` package
allows you to specify the directories to search for ``libyaml``:
.. code-block:: python
def install_options(self, spec, prefix):
options = []
if '+libyaml' in spec:
options.extend([
spec['libyaml'].libs.search_flags,
spec['libyaml'].headers.include_flags,
])
return options
^^^^^^^
@@ -669,9 +448,9 @@ a "package" is a directory containing files like:
whereas a "module" is a single Python file.
The ``PythonPackage`` base class automatically detects these module
names for you. If, for whatever reason, the module names detected
are wrong, you can provide the names yourself by overriding
The ``PythonPackage`` base class automatically detects these package
and module names for you. If, for whatever reason, the module names
detected are wrong, you can provide the names yourself by overriding
``import_modules`` like so:
.. code-block:: python
@@ -692,10 +471,8 @@ This can be expressed like so:
@property
def import_modules(self):
modules = ['yaml']
if '+libyaml' in self.spec:
modules.append('yaml.cyaml')
return modules
@@ -713,8 +490,8 @@ Unit tests
""""""""""
The package may have its own unit or regression tests. Spack can
run these tests during the installation by adding phase-appropriate
test methods.
run these tests during the installation by adding test methods after
installation.
For example, ``py-numpy`` adds the following as a check to run
after the ``install`` phase:
@@ -740,34 +517,14 @@ when testing is enabled during the installation (i.e., ``spack install
Setup file in a sub-directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In order to be compatible with package managers like ``pip``, the package
is required to place its ``setup.py`` in the root of the tarball. However,
not every Python package cares about ``pip`` or PyPI. If you are installing
a package that is not hosted on PyPI, you may find that it places its
``setup.py`` in a sub-directory. To handle this, add the directory containing
``setup.py`` to the package like so:
Many C/C++ libraries provide optional Python bindings in a
subdirectory. To tell pip which directory to build from, you can
override the ``build_directory`` attribute. For example, if a package
provides Python bindings in a ``python`` directory, you can use:
.. code-block:: python
build_directory = 'source'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Alternate names for setup.py
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As previously mentioned, packages need to call their setup script ``setup.py``
in order to be compatible with package managers like ``pip``. However, some
packages like
`py-meep <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-meep/package.py>`_ and
`py-adios <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-adios/package.py>`_
come with multiple setup scripts, one for a serial build and another for a
parallel build. You can override the default name to use like so:
.. code-block:: python
def setup_file(self):
return 'setup-mpi.py' if '+mpi' in self.spec else 'setup.py'
build_directory = 'python'
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -781,10 +538,14 @@ on Python are not necessarily ``PythonPackage``'s.
Choosing a build system
"""""""""""""""""""""""
First of all, you need to select a build system. ``spack create`` usually
does this for you, but if for whatever reason you need to do this manually,
choose ``PythonPackage`` if and only if the package contains a ``setup.py``
file.
First of all, you need to select a build system. ``spack create``
usually does this for you, but if for whatever reason you need to do
this manually, choose ``PythonPackage`` if and only if the package
contains one of the following files:
* ``pyproject.toml``
* ``setup.py``
* ``setup.cfg``
"""""""""""""""""""""""
Choosing a package name
@@ -857,10 +618,9 @@ having to add that module to ``PYTHONPATH``.
When deciding between ``extends`` and ``depends_on``, the best rule of
thumb is to check the installation prefix. If Python libraries are
installed to ``prefix/lib/python2.7/site-packages`` (where 2.7 is the
MAJOR.MINOR version of Python you used to install the package), then
you should use ``extends``. If Python libraries are installed elsewhere
or the only files that get installed reside in ``prefix/bin``, then
installed to ``<prefix>/lib/pythonX.Y/site-packages``, then you
should use ``extends``. If Python libraries are installed elsewhere
or the only files that get installed reside in ``<prefix>/bin``, then
don't use ``extends``, as symlinking the package wouldn't be useful.
^^^^^^^^^^^^^^^^^^^^^
@@ -893,4 +653,17 @@ External documentation
^^^^^^^^^^^^^^^^^^^^^^
For more information on Python packaging, see:
https://packaging.python.org/
* https://packaging.python.org/
For more information on build and installation frontend tools, see:
* pip: https://pip.pypa.io/
* build: https://pypa-build.readthedocs.io/
* installer: https://installer.readthedocs.io/
For more information on build backend tools, see:
* setuptools: https://setuptools.pypa.io/
* flit: https://flit.readthedocs.io/
* poetry: https://python-poetry.org/