zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Separate setting build environment and run environment in packages (#11115)

Browse Source 
* Methods setting the environment now do it separately for build and run

Before this commit the `*_environment` methods were setting
modifications to both the build-time and run-time environment
simultaneously. This might cause issues as the two environments
inherently rely on different preconditions:

1. The build-time environment is set before building a package, thus
the package prefix doesn't exist and can't be inspected

2. The run-time environment instead is set assuming the target package
has been already installed

Here we split each of these functions into two: one setting the
build-time environment, one the run-time.

We also adopt a fallback strategy that inspects for old methods and
executes them as before, but prints a deprecation warning to tty. This
permits to port packages to use the new methods in a distributed way,
rather than having to modify all the packages at once.

* Added a test that fails if any package uses the old API

Marked the test xfail for now as we have a lot of packages in that
state.

* Added a test to check that a package modified by a PR is up to date

This test can be used any time we deprecate a method call to ensure
that during the first modification of the package we update also
the deprecated calls.

* Updated documentation
This commit is contained in:
Massimiliano Culpo
2019-10-17 19:17:21 +02:00
committed by Greg Becker
parent cf9de058aa
commit 9ddc98e46a
14 changed files with 337 additions and 219 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
lib/spack/docs/module_file_support.rst
View File

@@ -303,8 +303,7 @@ content of the module files generated by Spack. The first one:
.. code-block:: python
def setup_environment(self, spack_env, run_env):
"""Set up the compile and runtime environments for a package."""
def setup_run_environment(self, env):
pass
can alter the content of the module file associated with the same package where it is overridden.
@@ -312,16 +311,15 @@ The second method:
.. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
"""Set up the environment of packages that depend on this one"""
def setup_dependent_run_environment(self, env, dependent_spec):
pass
can instead inject run-time environment modifications in the module files of packages
that depend on it. In both cases you need to fill ``run_env`` with the desired
list of environment modifications.
.. note::
The ``r`` package and callback APIs
.. admonition:: The ``r`` package and callback APIs
An example in which it is crucial to override both methods
is given by the ``r`` package. This package installs libraries and headers
in non-standard locations and it is possible to prepend the appropriate directory
@@ -336,14 +334,14 @@ list of environment modifications.
with the following snippet:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/r/package.py
:pyobject: R.setup_environment
:pyobject: R.setup_run_environment
The ``r`` package also knows which environment variable should be modified
to make language extensions provided by other packages available, and modifies
it appropriately in the override of the second method:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/r/package.py
:pyobject: R.setup_dependent_environment
:pyobject: R.setup_dependent_run_environment
.. _modules-yaml:

73
lib/spack/docs/packaging_guide.rst
View File

@@ -2032,55 +2032,58 @@ appear in the package file (or in this case, in the list).
.. _setup-dependent-environment:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``setup_dependent_environment()``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Influence how dependents are built or run
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Spack provides a mechanism for dependencies to provide variables that
can be used in their dependents' build. Any package can declare a
``setup_dependent_environment()`` function, and this function will be
called before the ``install()`` method of any dependent packages.
This allows dependencies to set up environment variables and other
properties to be used by dependents.
The function declaration should look like this:
Spack provides a mechanism for dependencies to influence the
environment of their dependents by overriding the
:meth:`setup_dependent_run_environment <spack.package.PackageBase.setup_dependent_run_environment>`
or the
:meth:`setup_dependent_build_environment <spack.package.PackageBase.setup_dependent_build_environment>`
methods.
The Qt package, for instance, uses this call:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/qt/package.py
:pyobject: Qt.setup_dependent_environment
:pyobject: Qt.setup_dependent_build_environment
:linenos:
Here, the Qt package sets the ``QTDIR`` environment variable so that
packages that depend on a particular Qt installation will find it.
The arguments to this function are:
* **spack_env**: List of environment modifications to be applied when
the dependent package is built within Spack.
* **run_env**: List of environment modifications to be applied when
the dependent package is run outside of Spack. These are added to the
resulting module file.
* **dependent_spec**: The spec of the dependent package about to be
built. This allows the extendee (self) to query the dependent's state.
Note that *this* package's spec is available as ``self.spec``.
A good example of using these is in the Python package:
to set the ``QTDIR`` environment variable so that packages
that depend on a particular Qt installation will find it.
Another good example of how a dependency can influence
the build environment of dependents is the Python package:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/python/package.py
:pyobject: Python.setup_dependent_environment
:pyobject: Python.setup_dependent_build_environment
:linenos:
The first thing that happens here is that the ``python`` command is
inserted into module scope of the dependent. This allows most python
packages to have a very simple install method, like this:
In the method above it is ensured that any package that depends on Python
will have the ``PYTHONPATH``, ``PYTHONHOME`` and ``PATH`` environment
variables set appropriately before starting the installation. To make things
even simpler the ``python setup.py`` command is also inserted into the module
scope of dependents by overriding a third method called
:meth:`setup_dependent_package <spack.package.PackageBase.setup_dependent_package>`
:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/python/package.py
:pyobject: Python.setup_dependent_package
:linenos:
This allows most python packages to have a very simple install procedure,
like the following:
.. code-block:: python
def install(self, spec, prefix):
python('setup.py', 'install', '--prefix={0}'.format(prefix))
setup_py('install', '--prefix={0}'.format(prefix))
Finally the Python package takes also care of the modifications to ``PYTHONPATH``
to allow dependencies to run correctly:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/python/package.py
:pyobject: Python.setup_dependent_run_environment
:linenos:
Python's ``setup_dependent_environment`` method also sets up some
other variables, creates a directory, and sets up the ``PYTHONPATH``
so that dependent packages can find their dependencies at build time.
.. _packaging_conflicts:

79
lib/spack/docs/tutorial_advanced_packaging.rst
View File

@@ -90,21 +90,23 @@ like py-numpy, Spack's ``python`` package will add it to ``PYTHONPATH``
so it is available at build time; this is required because the default setup
that spack does is not sufficient for python to import modules.
To provide environment setup for a dependent, a package can implement the
:py:func:`setup_dependent_environment <spack.package.PackageBase.setup_dependent_environment>`
function. This function takes as a parameter a :py:class:`EnvironmentModifications <spack.util.environment.EnvironmentModifications>`
Any package can override the
:py:func:`setup_dependent_build_environment <spack.package.PackageBase.setup_dependent_build_environment>`
method to setup the build environment for a dependent.
This method takes as an argument a :py:class:`EnvironmentModifications <spack.util.environment.EnvironmentModifications>`
object which includes convenience methods to update the environment. For
example, an MPI implementation can set ``MPICC`` for packages that depend on it:
.. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
spack_env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
def setup_dependent_build_environment(self, env, dependent_spec):
env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
In this case packages that depend on ``mpi`` will have ``MPICC`` defined in
their environment when they build. This section is focused on modifying the
build-time environment represented by ``spack_env``, but it's worth noting that
modifications to ``run_env`` are included in Spack's automatically-generated
their environment when they build. This section is focused on setting up the
build-time environment but it's worth noting that a similar method called
:py:func:`setup_dependent_run_environment <spack.package.PackageBase.setup_dependent_run_environment>`
can be used to code modifications that will be included in Spack's automatically-generated
module files.
We can practice by editing the ``mpich`` package to set the ``MPICC``
@@ -118,17 +120,17 @@ Once you're finished, the method should look like this:
.. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
spack_env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
spack_env.set('MPICXX', join_path(self.prefix.bin, 'mpic++'))
spack_env.set('MPIF77', join_path(self.prefix.bin, 'mpif77'))
spack_env.set('MPIF90', join_path(self.prefix.bin, 'mpif90'))
def setup_dependent_build_environment(self, env, dependent_spec):
env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
env.set('MPICXX', join_path(self.prefix.bin, 'mpic++'))
env.set('MPIF77', join_path(self.prefix.bin, 'mpif77'))
env.set('MPIF90', join_path(self.prefix.bin, 'mpif90'))
spack_env.set('MPICH_CC', spack_cc)
spack_env.set('MPICH_CXX', spack_cxx)
spack_env.set('MPICH_F77', spack_f77)
spack_env.set('MPICH_F90', spack_fc)
spack_env.set('MPICH_FC', spack_fc)
env.set('MPICH_CC', spack_cc)
env.set('MPICH_CXX', spack_cxx)
env.set('MPICH_F77', spack_f77)
env.set('MPICH_F90', spack_fc)
env.set('MPICH_FC', spack_fc)
At this point we can, for instance, install ``netlib-scalapack`` with
``mpich``:
@@ -155,25 +157,32 @@ set to the correct value.
Set environment variables in your own package
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Packages can modify their own build-time environment by implementing the
:py:func:`setup_environment <spack.package.PackageBase.setup_environment>` function.
For ``qt`` this looks like:
Packages can override the
:py:func:`setup_build_environment <spack.package.PackageBase.setup_build_environment>`
or the
:py:func:`setup_run_environment <spack.package.PackageBase.setup_run_environment>`
methods to modify their own build-time or run-time environment respectively.
An example of a package that overrides both methods is ``qt``:
.. code-block:: python
def setup_environment(self, spack_env, run_env):
spack_env.set('MAKEFLAGS', '-j{0}'.format(make_jobs))
run_env.set('QTDIR', self.prefix)
def setup_build_environment(self, env):
env.set('MAKEFLAGS', '-j{0}'.format(make_jobs))
When ``qt`` builds, ``MAKEFLAGS`` will be defined in the environment.
def setup_run_environment(self, env):
env.set('QTDIR', self.prefix)
To contrast with ``qt``'s :py:func:`setup_dependent_environment <spack.package.PackageBase.setup_dependent_environment>`
When ``qt`` builds, ``MAKEFLAGS`` will be defined in the environment. Likewise, when a
module file is created for ``qt`` it will contain commands to define ``QTDIR`` appropriately.
To contrast with ``qt``'s
:py:func:`setup_dependent_build_environment <spack.package.PackageBase.setup_dependent_build_environment>`
function:
.. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
spack_env.set('QTDIR', self.prefix)
def setup_dependent_build_environment(self, env, dependent_spec):
env.set('QTDIR', self.prefix)
Let's see how it works by completing the ``elpa`` package:
@@ -185,16 +194,16 @@ In the end your method should look like:
.. code-block:: python
def setup_environment(self, spack_env, run_env):
def setup_build_environment(self, env):
spec = self.spec
spack_env.set('CC', spec['mpi'].mpicc)
spack_env.set('FC', spec['mpi'].mpifc)
spack_env.set('CXX', spec['mpi'].mpicxx)
spack_env.set('SCALAPACK_LDFLAGS', spec['scalapack'].libs.joined())
env.set('CC', spec['mpi'].mpicc)
env.set('FC', spec['mpi'].mpifc)
env.set('CXX', spec['mpi'].mpicxx)
env.set('SCALAPACK_LDFLAGS', spec['scalapack'].libs.joined())
spack_env.append_flags('LDFLAGS', spec['lapack'].libs.search_flags)
spack_env.append_flags('LIBS', spec['lapack'].libs.link_flags)
env.append_flags('LDFLAGS', spec['lapack'].libs.search_flags)
env.append_flags('LIBS', spec['lapack'].libs.link_flags)
At this point it's possible to proceed with the installation of ``elpa ^mpich``