zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Separate setting build environment and run environment in packages (#11115)

Browse Source 
* Methods setting the environment now do it separately for build and run

Before this commit the `*_environment` methods were setting
modifications to both the build-time and run-time environment
simultaneously. This might cause issues as the two environments
inherently rely on different preconditions:

1. The build-time environment is set before building a package, thus
the package prefix doesn't exist and can't be inspected

2. The run-time environment instead is set assuming the target package
has been already installed

Here we split each of these functions into two: one setting the
build-time environment, one the run-time.

We also adopt a fallback strategy that inspects for old methods and
executes them as before, but prints a deprecation warning to tty. This
permits to port packages to use the new methods in a distributed way,
rather than having to modify all the packages at once.

* Added a test that fails if any package uses the old API

Marked the test xfail for now as we have a lot of packages in that
state.

* Added a test to check that a package modified by a PR is up to date

This test can be used any time we deprecate a method call to ensure
that during the first modification of the package we update also
the deprecated calls.

* Updated documentation
This commit is contained in:
Massimiliano Culpo 2019-10-17 19:17:21 +02:00 committed by Greg Becker
parent cf9de058aa
commit 9ddc98e46a
14 changed files with 337 additions and 219 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
lib/spack/docs/module_file_support.rst
View File

@ -303,8 +303,7 @@ content of the module files generated by Spack. The first one:
.. code-block:: python .. code-block:: python
def setup_environment(self, spack_env, run_env): def setup_run_environment(self, env):
"""Set up the compile and runtime environments for a package."""
pass pass
can alter the content of the module file associated with the same package where it is overridden. can alter the content of the module file associated with the same package where it is overridden.
@ -312,16 +311,15 @@ The second method:
.. code-block:: python .. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, dependent_spec): def setup_dependent_run_environment(self, env, dependent_spec):
"""Set up the environment of packages that depend on this one"""
pass pass
can instead inject run-time environment modifications in the module files of packages can instead inject run-time environment modifications in the module files of packages
that depend on it. In both cases you need to fill ``run_env`` with the desired that depend on it. In both cases you need to fill ``run_env`` with the desired
list of environment modifications. list of environment modifications.
.. note:: .. admonition:: The ``r`` package and callback APIs
The ``r`` package and callback APIs
An example in which it is crucial to override both methods An example in which it is crucial to override both methods
is given by the ``r`` package. This package installs libraries and headers is given by the ``r`` package. This package installs libraries and headers
in non-standard locations and it is possible to prepend the appropriate directory in non-standard locations and it is possible to prepend the appropriate directory
@ -336,14 +334,14 @@ list of environment modifications.
with the following snippet: with the following snippet:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/r/package.py .. literalinclude:: _spack_root/var/spack/repos/builtin/packages/r/package.py
:pyobject: R.setup_environment :pyobject: R.setup_run_environment
The ``r`` package also knows which environment variable should be modified The ``r`` package also knows which environment variable should be modified
to make language extensions provided by other packages available, and modifies to make language extensions provided by other packages available, and modifies
it appropriately in the override of the second method: it appropriately in the override of the second method:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/r/package.py .. literalinclude:: _spack_root/var/spack/repos/builtin/packages/r/package.py
:pyobject: R.setup_dependent_environment :pyobject: R.setup_dependent_run_environment
.. _modules-yaml: .. _modules-yaml:

73
lib/spack/docs/packaging_guide.rst
View File

@ -2032,55 +2032,58 @@ appear in the package file (or in this case, in the list).
.. _setup-dependent-environment: .. _setup-dependent-environment:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``setup_dependent_environment()`` Influence how dependents are built or run
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Spack provides a mechanism for dependencies to provide variables that Spack provides a mechanism for dependencies to influence the
can be used in their dependents' build. Any package can declare a environment of their dependents by overriding the
``setup_dependent_environment()`` function, and this function will be :meth:`setup_dependent_run_environment <spack.package.PackageBase.setup_dependent_run_environment>`
called before the ``install()`` method of any dependent packages. or the
This allows dependencies to set up environment variables and other :meth:`setup_dependent_build_environment <spack.package.PackageBase.setup_dependent_build_environment>`
properties to be used by dependents. methods.
The Qt package, for instance, uses this call:
The function declaration should look like this:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/qt/package.py .. literalinclude:: _spack_root/var/spack/repos/builtin/packages/qt/package.py
:pyobject: Qt.setup_dependent_environment :pyobject: Qt.setup_dependent_build_environment
:linenos: :linenos:
Here, the Qt package sets the ``QTDIR`` environment variable so that to set the ``QTDIR`` environment variable so that packages
packages that depend on a particular Qt installation will find it. that depend on a particular Qt installation will find it.
Another good example of how a dependency can influence
The arguments to this function are: the build environment of dependents is the Python package:
* **spack_env**: List of environment modifications to be applied when
the dependent package is built within Spack.
* **run_env**: List of environment modifications to be applied when
the dependent package is run outside of Spack. These are added to the
resulting module file.
* **dependent_spec**: The spec of the dependent package about to be
built. This allows the extendee (self) to query the dependent's state.
Note that *this* package's spec is available as ``self.spec``.
A good example of using these is in the Python package:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/python/package.py .. literalinclude:: _spack_root/var/spack/repos/builtin/packages/python/package.py
:pyobject: Python.setup_dependent_environment :pyobject: Python.setup_dependent_build_environment
:linenos: :linenos:
The first thing that happens here is that the ``python`` command is In the method above it is ensured that any package that depends on Python
inserted into module scope of the dependent. This allows most python will have the ``PYTHONPATH``, ``PYTHONHOME`` and ``PATH`` environment
packages to have a very simple install method, like this: variables set appropriately before starting the installation. To make things
even simpler the ``python setup.py`` command is also inserted into the module
scope of dependents by overriding a third method called
:meth:`setup_dependent_package <spack.package.PackageBase.setup_dependent_package>`
:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/python/package.py
:pyobject: Python.setup_dependent_package
:linenos:
This allows most python packages to have a very simple install procedure,
like the following:
.. code-block:: python .. code-block:: python
def install(self, spec, prefix): def install(self, spec, prefix):
python('setup.py', 'install', '--prefix={0}'.format(prefix)) setup_py('install', '--prefix={0}'.format(prefix))
Finally the Python package takes also care of the modifications to ``PYTHONPATH``
to allow dependencies to run correctly:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/python/package.py
:pyobject: Python.setup_dependent_run_environment
:linenos:
Python's ``setup_dependent_environment`` method also sets up some
other variables, creates a directory, and sets up the ``PYTHONPATH``
so that dependent packages can find their dependencies at build time.
.. _packaging_conflicts: .. _packaging_conflicts:

79
lib/spack/docs/tutorial_advanced_packaging.rst
View File

@ -90,21 +90,23 @@ like py-numpy, Spack's ``python`` package will add it to ``PYTHONPATH``
so it is available at build time; this is required because the default setup so it is available at build time; this is required because the default setup
that spack does is not sufficient for python to import modules. that spack does is not sufficient for python to import modules.
To provide environment setup for a dependent, a package can implement the Any package can override the
:py:func:`setup_dependent_environment <spack.package.PackageBase.setup_dependent_environment>` :py:func:`setup_dependent_build_environment <spack.package.PackageBase.setup_dependent_build_environment>`
function. This function takes as a parameter a :py:class:`EnvironmentModifications <spack.util.environment.EnvironmentModifications>` method to setup the build environment for a dependent.
This method takes as an argument a :py:class:`EnvironmentModifications <spack.util.environment.EnvironmentModifications>`
object which includes convenience methods to update the environment. For object which includes convenience methods to update the environment. For
example, an MPI implementation can set ``MPICC`` for packages that depend on it: example, an MPI implementation can set ``MPICC`` for packages that depend on it:
.. code-block:: python .. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, dependent_spec): def setup_dependent_build_environment(self, env, dependent_spec):
spack_env.set('MPICC', join_path(self.prefix.bin, 'mpicc')) env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
In this case packages that depend on ``mpi`` will have ``MPICC`` defined in In this case packages that depend on ``mpi`` will have ``MPICC`` defined in
their environment when they build. This section is focused on modifying the their environment when they build. This section is focused on setting up the
build-time environment represented by ``spack_env``, but it's worth noting that build-time environment but it's worth noting that a similar method called
modifications to ``run_env`` are included in Spack's automatically-generated :py:func:`setup_dependent_run_environment <spack.package.PackageBase.setup_dependent_run_environment>`
can be used to code modifications that will be included in Spack's automatically-generated
module files. module files.
We can practice by editing the ``mpich`` package to set the ``MPICC`` We can practice by editing the ``mpich`` package to set the ``MPICC``
@ -118,17 +120,17 @@ Once you're finished, the method should look like this:
.. code-block:: python .. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, dependent_spec): def setup_dependent_build_environment(self, env, dependent_spec):
spack_env.set('MPICC', join_path(self.prefix.bin, 'mpicc')) env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
spack_env.set('MPICXX', join_path(self.prefix.bin, 'mpic++')) env.set('MPICXX', join_path(self.prefix.bin, 'mpic++'))
spack_env.set('MPIF77', join_path(self.prefix.bin, 'mpif77')) env.set('MPIF77', join_path(self.prefix.bin, 'mpif77'))
spack_env.set('MPIF90', join_path(self.prefix.bin, 'mpif90')) env.set('MPIF90', join_path(self.prefix.bin, 'mpif90'))
spack_env.set('MPICH_CC', spack_cc) env.set('MPICH_CC', spack_cc)
spack_env.set('MPICH_CXX', spack_cxx) env.set('MPICH_CXX', spack_cxx)
spack_env.set('MPICH_F77', spack_f77) env.set('MPICH_F77', spack_f77)
spack_env.set('MPICH_F90', spack_fc) env.set('MPICH_F90', spack_fc)
spack_env.set('MPICH_FC', spack_fc) env.set('MPICH_FC', spack_fc)
At this point we can, for instance, install ``netlib-scalapack`` with At this point we can, for instance, install ``netlib-scalapack`` with
``mpich``: ``mpich``:
@ -155,25 +157,32 @@ set to the correct value.
Set environment variables in your own package Set environment variables in your own package
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Packages can modify their own build-time environment by implementing the Packages can override the
:py:func:`setup_environment <spack.package.PackageBase.setup_environment>` function. :py:func:`setup_build_environment <spack.package.PackageBase.setup_build_environment>`
For ``qt`` this looks like: or the
:py:func:`setup_run_environment <spack.package.PackageBase.setup_run_environment>`
methods to modify their own build-time or run-time environment respectively.
An example of a package that overrides both methods is ``qt``:
.. code-block:: python .. code-block:: python
def setup_environment(self, spack_env, run_env): def setup_build_environment(self, env):
spack_env.set('MAKEFLAGS', '-j{0}'.format(make_jobs)) env.set('MAKEFLAGS', '-j{0}'.format(make_jobs))
run_env.set('QTDIR', self.prefix)
When ``qt`` builds, ``MAKEFLAGS`` will be defined in the environment. def setup_run_environment(self, env):
env.set('QTDIR', self.prefix)
To contrast with ``qt``'s :py:func:`setup_dependent_environment <spack.package.PackageBase.setup_dependent_environment>` When ``qt`` builds, ``MAKEFLAGS`` will be defined in the environment. Likewise, when a
module file is created for ``qt`` it will contain commands to define ``QTDIR`` appropriately.
To contrast with ``qt``'s
:py:func:`setup_dependent_build_environment <spack.package.PackageBase.setup_dependent_build_environment>`
function: function:
.. code-block:: python .. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, dependent_spec): def setup_dependent_build_environment(self, env, dependent_spec):
spack_env.set('QTDIR', self.prefix) env.set('QTDIR', self.prefix)
Let's see how it works by completing the ``elpa`` package: Let's see how it works by completing the ``elpa`` package:
@ -185,16 +194,16 @@ In the end your method should look like:
.. code-block:: python .. code-block:: python
def setup_environment(self, spack_env, run_env): def setup_build_environment(self, env):
spec = self.spec spec = self.spec
spack_env.set('CC', spec['mpi'].mpicc) env.set('CC', spec['mpi'].mpicc)
spack_env.set('FC', spec['mpi'].mpifc) env.set('FC', spec['mpi'].mpifc)
spack_env.set('CXX', spec['mpi'].mpicxx) env.set('CXX', spec['mpi'].mpicxx)
spack_env.set('SCALAPACK_LDFLAGS', spec['scalapack'].libs.joined()) env.set('SCALAPACK_LDFLAGS', spec['scalapack'].libs.joined())
spack_env.append_flags('LDFLAGS', spec['lapack'].libs.search_flags) env.append_flags('LDFLAGS', spec['lapack'].libs.search_flags)
spack_env.append_flags('LIBS', spec['lapack'].libs.link_flags) env.append_flags('LIBS', spec['lapack'].libs.link_flags)
At this point it's possible to proceed with the installation of ``elpa ^mpich`` At this point it's possible to proceed with the installation of ``elpa ^mpich``

64
lib/spack/spack/build_environment.py
View File

@ -673,35 +673,26 @@ def load_external_modules(pkg):
def setup_package(pkg, dirty): def setup_package(pkg, dirty):
"""Execute all environment setup routines.""" """Execute all environment setup routines."""
spack_env = EnvironmentModifications() build_env = EnvironmentModifications()
run_env = EnvironmentModifications()
if not dirty: if not dirty:
clean_environment() clean_environment()
set_compiler_environment_variables(pkg, spack_env) set_compiler_environment_variables(pkg, build_env)
set_build_environment_variables(pkg, spack_env, dirty) set_build_environment_variables(pkg, build_env, dirty)
pkg.architecture.platform.setup_platform_environment(pkg, spack_env) pkg.architecture.platform.setup_platform_environment(pkg, build_env)
# traverse in postorder so package can use vars from its dependencies build_env.extend(
spec = pkg.spec modifications_from_dependencies(pkg.spec, context='build')
for dspec in pkg.spec.traverse(order='post', root=False, )
deptype=('build', 'test')):
spkg = dspec.package
set_module_variables_for_package(spkg)
# Allow dependencies to modify the module if (not dirty) and (not build_env.is_unset('CPATH')):
dpkg = dspec.package
dpkg.setup_dependent_package(pkg.module, spec)
dpkg.setup_dependent_environment(spack_env, run_env, spec)
if (not dirty) and (not spack_env.is_unset('CPATH')):
tty.debug("A dependency has updated CPATH, this may lead pkg-config" tty.debug("A dependency has updated CPATH, this may lead pkg-config"
" to assume that the package is part of the system" " to assume that the package is part of the system"
" includes and omit it when invoked with '--cflags'.") " includes and omit it when invoked with '--cflags'.")
set_module_variables_for_package(pkg) set_module_variables_for_package(pkg)
pkg.setup_environment(spack_env, run_env) pkg.setup_build_environment(build_env)
# Loading modules, in particular if they are meant to be used outside # Loading modules, in particular if they are meant to be used outside
# of Spack, can change environment variables that are relevant to the # of Spack, can change environment variables that are relevant to the
@ -711,7 +702,7 @@ def setup_package(pkg, dirty):
# unnecessary. Modules affecting these variables will be overwritten anyway # unnecessary. Modules affecting these variables will be overwritten anyway
with preserve_environment('CC', 'CXX', 'FC', 'F77'): with preserve_environment('CC', 'CXX', 'FC', 'F77'):
# All module loads that otherwise would belong in previous # All module loads that otherwise would belong in previous
# functions have to occur after the spack_env object has its # functions have to occur after the build_env object has its
# modifications applied. Otherwise the environment modifications # modifications applied. Otherwise the environment modifications
# could undo module changes, such as unsetting LD_LIBRARY_PATH # could undo module changes, such as unsetting LD_LIBRARY_PATH
# after a module changes it. # after a module changes it.
@ -727,8 +718,39 @@ def setup_package(pkg, dirty):
load_external_modules(pkg) load_external_modules(pkg)
# Make sure nothing's strange about the Spack environment. # Make sure nothing's strange about the Spack environment.
validate(spack_env, tty.warn) validate(build_env, tty.warn)
spack_env.apply_modifications() build_env.apply_modifications()
def modifications_from_dependencies(spec, context):
"""Returns the environment modifications that are required by
the dependencies of a spec and also applies modifications
to this spec's package at module scope, if need be.
Args:
spec (Spec): spec for which we want the modifications
context (str): either 'build' for build-time modifications or 'run'
for run-time modifications
"""
env = EnvironmentModifications()
pkg = spec.package
# Maps the context to deptype and method to be called
deptype_and_method = {
'build': (('build', 'link', 'test'),
'setup_dependent_build_environment'),
'run': (('link', 'run'), 'setup_dependent_run_environment')
}
deptype, method = deptype_and_method[context]
for dspec in spec.traverse(order='post', root=False, deptype=deptype):
dpkg = dspec.package
set_module_variables_for_package(dpkg)
# Allow dependencies to modify the module
dpkg.setup_dependent_package(pkg.module, spec)
getattr(dpkg, method)(env, spec)
return env
def fork(pkg, function, dirty, fake): def fork(pkg, function, dirty, fake):

9
lib/spack/spack/cmd/flake8.py
View File

@ -93,12 +93,11 @@ def is_package(f):
for file_pattern, error_dict in pattern_exemptions.items()) for file_pattern, error_dict in pattern_exemptions.items())
def changed_files(args): def changed_files(base=None, untracked=True, all_files=False):
"""Get list of changed files in the Spack repository.""" """Get list of changed files in the Spack repository."""
git = which('git', required=True) git = which('git', required=True)
base = args.base
if base is None: if base is None:
base = os.environ.get('TRAVIS_BRANCH', 'develop') base = os.environ.get('TRAVIS_BRANCH', 'develop')
@ -114,11 +113,11 @@ def changed_files(args):
] ]
# Add new files that are untracked # Add new files that are untracked
if args.untracked: if untracked:
git_args.append(['ls-files', '--exclude-standard', '--other']) git_args.append(['ls-files', '--exclude-standard', '--other'])
# add everything if the user asked for it # add everything if the user asked for it
if args.all: if all_files:
git_args.append(['ls-files', '--exclude-standard']) git_args.append(['ls-files', '--exclude-standard'])
excludes = [os.path.realpath(f) for f in exclude_directories] excludes = [os.path.realpath(f) for f in exclude_directories]
@ -246,7 +245,7 @@ def prefix_relative(path):
with working_dir(spack.paths.prefix): with working_dir(spack.paths.prefix):
if not file_list: if not file_list:
file_list = changed_files(args) file_list = changed_files(args.base, args.untracked, args.all)
print('=======================================================') print('=======================================================')
print('flake8: running flake8 code checks on spack.') print('flake8: running flake8 code checks on spack.')

17
lib/spack/spack/modules/common.py
View File

@ -634,23 +634,16 @@ def environment_modifications(self):
exclude=spack.util.environment.is_system_path exclude=spack.util.environment.is_system_path
) )
# Modifications that are coded at package level
_ = spack.util.environment.EnvironmentModifications()
# TODO : the code down below is quite similar to
# TODO : build_environment.setup_package and needs to be factored out
# TODO : to a single place
# Let the extendee/dependency modify their extensions/dependencies # Let the extendee/dependency modify their extensions/dependencies
# before asking for package-specific modifications # before asking for package-specific modifications
for item in dependencies(self.spec, 'all'): env.extend(
package = self.spec[item.name].package build_environment.modifications_from_dependencies(
build_environment.set_module_variables_for_package(package) self.spec, context='run'
package.setup_dependent_package(
self.spec.package.module, self.spec
) )
package.setup_dependent_environment(_, env, self.spec) )
# Package specific modifications # Package specific modifications
build_environment.set_module_variables_for_package(self.spec.package) build_environment.set_module_variables_for_package(self.spec.package)
self.spec.package.setup_environment(_, env) self.spec.package.setup_run_environment(env)
# Modifications required from modules.yaml # Modifications required from modules.yaml
env.extend(self.conf.env) env.extend(self.conf.env)

143
lib/spack/spack/package.py
View File

@ -45,6 +45,7 @@
import spack.multimethod import spack.multimethod
import spack.repo import spack.repo
import spack.url import spack.url
import spack.util.environment
import spack.util.web import spack.util.web
import spack.multimethod import spack.multimethod
import spack.binary_distribution as binary_distribution import spack.binary_distribution as binary_distribution
@ -1951,68 +1952,108 @@ def build_system_flags(cls, name, flags):
""" """
return (None, None, flags) return (None, None, flags)
def setup_environment(self, spack_env, run_env): def _get_legacy_environment_method(self, method_name):
"""Set up the compile and runtime environments for a package. legacy_fn = getattr(self, method_name, None)
name_prefix = method_name.split('_environment')[0]
if legacy_fn:
msg = '[DEPRECATED METHOD]\n"{0}" ' \
'still defines the deprecated method "{1}" ' \
'[should be split into "{2}_build_environment" and ' \
'"{2}_run_environment"]'
tty.debug(msg.format(self.name, method_name, name_prefix))
return legacy_fn
``spack_env`` and ``run_env`` are ``EnvironmentModifications`` def setup_build_environment(self, env):
objects. Package authors can call methods on them to alter """Sets up the build environment for a package.
the environment within Spack and at runtime.
Both ``spack_env`` and ``run_env`` are applied within the build This method will be called before the current package prefix exists in
process, before this package's ``install()`` method is called. Spack's store.
Modifications in ``run_env`` will *also* be added to the
generated environment modules for this package.
Default implementation does nothing, but this can be
overridden if the package needs a particular environment.
Example:
1. Qt extensions need ``QTDIR`` set.
Args: Args:
spack_env (EnvironmentModifications): List of environment env (EnvironmentModifications): environment modifications to be
modifications to be applied when this package is built applied when the package is built. Package authors can call
within Spack. methods on it to alter the build environment.
run_env (EnvironmentModifications): List of environment
modifications to be applied when this package is run outside
of Spack. These are added to the resulting module file.
""" """
pass legacy_fn = self._get_legacy_environment_method('setup_environment')
if legacy_fn:
_ = spack.util.environment.EnvironmentModifications()
legacy_fn(env, _)
def setup_dependent_environment(self, spack_env, run_env, dependent_spec): def setup_run_environment(self, env):
"""Set up the environment of packages that depend on this one. """Sets up the run environment for a package.
This is similar to ``setup_environment``, but it is used to
modify the compile and runtime environments of packages that
*depend* on this one. This gives packages like Python and
others that follow the extension model a way to implement
common environment or compile-time settings for dependencies.
This is useful if there are some common steps to installing
all extensions for a certain package.
Example:
1. Installing python modules generally requires ``PYTHONPATH`` to point
to the ``lib/pythonX.Y/site-packages`` directory in the module's
install prefix. This method could be used to set that variable.
Args: Args:
spack_env (EnvironmentModifications): List of environment env (EnvironmentModifications): environment modifications to be
modifications to be applied when the dependent package is applied when the package is run. Package authors can call
built within Spack. methods on it to alter the run environment.
run_env (EnvironmentModifications): List of environment """
modifications to be applied when the dependent package is legacy_fn = self._get_legacy_environment_method('setup_environment')
run outside of Spack. These are added to the resulting if legacy_fn:
module file. _ = spack.util.environment.EnvironmentModifications()
dependent_spec (Spec): The spec of the dependent package legacy_fn(_, env)
def setup_dependent_build_environment(self, env, dependent_spec):
"""Sets up the build environment of packages that depend on this one.
This is similar to ``setup_build_environment``, but it is used to
modify the build environments of packages that *depend* on this one.
This gives packages like Python and others that follow the extension
model a way to implement common environment or compile-time settings
for dependencies.
This method will be called before the dependent package prefix exists
in Spack's store.
Examples:
1. Installing python modules generally requires ``PYTHONPATH``
to point to the ``lib/pythonX.Y/site-packages`` directory in the
module's install prefix. This method could be used to set that
variable.
Args:
env (EnvironmentModifications): environment modifications to be
applied when the dependent package is built. Package authors
can call methods on it to alter the build environment.
dependent_spec (Spec): the spec of the dependent package
about to be built. This allows the extendee (self) to query about to be built. This allows the extendee (self) to query
the dependent's state. Note that *this* package's spec is the dependent's state. Note that *this* package's spec is
available as ``self.spec``. available as ``self.spec``
""" """
pass legacy_fn = self._get_legacy_environment_method(
'setup_dependent_environment'
)
if legacy_fn:
_ = spack.util.environment.EnvironmentModifications()
legacy_fn(env, _, dependent_spec)
def setup_dependent_run_environment(self, env, dependent_spec):
"""Sets up the run environment of packages that depend on this one.
This is similar to ``setup_run_environment``, but it is used to
modify the run environments of packages that *depend* on this one.
This gives packages like Python and others that follow the extension
model a way to implement common environment or run-time settings
for dependencies.
Args:
env (EnvironmentModifications): environment modifications to be
applied when the dependent package is run. Package authors
can call methods on it to alter the build environment.
dependent_spec (Spec): The spec of the dependent package
about to be run. This allows the extendee (self) to query
the dependent's state. Note that *this* package's spec is
available as ``self.spec``
"""
legacy_fn = self._get_legacy_environment_method(
'setup_dependent_environment'
)
if legacy_fn:
_ = spack.util.environment.EnvironmentModifications()
legacy_fn(_, env, dependent_spec)
def setup_dependent_package(self, module, dependent_spec): def setup_dependent_package(self, module, dependent_spec):
"""Set up Python module-scope variables for dependent packages. """Set up Python module-scope variables for dependent packages.

50
lib/spack/spack/test/package_sanity.py
View File

@ -4,6 +4,7 @@
# SPDX-License-Identifier: (Apache-2.0 OR MIT) # SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""This test does sanity checks on Spack's builtin package database.""" """This test does sanity checks on Spack's builtin package database."""
import os.path
import re import re
import pytest import pytest
@ -11,6 +12,10 @@
import spack.fetch_strategy import spack.fetch_strategy
import spack.paths import spack.paths
import spack.repo import spack.repo
import spack.util.executable as executable
# A few functions from this module are used to
# do sanity checks only on packagess modified by a PR
import spack.cmd.flake8 as flake8
import spack.util.crypto as crypto import spack.util.crypto as crypto
@ -134,3 +139,48 @@ def invalid_sha256_digest(fetcher):
) )
assert [] == errors assert [] == errors
@pytest.mark.xfail
def test_api_for_build_and_run_environment():
"""Ensure that every package uses the correct API to set build and
run environment, and not the old one.
"""
failing = []
for pkg in spack.repo.path.all_packages():
add_to_list = (hasattr(pkg, 'setup_environment') or
hasattr(pkg, 'setup_dependent_environment'))
if add_to_list:
failing.append(pkg)
msg = ('there are {0} packages using the old API to set build '
'and run environment [{1}]')
assert not failing, msg.format(
len(failing), ','.join(x.name for x in failing)
)
@pytest.mark.skipif(
not executable.which('git'), reason='requires git to be installed'
)
def test_prs_update_old_api():
"""Ensures that every package modified in a PR doesn't contain
deprecated calls to any method.
"""
changed_package_files = [
x for x in flake8.changed_files() if flake8.is_package(x)
]
failing = []
for file in changed_package_files:
name = os.path.basename(os.path.dirname(file))
pkg = spack.repo.get(name)
# Check for old APIs
failed = (hasattr(pkg, 'setup_environment') or
hasattr(pkg, 'setup_dependent_environment'))
if failed:
failing.append(pkg)
msg = 'there are {0} packages still using old APIs in this PR [{1}]'
assert not failing, msg.format(
len(failing), ','.join(x.name for x in failing)
)

4
var/spack/repos/builtin/packages/docbook-xml/package.py
View File

@ -20,6 +20,6 @@ class DocbookXml(Package):
def install(self, spec, prefix): def install(self, spec, prefix):
install_tree('.', prefix) install_tree('.', prefix)
def setup_environment(self, spack_env, run_env): def setup_run_environment(self, env):
catalog = os.path.join(self.prefix, 'catalog.xml') catalog = os.path.join(self.prefix, 'catalog.xml')
run_env.set('XML_CATALOG_FILES', catalog, separator=' ') env.set('XML_CATALOG_FILES', catalog, separator=' ')

14
var/spack/repos/builtin/packages/elpa/package.py
View File

@ -61,16 +61,16 @@ def headers(self):
build_directory = 'spack-build' build_directory = 'spack-build'
def setup_environment(self, spack_env, run_env): def setup_build_environment(self, env):
spec = self.spec spec = self.spec
spack_env.set('CC', spec['mpi'].mpicc) env.set('CC', spec['mpi'].mpicc)
spack_env.set('FC', spec['mpi'].mpifc) env.set('FC', spec['mpi'].mpifc)
spack_env.set('CXX', spec['mpi'].mpicxx) env.set('CXX', spec['mpi'].mpicxx)
spack_env.append_flags('LDFLAGS', spec['lapack'].libs.search_flags) env.append_flags('LDFLAGS', spec['lapack'].libs.search_flags)
spack_env.append_flags('LIBS', spec['lapack'].libs.link_flags) env.append_flags('LIBS', spec['lapack'].libs.link_flags)
spack_env.set('SCALAPACK_LDFLAGS', spec['scalapack'].libs.joined()) env.set('SCALAPACK_LDFLAGS', spec['scalapack'].libs.joined())
def configure_args(self): def configure_args(self):
# TODO: set optimum flags for platform+compiler combo, see # TODO: set optimum flags for platform+compiler combo, see

35
var/spack/repos/builtin/packages/mpich/package.py
View File

@ -110,29 +110,28 @@ class Mpich(AutotoolsPackage):
conflicts('pmi=pmi2', when='device=ch3 netmod=ofi') conflicts('pmi=pmi2', when='device=ch3 netmod=ofi')
conflicts('pmi=pmix', when='device=ch3') conflicts('pmi=pmix', when='device=ch3')
def setup_environment(self, spack_env, run_env): def setup_build_environment(self, env):
# mpich configure fails when F90 and F90FLAGS are set env.unset('F90')
spack_env.unset('F90') env.unset('F90FLAGS')
spack_env.unset('F90FLAGS')
def setup_dependent_environment(self, spack_env, run_env, dependent_spec): def setup_dependent_build_environment(self, env, dependent_spec):
# On Cray, the regular compiler wrappers *are* the MPI wrappers. # On Cray, the regular compiler wrappers *are* the MPI wrappers.
if 'platform=cray' in self.spec: if 'platform=cray' in self.spec:
spack_env.set('MPICC', spack_cc) env.set('MPICC', spack_cc)
spack_env.set('MPICXX', spack_cxx) env.set('MPICXX', spack_cxx)
spack_env.set('MPIF77', spack_fc) env.set('MPIF77', spack_fc)
spack_env.set('MPIF90', spack_fc) env.set('MPIF90', spack_fc)
else: else:
spack_env.set('MPICC', join_path(self.prefix.bin, 'mpicc')) env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
spack_env.set('MPICXX', join_path(self.prefix.bin, 'mpic++')) env.set('MPICXX', join_path(self.prefix.bin, 'mpic++'))
spack_env.set('MPIF77', join_path(self.prefix.bin, 'mpif77')) env.set('MPIF77', join_path(self.prefix.bin, 'mpif77'))
spack_env.set('MPIF90', join_path(self.prefix.bin, 'mpif90')) env.set('MPIF90', join_path(self.prefix.bin, 'mpif90'))
spack_env.set('MPICH_CC', spack_cc) env.set('MPICH_CC', spack_cc)
spack_env.set('MPICH_CXX', spack_cxx) env.set('MPICH_CXX', spack_cxx)
spack_env.set('MPICH_F77', spack_f77) env.set('MPICH_F77', spack_f77)
spack_env.set('MPICH_F90', spack_fc) env.set('MPICH_F90', spack_fc)
spack_env.set('MPICH_FC', spack_fc) env.set('MPICH_FC', spack_fc)
def setup_dependent_package(self, module, dependent_spec): def setup_dependent_package(self, module, dependent_spec):
if 'platform=cray' in self.spec: if 'platform=cray' in self.spec:

15
var/spack/repos/builtin/packages/python/package.py
View File

@ -183,7 +183,7 @@ def patch(self):
r'\1setup.py\2 --no-user-cfg \3\6' r'\1setup.py\2 --no-user-cfg \3\6'
) )
def setup_environment(self, spack_env, run_env): def setup_build_environment(self, env):
spec = self.spec spec = self.spec
# TODO: The '--no-user-cfg' option for Python installation is only in # TODO: The '--no-user-cfg' option for Python installation is only in
@ -195,7 +195,7 @@ def setup_environment(self, spack_env, run_env):
'user configurations are present.').format(self.version)) 'user configurations are present.').format(self.version))
# Need this to allow python build to find the Python installation. # Need this to allow python build to find the Python installation.
spack_env.set('MACOSX_DEPLOYMENT_TARGET', platform.mac_ver()[0]) env.set('MACOSX_DEPLOYMENT_TARGET', platform.mac_ver()[0])
def configure_args(self): def configure_args(self):
spec = self.spec spec = self.spec
@ -672,7 +672,7 @@ def site_packages_dir(self):
def easy_install_file(self): def easy_install_file(self):
return join_path(self.site_packages_dir, "easy-install.pth") return join_path(self.site_packages_dir, "easy-install.pth")
def setup_dependent_environment(self, spack_env, run_env, dependent_spec): def setup_dependent_build_environment(self, env, dependent_spec):
"""Set PYTHONPATH to include the site-packages directory for the """Set PYTHONPATH to include the site-packages directory for the
extension and any other python extensions it depends on.""" extension and any other python extensions it depends on."""
@ -680,11 +680,11 @@ def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
# python is found in the build environment. This to prevent cases # python is found in the build environment. This to prevent cases
# where a system provided python is run against the standard libraries # where a system provided python is run against the standard libraries
# of a Spack built python. See issue #7128 # of a Spack built python. See issue #7128
spack_env.set('PYTHONHOME', self.home) env.set('PYTHONHOME', self.home)
path = os.path.dirname(self.command.path) path = os.path.dirname(self.command.path)
if not is_system_path(path): if not is_system_path(path):
spack_env.prepend_path('PATH', path) env.prepend_path('PATH', path)
python_paths = [] python_paths = []
for d in dependent_spec.traverse( for d in dependent_spec.traverse(
@ -694,12 +694,13 @@ def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
self.site_packages_dir)) self.site_packages_dir))
pythonpath = ':'.join(python_paths) pythonpath = ':'.join(python_paths)
spack_env.set('PYTHONPATH', pythonpath) env.set('PYTHONPATH', pythonpath)
def setup_dependent_run_environment(self, env, dependent_spec):
# For run time environment set only the path for # For run time environment set only the path for
# dependent_spec and prepend it to PYTHONPATH # dependent_spec and prepend it to PYTHONPATH
if dependent_spec.package.extends(self.spec): if dependent_spec.package.extends(self.spec):
run_env.prepend_path('PYTHONPATH', join_path( env.prepend_path('PYTHONPATH', join_path(
dependent_spec.prefix, self.site_packages_dir)) dependent_spec.prefix, self.site_packages_dir))
def setup_dependent_package(self, module, dependent_spec): def setup_dependent_package(self, module, dependent_spec):

12
var/spack/repos/builtin/packages/qt/package.py
View File

@ -206,12 +206,14 @@ def url_for_version(self, version):
return url return url
def setup_environment(self, spack_env, run_env): def setup_build_environment(self, env):
spack_env.set('MAKEFLAGS', '-j{0}'.format(make_jobs)) env.set('MAKEFLAGS', '-j{0}'.format(make_jobs))
run_env.set('QTDIR', self.prefix)
def setup_dependent_environment(self, spack_env, run_env, dependent_spec): def setup_run_environment(self, env):
spack_env.set('QTDIR', self.prefix) env.set('QTDIR', self.prefix)
def setup_dependent_build_environment(self, env, dependent_spec):
env.set('QTDIR', self.prefix)
def setup_dependent_package(self, module, dependent_spec): def setup_dependent_package(self, module, dependent_spec):
module.qmake = Executable(join_path(self.spec.prefix.bin, 'qmake')) module.qmake = Executable(join_path(self.spec.prefix.bin, 'qmake'))

27
var/spack/repos/builtin/packages/r/package.py
View File

@ -168,7 +168,7 @@ def copy_makeconf(self):
def r_lib_dir(self): def r_lib_dir(self):
return join_path('rlib', 'R', 'library') return join_path('rlib', 'R', 'library')
def setup_dependent_environment(self, spack_env, run_env, dependent_spec): def setup_dependent_build_environment(self, env, dependent_spec):
# Set R_LIBS to include the library dir for the # Set R_LIBS to include the library dir for the
# extension and any other R extensions it depends on. # extension and any other R extensions it depends on.
r_libs_path = [] r_libs_path = []
@ -178,27 +178,28 @@ def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
r_libs_path.append(join_path(d.prefix, self.r_lib_dir)) r_libs_path.append(join_path(d.prefix, self.r_lib_dir))
r_libs_path = ':'.join(r_libs_path) r_libs_path = ':'.join(r_libs_path)
spack_env.set('R_LIBS', r_libs_path) env.set('R_LIBS', r_libs_path)
spack_env.set('R_MAKEVARS_SITE', env.set('R_MAKEVARS_SITE',
join_path(self.etcdir, 'Makeconf.spack')) join_path(self.etcdir, 'Makeconf.spack'))
# Use the number of make_jobs set in spack. The make program will # Use the number of make_jobs set in spack. The make program will
# determine how many jobs can actually be started. # determine how many jobs can actually be started.
spack_env.set('MAKEFLAGS', '-j{0}'.format(make_jobs)) env.set('MAKEFLAGS', '-j{0}'.format(make_jobs))
def setup_dependent_run_environment(self, env, dependent_spec):
# For run time environment set only the path for dependent_spec and # For run time environment set only the path for dependent_spec and
# prepend it to R_LIBS # prepend it to R_LIBS
if dependent_spec.package.extends(self.spec): if dependent_spec.package.extends(self.spec):
run_env.prepend_path('R_LIBS', join_path( env.prepend_path('R_LIBS', join_path(
dependent_spec.prefix, self.r_lib_dir)) dependent_spec.prefix, self.r_lib_dir))
def setup_environment(self, spack_env, run_env): def setup_run_environment(self, env):
run_env.prepend_path('LIBRARY_PATH', env.prepend_path('LIBRARY_PATH',
join_path(self.prefix, 'rlib', 'R', 'lib')) join_path(self.prefix, 'rlib', 'R', 'lib'))
run_env.prepend_path('LD_LIBRARY_PATH', env.prepend_path('LD_LIBRARY_PATH',
join_path(self.prefix, 'rlib', 'R', 'lib')) join_path(self.prefix, 'rlib', 'R', 'lib'))
run_env.prepend_path('CPATH', env.prepend_path('CPATH',
join_path(self.prefix, 'rlib', 'R', 'include')) join_path(self.prefix, 'rlib', 'R', 'include'))
def setup_dependent_package(self, module, dependent_spec): def setup_dependent_package(self, module, dependent_spec):
"""Called before R modules' install() methods. In most cases, """Called before R modules' install() methods. In most cases,