zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

shell support: spack load no longer needs modules (#14062)

Browse Source 
Previously the `spack load` command was a wrapper around `module load`. This required some bootstrapping of modules to make `spack load` work properly.

With this PR, the `spack` shell function handles the environment modifications necessary to add packages to your user environment. This removes the dependence on environment modules or lmod and removes the requirement to bootstrap spack (beyond using the setup-env scripts).

Included in this PR is support for MacOS when using Apple's System Integrity Protection (SIP), which is enabled by default in modern MacOS versions. SIP clears the `LD_LIBRARY_PATH` and `DYLD_LIBRARY_PATH` variables on process startup for executables that live in `/usr` (but not '/usr/local', `/System`, `/bin`, and `/sbin` among other system locations. Spack cannot know the `LD_LIBRARY_PATH` of the calling process when executed using `/bin/sh` and `/usr/bin/python`. The `spack` shell function now manually forwards these two variables, if they are present, as `SPACK_<VAR>` and recovers those values on startup.

- [x] spack load/unload no longer delegate to modules
- [x] refactor user_environment modification calculations
- [x] update documentation for spack load/unload

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
This commit is contained in:
Greg Becker
2020-01-22 22:36:02 -08:00
committed by Todd Gamblin
parent 5053dfa259
commit c9e01ff9d7
17 changed files with 516 additions and 184 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
lib/spack/docs/basic_usage.rst
View File

@@ -1317,10 +1317,9 @@ directly when you run ``python``:
Using Extensions
^^^^^^^^^^^^^^^^
There are three ways to get ``numpy`` working in Python. The first is
to use :ref:`shell-support`. You can simply ``load`` the
module for the extension, and it will be added to the ``PYTHONPATH``
in your current shell:
There are four ways to get ``numpy`` working in Python. The first is
to use :ref:`shell-support`. You can simply ``load`` the extension,
and it will be added to the ``PYTHONPATH`` in your current shell:
.. code-block:: console
@@ -1330,11 +1329,29 @@ in your current shell:
Now ``import numpy`` will succeed for as long as you keep your current
session open.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Loading Extensions via Modules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Instead of using Spack's environment modification capabilities through
the ``spack load`` command, you can load numpy through your
environment modules (using ``environment-modules`` or ``lmod``). This
will also add the extension to the ``PYTHONPATH`` in your current
shell.
.. code-block:: console
$ module load <name of numpy module>
If you do not know the name of the specific numpy module you wish to
load, you can use the ``spack module tcl|lmod loads`` command to get
the name of the module from the Spack spec.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Activating Extensions in a View
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The second way to use extensions is to create a view, which merges the
Another way to use extensions is to create a view, which merges the
python installation along with the extensions into a single prefix.
See :ref:`filesystem-views` for a more in-depth description of views and
:ref:`cmd-spack-view` for usage of the ``spack view`` command.

39
lib/spack/docs/module_file_support.rst
View File

@@ -119,7 +119,7 @@ For example this will add the ``mpich`` package built with ``gcc`` to your path:
# ... wait for install ...
$ spack load mpich %gcc@4.4.7 # modules
$ spack load mpich %gcc@4.4.7
$ which mpicc
~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpich@3.0.4/bin/mpicc
@@ -129,27 +129,29 @@ want to use a package, you can type unload or unuse similarly:
.. code-block:: console
$ spack unload mpich %gcc@4.4.7 # modules
$ spack unload mpich %gcc@4.4.7
.. note::
The ``load`` and ``unload`` subcommands are
only available if you have enabled Spack's shell support *and* you
have environment-modules installed on your machine.
The ``load`` and ``unload`` subcommands are only available if you
have enabled Spack's shell support. These command DO NOT use the
underlying Spack-generated module files.
^^^^^^^^^^^^^^^^^^^^^^
Ambiguous module names
^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^
Ambiguous specs
^^^^^^^^^^^^^^^
If a spec used with load/unload or use/unuse is ambiguous (i.e. more
than one installed package matches it), then Spack will warn you:
If a spec used with load/unload or is ambiguous (i.e. more than one
installed package matches it), then Spack will warn you:
.. code-block:: console
$ spack load libelf
==> Error: Multiple matches for spec libelf. Choose one:
libelf@0.8.13%gcc@4.4.7 arch=linux-debian7-x86_64
libelf@0.8.13%intel@15.0.0 arch=linux-debian7-x86_64
==> Error: libelf matches multiple packages.
Matching packages:
libelf@0.8.13%gcc@4.4.7 arch=linux-debian7-x86_64
libelf@0.8.13%intel@15.0.0 arch=linux-debian7-x86_64
Use a more specific spec
You can either type the ``spack load`` command again with a fully
qualified argument, or you can add just enough extra constraints to
@@ -171,8 +173,15 @@ To identify just the one built with the Intel compiler.
``spack module tcl loads``
^^^^^^^^^^^^^^^^^^^^^^^^^^
In some cases, it is desirable to load not just a module, but also all
the modules it depends on. This is not required for most modules
In some cases, it is desirable to use a Spack-generated module, rather
than relying on Spack's built-in user-environment modification
capabilities. To translate a spec into a module name, use ``spack
module tcl loads`` or ``spack module lmod loads`` depending on the
module system desired.
To load not just a module, but also all the modules it depends on, use
the ``--dependencies`` option. This is not required for most modules
because Spack builds binaries with RPATH support. However, not all
packages use RPATH to find their dependencies: this can be true in
particular for Python extensions, which are currently *not* built with

33
lib/spack/docs/workflows.rst
View File

@@ -253,14 +253,14 @@ However, other more powerful methods are generally preferred for user
environments.
^^^^^^^^^^^^^^^^^^^^^^^
Spack-Generated Modules
^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Using ``spack load`` to Manage the User Environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Suppose that Spack has been used to install a set of command-line
programs, which users now wish to use. One can in principle put a
number of ``spack load`` commands into ``.bashrc``, for example, to
load a set of Spack-generated modules:
load a set of Spack packages:
.. code-block:: sh
@@ -273,7 +273,7 @@ load a set of Spack-generated modules:
Although simple load scripts like this are useful in many cases, they
have some drawbacks:
1. The set of modules loaded by them will in general not be
1. The set of packages loaded by them will in general not be
consistent. They are a decent way to load commands to be called
from command shells. See below for better ways to assemble a
consistent set of packages for building application programs.
@@ -285,19 +285,24 @@ have some drawbacks:
other hand, are not very smart: if the user-supplied spec matches
more than one installed package, then ``spack module tcl loads`` will
fail. This may change in the future. For now, the workaround is to
be more specific on any ``spack module tcl loads`` lines that fail.
be more specific on any ``spack load`` commands that fail.
""""""""""""""""""""""
Generated Load Scripts
""""""""""""""""""""""
Another problem with using `spack load` is, it is slow; a typical user
environment could take several seconds to load, and would not be
appropriate to put into ``.bashrc`` directly. It is preferable to use
a series of ``spack module tcl loads`` commands to pre-compute which
modules to load. These can be put in a script that is run whenever
installed Spack packages change. For example:
Another problem with using `spack load` is, it can be slow; a typical
user environment could take several seconds to load, and would not be
appropriate to put into ``.bashrc`` directly. This is because it
requires the full start-up overhead of python/Spack for each command.
In some circumstances it is preferable to use a series of ``spack
module tcl loads`` (or ``spack module lmod loads``) commands to
pre-compute which modules to load. This will generate the modulenames
to load the packages using environment modules, rather than Spack's
built-in support for environment modifications. These can be put in a
script that is run whenever installed Spack packages change. For
example:
.. code-block:: sh
@@ -634,7 +639,7 @@ Global Activations
Python (and similar systems) packages directly or creating a view.
If extensions are globally activated, then ``spack load python`` will
also load all the extensions activated for the given ``python``.
This reduces the need for users to load a large number of modules.
This reduces the need for users to load a large number of packages.
However, Spack global activations have two potential drawbacks:
@@ -1254,7 +1259,7 @@ In order to build and run the image, execute:
RUN spack install tar \
&& spack clean -a
# need the modules already during image build?
# need the executables from a package already during image build?
#RUN /bin/bash -l -c ' \
# spack load tar \
# && which tar'