c4aa5cb5bc
fixes #15183 - Moved the container related content from workflows.rst into containers.rst - Deleted the docker_for_developers.rst file, since it describes an outdated procedure Co-authored-by: Axel Huebl <a.huebl@hzdr.de> Co-authored-by: Omar Padron <omar.padron@kitware.com>
1193 lines
44 KiB
ReStructuredText
1193 lines
44 KiB
ReStructuredText
.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
|
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
|
|
|
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
|
|
|
=========
|
|
Workflows
|
|
=========
|
|
|
|
The process of using Spack involves building packages, running
|
|
binaries from those packages, and developing software that depends on
|
|
those packages. For example, one might use Spack to build the
|
|
``netcdf`` package, use ``spack load`` to run the ``ncdump`` binary, and
|
|
finally, write a small C program to read/write a particular NetCDF file.
|
|
|
|
Spack supports a variety of workflows to suit a variety of situations
|
|
and user preferences, there is no single way to do all these things.
|
|
This chapter demonstrates different workflows that have been
|
|
developed, pointing out the pros and cons of them.
|
|
|
|
-----------
|
|
Definitions
|
|
-----------
|
|
|
|
First some basic definitions.
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
Package, Concrete Spec, Installed Package
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
In Spack, a package is an abstract recipe to build one piece of software.
|
|
Spack packages may be used to build, in principle, any version of that
|
|
software with any set of variants. Examples of packages include
|
|
``curl`` and ``zlib``.
|
|
|
|
A package may be *instantiated* to produce a concrete spec; one
|
|
possible realization of a particular package, out of combinatorially
|
|
many other realizations. For example, here is a concrete spec
|
|
instantiated from ``curl``:
|
|
|
|
.. command-output:: spack spec curl
|
|
|
|
Spack's core concretization algorithm generates concrete specs by
|
|
instantiating packages from its repo, based on a set of "hints",
|
|
including user input and the ``packages.yaml`` file. This algorithm
|
|
may be accessed at any time with the ``spack spec`` command.
|
|
|
|
Every time Spack installs a package, that installation corresponds to
|
|
a concrete spec. Only a vanishingly small fraction of possible
|
|
concrete specs will be installed at any one Spack site.
|
|
|
|
^^^^^^^^^^^^^^^
|
|
Consistent Sets
|
|
^^^^^^^^^^^^^^^
|
|
|
|
A set of Spack specs is said to be *consistent* if each package is
|
|
only instantiated one way within it --- that is, if two specs in the
|
|
set have the same package, then they must also have the same version,
|
|
variant, compiler, etc. For example, the following set is consistent:
|
|
|
|
.. code-block:: console
|
|
|
|
curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
^openssl@1.0.2k%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
^zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
|
|
The following set is not consistent:
|
|
|
|
.. code-block:: console
|
|
|
|
curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
^openssl@1.0.2k%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
^zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
zlib@1.2.7%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
|
|
The compatibility of a set of installed packages determines what may
|
|
be done with it. It is always possible to ``spack load`` any set of
|
|
installed packages, whether or not they are consistent, and run their
|
|
binaries from the command line. However, a set of installed packages
|
|
can only be linked together in one binary if it is consistent.
|
|
|
|
If the user produces a series of ``spack spec`` or ``spack load``
|
|
commands, in general there is no guarantee of consistency between
|
|
them. Spack's concretization procedure guarantees that the results of
|
|
any *single* ``spack spec`` call will be consistent. Therefore, the
|
|
best way to ensure a consistent set of specs is to create a Spack
|
|
package with dependencies, and then instantiate that package. We will
|
|
use this technique below.
|
|
|
|
-----------------
|
|
Building Packages
|
|
-----------------
|
|
|
|
Suppose you are tasked with installing a set of software packages on a
|
|
system in order to support one application -- both a core application
|
|
program, plus software to prepare input and analyze output. The
|
|
required software might be summed up as a series of ``spack install``
|
|
commands placed in a script. If needed, this script can always be run
|
|
again in the future. For example:
|
|
|
|
.. code-block:: sh
|
|
|
|
#!/bin/sh
|
|
spack install modele-utils
|
|
spack install emacs
|
|
spack install ncview
|
|
spack install nco
|
|
spack install modele-control
|
|
spack install py-numpy
|
|
|
|
In most cases, this script will not correctly install software
|
|
according to your specific needs: choices need to be made for
|
|
variants, versions and virtual dependency choices may be needed. It
|
|
*is* possible to specify these choices by extending specs on the
|
|
command line; however, the same choices must be specified repeatedly.
|
|
For example, if you wish to use ``openmpi`` to satisfy the ``mpi``
|
|
dependency, then ``^openmpi`` will have to appear on *every* ``spack
|
|
install`` line that uses MPI. It can get repetitive fast.
|
|
|
|
Customizing Spack installation options is easier to do in the
|
|
``~/.spack/packages.yaml`` file. In this file, you can specify
|
|
preferred versions and variants to use for packages. For example:
|
|
|
|
.. code-block:: yaml
|
|
|
|
packages:
|
|
python:
|
|
version: [3.5.1]
|
|
modele-utils:
|
|
version: [cmake]
|
|
|
|
everytrace:
|
|
version: [develop]
|
|
eigen:
|
|
variants: ~suitesparse
|
|
netcdf:
|
|
variants: +mpi
|
|
|
|
all:
|
|
compiler: [gcc@5.3.0]
|
|
providers:
|
|
mpi: [openmpi]
|
|
blas: [openblas]
|
|
lapack: [openblas]
|
|
|
|
|
|
This approach will work as long as you are building packages for just
|
|
one application.
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
Multiple Applications
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Suppose instead you're building multiple inconsistent applications.
|
|
For example, users want package A to be built with ``openmpi`` and
|
|
package B with ``mpich`` --- but still share many other lower-level
|
|
dependencies. In this case, a single ``packages.yaml`` file will not
|
|
work. Plans are to implement *per-project* ``packages.yaml`` files.
|
|
In the meantime, one could write shell scripts to switch
|
|
``packages.yaml`` between multiple versions as needed, using symlinks.
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
Combinatorial Sets of Installs
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Suppose that you are now tasked with systematically building many
|
|
incompatible versions of packages. For example, you need to build
|
|
``petsc`` 9 times for 3 different MPI implementations on 3 different
|
|
compilers, in order to support user needs. In this case, you will
|
|
need to either create 9 different ``packages.yaml`` files; or more
|
|
likely, create 9 different ``spack install`` command lines with the
|
|
correct options in the spec. Here is a real-life example of this kind
|
|
of usage:
|
|
|
|
.. code-block:: sh
|
|
|
|
#!/bin/bash
|
|
|
|
compilers=(
|
|
%gcc
|
|
%intel
|
|
%pgi
|
|
)
|
|
|
|
mpis=(
|
|
openmpi+psm~verbs
|
|
openmpi~psm+verbs
|
|
mvapich2+psm~mrail
|
|
mvapich2~psm+mrail
|
|
mpich+verbs
|
|
)
|
|
|
|
for compiler in "${compilers[@]}"
|
|
do
|
|
# Serial installs
|
|
spack install szip $compiler
|
|
spack install hdf $compiler
|
|
spack install hdf5 $compiler
|
|
spack install netcdf $compiler
|
|
spack install netcdf-fortran $compiler
|
|
spack install ncview $compiler
|
|
|
|
# Parallel installs
|
|
for mpi in "${mpis[@]}"
|
|
do
|
|
spack install $mpi $compiler
|
|
spack install hdf5~cxx+mpi $compiler ^$mpi
|
|
spack install parallel-netcdf $compiler ^$mpi
|
|
done
|
|
done
|
|
|
|
------------------------------
|
|
Running Binaries from Packages
|
|
------------------------------
|
|
|
|
Once Spack packages have been built, the next step is to use them. As
|
|
with building packages, there are many ways to use them, depending on
|
|
the use case.
|
|
|
|
^^^^^^^^^^^^
|
|
Find and Run
|
|
^^^^^^^^^^^^
|
|
|
|
The simplest way to run a Spack binary is to find it and run it!
|
|
In many cases, nothing more is needed because Spack builds binaries
|
|
with RPATHs. Spack installation directories may be found with ``spack
|
|
location --install-dir`` commands. For example:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack location --install-dir cmake
|
|
~/spack/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/cmake-3.6.0-7cxrynb6esss6jognj23ak55fgxkwtx7
|
|
|
|
This gives the root of the Spack package; relevant binaries may be
|
|
found within it. For example:
|
|
|
|
.. code-block:: console
|
|
|
|
$ CMAKE=`spack location --install-dir cmake`/bin/cmake
|
|
|
|
|
|
Standard UNIX tools can find binaries as well. For example:
|
|
|
|
.. code-block:: console
|
|
|
|
$ find ~/spack/opt -name cmake | grep bin
|
|
~/spack/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/cmake-3.6.0-7cxrynb6esss6jognj23ak55fgxkwtx7/bin/cmake
|
|
|
|
These methods are suitable, for example, for setting up build
|
|
processes or GUIs that need to know the location of particular tools.
|
|
However, other more powerful methods are generally preferred for user
|
|
environments.
|
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
Using ``spack load`` to Manage the User Environment
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Suppose that Spack has been used to install a set of command-line
|
|
programs, which users now wish to use. One can in principle put a
|
|
number of ``spack load`` commands into ``.bashrc``, for example, to
|
|
load a set of Spack packages:
|
|
|
|
.. code-block:: sh
|
|
|
|
spack load modele-utils
|
|
spack load emacs
|
|
spack load ncview
|
|
spack load nco
|
|
spack load modele-control
|
|
|
|
Although simple load scripts like this are useful in many cases, they
|
|
have some drawbacks:
|
|
|
|
1. The set of packages loaded by them will in general not be
|
|
consistent. They are a decent way to load commands to be called
|
|
from command shells. See below for better ways to assemble a
|
|
consistent set of packages for building application programs.
|
|
|
|
2. The ``spack spec`` and ``spack install`` commands use a
|
|
sophisticated concretization algorithm that chooses the "best"
|
|
among several options, taking into account ``packages.yaml`` file.
|
|
The ``spack load`` and ``spack module tcl loads`` commands, on the
|
|
other hand, are not very smart: if the user-supplied spec matches
|
|
more than one installed package, then ``spack module tcl loads`` will
|
|
fail. This default behavior may change in the future. For now,
|
|
the workaround is to either be more specific on any failing ``spack load``
|
|
commands or to use ``spack load --first`` to allow spack to load the
|
|
first matching spec.
|
|
|
|
|
|
""""""""""""""""""""""
|
|
Generated Load Scripts
|
|
""""""""""""""""""""""
|
|
|
|
Another problem with using `spack load` is, it can be slow; a typical
|
|
user environment could take several seconds to load, and would not be
|
|
appropriate to put into ``.bashrc`` directly. This is because it
|
|
requires the full start-up overhead of python/Spack for each command.
|
|
In some circumstances it is preferable to use a series of ``spack
|
|
module tcl loads`` (or ``spack module lmod loads``) commands to
|
|
pre-compute which modules to load. This will generate the modulenames
|
|
to load the packages using environment modules, rather than Spack's
|
|
built-in support for environment modifications. These can be put in a
|
|
script that is run whenever installed Spack packages change. For
|
|
example:
|
|
|
|
.. code-block:: sh
|
|
|
|
#!/bin/sh
|
|
#
|
|
# Generate module load commands in ~/env/spackenv
|
|
|
|
cat <<EOF | /bin/sh >$HOME/env/spackenv
|
|
FIND='spack module tcl loads --prefix linux-SuSE11-x86_64/'
|
|
|
|
\$FIND modele-utils
|
|
\$FIND emacs
|
|
\$FIND ncview
|
|
\$FIND nco
|
|
\$FIND modele-control
|
|
EOF
|
|
|
|
The output of this file is written in ``~/env/spackenv``:
|
|
|
|
.. code-block:: sh
|
|
|
|
# binutils@2.25%gcc@5.3.0+gold~krellpatch~libiberty arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/binutils-2.25-gcc-5.3.0-6w5d2t4
|
|
# python@2.7.12%gcc@5.3.0~tk~ucs4 arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/python-2.7.12-gcc-5.3.0-2azoju2
|
|
# ncview@2.1.7%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/ncview-2.1.7-gcc-5.3.0-uw3knq2
|
|
# nco@4.5.5%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/nco-4.5.5-gcc-5.3.0-7aqmimu
|
|
# modele-control@develop%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/modele-control-develop-gcc-5.3.0-7rddsij
|
|
# zlib@1.2.8%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/zlib-1.2.8-gcc-5.3.0-fe5onbi
|
|
# curl@7.50.1%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/curl-7.50.1-gcc-5.3.0-4vlev55
|
|
# hdf5@1.10.0-patch1%gcc@5.3.0+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/hdf5-1.10.0-patch1-gcc-5.3.0-pwnsr4w
|
|
# netcdf@4.4.1%gcc@5.3.0~hdf4+mpi arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/netcdf-4.4.1-gcc-5.3.0-rl5canv
|
|
# netcdf-fortran@4.4.4%gcc@5.3.0 arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/netcdf-fortran-4.4.4-gcc-5.3.0-stdk2xq
|
|
# modele-utils@cmake%gcc@5.3.0+aux+diags+ic arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/modele-utils-cmake-gcc-5.3.0-idyjul5
|
|
# everytrace@develop%gcc@5.3.0+fortran+mpi arch=linux-SuSE11-x86_64
|
|
module load linux-SuSE11-x86_64/everytrace-develop-gcc-5.3.0-p5wmb25
|
|
|
|
Users may now put ``source ~/env/spackenv`` into ``.bashrc``.
|
|
|
|
.. note ::
|
|
|
|
Some module systems put a prefix on the names of modules created
|
|
by Spack. For example, that prefix is ``linux-SuSE11-x86_64/`` in
|
|
the above case. If a prefix is not needed, you may omit the
|
|
``--prefix`` flag from ``spack module tcl loads``.
|
|
|
|
|
|
"""""""""""""""""""""""
|
|
Transitive Dependencies
|
|
"""""""""""""""""""""""
|
|
|
|
In the script above, each ``spack module tcl loads`` command generates a
|
|
*single* ``module load`` line. Transitive dependencies do not usually
|
|
need to be loaded, only modules the user needs in ``$PATH``. This is
|
|
because Spack builds binaries with RPATH. Spack's RPATH policy has
|
|
some nice features:
|
|
|
|
#. Modules for multiple inconsistent applications may be loaded
|
|
simultaneously. In the above example (Multiple Applications),
|
|
package A and package B can coexist together in the user's $PATH,
|
|
even though they use different MPIs.
|
|
|
|
#. RPATH eliminates a whole class of strange errors that can happen
|
|
in non-RPATH binaries when the wrong ``LD_LIBRARY_PATH`` is
|
|
loaded.
|
|
|
|
#. Recursive module systems such as LMod are not necessary.
|
|
|
|
#. Modules are not needed at all to execute binaries. If a path to a
|
|
binary is known, it may be executed. For example, the path for a
|
|
Spack-built compiler can be given to an IDE without requiring the
|
|
IDE to load that compiler's module.
|
|
|
|
Unfortunately, Spack's RPATH support does not work in all case. For example:
|
|
|
|
#. Software comes in many forms --- not just compiled ELF binaries,
|
|
but also as interpreted code in Python, R, JVM bytecode, etc.
|
|
Those systems almost universally use an environment variable
|
|
analogous to ``LD_LIBRARY_PATH`` to dynamically load libraries.
|
|
|
|
#. Although Spack generally builds binaries with RPATH, it does not
|
|
currently do so for compiled Python extensions (for example,
|
|
``py-numpy``). Any libraries that these extensions depend on
|
|
(``blas`` in this case, for example) must be specified in the
|
|
``LD_LIBRARY_PATH``.`
|
|
|
|
#. In some cases, Spack-generated binaries end up without a
|
|
functional RPATH for no discernible reason.
|
|
|
|
In cases where RPATH support doesn't make things "just work," it can
|
|
be necessary to load a module's dependencies as well as the module
|
|
itself. This is done by adding the ``--dependencies`` flag to the
|
|
``spack module tcl loads`` command. For example, the following line,
|
|
added to the script above, would be used to load SciPy, along with
|
|
Numpy, core Python, BLAS/LAPACK and anything else needed:
|
|
|
|
.. code-block:: sh
|
|
|
|
spack module tcl loads --dependencies py-scipy
|
|
|
|
^^^^^^^^^^^^^^
|
|
Dummy Packages
|
|
^^^^^^^^^^^^^^
|
|
|
|
As an alternative to a series of ``module load`` commands, one might
|
|
consider dummy packages as a way to create a *consistent* set of
|
|
packages that may be loaded as one unit. The idea here is pretty
|
|
simple:
|
|
|
|
#. Create a package (say, ``mydummy``) with no URL and no
|
|
``install()`` method, just dependencies.
|
|
|
|
#. Run ``spack install mydummy`` to install.
|
|
|
|
An advantage of this method is the set of packages produced will be
|
|
consistent. This means that you can reliably build software against
|
|
it. A disadvantage is the set of packages will be consistent; this
|
|
means you cannot load up two applications this way if they are not
|
|
consistent with each other.
|
|
|
|
.. _filesystem-views:
|
|
|
|
^^^^^^^^^^^^^^^^
|
|
Filesystem Views
|
|
^^^^^^^^^^^^^^^^
|
|
|
|
Filesystem views offer an alternative to environment modules, another
|
|
way to assemble packages in a useful way and load them into a user's
|
|
environment.
|
|
|
|
A single-prefix filesystem view is a single directory tree that is the
|
|
union of the directory hierarchies of a number of installed packages;
|
|
it is similar to the directory hierarchy that might exist under
|
|
``/usr/local``. The files of the view's installed packages are
|
|
brought into the view by symbolic or hard links, referencing the
|
|
original Spack installation.
|
|
|
|
A combinatorial filesystem view can contain more software than a
|
|
single-prefix view. Combinatorial filesystem views are created by
|
|
defining a projection for each spec or set of specs. The syntax for
|
|
this will be discussed in the section for the ``spack view`` command
|
|
under `adding_projections_to_views`_.
|
|
|
|
The projection for a spec or set of specs specifies the naming scheme
|
|
for the directory structure under the root of the view into which the
|
|
package will be linked. For example, the spec ``zlib@1.2.8%gcc@4.4.7``
|
|
could be projected to ``MYVIEW/zlib-1.2.8-gcc``.
|
|
|
|
When software is built and installed, absolute paths are frequently
|
|
"baked into" the software, making it non-relocatable. This happens
|
|
not just in RPATHs, but also in shebangs, configuration files, and
|
|
assorted other locations.
|
|
|
|
Therefore, programs run out of a Spack view will typically still look
|
|
in the original Spack-installed location for shared libraries and
|
|
other resources. This behavior is not easily changed; in general,
|
|
there is no way to know where absolute paths might be written into an
|
|
installed package, and how to relocate it. Therefore, the original
|
|
Spack tree must be kept in place for a filesystem view to work, even
|
|
if the view is built with hardlinks.
|
|
|
|
.. FIXME: reference the relocation work of Hegner and Gartung (PR #1013)
|
|
|
|
.. _cmd-spack-view:
|
|
|
|
""""""""""""""
|
|
``spack view``
|
|
""""""""""""""
|
|
|
|
A filesystem view is created, and packages are linked in, by the ``spack
|
|
view`` command's ``symlink`` and ``hardlink`` sub-commands. The
|
|
``spack view remove`` command can be used to unlink some or all of the
|
|
filesystem view.
|
|
|
|
The following example creates a filesystem view based
|
|
on an installed ``cmake`` package and then removes from the view the
|
|
files in the ``cmake`` package while retaining its dependencies.
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack view --verbose symlink myview cmake@3.5.2
|
|
==> Linking package: "ncurses"
|
|
==> Linking package: "zlib"
|
|
==> Linking package: "openssl"
|
|
==> Linking package: "cmake"
|
|
|
|
$ ls myview/
|
|
bin doc etc include lib share
|
|
|
|
$ ls myview/bin/
|
|
captoinfo clear cpack ctest infotocap openssl tabs toe tset
|
|
ccmake cmake c_rehash infocmp ncurses6-config reset tic tput
|
|
|
|
$ spack view --verbose --dependencies false rm myview cmake@3.5.2
|
|
==> Removing package: "cmake"
|
|
|
|
$ ls myview/bin/
|
|
captoinfo c_rehash infotocap openssl tabs toe tset
|
|
clear infocmp ncurses6-config reset tic tput
|
|
|
|
.. note::
|
|
|
|
If the set of packages being included in a view is inconsistent,
|
|
then it is possible that two packages will provide the same file. Any
|
|
conflicts of this type are handled on a first-come-first-served basis,
|
|
and a warning is printed.
|
|
|
|
.. note::
|
|
|
|
When packages are removed from a view, empty directories are
|
|
purged.
|
|
|
|
.. _adding_projections_to_views:
|
|
|
|
""""""""""""""""""""""""""""
|
|
Controlling View Projections
|
|
""""""""""""""""""""""""""""
|
|
|
|
The default projection into a view is to link every package into the
|
|
root of the view. This can be changed by adding a ``projections.yaml``
|
|
configuration file to the view. The projection configuration file for
|
|
a view located at ``/my/view`` is stored in
|
|
``/my/view/.spack/projections.yaml``.
|
|
|
|
When creating a view, the projection configuration file can also be
|
|
specified from the command line using the ``--projection-file`` option
|
|
to the ``spack view`` command.
|
|
|
|
The projections configuration file is a mapping of partial specs to
|
|
spec format strings, as shown in the example below.
|
|
|
|
.. code-block:: yaml
|
|
|
|
projections:
|
|
zlib: {name}-{version}
|
|
^mpi: {name}-{version}/{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version}
|
|
all: {name}-{version}/{compiler.name}-{compiler.version}
|
|
|
|
The entries in the projections configuration file must all be either
|
|
specs or the keyword ``all``. For each spec, the projection used will
|
|
be the first non-``all`` entry that the spec satisfies, or ``all`` if
|
|
there is an entry for ``all`` and no other entry is satisfied by the
|
|
spec. Where the keyword ``all`` appears in the file does not
|
|
matter. Given the example above, any spec satisfying ``zlib@1.2.8``
|
|
will be linked into ``/my/view/zlib-1.2.8/``, any spec satisfying
|
|
``hdf5@1.8.10+mpi %gcc@4.9.3 ^mvapich2@2.2`` will be linked into
|
|
``/my/view/hdf5-1.8.10/mvapich2-2.2-gcc-4.9.3``, and any spec
|
|
satisfying ``hdf5@1.8.10~mpi %gcc@4.9.3`` will be linked into
|
|
``/my/view/hdf5-1.8.10/gcc-4.9.3``.
|
|
|
|
If the keyword ``all`` does not appear in the projections
|
|
configuration file, any spec that does not satisfy any entry in the
|
|
file will be linked into the root of the view as in a single-prefix
|
|
view. Any entries that appear below the keyword ``all`` in the
|
|
projections configuration file will not be used, as all specs will use
|
|
the projection under ``all`` before reaching those entries.
|
|
|
|
""""""""""""""""""
|
|
Fine-Grain Control
|
|
""""""""""""""""""
|
|
|
|
The ``--exclude`` and ``--dependencies`` option flags allow for
|
|
fine-grained control over which packages and dependencies do or not
|
|
get included in a view. For example, suppose you are developing the
|
|
``appsy`` package. You wish to build against a view of all ``appsy``
|
|
dependencies, but not ``appsy`` itself:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack view --dependencies yes --exclude appsy symlink /path/to/MYVIEW/ appsy
|
|
|
|
Alternately, you wish to create a view whose purpose is to provide
|
|
binary executables to end users. You only need to include
|
|
applications they might want, and not those applications'
|
|
dependencies. In this case, you might use:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack view --dependencies no symlink /path/to/MYVIEW/ cmake
|
|
|
|
|
|
"""""""""""""""""""""""
|
|
Hybrid Filesystem Views
|
|
"""""""""""""""""""""""
|
|
|
|
Although filesystem views are usually created by Spack, users are free
|
|
to add to them by other means. For example, imagine a filesystem
|
|
view, created by Spack, that looks something like:
|
|
|
|
.. code-block:: console
|
|
|
|
/path/to/MYVIEW/bin/programA -> /path/to/spack/.../bin/programA
|
|
/path/to/MYVIEW/lib/libA.so -> /path/to/spack/.../lib/libA.so
|
|
|
|
Now, the user may add to this view by non-Spack means; for example, by
|
|
running a classic install script. For example:
|
|
|
|
.. code-block:: console
|
|
|
|
$ tar -xf B.tar.gz
|
|
$ cd B/
|
|
$ ./configure --prefix=/path/to/MYVIEW \
|
|
--with-A=/path/to/MYVIEW
|
|
$ make && make install
|
|
|
|
The result is a hybrid view:
|
|
|
|
.. code-block:: console
|
|
|
|
/path/to/MYVIEW/bin/programA -> /path/to/spack/.../bin/programA
|
|
/path/to/MYVIEW/bin/programB
|
|
/path/to/MYVIEW/lib/libA.so -> /path/to/spack/.../lib/libA.so
|
|
/path/to/MYVIEW/lib/libB.so
|
|
|
|
In this case, real files coexist, interleaved with the "view"
|
|
symlinks. At any time one can delete ``/path/to/MYVIEW`` or use
|
|
``spack view`` to manage it surgically. None of this will affect the
|
|
real Spack install area.
|
|
|
|
^^^^^^^^^^^^^^^^^^
|
|
Global Activations
|
|
^^^^^^^^^^^^^^^^^^
|
|
|
|
:ref:`cmd-spack-activate` may be used as an alternative to loading
|
|
Python (and similar systems) packages directly or creating a view.
|
|
If extensions are globally activated, then ``spack load python`` will
|
|
also load all the extensions activated for the given ``python``.
|
|
This reduces the need for users to load a large number of packages.
|
|
|
|
However, Spack global activations have two potential drawbacks:
|
|
|
|
#. Activated packages that involve compiled C extensions may still
|
|
need their dependencies to be loaded manually. For example,
|
|
``spack load openblas`` might be required to make ``py-numpy``
|
|
work.
|
|
|
|
#. Global activations "break" a core feature of Spack, which is that
|
|
multiple versions of a package can co-exist side-by-side. For example,
|
|
suppose you wish to run a Python package in two different
|
|
environments but the same basic Python --- one with
|
|
``py-numpy@1.7`` and one with ``py-numpy@1.8``. Spack extensions
|
|
will not support this potential debugging use case.
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
Discussion: Running Binaries
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Modules, extension packages and filesystem views are all ways to
|
|
assemble sets of Spack packages into a useful environment. They are
|
|
all semantically similar, in that conflicting installed packages
|
|
cannot simultaneously be loaded, activated or included in a view.
|
|
|
|
With all of these approaches, there is no guarantee that the
|
|
environment created will be consistent. It is possible, for example,
|
|
to simultaneously load application A that uses OpenMPI and application
|
|
B that uses MPICH. Both applications will run just fine in this
|
|
inconsistent environment because they rely on RPATHs, not the
|
|
environment, to find their dependencies.
|
|
|
|
In general, environments set up using modules vs. views will work
|
|
similarly. Both can be used to set up ephemeral or long-lived
|
|
testing/development environments. Operational differences between the
|
|
two approaches can make one or the other preferable in certain
|
|
environments:
|
|
|
|
* Filesystem views do not require environment module infrastructure.
|
|
Although Spack can install ``environment-modules``, users might be
|
|
hostile to its use. Filesystem views offer a good solution for
|
|
sysadmins serving users who just "want all the stuff I need in one
|
|
place" and don't want to hear about Spack.
|
|
|
|
* Although modern build systems will find dependencies wherever they
|
|
might be, some applications with hand-built make files expect their
|
|
dependencies to be in one place. One common problem is makefiles
|
|
that assume that ``netcdf`` and ``netcdf-fortran`` are installed in
|
|
the same tree. Or, one might use an IDE that requires tedious
|
|
configuration of dependency paths; and it's easier to automate that
|
|
administration in a view-building script than in the IDE itself.
|
|
For all these cases, a view will be preferable to other ways to
|
|
assemble an environment.
|
|
|
|
* On systems with I-node quotas, modules might be preferable to views
|
|
and extension packages.
|
|
|
|
* Views and activated extensions maintain state that is semantically
|
|
equivalent to the information in a ``spack module tcl loads`` script.
|
|
Administrators might find things easier to maintain without the
|
|
added "heavyweight" state of a view.
|
|
|
|
-------------------------------------
|
|
Using Spack to Replace Homebrew/Conda
|
|
-------------------------------------
|
|
|
|
Spack is an incredibly powerful package manager, designed for supercomputers
|
|
where users have diverse installation needs. But Spack can also be used to
|
|
handle simple single-user installations on your laptop. Most macOS users are
|
|
already familiar with package managers like Homebrew and Conda, where all
|
|
installed packages are symlinked to a single central location like ``/usr/local``.
|
|
In this section, we will show you how to emulate the behavior of Homebrew/Conda
|
|
using :ref:`environments`!
|
|
|
|
^^^^^
|
|
Setup
|
|
^^^^^
|
|
|
|
First, let's create a new environment. We'll assume that Spack is already set up
|
|
correctly, and that you've already sourced the setup script for your shell.
|
|
To create a new environment, simply run:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack env create myenv
|
|
==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
|
|
==> Created environment 'myenv' in /Users/me/spack/var/spack/environments/myenv
|
|
$ spack env activate myenv
|
|
|
|
Here, *myenv* can be anything you want to name your environment. Next, we can add
|
|
a list of packages we would like to install into our environment. Let's say we
|
|
want a newer version of Bash than the one that comes with macOS, and we want a
|
|
few Python libraries. We can run:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack add bash
|
|
==> Adding bash to environment myenv
|
|
==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
|
|
$ spack add python@3:
|
|
==> Adding python@3: to environment myenv
|
|
==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
|
|
$ spack add py-numpy py-scipy py-matplotlib
|
|
==> Adding py-numpy to environment myenv
|
|
==> Adding py-scipy to environment myenv
|
|
==> Adding py-matplotlib to environment myenv
|
|
==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
|
|
|
|
Each package can be listed on a separate line, or combined into a single line.
|
|
Notice that we're explicitly asking for Python 3 here. You can use any spec
|
|
you would normally use on the command line with other Spack commands.
|
|
|
|
Next, we want to manually configure a couple of things. In the ``myenv``
|
|
directory, we can find the ``spack.yaml`` that actually defines our environment.
|
|
|
|
.. code-block:: console
|
|
|
|
$ vim ~/spack/var/spack/environments/myenv/spack.yaml
|
|
|
|
.. code-block:: yaml
|
|
|
|
# This is a Spack Environment file.
|
|
#
|
|
# It describes a set of packages to be installed, along with
|
|
# configuration settings.
|
|
spack:
|
|
# add package specs to the `specs` list
|
|
specs: [bash, 'python@3:', py-numpy, py-scipy, py-matplotlib]
|
|
view:
|
|
default:
|
|
root: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
|
|
projections: {}
|
|
config: {}
|
|
mirrors: {}
|
|
modules:
|
|
enable: []
|
|
packages: {}
|
|
repos: []
|
|
upstreams: {}
|
|
definitions: []
|
|
concretization: separately
|
|
|
|
You can see the packages we added earlier in the ``specs:`` section. If you
|
|
ever want to add more packages, you can either use ``spack add`` or manually
|
|
edit this file.
|
|
|
|
We also need to change the ``concretization:`` option. By default, Spack
|
|
concretizes each spec *separately*, allowing multiple versions of the same
|
|
package to coexist. Since we want a single consistent environment, we want to
|
|
concretize all of the specs *together*.
|
|
|
|
Here is what your ``spack.yaml`` looks like with these new settings, and with
|
|
some of the sections we don't plan on using removed:
|
|
|
|
.. code-block:: diff
|
|
|
|
spack:
|
|
- specs: [bash, 'python@3:', py-numpy, py-scipy, py-matplotlib]
|
|
+ specs:
|
|
+ - bash
|
|
+ - 'python@3:'
|
|
+ - py-numpy
|
|
+ - py-scipy
|
|
+ - py-matplotlib
|
|
- view:
|
|
- default:
|
|
- root: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
|
|
- projections: {}
|
|
+ view: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
|
|
- config: {}
|
|
- mirrors: {}
|
|
- modules:
|
|
- enable: []
|
|
- packages: {}
|
|
- repos: []
|
|
- upstreams: {}
|
|
- definitions: []
|
|
+ concretization: together
|
|
- concretization: separately
|
|
|
|
""""""""""""""""
|
|
Symlink location
|
|
""""""""""""""""
|
|
|
|
In the ``spack.yaml`` file above, you'll notice that by default, Spack symlinks
|
|
all installations to ``/Users/me/spack/var/spack/environments/myenv/.spack-env/view``.
|
|
You can actually change this to any directory you want. For example, Homebrew
|
|
uses ``/usr/local``, while Conda uses ``/Users/me/anaconda``. In order to access
|
|
files in these locations, you need to update ``PATH`` and other environment variables
|
|
to point to them. Activating the Spack environment does this automatically, but
|
|
you can also manually set them in your ``.bashrc``.
|
|
|
|
.. warning::
|
|
|
|
There are several reasons why you shouldn't use ``/usr/local``:
|
|
|
|
1. If you are on macOS 10.11+ (El Capitan and newer), Apple makes it hard
|
|
for you. You may notice permissions issues on ``/usr/local`` due to their
|
|
`System Integrity Protection <https://support.apple.com/en-us/HT204899>`_.
|
|
By default, users don't have permissions to install anything in ``/usr/local``,
|
|
and you can't even change this using ``sudo chown`` or ``sudo chmod``.
|
|
2. Other package managers like Homebrew will try to install things to the
|
|
same directory. If you plan on using Homebrew in conjunction with Spack,
|
|
don't symlink things to ``/usr/local``.
|
|
3. If you are on a shared workstation, or don't have sudo privileges, you
|
|
can't do this.
|
|
|
|
If you still want to do this anyway, there are several ways around SIP.
|
|
You could disable SIP by booting into recovery mode and running
|
|
``csrutil disable``, but this is not recommended, as it can open up your OS
|
|
to security vulnerabilities. Another technique is to run ``spack concretize``
|
|
and ``spack install`` using ``sudo``. This is also not recommended.
|
|
|
|
The safest way I've found is to create your installation directories using
|
|
sudo, then change ownership back to the user like so:
|
|
|
|
.. code-block:: bash
|
|
|
|
for directory in .spack bin contrib include lib man share
|
|
do
|
|
sudo mkdir -p /usr/local/$directory
|
|
sudo chown $(id -un):$(id -gn) /usr/local/$directory
|
|
done
|
|
|
|
Depending on the packages you install in your environment, the exact list of
|
|
directories you need to create may vary. You may also find some packages
|
|
like Java libraries that install a single file to the installation prefix
|
|
instead of in a subdirectory. In this case, the action is the same, just replace
|
|
``mkdir -p`` with ``touch`` in the for-loop above.
|
|
|
|
But again, it's safer just to use the default symlink location.
|
|
|
|
|
|
^^^^^^^^^^^^
|
|
Installation
|
|
^^^^^^^^^^^^
|
|
|
|
To actually concretize the environment, run:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack concretize
|
|
|
|
This will tell you which if any packages are already installed, and alert you
|
|
to any conflicting specs.
|
|
|
|
To actually install these packages and symlink them to your ``view:``
|
|
directory, simply run:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack install
|
|
|
|
Now, when you type ``which python3``, it should find the one you just installed.
|
|
|
|
In order to change the default shell to our newer Bash installation, we first
|
|
need to add it to this list of acceptable shells. Run:
|
|
|
|
.. code-block:: console
|
|
|
|
$ sudo vim /etc/shells
|
|
|
|
and add the absolute path to your bash executable. Then run:
|
|
|
|
.. code-block:: console
|
|
|
|
$ chsh -s /path/to/bash
|
|
|
|
Now, when you log out and log back in, ``echo $SHELL`` should point to the
|
|
newer version of Bash.
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
Updating Installed Packages
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
Let's say you upgraded to a new version of macOS, or a new version of Python
|
|
was released, and you want to rebuild your entire software stack. To do this,
|
|
simply run the following commands:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack env activate myenv
|
|
$ spack concretize --force
|
|
$ spack install
|
|
|
|
The ``--force`` flag tells Spack to overwrite its previous concretization
|
|
decisions, allowing you to choose a new version of Python. If any of the new
|
|
packages like Bash are already installed, ``spack install`` won't re-install
|
|
them, it will keep the symlinks in place.
|
|
|
|
^^^^^^^^^^^^^^
|
|
Uninstallation
|
|
^^^^^^^^^^^^^^
|
|
|
|
If you decide that Spack isn't right for you, uninstallation is simple.
|
|
Just run:
|
|
|
|
.. code-block:: console
|
|
|
|
$ spack env activate myenv
|
|
$ spack uninstall --all
|
|
|
|
This will uninstall all packages in your environment and remove the symlinks.
|
|
|
|
------------------------
|
|
Using Spack on Travis-CI
|
|
------------------------
|
|
|
|
Spack can be deployed as a provider for userland software in
|
|
`Travis-CI <https://http://travis-ci.org>`_.
|
|
|
|
A starting-point for a ``.travis.yml`` file can look as follows.
|
|
It uses `caching <https://docs.travis-ci.com/user/caching/>`_ for
|
|
already built environments, so make sure to clean the Travis cache if
|
|
you run into problems.
|
|
|
|
The main points that are implemented below:
|
|
|
|
#. Travis is detected as having up to 34 cores available, but only 2
|
|
are actually allocated for the user. We limit the parallelism of
|
|
the spack builds in the config.
|
|
(The Travis yaml parser is a bit buggy on the echo command.)
|
|
|
|
#. Without control for the user, Travis jobs will run on various
|
|
``x86_64`` microarchitectures. If you plan to cache build results,
|
|
e.g. to accelerate dependency builds, consider building for the
|
|
generic ``x86_64`` target only.
|
|
Limiting the microarchitecture will also find more packages when
|
|
working with the
|
|
`E4S Spack build cache <https://oaciss.uoregon.edu/e4s/e4s_buildcache_inventory.html>`_.
|
|
|
|
#. Builds over 10 minutes need to be prefixed with ``travis_wait``.
|
|
Alternatively, generate output once with ``spack install -v``.
|
|
|
|
#. Travis builds are non-interactive. This prevents using bash
|
|
aliases and functions for modules. We fix that by sourcing
|
|
``/etc/profile`` first (or running everything in a subshell with
|
|
``bash -l -c '...'``).
|
|
|
|
.. code-block:: yaml
|
|
|
|
language: cpp
|
|
sudo: false
|
|
dist: trusty
|
|
|
|
cache:
|
|
apt: true
|
|
directories:
|
|
- $HOME/.cache
|
|
|
|
addons:
|
|
apt:
|
|
sources:
|
|
- ubuntu-toolchain-r-test
|
|
packages:
|
|
- g++-4.9
|
|
- environment-modules
|
|
|
|
env:
|
|
global:
|
|
- SPACK_ROOT: $HOME/.cache/spack
|
|
- PATH: $PATH:$HOME/.cache/spack/bin
|
|
|
|
before_install:
|
|
- export CXX=g++-4.9
|
|
- export CC=gcc-4.9
|
|
- export FC=gfortran-4.9
|
|
- export CXXFLAGS="-std=c++11"
|
|
|
|
install:
|
|
- |
|
|
if ! which spack >/dev/null; then
|
|
mkdir -p $SPACK_ROOT &&
|
|
git clone --depth 50 https://github.com/spack/spack.git $SPACK_ROOT &&
|
|
printf "config:\n build_jobs: 2\n" > $SPACK_ROOT/etc/spack/config.yaml &&
|
|
printf "packages:\n all:\n target: ['x86_64']\n" \
|
|
> $SPACK_ROOT/etc/spack/packages.yaml;
|
|
fi
|
|
- travis_wait spack install cmake@3.7.2~openssl~ncurses
|
|
- travis_wait spack install boost@1.62.0~graph~iostream~locale~log~wave
|
|
- spack clean -a
|
|
- source /etc/profile &&
|
|
source $SPACK_ROOT/share/spack/setup-env.sh
|
|
- spack load cmake
|
|
- spack load boost
|
|
|
|
script:
|
|
- mkdir -p $HOME/build
|
|
- cd $HOME/build
|
|
- cmake $TRAVIS_BUILD_DIR
|
|
- make -j 2
|
|
- make test
|
|
|
|
------------------
|
|
Upstream Bug Fixes
|
|
------------------
|
|
|
|
It is not uncommon to discover a bug in an upstream project while
|
|
trying to build with Spack. Typically, the bug is in a package that
|
|
serves a dependency to something else. This section describes
|
|
procedure to work around and ultimately resolve these bugs, while not
|
|
delaying the Spack user's main goal.
|
|
|
|
^^^^^^^^^^^^^^^^^
|
|
Buggy New Version
|
|
^^^^^^^^^^^^^^^^^
|
|
|
|
Sometimes, the old version of a package works fine, but a new version
|
|
is buggy. For example, it was once found that `Adios did not build
|
|
with hdf5@1.10 <https://github.com/spack/spack/issues/1683>`_. If the
|
|
old version of ``hdf5`` will work with ``adios``, the suggested
|
|
procedure is:
|
|
|
|
#. Revert ``adios`` to the old version of ``hdf5``. Put in its
|
|
``adios/package.py``:
|
|
|
|
.. code-block:: python
|
|
|
|
# Adios does not build with HDF5 1.10
|
|
# See: https://github.com/spack/spack/issues/1683
|
|
depends_on('hdf5@:1.9')
|
|
|
|
#. Determine whether the problem is with ``hdf5`` or ``adios``, and
|
|
report the problem to the appropriate upstream project. In this
|
|
case, the problem was with ``adios``.
|
|
|
|
#. Once a new version of ``adios`` comes out with the bugfix, modify
|
|
``adios/package.py`` to reflect it:
|
|
|
|
.. code-block:: python
|
|
|
|
# Adios up to v1.10.0 does not build with HDF5 1.10
|
|
# See: https://github.com/spack/spack/issues/1683
|
|
depends_on('hdf5@:1.9', when='@:1.10.0')
|
|
depends_on('hdf5', when='@1.10.1:')
|
|
|
|
^^^^^^^^^^^^^^^^
|
|
No Version Works
|
|
^^^^^^^^^^^^^^^^
|
|
|
|
Sometimes, *no* existing versions of a dependency work for a build.
|
|
This typically happens when developing a new project: only then does
|
|
the developer notice that existing versions of a dependency are all
|
|
buggy, or the non-buggy versions are all missing a critical feature.
|
|
|
|
In the long run, the upstream project will hopefully fix the bug and
|
|
release a new version. But that could take a while, even if a bugfix
|
|
has already been pushed to the project's repository. In the meantime,
|
|
the Spack user needs things to work.
|
|
|
|
The solution is to create an unofficial Spack release of the project,
|
|
as soon as the bug is fixed in *some* repository. A study of the `Git
|
|
history <https://github.com/citibeth/spack/commits/efischer/develop/var/spack/repos/builtin/packages/py-proj/package.py>`_
|
|
of ``py-proj/package.py`` is instructive here:
|
|
|
|
#. On `April 1 <https://github.com/citibeth/spack/commit/44a1d6a96706affe6ef0a11c3a780b91d21d105a>`_, an initial bugfix was identified for the PyProj project
|
|
and a pull request submitted to PyProj. Because the upstream
|
|
authors had not yet fixed the bug, the ``py-proj`` Spack package
|
|
downloads from a forked repository, set up by the package's author.
|
|
A non-numeric version number is used to make it easy to upgrade the
|
|
package without recomputing checksums; however, this is an
|
|
untrusted download method and should not be distributed. The
|
|
package author has now become, temporarily, a maintainer of the
|
|
upstream project:
|
|
|
|
.. code-block:: python
|
|
|
|
# We need the benefits of this PR
|
|
# https://github.com/jswhit/pyproj/pull/54
|
|
version('citibeth-latlong2',
|
|
git='https://github.com/citibeth/pyproj.git',
|
|
branch='latlong2')
|
|
|
|
|
|
#. By May 14, the upstream project had accepted a pull request with
|
|
the required bugfix. At this point, the forked repository was
|
|
deleted. However, the upstream project still had not released a
|
|
new version with a bugfix. Therefore, a Spack-only release was
|
|
created by specifying the desired hash in the main project
|
|
repository. The version number ``@1.9.5.1.1`` was chosen for this
|
|
"release" because it's a descendent of the officially released
|
|
version ``@1.9.5.1``. This is a trusted download method, and can
|
|
be released to the Spack community:
|
|
|
|
.. code-block:: python
|
|
|
|
# This is not a tagged release of pyproj.
|
|
# The changes in this "version" fix some bugs, especially with Python3 use.
|
|
version('1.9.5.1.1', 'd035e4bc704d136db79b43ab371b27d2',
|
|
url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f')
|
|
|
|
.. note::
|
|
|
|
It would have been simpler to use Spack's Git download method,
|
|
which is also a trusted download in this case:
|
|
|
|
.. code-block:: python
|
|
|
|
# This is not a tagged release of pyproj.
|
|
# The changes in this "version" fix some bugs, especially with Python3 use.
|
|
version('1.9.5.1.1',
|
|
git='https://github.com/jswhit/pyproj.git',
|
|
commit='0be612cc9f972e38b50a90c946a9b353e2ab140f')
|
|
|
|
.. note::
|
|
|
|
In this case, the upstream project fixed the bug in its
|
|
repository in a relatively timely manner. If that had not been
|
|
the case, the numbered version in this step could have been
|
|
released from the forked repository.
|
|
|
|
|
|
#. The author of the Spack package has now become an unofficial
|
|
release engineer for the upstream project. Depending on the
|
|
situation, it may be advisable to put ``preferred=True`` on the
|
|
latest *officially released* version.
|
|
|
|
#. As of August 31, the upstream project still had not made a new
|
|
release with the bugfix. In the meantime, Spack-built ``py-proj``
|
|
provides the bugfix needed by packages depending on it. As long as
|
|
this works, there is no particular need for the upstream project to
|
|
make a new official release.
|
|
|
|
#. If the upstream project releases a new official version with the
|
|
bugfix, then the unofficial ``version()`` line should be removed
|
|
from the Spack package.
|
|
|
|
^^^^^^^
|
|
Patches
|
|
^^^^^^^
|
|
|
|
Spack's source patching mechanism provides another way to fix bugs in
|
|
upstream projects. This has advantages and disadvantages compared to the procedures above.
|
|
|
|
Advantages:
|
|
|
|
1. It can fix bugs in existing released versions, and (probably)
|
|
future releases as well.
|
|
|
|
2. It is lightweight, does not require a new fork to be set up.
|
|
|
|
Disadvantages:
|
|
|
|
1. It is harder to develop and debug a patch, vs. a branch in a
|
|
repository. The user loses the automation provided by version
|
|
control systems.
|
|
|
|
2. Although patches of a few lines work OK, large patch files can be
|
|
hard to create and maintain.
|