Packaging docs for MPI (#2838)
* Add MPI docs to packaging guide and simplify packaging guide TOC a bit.
This commit is contained in:
Todd Gamblin
GitHub
parent
81a5146b1d
commit
ffb8aaa3f3
@ -405,70 +405,22 @@ For tarball downloads, Spack can currently support checksums using the
|
|||||||
MD5, SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 algorithms. It
|
MD5, SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 algorithms. It
|
||||||
determines the algorithm to use based on the hash length.
|
determines the algorithm to use based on the hash length.
|
||||||
|
|
||||||
-----------------------
|
---------------------
|
||||||
Package Version Numbers
|
Versions and fetching
|
||||||
-----------------------
|
---------------------
|
||||||
|
|
||||||
Most Spack versions are numeric, a tuple of integers; for example,
|
The most straightforward way to add new versions to your package is to
|
||||||
``apex@0.1``, ``ferret@6.96`` or ``py-netcdf@1.2.3.1``. Spack knows
|
add a line like this in the package class:
|
||||||
how to compare and sort numeric versions.
|
|
||||||
|
|
||||||
Some Spack versions involve slight extensions of numeric syntax; for
|
.. code-block:: python
|
||||||
example, ``py-sphinx-rtd-theme@0.1.10a0``. In this case, numbers are
|
:linenos:
|
||||||
always considered to be "newer" than letters. This is for consistency
|
|
||||||
with `RPM <https://bugzilla.redhat.com/show_bug.cgi?id=50977>`_.
|
|
||||||
|
|
||||||
Spack versions may also be arbitrary non-numeric strings; any string
|
class Foo(Package):
|
||||||
here will suffice; for example, ``@develop``, ``@master``, ``@local``.
|
url = 'http://example.com/foo-1.0.tar.gz'
|
||||||
The following rules determine the sort order of numeric
|
version('8.2.1', '4136d7b4c04df68b686570afa26988ac')
|
||||||
vs. non-numeric versions:
|
...
|
||||||
|
|
||||||
#. The non-numeric versions ``@develop`` is considered greatest (newest).
|
Versions should be listed with the newest version first.
|
||||||
|
|
||||||
#. Numeric versions are all less than ``@develop`` version, and are
|
|
||||||
sorted numerically.
|
|
||||||
|
|
||||||
#. All other non-numeric versions are less than numeric versions, and
|
|
||||||
are sorted alphabetically.
|
|
||||||
|
|
||||||
The logic behind this sort order is two-fold:
|
|
||||||
|
|
||||||
#. Non-numeric versions are usually used for special cases while
|
|
||||||
developing or debugging a piece of software. Keeping most of them
|
|
||||||
less than numeric versions ensures that Spack choose numeric
|
|
||||||
versions by default whenever possible.
|
|
||||||
|
|
||||||
#. The most-recent development version of a package will usually be
|
|
||||||
newer than any released numeric versions. This allows the
|
|
||||||
``develop`` version to satisfy dependencies like ``depends_on(abc,
|
|
||||||
when="@x.y.z:")``
|
|
||||||
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
Concretization Version Selection
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
|
|
||||||
When concretizing, many versions might match a user-supplied spec.
|
|
||||||
For example, the spec ``python`` matches all available versions of the
|
|
||||||
package ``python``. Similarly, ``python@3:`` matches all versions of
|
|
||||||
Python3. Given a set of versions that match a spec, Spack
|
|
||||||
concretization uses the following priorities to decide which one to
|
|
||||||
use:
|
|
||||||
|
|
||||||
#. If the user provided a list of versions in ``packages.yaml``, the
|
|
||||||
first matching version in that list will be used.
|
|
||||||
|
|
||||||
#. If one or more versions is specified as ``preferred=True``, in
|
|
||||||
either ``packages.yaml`` or ``package.py``, the largest matching
|
|
||||||
version will be used. ("Latest" is defined by the sort order
|
|
||||||
above).
|
|
||||||
|
|
||||||
#. If no preferences in particular are specified in the package or in
|
|
||||||
``packages.yaml``, then the largest matching non-develop version
|
|
||||||
will be used. By avoiding ``@develop``, this prevents users from
|
|
||||||
accidentally installing a ``@develop`` version.
|
|
||||||
|
|
||||||
#. If all else fails and ``@develop`` is the only matching version, it
|
|
||||||
will be used.
|
|
||||||
|
|
||||||
^^^^^^^^^^^^^
|
^^^^^^^^^^^^^
|
||||||
Date Versions
|
Date Versions
|
||||||
@ -484,24 +436,6 @@ Alternately, you might use a hybrid release-version / date scheme.
|
|||||||
For example, ``@1.3.2016.08.31`` would mean the version from the
|
For example, ``@1.3.2016.08.31`` would mean the version from the
|
||||||
``1.3`` branch, as of August 31, 2016.
|
``1.3`` branch, as of August 31, 2016.
|
||||||
|
|
||||||
|
|
||||||
-------------------
|
|
||||||
Adding new versions
|
|
||||||
-------------------
|
|
||||||
|
|
||||||
The most straightforward way to add new versions to your package is to
|
|
||||||
add a line like this in the package class:
|
|
||||||
|
|
||||||
.. code-block:: python
|
|
||||||
:linenos:
|
|
||||||
|
|
||||||
class Foo(Package):
|
|
||||||
url = 'http://example.com/foo-1.0.tar.gz'
|
|
||||||
version('8.2.1', '4136d7b4c04df68b686570afa26988ac')
|
|
||||||
...
|
|
||||||
|
|
||||||
Versions should be listed with the newest version first.
|
|
||||||
|
|
||||||
^^^^^^^^^^^^
|
^^^^^^^^^^^^
|
||||||
Version URLs
|
Version URLs
|
||||||
^^^^^^^^^^^^
|
^^^^^^^^^^^^
|
||||||
@ -566,6 +500,37 @@ way to guess the URL systematically.
|
|||||||
When you supply a custom URL for a version, Spack uses that URL
|
When you supply a custom URL for a version, Spack uses that URL
|
||||||
*verbatim* and does not perform extrapolation.
|
*verbatim* and does not perform extrapolation.
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
PyPI and version URLs
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
In addition to their developer websites, many python packages are hosted at the
|
||||||
|
`Python Package Index (PyPi) <https://pypi.python.org/pypi>`_. Although links to
|
||||||
|
these individual files are typically `generated using a hash
|
||||||
|
<https://bitbucket.org/pypa/pypi/issues/438>`_ it is often possible to find a
|
||||||
|
reliable link of the format
|
||||||
|
|
||||||
|
.. code-block:: sh
|
||||||
|
|
||||||
|
https://pypi.python.org/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
|
||||||
|
|
||||||
|
Packages hosted on GitHub and the like are often developer versions that do not
|
||||||
|
contain all of the files (e.g. configuration scripts) necessary to support
|
||||||
|
compilation. For this reason it is ideal to link to a repository such as PyPi
|
||||||
|
if possible.
|
||||||
|
|
||||||
|
More recently, sources are being indexed at `pypi.io <https://pypi.io>`_ as
|
||||||
|
well. Links obtained from this site follow a similar pattern, namely
|
||||||
|
|
||||||
|
.. code-block:: sh
|
||||||
|
|
||||||
|
https://pypi.io/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
|
||||||
|
|
||||||
|
These links currently redirect back to `pypi.python.org
|
||||||
|
<https://pypi.python.org>`_, but this `may change in the future
|
||||||
|
<https://bitbucket.org/pypa/pypi/issues/438#comment-27243225>`_.
|
||||||
|
|
||||||
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
Skipping the expand step
|
Skipping the expand step
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
@ -596,6 +561,79 @@ it executable, then runs it with some arguments.
|
|||||||
installer = Executable(self.stage.archive_file)
|
installer = Executable(self.stage.archive_file)
|
||||||
installer('--prefix=%s' % prefix, 'arg1', 'arg2', 'etc.')
|
installer('--prefix=%s' % prefix, 'arg1', 'arg2', 'etc.')
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^
|
||||||
|
Download caching
|
||||||
|
^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Spack maintains a cache (described :ref:`here <caching>`) which saves files
|
||||||
|
retrieved during package installations to avoid re-downloading in the case that
|
||||||
|
a package is installed with a different specification (but the same version) or
|
||||||
|
reinstalled on account of a change in the hashing scheme.
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^^
|
||||||
|
Version comparison
|
||||||
|
^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Most Spack versions are numeric, a tuple of integers; for example,
|
||||||
|
``apex@0.1``, ``ferret@6.96`` or ``py-netcdf@1.2.3.1``. Spack knows
|
||||||
|
how to compare and sort numeric versions.
|
||||||
|
|
||||||
|
Some Spack versions involve slight extensions of numeric syntax; for
|
||||||
|
example, ``py-sphinx-rtd-theme@0.1.10a0``. In this case, numbers are
|
||||||
|
always considered to be "newer" than letters. This is for consistency
|
||||||
|
with `RPM <https://bugzilla.redhat.com/show_bug.cgi?id=50977>`_.
|
||||||
|
|
||||||
|
Spack versions may also be arbitrary non-numeric strings; any string
|
||||||
|
here will suffice; for example, ``@develop``, ``@master``, ``@local``.
|
||||||
|
The following rules determine the sort order of numeric
|
||||||
|
vs. non-numeric versions:
|
||||||
|
|
||||||
|
#. The non-numeric versions ``@develop`` is considered greatest (newest).
|
||||||
|
|
||||||
|
#. Numeric versions are all less than ``@develop`` version, and are
|
||||||
|
sorted numerically.
|
||||||
|
|
||||||
|
#. All other non-numeric versions are less than numeric versions, and
|
||||||
|
are sorted alphabetically.
|
||||||
|
|
||||||
|
The logic behind this sort order is two-fold:
|
||||||
|
|
||||||
|
#. Non-numeric versions are usually used for special cases while
|
||||||
|
developing or debugging a piece of software. Keeping most of them
|
||||||
|
less than numeric versions ensures that Spack choose numeric
|
||||||
|
versions by default whenever possible.
|
||||||
|
|
||||||
|
#. The most-recent development version of a package will usually be
|
||||||
|
newer than any released numeric versions. This allows the
|
||||||
|
``develop`` version to satisfy dependencies like ``depends_on(abc,
|
||||||
|
when="@x.y.z:")``
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^
|
||||||
|
Version selection
|
||||||
|
^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
When concretizing, many versions might match a user-supplied spec.
|
||||||
|
For example, the spec ``python`` matches all available versions of the
|
||||||
|
package ``python``. Similarly, ``python@3:`` matches all versions of
|
||||||
|
Python3. Given a set of versions that match a spec, Spack
|
||||||
|
concretization uses the following priorities to decide which one to
|
||||||
|
use:
|
||||||
|
|
||||||
|
#. If the user provided a list of versions in ``packages.yaml``, the
|
||||||
|
first matching version in that list will be used.
|
||||||
|
|
||||||
|
#. If one or more versions is specified as ``preferred=True``, in
|
||||||
|
either ``packages.yaml`` or ``package.py``, the largest matching
|
||||||
|
version will be used. ("Latest" is defined by the sort order
|
||||||
|
above).
|
||||||
|
|
||||||
|
#. If no preferences in particular are specified in the package or in
|
||||||
|
``packages.yaml``, then the largest matching non-develop version
|
||||||
|
will be used. By avoiding ``@develop``, this prevents users from
|
||||||
|
accidentally installing a ``@develop`` version.
|
||||||
|
|
||||||
|
#. If all else fails and ``@develop`` is the only matching version, it
|
||||||
|
will be used.
|
||||||
|
|
||||||
^^^^^^^^^^^^^
|
^^^^^^^^^^^^^
|
||||||
``spack md5``
|
``spack md5``
|
||||||
@ -693,9 +731,9 @@ versions. See the documentation on `attribute_list_url`_ and
|
|||||||
|
|
||||||
.. _vcs-fetch:
|
.. _vcs-fetch:
|
||||||
|
|
||||||
------------------------------
|
-------------------------------
|
||||||
Fetching from VCS repositories
|
Fetching from code repositories
|
||||||
------------------------------
|
-------------------------------
|
||||||
|
|
||||||
For some packages, source code is provided in a Version Control System
|
For some packages, source code is provided in a Version Control System
|
||||||
(VCS) repository rather than in a tarball. Spack can fetch packages
|
(VCS) repository rather than in a tarball. Spack can fetch packages
|
||||||
@ -793,9 +831,9 @@ Submodules
|
|||||||
|
|
||||||
.. _github-fetch:
|
.. _github-fetch:
|
||||||
|
|
||||||
""""""
|
^^^^^^
|
||||||
GitHub
|
GitHub
|
||||||
""""""
|
^^^^^^
|
||||||
|
|
||||||
If a project is hosted on GitHub, *any* valid Git branch, tag or hash
|
If a project is hosted on GitHub, *any* valid Git branch, tag or hash
|
||||||
may be downloaded as a tarball. This is accomplished simply by
|
may be downloaded as a tarball. This is accomplished simply by
|
||||||
@ -875,38 +913,8 @@ Fetching a revision
|
|||||||
Subversion branches are handled as part of the directory structure, so
|
Subversion branches are handled as part of the directory structure, so
|
||||||
you can check out a branch or tag by changing the ``url``.
|
you can check out a branch or tag by changing the ``url``.
|
||||||
|
|
||||||
-----------------------------------------
|
|
||||||
Standard repositories for python packages
|
|
||||||
-----------------------------------------
|
|
||||||
|
|
||||||
In addition to their developer websites, many python packages are hosted at the
|
|
||||||
`Python Package Index (PyPi) <https://pypi.python.org/pypi>`_. Although links to
|
|
||||||
these individual files are typically `generated using a hash
|
|
||||||
<https://bitbucket.org/pypa/pypi/issues/438>`_ it is often possible to find a
|
|
||||||
reliable link of the format
|
|
||||||
|
|
||||||
.. code-block:: sh
|
|
||||||
|
|
||||||
https://pypi.python.org/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
|
|
||||||
|
|
||||||
Packages hosted on GitHub and the like are often developer versions that do not
|
|
||||||
contain all of the files (e.g. configuration scripts) necessary to support
|
|
||||||
compilation. For this reason it is ideal to link to a repository such as PyPi
|
|
||||||
if possible.
|
|
||||||
|
|
||||||
More recently, sources are being indexed at `pypi.io <https://pypi.io>`_ as
|
|
||||||
well. Links obtained from this site follow a similar pattern, namely
|
|
||||||
|
|
||||||
.. code-block:: sh
|
|
||||||
|
|
||||||
https://pypi.io/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
|
|
||||||
|
|
||||||
These links currently redirect back to `pypi.python.org
|
|
||||||
<https://pypi.python.org>`_, but this `may change in the future
|
|
||||||
<https://bitbucket.org/pypa/pypi/issues/438#comment-27243225>`_.
|
|
||||||
|
|
||||||
-------------------------------------------------
|
-------------------------------------------------
|
||||||
Expanding additional resources in the source tree
|
Resources (expanding extra tarballs)
|
||||||
-------------------------------------------------
|
-------------------------------------------------
|
||||||
|
|
||||||
Some packages (most notably compilers) provide optional features if additional
|
Some packages (most notably compilers) provide optional features if additional
|
||||||
@ -926,15 +934,6 @@ Based on the keywords present among the arguments the appropriate ``FetchStrateg
|
|||||||
will be used for the resource. The keyword ``destination`` is relative to the source
|
will be used for the resource. The keyword ``destination`` is relative to the source
|
||||||
root of the package and should point to where the resource is to be expanded.
|
root of the package and should point to where the resource is to be expanded.
|
||||||
|
|
||||||
------------------------------------------------------
|
|
||||||
Automatic caching of files fetched during installation
|
|
||||||
------------------------------------------------------
|
|
||||||
|
|
||||||
Spack maintains a cache (described :ref:`here <caching>`) which saves files
|
|
||||||
retrieved during package installations to avoid re-downloading in the case that
|
|
||||||
a package is installed with a different specification (but the same version) or
|
|
||||||
reinstalled on account of a change in the hashing scheme.
|
|
||||||
|
|
||||||
.. _license:
|
.. _license:
|
||||||
|
|
||||||
-----------------
|
-----------------
|
||||||
@ -1099,20 +1098,28 @@ structure like this:
|
|||||||
package.py
|
package.py
|
||||||
ad_lustre_rwcontig_open_source.patch
|
ad_lustre_rwcontig_open_source.patch
|
||||||
|
|
||||||
If you supply a URL instead of a filename, the patch will be fetched
|
If you supply a URL instead of a filename, you need to supply a checksum,
|
||||||
from the URL and then applied to your source code.
|
like this:
|
||||||
|
|
||||||
.. warning::
|
.. code-block:: python
|
||||||
|
|
||||||
It is generally better to use a filename rather than a URL for your
|
patch('http://www.nwchem-sw.org/images/Tddft_mxvec20.patch.gz',
|
||||||
patch. Patches fetched from URLs are not currently checksummed,
|
md5='f91c6a04df56e228fe946291d2f38c9a')
|
||||||
and adding checksums for them is tedious for the package builder.
|
|
||||||
File patches go into the spack repository, which gives you git's
|
This directive provides an ``md5`` checksum. You can use other hashing
|
||||||
integrity guarantees. URL patches may be removed in a future spack
|
algorihtms like ``sha256`` as well. The patch will be fetched from the
|
||||||
version.
|
URL, checked, and applied to your source code. You can use the ``spack
|
||||||
|
md5`` command to generate a checksum for a patch file.
|
||||||
|
|
||||||
``patch`` can take two options keyword arguments. They are:
|
``patch`` can take two options keyword arguments. They are:
|
||||||
|
|
||||||
|
""""""""""""""""""""""""""""""""""""""
|
||||||
|
``md5``, ``sha256``, ``sha512``, etc.
|
||||||
|
""""""""""""""""""""""""""""""""""""""
|
||||||
|
|
||||||
|
Use one of these when you supply a patch to be downloaded from a remote
|
||||||
|
site. The downloaded file will be validated using the given checksum.
|
||||||
|
|
||||||
""""""""
|
""""""""
|
||||||
``when``
|
``when``
|
||||||
""""""""
|
""""""""
|
||||||
@ -1960,7 +1967,7 @@ See the :ref:`concretization-preferences` section for more details.
|
|||||||
.. _install-method:
|
.. _install-method:
|
||||||
|
|
||||||
------------------
|
------------------
|
||||||
Inconsistent Specs
|
Conflicting Specs
|
||||||
------------------
|
------------------
|
||||||
|
|
||||||
Suppose a user needs to install package C, which depends on packages A
|
Suppose a user needs to install package C, which depends on packages A
|
||||||
@ -2151,7 +2158,7 @@ built with. These parameters give you access to this type of information.
|
|||||||
.. _install-environment:
|
.. _install-environment:
|
||||||
|
|
||||||
-----------------------
|
-----------------------
|
||||||
The install environment
|
The build environment
|
||||||
-----------------------
|
-----------------------
|
||||||
|
|
||||||
In general, you should not have to do much differently in your install
|
In general, you should not have to do much differently in your install
|
||||||
@ -2169,6 +2176,17 @@ custom Makefiles, you may need to add logic to modify the makefiles.
|
|||||||
The remainder of the section covers the way Spack's build environment
|
The remainder of the section covers the way Spack's build environment
|
||||||
works.
|
works.
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
Forking ``install()``
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
To give packagers free reign over their install environment, Spack forks
|
||||||
|
a new process each time it invokes a package's ``install()`` method.
|
||||||
|
This allows packages to have a sandboxed build environment, without
|
||||||
|
impacting the environments ofother jobs that the main Spack process runs.
|
||||||
|
Packages are free to change the environment or to modify Spack internals,
|
||||||
|
because each ``install()`` call has its own dedicated process.
|
||||||
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
Environment variables
|
Environment variables
|
||||||
^^^^^^^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
@ -2190,6 +2208,10 @@ The Compiler environment variables that Spack sets are:
|
|||||||
``FC`` Fortran 90 and above compiler
|
``FC`` Fortran 90 and above compiler
|
||||||
============ ===============================
|
============ ===============================
|
||||||
|
|
||||||
|
Spack sets these variables so that they point to *compiler
|
||||||
|
wrappers*. These are covered in :ref:`their own section
|
||||||
|
<compiler-wrappers>` below.
|
||||||
|
|
||||||
All of these are standard variables respected by most build systems.
|
All of these are standard variables respected by most build systems.
|
||||||
If your project uses ``Autotools`` or ``CMake``, then it should pick
|
If your project uses ``Autotools`` or ``CMake``, then it should pick
|
||||||
them up automatically when you run ``configure`` or ``cmake`` in the
|
them up automatically when you run ``configure`` or ``cmake`` in the
|
||||||
@ -2237,145 +2259,9 @@ if you want to run commands in that environment to test them out, you
|
|||||||
can use the :ref:`cmd-spack-env` command, documented
|
can use the :ref:`cmd-spack-env` command, documented
|
||||||
below.
|
below.
|
||||||
|
|
||||||
.. _compiler-wrappers:
|
|
||||||
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
Compiler interceptors
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
|
|
||||||
As mentioned, ``CC``, ``CXX``, ``F77``, and ``FC`` are set to point to
|
|
||||||
Spack's compiler wrappers. These are simply called ``cc``, ``c++``,
|
|
||||||
``f77``, and ``f90``, and they live in ``$SPACK_ROOT/lib/spack/env``.
|
|
||||||
|
|
||||||
``$SPACK_ROOT/lib/spack/env`` is added first in the ``PATH``
|
|
||||||
environment variable when ``install()`` runs so that system compilers
|
|
||||||
are not picked up instead.
|
|
||||||
|
|
||||||
All of these compiler wrappers point to a single compiler wrapper
|
|
||||||
script that figures out which *real* compiler it should be building
|
|
||||||
with. This comes either from spec `concretization
|
|
||||||
<abstract-and-concrete>`_ or from a user explicitly asking for a
|
|
||||||
particular compiler using, e.g., ``%intel`` on the command line.
|
|
||||||
|
|
||||||
In addition to invoking the right compiler, the compiler wrappers add
|
|
||||||
flags to the compile line so that dependencies can be easily found.
|
|
||||||
These flags are added for each dependency, if they exist:
|
|
||||||
|
|
||||||
Compile-time library search paths
|
|
||||||
* ``-L$dep_prefix/lib``
|
|
||||||
* ``-L$dep_prefix/lib64``
|
|
||||||
|
|
||||||
Runtime library search paths (RPATHs)
|
|
||||||
* ``$rpath_flag$dep_prefix/lib``
|
|
||||||
* ``$rpath_flag$dep_prefix/lib64``
|
|
||||||
|
|
||||||
Include search paths
|
|
||||||
* ``-I$dep_prefix/include``
|
|
||||||
|
|
||||||
An example of this would be the ``libdwarf`` build, which has one
|
|
||||||
dependency: ``libelf``. Every call to ``cc`` in the ``libdwarf``
|
|
||||||
build will have ``-I$LIBELF_PREFIX/include``,
|
|
||||||
``-L$LIBELF_PREFIX/lib``, and ``$rpath_flag$LIBELF_PREFIX/lib``
|
|
||||||
inserted on the command line. This is done transparently to the
|
|
||||||
project's build system, which will just think it's using a system
|
|
||||||
where ``libelf`` is readily available. Because of this, you **do
|
|
||||||
not** have to insert extra ``-I``, ``-L``, etc. on the command line.
|
|
||||||
|
|
||||||
Another useful consequence of this is that you often do *not* have to
|
|
||||||
add extra parameters on the ``configure`` line to get autotools to
|
|
||||||
find dependencies. The ``libdwarf`` install method just calls
|
|
||||||
configure like this:
|
|
||||||
|
|
||||||
.. code-block:: python
|
|
||||||
|
|
||||||
configure("--prefix=" + prefix)
|
|
||||||
|
|
||||||
Because of the ``-L`` and ``-I`` arguments, configure will
|
|
||||||
successfully find ``libdwarf.h`` and ``libdwarf.so``, without the
|
|
||||||
packager having to provide ``--with-libdwarf=/path/to/libdwarf`` on
|
|
||||||
the command line.
|
|
||||||
|
|
||||||
.. note::
|
|
||||||
|
|
||||||
For most compilers, ``$rpath_flag`` is ``-Wl,-rpath,``. However, NAG
|
|
||||||
passes its flags to GCC instead of passing them directly to the linker.
|
|
||||||
Therefore, its ``$rpath_flag`` is doubly wrapped: ``-Wl,-Wl,,-rpath,``.
|
|
||||||
``$rpath_flag`` can be overriden on a compiler specific basis in
|
|
||||||
``lib/spack/spack/compilers/$compiler.py``.
|
|
||||||
|
|
||||||
The compiler wrappers also pass the compiler flags specified by the user from
|
|
||||||
the command line (``cflags``, ``cxxflags``, ``fflags``, ``cppflags``, ``ldflags``,
|
|
||||||
and/or ``ldlibs``). They do not override the canonical autotools flags with the
|
|
||||||
same names (but in ALL-CAPS) that may be passed into the build by particularly
|
|
||||||
challenging package scripts.
|
|
||||||
|
|
||||||
^^^^^^^^^^^^^^
|
|
||||||
Compiler flags
|
|
||||||
^^^^^^^^^^^^^^
|
|
||||||
|
|
||||||
In rare circumstances such as compiling and running small unit tests, a package
|
|
||||||
developer may need to know what are the appropriate compiler flags to enable
|
|
||||||
features like ``OpenMP``, ``c++11``, ``c++14`` and alike. To that end the
|
|
||||||
compiler classes in ``spack`` implement the following **properties**:
|
|
||||||
``openmp_flag``, ``cxx11_flag``, ``cxx14_flag``, which can be accessed in a
|
|
||||||
package by ``self.compiler.cxx11_flag`` and alike. Note that the implementation
|
|
||||||
is such that if a given compiler version does not support this feature, an
|
|
||||||
error will be produced. Therefore package developers can also use these properties
|
|
||||||
to assert that a compiler supports the requested feature. This is handy when a
|
|
||||||
package supports additional variants like
|
|
||||||
|
|
||||||
.. code-block:: python
|
|
||||||
|
|
||||||
variant('openmp', default=True, description="Enable OpenMP support.")
|
|
||||||
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
Message Parsing Interface (MPI)
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
|
|
||||||
It is common for high performance computing software/packages to use ``MPI``.
|
|
||||||
As a result of conretization, a given package can be built using different
|
|
||||||
implementations of MPI such as ``Openmpi``, ``MPICH`` or ``IntelMPI``.
|
|
||||||
In some scenarios, to configure a package, one has to provide it with appropriate MPI
|
|
||||||
compiler wrappers such as ``mpicc``, ``mpic++``.
|
|
||||||
However different implementations of ``MPI`` may have different names for those
|
|
||||||
wrappers. In order to make package's ``install()`` method indifferent to the
|
|
||||||
choice ``MPI`` implementation, each package which implements ``MPI`` sets up
|
|
||||||
``self.spec.mpicc``, ``self.spec.mpicxx``, ``self.spec.mpifc`` and ``self.spec.mpif77``
|
|
||||||
to point to ``C``, ``C++``, ``Fortran 90`` and ``Fortran 77`` ``MPI`` wrappers.
|
|
||||||
Package developers are advised to use these variables, for example ``self.spec['mpi'].mpicc``
|
|
||||||
instead of hard-coding ``join_path(self.spec['mpi'].prefix.bin, 'mpicc')`` for
|
|
||||||
the reasons outlined above.
|
|
||||||
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
Blas and Lapack libraries
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
|
|
||||||
Different packages provide implementation of ``Blas`` and ``Lapack`` routines.
|
|
||||||
The names of the resulting static and/or shared libraries differ from package
|
|
||||||
to package. In order to make the ``install()`` method independent of the
|
|
||||||
choice of ``Blas`` implementation, each package which provides it
|
|
||||||
sets up ``self.spec.blas_libs`` to point to the correct ``Blas`` libraries.
|
|
||||||
The same applies to packages which provide ``Lapack``. Package developers are advised to
|
|
||||||
use these variables, for example ``spec['blas'].blas_libs.joined()`` instead of
|
|
||||||
hard-coding ``join_path(spec['blas'].prefix.lib, 'libopenblas.so')``.
|
|
||||||
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
Forking ``install()``
|
|
||||||
^^^^^^^^^^^^^^^^^^^^^
|
|
||||||
|
|
||||||
To give packagers free reign over their install environment, Spack
|
|
||||||
forks a new process each time it invokes a package's ``install()``
|
|
||||||
method. This allows packages to have their own completely sandboxed
|
|
||||||
build environment, without impacting other jobs that the main Spack
|
|
||||||
process runs. Packages are free to change the environment or to
|
|
||||||
modify Spack internals, because each ``install()`` call has its own
|
|
||||||
dedicated process.
|
|
||||||
|
|
||||||
.. _prefix-objects:
|
|
||||||
|
|
||||||
-----------------
|
|
||||||
Failing the build
|
Failing the build
|
||||||
-----------------
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
Sometimes you don't want a package to successfully install unless some
|
Sometimes you don't want a package to successfully install unless some
|
||||||
condition is true. You can explicitly cause the build to fail from
|
condition is true. You can explicitly cause the build to fail from
|
||||||
@ -2386,9 +2272,89 @@ condition is true. You can explicitly cause the build to fail from
|
|||||||
if spec.architecture.startswith('darwin'):
|
if spec.architecture.startswith('darwin'):
|
||||||
raise InstallError('This package does not build on Mac OS X!')
|
raise InstallError('This package does not build on Mac OS X!')
|
||||||
|
|
||||||
--------------
|
.. _shell-wrappers:
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
Shell command functions
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Recall the install method from ``libelf``:
|
||||||
|
|
||||||
|
.. literalinclude:: ../../../var/spack/repos/builtin/packages/libelf/package.py
|
||||||
|
:pyobject: Libelf.install
|
||||||
|
:linenos:
|
||||||
|
|
||||||
|
Normally in Python, you'd have to write something like this in order
|
||||||
|
to execute shell commands:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
import subprocess
|
||||||
|
subprocess.check_call('configure', '--prefix={0}'.format(prefix))
|
||||||
|
|
||||||
|
We've tried to make this a bit easier by providing callable wrapper
|
||||||
|
objects for some shell commands. By default, ``configure``,
|
||||||
|
``cmake``, and ``make`` wrappers are are provided, so you can call
|
||||||
|
them more naturally in your package files.
|
||||||
|
|
||||||
|
If you need other commands, you can use ``which`` to get them:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
sed = which('sed')
|
||||||
|
sed('s/foo/bar/', filename)
|
||||||
|
|
||||||
|
The ``which`` function will search the ``PATH`` for the application.
|
||||||
|
|
||||||
|
Callable wrappers also allow spack to provide some special features.
|
||||||
|
For example, in Spack, ``make`` is parallel by default, and Spack
|
||||||
|
figures out the number of cores on your machine and passes an
|
||||||
|
appropriate value for ``-j<numjobs>`` when it calls ``make`` (see the
|
||||||
|
``parallel`` `package attribute <attribute_parallel>`). In
|
||||||
|
a package file, you can supply a keyword argument, ``parallel=False``,
|
||||||
|
to the ``make`` wrapper to disable parallel make. In the ``libelf``
|
||||||
|
package, this allows us to avoid race conditions in the library's
|
||||||
|
build system.
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^
|
||||||
|
Compiler flags
|
||||||
|
^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
In rare circumstances such as compiling and running small unit tests, a
|
||||||
|
package developer may need to know what are the appropriate compiler
|
||||||
|
flags to enable features like ``OpenMP``, ``c++11``, ``c++14`` and
|
||||||
|
alike. To that end the compiler classes in ``spack`` implement the
|
||||||
|
following **properties**: ``openmp_flag``, ``cxx11_flag``,
|
||||||
|
``cxx14_flag``, which can be accessed in a package by
|
||||||
|
``self.compiler.cxx11_flag`` and alike. Note that the implementation is
|
||||||
|
such that if a given compiler version does not support this feature, an
|
||||||
|
error will be produced. Therefore package developers can also use these
|
||||||
|
properties to assert that a compiler supports the requested feature. This
|
||||||
|
is handy when a package supports additional variants like
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
variant('openmp', default=True, description="Enable OpenMP support.")
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
Blas and Lapack libraries
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Different packages provide implementation of ``Blas`` and ``Lapack``
|
||||||
|
routines. The names of the resulting static and/or shared libraries
|
||||||
|
differ from package to package. In order to make the ``install()`` method
|
||||||
|
independent of the choice of ``Blas`` implementation, each package which
|
||||||
|
provides it sets up ``self.spec.blas_libs`` to point to the correct
|
||||||
|
``Blas`` libraries. The same applies to packages which provide
|
||||||
|
``Lapack``. Package developers are advised to use these variables, for
|
||||||
|
example ``spec['blas'].blas_libs.joined()`` instead of hard-coding
|
||||||
|
``join_path(spec['blas'].prefix.lib, 'libopenblas.so')``.
|
||||||
|
|
||||||
|
.. _prefix-objects:
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
Prefix objects
|
Prefix objects
|
||||||
--------------
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
Spack passes the ``prefix`` parameter to the install method so that
|
Spack passes the ``prefix`` parameter to the install method so that
|
||||||
you can pass it to ``configure``, ``cmake``, or some other installer,
|
you can pass it to ``configure``, ``cmake``, or some other installer,
|
||||||
@ -2657,50 +2623,248 @@ method (the one without the ``@when`` decorator) will be called.
|
|||||||
versions. There's not much we can do to get around this because of
|
versions. There's not much we can do to get around this because of
|
||||||
the way decorators work.
|
the way decorators work.
|
||||||
|
|
||||||
|
.. _compiler-wrappers:
|
||||||
|
|
||||||
.. _shell-wrappers:
|
---------------------
|
||||||
|
Compiler wrappers
|
||||||
|
---------------------
|
||||||
|
|
||||||
-----------------------
|
As mentioned, ``CC``, ``CXX``, ``F77``, and ``FC`` are set to point to
|
||||||
Shell command functions
|
Spack's compiler wrappers. These are simply called ``cc``, ``c++``,
|
||||||
-----------------------
|
``f77``, and ``f90``, and they live in ``$SPACK_ROOT/lib/spack/env``.
|
||||||
|
|
||||||
Recall the install method from ``libelf``:
|
``$SPACK_ROOT/lib/spack/env`` is added first in the ``PATH``
|
||||||
|
environment variable when ``install()`` runs so that system compilers
|
||||||
|
are not picked up instead.
|
||||||
|
|
||||||
.. literalinclude:: ../../../var/spack/repos/builtin/packages/libelf/package.py
|
All of these compiler wrappers point to a single compiler wrapper
|
||||||
:pyobject: Libelf.install
|
script that figures out which *real* compiler it should be building
|
||||||
:linenos:
|
with. This comes either from spec `concretization
|
||||||
|
<abstract-and-concrete>`_ or from a user explicitly asking for a
|
||||||
|
particular compiler using, e.g., ``%intel`` on the command line.
|
||||||
|
|
||||||
Normally in Python, you'd have to write something like this in order
|
In addition to invoking the right compiler, the compiler wrappers add
|
||||||
to execute shell commands:
|
flags to the compile line so that dependencies can be easily found.
|
||||||
|
These flags are added for each dependency, if they exist:
|
||||||
|
|
||||||
|
Compile-time library search paths
|
||||||
|
* ``-L$dep_prefix/lib``
|
||||||
|
* ``-L$dep_prefix/lib64``
|
||||||
|
|
||||||
|
Runtime library search paths (RPATHs)
|
||||||
|
* ``$rpath_flag$dep_prefix/lib``
|
||||||
|
* ``$rpath_flag$dep_prefix/lib64``
|
||||||
|
|
||||||
|
Include search paths
|
||||||
|
* ``-I$dep_prefix/include``
|
||||||
|
|
||||||
|
An example of this would be the ``libdwarf`` build, which has one
|
||||||
|
dependency: ``libelf``. Every call to ``cc`` in the ``libdwarf``
|
||||||
|
build will have ``-I$LIBELF_PREFIX/include``,
|
||||||
|
``-L$LIBELF_PREFIX/lib``, and ``$rpath_flag$LIBELF_PREFIX/lib``
|
||||||
|
inserted on the command line. This is done transparently to the
|
||||||
|
project's build system, which will just think it's using a system
|
||||||
|
where ``libelf`` is readily available. Because of this, you **do
|
||||||
|
not** have to insert extra ``-I``, ``-L``, etc. on the command line.
|
||||||
|
|
||||||
|
Another useful consequence of this is that you often do *not* have to
|
||||||
|
add extra parameters on the ``configure`` line to get autotools to
|
||||||
|
find dependencies. The ``libdwarf`` install method just calls
|
||||||
|
configure like this:
|
||||||
|
|
||||||
.. code-block:: python
|
.. code-block:: python
|
||||||
|
|
||||||
import subprocess
|
configure("--prefix=" + prefix)
|
||||||
subprocess.check_call('configure', '--prefix={0}'.format(prefix))
|
|
||||||
|
|
||||||
We've tried to make this a bit easier by providing callable wrapper
|
Because of the ``-L`` and ``-I`` arguments, configure will
|
||||||
objects for some shell commands. By default, ``configure``,
|
successfully find ``libdwarf.h`` and ``libdwarf.so``, without the
|
||||||
``cmake``, and ``make`` wrappers are are provided, so you can call
|
packager having to provide ``--with-libdwarf=/path/to/libdwarf`` on
|
||||||
them more naturally in your package files.
|
the command line.
|
||||||
|
|
||||||
If you need other commands, you can use ``which`` to get them:
|
.. note::
|
||||||
|
|
||||||
|
For most compilers, ``$rpath_flag`` is ``-Wl,-rpath,``. However, NAG
|
||||||
|
passes its flags to GCC instead of passing them directly to the linker.
|
||||||
|
Therefore, its ``$rpath_flag`` is doubly wrapped: ``-Wl,-Wl,,-rpath,``.
|
||||||
|
``$rpath_flag`` can be overriden on a compiler specific basis in
|
||||||
|
``lib/spack/spack/compilers/$compiler.py``.
|
||||||
|
|
||||||
|
The compiler wrappers also pass the compiler flags specified by the user from
|
||||||
|
the command line (``cflags``, ``cxxflags``, ``fflags``, ``cppflags``, ``ldflags``,
|
||||||
|
and/or ``ldlibs``). They do not override the canonical autotools flags with the
|
||||||
|
same names (but in ALL-CAPS) that may be passed into the build by particularly
|
||||||
|
challenging package scripts.
|
||||||
|
|
||||||
|
---------------------
|
||||||
|
MPI support in Spack
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
It is common for high performance computing software/packages to use the
|
||||||
|
Message Passing Interface ( ``MPI``). As a result of conretization, a
|
||||||
|
given package can be built using different implementations of MPI such as
|
||||||
|
``Openmpi``, ``MPICH`` or ``IntelMPI``. That is, when your package
|
||||||
|
declares that it ``depends_on('mpi')``, it can be built with any of these
|
||||||
|
``mpi`` implementations. In some scenarios, to configure a package, one
|
||||||
|
has to provide it with appropriate MPI compiler wrappers such as
|
||||||
|
``mpicc``, ``mpic++``. However different implementations of ``MPI`` may
|
||||||
|
have different names for those wrappers.
|
||||||
|
|
||||||
|
Spack provides an idiomatic way to use MPI compilers in your package. To
|
||||||
|
use MPI wrappers to compile your whole build, do this in your
|
||||||
|
``install()`` method:
|
||||||
|
|
||||||
.. code-block:: python
|
.. code-block:: python
|
||||||
|
|
||||||
sed = which('sed')
|
env['CC'] = spec['mpi'].mpicc
|
||||||
sed('s/foo/bar/', filename)
|
env['CXX'] = spec['mpi'].mpicxx
|
||||||
|
env['F77'] = spec['mpi'].mpif77
|
||||||
|
env['FC'] = spec['mpi'].mpifc
|
||||||
|
|
||||||
The ``which`` function will search the ``PATH`` for the application.
|
That's all. A longer explanation of why this works is below.
|
||||||
|
|
||||||
Callable wrappers also allow spack to provide some special features.
|
We don't try to force any particular build method on packagers. The
|
||||||
For example, in Spack, ``make`` is parallel by default, and Spack
|
decision to use MPI wrappers depends on the way the package is written,
|
||||||
figures out the number of cores on your machine and passes an
|
on common practice, and on "what works". Loosely, There are three types
|
||||||
appropriate value for ``-j<numjobs>`` when it calls ``make`` (see the
|
of MPI builds:
|
||||||
``parallel`` `package attribute <attribute_parallel>`). In
|
|
||||||
a package file, you can supply a keyword argument, ``parallel=False``,
|
1. Some build systems work well without the wrappers and can treat MPI
|
||||||
to the ``make`` wrapper to disable parallel make. In the ``libelf``
|
as an external library, where the person doing the build has to
|
||||||
package, this allows us to avoid race conditions in the library's
|
supply includes/libs/etc. This is fairly uncommon.
|
||||||
build system.
|
|
||||||
|
2. Others really want the wrappers and assume you're using an MPI
|
||||||
|
"compiler" – i.e., they have no mechanism to add MPI
|
||||||
|
includes/libraries/etc.
|
||||||
|
|
||||||
|
3. CMake's ``FindMPI`` needs the compiler wrappers, but it uses them to
|
||||||
|
extract ``–I`` / ``-L`` / ``-D`` arguments, then treats MPI like a
|
||||||
|
regular library.
|
||||||
|
|
||||||
|
Note that some CMake builds fall into case 2 because they either don't
|
||||||
|
know about or don't like CMake's ``FindMPI`` support – they just assume
|
||||||
|
an MPI compiler. Also, some autotools builds fall into case 3 (e.g. `here
|
||||||
|
is an autotools version of CMake's FindMPI
|
||||||
|
<https://github.com/tgamblin/libra/blob/master/m4/lx_find_mpi.m4>`_).
|
||||||
|
|
||||||
|
Given all of this, we leave the use of the wrappers up to the packager.
|
||||||
|
Spack will support all three ways of building MPI packages.
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
Packaging Conventions
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
As mentioned above, in the ``install()`` method, ``CC``, ``CXX``,
|
||||||
|
``F77``, and ``FC`` point to Spack's wrappers around the chosen compiler.
|
||||||
|
Spack's wrappers are not the MPI compiler wrappers, though they do
|
||||||
|
automatically add ``–I``, ``–L``, and ``–Wl,-rpath`` args for
|
||||||
|
dependencies in a similar way. The MPI wrappers are a bit different in
|
||||||
|
that they also add ``-l`` arguments for the MPI libraries, and some add
|
||||||
|
special ``-D`` arguments to trigger build options in MPI programs.
|
||||||
|
|
||||||
|
For case 1 above, you generally don't need to do more than patch your
|
||||||
|
Makefile or add configure args as you normally would.
|
||||||
|
|
||||||
|
For case 3, you don't need to do much of anything, as Spack puts the MPI
|
||||||
|
compiler wrappers in the PATH, and the build will find them and
|
||||||
|
interrogate them.
|
||||||
|
|
||||||
|
For case 2, things are a bit more complicated, as you'll need to tell the
|
||||||
|
build to use the MPI compiler wrappers instead of Spack's compiler
|
||||||
|
wrappers. All it takes some lines like this:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
env['CC'] = spec['mpi'].mpicc
|
||||||
|
env['CXX'] = spec['mpi'].mpicxx
|
||||||
|
env['F77'] = spec['mpi'].mpif77
|
||||||
|
env['FC'] = spec['mpi'].mpifc
|
||||||
|
|
||||||
|
Or, if you pass CC, CXX, etc. directly to your build with, e.g.,
|
||||||
|
`--with-cc=<path>`, you'll want to substitute `spec['mpi'].mpicc` in
|
||||||
|
there instead, e.g.:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
configure('—prefix=%s' % prefix,
|
||||||
|
'—with-cc=%s' % spec['mpi'].mpicc)
|
||||||
|
|
||||||
|
Now, you may think that doing this will lose the includes, library paths,
|
||||||
|
and RPATHs that Spack's compiler wrapper get you, but we've actually set
|
||||||
|
things up so that the MPI compiler wrappers use Spack's compiler wrappers
|
||||||
|
when run from within Spack. So using the MPI wrappers should really be as
|
||||||
|
simple as the code above.
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
``spec['mpi']``
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Ok, so how does all this work?
|
||||||
|
|
||||||
|
If your package has a virtual dependency like ``mpi``, then referring to
|
||||||
|
``spec['mpi']`` within ``install()`` will get you the concrete ``mpi``
|
||||||
|
implementation in your dependency DAG. That is a spec object just like
|
||||||
|
the one passed to install, only the MPI implementations all set some
|
||||||
|
additional properties on it to help you out. E.g., in mvapich2, you'll
|
||||||
|
find this:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
def setup_dependent_package(self, module, dep_spec):
|
||||||
|
self.spec.mpicc = join_path(self.prefix.bin, 'mpicc')
|
||||||
|
# … etc …
|
||||||
|
|
||||||
|
That code allows the mvapich2 package to associate an ``mpicc`` property
|
||||||
|
with the ``mvapich2`` node in the DAG, so that dependents can access it.
|
||||||
|
``openmpi`` and ``mpich`` do similar things. So, no matter what MPI
|
||||||
|
you're using, spec['mpi'].mpicc gets you the location of the MPI
|
||||||
|
compilers. This allows us to have a fairly simple polymorphic interface
|
||||||
|
for information about virtual dependencies like MPI.
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
Wrapping wrappers
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
Spack likes to use its own compiler wrappers to make it easy to add
|
||||||
|
``RPATHs`` to builds, and to try hard to ensure that your builds use the
|
||||||
|
right dependencies. This doesn't play nicely by default with MPI, so we
|
||||||
|
have to do a couple tricks.
|
||||||
|
|
||||||
|
1. If we build MPI with Spack's wrappers, mpicc and friends will be
|
||||||
|
installed with hard-coded paths to Spack's wrappers, and using them
|
||||||
|
from outside of Spack will fail because they only work within Spack.
|
||||||
|
To fix this, we patch mpicc and friends to use the regular
|
||||||
|
compilers. Look at the filter_compilers method in mpich, openmpi,
|
||||||
|
or mvapich2 for details.
|
||||||
|
|
||||||
|
2. We still want to use the Spack compiler wrappers when Spack is
|
||||||
|
calling mpicc. Luckily, wrappers in all mainstream MPI
|
||||||
|
implementations provide environment variables that allow us to
|
||||||
|
dynamically set the compiler to be used by mpicc, mpicxx, etc.
|
||||||
|
Denis pasted some code from this below – Spack's build environment
|
||||||
|
sets ``MPICC``, ``MPICXX``, etc. for mpich derivatives and
|
||||||
|
``OMPI_CC``, ``OMPI_CXX``, etc. for OpenMPI. This makes the MPI
|
||||||
|
compiler wrappers use the Spack compiler wrappers so that your
|
||||||
|
dependencies still get proper RPATHs even if you use the MPI
|
||||||
|
wrappers.
|
||||||
|
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
MPI on Cray machines
|
||||||
|
^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
|
||||||
|
The Cray programming environment notably uses ITS OWN compiler wrappers,
|
||||||
|
which function like MPI wrappers. On Cray systems, the ``CC``, ``cc``,
|
||||||
|
and ``ftn`` wrappers ARE the MPI compiler wrappers, and it's assumed that
|
||||||
|
you'll use them for all of your builds. So on Cray we don't bother with
|
||||||
|
``mpicc``, ``mpicxx``, etc, Spack MPI implementations set
|
||||||
|
``spec['mpi'].mpicc`` to point to Spack's wrappers, which wrap the Cray
|
||||||
|
wrappers, which wrap the regular compilers and include MPI flags. That
|
||||||
|
may seem complicated, but for packagers, that means the same code for
|
||||||
|
using MPI wrappers will work, even on even on a Cray:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
env['CC'] = spec['mpi'].mpicc
|
||||||
|
|
||||||
|
This is because on Cray, ``spec['mpi'].mpicc`` is just ``spack_cc``.
|
||||||
|
|
||||||
.. _sanity-checks:
|
.. _sanity-checks:
|
||||||
|
|
||||||
@ -2963,9 +3127,9 @@ File functions
|
|||||||
|
|
||||||
.. _package-lifecycle:
|
.. _package-lifecycle:
|
||||||
|
|
||||||
-----------------------
|
-----------------------------
|
||||||
Coding Style Guidelines
|
Style guidelines for packages
|
||||||
-----------------------
|
-----------------------------
|
||||||
|
|
||||||
The following guidelines are provided, in the interests of making
|
The following guidelines are provided, in the interests of making
|
||||||
Spack packages work in a consistent manner:
|
Spack packages work in a consistent manner:
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user