|
|
|
|
@@ -703,400 +703,6 @@ environments:
|
|
|
|
|
Administrators might find things easier to maintain without the
|
|
|
|
|
added "heavyweight" state of a view.
|
|
|
|
|
|
|
|
|
|
------------------------------
|
|
|
|
|
Developing Software with Spack
|
|
|
|
|
------------------------------
|
|
|
|
|
|
|
|
|
|
For any project, one needs to assemble an
|
|
|
|
|
environment of that application's dependencies. You might consider
|
|
|
|
|
loading a series of modules or creating a filesystem view. This
|
|
|
|
|
approach, while obvious, has some serious drawbacks:
|
|
|
|
|
|
|
|
|
|
1. There is no guarantee that an environment created this way will be
|
|
|
|
|
consistent. Your application could end up with dependency A
|
|
|
|
|
expecting one version of MPI, and dependency B expecting another.
|
|
|
|
|
The linker will not be happy...
|
|
|
|
|
|
|
|
|
|
2. Suppose you need to debug a package deep within your software DAG.
|
|
|
|
|
If you build that package with a manual environment, then it
|
|
|
|
|
becomes difficult to have Spack auto-build things that depend on
|
|
|
|
|
it. That could be a serious problem, depending on how deep the
|
|
|
|
|
package in question is in your dependency DAG.
|
|
|
|
|
|
|
|
|
|
3. At its core, Spack is a sophisticated concretization algorithm that
|
|
|
|
|
matches up packages with appropriate dependencies and creates a
|
|
|
|
|
*consistent* environment for the package it's building. Writing a
|
|
|
|
|
list of ``spack load`` commands for your dependencies is at least
|
|
|
|
|
as hard as writing the same list of ``depends_on()`` declarations
|
|
|
|
|
in a Spack package. But it makes no use of Spack concretization
|
|
|
|
|
and is more error-prone.
|
|
|
|
|
|
|
|
|
|
4. Spack provides an automated, systematic way not just to find a
|
|
|
|
|
packages's dependencies --- but also to build other packages on
|
|
|
|
|
top. Any Spack package can become a dependency for another Spack
|
|
|
|
|
package, offering a powerful vision of software re-use. If you
|
|
|
|
|
build your package A outside of Spack, then your ability to use it
|
|
|
|
|
as a building block for other packages in an automated way is
|
|
|
|
|
diminished: other packages depending on package A will not
|
|
|
|
|
be able to use Spack to fulfill that dependency.
|
|
|
|
|
|
|
|
|
|
5. If you are reading this manual, you probably love Spack. You're
|
|
|
|
|
probably going to write a Spack package for your software so
|
|
|
|
|
prospective users can install it with the least amount of pain.
|
|
|
|
|
Why should you go to additional work to find dependencies in your
|
|
|
|
|
development environment? Shouldn't Spack be able to help you build
|
|
|
|
|
your software based on the package you've already written?
|
|
|
|
|
|
|
|
|
|
In this section, we show how Spack can be used in the software
|
|
|
|
|
development process to greatest effect, and how development packages
|
|
|
|
|
can be seamlessly integrated into the Spack ecosystem. We will show
|
|
|
|
|
how this process works by example, assuming the software you are
|
|
|
|
|
creating is called ``mylib``.
|
|
|
|
|
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
Write the CMake Build
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
For now, the techniques in this section only work for CMake-based
|
|
|
|
|
projects, although they could be easily extended to other build
|
|
|
|
|
systems in the future. We will therefore assume you are using CMake
|
|
|
|
|
to build your project.
|
|
|
|
|
|
|
|
|
|
The ``CMakeLists.txt`` file should be written as normal. A few caveats:
|
|
|
|
|
|
|
|
|
|
1. Your project should produce binaries with RPATHs. This will ensure
|
|
|
|
|
that they work the same whether built manually or automatically by
|
|
|
|
|
Spack. For example:
|
|
|
|
|
|
|
|
|
|
.. code-block:: cmake
|
|
|
|
|
|
|
|
|
|
# enable @rpath in the install name for any shared library being built
|
|
|
|
|
# note: it is planned that a future version of CMake will enable this by default
|
|
|
|
|
set(CMAKE_MACOSX_RPATH 1)
|
|
|
|
|
|
|
|
|
|
# Always use full RPATH
|
|
|
|
|
# http://www.cmake.org/Wiki/CMake_RPATH_handling
|
|
|
|
|
# http://www.kitware.com/blog/home/post/510
|
|
|
|
|
|
|
|
|
|
# use, i.e. don't skip the full RPATH for the build tree
|
|
|
|
|
SET(CMAKE_SKIP_BUILD_RPATH FALSE)
|
|
|
|
|
|
|
|
|
|
# when building, don't use the install RPATH already
|
|
|
|
|
# (but later on when installing)
|
|
|
|
|
SET(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)
|
|
|
|
|
|
|
|
|
|
# add the automatically determined parts of the RPATH
|
|
|
|
|
# which point to directories outside the build tree to the install RPATH
|
|
|
|
|
SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
|
|
|
|
|
|
|
|
|
|
# the RPATH to be used when installing, but only if it's not a system directory
|
|
|
|
|
LIST(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES "${CMAKE_INSTALL_PREFIX}/lib" isSystemDir)
|
|
|
|
|
IF("${isSystemDir}" STREQUAL "-1")
|
|
|
|
|
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")
|
|
|
|
|
ENDIF("${isSystemDir}" STREQUAL "-1")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2. Spack provides a CMake variable called
|
|
|
|
|
``SPACK_TRANSITIVE_INCLUDE_PATH``, which contains the ``include/``
|
|
|
|
|
directory for all of your project's transitive dependencies. It
|
|
|
|
|
can be useful if your project ``#include``s files from package B,
|
|
|
|
|
which ``#include`` files from package C, but your project only
|
|
|
|
|
lists project B as a dependency. This works in traditional
|
|
|
|
|
single-tree build environments, in which B and C's include files
|
|
|
|
|
live in the same place. In order to make it work with Spack as
|
|
|
|
|
well, you must add the following to ``CMakeLists.txt``. It will
|
|
|
|
|
have no effect when building without Spack:
|
|
|
|
|
|
|
|
|
|
.. code-block:: cmake
|
|
|
|
|
|
|
|
|
|
# Include all the transitive dependencies determined by Spack.
|
|
|
|
|
# If we're not running with Spack, this does nothing...
|
|
|
|
|
include_directories($ENV{SPACK_TRANSITIVE_INCLUDE_PATH})
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
|
|
|
|
|
Note that this feature is controversial and could break with
|
|
|
|
|
future versions of GNU ld. The best practice is to make sure
|
|
|
|
|
anything you ``#include`` is listed as a dependency in your
|
|
|
|
|
CMakeLists.txt (and Spack package).
|
|
|
|
|
|
|
|
|
|
.. _write-the-spack-package:
|
|
|
|
|
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
Write the Spack Package
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
The Spack package also needs to be written, in tandem with setting up
|
|
|
|
|
the build (for example, CMake). The most important part of this task
|
|
|
|
|
is declaring dependencies. Here is an example of the Spack package
|
|
|
|
|
for the ``mylib`` package (ellipses for brevity):
|
|
|
|
|
|
|
|
|
|
.. code-block:: python
|
|
|
|
|
|
|
|
|
|
class Mylib(CMakePackage):
|
|
|
|
|
"""Misc. reusable utilities used by Myapp."""
|
|
|
|
|
|
|
|
|
|
homepage = "https://github.com/citibeth/mylib"
|
|
|
|
|
url = "https://github.com/citibeth/mylib/tarball/123"
|
|
|
|
|
|
|
|
|
|
version('0.1.2', '3a6acd70085e25f81b63a7e96c504ef9')
|
|
|
|
|
version('develop', git='https://github.com/citibeth/mylib.git',
|
|
|
|
|
branch='develop')
|
|
|
|
|
|
|
|
|
|
variant('everytrace', default=False,
|
|
|
|
|
description='Report errors through Everytrace')
|
|
|
|
|
...
|
|
|
|
|
|
|
|
|
|
extends('python')
|
|
|
|
|
|
|
|
|
|
depends_on('eigen')
|
|
|
|
|
depends_on('everytrace', when='+everytrace')
|
|
|
|
|
depends_on('proj', when='+proj')
|
|
|
|
|
...
|
|
|
|
|
depends_on('cmake', type='build')
|
|
|
|
|
depends_on('doxygen', type='build')
|
|
|
|
|
|
|
|
|
|
def cmake_args(self):
|
|
|
|
|
spec = self.spec
|
|
|
|
|
return [
|
|
|
|
|
'-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'),
|
|
|
|
|
'-DUSE_PROJ4=%s' % ('YES' if '+proj' in spec else 'NO'),
|
|
|
|
|
...
|
|
|
|
|
'-DUSE_UDUNITS2=%s' % ('YES' if '+udunits2' in spec else 'NO'),
|
|
|
|
|
'-DUSE_GTEST=%s' % ('YES' if '+googletest' in spec else 'NO')]
|
|
|
|
|
|
|
|
|
|
This is a standard Spack package that can be used to install
|
|
|
|
|
``mylib`` in a production environment. The list of dependencies in
|
|
|
|
|
the Spack package will generally be a repeat of the list of CMake
|
|
|
|
|
dependencies. This package also has some features that allow it to be
|
|
|
|
|
used for development:
|
|
|
|
|
|
|
|
|
|
1. It subclasses ``CMakePackage`` instead of ``Package``. This
|
|
|
|
|
eliminates the need to write an ``install()`` method, which is
|
|
|
|
|
defined in the superclass. Instead, one just needs to write the
|
|
|
|
|
``configure_args()`` method. That method should return the
|
|
|
|
|
arguments needed for the ``cmake`` command (beyond the standard
|
|
|
|
|
CMake arguments, which Spack will include already). These
|
|
|
|
|
arguments are typically used to turn features on/off in the build.
|
|
|
|
|
|
|
|
|
|
2. It specifies a non-checksummed version ``develop``. Running
|
|
|
|
|
``spack install mylib@develop`` the ``@develop`` version will
|
|
|
|
|
install the latest version off the develop branch. This method of
|
|
|
|
|
download is useful for the developer of a project while it is in
|
|
|
|
|
active development; however, it should only be used by developers
|
|
|
|
|
who control and trust the repository in question!
|
|
|
|
|
|
|
|
|
|
3. The ``url``, ``url_for_version()`` and ``homepage`` attributes are
|
|
|
|
|
not used in development. Don't worry if you don't have any, or if
|
|
|
|
|
they are behind a firewall.
|
|
|
|
|
|
|
|
|
|
^^^^^^^^^^^^^^^^
|
|
|
|
|
Build with Spack
|
|
|
|
|
^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
Now that you have a Spack package, you can use Spack to find its
|
|
|
|
|
dependencies automatically. For example:
|
|
|
|
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
|
|
|
|
|
|
$ cd mylib
|
|
|
|
|
$ spack setup mylib@local
|
|
|
|
|
|
|
|
|
|
The result will be a file ``spconfig.py`` in the top-level
|
|
|
|
|
``mylib/`` directory. It is a short script that calls CMake with the
|
|
|
|
|
dependencies and options determined by Spack --- similar to what
|
|
|
|
|
happens in ``spack install``, but now written out in script form.
|
|
|
|
|
From a developer's point of view, you can think of ``spconfig.py`` as
|
|
|
|
|
a stand-in for the ``cmake`` command.
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
|
|
|
|
|
You can invent any "version" you like for the ``spack setup``
|
|
|
|
|
command.
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
|
|
|
|
|
Although ``spack setup`` does not build your package, it does
|
|
|
|
|
create and install a module file, and mark in the database that
|
|
|
|
|
your package has been installed. This can lead to errors, of
|
|
|
|
|
course, if you don't subsequently install your package.
|
|
|
|
|
Also... you will need to ``spack uninstall`` before you run
|
|
|
|
|
``spack setup`` again.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
You can now build your project as usual with CMake:
|
|
|
|
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
|
|
|
|
|
|
$ mkdir build; cd build
|
|
|
|
|
$ ../spconfig.py .. # Instead of cmake ..
|
|
|
|
|
$ make
|
|
|
|
|
$ make install
|
|
|
|
|
|
|
|
|
|
Once your ``make install`` command is complete, your package will be
|
|
|
|
|
installed, just as if you'd run ``spack install``. Except you can now
|
|
|
|
|
edit, re-build and re-install as often as needed, without checking
|
|
|
|
|
into Git or downloading tarballs.
|
|
|
|
|
|
|
|
|
|
.. note::
|
|
|
|
|
|
|
|
|
|
The build you get this way will be *almost* the same as the build
|
|
|
|
|
from ``spack install``. The only difference is, you will not be
|
|
|
|
|
using Spack's compiler wrappers. This difference has not caused
|
|
|
|
|
problems in our experience, as long as your project sets
|
|
|
|
|
RPATHs as shown above. You DO use RPATHs, right?
|
|
|
|
|
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
Build Other Software
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
Now that you've built ``mylib`` with Spack, you might want to build
|
|
|
|
|
another package that depends on it --- for example, ``myapp``. This
|
|
|
|
|
is accomplished easily enough:
|
|
|
|
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
|
|
|
|
|
|
$ spack install myapp ^mylib@local
|
|
|
|
|
|
|
|
|
|
Note that auto-built software has now been installed *on top of*
|
|
|
|
|
manually-built software, without breaking Spack's "web." This
|
|
|
|
|
property is useful if you need to debug a package deep in the
|
|
|
|
|
dependency hierarchy of your application. It is a *big* advantage of
|
|
|
|
|
using ``spack setup`` to build your package's environment.
|
|
|
|
|
|
|
|
|
|
If you feel your software is stable, you might wish to install it with
|
|
|
|
|
``spack install`` and skip the source directory. You can just use,
|
|
|
|
|
for example:
|
|
|
|
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
|
|
|
|
|
|
$ spack install mylib@develop
|
|
|
|
|
|
|
|
|
|
.. _release-your-software:
|
|
|
|
|
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
Release Your Software
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
You are now ready to release your software as a tarball with a
|
|
|
|
|
numbered version, and a Spack package that can build it. If you're
|
|
|
|
|
hosted on GitHub, this process will be a bit easier.
|
|
|
|
|
|
|
|
|
|
#. Put tag(s) on the version(s) in your GitHub repo you want to be
|
|
|
|
|
release versions. For example, a tag ``v0.1.0`` for version 0.1.0.
|
|
|
|
|
|
|
|
|
|
#. Set the ``url`` in your ``package.py`` to download a tarball for
|
|
|
|
|
the appropriate version. GitHub will give you a tarball for any
|
|
|
|
|
commit in the repo, if you tickle it the right way. For example:
|
|
|
|
|
|
|
|
|
|
.. code-block:: python
|
|
|
|
|
|
|
|
|
|
url = 'https://github.com/citibeth/mylib/tarball/v0.1.2'
|
|
|
|
|
|
|
|
|
|
#. Use Spack to determine your version's hash, and cut'n'paste it into
|
|
|
|
|
your ``package.py``:
|
|
|
|
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
|
|
|
|
|
|
$ spack checksum mylib 0.1.2
|
|
|
|
|
==> Found 1 versions of mylib
|
|
|
|
|
0.1.2 https://github.com/citibeth/mylib/tarball/v0.1.2
|
|
|
|
|
|
|
|
|
|
How many would you like to checksum? (default is 5, q to abort)
|
|
|
|
|
==> Downloading...
|
|
|
|
|
==> Trying to fetch from https://github.com/citibeth/mylib/tarball/v0.1.2
|
|
|
|
|
######################################################################## 100.0%
|
|
|
|
|
==> Checksummed new versions of mylib:
|
|
|
|
|
version('0.1.2', '3a6acd70085e25f81b63a7e96c504ef9')
|
|
|
|
|
|
|
|
|
|
#. You should now be able to install released version 0.1.2 of your package with:
|
|
|
|
|
|
|
|
|
|
.. code-block:: console
|
|
|
|
|
|
|
|
|
|
$ spack install mylib@0.1.2
|
|
|
|
|
|
|
|
|
|
#. There is no need to remove the `develop` version from your package.
|
|
|
|
|
Spack concretization will always prefer numbered version to
|
|
|
|
|
non-numeric versions. Users will only get it if they ask for it.
|
|
|
|
|
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
Distribute Your Software
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
Once you've released your software, other people will want to build
|
|
|
|
|
it; and you will need to tell them how. In the past, that has meant a
|
|
|
|
|
few paragraphs of prose explaining which dependencies to install. But
|
|
|
|
|
now you use Spack, and those instructions are written in executable
|
|
|
|
|
Python code. But your software has many dependencies, and you know
|
|
|
|
|
Spack is the best way to install it:
|
|
|
|
|
|
|
|
|
|
#. First, you will want to fork Spack's ``develop`` branch. Your aim
|
|
|
|
|
is to provide a stable version of Spack that you KNOW will install
|
|
|
|
|
your software. If you make changes to Spack in the process, you
|
|
|
|
|
will want to submit pull requests to Spack core.
|
|
|
|
|
|
|
|
|
|
#. Add your software's ``package.py`` to that fork. You should submit
|
|
|
|
|
a pull request for this as well, unless you don't want the public
|
|
|
|
|
to know about your software.
|
|
|
|
|
|
|
|
|
|
#. Prepare instructions that read approximately as follows:
|
|
|
|
|
|
|
|
|
|
#. Download Spack from your forked repo.
|
|
|
|
|
|
|
|
|
|
#. Install Spack; see :ref:`getting_started`.
|
|
|
|
|
|
|
|
|
|
#. Set up an appropriate ``packages.yaml`` file. You should tell
|
|
|
|
|
your users to include in this file whatever versions/variants
|
|
|
|
|
are needed to make your software work correctly (assuming those
|
|
|
|
|
are not already in your ``packages.yaml``).
|
|
|
|
|
|
|
|
|
|
#. Run ``spack install mylib``.
|
|
|
|
|
|
|
|
|
|
#. Run this script to generate the ``module load`` commands or
|
|
|
|
|
filesystem view needed to use this software.
|
|
|
|
|
|
|
|
|
|
#. Be aware that your users might encounter unexpected bootstrapping
|
|
|
|
|
issues on their machines, especially if they are running on older
|
|
|
|
|
systems. The :ref:`getting_started` section should cover this, but
|
|
|
|
|
there could always be issues.
|
|
|
|
|
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
Other Build Systems
|
|
|
|
|
^^^^^^^^^^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
``spack setup`` currently only supports CMake-based builds, in
|
|
|
|
|
packages that subclass ``CMakePackage``. The intent is that this
|
|
|
|
|
mechanism should support a wider range of build systems; for example,
|
|
|
|
|
GNU Autotools. Someone well-versed in Autotools is needed to develop
|
|
|
|
|
this patch and test it out.
|
|
|
|
|
|
|
|
|
|
Python Distutils is another popular build system that should get
|
|
|
|
|
``spack setup`` support. For non-compiled languages like Python,
|
|
|
|
|
``spack diy`` may be used. Even better is to put the source directory
|
|
|
|
|
directly in the user's ``PYTHONPATH``. Then, edits in source files
|
|
|
|
|
are immediately available to run without any install process at all!
|
|
|
|
|
|
|
|
|
|
^^^^^^^^^^
|
|
|
|
|
Conclusion
|
|
|
|
|
^^^^^^^^^^
|
|
|
|
|
|
|
|
|
|
The ``spack setup`` development workflow provides better automation,
|
|
|
|
|
flexibility and safety than workflows relying on environment modules
|
|
|
|
|
or filesystem views. However, it has some drawbacks:
|
|
|
|
|
|
|
|
|
|
#. It currently works only with projects that use the CMake build
|
|
|
|
|
system. Support for other build systems is not hard to build, but
|
|
|
|
|
will require a small amount of effort for each build system to be
|
|
|
|
|
supported. It might not work well with some IDEs.
|
|
|
|
|
|
|
|
|
|
#. It only works with packages that sub-class ``StagedPackage``.
|
|
|
|
|
Currently, most Spack packages do not. Converting them is not
|
|
|
|
|
hard; but must be done on a package-by-package basis.
|
|
|
|
|
|
|
|
|
|
#. It requires that users are comfortable with Spack, as they
|
|
|
|
|
integrate Spack explicitly in their workflow. Not all users are
|
|
|
|
|
willing to do this.
|
|
|
|
|
|
|
|
|
|
-------------------------------------
|
|
|
|
|
Using Spack to Replace Homebrew/Conda
|
|
|
|
|
-------------------------------------
|
|
|
|
|
|
Adam J. Stewart