zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

SPACK-59: Documentation updates, bugfix in fetching.

Browse Source
This commit is contained in:
Todd Gamblin 2015-02-24 02:33:29 -08:00
parent 049808a34f
commit daa38d2ff4
14 changed files with 587 additions and 226 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
lib/spack/docs/.gitignore vendored
View File

@ -1,3 +1,4 @@
package_list.rst package_list.rst
command_index.rst
spack*.rst spack*.rst
_build _build

16
lib/spack/docs/Makefile
View File

@ -27,6 +27,18 @@ all: html
package_list: package_list:
spack package-list > package_list.rst spack package-list > package_list.rst
#
# Generate a command index
#
command_index:
cp command_index.in command_index.rst
echo >> command_index.rst
grep -ho '.. _spack-.*:' *rst \
| perl -pe 's/.. _([^:]*):/ * :ref:`\1`/' \
| sort >> command_index.rst
custom_targets: package_list command_index
# #
# This creates a git repository and commits generated html docs. # This creates a git repository and commits generated html docs.
# It them pushes the new branch into THIS repository as gh-pages. # It them pushes the new branch into THIS repository as gh-pages.
@ -77,10 +89,10 @@ help:
@echo " doctest to run all doctests embedded in the documentation (if enabled)" @echo " doctest to run all doctests embedded in the documentation (if enabled)"
clean: clean:
-rm -f package_list.rst -rm -f package_list.rst command_index.rst
-rm -rf $(BUILDDIR)/* $(APIDOC_FILES) -rm -rf $(BUILDDIR)/* $(APIDOC_FILES)
html: apidoc package_list html: apidoc custom_targets
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo @echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html." @echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

37
lib/spack/docs/basic_usage.rst
View File

@ -16,6 +16,8 @@ software. Before that, you need to know what's available. You can
see avaialble package names either using the :ref:`package-list`, or see avaialble package names either using the :ref:`package-list`, or
using the commands below. using the commands below.
.. _spack-list:
``spack list`` ``spack list``
~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~
@ -31,6 +33,7 @@ do wildcard searches using ``*``:
.. command-output:: spack list *util* .. command-output:: spack list *util*
.. _spack-info:
``spack info`` ``spack info``
~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~
@ -47,6 +50,8 @@ attacks. :ref:`Dependencies <sec-specs>` and :ref:`virtual
dependencies <sec-virtual-dependencies>`, are described in more detail dependencies <sec-virtual-dependencies>`, are described in more detail
later. later.
.. _spack-versions:
``spack versions`` ``spack versions``
~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~
@ -69,6 +74,8 @@ Installing and uninstalling
Now that you know how to list avaiable packages and versions, you're Now that you know how to list avaiable packages and versions, you're
ready to start installing things. ready to start installing things.
.. _spack-install:
``spack install`` ``spack install``
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
@ -138,6 +145,7 @@ configuration a **spec**. In the command lines above, both
``mpileaks`` and ``mpileaks@3.0.4`` are specs. Specs are described in ``mpileaks`` and ``mpileaks@3.0.4`` are specs. Specs are described in
detail in :ref:`sec-specs`. detail in :ref:`sec-specs`.
.. _spack-uninstall:
``spack uninstall`` ``spack uninstall``
~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~
@ -170,6 +178,7 @@ Seeing installed packages
We know that ``spack list`` shows you the names of available packages, We know that ``spack list`` shows you the names of available packages,
but how do you figure out which are installed? but how do you figure out which are installed?
.. _spack-find:
``spack find`` ``spack find``
~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~
@ -303,7 +312,7 @@ of libelf would look like this:
The full spec syntax is discussed in detail in :ref:`sec-specs`. The full spec syntax is discussed in detail in :ref:`sec-specs`.
Compiler Configuration Compiler configuration
----------------------------------- -----------------------------------
Spack has the ability to build packages with multiple compilers and Spack has the ability to build packages with multiple compilers and
@ -311,6 +320,8 @@ compiler versions. Spack searches for compilers on your machine
automatically the first time it is run. It does this by inspecting automatically the first time it is run. It does this by inspecting
your path. your path.
.. _spack-compilers:
``spack compilers`` ``spack compilers``
~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
@ -337,6 +348,8 @@ compilers`` or ``spack compiler list``::
Any of these compilers can be used to build Spack packages. More on Any of these compilers can be used to build Spack packages. More on
how this is done is in :ref:`sec-specs`. how this is done is in :ref:`sec-specs`.
.. _spack-compiler-add:
``spack compiler add`` ``spack compiler add``
~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
@ -361,6 +374,7 @@ installed, but you know that new compilers have been added to your
This loads the environment module for gcc-4.9.0 to get it into the This loads the environment module for gcc-4.9.0 to get it into the
``PATH``, and then it adds the compiler to Spack. ``PATH``, and then it adds the compiler to Spack.
.. _spack-compiler-info:
``spack compiler info`` ``spack compiler info``
~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
@ -382,7 +396,7 @@ matching Intel compiler was displayed.
Manual compiler configuration Manual compiler configuration
~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If autodetection fails, you can manually conigure a compiler by If autodetection fails, you can manually conigure a compiler by
editing your ``~/.spackconfig`` file. You can do this by running editing your ``~/.spackconfig`` file. You can do this by running
@ -413,7 +427,7 @@ list displayed by ``spack compilers``.
.. _sec-specs: .. _sec-specs:
Specs & Dependencies Specs & dependencies
------------------------- -------------------------
We know that ``spack install``, ``spack uninstall``, and other We know that ``spack install``, ``spack uninstall``, and other
@ -720,6 +734,8 @@ any MPI implementation will do. If another package depends on
error. Likewise, if you try to plug in some package that doesn't error. Likewise, if you try to plug in some package that doesn't
provide MPI, Spack will raise an error. provide MPI, Spack will raise an error.
.. _spack-providers:
``spack providers`` ``spack providers``
~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -739,7 +755,7 @@ versions are now filtered out.
.. _shell-support: .. _shell-support:
Environment Modules Environment modules
------------------------------- -------------------------------
.. note:: .. note::
@ -787,6 +803,7 @@ The directories are automatically added to your ``MODULEPATH`` and
``DK_NODE`` environment variables when you enable Spack's `shell ``DK_NODE`` environment variables when you enable Spack's `shell
support <shell-support_>`_. support <shell-support_>`_.
Using Modules & Dotkits Using Modules & Dotkits
~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -934,6 +951,8 @@ if newer, fancier module support is added to Spack at some later date,
you may want to regenerate all the modules to take advantage of these you may want to regenerate all the modules to take advantage of these
new features. new features.
.. _spack-module:
``spack module refresh`` ``spack module refresh``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -950,7 +969,7 @@ regenerate all module and dotkit files from scratch:
.. _extensions: .. _extensions:
Extensions & Python Support Extensions & Python support
------------------------------------ ------------------------------------
Spack's installation model assumes that each package will live in its Spack's installation model assumes that each package will live in its
@ -971,6 +990,8 @@ an *extension*. Suppose you have Python installed like so:
-- chaos_5_x86_64_ib / gcc@4.4.7 -------------------------------- -- chaos_5_x86_64_ib / gcc@4.4.7 --------------------------------
python@2.7.8 python@2.7.8
.. _spack-extensions:
``spack extensions`` ``spack extensions``
~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~
@ -1004,6 +1025,7 @@ they are packages like any ohter. They are installed into their own
prefixes, and you can see this with ``spack find -p``: prefixes, and you can see this with ``spack find -p``:
.. code-block:: sh .. code-block:: sh
$ spack find -p py-numpy $ spack find -p py-numpy
==> 1 installed packages. ==> 1 installed packages.
-- chaos_5_x86_64_ib / gcc@4.4.7 -------------------------------- -- chaos_5_x86_64_ib / gcc@4.4.7 --------------------------------
@ -1060,6 +1082,8 @@ for this case. Instead of requiring users to load particular
environment modules, you can *activate* the package within the Python environment modules, you can *activate* the package within the Python
installation: installation:
.. _spack-activate:
``spack activate`` ``spack activate``
^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^
@ -1133,6 +1157,7 @@ dependencies, you can use ``spack activate -f``:
$ spack activate -f py-numpy $ spack activate -f py-numpy
==> Activated extension py-numpy@1.9.1%gcc@4.4.7=chaos_5_x86_64_ib-66733244 for python@2.7.8%gcc@4.4.7. ==> Activated extension py-numpy@1.9.1%gcc@4.4.7=chaos_5_x86_64_ib-66733244 for python@2.7.8%gcc@4.4.7.
.. _spack-deactivate:
``spack deactivate`` ``spack deactivate``
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
@ -1159,6 +1184,8 @@ several variants:
Getting Help Getting Help
----------------------- -----------------------
.. _spack-help:
``spack help`` ``spack help``
~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~

10
lib/spack/docs/command_index.in Normal file
View File

@ -0,0 +1,10 @@
.. _command_index:
Command index
=================
This is an alphabetical list of commands with links to the places they
appear in the documentation.
.. hlist::
:columns: 3

2
lib/spack/docs/features.rst
View File

@ -1,4 +1,4 @@
Feature Overview Feature overview
================== ==================
This is a high-level overview of features that make Spack different This is a high-level overview of features that make Spack different

2
lib/spack/docs/index.rst
View File

@ -46,8 +46,10 @@ Table of Contents
getting_started getting_started
basic_usage basic_usage
packaging_guide packaging_guide
mirrors
site_configuration site_configuration
developer_guide developer_guide
command_index
package_list package_list
API Docs <spack> API Docs <spack>

217
lib/spack/docs/mirrors.rst Normal file
View File

@ -0,0 +1,217 @@
.. _mirrors:
Mirrors
============================
Some sites may not have access to the internet for fetching packages.
These sites will need a local repository of tarballs from which they
can get their files. Spack has support for this with *mirrors*. A
mirror is a URL that points to a directory, either on the local
filesystem or on some server, containing tarballs for all of Spack's
packages.
Here's an example of a mirror's directory structure::
mirror/
cmake/
cmake-2.8.10.2.tar.gz
dyninst/
dyninst-8.1.1.tgz
dyninst-8.1.2.tgz
libdwarf/
libdwarf-20130126.tar.gz
libdwarf-20130207.tar.gz
libdwarf-20130729.tar.gz
libelf/
libelf-0.8.12.tar.gz
libelf-0.8.13.tar.gz
libunwind/
libunwind-1.1.tar.gz
mpich/
mpich-3.0.4.tar.gz
mvapich2/
mvapich2-1.9.tgz
The structure is very simple. There is a top-level directory. The
second level directories are named after packages, and the third level
contains tarballs for each package, named after each package.
.. note::
Archives are **not** named exactly they were in the package's fetch
URL. They have the form ``<name>-<version>.<extension>``, where
``<name>`` is Spack's name for the package, ``<version>`` is the
version of the tarball, and ``<extension>`` is whatever format the
package's fetch URL contains.
In order to make mirror creation reasonably fast, we copy the
tarball in its original format to the mirror directory, but we do
not standardize on a particular compression algorithm, because this
would potentially require expanding and recompressing each archive.
.. _spack-mirror:
``spack mirror``
----------------------------
Mirrors are managed with the ``spack mirror`` command. The help for
``spack mirror`` looks like this::
$ spack mirror -h
usage: spack mirror [-h] SUBCOMMAND ...
positional arguments:
SUBCOMMAND
create Create a directory to be used as a spack mirror, and fill
it with package archives.
add Add a mirror to Spack.
remove Remove a mirror by name.
list Print out available mirrors to the console.
optional arguments:
-h, --help show this help message and exit
The ``create`` command actually builds a mirror by fetching all of its
packages from the internet and checksumming them.
The other three commands are for managing mirror configuration. They
control the URL(s) from which Spack downloads its packages.
.. _spack-mirror-create:
``spack mirror create``
----------------------------
You can create a mirror using the ``spack mirror create`` command, assuming
you're on a machine where you can access the internet.
The command will iterate through all of Spack's packages and download
the safe ones into a directory structure like the one above. Here is
what it looks like:
.. code-block:: bash
$ spack mirror create libelf libdwarf
==> Created new mirror in spack-mirror-2014-06-24
==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.13.tar.gz
########################################################## 81.6%
==> Checksum passed for libelf@0.8.13
==> Added libelf@0.8.13
==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.12.tar.gz
###################################################################### 98.6%
==> Checksum passed for libelf@0.8.12
==> Added libelf@0.8.12
==> Trying to fetch from http://www.prevanders.net/libdwarf-20130207.tar.gz
###################################################################### 97.3%
==> Checksum passed for libdwarf@20130207
==> Added libdwarf@20130207
==> Trying to fetch from http://www.prevanders.net/libdwarf-20130126.tar.gz
######################################################## 78.9%
==> Checksum passed for libdwarf@20130126
==> Added libdwarf@20130126
==> Trying to fetch from http://www.prevanders.net/libdwarf-20130729.tar.gz
############################################################# 84.7%
==> Added libdwarf@20130729
==> Added spack-mirror-2014-06-24/libdwarf/libdwarf-20130729.tar.gz to mirror
==> Added python@2.7.8.
==> Successfully updated mirror in spack-mirror-2015-02-24.
Archive stats:
0 already present
5 added
0 failed to fetch.
Once this is done, you can tar up the ``spack-mirror-2014-06-24`` directory and
copy it over to the machine you want it hosted on.
Custom package sets
~~~~~~~~~~~~~~~~~~~~~~~
Normally, ``spack mirror create`` downloads all the archives it has
checksums for. If you want to only create a mirror for a subset of
packages, you can do that by supplying a list of package specs on the
command line after ``spack mirror create``. For example, this
command::
$ spack mirror create libelf@0.8.12: boost@1.44:
Will create a mirror for libelf versions greater than or equal to
0.8.12 and boost versions greater than or equal to 1.44.
Mirror files
~~~~~~~~~~~~~~~~~~~~~~~
If you have a *very* large number of packages you want to mirror, you
can supply a file with specs in it, one per line::
$ cat specs.txt
libdwarf
libelf@0.8.12:
boost@1.44:
boost@1.39.0
...
$ spack mirror create -f specs.txt
...
This is useful if there is a specific suite of software managed by
your site.
.. _spack-mirror-add:
``spack mirror add``
----------------------------
Once you have a mirrror, you need to let spack know about it. This is
relatively simple. First, figure out the URL for the mirror. If it's
a file, you can use a file URL like this one::
file:///Users/gamblin2/spack-mirror-2014-06-24
That points to the directory on the local filesystem. If it were on a
web server, you could use a URL like this one:
https://example.com/some/web-hosted/directory/spack-mirror-2014-06-24
Spack will use the URL as the root for all of the packages it fetches.
You can tell your Spack installation to use that mirror like this:
.. code-block:: bash
$ spack mirror add local_filesystem file:///Users/gamblin2/spack-mirror-2014-06-24
Each mirror has a name so that you can refer to it again later.
.. _spack-mirror-list:
``spack mirror list``
----------------------------
If you want to see all the mirrors Spack knows about you can run ``spack mirror list``::
$ spack mirror list
local_filesystem file:///Users/gamblin2/spack-mirror-2014-06-24
.. _spack-mirror-remove:
``spack mirror remove``
----------------------------
And, if you want to remove a mirror, just remove it by name::
$ spack mirror remove local_filesystem
$ spack mirror list
==> No mirrors configured.
Mirror precedence
----------------------------
Adding a mirror really just adds a section in ``~/.spackconfig``::
[mirror "local_filesystem"]
url = file:///Users/gamblin2/spack-mirror-2014-06-24
[mirror "remote_server"]
url = https://example.com/some/web-hosted/directory/spack-mirror-2014-06-24
If you want to change the order in which mirrors are searched for
packages, you can edit this file and reorder the sections. Spack will
search the topmost mirror first and the bottom-most mirror last.

316
lib/spack/docs/packaging_guide.rst
View File

@ -28,7 +28,7 @@ ubiquitous in the scientific software community. Second, it's a modern
language and has many powerful features to help make package writing language and has many powerful features to help make package writing
easy. easy.
Creating & Editing Packages Creating & editing packages
---------------------------------- ----------------------------------
.. _spack-create: .. _spack-create:
@ -254,7 +254,7 @@ This is useful when ``spack create`` cannot figure out the name and
version of your package from the archive URL. version of your package from the archive URL.
Naming & Directory Structure Naming & directory structure
-------------------------------------- --------------------------------------
.. note:: .. note::
@ -497,7 +497,7 @@ versions. See the documentation on `attribute_list_url`_ and
.. _vcs-fetch: .. _vcs-fetch:
Fetching from VCS Repositories Fetching from VCS repositories
-------------------------------------- --------------------------------------
For some packages, source code is provided in a Version Control System For some packages, source code is provided in a Version Control System
@ -774,6 +774,8 @@ to patch a package's source. For example, the ``py-pyside`` package
contains some custom code for tweaking the way the PySide build contains some custom code for tweaking the way the PySide build
handles ``RPATH``: handles ``RPATH``:
.. _pyside-patch:
.. code-block:: python .. code-block:: python
:linenos: :linenos:
@ -810,14 +812,59 @@ You could put this logic in ``install()``, but putting it in a patch
function gives you some benefits. First, spack ensures that the function gives you some benefits. First, spack ensures that the
``patch()`` function is run once per code checkout. That means that ``patch()`` function is run once per code checkout. That means that
if you run install, hit ctrl-C, and run install again, the code in the if you run install, hit ctrl-C, and run install again, the code in the
patch function is only run once. Also, you can tell Spack to run only the patching part of the build using the .. patch function is only run once. Also, you can tell Spack to run only
the patching part of the build using the :ref:`spack-patch` command.
Handling RPATHs
Finding Package Downloads
---------------------------- ----------------------------
We've already seen the ``homepage`` and ``url`` package attributes: Spack installs each package in a way that ensures that all of its
dependencies are found when it runs. It does this using `RPATHs
<http://en.wikipedia.org/wiki/Rpath>`_. An RPATH is a search
path, stored in a binary (an executable or library), that tells the
dynamic loader where to find its dependencies at runtime. You may be
familiar with ```LD_LIBRARY_PATH``
<http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`_
on Linux or ```DYLD_LIBRARY_PATH``
<https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/dyld.1.html>`
on Mac OS X. RPATH is similar to these paths, in that it tells
the loader where to find libraries. Unlike them, it is embedded in
the binary and not set in each user's environment.
RPATHs in Spack are handled in one of three ways:
1. For most packages, RPATHs are handled automatically using Spack's
:ref:`compiler wrappers <compiler-wrappers>`. These wrappers are
set in standard variables like ``CC``, ``CXX``, and ``FC``, so
most build systems (autotools and many gmake systems) pick them
up and use them.
2. CMake also respects Spack's compiler wrappers, but many CMake
builds have logic to overwrite RPATHs when binaries are
installed. Spack provides the ``std_cmake_args`` variable, which
includes parameters necessary for CMake build use the right
installation RPATH. It can be used like this when ``cmake`` is
invoked:
.. code-block:: python
class MyPackage(Package):
...
def install(self, spec, prefix):
cmake('..', *std_cmake_args)
make()
make('install')
3. If you need to modify the build to add your own RPATHs, you can
use the ``self.rpath`` property of your package, which will
return a list of all the RPATHs that Spack will use when it
links. You can see this how this is used in the :ref:`PySide
example <pyside-patch>` above.
Finding new versions
----------------------------
You've already seen the ``homepage`` and ``url`` package attributes:
.. code-block:: python .. code-block:: python
:linenos: :linenos:
@ -853,7 +900,7 @@ url is:
url = "http://www.mr511.de/software/libelf-0.8.13.tar.gz" url = "http://www.mr511.de/software/libelf-0.8.13.tar.gz"
Spack spiders ``http://www.mr511.de/software/`` to find similar Here, Spack spiders ``http://www.mr511.de/software/`` to find similar
tarball links and ultimately to make a list of available versions of tarball links and ultimately to make a list of available versions of
``libelf``. ``libelf``.
@ -907,7 +954,7 @@ when spidering the page.
.. _attribute_parallel: .. _attribute_parallel:
Parallel Builds Parallel builds
------------------ ------------------
By default, Spack will invoke ``make()`` with a ``-j <njobs>`` By default, Spack will invoke ``make()`` with a ``-j <njobs>``
@ -1036,6 +1083,203 @@ command line to find installed packages or to install packages with
particular constraints, and package authors can use specs to describe particular constraints, and package authors can use specs to describe
relationships between packages. relationships between packages.
.. _setup-dependent-environment:
``setup_dependent_environment()``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Spack provides a mechanism for dependencies to provide variables that
can be used in their dependents' build. Any package can declare a
``setup_dependent_environment()`` function, and this function will be
called before the ``install()`` method of any dependent packages.
This allows dependencies to set up environment variables and other
properties to be used by dependents.
The funciton declaration should look like this:
.. code-block:: python
class Qt(Package):
...
def setup_dependent_environment(self, module, spec, dep_spec):
"""Dependencies of Qt find it using the QTDIR environment variable."""
os.environ['QTDIR'] = self.prefix
Here, the Qt package sets the ``QTDIR`` environment variable so that
packages that depend on a particular Qt installation will find it.
The arguments to this function are:
* **module**: the module of the dependent package, where global
properties can be assigned.
* **spec**: the spec of the *dependency package* (the one the function is called on).
* **dep_spec**: the spec of the dependent package (i.e. dep_spec depends on spec).
A goo example of using these is in the Python packge:
.. code-block:: python
def setup_dependent_environment(self, module, spec, dep_spec):
# Python extension builds can have a global python executable function
module.python = Executable(join_path(spec.prefix.bin, 'python'))
# Add variables for lib/pythonX.Y and lib/pythonX.Y/site-packages dirs.
module.python_lib_dir = os.path.join(dep_spec.prefix, self.python_lib_dir)
module.python_include_dir = os.path.join(dep_spec.prefix, self.python_include_dir)
module.site_packages_dir = os.path.join(dep_spec.prefix, self.site_packages_dir)
# Make the site packages directory if it does not exist already.
mkdirp(module.site_packages_dir)
# Set PYTHONPATH to include site-packages dir for the
# extension and any other python extensions it depends on.
python_paths = []
for d in dep_spec.traverse():
if d.package.extends(self.spec):
python_paths.append(os.path.join(d.prefix, self.site_packages_dir))
os.environ['PYTHONPATH'] = ':'.join(python_paths)
The first thing that happens here is that the ``python`` command is
inserted into module scope of the dependent. This allows most python
packages to have a very simple install method, like this:
.. code-block:: python
def install(self, spec, prefix):
python('setup.py', 'install', '--prefix=%s' % prefix)
Python's ``setup_dependent_environment`` method also sets up smoe
other variables, creates a directory, and sets up the ``PYTHONPATH``
so that dependent packages can find their dependencies at build time.
.. _packaging_extensions:
Extensions
-------------------------
Spack's support for package extensions is documented extensively in
:ref:`extensions`. This section documents how to make your own
extendable packages and extensions.
To support extensions, a package needs to set its ``extendable``
property to ``True``, e.g.:
.. code-block:: python
class Python(Package):
...
extendable = True
...
To make a package into an extension, simply add simply add an
``extends`` call in the package definition, and pass it the name of an
extendable package:
.. code-block:: python
class PyNumpy(Package):
...
extends('python')
...
Now, the ``py-numpy`` package can be used as an argument to ``spack
activate``. When it is activated, all the files in its prefix will be
symbolically linked into the prefix of the python package.
Sometimes, certain files in one package will conflict with those in
another, which means they cannot both be activated (symlinked) at the
same time. In this case, you can tell Spack to ignore those files
when it does the activation:
.. code-block:: python
class PyNose(Package):
...
extends('python', ignore=r'bin/nosetests.*$')
...
The code above will prevent ``$prefix/bin/nosetests`` from being
linked in at activation time.
.. note::
You can call *either* ``depends_on`` or ``extends`` on any one
package, but not both. For example you cannot both
``depends_on('python')`` and ``extends(python)`` in the same
package. ``extends`` implies ``depends_on``.
Activation & deactivation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Spack's ``Package`` class has default ``activate`` and ``deactivate``
implementations that handle symbolically linking extensions' prefixes
into the directory of the parent package. However, extendable
packages can override these methdos to add custom activate/deactivate
logic of their own. For example, the ``activate`` and ``deactivate``
methods in the Python class use the symbolic linking, but they also
handle details surrounding Python's ``.pth`` files, and other aspects
of Python packaging.
Spack's extensions mechanism is designed to be extensible, so that
other packages (like Ruby, R, Perl, etc.) can provide their own
custom extension management logic, as they may not handle modules the
same way that Python does.
Let's look at Python's activate function:
.. code-block:: python
def activate(self, ext_pkg, **kwargs):
kwargs.update(ignore=self.python_ignore(ext_pkg, kwargs))
super(Python, self).activate(ext_pkg, **kwargs)
exts = spack.install_layout.extension_map(self.spec)
exts[ext_pkg.name] = ext_pkg.spec
self.write_easy_install_pth(exts)
This function is called on the *extendee* (Python). It first calls
``activate`` in the superclass, which handles symlinking the
extension package's prefix into this package's prefix. It then does
some special handling of the ``easy-install.pth`` file, part of
Python's setuptools.
Deactivate behaves similarly to activate, but it unlinks files:
.. code-block:: python
def deactivate(self, ext_pkg, **kwargs):
kwargs.update(ignore=self.python_ignore(ext_pkg, kwargs))
super(Python, self).deactivate(ext_pkg, **kwargs)
exts = spack.install_layout.extension_map(self.spec)
if ext_pkg.name in exts: # Make deactivate idempotent.
del exts[ext_pkg.name]
self.write_easy_install_pth(exts)
Both of these methods call some custom functions in the Python
package. See the source for Spack's Python package for details.
Activation arguments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You may have noticed that the ``activate`` function defined above
takes keyword arguments. These are the keyword arguments from
``extends()``, and they are passed to both activate and deactivate.
This capability allows an extension to customize its own activation by
passing arguments to the extendee. Extendees can likewise implement
custom ``activate()`` and ``deactivate()`` functions to suit their
needs.
The only keyword argument supported by default is the ``ignore``
argument, which can take a regex, list of regexes, or a predicate to
determine which files *not* to symlink during activation.
.. _virtual-dependencies: .. _virtual-dependencies:
Virtual dependencies Virtual dependencies
@ -1257,6 +1501,7 @@ explicitly. Concretization policies are discussed in more detail in
:ref:`site-configuration`. Sites using Spack can customize them to :ref:`site-configuration`. Sites using Spack can customize them to
match the preferences of their own users. match the preferences of their own users.
.. _spack-spec:
``spack spec`` ``spack spec``
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
@ -1354,7 +1599,7 @@ information.
.. _install-environment: .. _install-environment:
The Install environment The install environment
-------------------------- --------------------------
In general, you should not have to do much differently in your install In general, you should not have to do much differently in your install
@ -1414,6 +1659,7 @@ easily:
``PATH`` Set to point to ``/bin`` directories of dpeendencies ``PATH`` Set to point to ``/bin`` directories of dpeendencies
``CMAKE_PREFIX_PATH`` Path to dependency prefixes for CMake ``CMAKE_PREFIX_PATH`` Path to dependency prefixes for CMake
``PKG_CONFIG_PATH`` Path to any pkgconfig directories for dependencies ``PKG_CONFIG_PATH`` Path to any pkgconfig directories for dependencies
``PYTHONPATH`` Path to site-packages dir of any python dependencies
======================= ============================= ======================= =============================
``PATH`` is set up to point to dependencies ``/bin`` directories so ``PATH`` is set up to point to dependencies ``/bin`` directories so
@ -1433,6 +1679,12 @@ dependencies using the GNU ``pkg-config`` tool. It is similar to
``CMAKE_PREFIX_PATH`` in that it allows a build to automatically ``CMAKE_PREFIX_PATH`` in that it allows a build to automatically
discover its dependencies. discover its dependencies.
If you want to see the environment that a package will build with, or
if you want to run commands in that environment to test them out, you
can use the :ref:```spack env`` <spack-env>` command, documented
below.
.. _compiler-wrappers:
Compiler interceptors Compiler interceptors
~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~
@ -1498,7 +1750,6 @@ process runs. Packages are free to change the environment or to
modify Spack internals, because each ``install()`` call has its own modify Spack internals, because each ``install()`` call has its own
dedicated process. dedicated process.
.. _prefix-objects: .. _prefix-objects:
Prefix objects Prefix objects
@ -1970,7 +2221,7 @@ File functions
.. _pacakge-lifecycle: .. _pacakge-lifecycle:
Package Workflow Commands Packaging workflow commands
--------------------------------- ---------------------------------
When you are building packages, you will likely not get things When you are building packages, you will likely not get things
@ -2036,6 +2287,8 @@ this step if they have been. If Spack discovers that patches didn't
apply cleanly on some previous run, then it will restage the entire apply cleanly on some previous run, then it will restage the entire
package before patching. package before patching.
.. _spack-restage:
``spack restage`` ``spack restage``
~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~
Restores the source code to pristine state, as it was before building. Restores the source code to pristine state, as it was before building.
@ -2048,6 +2301,7 @@ Does this in one of two ways:
2. If the source was checked out from a repository, this deletes the 2. If the source was checked out from a repository, this deletes the
build directory and checks it out again. build directory and checks it out again.
.. _spack-clean:
``spack clean`` ``spack clean``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -2057,6 +2311,8 @@ expanded/checked out source code *and* any downloaded archive. If
build process will start from scratch. build process will start from scratch.
.. _spack-purge:
``spack purge`` ``spack purge``
~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~
Cleans up all of Spack's temporary files. Use this to recover disk Cleans up all of Spack's temporary files. Use this to recover disk
@ -2102,7 +2358,7 @@ to get rid of the install prefix before you build again:
spack uninstall -f <spec> spack uninstall -f <spec>
Graphing Dependencies Graphing dependencies
-------------------------- --------------------------
.. _spack-graph: .. _spack-graph:
@ -2181,7 +2437,7 @@ example::
This graph can be provided as input to other graphing tools, such as This graph can be provided as input to other graphing tools, such as
those in `Graphviz <http://www.graphviz.org>`_. those in `Graphviz <http://www.graphviz.org>`_.
Interactive Shell Support Interactive shell support
-------------------------- --------------------------
Spack provides some limited shell support to make life easier for Spack provides some limited shell support to make life easier for
@ -2197,6 +2453,7 @@ For ``csh`` and ``tcsh`` run:
``spack cd`` will then be available. ``spack cd`` will then be available.
.. _spack-cd:
``spack cd`` ``spack cd``
~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~
@ -2227,6 +2484,35 @@ directory, install directory, package directory) and others change to
core spack locations. For example, ``spack cd -m`` will take you to core spack locations. For example, ``spack cd -m`` will take you to
the main python source directory of your spack install. the main python source directory of your spack install.
.. _spack-env:
``spack env``
~~~~~~~~~~~~~~~~~~~~~~
``spack env`` functions much like the standard unix ``env`` command,
but it takes a spec as an argument. You can use it to see the
environment variables that will be set when a particular build runs,
for example:
.. code-block:: sh
$ spack env mpileaks@1.1%intel
This will display the entire environment that will be set when the
``mpileaks@1.1%intel`` build runs.
To run commands in a package's build environment, you can simply provided them after the spec argument to ``spack env``:
.. code-block:: sh
$ spack cd mpileaks@1.1%intel
$ spack env mpileaks@1.1%intel ./configure
This will cd to the build directory and then run ``configure`` in the
package's build environment.
.. _spack-location:
``spack location`` ``spack location``
~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~

193
lib/spack/docs/site_configuration.rst
View File

@ -3,199 +3,6 @@
Site-specific configuration Site-specific configuration
=================================== ===================================
.. _mirrors:
Mirrors
----------------------------
Some sites may not have access to the internet for fetching packages.
These sites will need a local repository of tarballs from which they
can get their files. Spack has support for this with *mirrors*. A
mirror is a URL that points to a directory, either on the local
filesystem or on some server, containing tarballs for all of Spack's
packages.
Here's an example of a mirror's directory structure::
mirror/
cmake/
cmake-2.8.10.2.tar.gz
dyninst/
DyninstAPI-8.1.1.tgz
DyninstAPI-8.1.2.tgz
libdwarf/
libdwarf-20130126.tar.gz
libdwarf-20130207.tar.gz
libdwarf-20130729.tar.gz
libelf/
libelf-0.8.12.tar.gz
libelf-0.8.13.tar.gz
libunwind/
libunwind-1.1.tar.gz
mpich/
mpich-3.0.4.tar.gz
mvapich2/
mvapich2-1.9.tgz
The structure is very simple. There is a top-level directory. The
second level directories are named after packages, and the third level
contains tarballs for each package, named as they were in the
package's fetch URL.
``spack mirror``
~~~~~~~~~~~~~~~~~~~~~~~
Mirrors are managed with the ``spack mirror`` command. The help for
``spack mirror`` looks like this::
$ spack mirror -h
usage: spack mirror [-h] SUBCOMMAND ...
positional arguments:
SUBCOMMAND
create Create a directory to be used as a spack mirror, and fill
it with package archives.
add Add a mirror to Spack.
remove Remove a mirror by name.
list Print out available mirrors to the console.
optional arguments:
-h, --help show this help message and exit
The ``create`` command actually builds a mirror by fetching all of its
packages from the internet and checksumming them.
The other three commands are for managing mirror configuration. They
control the URL(s) from which Spack downloads its packages.
``spack mirror create``
~~~~~~~~~~~~~~~~~~~~~~~
You can create a mirror using the ``spack mirror create`` command, assuming
you're on a machine where you can access the internet.
The command will iterate through all of Spack's packages and download
the safe ones into a directory structure like the one above. Here is
what it looks like:
.. code-block:: bash
$ spack mirror create libelf libdwarf
==> Created new mirror in spack-mirror-2014-06-24
==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.13.tar.gz
########################################################## 81.6%
==> Checksum passed for libelf@0.8.13
==> Added spack-mirror-2014-06-24/libelf/libelf-0.8.13.tar.gz to mirror
==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.12.tar.gz
###################################################################### 98.6%
==> Checksum passed for libelf@0.8.12
==> Added spack-mirror-2014-06-24/libelf/libelf-0.8.12.tar.gz to mirror
==> Trying to fetch from http://www.prevanders.net/libdwarf-20130207.tar.gz
###################################################################### 97.3%
==> Checksum passed for libdwarf@20130207
==> Added spack-mirror-2014-06-24/libdwarf/libdwarf-20130207.tar.gz to mirror
==> Trying to fetch from http://www.prevanders.net/libdwarf-20130126.tar.gz
######################################################## 78.9%
==> Checksum passed for libdwarf@20130126
==> Added spack-mirror-2014-06-24/libdwarf/libdwarf-20130126.tar.gz to mirror
==> Trying to fetch from http://www.prevanders.net/libdwarf-20130729.tar.gz
############################################################# 84.7%
==> Checksum passed for libdwarf@20130729
==> Added spack-mirror-2014-06-24/libdwarf/libdwarf-20130729.tar.gz to mirror
Once this is done, you can tar up the ``spack-mirror-2014-06-24`` directory and
copy it over to the machine you want it hosted on.
Custom package sets
^^^^^^^^^^^^^^^^^^^^^^^^
Normally, ``spack mirror create`` downloads all the archives it has
checksums for. If you want to only create a mirror for a subset of
packages, you can do that by supplying a list of package specs on the
command line after ``spack mirror create``. For example, this
command::
$ spack mirror create libelf@0.8.12: boost@1.44:
Will create a mirror for libelf versions greater than or equal to
0.8.12 and boost versions greater than or equal to 1.44.
Mirror files
^^^^^^^^^^^^^^^^^^^^^^^^
If you have a *very* large number of packages you want to mirror, you
can supply a file with specs in it, one per line::
$ cat specs.txt
libdwarf
libelf@0.8.12:
boost@1.44:
boost@1.39.0
...
$ spack mirror create -f specs.txt
...
This is useful if there is a specific suite of software managed by
your site.
``spack mirror add``
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Once you have a mirrror, you need to let spack know about it. This is
relatively simple. First, figure out the URL for the mirror. If it's
a file, you can use a file URL like this one::
file:///Users/gamblin2/spack-mirror-2014-06-24
That points to the directory on the local filesystem. If it were on a
web server, you could use a URL like this one:
https://example.com/some/web-hosted/directory/spack-mirror-2014-06-24
Spack will use the URL as the root for all of the packages it fetches.
You can tell your Spack installation to use that mirror like this:
.. code-block:: bash
$ spack mirror add local_filesystem file:///Users/gamblin2/spack-mirror-2014-06-24
Each mirror has a name so that you can refer to it again later.
``spack mirror list``
~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to see all the mirrors Spack knows about you can run ``spack mirror list``::
$ spack mirror list
local_filesystem file:///Users/gamblin2/spack-mirror-2014-06-24
``spack mirror remove``
~~~~~~~~~~~~~~~~~~~~~~~~~~~
And, if you want to remove a mirror, just remove it by name::
$ spack mirror remove local_filesystem
$ spack mirror list
==> No mirrors configured.
Mirror precedence
~~~~~~~~~~~~~~~~~~~~~~~~~
Adding a mirror really just adds a section in ``~/.spackconfig``::
[mirror "local_filesystem"]
url = file:///Users/gamblin2/spack-mirror-2014-06-24
[mirror "remote_server"]
url = https://example.com/some/web-hosted/directory/spack-mirror-2014-06-24
If you want to change the order in which mirrors are searched for
packages, you can edit this file and reorder the sections. Spack will
search the topmost mirror first and the bottom-most mirror last.
.. _temp-space: .. _temp-space:
Temporary space Temporary space

2
lib/spack/spack/mirror.py
View File

@ -146,7 +146,7 @@ def create(path, specs, **kwargs):
stage = None stage = None
try: try:
# create a subdirectory for the current package@version # create a subdirectory for the current package@version
archive_path = join_path(path, mirror_archive_path(spec)) archive_path = os.path.abspath(join_path(path, mirror_archive_path(spec)))
subdir = os.path.dirname(archive_path) subdir = os.path.dirname(archive_path)
mkdirp(subdir) mkdirp(subdir)

5
lib/spack/spack/package.py
View File

@ -287,10 +287,9 @@ class SomePackage(Package):
.. code-block:: python .. code-block:: python
p.do_clean() # runs make clean p.do_clean() # removes the stage directory entirely
p.do_clean_work() # removes the build directory and p.do_restage() # removes the build directory and
# re-expands the archive. # re-expands the archive.
p.do_clean_dist() # removes the stage directory entirely
The convention used here is that a do_* function is intended to be called The convention used here is that a do_* function is intended to be called
internally by Spack commands (in spack.cmd). These aren't for package internally by Spack commands (in spack.cmd). These aren't for package

4
lib/spack/spack/test/git_fetch.py
View File

@ -61,7 +61,7 @@ def tearDown(self):
if self.repo.stage is not None: if self.repo.stage is not None:
self.repo.stage.destroy() self.repo.stage.destroy()
self.pkg.do_clean_dist() self.pkg.do_clean()
def assert_rev(self, rev): def assert_rev(self, rev):
@ -93,7 +93,7 @@ def try_fetch(self, rev, test_file, args):
untracked_file = 'foobarbaz' untracked_file = 'foobarbaz'
touch(untracked_file) touch(untracked_file)
self.assertTrue(os.path.isfile(untracked_file)) self.assertTrue(os.path.isfile(untracked_file))
self.pkg.do_clean_work() self.pkg.do_restage()
self.assertFalse(os.path.isfile(untracked_file)) self.assertFalse(os.path.isfile(untracked_file))
self.assertTrue(os.path.isdir(self.pkg.stage.source_path)) self.assertTrue(os.path.isdir(self.pkg.stage.source_path))

4
lib/spack/spack/test/hg_fetch.py
View File

@ -60,7 +60,7 @@ def tearDown(self):
if self.repo.stage is not None: if self.repo.stage is not None:
self.repo.stage.destroy() self.repo.stage.destroy()
self.pkg.do_clean_dist() self.pkg.do_clean()
def try_fetch(self, rev, test_file, args): def try_fetch(self, rev, test_file, args):
@ -87,7 +87,7 @@ def try_fetch(self, rev, test_file, args):
untracked = 'foobarbaz' untracked = 'foobarbaz'
touch(untracked) touch(untracked)
self.assertTrue(os.path.isfile(untracked)) self.assertTrue(os.path.isfile(untracked))
self.pkg.do_clean_work() self.pkg.do_restage()
self.assertFalse(os.path.isfile(untracked)) self.assertFalse(os.path.isfile(untracked))
self.assertTrue(os.path.isdir(self.pkg.stage.source_path)) self.assertTrue(os.path.isdir(self.pkg.stage.source_path))

4
lib/spack/spack/test/svn_fetch.py
View File

@ -60,7 +60,7 @@ def tearDown(self):
if self.repo.stage is not None: if self.repo.stage is not None:
self.repo.stage.destroy() self.repo.stage.destroy()
self.pkg.do_clean_dist() self.pkg.do_clean()
def assert_rev(self, rev): def assert_rev(self, rev):
@ -99,7 +99,7 @@ def try_fetch(self, rev, test_file, args):
untracked = 'foobarbaz' untracked = 'foobarbaz'
touch(untracked) touch(untracked)
self.assertTrue(os.path.isfile(untracked)) self.assertTrue(os.path.isfile(untracked))
self.pkg.do_clean_work() self.pkg.do_restage()
self.assertFalse(os.path.isfile(untracked)) self.assertFalse(os.path.isfile(untracked))
self.assertTrue(os.path.isdir(self.pkg.stage.source_path)) self.assertTrue(os.path.isdir(self.pkg.stage.source_path))