zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Start of basic documentation

Browse Source
This commit is contained in:
Todd Gamblin
2013-12-12 04:25:31 -08:00
parent c7539fe950
commit c9e40b725d
23 changed files with 945 additions and 130 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
lib/spack/docs/.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
spack*.rst
_build

12
lib/spack/docs/Makefile
View File

@@ -7,6 +7,9 @@ SPHINXBUILD = sphinx-build
PAPER =
BUILDDIR = _build
export PYTHONPATH = ../../spack
APIDOC_FILES = spack*.rst
# Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter
@@ -14,10 +17,13 @@ ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext apidoc
all: html
apidoc:
sphinx-apidoc -T -o . $(PYTHONPATH)/spack
help:
@echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files"
@@ -41,9 +47,9 @@ help:
@echo " doctest to run all doctests embedded in the documentation (if enabled)"
clean:
-rm -rf $(BUILDDIR)/*
-rm -rf $(BUILDDIR)/* $(APIDOC_FILES)
html:
html: apidoc
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

4
lib/spack/docs/_themes/sphinx_rtd_theme/footer.html vendored
View File

@@ -24,7 +24,7 @@
{%- if last_updated %}
{% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %}
{%- endif %}
</p>
&nbsp;&nbsp;
{% trans %}<a href="https://www.github.com/snide/sphinx_rtd_theme">Sphinx theme</a> provided by <a href="http://readthedocs.org">Read the Docs</a>{% endtrans %}
</p>
</footer>

233
lib/spack/docs/basic_usage.rst Normal file
View File

@@ -0,0 +1,233 @@
Basic usage
=====================
Nearly everything you do wtih spack will involve the ``spack``
command. Like many well-known tools (``git``, ``cvs``, ``svn``,
``yum``, ``port``, ``apt-get``, etc.), ``spack`` is generally called
with a *subcommand* indicating the action you want to perform.
Getting Help
-----------------------
``spack help``
~~~~~~~~~~~~~~~~~~~~~~
The first subcommand you should know is ``spack help``. Run with no
arguments, it will give a list of all spack options and subcommands:
.. command-output:: spack help
If you want help on the usage of a particular subcommand, you can pass
it as an argument to ``spack help``:
.. command-output:: spack help install
Alternately, you can use ``spack -h`` in place of ``spack help``, or
``spack <subcommand> -h`` to get help on a particular subcommand.
Viewing available packages
------------------------------
The first thing you will likely want to do with spack is find out what
software is available to install. There are two main commands for
this: ``spack list`` and ``spack info``.
``spack list``
~~~~~~~~~~~~~~~~
The ``spack list`` command does what you might expect. it prints out a
list of all the available packages you can install. Use it like
this:
.. command-output:: spack list
The packages are listed by name in alphabetical order. If you just
want to see *installed* packages, you should use ``spack list -i``
``spack info``
~~~~~~~~~~~~~~~~
To get information on a particular package from the full list, you can
run ``spack info <package name>``. e.g., for ``mpich``:
.. command-output:: spack info mpich
This gives basic information about the package, such as where it can
be downloaded, what other packages it depends on, virtual package
information, and a text description, if one is available. We'll give
more details on dependencies and virtual dependencies later in this
guide.
Installing and uninstalling
------------------------------
``spack install``
~~~~~~~~~~~~~~~~~~~~~
You can install any package from ``spack list``, using ``spack
install``. In the simplest case, if you just want the latest version
and you don't care about any configuration, you can just run ``spack
install <package>``:
.. code-block:: sh
spack install mpileaks
This will fetch the tarball for ``mpileaks``, expand it, verify that
it was donwloaded without errors, build the package, and install it in
its own directory in ``$SPACK_HOME/opt``. If the requested packages
depends on other packages in order to build, then they will also be
fetched and installed.
Spack also allows you to ask for *specific* configurations of a
package. For example, if you want to install something with a
specific version, you can add ``@`` after the package name, followed
by the version you want:
.. code-block:: sh
spack install mpich@3.0.4
You can install as many versions of the same pacakge as you want, and
they will not interfere with each other. Spack installs each package
into its own unique prefix. If you or another user links a library
against soething you install using Spack, it will continue to work
until you explicitly uninstall it.
The version isn't all that you can customize on a spack command line.
Spack can install many configurations, with different versions,
compilers, compiler versions, compile-time options (variants), and
even architectures (e.g., on a machine that requires cross-compiling).
Spack is also unique in that it lets you customize the *dependencies*
you build a package with. That is, you could have two configurations
of the same version of a package: one built with boost 1.39.0, and the
other version built with version 1.43.0.
Spack calls the descriptor used to refer to a particular package
configuration a **spec**. In the command lines above, both
``mpileaks`` and ``mpileaks@3.0.4`` are specs. Specs and their syntax
are covered in more detail in :ref:`sec-specs`.
``spack uninstall``
~~~~~~~~~~~~~~~~~~~~~
To uninstall a package, just type ``spack uninstall <package>``. This
will completely remove the directory in which the package was installed.
.. code-block:: sh
spack uninstall mpich
If there are other installed packages depend on the package you're
uninstalling, spack will issue a warning to this effect. In general,
you should remove the other packages *before* removing the package
they depend on, or you risk breaking packages on your system. If you
still want to remove the package without regard for its dependencies,
you can run ``spack uninstall -f <package>`` to override Spack's
warning.
If you have more than one version of the same package installed, spack
may not be able to figure out which on eyou want uninstalled. For
example, if you have both ``mpich@3.0.2`` and ``mpich@3.1`` installed,
and you type ``spack uninstall mpich``, then Spack will not know which
one you're referring to, and it will ask you to be more specific by
providing a version to differentiate, For example, ``spack uninstall
mpich@3.1`` is unambiguous.
.. _sec-specs:
Specs
-------------------------
Dependencies
-------------------------
Virtual dependencies
-------------------------
Versions, compilers, and architectures
----------------------------------------
``spack versions``
~~~~~~~~~~~~~~~~~~~~~~~~
``spack compilers``
~~~~~~~~~~~~~~~~~~~~~~~~
Architectures
~~~~~~~~~~~~~~~~~~~~~~~~
Spack's specs allow insatllations for multiple architectures to coexist
within the same prefix. It is also intended to support multiple
architecutres for cross-compilation.
Package lifecycle
------------------------------
The ``spack install`` command performs a number of tasks before it
finally installs each package. It downloads an archive, expands it in
a temporary directory, and only then performs the installation. Spack
has several commands that allow finer-grained control over each of
these stages of the build process.
``spack fetch``
~~~~~~~~~~~~~~~~~
This is the first step of ``spack install``. It takes a spec and
determines the correct download URL to use for the requested package
version. It then downloads the archive, checks it against an MD5
checksum, and stores it in a staging directory if the check was
successful. The staging directory will be located under
``$SPACK_HOME/var/spack``.
If run after the archive has already been downloaded, ``spack fetch``
is idempotent and will not download the archive again.
``spack stage``
~~~~~~~~~~~~~~~~~
This is the second step in installation after ``spack fetch``. It
expands the downloaded archive in its temporary directory, where it
will be built by ``spack install``. If the archive has already been
expanded, then this command does nothing.
``spack clean``
~~~~~~~~~~~~~~~~~
This command has several variations, each undoing one of the
installation tasks. They are:
``spack clean``
Runs ``make clean`` in the expanded archive directory. This is useful
if an attempted build failed, and something needs to be changed to get
a package to build. If a particular package does not have a ``make clean``
target, this will do nothing.
``spack clean -w`` or ``spack clean --work``
This deletes the entire build directory and re-expands it from the downloaded
archive. This is useful if a package does not support a proper ``make clean``
target.
``spack clean -d`` or ``spack clean --dist``
This deletes the build directory *and* the downloaded archive. If
``fetch``, ``stage``, or ``install`` are run again after this, the
process will start from scratch, and the archive archive will be
downloaded again. Useful if somehow a bad archive is downloaded
accidentally and needs to be cleaned out of the staging area.
``spack purge``
~~~~~~~~~~~~~~~~~

10
lib/spack/docs/conf.py
View File

@@ -16,7 +16,10 @@
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath('exts'))
os.environ['PATH'] += os.pathsep + '../../../bin'
todo_include_todos = True
# -- General configuration -----------------------------------------------------
@@ -25,7 +28,10 @@
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc']
extensions = ['sphinx.ext.autodoc',
'sphinx.ext.graphviz',
'sphinx.ext.todo',
'sphinxcontrib.programoutput']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

31
lib/spack/docs/developer_guide.rst Normal file
View File

@@ -0,0 +1,31 @@
Developer Guide
=====================
Spec objects
-------------------------
Package objects
-------------------------
Stage objects
-------------------------
Writing commands
-------------------------
Unit tests
-------------------------
Unit testing
-------------------------
Developer commands
-------------------------
``spack doc``
~~~~~~~~~~~~~~~~~
``spack test``
~~~~~~~~~~~~~~~~~

25
lib/spack/docs/exts/sphinxcontrib/LICENSE Normal file
View File

@@ -0,0 +1,25 @@
Copyright (c) 2010, 2011, 2012 Sebastian Wiesner <lunaryorn@googlemail.com>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

9
lib/spack/docs/exts/sphinxcontrib/__init__.py Normal file
View File

@@ -0,0 +1,9 @@
# -*- coding: utf-8 -*-
"""
sphinxcontrib
~~~~~~~~~~~~~
Contains 3rd party Sphinx extensions.
"""
__import__('pkg_resources').declare_namespace(__name__)

263
lib/spack/docs/exts/sphinxcontrib/programoutput.py Normal file
View File

@@ -0,0 +1,263 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2010, 2011, 2012, Sebastian Wiesner <lunaryorn@gmail.com>
# All rights reserved.
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
"""
sphinxcontrib.programoutput
===========================
This extension provides a directive to include the output of commands as
literal block while building the docs.
.. moduleauthor:: Sebastian Wiesner <lunaryorn@gmail.com>
"""
from __future__ import (print_function, division, unicode_literals,
absolute_import)
import sys
import os
import shlex
from subprocess import Popen, PIPE, STDOUT
from collections import defaultdict, namedtuple
from docutils import nodes
from docutils.parsers import rst
from docutils.parsers.rst.directives import flag, unchanged, nonnegative_int
__version__ = '0.9'
class program_output(nodes.Element):
pass
def _slice(value):
parts = [int(v.strip()) for v in value.split(',')]
if len(parts) > 2:
raise ValueError('too many slice parts')
return tuple((parts + [None] * 2)[:2])
class ProgramOutputDirective(rst.Directive):
has_content = False
final_argument_whitespace = True
required_arguments = 1
option_spec = dict(shell=flag, prompt=flag, nostderr=flag,
ellipsis=_slice, extraargs=unchanged,
returncode=nonnegative_int, cwd=unchanged)
def run(self):
env = self.state.document.settings.env
node = program_output()
node.line = self.lineno
node['command'] = self.arguments[0]
if self.name == 'command-output':
node['show_prompt'] = True
else:
node['show_prompt'] = 'prompt' in self.options
node['hide_standard_error'] = 'nostderr' in self.options
node['extraargs'] = self.options.get('extraargs', '')
_, cwd = env.relfn2path(self.options.get('cwd', '/'))
node['working_directory'] = cwd
node['use_shell'] = 'shell' in self.options
node['returncode'] = self.options.get('returncode', 0)
if 'ellipsis' in self.options:
node['strip_lines'] = self.options['ellipsis']
return [node]
_Command = namedtuple(
'Command', 'command shell hide_standard_error working_directory')
class Command(_Command):
"""
A command to be executed.
"""
def __new__(cls, command, shell=False, hide_standard_error=False,
working_directory='/'):
if isinstance(command, list):
command = tuple(command)
# `chdir()` resolves symlinks, so we need to resolve them too for
# caching to make sure that different symlinks to the same directory
# don't result in different cache keys. Also normalize paths to make
# sure that identical paths are also equal as strings.
working_directory = os.path.normpath(os.path.realpath(
working_directory))
return _Command.__new__(cls, command, shell, hide_standard_error,
working_directory)
@classmethod
def from_program_output_node(cls, node):
"""
Create a command from a :class:`program_output` node.
"""
extraargs = node.get('extraargs', '')
command = (node['command'] + ' ' + extraargs).strip()
return cls(command, node['use_shell'],
node['hide_standard_error'], node['working_directory'])
def execute(self):
"""
Execute this command.
Return the :class:`~subprocess.Popen` object representing the running
command.
"""
if self.shell:
if sys.version_info[0] < 3 and isinstance(self.command, unicode):
command = self.command.encode(sys.getfilesystemencoding())
else:
command = self.command
else:
if sys.version_info[0] < 3 and isinstance(self.command, unicode):
command = shlex.split(self.command.encode(
sys.getfilesystemencoding()))
elif isinstance(self.command, str):
command = shlex.split(self.command)
else:
command = self.command
return Popen(command, shell=self.shell, stdout=PIPE,
stderr=PIPE if self.hide_standard_error else STDOUT,
cwd=self.working_directory)
def get_output(self):
"""
Get the output of this command.
Return a tuple ``(returncode, output)``. ``returncode`` is the
integral return code of the process, ``output`` is the output as
unicode string, with final trailing spaces and new lines stripped.
"""
process = self.execute()
output = process.communicate()[0].decode(
sys.getfilesystemencoding(), 'replace').rstrip()
return process.returncode, output
def __str__(self):
if isinstance(self.command, tuple):
return repr(list(self.command))
return repr(self.command)
class ProgramOutputCache(defaultdict):
"""
Execute command and cache their output.
This class is a mapping. Its keys are :class:`Command` objects represeting
command invocations. Its values are tuples of the form ``(returncode,
output)``, where ``returncode`` is the integral return code of the command,
and ``output`` is the output as unicode string.
The first time, a key is retrieved from this object, the command is
invoked, and its result is cached. Subsequent access to the same key
returns the cached value.
"""
def __missing__(self, command):
"""
Called, if a command was not found in the cache.
``command`` is an instance of :class:`Command`.
"""
result = command.get_output()
self[command] = result
return result
def run_programs(app, doctree):
"""
Execute all programs represented by ``program_output`` nodes in
``doctree``. Each ``program_output`` node in ``doctree`` is then
replaced with a node, that represents the output of this program.
The program output is retrieved from the cache in
``app.env.programoutput_cache``.
"""
if app.config.programoutput_use_ansi:
# enable ANSI support, if requested by config
from sphinxcontrib.ansi import ansi_literal_block
node_class = ansi_literal_block
else:
node_class = nodes.literal_block
cache = app.env.programoutput_cache
for node in doctree.traverse(program_output):
command = Command.from_program_output_node(node)
try:
returncode, output = cache[command]
except EnvironmentError as error:
error_message = 'Command {0} failed: {1}'.format(command, error)
error_node = doctree.reporter.error(error_message, base_node=node)
node.replace_self(error_node)
else:
if returncode != node['returncode']:
app.warn('Unexpected return code {0} from command {1}'.format(
returncode, command))
# replace lines with ..., if ellipsis is specified
if 'strip_lines' in node:
lines = output.splitlines()
start, stop = node['strip_lines']
lines[start:stop] = ['...']
output = '\n'.join(lines)
if node['show_prompt']:
tmpl = app.config.programoutput_prompt_template
output = tmpl.format(command=node['command'], output=output,
returncode=returncode)
new_node = node_class(output, output)
new_node['language'] = 'text'
node.replace_self(new_node)
def init_cache(app):
"""
Initialize the cache for program output at
``app.env.programoutput_cache``, if not already present (e.g. being
loaded from a pickled environment).
The cache is of type :class:`ProgramOutputCache`.
"""
if not hasattr(app.env, 'programoutput_cache'):
app.env.programoutput_cache = ProgramOutputCache()
def setup(app):
app.add_config_value('programoutput_use_ansi', False, 'env')
app.add_config_value('programoutput_prompt_template',
'$ {command}\n{output}', 'env')
app.add_directive('program-output', ProgramOutputDirective)
app.add_directive('command-output', ProgramOutputDirective)
app.connect(str('builder-inited'), init_cache)
app.connect(str('doctree-read'), run_programs)

96
lib/spack/docs/features.rst Normal file
View File

@@ -0,0 +1,96 @@
Feature Overview
==================
This is an overview of features that make Spack different from other
`package managers <http://en.wikipedia.org/wiki/Package_management_system>`_
and `port systems <http://en.wikipedia.org/wiki/Ports_collection>`_.
Simple package installation
----------------------------
Installing packages is easy with Spack when you just want the default
version. This installs the latest version of mpileaks and all of its
dependencies:
.. code-block:: sh
$ spack install mpileaks
Custom versions & configurations
-------------------------------------------
If there's some aspect of your package that you want to customize, you
can do that too.
.. code-block:: sh
# Install a particular version by appending @
$ spack install mpileaks@1.1.2
# Or your favorite compiler (and its version), with %
$ spack install mpileaks@1.1.2 %gcc@4.7.3
# Add some special compile-time options with +
$ spack install mpileaks@1.1.2 %gcc@4.7.3 +debug
# Cross-compile for a different architecture with =
$ spack install mpileaks@1.1.2 =bgqos_0
Customize dependencies
-------------------------------------
You can customize package dependencies with ``^``. Suppose that
``mpileaks`` depends indirectly on ``libelf`` and ``libdwarf``. Using
``^``, you can add custom configurations for the dependencies, too.
.. code-block:: sh
# Install mpileaks and link it with specific versions of libelf and libdwarf
$ spack install mpileaks@1.1.2 %gcc@4.7.3 +debug ^libelf@0.8.12 ^libdwarf@20130729+debug
Non-destructive installs
-------------------------------------
Spack installs every unique package configuration in its own prefix,
so you can install as many different versions and configurations as
you want. New installs will not break existing ones.
Packages can peacefully coexist
-------------------------------------
Spack uses ``RPATH`` everywhere, so users do not need to customize
``LD_LIBRARY_PATH``. If you use a library or run a program, it will
run the way you built it.
Creating packages is easy
-------------------------------------
To create your own packages, give spack the tarball URL. Spack
creates all the boilerplate for you.
.. code-block:: sh
$ spack create http://scalability.llnl.gov/mpileaks/downloads/mpileaks-1.0.tar.gz
Creates ``mpileaks.py``:
.. code-block:: python
from spack import *
class Mpileaks(Package):
homepage = "http://www.example.com/"
url = "http://scalability.llnl.gov/mpileaks/downloads/mpileaks-1.0.tar.gz"
md5 = "4136d7b4c04df68b686570afa26988ac"
def install(self, prefix):
configure("--prefix=%s" % prefix)
make()
make("install")
Packages are pure python, so you have complete freedom when writing
build code. Spack also provides a number of feature that make it
easier to write packages.

44
lib/spack/docs/getting_started.rst Normal file
View File

@@ -0,0 +1,44 @@
Getting Started
====================
Download
--------------------
Getting spack is easy. Clone it using `git <http://git-scm.com->`_
with the following command:
.. code-block:: sh
$ git clone ssh://git@cz-stash.llnl.gov:7999/scale/spack.git
This will create a directory called ``spack``. We'll assume that the
full path to this directory is in some environment called
``SPACK_HOME``. Add ``$SPACK_HOME/bin`` to your path and you're ready
to go:
.. code-block:: sh
$ export PATH=spack/bin:$SPACK_HOME
$ spack install mpich
In general, most of your interactions with Spack will be through the
``spack`` command.
Install
--------------------
You don't need to install Spack; it's ready to run as soon as you
clone it from git.
You may want to run it out of a prefix other than the git repository
you cloned. The ``spack bootstrap`` command provides this
functionality. To install spack in a new directory, simply type:
.. code-block:: sh
$ spack bootstrap /my/favorite/prefix
This will install a new spack script in /my/favorite/prefix/bin, which
you can use just like you would the regular spack script. Each copy
of spack installs packages into its own ``$PREFIX/opt`` directory.

31
lib/spack/docs/index.rst
View File

@@ -3,15 +3,39 @@
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to Spack's documentation!
Spack Documentation
=================================
Contents:
Spack builds and installs software the way you want it. Other tools
let you install the latest version once. Spack lets you install the
versions you want, built with the compilers, libraries, and options
you want. Spack is non-destructive; installing a new version does not
break your old installs. See the :doc:`features` for more highlights.
Get spack and install your first package:
.. code-block:: sh
$ git clone ssh://git@cz-stash.llnl.gov:7999/scale/spack.git
$ cd spack/bin
$ ./spack install mpich
If you're new to spack and want to start using it, see :doc:`getting_started`,
or refer to the full manual below.
Table of Contents
---------------------
.. toctree::
:maxdepth: 2
features
getting_started
basic_usage
packaging_guide
site_configuration
developer_guide
API Docs <spack>
Indices and tables
==================
@@ -19,4 +43,3 @@ Indices and tables
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

35
lib/spack/docs/packaging_guide.rst Normal file
View File

@@ -0,0 +1,35 @@
Packaging Guide
=====================
Package files
-------------------------
Dependencies
-------------------------
Virtual dependencies
-------------------------
Virtual dependencies
-------------------------
Packaging commands
-------------------------
``spack edit``
~~~~~~~~~~~~~~~~~~~~
``spack create``
~~~~~~~~~~~~~~~~~~~~
``spack checksum``
~~~~~~~~~~~~~~~~~~~~
``spack graph``
~~~~~~~~~~~~~~~~~~~~

9
lib/spack/docs/site_configuration.rst Normal file
View File

@@ -0,0 +1,9 @@
Site-specific configuration
===================================
Temporary space
----------------------------
Concretization policies
----------------------------

6
lib/spack/spack/cmd/install-spack.py → lib/spack/spack/cmd/bootstrap.py
View File

@@ -17,7 +17,7 @@ def get_origin_url():
return origin_url.strip()
def install_spack(parser, args):
def bootstrap(parser, args):
origin_url = get_origin_url()
prefix = args.prefix
@@ -28,7 +28,7 @@ def install_spack(parser, args):
files_in_the_way = os.listdir(prefix)
if files_in_the_way:
tty.die("There are already files there! Delete these files before installing spack.",
tty.die("There are already files there! Delete these files before boostrapping spack.",
*files_in_the_way)
tty.msg("Installing:",
@@ -41,5 +41,5 @@ def install_spack(parser, args):
check_call(['git', 'fetch', 'origin', 'master:refs/remotes/origin/master', '-n', '-q'])
check_call(['git', 'reset', '--hard', 'origin/master', '-q'])
tty.msg("Successfully installed spack in %s" % prefix,
tty.msg("Successfully created a new spack in %s" % prefix,
"Run %s/bin/spack to use this installation." % prefix)

4
lib/spack/spack/cmd/clean.py
View File

@@ -9,9 +9,9 @@
def setup_parser(subparser):
subparser.add_argument('-c', "--clean", action="store_true", dest='clean',
help="run make clean in the stage directory (default)")
help="run make clean in the build directory (default)")
subparser.add_argument('-w', "--work", action="store_true", dest='work',
help="delete and re-expand the entire stage directory")
help="delete the build directory and re-expand it from its archive.")
subparser.add_argument('-d', "--dist", action="store_true", dest='dist',
help="delete the downloaded archive.")
subparser.add_argument('packages', nargs=argparse.REMAINDER,

0
lib/spack/spack/cmd/sys-type.py → lib/spack/spack/cmd/sys_type.py
View File

151
lib/spack/spack/package.py
View File

@@ -35,8 +35,8 @@
class Package(object):
"""This is the superclass for all spack packages.
The Package class
==================
***The Package class***
Package is where the bulk of the work of installing packages is done.
A package defines how to fetch, verfiy (via, e.g., md5), build, and
@@ -53,35 +53,39 @@ class Package(object):
in this directory. Spack automatically scans the python files there
and figures out which one to import when you invoke it.
An example package
====================
**An example package**
Let's look at the cmake package to start with. This package lives in
$prefix/lib/spack/spack/packages/cmake.py:
from spack import *
class Cmake(Package):
homepage = 'https://www.cmake.org'
url = 'http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz'
md5 = '097278785da7182ec0aea8769d06860c'
.. code-block:: python
def install(self, prefix):
configure('--prefix=%s' % prefix,
'--parallel=%s' % make_jobs)
make()
make('install')
from spack import *
class Cmake(Package):
homepage = 'https://www.cmake.org'
url = 'http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz'
md5 = '097278785da7182ec0aea8769d06860c'
def install(self, prefix):
configure('--prefix=%s' % prefix,
'--parallel=%s' % make_jobs)
make()
make('install')
**Naming conventions**
Naming conventions
---------------------
There are two names you should care about:
1. The module name, 'cmake'.
- User will refers to this name, e.g. 'spack install cmake'.
- Corresponds to the name of the file, 'cmake.py', and it can
include _, -, and numbers (it can even start with a number).
1. The module name, ``cmake``.
2. The class name, "Cmake". This is formed by converting -'s or _'s
in the module name to camel case. If the name starts with a number,
we prefix the class name with 'Num_'. Examples:
* User will refers to this name, e.g. 'spack install cmake'.
* Corresponds to the name of the file, 'cmake.py', and it can
include ``_``, ``-``, and numbers (it can even start with a
number).
2. The class name, "Cmake". This is formed by converting `-` or
``_`` in the module name to camel case. If the name starts with
a number, we prefix the class name with ``Num_``. Examples:
Module Name Class Name
foo_bar FooBar
@@ -91,23 +95,28 @@ def install(self, prefix):
The class name is what spack looks for when it loads a package module.
Required Attributes
---------------------
**Required Attributes**
Aside from proper naming, here is the bare minimum set of things you
need when you make a package:
homepage informational URL, so that users know what they're
installing.
url URL of the source archive that spack will fetch.
homepage
informational URL, so that users know what they're
installing.
md5 md5 hash of the source archive, so that we can
verify that it was downloaded securely and correctly.
url
URL of the source archive that spack will fetch.
install() This function tells spack how to build and install the
software it downloaded.
md5
md5 hash of the source archive, so that we can
verify that it was downloaded securely and correctly.
install()
This function tells spack how to build and install the
software it downloaded.
**Optional Attributes**
Optional Attributes
---------------------
You can also optionally add these attributes, if needed:
list_url
Webpage to scrape for available version strings. Default is the
@@ -121,8 +130,8 @@ def install(self, prefix):
your package needs special version formatting in its URL. boost
is an example of a package that needs this.
Creating Packages
===================
***Creating Packages***
As a package creator, you can probably ignore most of the preceding
information, because you can use the 'spack create' command to do it
all automatically.
@@ -130,8 +139,8 @@ def install(self, prefix):
You as the package creator generally only have to worry about writing
your install function and specifying dependencies.
spack create
----------------
**spack create**
Most software comes in nicely packaged tarballs, like this one:
http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz
@@ -147,15 +156,17 @@ def install(self, prefix):
Once this skeleton code is generated, spack pops up the new package in
your $EDITOR so that you can modify the parts that need changes.
Dependencies
---------------
**Dependencies**
If your package requires another in order to build, you can specify that
like this:
class Stackwalker(Package):
...
depends_on("libdwarf")
...
.. code-block:: python
class Stackwalker(Package):
...
depends_on("libdwarf")
...
This tells spack that before it builds stackwalker, it needs to build
the libdwarf package as well. Note that this is the module name, not
@@ -170,8 +181,8 @@ class Stackwalker(Package):
it will find the dependencies automatically.
The Install Function
----------------------
**The Install Function**
The install function is designed so that someone not too terribly familiar
with Python could write a package installer. For example, we put a number
of commands in install scope that you can use almost like shell commands.
@@ -195,47 +206,58 @@ class Stackwalker(Package):
them are created and set on the module.
Parallel Builds
-------------------
**Parallel Builds**
By default, Spack will run make in parallel when you run make() in your
install function. Spack figures out how many cores are available on
your system and runs make with -j<cores>. If you do not want this behavior,
you can explicitly mark a package not to use parallel make:
class SomePackage(Package):
...
parallel = False
...
.. code-block:: python
class SomePackage(Package):
...
parallel = False
...
This changes thd default behavior so that make is sequential. If you still
want to build some parts in parallel, you can do this in your install function:
make(parallel=True)
.. code-block:: python
make(parallel=True)
Likewise, if you do not supply parallel = True in your Package, you can keep
the default parallel behavior and run make like this when you want a
sequential build:
make(parallel=False)
.. code-block:: python
make(parallel=False)
**Package Lifecycle**
Package Lifecycle
==================
This section is really only for developers of new spack commands.
A package's lifecycle over a run of Spack looks something like this:
p = Package() # Done for you by spack
.. code-block:: python
p.do_fetch() # called by spack commands in spack/cmd.
p.do_stage() # see spack.stage.Stage docs.
p.do_install() # calls package's install() function
p.do_uninstall()
p = Package() # Done for you by spack
p.do_fetch() # called by spack commands in spack/cmd.
p.do_stage() # see spack.stage.Stage docs.
p.do_install() # calls package's install() function
p.do_uninstall()
There are also some other commands that clean the build area:
p.do_clean() # runs make clean
p.do_clean_work() # removes the build directory and
.. code-block:: python
p.do_clean() # runs make clean
p.do_clean_work() # removes the build directory and
# re-expands the archive.
p.do_clean_dist() # removes the stage directory entirely
p.do_clean_dist() # removes the stage directory entirely
The convention used here is that a do_* function is intended to be called
internally by Spack commands (in spack.cmd). These aren't for package
@@ -244,6 +266,7 @@ class SomePackage(Package):
Package creators override functions like install() (all of them do this),
clean() (some of them do this), and others to provide custom behavior.
"""
#

17
lib/spack/spack/relations.py
View File

@@ -68,9 +68,11 @@ def _caller_locals():
del stack
def _ensure_caller_is_spack_package():
"""Make sure that the caller is a spack package. If it's not,
raise ScopeError. if it is, return its name."""
def _get_calling_package_name():
"""Make sure that the caller is a class definition, and return
the module's name. This is useful for getting the name of
spack packages from inside a relation function.
"""
stack = inspect.stack()
try:
# get calling function name (the relation)
@@ -85,9 +87,6 @@ def _ensure_caller_is_spack_package():
raise ScopeError(relation)
module_name = caller_locals['__module__']
if not module_name.startswith(packages_module()):
raise ScopeError(relation)
base_name = module_name.split('.')[-1]
return base_name
@@ -121,7 +120,7 @@ def _parse_local_spec(spec_like, pkg_name):
"""Adds a dependencies local variable in the locals of
the calling class, based on args. """
def depends_on(*specs):
pkg = _ensure_caller_is_spack_package()
pkg = _get_calling_package_name()
dependencies = _caller_locals().setdefault('dependencies', {})
for string in specs:
@@ -136,7 +135,7 @@ def provides(*specs, **kwargs):
'mpi', other packages can declare that they depend on "mpi", and spack
can use the providing package to satisfy the dependency.
"""
pkg = _ensure_caller_is_spack_package()
pkg = _get_calling_package_name()
spec_string = kwargs.get('when', pkg)
provider_spec = _parse_local_spec(spec_string, pkg)
@@ -170,7 +169,7 @@ class ScopeError(RelationError):
def __init__(self, relation):
super(ScopeError, self).__init__(
relation,
"Cannot inovke '%s' from outside of a Spack package!" % relation)
"Must invoke '%s' from inside a class definition!" % relation)
class CircularReferenceError(RelationError):

59
lib/spack/spack/spec.py
View File

@@ -6,8 +6,10 @@
The syntax looks like this:
spack install mpileaks ^openmpi @1.2:1.4 +debug %intel @12.1
0 1 2 3 4 5
.. code-block:: sh
$ spack install mpileaks ^openmpi @1.2:1.4 +debug %intel @12.1 =bgqos_0
0 1 2 3 4 5 6
The first part of this is the command, 'spack install'. The rest of the
line is a spec for a particular installation of the mpileaks package.
@@ -37,7 +39,10 @@
if it comes immediately after the compiler name. Otherwise it will be
associated with the current package spec.
Here is the EBNF grammar for a spec:
6. The architecture to build with. This is needed on machines where
cross-compilation is required
Here is the EBNF grammar for a spec::
spec-list = { spec [ dep-list ] }
dep_list = { ^ spec }
@@ -80,8 +85,7 @@
"""This map determines the coloring of specs when using color output.
We make the fields different colors to enhance readability.
See spack.color for descriptions of the color codes.
"""
See spack.color for descriptions of the color codes. """
color_formats = {'%' : '@g', # compiler
'@' : '@c', # version
'=' : '@m', # architecture
@@ -129,8 +133,7 @@ def __call__(self, match):
class Compiler(object):
"""The Compiler field represents the compiler or range of compiler
versions that a package should be built with. Compilers have a
name and a version list.
"""
name and a version list. """
def __init__(self, name, version=None):
if name not in spack.compilers.supported_compilers():
raise UnknownCompilerError(name)
@@ -348,10 +351,11 @@ def package(self):
@property
def virtual(self):
"""Right now, a spec is virtual if no package exists with its name.
TODO: revisit this -- might need to use a separate namespace and
be more explicit about this.
Possible idea: just use conventin and make virtual deps all
caps, e.g., MPI vs mpi.
be more explicit about this.
Possible idea: just use conventin and make virtual deps all
caps, e.g., MPI vs mpi.
"""
return not packages.exists(self.name)
@@ -463,13 +467,15 @@ def _expand_virtual_packages(self):
and normalize again to include the provider's (potentially virtual)
dependencies. Repeat until there are no virtual deps.
TODO: If a provider depends on something that conflicts with
other dependencies in the spec being expanded, this can
produce a conflicting spec. For example, if mpich depends
on hwloc@:1.3 but something in the spec needs hwloc1.4:,
then we should choose an MPI other than mpich. Cases like
this are infrequent, but should implement this before it is
a problem.
.. todo::
If a provider depends on something that conflicts with
other dependencies in the spec being expanded, this can
produce a conflicting spec. For example, if mpich depends
on hwloc@:1.3 but something in the spec needs hwloc1.4:,
then we should choose an MPI other than mpich. Cases like
this are infrequent, but should implement this before it is
a problem.
"""
while True:
virtuals =[v for v in self.preorder_traversal() if v.virtual]
@@ -625,16 +631,19 @@ def normalize(self):
"""When specs are parsed, any dependencies specified are hanging off
the root, and ONLY the ones that were explicitly provided are there.
Normalization turns a partial flat spec into a DAG, where:
1) ALL dependencies of the root package are in the DAG.
2) Each node's dependencies dict only contains its direct deps.
3) There is only ONE unique spec for each package in the DAG.
- This includes virtual packages. If there a non-virtual
package that provides a virtual package that is in the spec,
then we replace the virtual package with the non-virtual one.
4) The spec DAG matches package DAG.
1. ALL dependencies of the root package are in the DAG.
2. Each node's dependencies dict only contains its direct deps.
3. There is only ONE unique spec for each package in the DAG.
* This includes virtual packages. If there a non-virtual
package that provides a virtual package that is in the spec,
then we replace the virtual package with the non-virtual one.
4. The spec DAG matches package DAG.
TODO: normalize should probably implement some form of cycle detection,
to ensure that the spec is actually a DAG.
to ensure that the spec is actually a DAG.
"""
# Ensure first that all packages in the DAG exist.
self.validate_package_names()

16
lib/spack/spack/stage.py
View File

@@ -18,12 +18,16 @@ class Stage(object):
expanded, and built before being installed. It also handles downloading
the archive. A stage's lifecycle looks like this:
setup() Create the stage directory.
fetch() Fetch a source archive into the stage.
expand_archive() Expand the source archive.
<install> Build and install the archive. This is handled
by the Package class.
destroy() Remove the stage once the package has been installed.
setup()
Create the stage directory.
fetch()
Fetch a source archive into the stage.
expand_archive()
Expand the source archive.
<install>
Build and install the archive. This is handled by the Package class.
destroy()
Remove the stage once the package has been installed.
If spack.use_tmp_stage is True, spack will attempt to create stages
in a tmp directory. Otherwise, stages are created directly in

16
lib/spack/spack/version.py
View File

@@ -1,15 +1,15 @@
"""
This file implements Version and version-ish objects. These are:
Version
A single version of a package.
VersionRange
A range of versions of a package.
VersionList
A list of Versions and VersionRanges.
Version
A single version of a package.
VersionRange
A range of versions of a package.
VersionList
A list of Versions and VersionRanges.
All of these types support the following operations, which can
be called on any of the types:
be called on any of the types::
__eq__, __ne__, __lt__, __gt__, __ge__, __le__, __hash__
__contains__
@@ -18,8 +18,6 @@
union
intersection
concrete
True if the Version, VersionRange or VersionList represents
a single version.
"""
import os
import sys