zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add docs on new top-level VCS attributes

Browse Source
This commit is contained in:
Adam J. Stewart 2018-07-25 20:35:12 -05:00 committed by Todd Gamblin
parent 5d07042f23
commit 7902f3eb49
2 changed files with 188 additions and 116 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

297
lib/spack/docs/packaging_guide.rst
View File

@ -413,27 +413,26 @@ The most straightforward way to add new versions to your package is to
add a line like this in the package class:
.. code-block:: python
:linenos:
class Foo(Package):
url = 'http://example.com/foo-1.0.tar.gz'
version('8.2.1', '4136d7b4c04df68b686570afa26988ac')
...
Versions should be listed with the newest version first.
url = "http://example.com/foo-1.0.tar.gz"
version('8.2.1', '4136d7b4c04df68b686570afa26988ac')
version('8.2.0', '1c9f62f0778697a09d36121ead88e08e')
version('8.1.2', 'd47dd09ed7ae6e7fd6f9a816d7f5fdf6')
Versions should be listed in descending order, from newest to oldest.
^^^^^^^^^^^^^
Date Versions
^^^^^^^^^^^^^
If you wish to use dates as versions, it is best to use the format
``@date-yyyy-mm-dd``. This will ensure they sort in the correct
order. If you want your date versions to be numeric (assuming they
don't conflict with other numeric versions), you can use just
``yyyy.mm.dd``.
``@yyyy-mm-dd``. This will ensure they sort in the correct order.
Alternately, you might use a hybrid release-version / date scheme.
For example, ``@1.3.2016.08.31`` would mean the version from the
For example, ``@1.3_2016-08-31`` would mean the version from the
``1.3`` branch, as of August 31, 2016.
^^^^^^^^^^^^
@ -442,7 +441,7 @@ Version URLs
By default, each version's URL is extrapolated from the ``url`` field
in the package. For example, Spack is smart enough to download
version ``8.2.1.`` of the ``Foo`` package above from
version ``8.2.1`` of the ``Foo`` package above from
http://example.com/foo-8.2.1.tar.gz.
If the URL is particularly complicated or changes based on the release,
@ -515,44 +514,28 @@ of its versions, you can add an explicit URL for a particular version:
When you supply a custom URL for a version, Spack uses that URL
*verbatim* and does not perform extrapolation.
*verbatim* and does not perform extrapolation. The order of precedence
of these methods is:
^^^^^^^^^^^^^^^^^^^^^
PyPI and version URLs
^^^^^^^^^^^^^^^^^^^^^
#. package-level ``url``
#. ``url_for_version()``
#. version-specific ``url``
In addition to their developer websites, many python packages are hosted at the
`Python Package Index (PyPi) <https://pypi.python.org/pypi>`_. Although links to
these individual files are typically `generated using a hash
<https://bitbucket.org/pypa/pypi/issues/438>`_ it is often possible to find a
reliable link of the format
.. code-block:: sh
https://pypi.python.org/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
Packages hosted on GitHub and the like are often developer versions that do not
contain all of the files (e.g. configuration scripts) necessary to support
compilation. For this reason it is ideal to link to a repository such as PyPi
if possible.
More recently, sources are being indexed at `pypi.io <https://pypi.io>`_ as
well. Links obtained from this site follow a similar pattern, namely
.. code-block:: sh
https://pypi.io/packages/source/<first letter of package>/<package>/<package>-<version>.<extension>
These links currently redirect back to `pypi.python.org
<https://pypi.python.org>`_, but this `may change in the future
<https://bitbucket.org/pypa/pypi/issues/438#comment-27243225>`_.
so if your package contains a ``url_for_version()``, it can be overridden
by a version-specific ``url``.
If your package does not contain a package-level ``url`` or ``url_for_version()``,
Spack can determine which URL to download from even if only some of the versions
specify their own ``url``. Spack will use the nearest URL *before* the requested
version. This is useful for packages that have an easy to extrapolate URL, but
keep changing their URL format every few releases. With this method, you only
need to specify the ``url`` when the URL changes.
^^^^^^^^^^^^^^^^^^^^^^^^
Skipping the expand step
^^^^^^^^^^^^^^^^^^^^^^^^
Spack normally expands archives (e.g. `*.tar.gz` and `*.zip`) automatically
Spack normally expands archives (e.g. ``*.tar.gz`` and ``*.zip``) automatically
after downloading them. If you want to skip this step (e.g., for
self-extracting executables and other custom archive types), you can add
``expand=False`` to a ``version`` directive.
@ -560,7 +543,7 @@ self-extracting executables and other custom archive types), you can add
.. code-block:: python
version('8.2.1', '4136d7b4c04df68b686570afa26988ac',
url='http://example.com/foo-8.2.1-special-version.tar.gz', expand=False)
url='http://example.com/foo-8.2.1-special-version.sh', expand=False)
When ``expand`` is set to ``False``, Spack sets the current working
directory to the directory containing the downloaded archive before it
@ -605,7 +588,7 @@ here will suffice; for example, ``@develop``, ``@master``, ``@local``.
The following rules determine the sort order of numeric
vs. non-numeric versions:
#. The non-numeric versions ``@develop`` is considered greatest (newest).
#. The non-numeric version ``@develop`` is considered greatest (newest).
#. Numeric versions are all less than ``@develop`` version, and are
sorted numerically.
@ -617,7 +600,7 @@ The logic behind this sort order is two-fold:
#. Non-numeric versions are usually used for special cases while
developing or debugging a piece of software. Keeping most of them
less than numeric versions ensures that Spack choose numeric
less than numeric versions ensures that Spack chooses numeric
versions by default whenever possible.
#. The most-recent development version of a package will usually be
@ -679,7 +662,7 @@ example for ``libelf``:
...
0.5.2 http://www.mr511.de/software/libelf-0.5.2.tar.gz
How many would you like to checksum? (default is 5, q to abort)
How many would you like to checksum? (default is 1, q to abort)
This does the same thing that ``spack create`` does, but it allows you
to go back and add new versions easily as you need them (e.g., as
@ -805,12 +788,12 @@ But its downloads are in many different subdirectories of
homepage = "http://www.mpich.org"
url = "http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz"
list_url = "http://www.mpich.org/static/downloads/"
list_depth = 2
list_depth = 1
By default, Spack only looks at the top-level page available at
``list_url``. ``list_depth`` tells it to follow up to 2 levels of
links from the top-level page. Note that here, this implies two
levels of subdirectories, as the ``mpich`` website is structured much
``list_url``. ``list_depth = 1`` tells it to follow up to 1 level of
links from the top-level page. Note that here, this implies 1
level of subdirectories, as the ``mpich`` website is structured much
like a filesystem. But ``list_depth`` really refers to link depth
when spidering the page.
@ -823,13 +806,57 @@ Fetching from code repositories
For some packages, source code is provided in a Version Control System
(VCS) repository rather than in a tarball. Spack can fetch packages
from VCS repositories. Currently, Spack supports fetching with `Git
<git-fetch_>`_, `Mercurial (hg) <hg-fetch_>`_, and `Subversion (SVN)
<svn-fetch_>`_.
<git-fetch_>`_, `Mercurial (hg) <hg-fetch_>`_, `Subversion (svn)
<svn-fetch_>`_, and `Go <go-fetch_>`_.
To fetch a package from a source repository, Spack needs to know which
VCS to use and where to download from. Much like with ``url``, package
authors can specify a class-level ``git``, ``hg``, ``svn``, or ``go``
attribute containing the correct download location.
Many packages developed with Git have both a Git repository as well as
release tarballs available for download. Packages can define both a
class-level tarball URL and VCS. For example:
.. code-block:: python
class Trilinos(CMakePackage):
homepage = "https://trilinos.org/"
url = "https://github.com/trilinos/Trilinos/archive/trilinos-release-12-12-1.tar.gz"
git = "https://github.com/trilinos/Trilinos.git"
version('develop', branch='develop')
version('master', branch='master')
version('12.12.1', 'ecd4606fa332212433c98bf950a69cc7')
version('12.10.1', '667333dbd7c0f031d47d7c5511fd0810')
version('12.8.1', '9f37f683ee2b427b5540db8a20ed6b15')
If a package contains both a ``url`` and ``git`` class-level attribute,
Spack decides which to use based on the arguments to the ``version()``
directive. Versions containing a specific branch, tag, or revision are
assumed to be for VCS download methods, while versions containing a
checksum are assumed to be for URL download methods.
Like ``url``, if a specific version downloads from a different repository
than the default repo, it can be overridden with a version-specific argument.
.. note::
In order to reduce ambiguity, each package can only have a single VCS
top-level attribute in addition to ``url``. In the rare case that a
package uses multiple VCS, a fetch strategy can be specified for each
version. For example, the ``rockstar`` package contains:
.. code-block:: python
class Rockstar(MakefilePackage):
homepage = "https://bitbucket.org/gfcstanford/rockstar"
version('develop', git='https://bitbucket.org/gfcstanford/rockstar.git')
version('yt', hg='https://bitbucket.org/MatthewTurk/rockstar')
To fetch a package from a source repository, you add a ``version()``
call to your package with parameters indicating the repository URL and
any branch, tag, or revision to fetch. See below for the parameters
you'll need for each VCS system.
.. _git-fetch:
@ -837,11 +864,11 @@ you'll need for each VCS system.
Git
^^^
Git fetching is enabled with the following parameters to ``version``:
Git fetching supports the following parameters to ``version``:
* ``git``: URL of the git repository.
* ``tag``: name of a tag to fetch.
* ``branch``: name of a branch to fetch.
* ``git``: URL of the git repository, if different than the class-level ``git``.
* ``branch``: Name of a branch to fetch.
* ``tag``: Name of a tag to fetch.
* ``commit``: SHA hash (or prefix) of a commit to fetch.
* ``submodules``: Also fetch submodules recursively when checking out this repository.
@ -853,66 +880,72 @@ Default branch
.. code-block:: python
class Example(Package):
...
version('develop', git='https://github.com/example-project/example.git')
This download method is untrusted, and is not recommended.
git = "https://github.com/example-project/example.git"
Tags
To fetch from a particular tag, use the ``tag`` parameter along with
``git``:
version('develop')
.. code-block:: python
version('1.0.1', git='https://github.com/example-project/example.git',
tag='v1.0.1')
This download method is untrusted, and is not recommended.
This download method is untrusted, and is not recommended. Aside from HTTPS,
there is no way to verify that the repository has not been compromised, and
the commit you get when you install the package likely won't be the same
commit that was used when the package was first written. Additionally, the
default branch may change. It is best to at least specify a branch name.
Branches
To fetch a particular branch, use ``branch`` instead:
To fetch a particular branch, use the ``branch`` parameter:
.. code-block:: python
version('experimental', git='https://github.com/example-project/example.git',
branch='experimental')
version('experimental', branch='experimental')
This download method is untrusted, and is not recommended.
This download method is untrusted, and is not recommended. Branches are
moving targets, so the commit you get when you install the package likely
won't be the same commit that was used when the package was first written.
Tags
To fetch from a particular tag, use ``tag`` instead:
.. code-block:: python
version('1.0.1', tag='v1.0.1')
This download method is untrusted, and is not recommended. Although tags
are generally more stable than branches, Git allows tags to be moved.
Many developers use tags to denote rolling releases, and may move the
tag when a bug is patched.
Commits
Finally, to fetch a particular commit, use ``commit``:
.. code-block:: python
version('2014-10-08', git='https://github.com/example-project/example.git',
commit='9d38cd4e2c94c3cea97d0e2924814acc')
version('2014-10-08', commit='9d38cd4e2c94c3cea97d0e2924814acc')
This doesn't have to be a full hash; you can abbreviate it as you'd
expect with git:
.. code-block:: python
version('2014-10-08', git='https://github.com/example-project/example.git',
commit='9d38cd')
version('2014-10-08', commit='9d38cd')
This download method *is trusted*. It is the recommended way to
securely download from a Git repository.
It may be useful to provide a saner version for commits like this,
e.g. you might use the date as the version, as done above. Or you
could just use the abbreviated commit hash. It's up to the package
author to decide what makes the most sense.
e.g. you might use the date as the version, as done above. Or, if you
know the commit at which a release was cut, you can use the release
version. It's up to the package author to decide what makes the most
sense. Although you can use the commit hash as the version number,
this is not recommended, as it won't sort properly.
Submodules
You can supply ``submodules=True`` to cause Spack to fetch submodules
recursively along with the repository at fetch time. For more information
about git submodules see the manpage of git: ``man git-submodule``.
.. code-block:: python
version('1.0.1', git='https://github.com/example-project/example.git',
tag='v1.0.1', submodules=True)
version('1.0.1', tag='v1.0.1', submodules=True)
.. _github-fetch:
@ -921,7 +954,7 @@ Submodules
GitHub
^^^^^^
If a project is hosted on GitHub, *any* valid Git branch, tag or hash
If a project is hosted on GitHub, *any* valid Git branch, tag, or hash
may be downloaded as a tarball. This is accomplished simply by
constructing an appropriate URL. Spack can checksum any package
downloaded this way, thereby producing a trusted download. For
@ -931,7 +964,7 @@ checksum.
.. code-block:: python
version('1.9.5.1.1', 'd035e4bc704d136db79b43ab371b27d2',
url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f')
url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f')
.. _hg-fetch:
@ -939,34 +972,38 @@ checksum.
Mercurial
^^^^^^^^^
Fetching with mercurial works much like `git <git-fetch>`_, but you
Fetching with Mercurial works much like `Git <git-fetch>`_, but you
use the ``hg`` parameter.
Default
Add the ``hg`` parameter with no ``revision``:
Default branch
Add the ``hg`` attribute with no ``revision`` passed to ``version``:
.. code-block:: python
version('develop', hg='https://jay.grs.rwth-aachen.de/hg/example')
class Example(Package):
This download method is untrusted, and is not recommended.
hg = "https://bitbucket.org/example-project/example"
version('develop')
This download method is untrusted, and is not recommended. As with
Git's default fetching strategy, there is no way to verify the
integrity of the download.
Revisions
Add ``hg`` and ``revision`` parameters:
To fetch a particular revision, use the ``revision`` parameter:
.. code-block:: python
version('1.0', hg='https://jay.grs.rwth-aachen.de/hg/example',
revision='v1.0')
This download method is untrusted, and is not recommended.
version('1.0', revision='v1.0')
Unlike ``git``, which has special parameters for different types of
revisions, you can use ``revision`` for branches, tags, and commits
when you fetch with Mercurial.
when you fetch with Mercurial. Like Git, fetching specific branches
or tags is an untrusted download method, and is not recommended.
The recommended fetch strategy is to specify a particular commit
hash as the revision.
As with git, you can fetch these versions using the ``spack install
example@<version>`` command-line syntax.
.. _svn-fetch:
@ -974,34 +1011,70 @@ example@<version>`` command-line syntax.
Subversion
^^^^^^^^^^
To fetch with subversion, use the ``svn`` and ``revision`` parameters:
To fetch with subversion, use the ``svn`` and ``revision`` parameters.
Fetching the head
Simply add an ``svn`` parameter to ``version``:
Simply add an ``svn`` parameter to the package:
.. code-block:: python
version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk')
class Example(Package):
This download method is untrusted, and is not recommended.
svn = "https://outreach.scidac.gov/svn/example/trunk"
version('develop')
This download method is untrusted, and is not recommended for the
same reasons as mentioned above.
Fetching a revision
To fetch a particular revision, add a ``revision`` to the
version call:
To fetch a particular revision, add a ``revision`` argument to the
version directive:
.. code-block:: python
version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk',
revision=128)
version('develop', revision=128)
This download method is untrusted, and is not recommended.
Subversion branches are handled as part of the directory structure, so
you can check out a branch or tag by changing the ``url``.
Unfortunately, Subversion has no commit hashing scheme like Git and
Mercurial do, so there is no way to guarantee that the download you
get is the same as the download used when the package was created.
Use at your own risk.
-------------------------------------------------
Subversion branches are handled as part of the directory structure, so
you can check out a branch or tag by changing the URL. If you want to
package multiple branches, simply add a ``svn`` argument to each
version directive.
.. _go-fetch:
^^
Go
^^
Go isn't a VCS, it is a programming language with a builtin command,
`go get <https://golang.org/cmd/go/#hdr-Download_and_install_packages_and_dependencies>`_,
that fetches packages and their dependencies automatically.
It can clone a Git repository, or download from another source location.
For example:
.. code-block:: python
class ThePlatinumSearcher(Package):
homepage = "https://github.com/monochromegane/the_platinum_searcher"
go = "github.com/monochromegane/the_platinum_searcher/..."
version('head')
Go cannot be used to fetch a particular commit or branch, it always
downloads the head of the repository. This download method is untrusted,
and is not recommended. Use another fetch strategy whenever possible.
------------------------------------
Resources (expanding extra tarballs)
-------------------------------------------------
------------------------------------
Some packages (most notably compilers) provide optional features if additional
resources are expanded within their source tree before building. In Spack it is

7
var/spack/repos/builtin/packages/the-platinum-searcher/package.py
View File

@ -29,12 +29,11 @@
class ThePlatinumSearcher(Package):
"""Fast parallel recursive grep alternative"""
homepage = "https://github.com/monochromegane/the_platinum_searcher"
url = "https://github.com/monochromegane/the_platinum_searcher"
go = "github.com/monochromegane/the_platinum_searcher/..."
package = 'github.com/monochromegane/the_platinum_searcher/...'
version('head', go=package)
version('head')
extends("go", deptypes='build')