zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Documentation for dependency patching.

Browse Source
This commit is contained in:
Todd Gamblin 2017-09-27 20:03:30 -07:00
parent 4f8c7d57eb
commit 96d2488e0c
3 changed files with 189 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

152
lib/spack/docs/packaging_guide.rst
View File

@ -1212,27 +1212,47 @@ structure like this:
package.py
ad_lustre_rwcontig_open_source.patch
If you supply a URL instead of a filename, you need to supply a checksum,
like this:
If you supply a URL instead of a filename, you need to supply a
``sha256`` checksum, like this:
.. code-block:: python
patch('http://www.nwchem-sw.org/images/Tddft_mxvec20.patch',
sha256='252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866')
Spack includes the hashes of patches in its versioning information, so
that the same package with different patches applied will have different
hash identifiers. To ensure that the hashing scheme is consistent, you
must use a ``sha256`` checksum for the patch. Patches will be fetched
from their URLs, checked, and applied to your source code. You can use
the ``spack sha256`` command to generate a checksum for a patch file or
URL.
Spack can also handle compressed patches. If you use these, Spack needs
a little more help. Specifically, it needs *two* checksums: the
``sha256`` of the patch and ``archive_sha256`` for the compressed
archive. ``archive_sha256`` helps Spack ensure that the downloaded
file is not corrupted or malicious, before running it through a tool like
``tar`` or ``zip``. The ``sha256`` of the patch is still required so
that it can be included in specs. Providing it in the package file
ensures that Spack won't have to download and decompress patches it won't
end up using at install time. Both the archive and patch checksum are
checked when patch archives are downloaded.
.. code-block:: python
patch('http://www.nwchem-sw.org/images/Tddft_mxvec20.patch.gz',
md5='f91c6a04df56e228fe946291d2f38c9a')
sha256='252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866',
archive_sha256='4e8092a161ec6c3a1b5253176fcf33ce7ba23ee2ff27c75dbced589dabacd06e')
This directive provides an ``md5`` checksum. You can use other hashing
algorihtms like ``sha256`` as well. The patch will be fetched from the
URL, checked, and applied to your source code. You can use the ``spack
md5`` command to generate a checksum for a patch file.
``patch`` keyword arguments are described below.
``patch`` can take two options keyword arguments. They are:
""""""""""""""""""""""""""""""
``sha256``, ``archive_sha256``
""""""""""""""""""""""""""""""
""""""""""""""""""""""""""""""""""""""
``md5``, ``sha256``, ``sha512``, etc.
""""""""""""""""""""""""""""""""""""""
Use one of these when you supply a patch to be downloaded from a remote
site. The downloaded file will be validated using the given checksum.
Hashes of downloaded patch and compressed archive, respectively. Only
needed for patches fetched from URLs.
""""""""
``when``
@ -1309,6 +1329,21 @@ if you run install, hit ctrl-C, and run install again, the code in the
patch function is only run once. Also, you can tell Spack to run only
the patching part of the build using the :ref:`cmd-spack-patch` command.
.. _patch_dependency_patching:
^^^^^^^^^^^^^^^^^^^
Dependency patching
^^^^^^^^^^^^^^^^^^^
So far we've covered how the ``patch`` directive can be used by a package
to patch *its own* source code. Packages can *also* specify patches to be
applied to their dependencies, if they require special modifications. As
with all packages in Spack, a patched dependency library can coexist with
other versions of that library. See the `section on depends_on
<dependency_dependency_patching_>`_ for more details.
.. _handling_rpaths:
---------------
Handling RPATHs
---------------
@ -1482,7 +1517,7 @@ particular constraints, and package authors can use specs to describe
relationships between packages.
^^^^^^^^^^^^^^
Version Ranges
Version ranges
^^^^^^^^^^^^^^
Although some packages require a specific version for their dependencies,
@ -1530,7 +1565,7 @@ correct way to specify this would be:
^^^^^^^^^^^^^^^^
Dependency Types
Dependency types
^^^^^^^^^^^^^^^^
Not all dependencies are created equal, and Spack allows you to specify
@ -1566,6 +1601,91 @@ inject the dependency's ``prefix/lib`` directory, but the package needs to
be in ``PATH`` and ``PYTHONPATH`` during the build process and later when
a user wants to run the package.
.. _dependency_dependency_patching:
^^^^^^^^^^^^^^^^^^^
Dependency patching
^^^^^^^^^^^^^^^^^^^
Some packages maintain special patches on their dependencies, either to
add new features or to fix bugs. This typically makes a package harder
to maintain, and we encourage developers to upstream (contribute back)
their changes rather than maintaining patches. However, in some cases
it's not possible to upstream. Maybe the dependency's developers don't
accept changes, or maybe they just haven't had time to integrate them.
For times like these, Spack's ``depends_on`` directive can optionally
take a patch or list of patches:
.. code-block:: python
class SpecialTool(Package):
...
depends_on('binutils', patches='special-binutils-feature.patch')
...
Here, the ``special-tool`` package requires a special feature in
``binutils``, so it provides an extra ``patches=<filename>`` keyword
argument. This is similar to the `patch directive <patching_>`_, with
one small difference. Here, ``special-tool`` is responsible for the
patch, so it should live in ``special-tool``'s directory in the package
repository, not the ``binutils`` directory.
If you need something more sophisticated than this, you can simply nest a
``patch()`` directive inside of ``depends_on``:
.. code-block:: python
class SpecialTool(Package):
...
depends_on(
'binutils',
patches=patch('special-binutils-feature.patch',
level=3,
when='@:1.3'), # condition on binutils
when='@2.0:') # condition on special-tool
...
Note that there are two optional ``when`` conditions here -- one on the
``patch`` directive and the other on ``depends_on``. The condition in
the ``patch`` directive applies to ``binutils`` (the package being
patched), while the condition in ``depends_on`` applies to
``special-tool``. See `patch directive <patching_>`_ for details on all
the arguments the ``patch`` directive can take.
Finally, if you need *multiple* patches on a dependency, you can provide
a list for ``patches``, e.g.:
.. code-block:: python
class SpecialTool(Package):
...
depends_on(
'binutils',
patches=[
'binutils-bugfix1.patch',
'binutils-bugfix2.patch',
patch('https://example.com/special-binutils-feature.patch',
sha256='252c0af58be3d90e5dc5e0d16658434c9efa5d20a5df6c10bf72c2d77f780866',
when='@:1.3')],
when='@2.0:')
...
As with ``patch`` directives, patches are applied in the order they
appear in the package file (or in this case, in the list).
.. note::
You may wonder whether dependency patching will interfere with other
packages that depend on ``binutils``. It won't.
As described in patching_, Patching a package adds the ``sha256`` of
the patch to the package's spec, which means it will have a
*different* unique hash than other versions without the patch. The
patched version coexists with unpatched versions, and Spack's support
for handling_rpaths_ guarantees that each installation finds the
right version. If two packages depend on ``binutils`` patched *the
same* way, they can both use a single installation of ``binutils``.
.. _setup-dependent-environment:

20
lib/spack/spack/cmd/md5.py
View File

@ -42,13 +42,15 @@ def setup_parser(subparser):
help="files/urls to checksum")
def compute_md5_checksum(url):
def compute_checksum(url, algo):
algo = getattr(hashlib, algo)
if not os.path.isfile(url):
with Stage(url) as stage:
stage.fetch()
value = spack.util.crypto.checksum(hashlib.md5, stage.archive_file)
value = spack.util.crypto.checksum(algo, stage.archive_file)
else:
value = spack.util.crypto.checksum(hashlib.md5, url)
value = spack.util.crypto.checksum(algo, url)
return value
@ -61,7 +63,7 @@ def normalized(files):
yield value
def md5(parser, args):
def do_checksum(parser, args, algo):
if not args.files:
setup_parser.parser.print_help()
return 1
@ -70,7 +72,7 @@ def md5(parser, args):
results = []
for url in urls:
try:
checksum = compute_md5_checksum(url)
checksum = compute_checksum(url, algo)
results.append((checksum, url))
except FailedDownloadError as e:
tty.warn("Failed to fetch %s" % url)
@ -79,8 +81,12 @@ def md5(parser, args):
tty.warn("Error when reading %s" % url)
tty.warn("%s" % e)
# Dump the MD5s at last without interleaving them with downloads
# Dump the hashes last, without interleaving them with downloads
checksum = 'checksum' if len(results) == 1 else 'checksums'
tty.msg("%d MD5 %s:" % (len(results), checksum))
tty.msg("%d %s %s:" % (len(results), algo, checksum))
for checksum, url in results:
print("{0} {1}".format(checksum, url))
def md5(parser, args):
do_checksum(parser, args, 'md5')

40
lib/spack/spack/cmd/sha256.py Normal file
View File

@ -0,0 +1,40 @@
##############################################################################
# Copyright (c) 2013-2017, Lawrence Livermore National Security, LLC.
# Produced at the Lawrence Livermore National Laboratory.
#
# This file is part of Spack.
# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
# LLNL-CODE-647188
#
# For details, see https://github.com/llnl/spack
# Please also see the NOTICE and LICENSE files for our notice and the LGPL.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU Lesser General Public License (as
# published by the Free Software Foundation) version 2.1, February 1999.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
# conditions of the GNU Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
##############################################################################
import argparse
from spack.cmd.md5 import do_checksum
description = "calculate sha256 checksums for files/urls"
section = "packaging"
level = "long"
def setup_parser(subparser):
setup_parser.parser = subparser
subparser.add_argument('files', nargs=argparse.REMAINDER,
help="files/urls to checksum")
def sha256(parser, args):
do_checksum(parser, args, 'sha256')