eef041abee
We've included a package hash in Spack since #7193 for CI, and we started using it on the spec in #28504. However, what goes into the package hash is a bit opaque. Here's what `spec.json` looks like now: ```json { "spec": { "_meta": { "version": 3 }, "nodes": [ { "name": "zlib", "version": "1.2.12", ... "patches": [ "0d38234384870bfd34dfcb738a9083952656f0c766a0f5990b1893076b084b76" ], "package_hash": "pthf7iophdyonixxeed7gyqiksopxeklzzjbxtjrw7nzlkcqleba====", "hash": "ke4alug7ypoxp37jb6namwlxssmws4kp" } ] } } ``` The `package_hash` there is a hash of the concatenation of: * A canonical hash of the `package.py` recipe, as implemented in #28156; * `sha256`'s of patches applied to the spec; and * Archive `sha256` sums of archives or commits/revisions of repos used to build the spec. There are some issues with this: patches are counted twice in this spec (in `patches` and in the `package_hash`), the hashes of sources used to build are conflated with the `package.py` hash, and we don't actually include resources anywhere. With this PR, I've expanded the package hash out in the `spec.json` body. Here is the "same" spec with the new fields: ```json { "spec": { "_meta": { "version": 3 }, "nodes": [ { "name": "zlib", "version": "1.2.12", ... "package_hash": "6kkliqdv67ucuvfpfdwaacy5bz6s6en4", "sources": [ { "type": "archive", "sha256": "91844808532e5ce316b3c010929493c0244f3d37593afd6de04f71821d5136d9" } ], "patches": [ "0d38234384870bfd34dfcb738a9083952656f0c766a0f5990b1893076b084b76" ], "hash": "ts3gkpltbgzr5y6nrfy6rzwbjmkscein" } ] } } ``` Now: * Patches and archive hashes are no longer included in the `package_hash`; * Artifacts used in the build go in `sources`, and we tell you their checksum in the `spec.json`; * `sources` will include resources for packages that have it; * Patches are the same as before -- but only represented once; and * The `package_hash` is a base32-encoded `sha1`, like other hashes in Spack, and it only tells you that the `package.py` changed. The behavior of the DAG hash (which includes the `package_hash`) is basically the same as before, except now resources are included, and we can see differences in archives and resources directly in the `spec.json` Note that we do not need to bump the spec meta version on this, as past versions of Spack can still read the new specs; they just will not notice the new fields (which is fine, since we currently do not do anything with them). Among other things, this will more easily allow us to convert Spack specs to SBOM and track relevant security information (like `sha256`'s of archives). For example, we could do continuous scanning of a Spack installation based on these hashes, and if the `sha256`'s become associated with CVE's, we'll know we're affected. - [x] Add a method, `spec_attrs()` to `FetchStrategy` that can be used to describe a fetcher for a `spec.json`. - [x] Simplify the way package_hash() is handled in Spack. Previously, it was handled as a special-case spec hash in `hash_types.py`, but it really doesn't belong there. Now, it's handled as part of `Spec._finalize_concretization()` and `hash_types.py` is much simpler. - [x] Change `PackageBase.content_hash()` to `PackageBase.artifact_hashes()`, and include more information about artifacts in it. - [x] Update package hash tests and make them check for artifact and resource hashes. Signed-off-by: Todd Gamblin <tgamblin@llnl.gov> |
||
|---|---|---|
| .devcontainer | ||
| .github | ||
| bin | ||
| etc/spack/defaults | ||
| lib/spack | ||
| share/spack | ||
| var/spack | ||
| .codecov.yml | ||
| .dockerignore | ||
| .flake8 | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .mailmap | ||
| .readthedocs.yml | ||
| CHANGELOG.md | ||
| CITATION.cff | ||
| COPYRIGHT | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| NOTICE | ||
| pyproject.toml | ||
| pytest.ini | ||
| README.md | ||
| SECURITY.md | ||
Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, Windows, and many supercomputers. Spack is non-destructive: installing a new version of a package does not break existing installations, so many configurations of the same package can coexist.
Spack offers a simple "spec" syntax that allows users to specify versions and configuration options. Package files are written in pure Python, and specs allow package authors to write a single script for many different builds of the same package. With Spack, you can build your software all the ways you want to.
See the Feature Overview for examples and highlights.
To install spack and your first package, make sure you have Python & Git. Then:
$ git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git
$ cd spack/bin
$ ./spack install zlib
Tip
-c feature.manyFiles=trueimproves git's performance on repositories with 1,000+ files.
--depth=2prunes the git history to reduce the size of the Spack installation.
Documentation
Full documentation is available, or
run spack help or spack help --all.
For a cheat sheet on Spack syntax, run spack help --spec.
Tutorial
We maintain a hands-on tutorial. It covers basic to advanced usage, packaging, developer features, and large HPC deployments. You can do all of the exercises on your own laptop using a Docker container.
Feel free to use these materials to teach users at your organization about Spack.
Community
Spack is an open source project. Questions, discussion, and contributions are welcome. Contributions can be anything from new packages to bugfixes, documentation, or even new core features.
Resources:
- Slack workspace: spackpm.slack.com. To get an invitation, visit slack.spack.io.
- Matrix space: #spack-space:matrix.org: bridged to Slack.
- Github Discussions: for Q&A and discussions. Note the pinned discussions for announcements.
- X: @spackpm. Be sure to
@mentionus! - Mailing list: groups.google.com/d/forum/spack: only for announcements. Please use other venues for discussions.
Contributing
Contributing to Spack is relatively easy. Just send us a
pull request.
When you send your request, make develop the destination branch on the
Spack repository.
Your PR must pass Spack's unit tests and documentation tests, and must be PEP 8 compliant. We enforce these guidelines with our CI process. To run these tests locally, and for helpful tips on git, see our Contribution Guide.
Spack's develop branch has the latest contributions. Pull requests
should target develop, and users who want the latest package versions,
features, etc. can use develop.
Releases
For multi-user site deployments or other use cases that need very stable software installations, we recommend using Spack's stable releases.
Each Spack release series also has a corresponding branch, e.g.
releases/v0.14 has 0.14.x versions of Spack, and releases/v0.13 has
0.13.x versions. We backport important bug fixes to these branches but
we do not advance the package versions or make other changes that would
change the way Spack concretizes dependencies within a release branch.
So, you can base your Spack deployment on a release branch and git pull
to get fixes, without the package churn that comes with develop.
The latest release is always available with the releases/latest tag.
See the docs on releases for more details.
Code of Conduct
Please note that Spack has a Code of Conduct. By participating in the Spack community, you agree to abide by its rules.
Authors
Many thanks go to Spack's contributors.
Spack was created by Todd Gamblin, tgamblin@llnl.gov.
Citing Spack
If you are referencing Spack in a publication, please cite the following paper:
- Todd Gamblin, Matthew P. LeGendre, Michael R. Collette, Gregory L. Lee, Adam Moody, Bronis R. de Supinski, and W. Scott Futral. The Spack Package Manager: Bringing Order to HPC Software Chaos. In Supercomputing 2015 (SC’15), Austin, Texas, November 15-20 2015. LLNL-CONF-669890.
On GitHub, you can copy this citation in APA or BibTeX format via the "Cite this repository"
button. Or, see the comments in CITATION.cff for the raw BibTeX.
License
Spack is distributed under the terms of both the MIT license and the Apache License (Version 2.0). Users may choose either license, at their option.
All new contributions must be made under both the MIT and Apache-2.0 licenses.
See LICENSE-MIT, LICENSE-APACHE, COPYRIGHT, and NOTICE for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
LLNL-CODE-811652