9818002219
Continuing the work started in #40326, his changes the structure of Variant metadata on Packages from a single variant definition per name with a list of `when` specs: ``` name: (Variant, [when_spec, ...]) ``` to a Variant definition per `when_spec` per name: ``` when_spec: { name: Variant } ``` With this change, everything on a package *except* versions is keyed by `when` spec. This: 1. makes things consistent, in that conditional things are (nearly) all modeled in the same way; and 2. fixes an issue where we would lose information about multiple variant definitions in a package (see #38302). We can now have, e.g., different defaults for the same variant in different versions of a package. Some notes: 1. This required some pretty deep changes to the solver. Previously, the solver's job was to select value(s) for a single variant definition per name per package. Now, the solver needs to: a. Determine which variant definition should be used for a given node, which can depend on the node's version, compiler, target, other variants, etc. b. Select valid value(s) for variants for each node based on the selected variant definition. When multiple variant definitions are enabled via their `when=` clause, we will always prefer the *last* matching definition, by declaration order in packages. This is implemented by adding a `precedence` to each variant at definition time, and we ensure they are added to the solver in order of precedence. This has the effect that variant definitions from derived classes are preferred over definitions from superclasses, and the last definition within the same class sticks. This matches python semantics. Some examples: ```python class ROCmPackage(PackageBase): variant("amdgpu_target", ..., when="+rocm") class Hipblas(ROCmPackage): variant("amdgpu_target", ...) ``` The global variant in `hipblas` will always supersede the `when="+rocm"` variant in `ROCmPackage`. If `hipblas`'s variant was also conditional on `+rocm` (as it probably should be), we would again filter out the definition from `ROCmPackage` because it could never be activated. If you instead have: ```python class ROCmPackage(PackageBase): variant("amdgpu_target", ..., when="+rocm") class Hipblas(ROCmPackage): variant("amdgpu_target", ..., when="+rocm+foo") ``` The variant on `hipblas` will win for `+rocm+foo` but the one on `ROCmPackage` will win with `rocm~foo`. So, *if* we can statically determine if a variant is overridden, we filter it out. This isn't strictly necessary, as the solver can handle many definitions fine, but this reduces the complexity of the problem instance presented to `clingo`, and simplifies output in `spack info` for derived packages. e.g., `spack info hipblas` now shows only one definition of `amdgpu_target` where before it showed two, one of which would never be used. 2. Nearly all access to the `variants` dictionary on packages has been refactored to use the following class methods on `PackageBase`: * `variant_names(cls) -> List[str]`: get all variant names for a package * `has_variant(cls, name) -> bool`: whether a package has a variant with a given name * `variant_definitions(cls, name: str) -> List[Tuple[Spec, Variant]]`: all definitions of variant `name` that are possible, along with their `when` specs. * `variant_items() -> `: iterate over `pkg.variants.items()`, with impossible variants filtered out. Consolidating to these methods seems to simplify the code a lot. 3. The solver does a lot more validation on variant values at setup time now. In particular, it checks whether a variant value on a spec is valid given the other constraints on that spec. This allowed us to remove the crufty logic in `update_variant_validate`, which was needed because we previously didn't *know* after a solve which variant definition had been used. Now, variant values from solves are constructed strictly based on which variant definition was selected -- no more heuristics. 4. The same prevalidation can now be done in package audits, and you can run: ``` spack audit packages --strict-variants ``` This turns up around 18 different places where a variant specification isn't valid given the conditions on variant definitions in packages. I haven't fixed those here but will open a separate PR to iterate on them. I plan to make strict checking the defaults once all existing package issues are resolved. It's not clear to me that strict checking should be the default for the prevalidation done at solve time. There are a few other changes here that might be of interest: 1. The `generator` variant in `CMakePackage` is now only defined when `build_system=cmake`. 2. `spack info` has been updated to support the new metadata layout. 3. split out variant propagation into its own `.lp` file in the `solver` code. 4. Add better typing and clean up code for variant types in `variant.py`. 5. Add tests for new variant behavior. |
||
|---|---|---|
| .devcontainer | ||
| .github | ||
| bin | ||
| etc/spack/defaults | ||
| lib/spack | ||
| share/spack | ||
| var/spack | ||
| .codecov.yml | ||
| .dockerignore | ||
| .flake8 | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .mailmap | ||
| .readthedocs.yml | ||
| CHANGELOG.md | ||
| CITATION.cff | ||
| COPYRIGHT | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| NOTICE | ||
| pyproject.toml | ||
| pytest.ini | ||
| README.md | ||
| SECURITY.md | ||
Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, Windows, and many supercomputers. Spack is non-destructive: installing a new version of a package does not break existing installations, so many configurations of the same package can coexist.
Spack offers a simple "spec" syntax that allows users to specify versions and configuration options. Package files are written in pure Python, and specs allow package authors to write a single script for many different builds of the same package. With Spack, you can build your software all the ways you want to.
See the Feature Overview for examples and highlights.
To install spack and your first package, make sure you have Python. Then:
$ git clone -c feature.manyFiles=true https://github.com/spack/spack.git
$ cd spack/bin
$ ./spack install zlib
Documentation
Full documentation is available, or
run spack help or spack help --all.
For a cheat sheet on Spack syntax, run spack help --spec.
Tutorial
We maintain a hands-on tutorial. It covers basic to advanced usage, packaging, developer features, and large HPC deployments. You can do all of the exercises on your own laptop using a Docker container.
Feel free to use these materials to teach users at your organization about Spack.
Community
Spack is an open source project. Questions, discussion, and contributions are welcome. Contributions can be anything from new packages to bugfixes, documentation, or even new core features.
Resources:
- Slack workspace: spackpm.slack.com. To get an invitation, visit slack.spack.io.
- Matrix space: #spack-space:matrix.org: bridged to Slack.
- Github Discussions: for Q&A and discussions. Note the pinned discussions for announcements.
- X: @spackpm. Be sure to
@mentionus! - Mailing list: groups.google.com/d/forum/spack: only for announcements. Please use other venues for discussions.
Contributing
Contributing to Spack is relatively easy. Just send us a
pull request.
When you send your request, make develop the destination branch on the
Spack repository.
Your PR must pass Spack's unit tests and documentation tests, and must be PEP 8 compliant. We enforce these guidelines with our CI process. To run these tests locally, and for helpful tips on git, see our Contribution Guide.
Spack's develop branch has the latest contributions. Pull requests
should target develop, and users who want the latest package versions,
features, etc. can use develop.
Releases
For multi-user site deployments or other use cases that need very stable software installations, we recommend using Spack's stable releases.
Each Spack release series also has a corresponding branch, e.g.
releases/v0.14 has 0.14.x versions of Spack, and releases/v0.13 has
0.13.x versions. We backport important bug fixes to these branches but
we do not advance the package versions or make other changes that would
change the way Spack concretizes dependencies within a release branch.
So, you can base your Spack deployment on a release branch and git pull
to get fixes, without the package churn that comes with develop.
The latest release is always available with the releases/latest tag.
See the docs on releases for more details.
Code of Conduct
Please note that Spack has a Code of Conduct. By participating in the Spack community, you agree to abide by its rules.
Authors
Many thanks go to Spack's contributors.
Spack was created by Todd Gamblin, tgamblin@llnl.gov.
Citing Spack
If you are referencing Spack in a publication, please cite the following paper:
- Todd Gamblin, Matthew P. LeGendre, Michael R. Collette, Gregory L. Lee, Adam Moody, Bronis R. de Supinski, and W. Scott Futral. The Spack Package Manager: Bringing Order to HPC Software Chaos. In Supercomputing 2015 (SC’15), Austin, Texas, November 15-20 2015. LLNL-CONF-669890.
On GitHub, you can copy this citation in APA or BibTeX format via the "Cite this repository"
button. Or, see the comments in CITATION.cff for the raw BibTeX.
License
Spack is distributed under the terms of both the MIT license and the Apache License (Version 2.0). Users may choose either license, at their option.
All new contributions must be made under both the MIT and Apache-2.0 licenses.
See LICENSE-MIT, LICENSE-APACHE, COPYRIGHT, and NOTICE for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
LLNL-CODE-811652