5b3942a489
## Summary Compilers stop being a *node attribute*, and become a *build-only* dependency. Packages may declare a dependency on the `c`, `cxx`, or `fortran` languages, which are now treated as virtuals, and compilers would be *providers* for one or more of those languages. Compilers can also inject runtime dependency, on the node being compiled. An example graph for something as simple as `zlib-ng` is the following: <p align="center"> <img src="https://github.com/user-attachments/assets/ee6471cb-09fd-4127-9f16-b9fe6d1338ac" alt="zlib-ng DAG" width="80%" height="auto"> </p> Here `gcc` is used for both the `c`, and `cxx` languages. Edges are annotated with the virtuals they satisfy (`c`, `cxx`, `libc`). `gcc` injects `gcc-runtime` on the nodes being compiled. `glibc` is also injected for packages that require `c`. The `compiler-wrapper` is explicitly represented as a node in the DAG, and is included in the hash. This change in the model has implications on the semantics of the `%` sigil, as discussed in #44379, and requires a version bump for our `Specfile`, `Database`, and `Lockfile` formats. ## Breaking changes Breaking changes below may impact users of this branch. ### 1. Custom, non-numeric version of compilers are not supported Currently, users can assign to compilers any custom version they want, and Spack will try to recover the "real version" whenever the custom version fails some operation. To deduce the "real version" Spack must run the compiler, which can add needless overhead to common operations. Since any information that a version like `gcc@foo` might give to the user, can also be suffixed while retaining the correct numeric version, e.g. `gcc@10.5.0-foo`, Spack will **not try** anymore to deduce real versions for compilers. Said otherwise, users should have no expectation that `gcc@foo` behaves as `gcc@X.Y.Z` internally. ### 2. The `%` sigil in the spec syntax means "direct build dependency" The `%` sigil in the spec syntax means *"direct build dependency"*, and is not a node attribute anymore. This means that: ```python node.satisfies("%gcc") ``` is true only if `gcc` is a direct build dependency of the node. *Nodes without a compiler dependency are allowed.* ### `parent["child"]`, and `node in spec`, will now only inspect the link/run sub-DAG and direct build dependencies The subscript notation for `Spec`: ```python parent["child"] ``` will look for a `child` node only in the link/run transitive graph of `parent`, and in its direct build dependencies. This means that to reach a transitive build dependency, we must first pass through the node it is associated with. Assuming `parent` does not depend on `cmake`, but depends on a `CMakePackage`, e.g. `hdf5`, then we have the following situation: ```python # This one raises an Exception, since "parent" does not depend on cmake parent["cmake"] # This one is ok cmake = parent["hdf5"]["cmake"] ``` ### 3. Externals differing by just the compiler attribute Externals are nodes where dependencies are trimmed, and that _is not planned to change_ in this branch. Currently, on `develop` it is ok to write: ```yaml packages: hdf5: externals: - spec: hdf5@1.12 %gcc prefix: /prefix/gcc - spec: hdf5@1.12 %clang prefix: /prefix/clang ``` and Spack will account for the compiler node attribute when computing the optimal spec. In this branch, using externals with a compiler specified is allowed only if any compiler in the dag matches the constraints specified on the external. _The external will be still represented as a single node without dependencies_. ### 4. Spec matrices enforcing a compiler Currently we can have matrices of the form: ```yaml matrix: - [x, y, z] - [%gcc, %clang] ``` to get the cross-product of specs and compilers. We can disregard the nature of the packages in the first row, since the compiler is a node attribute required on each node. In this branch, instead, we require a spec to depend on `c`, `cxx`, or `fortran` for the `%` to have any meaning. If any of the specs in the first row doesn't depend on these languages, there will be a concretization error. ## Deprecations * The entire `compilers` section in the configuration (i.e., `compilers.yaml`) has been deprecated, and current entries will be removed in v1.2.0. For the time being, if Spack finds any `compilers` configuration, it will try to convert it automatically to a set of external packages. * The `packages:compiler` soft-preference has been deprecated. It will be removed in v1.1.0. ## Other notable changes * The tokens `{compiler}`, `{compiler.version}`, and `{compiler.name}` in `Spec.format` expand to `"none"` if a Spec does not depend on C, C++, or Fortran. * The default install tree layout is now `"{architecture.platform}-{architecture.target}/{name}-{version}-{hash}"` ## Known limitations The major known limitations of this branch that we intend to fix before v1.0 is that compilers cannot be bootstrapped directly. In this branch we can build a new compiler using an existing external compiler, for instance: ``` $ spack install gcc@14 %gcc@10.5.0 ``` where `gcc@10.5.0` is external, and `gcc@14` is to be built. What we can't do at the moment is use a yet to be built compiler, and expect it will be bootstrapped, e.g. : ``` spack install hdf5 %gcc@14 ``` We plan to tackle this issue in a following PR. --------- Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> Signed-off-by: Todd Gamblin <tgamblin@llnl.gov> Signed-off-by: Harmen Stoppels <me@harmenstoppels.nl> Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl> Co-authored-by: Todd Gamblin <tgamblin@llnl.gov> |
||
|---|---|---|
| .devcontainer | ||
| .github | ||
| bin | ||
| etc/spack/defaults | ||
| lib/spack | ||
| share/spack | ||
| var/spack | ||
| .codecov.yml | ||
| .dockerignore | ||
| .flake8 | ||
| .git-blame-ignore-revs | ||
| .gitattributes | ||
| .gitignore | ||
| .mailmap | ||
| .readthedocs.yml | ||
| CHANGELOG.md | ||
| CITATION.cff | ||
| COPYRIGHT | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| NOTICE | ||
| pyproject.toml | ||
| pytest.ini | ||
| README.md | ||
| SECURITY.md | ||
Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, Windows, and many supercomputers. Spack is non-destructive: installing a new version of a package does not break existing installations, so many configurations of the same package can coexist.
Spack offers a simple "spec" syntax that allows users to specify versions and configuration options. Package files are written in pure Python, and specs allow package authors to write a single script for many different builds of the same package. With Spack, you can build your software all the ways you want to.
See the Feature Overview for examples and highlights.
To install spack and your first package, make sure you have Python & Git. Then:
$ git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git
$ cd spack/bin
$ ./spack install zlib
Tip
-c feature.manyFiles=trueimproves git's performance on repositories with 1,000+ files.
--depth=2prunes the git history to reduce the size of the Spack installation.
Documentation
Full documentation is available, or
run spack help or spack help --all.
For a cheat sheet on Spack syntax, run spack help --spec.
Tutorial
We maintain a hands-on tutorial. It covers basic to advanced usage, packaging, developer features, and large HPC deployments. You can do all of the exercises on your own laptop using a Docker container.
Feel free to use these materials to teach users at your organization about Spack.
Community
Spack is an open source project. Questions, discussion, and contributions are welcome. Contributions can be anything from new packages to bugfixes, documentation, or even new core features.
Resources:
- Slack workspace: spackpm.slack.com. To get an invitation, visit slack.spack.io.
- Matrix space: #spack-space:matrix.org: bridged to Slack.
- Github Discussions: for Q&A and discussions. Note the pinned discussions for announcements.
- X: @spackpm. Be sure to
@mentionus! - Mailing list: groups.google.com/d/forum/spack: only for announcements. Please use other venues for discussions.
Contributing
Contributing to Spack is relatively easy. Just send us a
pull request.
When you send your request, make develop the destination branch on the
Spack repository.
Your PR must pass Spack's unit tests and documentation tests, and must be PEP 8 compliant. We enforce these guidelines with our CI process. To run these tests locally, and for helpful tips on git, see our Contribution Guide.
Spack's develop branch has the latest contributions. Pull requests
should target develop, and users who want the latest package versions,
features, etc. can use develop.
Releases
For multi-user site deployments or other use cases that need very stable software installations, we recommend using Spack's stable releases.
Each Spack release series also has a corresponding branch, e.g.
releases/v0.14 has 0.14.x versions of Spack, and releases/v0.13 has
0.13.x versions. We backport important bug fixes to these branches but
we do not advance the package versions or make other changes that would
change the way Spack concretizes dependencies within a release branch.
So, you can base your Spack deployment on a release branch and git pull
to get fixes, without the package churn that comes with develop.
The latest release is always available with the releases/latest tag.
See the docs on releases for more details.
Code of Conduct
Please note that Spack has a Code of Conduct. By participating in the Spack community, you agree to abide by its rules.
Authors
Many thanks go to Spack's contributors.
Spack was created by Todd Gamblin, tgamblin@llnl.gov.
Citing Spack
If you are referencing Spack in a publication, please cite the following paper:
- Todd Gamblin, Matthew P. LeGendre, Michael R. Collette, Gregory L. Lee, Adam Moody, Bronis R. de Supinski, and W. Scott Futral. The Spack Package Manager: Bringing Order to HPC Software Chaos. In Supercomputing 2015 (SC’15), Austin, Texas, November 15-20 2015. LLNL-CONF-669890.
On GitHub, you can copy this citation in APA or BibTeX format via the "Cite this repository"
button. Or, see the comments in CITATION.cff for the raw BibTeX.
License
Spack is distributed under the terms of both the MIT license and the Apache License (Version 2.0). Users may choose either license, at their option.
All new contributions must be made under both the MIT and Apache-2.0 licenses.
See LICENSE-MIT, LICENSE-APACHE, COPYRIGHT, and NOTICE for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
LLNL-CODE-811652