Builders and package classes refer to packages from the builtin package
repo and are often modified together with packages. That means that
these classes should move into `spack_repo.builtin`.
* move `spack.build_systems` -> `spack_repo.builtin.build_systems`
* Remove the following re-exports from the `spack.package` module:
- `AspellDictPackage` - `LuaPackage`
- `AutotoolsPackage` - `MakefilePackage`
- `BundlePackage` - `MavenPackage`
- `CachedCMakePackage` - `MesonPackage`
- `cmake_cache_filepath` - `MSBuildPackage`
- `cmake_cache_option` - `NMakePackage`
- `cmake_cache_path` - `OctavePackage`
- `cmake_cache_string` - `PerlPackage`
- `CargoPackage` - `PythonExtension`
- `CMakePackage` - `PythonPackage`
- `generator` - `QMakePackage`
- `CompilerPackage` - `RacketPackage`
- `CudaPackage` - `RPackage`
- `Package` - `ROCmPackage`
- `GNUMirrorPackage` - `RubyPackage`
- `GoPackage` - `SConsPackage`
- `IntelPackage` - `SIPPackage`
- `IntelOneApiLibraryPackageWithSdk` - `SourceforgePackage`
- `IntelOneApiLibraryPackage` - `SourcewarePackage`
- `IntelOneApiStaticLibraryList` - `WafPackage`
- `IntelOneApiPackage` - `XorgPackage`
- `INTEL_MATH_LIBRARIES`
* update mock packages to repo v2.0 and add copies of packages/build
systems they use from builtin
* add missing imports to build systems in `package.py` from builtin
and test repos
* update various tests
This PR is breaking because of removal of various names from
`spack.package`, but breakage should be minimal thanks to #50496, which
ensures the above names are always imported in repo v1 packages.
Specifically this PR breaks imports like the following in `package.py` files:
```python
from spack.package import Package
```
but if your repo is v1.0 (see `spack repo list`) and has the following
much more common pattern:
```python
from spack.package import *
```
nothing should break.
This PR modifies the parser, so that `%` is parsed as a `DEPENDENCY`, and all
node properties that follow are associated to the name after the `%`. e.g.,
in `foo %gcc +binutils` the `+binutils` refers to `gcc` and not to `foo`.
`%` is still parsed as a build-type dependency, at the moment.
Environments, config files and `package.py` files from before Spack v1.0 may have
spec strings with package variants, targets, etc. *after* a build dependency, and these
will need to be updated. You can use the `spack style --spec-strings` command to do this.
To see what strings will be parsed differently under Spack v1.0, run:
```
spack style --spec-strings FILES
```
where `FILES` is a list of filenames that may contain old specs. To update these spec
strings so that they parse correctly under both Spack 1.0 and Spack 0.x, you can run:
```
spack style --fix --spec-strings FILES
```
In the example above, `foo %gcc +binutils` would be rewritten as `foo +binutils %gcc`,
which parses the same in any Spack version.
In addition, this PR fixes several issues with `%` dependencies:
- [x] Ensure we can still constrain compilers on reuse
- [x] Ensure we can reuse a compiler by hash
- [x] Add tests
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Fixes a bug where `custom_repo.get_pkg_class("foo")` failed executing `import spack_repo.builtin` even if the builtin repo was configured globally.
Instead of assignment of `spack.repo.PATH`, the `spack.repo.enable_repo` function now assigns and enables the repo, meaning that also Python module search paths are modified.
binary_distribution: content addressable url buildcache
Change how binary mirrors are laid out, adopting content addressing for every
piece of data spack stores in a binary mirror. Items (e.g. tarballs, specfiles, public
keys, indices, etc) are now discoverable via manifest files which give the size,
checksum, compression type, etc of the the stored item. The information in the
manifest, in turn, is used to find the actual data, which is stored by its content
address in the blobs directory. Additionally, signing is now applied to the manifest
files, rather than to the spec files themselves.
* Bump the package API of the `builtin` repo to `v2.0`
* Move `var/spack/repos/builtin` -> `var/spack/repos/spack_repo/builtin`
* Move test repos `var/spack/repos/{builtin.mock,tutorial,...}` -> `var/spack/test_repos/`
* Update package dir names to v2 format (`-` -> `_` etc)
* Change absolute imports `from spack.pkg.builtin.my_pkg ...` to relative imports `from ..my_pkg.package ...`
Users who have a repo on top of builtin should change imports from
```python
from spack.pkg.builtin.my_pkg import MyPkg
```
to
```python
from spack_repo.builtin.packages.my_pkg.package import MyPkg
```
and can configure their editors with
```
PYTHONPATH=$spack/lib/spack:$spack/var/spack/repos
```
[skip-verify-checksums]
This PR improves compatibility with specs installed before #45189, and with externals specifying a compiler, by using the annotated compiler to "satisfy" a spec query.
On top of that, the PR adds a new flag for:
```console
$ spack find --specfile-format -I %gcc
-- linux-ubuntu20.04-icelake / gcc@10.5.0 -----------------------
[+] [v4] ca-certificates-mozilla@2023-05-30 [e] [v4] cmake@3.31.6 [+] [v4] gcc-runtime@10.5.0 [e] [v4] glibc@2.31 [+] [v4] gmake@4.4.1 [+] [v4] hdf5@1.14.5 [+] [v4] pkgconf@2.2.0 [+] [v4] zlib-ng@2.2.1
==> 8 installed packages
```
which shows the specfile format of the specs being retrieved.
* geomodel: depend on c
* hep: add geomodel
* hep: geomodel +fullsimlight
* geomodel: depends on virtual gl, not opengl
* soqt: depends on gl and glu instead of opengl
* geomodel: rm generated comments on language dependencies
* TestSuite: add type hints
* spack test run: add a --timeout argument
* pipelines: allow 2 minutes to run tests
* Fix docstrings, increase maximum pipelines time for tests to 5 mins.
* Use SIGTERM first, SIGKILL shortly after
* Add unit-tests for "start_build_process"
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
neoverse_v1 matches the name of the stack and more accurately captures
the requirement for these jobs. The relevant runners in GitLab already
bear both tags, so this shouldn't affect how jobs get assigned to runners.
* e4s oneapi: upgrade to latest compilers oneapi@2025.1
* update specs and package preferences
* enable some more dav packages
* enable additional specs
* e4s oneapi: packages: elfutils does not have bzip2 variant
* e4s oneapi: packages: elfutils does not have xz variant
* e4s oneapi: comment out heffte+sycl
* comment out e4s oneapi failures
* comment out more failures
* comment out failing spec
* py-jaxlib: add spack-built ROCm support
* fix style
* py-jaxlib 0.4.38 rocm support
* py-jaxlib 0.4.38 rocm support
* add comgr dependency
* changes for ROCm external and enable till 0.4.38
* enable version of py-jax
* add jax+rocm to ci
* add conflict for cuda and remove py-jaxlib from aarch64 pipeline
* Update var/spack/repos/builtin/packages/py-jaxlib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* add conflict for aarch64
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
## Summary
Compilers stop being a *node attribute*, and become a *build-only* dependency.
Packages may declare a dependency on the `c`, `cxx`, or `fortran` languages, which
are now treated as virtuals, and compilers would be *providers* for one or more of
those languages. Compilers can also inject runtime dependency, on the node being
compiled. An example graph for something as simple as `zlib-ng` is the following:
<p align="center">
<img src="https://github.com/user-attachments/assets/ee6471cb-09fd-4127-9f16-b9fe6d1338ac" alt="zlib-ng DAG" width="80%" height="auto">
</p>
Here `gcc` is used for both the `c`, and `cxx` languages. Edges are annotated with
the virtuals they satisfy (`c`, `cxx`, `libc`). `gcc` injects `gcc-runtime` on the nodes
being compiled. `glibc` is also injected for packages that require `c`. The
`compiler-wrapper` is explicitly represented as a node in the DAG, and is included in
the hash.
This change in the model has implications on the semantics of the `%` sigil, as
discussed in #44379, and requires a version bump for our `Specfile`, `Database`,
and `Lockfile` formats.
## Breaking changes
Breaking changes below may impact users of this branch.
### 1. Custom, non-numeric version of compilers are not supported
Currently, users can assign to compilers any custom version they want, and Spack
will try to recover the "real version" whenever the custom version fails some operation.
To deduce the "real version" Spack must run the compiler, which can add needless
overhead to common operations.
Since any information that a version like `gcc@foo` might give to the user, can also
be suffixed while retaining the correct numeric version, e.g. `gcc@10.5.0-foo`, Spack
will **not try** anymore to deduce real versions for compilers.
Said otherwise, users should have no expectation that `gcc@foo` behaves as
`gcc@X.Y.Z` internally.
### 2. The `%` sigil in the spec syntax means "direct build dependency"
The `%` sigil in the spec syntax means *"direct build dependency"*, and is not a node
attribute anymore. This means that:
```python
node.satisfies("%gcc")
```
is true only if `gcc` is a direct build dependency of the node. *Nodes without a compiler
dependency are allowed.*
### `parent["child"]`, and `node in spec`, will now only inspect the link/run sub-DAG
and direct build dependencies
The subscript notation for `Spec`:
```python
parent["child"]
```
will look for a `child` node only in the link/run transitive graph of `parent`, and in its
direct build dependencies. This means that to reach a transitive build dependency,
we must first pass through the node it is associated with.
Assuming `parent` does not depend on `cmake`, but depends on a `CMakePackage`,
e.g. `hdf5`, then we have the following situation:
```python
# This one raises an Exception, since "parent" does not depend on cmake
parent["cmake"]
# This one is ok
cmake = parent["hdf5"]["cmake"]
```
### 3. Externals differing by just the compiler attribute
Externals are nodes where dependencies are trimmed, and that _is not planned to
change_ in this branch. Currently, on `develop` it is ok to write:
```yaml
packages:
hdf5:
externals:
- spec: hdf5@1.12 %gcc
prefix: /prefix/gcc
- spec: hdf5@1.12 %clang
prefix: /prefix/clang
```
and Spack will account for the compiler node attribute when computing the optimal
spec. In this branch, using externals with a compiler specified is allowed only if any
compiler in the dag matches the constraints specified on the external. _The external
will be still represented as a single node without dependencies_.
### 4. Spec matrices enforcing a compiler
Currently we can have matrices of the form:
```yaml
matrix:
- [x, y, z]
- [%gcc, %clang]
```
to get the cross-product of specs and compilers. We can disregard the nature of the
packages in the first row, since the compiler is a node attribute required on each node.
In this branch, instead, we require a spec to depend on `c`, `cxx`, or `fortran` for the
`%` to have any meaning. If any of the specs in the first row doesn't depend on these
languages, there will be a concretization error.
## Deprecations
* The entire `compilers` section in the configuration (i.e., `compilers.yaml`) has been
deprecated, and current entries will be removed in v1.2.0. For the time being, if Spack
finds any `compilers` configuration, it will try to convert it automatically to a set of
external packages.
* The `packages:compiler` soft-preference has been deprecated. It will be removed
in v1.1.0.
## Other notable changes
* The tokens `{compiler}`, `{compiler.version}`, and `{compiler.name}` in `Spec.format`
expand to `"none"` if a Spec does not depend on C, C++, or Fortran.
* The default install tree layout is now
`"{architecture.platform}-{architecture.target}/{name}-{version}-{hash}"`
## Known limitations
The major known limitations of this branch that we intend to fix before v1.0 is that compilers
cannot be bootstrapped directly.
In this branch we can build a new compiler using an existing external compiler, for instance:
```
$ spack install gcc@14 %gcc@10.5.0
```
where `gcc@10.5.0` is external, and `gcc@14` is to be built.
What we can't do at the moment is use a yet to be built compiler, and expect it will be
bootstrapped, e.g. :
```
spack install hdf5 %gcc@14
```
We plan to tackle this issue in a following PR.
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Signed-off-by: Harmen Stoppels <me@harmenstoppels.nl>
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
A user had `grep` aliased to `grep -n`, which was causing `csh` setup to
fail due to number prefixes in `SPACK_ROOT`.
- [x] Prefix invocations of `grep` and `sed` (which are not builtin) with `\`
to avoid any aliases.
- [x] Avoid using `dirname` altogether -- use csh's `:h` modifier (which does
the same thing) instead.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
* Add recursive argument to spack develop
This effort allows for a recursive develop call
which will traverse from the develop spec given back to the root(s)
and mark all packages along the path as develop.
If people are doing development across the graph then paying
fetch and full rebuild costs every time spack develop is called
is unnecessary and expensive.
Also remove the constraint for concrete specs and simply take the
max(version) if a version is not given. This should default to the
highest infinity version which is also the logical best guess for
doing development.
Add a CI check to automatically verify the checksums of newly added
package versions:
- [x] a new command, `spack ci verify-versions`
- [x] a GitHub actions check to run the command
- [x] tests for the new command
This also eliminates the suggestion for maintainers to manually verify added
checksums in the case of accidental version <--> checksum mismatches.
----
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
* style.py: add spack style --spec-strings for compat with v1.0
* add --fix also, and avoid infinite recursion and too large files
* tests: check identify and check edit files
This should help resolve the "No binary found when cache-only was specified"
errors we've recently seen in our GitLab CI pipelines.
example failing job here:
https://gitlab.spack.io/spack/spack/-/jobs/15570931#L370
This error is caused when a generate job finds a spec in the local root
binary mirror, and that spec does not yet exist in the stack-specific mirror.
The fix here is to instead locally cache the stack-specific mirrors and only
use the root-level mirror for public use.
When requiring a constraint on a virtual package, it makes little
sense to use anonymous specs, and our documentation shows no example
of requirements on virtual packages starting with `^`.
Right now, due to how `^` is implemented in the solver, writing:
```yaml
mpi:
require: "^openmpi"
```
is equivalent to the more correct form:
```yaml
mpi:
require: "openmpi"
```
but the situation will change when `%` will shift its meaning to be a
direct dependency.
To avoid later errors that are both unclear, and quite slow to get to the user,
this commit makes anonymous specs under virtual requirements an error,
and shows a clear error message pointing to the file and line where the
spec needs to be changed.
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Supersedes #46792.
Closes#40018.
Closes#31026.
Closes#2700.
There were a number of feature requests for os-specific config. This enables os-specific
config without adding a lot of special sub-scopes.
Support `include:` as an independent configuration schema, allowing users to include
configuration scopes from files or directories. Includes can be:
* conditional (similar to definitions in environments), and/or
* optional (i.e., the include will be skipped if it does not exist).
Includes can be paths or URLs (`ftp`, `https`, `http` or `file`). Paths can be absolute or
relative . Environments can include configuration files using the same schema. Remote includes
must be checked by `sha256`.
Includes can also be recursive, and this modifies the config system accordingly so that
we push included configuration scopes on the stack *before* their including scopes, and
we remove configuration scopes from the stack when their including scopes are removed.
For example, you could have an `include.yaml` file (e.g., under `$HOME/.spack`) to specify
global includes:
```
include:
- ./enable_debug.yaml
- path: https://github.com/spack/spack-configs/blob/main/NREL/configs/mac/config.yaml
sha256: 37f982915b03de18cc4e722c42c5267bf04e46b6a6d6e0ef3a67871fcb1d258b
```
Or an environment `spack.yaml`:
```
spack:
include:
- path: "/path/to/a/config-dir-or-file"
when: os == "ventura"
- ./path/relative/to/containing/file/that/is/required
- path: "/path/with/spack/variables/$os/$target"
optional: true
- path: https://raw.githubusercontent.com/spack/spack-configs/refs/heads/main/path/to/required/raw/config.yaml
sha256: 26e871804a92cd07bb3d611b31b4156ae93d35b6a6d6e0ef3a67871fcb1d258b
```
Updated TODO:
- [x] Get existing unit tests to pass with Todd's changes
- [x] Resolve new (or old) circular imports
- [x] Ensure remote includes (global) work
- [x] Ensure remote includes for environments work (note: caches remote
files under user cache root)
- [x] add sha256 field to include paths, validate, and require for remote includes
- [x] add sha256 remote file unit tests
- [x] revisit how diamond includes should work
- [x] support recursive includes
- [x] add recursive include unit tests
- [x] update docs and unit test to indicate ordering of recursive includes with
conflicting options is deferred to follow-on work
---------
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Co-authored-by: Peter Scheibel <scheibel1@llnl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
The SPACK_USER_CACHE_PATH was being overwritten in the windows CI
before_script. This should set the path for all systems unless
explicitly overridden.
* Split requirements to get better error messages in case of unsat solves.
* use list requirements instead of string
* activate static_analysis in a few pipelines
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* ci: darwin stacks: update tags following system updates
* disable SPACK_CI_DISABLE_STACKS; only enable *darwin* stacks for testing
* manually chmod u+w tmp/ before cleanup due to issue#49147
* comment out failing specs for now
* re-enable logic for disabling stacks
* add explanatory comment for darwin after_script additions
* remove more darwin-only targetting
* restore build_stage to default location
* move build-job-remove out of individual darwin stacks into darwin top level config
* keep build_stage in $spack/tmp for now
Currently, the custom config scopes are pushed at the top when constructing
configuration, and are demoted whenever a context manager activating an
environment is used - see #48414 for details. Workflows that rely on the order
in the [docs](https://spack.readthedocs.io/en/latest/configuration.html#custom-scopes)
are thus fragile, and may break
This PR allows to assign priorities to scopes, and ensures that scopes of lower priorities
are always "below" scopes of higher priorities. When scopes have the same priority,
what matters is the insertion order.
Modifications:
- [x] Add a mapping that iterates over keys according to priorities set when
adding the key/value pair
- [x] Use that mapping to allow assigning priorities to configuration scopes
- [x] Assign different priorities for different kind of scopes, to fix a bug, and
add a regression test
- [x] Simplify `Configuration` constructor
- [x] Remove `Configuration.pop_scope`
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
All the build jobs in pipelines are apparently relying on the bug that was fixed.
The issue was not caught in the PR because generation jobs were fine, and
there was nothing to rebuild.
Reverting to fix pipelines in a new PR.
This reverts commit 3ad99d75f9.
