Concretization setup was checking whether any input spec has a dependency
that's *not* in the set of possible dependencies for all roots in the solve.
There are two reasons to check this:
1. The user could be asking for a dependency that none of the roots has, or
2. The user could be asking for a dependency that doesn't exist.
For abstract roots, (2) implies (1), and the check makes sense. For concrete
roots, we don't care, because the spec has already been built. If a `package.py`
no longer depends on something it did before, it doesn't matter -- it's already
built. If the dependency no longer exists, we also do not care -- we already
built it and there's an installation for it somewhere.
When you concretize an environment with a lockfile, *many* of the input specs
are concrete, and we don't need to build them. If a package changes its
dependencies, or if a `package.py` is removed for a concrete input spec, that
shouldn't cause an already-built environment to fail concretization.
A user reported that this was happening with an error like:
```console
spack concretize
==> Error: Package chapel does not depend on py-protobuf@5.28.2/a4rf4glr2tntfwsz6myzwmlk5iu25t74
```
Or, with traceback:
```console
File "/apps/other/spack-devel/lib/spack/spack/solver/asp.py", line 3014, in setup
raise spack.spec.InvalidDependencyError(spec.name, missing_deps)
spack.spec.InvalidDependencyError: Package chapel does not depend on py-protobuf@5.28.2/a4rf4glr2tntfwsz6myzwmlk5iu25t74
```
Fix this by skipping the check for concrete input specs. We already ignore conflicts,
etc. for concrete/external specs, and we do not need metadata in the solve for
concrete dependencies b/c they're imposed by hash constraints.
- [x] Ignore the package existence check for concrete input specs.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
* Skip packages removed for automatic checksum verification
* Unify finding modified or added packages with spack.repo logic
* Remove unused imports
* Fix unit-tests using shared modified function
* Update last remaining unit test to new format
* glab: add v1.54.0 and v1.55.0
* glab: uniform go deps
Co-authored-by: Alec Scott <hi@alecbcs.com>
---------
Co-authored-by: Alec Scott <hi@alecbcs.com>
* py-jaxlib: add spack-built ROCm support
* fix style
* py-jaxlib 0.4.38 rocm support
* py-jaxlib 0.4.38 rocm support
* add comgr dependency
* changes for ROCm external and enable till 0.4.38
* enable version of py-jax
* add jax+rocm to ci
* add conflict for cuda and remove py-jaxlib from aarch64 pipeline
* Update var/spack/repos/builtin/packages/py-jaxlib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* add conflict for aarch64
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
## Summary
Compilers stop being a *node attribute*, and become a *build-only* dependency.
Packages may declare a dependency on the `c`, `cxx`, or `fortran` languages, which
are now treated as virtuals, and compilers would be *providers* for one or more of
those languages. Compilers can also inject runtime dependency, on the node being
compiled. An example graph for something as simple as `zlib-ng` is the following:
<p align="center">
<img src="https://github.com/user-attachments/assets/ee6471cb-09fd-4127-9f16-b9fe6d1338ac" alt="zlib-ng DAG" width="80%" height="auto">
</p>
Here `gcc` is used for both the `c`, and `cxx` languages. Edges are annotated with
the virtuals they satisfy (`c`, `cxx`, `libc`). `gcc` injects `gcc-runtime` on the nodes
being compiled. `glibc` is also injected for packages that require `c`. The
`compiler-wrapper` is explicitly represented as a node in the DAG, and is included in
the hash.
This change in the model has implications on the semantics of the `%` sigil, as
discussed in #44379, and requires a version bump for our `Specfile`, `Database`,
and `Lockfile` formats.
## Breaking changes
Breaking changes below may impact users of this branch.
### 1. Custom, non-numeric version of compilers are not supported
Currently, users can assign to compilers any custom version they want, and Spack
will try to recover the "real version" whenever the custom version fails some operation.
To deduce the "real version" Spack must run the compiler, which can add needless
overhead to common operations.
Since any information that a version like `gcc@foo` might give to the user, can also
be suffixed while retaining the correct numeric version, e.g. `gcc@10.5.0-foo`, Spack
will **not try** anymore to deduce real versions for compilers.
Said otherwise, users should have no expectation that `gcc@foo` behaves as
`gcc@X.Y.Z` internally.
### 2. The `%` sigil in the spec syntax means "direct build dependency"
The `%` sigil in the spec syntax means *"direct build dependency"*, and is not a node
attribute anymore. This means that:
```python
node.satisfies("%gcc")
```
is true only if `gcc` is a direct build dependency of the node. *Nodes without a compiler
dependency are allowed.*
### `parent["child"]`, and `node in spec`, will now only inspect the link/run sub-DAG
and direct build dependencies
The subscript notation for `Spec`:
```python
parent["child"]
```
will look for a `child` node only in the link/run transitive graph of `parent`, and in its
direct build dependencies. This means that to reach a transitive build dependency,
we must first pass through the node it is associated with.
Assuming `parent` does not depend on `cmake`, but depends on a `CMakePackage`,
e.g. `hdf5`, then we have the following situation:
```python
# This one raises an Exception, since "parent" does not depend on cmake
parent["cmake"]
# This one is ok
cmake = parent["hdf5"]["cmake"]
```
### 3. Externals differing by just the compiler attribute
Externals are nodes where dependencies are trimmed, and that _is not planned to
change_ in this branch. Currently, on `develop` it is ok to write:
```yaml
packages:
hdf5:
externals:
- spec: hdf5@1.12 %gcc
prefix: /prefix/gcc
- spec: hdf5@1.12 %clang
prefix: /prefix/clang
```
and Spack will account for the compiler node attribute when computing the optimal
spec. In this branch, using externals with a compiler specified is allowed only if any
compiler in the dag matches the constraints specified on the external. _The external
will be still represented as a single node without dependencies_.
### 4. Spec matrices enforcing a compiler
Currently we can have matrices of the form:
```yaml
matrix:
- [x, y, z]
- [%gcc, %clang]
```
to get the cross-product of specs and compilers. We can disregard the nature of the
packages in the first row, since the compiler is a node attribute required on each node.
In this branch, instead, we require a spec to depend on `c`, `cxx`, or `fortran` for the
`%` to have any meaning. If any of the specs in the first row doesn't depend on these
languages, there will be a concretization error.
## Deprecations
* The entire `compilers` section in the configuration (i.e., `compilers.yaml`) has been
deprecated, and current entries will be removed in v1.2.0. For the time being, if Spack
finds any `compilers` configuration, it will try to convert it automatically to a set of
external packages.
* The `packages:compiler` soft-preference has been deprecated. It will be removed
in v1.1.0.
## Other notable changes
* The tokens `{compiler}`, `{compiler.version}`, and `{compiler.name}` in `Spec.format`
expand to `"none"` if a Spec does not depend on C, C++, or Fortran.
* The default install tree layout is now
`"{architecture.platform}-{architecture.target}/{name}-{version}-{hash}"`
## Known limitations
The major known limitations of this branch that we intend to fix before v1.0 is that compilers
cannot be bootstrapped directly.
In this branch we can build a new compiler using an existing external compiler, for instance:
```
$ spack install gcc@14 %gcc@10.5.0
```
where `gcc@10.5.0` is external, and `gcc@14` is to be built.
What we can't do at the moment is use a yet to be built compiler, and expect it will be
bootstrapped, e.g. :
```
spack install hdf5 %gcc@14
```
We plan to tackle this issue in a following PR.
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Signed-off-by: Harmen Stoppels <me@harmenstoppels.nl>
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
The `umea.se` mirror seems to have gone down (or at least is forbidden for now).
Revert the checksum changes in #47825; points at the official GNOME mirror
instead of the prior two places we were getting `gdk-pixbuf`.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Since we moved from creating clingo symbols directly to constructing a pure string
representation of the program, we don't need to make `AspFunctions` into symbols before
turning them into strings. We can just write strings like clingo would.
This cuts about 25% off the setup time by avoiding an unnecessary round trip.
- [x] create strings directly from `AspFunctions`
- [x] remove unused `symbol()` method on `AspFunction`
- [x] setup no longer tries to call `symbol()`
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Signed-off-by: Greg Becker <becker33@llnl.gov>
---------
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Co-authored-by: Greg Becker <becker33@llnl.gov>
* trilinos: add equals sign to kokkos dependencies.
* Fix some license headers to pass style check.
* Generalize a bit.
* Generalize a bit more.
* datatransferkit: constraing to maximum of trilinos@16.0.
* limit some patches by chapel version
* fix short output version if building main
* update patches, remove unneeded 'self' refs
* fix spack style
* update patches with changes from PR
* change py-protobuf to just protobuf dep
* add PR numbers for patches
* fix spack style
* update 2.4 sha256
A user had `grep` aliased to `grep -n`, which was causing `csh` setup to
fail due to number prefixes in `SPACK_ROOT`.
- [x] Prefix invocations of `grep` and `sed` (which are not builtin) with `\`
to avoid any aliases.
- [x] Avoid using `dirname` altogether -- use csh's `:h` modifier (which does
the same thing) instead.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
* Fix problem at least with datatransferkit
* Include patch 11676 from trilinos
* Add patches for trilinos 13.4.1
* style check failed
* Update links for patches
* additional style check failed
* Add recursive argument to spack develop
This effort allows for a recursive develop call
which will traverse from the develop spec given back to the root(s)
and mark all packages along the path as develop.
If people are doing development across the graph then paying
fetch and full rebuild costs every time spack develop is called
is unnecessary and expensive.
Also remove the constraint for concrete specs and simply take the
max(version) if a version is not given. This should default to the
highest infinity version which is also the logical best guess for
doing development.
* sst-core: fix for > 14.0.0 requiring ncurses
* sst-core: backport fix for curses detection
* sst-core: ensure HDF5 is ignored if not specified
* sst-core: HDF5 integration is via C++
* sst-core: switch to with_or_without for configure
* sst-core: switch to enable_or_disable for configure
* sst-core: control memory pools and debug output with variants
