The solves now supports key:=val syntax for multivalued variants in specs originating from input, externals, requirements, directives and when conditions
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
This PR is a step towards treating externals as concrete specs. Specifically, it moves the optimization weights of external nodes into the group of "reused" specs, and doesn't count externals as specs to be built.
It still keeps the one to many mapping between an external spec in `packages.yaml` and the corresponding specs in the DB. To make it such that an hashed external is preferred to a non-hashed one, the version weights of externals are demoted at lowest priority.
**Change in behavior**:
- Having the possibility, Spack will now prefer to mix compilers in a DAG, and use the latest version possible for each node, rather than using a single compiler and employ old versions of some nodes because of conflicts
- In general, using externals by default is now triggered by putting their weights in the "reused" group. This means that any penalty they might induce will never have the same priority as something that needs to be built. This behavior reflects reality, but changes some default choices from the previous state.
Modifications:
- External nodes weights are now moved to the group of "reused" specs
- Runtimes are treated as concrete as well (to avoid that e.g.`gcc` is not selected because it "builds" the runtime package)
- Shuffle version weights, so that externals are least preferred (counterbalanced by the fact that they are in the "reused" groups)
- Split provider weights on edges from version badness on edges
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
When indexing top level specs, eg, in s3://spack-binaries/develop,
do not sync manifests from all the stacks. Instead, add the path
to the spec manifests to the url to sync, so that only items in
s3://spack-binaries/develop/v3/manifests/spec are copied to the
local system.
Allow rebuild-index to continue if fetching some specs fails
for any reason, and issue a warning indicating which manifest
is associated with the failed fetch.
Fixes a bug where `custom_repo.get_pkg_class("foo")` failed executing `import spack_repo.builtin` even if the builtin repo was configured globally.
Instead of assignment of `spack.repo.PATH`, the `spack.repo.enable_repo` function now assigns and enables the repo, meaning that also Python module search paths are modified.
binary_distribution: content addressable url buildcache
Change how binary mirrors are laid out, adopting content addressing for every
piece of data spack stores in a binary mirror. Items (e.g. tarballs, specfiles, public
keys, indices, etc) are now discoverable via manifest files which give the size,
checksum, compression type, etc of the the stored item. The information in the
manifest, in turn, is used to find the actual data, which is stored by its content
address in the blobs directory. Additionally, signing is now applied to the manifest
files, rather than to the spec files themselves.
* Bump the package API of the `builtin` repo to `v2.0`
* Move `var/spack/repos/builtin` -> `var/spack/repos/spack_repo/builtin`
* Move test repos `var/spack/repos/{builtin.mock,tutorial,...}` -> `var/spack/test_repos/`
* Update package dir names to v2 format (`-` -> `_` etc)
* Change absolute imports `from spack.pkg.builtin.my_pkg ...` to relative imports `from ..my_pkg.package ...`
Users who have a repo on top of builtin should change imports from
```python
from spack.pkg.builtin.my_pkg import MyPkg
```
to
```python
from spack_repo.builtin.packages.my_pkg.package import MyPkg
```
and can configure their editors with
```
PYTHONPATH=$spack/lib/spack:$spack/var/spack/repos
```
[skip-verify-checksums]
This implements Package API v2.0, and is an opt-in feature for repos. It can be enabled with
```yaml
repo:
...
api: v2.0
```
It differs from the current default v1.0 as follows:
1. Package names can only contain `-` as a separator.
2. Package names can only be lowercase.
3. Package directory names are valid Python module names.
4. The repo namespace and its directory name are the same.
5. The `packages` subdir, which is configurable, should be a directory
name that is also a valid Python module name.
6. There is a one to one mapping between Spack package names and Python
module names.
7. Import statements `import spack.pkg.namespace.package_module` in
`package.py` files need to specify the canonical package module.
To go from Spack package name to Python module name:
- Replace `-` by `_`
- Add a leading `_` if the package name starts with a digit
To go from Python module name to Spack package name:
- Strip leading `_`
- Replace `_` by `-`.
This PR improves compatibility with specs installed before #45189, and with externals specifying a compiler, by using the annotated compiler to "satisfy" a spec query.
On top of that, the PR adds a new flag for:
```console
$ spack find --specfile-format -I %gcc
-- linux-ubuntu20.04-icelake / gcc@10.5.0 -----------------------
[+] [v4] ca-certificates-mozilla@2023-05-30 [e] [v4] cmake@3.31.6 [+] [v4] gcc-runtime@10.5.0 [e] [v4] glibc@2.31 [+] [v4] gmake@4.4.1 [+] [v4] hdf5@1.14.5 [+] [v4] pkgconf@2.2.0 [+] [v4] zlib-ng@2.2.1
==> 8 installed packages
```
which shows the specfile format of the specs being retrieved.
Fixes a logic bug where a -> b was assumed to imply not a -> not b
in conditional requirements.
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
When trying to use Spack to build Intel TBB on the EOS network file
system we use at CERN, I am facing the following error:
```
Launching the installer...
Installation directory is not empty.
The product cannot be installed into nonempty directory
'/eos/home-s/sswatman/spack/opt/spack/linux-x86_64_v2/...
intel-oneapi-tbb-2021.12.0-3jlx6hlr3z6di42f3qy35eizcse7u2tk'.
```
This error appears to happen because Spack tries to set `$HOME` to the
prefix of the package, into which the Intel installer is also trying to
install it's software. I've found that an easy fix is to set `$HOME` to
a directory in the build stage instead, as this can be guaranteed to
remain empty.
