binary_distribution: content addressable url buildcache
Change how binary mirrors are laid out, adopting content addressing for every
piece of data spack stores in a binary mirror. Items (e.g. tarballs, specfiles, public
keys, indices, etc) are now discoverable via manifest files which give the size,
checksum, compression type, etc of the the stored item. The information in the
manifest, in turn, is used to find the actual data, which is stored by its content
address in the blobs directory. Additionally, signing is now applied to the manifest
files, rather than to the spec files themselves.
* Bump the package API of the `builtin` repo to `v2.0`
* Move `var/spack/repos/builtin` -> `var/spack/repos/spack_repo/builtin`
* Move test repos `var/spack/repos/{builtin.mock,tutorial,...}` -> `var/spack/test_repos/`
* Update package dir names to v2 format (`-` -> `_` etc)
* Change absolute imports `from spack.pkg.builtin.my_pkg ...` to relative imports `from ..my_pkg.package ...`
Users who have a repo on top of builtin should change imports from
```python
from spack.pkg.builtin.my_pkg import MyPkg
```
to
```python
from spack_repo.builtin.packages.my_pkg.package import MyPkg
```
and can configure their editors with
```
PYTHONPATH=$spack/lib/spack:$spack/var/spack/repos
```
[skip-verify-checksums]
This implements Package API v2.0, and is an opt-in feature for repos. It can be enabled with
```yaml
repo:
...
api: v2.0
```
It differs from the current default v1.0 as follows:
1. Package names can only contain `-` as a separator.
2. Package names can only be lowercase.
3. Package directory names are valid Python module names.
4. The repo namespace and its directory name are the same.
5. The `packages` subdir, which is configurable, should be a directory
name that is also a valid Python module name.
6. There is a one to one mapping between Spack package names and Python
module names.
7. Import statements `import spack.pkg.namespace.package_module` in
`package.py` files need to specify the canonical package module.
To go from Spack package name to Python module name:
- Replace `-` by `_`
- Add a leading `_` if the package name starts with a digit
To go from Python module name to Spack package name:
- Strip leading `_`
- Replace `_` by `-`.
This PR improves compatibility with specs installed before #45189, and with externals specifying a compiler, by using the annotated compiler to "satisfy" a spec query.
On top of that, the PR adds a new flag for:
```console
$ spack find --specfile-format -I %gcc
-- linux-ubuntu20.04-icelake / gcc@10.5.0 -----------------------
[+] [v4] ca-certificates-mozilla@2023-05-30 [e] [v4] cmake@3.31.6 [+] [v4] gcc-runtime@10.5.0 [e] [v4] glibc@2.31 [+] [v4] gmake@4.4.1 [+] [v4] hdf5@1.14.5 [+] [v4] pkgconf@2.2.0 [+] [v4] zlib-ng@2.2.1
==> 8 installed packages
```
which shows the specfile format of the specs being retrieved.
Fixes a logic bug where a -> b was assumed to imply not a -> not b
in conditional requirements.
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
When trying to use Spack to build Intel TBB on the EOS network file
system we use at CERN, I am facing the following error:
```
Launching the installer...
Installation directory is not empty.
The product cannot be installed into nonempty directory
'/eos/home-s/sswatman/spack/opt/spack/linux-x86_64_v2/...
intel-oneapi-tbb-2021.12.0-3jlx6hlr3z6di42f3qy35eizcse7u2tk'.
```
This error appears to happen because Spack tries to set `$HOME` to the
prefix of the package, into which the Intel installer is also trying to
install it's software. I've found that an easy fix is to set `$HOME` to
a directory in the build stage instead, as this can be guaranteed to
remain empty.
* `x=*` constrained by `+x` now produces a boolean valued variant instead of a multi-valued variant.
* Values are now always stored as a tuple internally, whether bool, single or multi-valued.
* Value assignment has a stricter api to prevent ambiguity / type issues related to
`variant.value = "x"` / `variant.value = ["x"]` / `variant.value = ("x",)`. It's now `variant.set("x", ...)` for
single and multi-valued variants.
* The `_original_value` prop is dropped, since it was unused.
* The wildcard `*` is no longer a possible variant value in any type of variant, since the *parser*
deals with it and creates a variant with no values.
The second change technically affects non-Windows, but the
behavior should be exactly the same:
* Packages no longer have access to `.msbuild` and `.nmake`
automatically; they now get them via a dependency on `msvc`.
* Update two CMake-based packages that call `make test` to
instead call `ctest` (`netcdf-cxx4` and `pegtl`).
CMake-based packages should do this because on Windows
`make test` will not generally work, but `ctest` does.
* Fix `openssl` "make test" on Windows (WRT prior point: not
a CMake-based package).
Show progress meter for fetches when `stdout` is a `tty`.
* fetch_strategy.py: show progress
* "Fetched: x MB at y MB/s"
* add tests, show % if content-length
This PR fixes the issues with `%` and reused specs, due to https://github.com/spack/spack/issues/49847#issuecomment-2774640234
It does so by adding another layer of indirection, so that whenever a spec
`foo %bar` is encountered, the `%bar` part is encoded as an
`attr("build_requirement", ...)`.
Then:
1. If `foo` is a node to be built, then the build requirement implies a dependency
2. Otherwise it implies looking e.g. reused specs metadata, and ensure it matches
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
When the solver produces specs that do not satisfy the input
constraints, dump both input and output specs as json in an temporary
dir and ask the user to upload these files in a bug report.
