- [x] don't allow calling `spec.package_hash()` on abstract specs
- [x] get rid of last few uses of `ht.package_hash`, which was removed.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
The submodules argument for git versions supports callbacks (which are used on some key
packages to conditionally fetch certain submodules). Callbacks can't be serialized to
JSON, so we need to ensure that these things are resolved (by calling them) at
concretization time.
- [x] ensure that submodule callbacks are called when creating spec JSON
- [x] add tests that try serializing all types of git versions
`installer.py` currently swallows the traceback and preserves only
the error messaege if a build process fails.
- [x] preserve exceptions from failed build processes
- [x] print a full traceback for each one when running with `spack -d`
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
We've included a package hash in Spack since #7193 for CI, and we started using it on
the spec in #28504. However, what goes into the package hash is a bit opaque. Here's
what `spec.json` looks like now:
```json
{
"spec": {
"_meta": {
"version": 3
},
"nodes": [
{
"name": "zlib",
"version": "1.2.12",
...
"patches": [
"0d38234384870bfd34dfcb738a9083952656f0c766a0f5990b1893076b084b76"
],
"package_hash": "pthf7iophdyonixxeed7gyqiksopxeklzzjbxtjrw7nzlkcqleba====",
"hash": "ke4alug7ypoxp37jb6namwlxssmws4kp"
}
]
}
}
```
The `package_hash` there is a hash of the concatenation of:
* A canonical hash of the `package.py` recipe, as implemented in #28156;
* `sha256`'s of patches applied to the spec; and
* Archive `sha256` sums of archives or commits/revisions of repos used to build the spec.
There are some issues with this: patches are counted twice in this spec (in `patches`
and in the `package_hash`), the hashes of sources used to build are conflated with the
`package.py` hash, and we don't actually include resources anywhere.
With this PR, I've expanded the package hash out in the `spec.json` body. Here is the
"same" spec with the new fields:
```json
{
"spec": {
"_meta": {
"version": 3
},
"nodes": [
{
"name": "zlib",
"version": "1.2.12",
...
"package_hash": "6kkliqdv67ucuvfpfdwaacy5bz6s6en4",
"sources": [
{
"type": "archive",
"sha256": "91844808532e5ce316b3c010929493c0244f3d37593afd6de04f71821d5136d9"
}
],
"patches": [
"0d38234384870bfd34dfcb738a9083952656f0c766a0f5990b1893076b084b76"
],
"hash": "ts3gkpltbgzr5y6nrfy6rzwbjmkscein"
}
]
}
}
```
Now:
* Patches and archive hashes are no longer included in the `package_hash`;
* Artifacts used in the build go in `sources`, and we tell you their checksum in the `spec.json`;
* `sources` will include resources for packages that have it;
* Patches are the same as before -- but only represented once; and
* The `package_hash` is a base32-encoded `sha1`, like other hashes in Spack, and it only
tells you that the `package.py` changed.
The behavior of the DAG hash (which includes the `package_hash`) is basically the same
as before, except now resources are included, and we can see differences in archives and
resources directly in the `spec.json`
Note that we do not need to bump the spec meta version on this, as past versions of
Spack can still read the new specs; they just will not notice the new fields (which is
fine, since we currently do not do anything with them).
Among other things, this will more easily allow us to convert Spack specs to SBOM and
track relevant security information (like `sha256`'s of archives). For example, we could
do continuous scanning of a Spack installation based on these hashes, and if the
`sha256`'s become associated with CVE's, we'll know we're affected.
- [x] Add a method, `spec_attrs()` to `FetchStrategy` that can be used to describe a
fetcher for a `spec.json`.
- [x] Simplify the way package_hash() is handled in Spack. Previously, it was handled as
a special-case spec hash in `hash_types.py`, but it really doesn't belong there.
Now, it's handled as part of `Spec._finalize_concretization()` and `hash_types.py`
is much simpler.
- [x] Change `PackageBase.content_hash()` to `PackageBase.artifact_hashes()`, and
include more information about artifacts in it.
- [x] Update package hash tests and make them check for artifact and resource hashes.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Originally, concretization failed if the splice config points to an invalid replacement.
This PR defers the check until we know the splice is needed, so that irrelevant splices
with bad config cannot stop concretization.
While I was at it, I improved an error message from an assert to a ValueError.
* Normalize Spack Win entrypoints
Currently Spack has multiple entrypoints on Windows that in addition to
differing from *nix implementations, differ from shell to shell on
Windows. This is a bit confusing for new users and in general
unnecessary.
This PR adds a normal setup script for the batch shell while preserving
the previous "click from file explorer for spack shell" behavior.
Additionally adds a shell title to both powershell and cmd letting users
know this is a Spack shell
* remove doskeys
#44588 we added logic to suppress deprecation warnings for the
Intel classic compilers. This depended on matching against
* The compiler names (looking for icc, icpc, ifort)
* The compiler version
When using an Intel compiler with fortran wrappers, the first check
always fails. To support using the fortran wrappers (in combination
with the classic Intel compilers), we remove the first check and
suppress if just the version matches. This works because:
* The newer compilers like icx can handle (ignore) the flags that
suppress deprecation warnings
* The Cray wrappers pass the underlying compiler version (e.g. they
report what icc would report)
* new openjdk variant to symlink system certificate
* new openjdk variant to symlink system certificate
* new openjdk variant to symlink system certificate
* new openjdk variant to symlink system certificate
* Update var/spack/repos/builtin/packages/openjdk/package.py
Co-authored-by: Alec Scott <hi@alecbcs.com>
---------
Co-authored-by: Alec Scott <hi@alecbcs.com>
Connection objects are Python version, platform and multiprocessing
start method independent, so better to use those than a mix of plain
file descriptors and inadequate guesses in the child process whether it
was forked or not.
This also allows us to delete the now redundant MultiProcessFd class,
hopefully making things a bit easier to follow.
This allows the following
```python
cache.init_entry("my/cache")
with cache.read_transaction("my/cache") as f:
data = f.read() if f is not None else None
```
mirroring `write_transaction`, which returns a tuple `(old, new)` where
`old` is `None` if the cache file did not exist yet.
The alternative that requires less defensive programming on the call
site would be to create the "old" file upon first read, but I did not
want to think about how to safely atomically create the file, and it's
not unthinkable that an empty file is an invalid format (for instance
the call site may expect a json file, which requires at least {} bytes).
This PR is in response to a question in the `environments` slack channel (https://spackpm.slack.com/archives/CMHK7MF51/p1729200068557219) about inadequate CLI help/documentation for one specific subcommand.
This PR uses the approach I took for the descriptions and help for `spack test` subcommands. Namely, I use the first line of the relevant docstring as the description, which is shown per subcommand in `spack env -h`, and the entire docstring as the help. I then added, where it seemed appropriate, help. I also tweaked argument docstrings to tighten them up, make consistent with similar arguments elsewhere in the command, and elaborate when it seemed important. (The only subcommand I didn't touch is `loads`.)
For example, before:
```
$ spack env update -h
usage: spack env update [-hy] env
positional arguments:
env name or directory of the environment to activate
optional arguments:
-h, --help show this help message and exit
-y, --yes-to-all assume "yes" is the answer to every confirmation request
```
After the changes in this PR:
```
$ spack env update -h
usage: spack env update [-hy] env
update the environment manifest to the latest schema format
update the environment to the latest schema format, which may not be
readable by older versions of spack
a backup copy of the manifest is retained in case there is a need to revert
this operation
positional arguments:
env name or directory of the environment
optional arguments:
-h, --help show this help message and exit
-y, --yes-to-all assume "yes" is the answer to every confirmation request
```
---------
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
