zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

specs: move to new spec.json format with build provenance (#22845)

Browse Source 
This is a major rework of Spack's core core `spec.yaml` metadata format.  It moves from `spec.yaml` to `spec.json` for speed, and it changes the format in several ways. Specifically:

1. The spec format now has a `_meta` section with a version (now set to version `2`).  This will simplify major changes like this one in the future.
2. The node list in spec dictionaries is no longer keyed by name. Instead, it is a list of records with no required key. The name, hash, etc. are fields in the dictionary records like any other.
3. Dependencies can be keyed by any hash (`hash`, `full_hash`, `build_hash`).
4. `build_spec` provenance from #20262 is included in the spec format. This means that, for spliced specs, we preserve the *full* provenance of how to build, and we can reproduce a spliced spec from the original builds that produced it.

**NOTE**: Because we have switched the spec format, this PR changes Spack's hashing algorithm.  This means that after this commit, Spack will think a lot of things need rebuilds.

There are two major benefits this PR provides:
* The switch to JSON format speeds up Spack significantly, as Python's builtin JSON implementation is orders of magnitude faster than YAML. 
* The new Spec format will soon allow us to represent DAGs with potentially multiple versions of the same dependency -- e.g., for build dependencies or for compilers-as-dependencies.  This PR lays the necessary groundwork for those features.

The old `spec.yaml` format continues to be supported, but is now considered a legacy format, and Spack will opportunistically convert these to the new `spec.json` format.
This commit is contained in:
Nathan Hanford
2021-09-09 01:48:30 -07:00
committed by GitHub
parent edb1d75b1b
commit d83f7110d5
37 changed files with 1357 additions and 768 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
share/spack/spack-completion.bash
View File

@@ -491,14 +491,14 @@ _spack_buildcache() {
then
SPACK_COMPREPLY="-h --help"
else
SPACK_COMPREPLY="create install list keys preview check download get-buildcache-name save-yaml copy sync update-index"
SPACK_COMPREPLY="create install list keys preview check download get-buildcache-name save-specfile copy sync update-index"
fi
}
_spack_buildcache_create() {
if $list_options
then
SPACK_COMPREPLY="-h --help -r --rel -f --force -u --unsigned -a --allow-root -k --key -d --directory -m --mirror-name --mirror-url --rebuild-index -y --spec-yaml --only"
SPACK_COMPREPLY="-h --help -r --rel -f --force -u --unsigned -a --allow-root -k --key -d --directory -m --mirror-name --mirror-url --rebuild-index --spec-file --only"
else
_all_packages
fi
@@ -536,23 +536,23 @@ _spack_buildcache_preview() {
}
_spack_buildcache_check() {
SPACK_COMPREPLY="-h --help -m --mirror-url -o --output-file --scope -s --spec -y --spec-yaml --rebuild-on-error"
SPACK_COMPREPLY="-h --help -m --mirror-url -o --output-file --scope -s --spec --spec-file --rebuild-on-error"
}
_spack_buildcache_download() {
SPACK_COMPREPLY="-h --help -s --spec -y --spec-yaml -p --path -c --require-cdashid"
SPACK_COMPREPLY="-h --help -s --spec --spec-file -p --path -c --require-cdashid"
}
_spack_buildcache_get_buildcache_name() {
SPACK_COMPREPLY="-h --help -s --spec -y --spec-yaml"
SPACK_COMPREPLY="-h --help -s --spec --spec-file"
}
_spack_buildcache_save_yaml() {
SPACK_COMPREPLY="-h --help --root-spec --root-spec-yaml -s --specs -y --yaml-dir"
_spack_buildcache_save_specfile() {
SPACK_COMPREPLY="-h --help --root-spec --root-specfile -s --specs --specfile-dir"
}
_spack_buildcache_copy() {
SPACK_COMPREPLY="-h --help --base-dir --spec-yaml --destination-url"
SPACK_COMPREPLY="-h --help --base-dir --spec-file --destination-url"
}
_spack_buildcache_sync() {