This feature generates a verification manifest for each installed
package and provides a command, "spack verify", which can be used to
compare the current file checksums/permissions with those calculated
at installed time.
Verification includes
* Checksums of files
* File permissions
* Modification time
* File size
Packages installed before this PR will be skipped during verification.
To verify such a package you must reinstall it.
The spack verify command has three modes.
* With the -a,--all option it will check every installed package.
* With the -f,--files option, it will check some specific files,
determine which package they belong to, and confirm that they have
not been changed.
* With the -s,--specs option or by default, it will check some
specific packages that no files havae changed.
This PR adds a 'concretize' entry to an environment's spec.yaml file
which controls how user specs are concretized. By default it is
set to 'separately' which means that each spec added by the user is
concretized separately (the behavior of environments before this PR).
If set to 'together', the environment will concretize all of the
added user specs together; this means that all specs and their
dependencies will be consistent with each other (for example, a
user could develop code linked against the set of libraries in the
environment without conflicts).
If the environment was previously concretized, this will re-concretize
all specs, in which case previously-installed specs may no longer be
used by the environment (in this sense, adding a new spec to an
environment with 'concretize: together' can be significantly more
expensive).
The 'concretize: together' setting is not compatible with Spec
matrices; this PR adds a check to look for multiple instances of the
same package added to the environment and fails early when
'concretize: together' is set (to avoid confusing messages about
conflicts later on).
While the build environment already takes share/pkgconfig into account,
the generated module files etc. only consider lib/pkgconfig and
lib64/pkgconfig.
Dotkit is being used only at a few sites and has been deprecated on new
machines. This commit removes all the code that provide support for the
generation of dotkit module files.
A new validator named "deprecatedProperties" has been added to the
jsonschema validators. It permits to prompt a warning message or exit
with an error if a property that has been marked as deprecated is
encountered.
* Removed references to dotkit in the docs
* Removed references to dotkit in setup-env-test.sh
* Added a unit test for the 'deprecatedProperties' schema validator
Preferred targets were failing because we were looking them up by
Microarchitecture object, not by string.
- [x] Add a call to `str()` to fix target lookup.
- [x] Add a test to exercise this part of concretization.
- [x] Add documentation for setting `target` in `packages.yaml`
Seamless translation from 'target=<generic>' to either
- target.family == <generic> (in methods)
- 'target=<generic>:' (in directives)
Also updated docs to show ranges in directives.
Spack can now:
- label ppc64, ppc64le, x86_64, etc. builds with specific
microarchitecture-specific names, like 'haswell', 'skylake' or
'icelake'.
- detect the host architecture of a machine from /proc/cpuinfo or similar
tools.
- Understand which microarchitectures are compatible with which (for
binary reuse)
- Understand which compiler flags are needed (for GCC, so far) to build
binaries for particular microarchitectures.
All of this is managed through a JSON file (microarchitectures.json) that
contains detailed auto-detection, compiler flag, and compatibility
information for specific microarchitecture targets. The `llnl.util.cpu`
module implements a library that allows detection and comparison of
microarchitectures based on the data in this file.
The `target` part of Spack specs is now essentially a Microarchitecture
object, and Specs' targets can be compared for compatibility as well.
This allows us to label optimized binary packages at a granularity that
enables them to be reused on compatible machines. Previously, we only
knew that a package was built for x86_64, NOT which x86_64 machines it
was usable on.
Currently this feature supports Intel, Power, and AMD chips. Support for
ARM is forthcoming.
Specifics:
- Add microarchitectures.json with descriptions of architectures
- Relaxed semantic of compiler's "target" attribute. Before this change
the semantic to check if a compiler could be viable for a given target
was exact match. This made sense as the finest granularity of targets
was architecture families. As now we can target micro-architectures,
this commit changes the semantic by interpreting as the architecture
family what is stored in the compiler's "target" attribute. A compiler
is then a viable choice if the target being concretized belongs to the
same family. Similarly when a new compiler is detected the architecture
family is stored in the "target" attribute.
- Make Spack's `cc` compiler wrapper inject target-specific flags on the
command line
- Architecture concretization updated to use the same algorithm as
compiler concretization
- Micro-architecture features, vendor, generation etc. are included in
the package hash. Generic architectures, such as x86_64 or ppc64, are
still dumped using the name only.
- If the compiler for a target is not supported exit with an intelligible
error message. If the compiler support is unknown don't try to use
optimization flags.
- Support and define feature aliases (e.g., sse3 -> ssse3) in
microarchitectures.json and on Microarchitecture objects. Feature
aliases are defined in targets.json and map a name (the "alias") to a
list of rules that must be met for the test to be successful. The rules
that are available can be extended later using a decorator.
- Implement subset semantics for comparing microarchitectures (treat
microarchitectures as a partial order, i.e. (a < b), (a == b) and (b <
a) can all be false.
- Implement logic to automatically demote the default target if the
compiler being used is too old to optimize for it. Updated docs to make
this behavior explicit. This avoids surprising the user if the default
compiler is older than the host architecture.
This commit adds unit tests to verify the semantics of target ranges and
target lists in constraints. The implementation to allow target ranges
and lists is minimal and doesn't add any new type. A more careful
refactor that takes into account the type system might be due later.
Co-authored-by: Gregory Becker <becker33.llnl.gov>
* When cleaning the stage root, only remove directories that appear
to be used for staging Spack packages. Previously Spack was clearing
all directories in the stage root, which could remove content not
related to Spack if the user chose a staging root which contains
files/directories not managed by Spack.
* The documentation is updated with warnings about choosing a stage
directory that is only managed by Spack (although generally the
check added in this PR for "spack clean" should avoid removing
content that was not created by Spack)
* The default stage directory (in config.yaml) is now
$tempdir/$user/spack-stage and the logic is updated to omit the
$user portion of this path if $tempdir already contains a $user
directory.
* When creating stage root assign user read/write permissions to all
directories in the path under $user. Previously Spack was assigning
the permissions of the first existing parent directory
* All fetch strategies now accept the Boolean version keyword option `no_cache` in order to allow per-version control of cache-ability.
* New git-specific version keyword option `get_full_repo` (Boolean). When true, disables the default `--depth 1` and `--single-branch` optimizations that are applied if supported by the git version and (in the former case) transport protocol.
* The try / catch blog attempting `--depth 1` and retrying on failure has been removed in favor of more accurately ascertaining when the `--depth` option should work based on git version and protocol choice. Any failure is now treated as a real problem, and the clone is only attempted once.
* Test improvements:
* `mock_git_repository.checks[type_of_test].args['git']` is now specified as the URL (with leading `file://`) in order to avoid complaints when using `--depth`.
* New type_of_test `tag-branch`.
* mock_git_repository now provides `git_exe`.
* Improved the action of the `git_version` fixture, which was previously hard-wired.
* New tests of `--single-branch` and `--depth 1` behavior.
* Add documentation of new options to the packaging guide.
Fixes#11163
The goal of this work is to simplify stage directory structures by eliminating use of symbolic links. This means, among other things, that` $spack/var/spack/stage` will no longer be the core staging directory. Instead, the first accessible `config:build_stage` path will be used.
Spack will no longer automatically append `spack-stage` (or the like) to configured build stage directories so the onus of distinguishing the directory from other work -- so the other work is not automatically removed with a `spack clean` operation -- falls on the user.
It's no longer possible to set compiler flags under as an entry under
"paths" in compilers.yaml; instead the user must list these under the
"flags" section. This updates the docs accordingly.
Using "compilers" with the "s" is an invalid config section and throws an error.
Traceback (most recent call last):
File "spack/bin/spack", line 48, in <module>
sys.exit(spack.main.main())
File "/home/omsai/src/libkmap/spack/lib/spack/spack/main.py", line 633, in main
env = ev.find_environment(args)
File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 263, in find_environment
return Environment(env)
File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 534, in __init__
self._read_manifest(f)
File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 561, in _read_manifest
self.yaml = _read_yaml(f)
File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 402, in _read_yaml
validate(data, filename)
File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 395, in validate
e, data, filename, e.instance.lc.line + 1)
spack.config.ConfigFormatError: /home/omsai/src/libkmap/spack.yaml:15: Additional properties are not allowed ('compilers' was unexpected)
Fixes#11781
* Rename build log to spack-build-log.txt
* Rename environment variables file to spack-build-env.txt
* The name of the log and env files is now the same during the build
and after the build completes
* Update packages which referred to the build log/env files
* For packages installed before this commit using older names for the
build and env files, search for the older names
Add an example of a 'modules:' entry for an external package in
packages.yaml. The 'External Packages' section of 'Build
Customization' mentions 'paths:' and 'modules:' and gives an
example of paths, but not modules.
* config:build_jobs now controls the number of parallel jobs to spawn during
builds, but cannot ever exceed the number of cores on the machine.
* The default is set to 16 or the number of available cores, whatever
is lowest.
* Updated docs to reflect the changes done to limit parallel builds
- `gettext_uuid=True` makes every commit update every .pot file in spack/localized-docs,
and speeds up the internationalized doc build slightly.
- Optimize for less repository churn, and use `python-levenshtein` to accelerate
the build instead.
- make all Spack paths relative to a `_spack_root` symlink, so that we
can easily relocate the docs build *outside* lib/spack/docs
- set some useful defaults for gettext translation variables in conf.py
- update `relativeinclude` and other references to the spack root in the
RST files to use _spack_root
- Add a `--update FILE` option to `spack list`
- Output is written to the file only if any package is newer than the file
- Simplify the code in docs/conf.py using this new option
The Spack documentation currently hard-codes some functionality in
`conf.py`, which makes the doc build less "pluggable" for things like
localized doc builds.
In particular, we unconditionally generate an index of commands and a
package list as part of the docs, but those should really only be done if
things are not up to date.
This commit does the following:
- Add `--header` option to `spack commands` so that it can do the work of
prepending text to its output.
- Add `--update FILE` option to `spack commands` that makes it generate a
new command index *only* if FILE is out of date w.r.t. commands in the
Spack source.
- Simplify code in `conf.py` to use these options and only update the
command index when needed.
