* style.py: add spack style --spec-strings for compat with v1.0
* add --fix also, and avoid infinite recursion and too large files
* tests: check identify and check edit files
This should help resolve the "No binary found when cache-only was specified"
errors we've recently seen in our GitLab CI pipelines.
example failing job here:
https://gitlab.spack.io/spack/spack/-/jobs/15570931#L370
This error is caused when a generate job finds a spec in the local root
binary mirror, and that spec does not yet exist in the stack-specific mirror.
The fix here is to instead locally cache the stack-specific mirrors and only
use the root-level mirror for public use.
When requiring a constraint on a virtual package, it makes little
sense to use anonymous specs, and our documentation shows no example
of requirements on virtual packages starting with `^`.
Right now, due to how `^` is implemented in the solver, writing:
```yaml
mpi:
require: "^openmpi"
```
is equivalent to the more correct form:
```yaml
mpi:
require: "openmpi"
```
but the situation will change when `%` will shift its meaning to be a
direct dependency.
To avoid later errors that are both unclear, and quite slow to get to the user,
this commit makes anonymous specs under virtual requirements an error,
and shows a clear error message pointing to the file and line where the
spec needs to be changed.
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Supersedes #46792.
Closes#40018.
Closes#31026.
Closes#2700.
There were a number of feature requests for os-specific config. This enables os-specific
config without adding a lot of special sub-scopes.
Support `include:` as an independent configuration schema, allowing users to include
configuration scopes from files or directories. Includes can be:
* conditional (similar to definitions in environments), and/or
* optional (i.e., the include will be skipped if it does not exist).
Includes can be paths or URLs (`ftp`, `https`, `http` or `file`). Paths can be absolute or
relative . Environments can include configuration files using the same schema. Remote includes
must be checked by `sha256`.
Includes can also be recursive, and this modifies the config system accordingly so that
we push included configuration scopes on the stack *before* their including scopes, and
we remove configuration scopes from the stack when their including scopes are removed.
For example, you could have an `include.yaml` file (e.g., under `$HOME/.spack`) to specify
global includes:
```
include:
- ./enable_debug.yaml
- path: https://github.com/spack/spack-configs/blob/main/NREL/configs/mac/config.yaml
sha256: 37f982915b03de18cc4e722c42c5267bf04e46b6a6d6e0ef3a67871fcb1d258b
```
Or an environment `spack.yaml`:
```
spack:
include:
- path: "/path/to/a/config-dir-or-file"
when: os == "ventura"
- ./path/relative/to/containing/file/that/is/required
- path: "/path/with/spack/variables/$os/$target"
optional: true
- path: https://raw.githubusercontent.com/spack/spack-configs/refs/heads/main/path/to/required/raw/config.yaml
sha256: 26e871804a92cd07bb3d611b31b4156ae93d35b6a6d6e0ef3a67871fcb1d258b
```
Updated TODO:
- [x] Get existing unit tests to pass with Todd's changes
- [x] Resolve new (or old) circular imports
- [x] Ensure remote includes (global) work
- [x] Ensure remote includes for environments work (note: caches remote
files under user cache root)
- [x] add sha256 field to include paths, validate, and require for remote includes
- [x] add sha256 remote file unit tests
- [x] revisit how diamond includes should work
- [x] support recursive includes
- [x] add recursive include unit tests
- [x] update docs and unit test to indicate ordering of recursive includes with
conflicting options is deferred to follow-on work
---------
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Co-authored-by: Peter Scheibel <scheibel1@llnl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
The SPACK_USER_CACHE_PATH was being overwritten in the windows CI
before_script. This should set the path for all systems unless
explicitly overridden.
* Split requirements to get better error messages in case of unsat solves.
* use list requirements instead of string
* activate static_analysis in a few pipelines
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* ci: darwin stacks: update tags following system updates
* disable SPACK_CI_DISABLE_STACKS; only enable *darwin* stacks for testing
* manually chmod u+w tmp/ before cleanup due to issue#49147
* comment out failing specs for now
* re-enable logic for disabling stacks
* add explanatory comment for darwin after_script additions
* remove more darwin-only targetting
* restore build_stage to default location
* move build-job-remove out of individual darwin stacks into darwin top level config
* keep build_stage in $spack/tmp for now
Currently, the custom config scopes are pushed at the top when constructing
configuration, and are demoted whenever a context manager activating an
environment is used - see #48414 for details. Workflows that rely on the order
in the [docs](https://spack.readthedocs.io/en/latest/configuration.html#custom-scopes)
are thus fragile, and may break
This PR allows to assign priorities to scopes, and ensures that scopes of lower priorities
are always "below" scopes of higher priorities. When scopes have the same priority,
what matters is the insertion order.
Modifications:
- [x] Add a mapping that iterates over keys according to priorities set when
adding the key/value pair
- [x] Use that mapping to allow assigning priorities to configuration scopes
- [x] Assign different priorities for different kind of scopes, to fix a bug, and
add a regression test
- [x] Simplify `Configuration` constructor
- [x] Remove `Configuration.pop_scope`
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
All the build jobs in pipelines are apparently relying on the bug that was fixed.
The issue was not caught in the PR because generation jobs were fine, and
there was nothing to rebuild.
Reverting to fix pipelines in a new PR.
This reverts commit 3ad99d75f9.
Currently, environments can end up with higher priority than `-C` custom
config scopes and `-c` command line arguments sometimes. This shouldn't
happen -- those explicit CLI scopes should override active environments.
Up to now configuration behaved like a stack, where scopes could be only be
pushed at the top. This PR allows to assign priorities to scopes, and ensures
that scopes of lower priorities are always "below" scopes of higher priorities.
When scopes have the same priority, what matters is the insertion order.
Modifications:
- [x] Add a mapping that iterates over keys according to priorities set when
adding the key/value pair
- [x] Use that mapping to allow assigning priorities to configuration scopes
- [x] Assign different priorities for different kind of scopes, to fix a bug, and
add a regression test
- [x] Simplify `Configuration` constructor
- [x] Remove `Configuration.pop_scope`
- [x] Remove `unify:false` from custom `-C` scope in pipelines
On the last modification: on `develop`, pipelines are relying on the environment
being able to override `-C` scopes, which is a bug. After this fix, we need to be
explicit about the unification strategy in each stack, and remove the blanket
`unify:false` from the highest priority scope
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Currently, we have `config:shared_linking:missing_library_policy` to error
or warn when shared libraries cannot be resolved upon install.
The new `spack verify libraries` command allows users to run this post
install hook at any point in time to check whether their current
installations can resolve shared libs in rpaths.
* Reproducer should decude artifact root from concrete environment
* Add documentation on the layout of the artifacts directory
* Use dag hash in the container name
* Add reproducer options to improve local testing
* --use-local-head allows running reproducer with
the current Spack HEAD commit rather than computing
a commit for the reproducer
* Add test to verify commits and recreating reproduction environment
* Add test for non-merge commit case
* ci reproduce-build: Drop overwrite option
in favor of throwing an error if the working dir is non-empty
Currently, when we setup the ASP problem for `clingo`, we don't take into account the configuration. This results in setting up ASP problems that are larger than necessary, with possibly redundant information, and higher concretization times.
This PR tries to improve things by adding an opt-in feature that computes the _possible dependencies_ of a solve taking also into account the current configuration, and avoids adding possible dependencies that we are certain can't be in the final solution.
The feature can be activated with:
```yaml
concretizer:
static_analysis: true
```
Examples of simple rules to discard dependencies are:
- Dependencies that are not buildable, and for which no binary is present (e.g. `cray-mpich` etc. on non Cray systems)
- Dependencies that are not for the current platform (e.g. `msmpi` on non Windows platforms)
- Conditional dependencies that cannot be activated, because of some user requirement (e.g. `cuda` etc. if the user requires `~cuda` in configuration)
- Virtual providers that cannot be used, because of a requirement on a virtual
The speed-up these rules seem to give depends on the use case at hand, but if the configuration is updated properly, they are noticeable.
Since in cases where there is no rule to exclude packages upfront, reuse is active, and this option is activated, it's possible to see some minor slow down, the feature has been added as opt-in, so it's turned off by default.
This adds a new configuration section called `env_vars:` that can be set in an environment.
It looks very similar to the existing `environment:` section that can be added to `modules.yaml`,
but it is global for an entire spack environment. It's called `env_vars:` to deconflate it with spack
environments (the term was too overloaded).
The syntax looks like this:
```yaml
spack:
specs:
- cmake%gcc
env_vars:
set:
ENVAR_SET_IN_ENV_LOAD: "True"
```
Any of our standard environment modifications can be added to the `env_vars` section, e.g.
`prepend_path:`, `unset:`, `append_path:`, etc. Operations in `env_vars:` are performed
on `spack env activate` and undone on `spack env deactivate`.
* e4s cray rhel ci stack: re-enable and update for new cpe, should fix cray libsci issue
* only run e4s-cray-rhel stack
* Mkae Autotools build_system point at correct build_directory
* remove selective enable of cray-rhel stacks
* restore SPACK_CI_DISABLE_STACKS
* use dot prefix to hide cray-sles jobs instead of comment-out
---------
Co-authored-by: Ryan Krattiger <ryan.krattiger@kitware.com>
