The SPACK_USER_CACHE_PATH was being overwritten in the windows CI
before_script. This should set the path for all systems unless
explicitly overridden.
* Split requirements to get better error messages in case of unsat solves.
* use list requirements instead of string
* activate static_analysis in a few pipelines
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* ci: darwin stacks: update tags following system updates
* disable SPACK_CI_DISABLE_STACKS; only enable *darwin* stacks for testing
* manually chmod u+w tmp/ before cleanup due to issue#49147
* comment out failing specs for now
* re-enable logic for disabling stacks
* add explanatory comment for darwin after_script additions
* remove more darwin-only targetting
* restore build_stage to default location
* move build-job-remove out of individual darwin stacks into darwin top level config
* keep build_stage in $spack/tmp for now
Currently, the custom config scopes are pushed at the top when constructing
configuration, and are demoted whenever a context manager activating an
environment is used - see #48414 for details. Workflows that rely on the order
in the [docs](https://spack.readthedocs.io/en/latest/configuration.html#custom-scopes)
are thus fragile, and may break
This PR allows to assign priorities to scopes, and ensures that scopes of lower priorities
are always "below" scopes of higher priorities. When scopes have the same priority,
what matters is the insertion order.
Modifications:
- [x] Add a mapping that iterates over keys according to priorities set when
adding the key/value pair
- [x] Use that mapping to allow assigning priorities to configuration scopes
- [x] Assign different priorities for different kind of scopes, to fix a bug, and
add a regression test
- [x] Simplify `Configuration` constructor
- [x] Remove `Configuration.pop_scope`
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
All the build jobs in pipelines are apparently relying on the bug that was fixed.
The issue was not caught in the PR because generation jobs were fine, and
there was nothing to rebuild.
Reverting to fix pipelines in a new PR.
This reverts commit 3ad99d75f9.
Currently, environments can end up with higher priority than `-C` custom
config scopes and `-c` command line arguments sometimes. This shouldn't
happen -- those explicit CLI scopes should override active environments.
Up to now configuration behaved like a stack, where scopes could be only be
pushed at the top. This PR allows to assign priorities to scopes, and ensures
that scopes of lower priorities are always "below" scopes of higher priorities.
When scopes have the same priority, what matters is the insertion order.
Modifications:
- [x] Add a mapping that iterates over keys according to priorities set when
adding the key/value pair
- [x] Use that mapping to allow assigning priorities to configuration scopes
- [x] Assign different priorities for different kind of scopes, to fix a bug, and
add a regression test
- [x] Simplify `Configuration` constructor
- [x] Remove `Configuration.pop_scope`
- [x] Remove `unify:false` from custom `-C` scope in pipelines
On the last modification: on `develop`, pipelines are relying on the environment
being able to override `-C` scopes, which is a bug. After this fix, we need to be
explicit about the unification strategy in each stack, and remove the blanket
`unify:false` from the highest priority scope
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Currently, we have `config:shared_linking:missing_library_policy` to error
or warn when shared libraries cannot be resolved upon install.
The new `spack verify libraries` command allows users to run this post
install hook at any point in time to check whether their current
installations can resolve shared libs in rpaths.
* Reproducer should decude artifact root from concrete environment
* Add documentation on the layout of the artifacts directory
* Use dag hash in the container name
* Add reproducer options to improve local testing
* --use-local-head allows running reproducer with
the current Spack HEAD commit rather than computing
a commit for the reproducer
* Add test to verify commits and recreating reproduction environment
* Add test for non-merge commit case
* ci reproduce-build: Drop overwrite option
in favor of throwing an error if the working dir is non-empty
Currently, when we setup the ASP problem for `clingo`, we don't take into account the configuration. This results in setting up ASP problems that are larger than necessary, with possibly redundant information, and higher concretization times.
This PR tries to improve things by adding an opt-in feature that computes the _possible dependencies_ of a solve taking also into account the current configuration, and avoids adding possible dependencies that we are certain can't be in the final solution.
The feature can be activated with:
```yaml
concretizer:
static_analysis: true
```
Examples of simple rules to discard dependencies are:
- Dependencies that are not buildable, and for which no binary is present (e.g. `cray-mpich` etc. on non Cray systems)
- Dependencies that are not for the current platform (e.g. `msmpi` on non Windows platforms)
- Conditional dependencies that cannot be activated, because of some user requirement (e.g. `cuda` etc. if the user requires `~cuda` in configuration)
- Virtual providers that cannot be used, because of a requirement on a virtual
The speed-up these rules seem to give depends on the use case at hand, but if the configuration is updated properly, they are noticeable.
Since in cases where there is no rule to exclude packages upfront, reuse is active, and this option is activated, it's possible to see some minor slow down, the feature has been added as opt-in, so it's turned off by default.
This adds a new configuration section called `env_vars:` that can be set in an environment.
It looks very similar to the existing `environment:` section that can be added to `modules.yaml`,
but it is global for an entire spack environment. It's called `env_vars:` to deconflate it with spack
environments (the term was too overloaded).
The syntax looks like this:
```yaml
spack:
specs:
- cmake%gcc
env_vars:
set:
ENVAR_SET_IN_ENV_LOAD: "True"
```
Any of our standard environment modifications can be added to the `env_vars` section, e.g.
`prepend_path:`, `unset:`, `append_path:`, etc. Operations in `env_vars:` are performed
on `spack env activate` and undone on `spack env deactivate`.
* e4s cray rhel ci stack: re-enable and update for new cpe, should fix cray libsci issue
* only run e4s-cray-rhel stack
* Mkae Autotools build_system point at correct build_directory
* remove selective enable of cray-rhel stacks
* restore SPACK_CI_DISABLE_STACKS
* use dot prefix to hide cray-sles jobs instead of comment-out
---------
Co-authored-by: Ryan Krattiger <ryan.krattiger@kitware.com>
A few changes to tarball creation (for build caches):
- do not run file to distinguish binary from text
- file is slow, even when running it in a batched fashion -- it usually reads all bytes and has slow logic to categorize specific types
- we don't need a highly detailed file categorization; a crude categorization of elf, mach-o, text suffices.
detecting elf and mach-o is straightforward and cheap
- detecting utf-8 (and with that ascii) is highly accurate: false positive rate decays exponentially as file size increases. Further it's not only the most common encoding, but the most common file type in package prefixes.
iso-8859-1 is cheaply (but heuristically) detected too, and sufficiently accurate after binaries and utf-8 files are classified earlier
- remove file as a dependency of Spack in general, which makes Spack itself easier to install
- detect file type and need to relocate as part of creating the tarball, which is more cache friendly and thus faster
`kcov` was removed in Ubuntu 24.04, and it is no longer
installable via `apt` in our CI images. Instal it via
Linuxbrew instead, at least until it comes back to Ubuntu.
`subversion` is also not installed on ubuntu 24 by default,
so we have to install it manually.
- [x] Add linuxbrew to linux tests
- [x] Install `kcov` with brew
- [x] Install subversion with `apt`
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
