Python was removed from being a build tool in #46980, due to issues
when reusing specs. This PR adds a new rule to match the interpreter
among different Python packages, in clingo.
It also adds a bunch of new "build-tools", so that specs like:
```
py-matplotlib backend=tkagg
```
can be concretized in one go.
Modifications:
- [x] Make `py-matplotlib backend=tkagg` concretizable
- [x] Add unit-tests to ensure situations like in #46980 do not happen
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Currently, the custom config scopes are pushed at the top when constructing
configuration, and are demoted whenever a context manager activating an
environment is used - see #48414 for details. Workflows that rely on the order
in the [docs](https://spack.readthedocs.io/en/latest/configuration.html#custom-scopes)
are thus fragile, and may break
This PR allows to assign priorities to scopes, and ensures that scopes of lower priorities
are always "below" scopes of higher priorities. When scopes have the same priority,
what matters is the insertion order.
Modifications:
- [x] Add a mapping that iterates over keys according to priorities set when
adding the key/value pair
- [x] Use that mapping to allow assigning priorities to configuration scopes
- [x] Assign different priorities for different kind of scopes, to fix a bug, and
add a regression test
- [x] Simplify `Configuration` constructor
- [x] Remove `Configuration.pop_scope`
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
All the build jobs in pipelines are apparently relying on the bug that was fixed.
The issue was not caught in the PR because generation jobs were fine, and
there was nothing to rebuild.
Reverting to fix pipelines in a new PR.
This reverts commit 3ad99d75f9.
VariantMap.concrete is unused, and would be incorrect if it were used
due to conditional variants.
Just let the Spec dictate what is concrete and what is not.
Currently, environments can end up with higher priority than `-C` custom
config scopes and `-c` command line arguments sometimes. This shouldn't
happen -- those explicit CLI scopes should override active environments.
Up to now configuration behaved like a stack, where scopes could be only be
pushed at the top. This PR allows to assign priorities to scopes, and ensures
that scopes of lower priorities are always "below" scopes of higher priorities.
When scopes have the same priority, what matters is the insertion order.
Modifications:
- [x] Add a mapping that iterates over keys according to priorities set when
adding the key/value pair
- [x] Use that mapping to allow assigning priorities to configuration scopes
- [x] Assign different priorities for different kind of scopes, to fix a bug, and
add a regression test
- [x] Simplify `Configuration` constructor
- [x] Remove `Configuration.pop_scope`
- [x] Remove `unify:false` from custom `-C` scope in pipelines
On the last modification: on `develop`, pipelines are relying on the environment
being able to override `-C` scopes, which is a bug. After this fix, we need to be
explicit about the unification strategy in each stack, and remove the blanket
`unify:false` from the highest priority scope
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Currently, we have `config:shared_linking:missing_library_policy` to error
or warn when shared libraries cannot be resolved upon install.
The new `spack verify libraries` command allows users to run this post
install hook at any point in time to check whether their current
installations can resolve shared libs in rpaths.
Installing `rust@nightly` fails because the package file declares a conflict of rust versions older than `:1.64` with `gcc>=13`. However, because `nightly` is alphanumerically smaller than any actual version number, `nightly` is incorrectly detected to have a conflict with `gcc>=13` as well. Marking `nightly` as an infinity version instead solves this.
* Reproducer should decude artifact root from concrete environment
* Add documentation on the layout of the artifacts directory
* Use dag hash in the container name
* Add reproducer options to improve local testing
* --use-local-head allows running reproducer with
the current Spack HEAD commit rather than computing
a commit for the reproducer
* Add test to verify commits and recreating reproduction environment
* Add test for non-merge commit case
* ci reproduce-build: Drop overwrite option
in favor of throwing an error if the working dir is non-empty
A directory and a symlink to it under the same relative path in a
different prefix
```
/prefix1/dir/
/prefix1/dir/file
/prefix2/dir -> /prefix1/dir/
```
are not a blocker to create a view. The view structure simply looks like
this:
```
/view/dir/
/view/dir/file
```
This should be the case independently of the order in which we visit
prefixes, so we could in principle create views order independently.
Up to now, Spack was allowing all build-tools that
may appear in the DAG to have 2 max_dupes.
This is not needed in practice for most of them,
and adding them out of caution just increases
grounding and concretization time.
This PR makes the value of max_dupes configurable
per package, and sets only a few known packages to
2 max_dupes by default.
In case user needs different values, they can
tune the configuration for their use case.
On macOS, prefix_a/file and prefix_b/FILE map to the same file view/file or view/FILE.
This commit ensures that we test whether a view is created on a case insensitive filesystem and handle projection conflicts accordingly.
With this change spec["pkg"] searches only direct dependencies and transitive link/run
dependencies, ordered by depth. This avoids situations where we pick up unwanted
deps of build/test deps.
To reach those, you need to do spec["build_dep"]["pkg"] explicitly.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
meson's `dependency` function often uses pkg-config to locate a dependency, and may even fall back to cmake. The former case is very common, and since packagers often forget to add the tiny pkgconfig package as a build dep, we do it for them.
This PR is effectively a breaking change extracted from #45189, which removes
support for spec["mpi"] if spec itself is openmpi / mpich that could provide mpi;
from the Spec instance we don't have any parent it provides it to,
hence it's a KeyError.
Currently, when we setup the ASP problem for `clingo`, we don't take into account the configuration. This results in setting up ASP problems that are larger than necessary, with possibly redundant information, and higher concretization times.
This PR tries to improve things by adding an opt-in feature that computes the _possible dependencies_ of a solve taking also into account the current configuration, and avoids adding possible dependencies that we are certain can't be in the final solution.
The feature can be activated with:
```yaml
concretizer:
static_analysis: true
```
Examples of simple rules to discard dependencies are:
- Dependencies that are not buildable, and for which no binary is present (e.g. `cray-mpich` etc. on non Cray systems)
- Dependencies that are not for the current platform (e.g. `msmpi` on non Windows platforms)
- Conditional dependencies that cannot be activated, because of some user requirement (e.g. `cuda` etc. if the user requires `~cuda` in configuration)
- Virtual providers that cannot be used, because of a requirement on a virtual
The speed-up these rules seem to give depends on the use case at hand, but if the configuration is updated properly, they are noticeable.
Since in cases where there is no rule to exclude packages upfront, reuse is active, and this option is activated, it's possible to see some minor slow down, the feature has been added as opt-in, so it's turned off by default.
`relocate_links` warns when the target is absolute and not matched by
any prefix from the prefix to prefix map.
This can lead to false positives, cause the prefix to prefix map does
not contain trivial/identity entries whenever a package is installed to
its original location.
Since relocate_links is the odd one out there (we don't warn about
similar issues with rpaths, etc), just remove the warning.
* Remove variable from cmake.py
#48775 left a dangling variable that was not caught in CI but by the eyes of @haampie. Restructure variable to local method.
* [@spackbot] updating style on behalf of psakievich
* Update cmake.py
* Update lib/spack/spack/build_systems/cmake.py
* Update lib/spack/spack/build_systems/cmake.py
---------
Co-authored-by: psakievich <psakievich@users.noreply.github.com>
Add ruff configuration to `pyproject.toml`.
This allows `ruff format` in the Spack repository to format all the files we care about,
with our line length of 99, the exceptions we already put in place, and excluding things
we don't auto-format, like vendored dependencies.
Right now it'll reformat 175 or so files, but only slightly, in places where `ruff` differs from
`black`. For the most part I like the ruff format decisions better than `black`, but none of
the changes seem too severe.
This does not change `spack style` -- I figure that can come later but this at least will
let people start playing with `ruff`.
---------
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
These are some changes that `ruff check --fix` would make that the current
`spack style` also agrees with. Make the changes now so that the `ruff`
change is less disruptive.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Currently, environments created from manifest files with relative includes result in broken
references to config files.
This PR modifies `spack env create` to create local copies in the new environment of any local
config files from relative paths in the environment manifest passed as an init file.
This PR does not change the behavior if the include is an absolute path or if the include is from
a relative path outside the environment directory, but it does warn about missing relative includes if
they are inside the environment directory.
Includes regression test and short blurb in docs.
* package api: drop wildcard re-export
To ensure package repos are forward/backward compatibility with Spack,
we should explicitly export all symbols we want to expose in the public
package API, and drop `from spack.something import *` because
removal/addition to the public API will go unnoticed.
Also `llnl.util.filesystem` has some methods that shouldn't be exposed
in the package API, so better to enumerate a subset explicitly.
* remove flatten_dependencies / install_dependency_symlinks
Regressed in #47126
Spack was not interpreting mirrors using relative path with respect to the
metadata directory.
---------
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* test_no_matching_compiler_specs: does not need mock_low_high_config,
since mutable_config is already used at class level
* bindist.py: setup a configuration that doesn't super-impose builtin.mock
over builtin
* builder.py: use a mock configuration for the tests
This adds a new configuration section called `env_vars:` that can be set in an environment.
It looks very similar to the existing `environment:` section that can be added to `modules.yaml`,
but it is global for an entire spack environment. It's called `env_vars:` to deconflate it with spack
environments (the term was too overloaded).
The syntax looks like this:
```yaml
spack:
specs:
- cmake%gcc
env_vars:
set:
ENVAR_SET_IN_ENV_LOAD: "True"
```
Any of our standard environment modifications can be added to the `env_vars` section, e.g.
`prepend_path:`, `unset:`, `append_path:`, etc. Operations in `env_vars:` are performed
on `spack env activate` and undone on `spack env deactivate`.
* Reduce the size of outputted go built binaries
* Remove unused import from go package
* go: remove comment from setup dependents build env
* Add back missing imports after rebase
* Backward compat with Python 3.9 for socket.timeout
* Forward compat with Python [unknown] as HTTPResponse.geturl is deprecated
* Catch timeout etc from .read()
* Some minor simplifications: json.load(...) takes file object in binary mode.
* Fix CDash code which does error handling wrong: non-2XX responses raise.
Should be sufficient to set CONFIG_SITE to /dev/null before running configure to
prevent any config site file to be loaded, which currently causes non-determinism in builds.
Setting the variable CONFIG_SITE prevents configure files to check $prefix/share/config.site,
$prefix/etc/config.site, $ac_default_prefix/share/config.site, $ac_default_prefix/etc/config.site
so we don't have to patch a block of code.
* Address quoting issue that casuses dev_paths containing @ symbols to parse as versions
* Fix additional location were dev_path was imporperly constructed
The dev_path must be quoted to avoid parsing issues when
paths contain '@' symbols. Also add tests to catch
regression of this behavior
* Format to fix line length
* fix failing tests
* Fix whitespace error
* Add binary_compatibility fixture to test
* Change string formatting to avoid multiline f string
* Update lib/spack/spack/test/concretization/core.py
* Add tmate debug session
* Revert "Add tmate debug session"
This reverts commit 24e2f77e3c.
* Move test and refactor to use env methods
* Update lib/spack/spack/test/cmd/develop.py
---------
Co-authored-by: psakievich <psakiev@sandia.gov>
Since macOS 15 `ld -single_module` warns with a deprecation message,
which makes configure scripts believe the flag is unsupported. That
in turn triggers a code path where `archive_cmds` is set to
```
$CC -r -keep_private_externs -nostdlib ... -dynamiclib
```
instead of just
```
$CC -dynamiclib ...
```
This code path was meant to trigger only on ancient macOS <= 14.4 where
libtool had to add `-single_module`, which is the default since macos
14.4, and is now apparently deprecated because the flag is a no-op for
more than 15 years.
The wrong `archive_cmds` causes actual problems combined with a bug in
OpenMPI's compiler wrapper (`CC=mpicc`), which appends `-rpath` flags,
which cause an error when combined with the `-r` flag added by the
autotools.
Spack's compiler wrapper doesn't do this, but it's likely there are
other compiler wrappers out there that are not aware that `-r` and
`-rpath` cannot be combined.
The fix is to change defaults: `lt_cv_apple_cc_single_mod=yes`.
* for config: let `syaml_dict` inherit from `dict` instead of `OrderedDict`. `syaml_dict` now only exists as a mutable wrapper for yaml related metadata.
* for spec serialization / hashing: use `dict` directly
This is possible since we only support cpython 3.6+ in which dicts are ordered.
This improves performance of hash computation a bit. For a larger spec I'm getting 9.22ms instead of 11.9ms, so 22.5% reduction in runtime.
