Commit Graph

7614 Commits

Author SHA1 Message Date
Massimiliano Culpo
9e508b0321 Revert "Assign priorities to configuration scopes (#48420)" (#49185)
All the build jobs in pipelines are apparently relying on the bug that was fixed.

The issue was not caught in the PR because generation jobs were fine, and
there was nothing to rebuild.

Reverting to fix pipelines in a new PR.

This reverts commit 3ad99d75f9.
2025-02-25 02:33:41 -08:00
dependabot[bot]
1cc63e2b7c build(deps): bump sphinx from 8.2.0 to 8.2.1 in /lib/spack/docs (#49180)
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 8.2.0 to 8.2.1.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/v8.2.1/CHANGES.rst)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v8.2.0...v8.2.1)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-25 02:33:03 -07:00
Harmen Stoppels
4e311a22d0 spec.py: remove VariantMap.concrete (#49170)
VariantMap.concrete is unused, and would be incorrect if it were used
due to conditional variants.

Just let the Spec dictate what is concrete and what is not.
2025-02-25 10:18:06 +01:00
Massimiliano Culpo
3ad99d75f9 Assign priorities to configuration scopes (#48420)
Currently, environments can end up with higher priority than `-C` custom
config scopes and `-c` command line arguments sometimes. This shouldn't
happen -- those explicit CLI scopes should override active environments.

Up to now configuration behaved like a stack, where scopes could be only be
pushed at the top. This PR allows to assign priorities to scopes, and ensures
that scopes of lower priorities are always "below" scopes of higher priorities.

When scopes have the same priority, what matters is the insertion order.

Modifications:
- [x] Add a mapping that iterates over keys according to priorities set when
      adding the key/value pair
- [x] Use that mapping to allow assigning priorities to configuration scopes
- [x] Assign different priorities for different kind of scopes, to fix a bug, and
      add a regression test
- [x] Simplify `Configuration` constructor
- [x] Remove `Configuration.pop_scope`
- [x] Remove `unify:false` from custom `-C` scope in pipelines

On the last modification: on `develop`, pipelines are relying on the environment
being able to override `-C` scopes, which is a bug. After this fix, we need to be
explicit about the unification strategy in each stack, and remove the blanket
`unify:false` from the highest priority scope

Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2025-02-25 00:58:16 -08:00
Harmen Stoppels
e783a2851d Revert "Repo.packages_with_tags: do not construct a set of all packages (#49141)" (#49172)
This reverts commit 0da5bafaf2.
2025-02-24 16:46:41 +01:00
Harmen Stoppels
4e7a5e9362 spack verify libraries: verify dependencies of installed packages can be resolved (#49124)
Currently, we have `config:shared_linking:missing_library_policy` to error
or warn when shared libraries cannot be resolved upon install.

The new `spack verify libraries` command allows users to run this post
install hook at any point in time to check whether their current
installations can resolve shared libs in rpaths.
2025-02-24 11:28:06 +01:00
Harmen Stoppels
974abc8067 Add typehints for directory_layout / Spec.prefix (#48652) 2025-02-24 09:47:07 +00:00
Massimiliano Culpo
2f9ad5f34d spec.py: fix virtual reconstruction for old specs (#49103)
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2025-02-24 01:23:37 -07:00
Harmen Stoppels
6cd74efa90 Spec.ensure_external_path_if_external, Spec.inject_patches_variant -> spack.solver.asp (#48988)
y
2025-02-24 08:36:23 +01:00
ByteHamster
b304a2d854 Fix installing rust@nightly (#49098)
Installing `rust@nightly` fails because the package file declares a conflict of rust versions older than `:1.64` with `gcc>=13`. However, because `nightly` is alphanumerically smaller than any actual version number, `nightly` is incorrectly detected to have a conflict with `gcc>=13` as well. Marking `nightly` as an infinity version instead solves this.
2025-02-21 09:53:46 -08:00
Ryan Krattiger
1fa1864b37 Reproducer should deduce artifact root from concrete environment (#45281)
* Reproducer should decude artifact root from concrete environment

* Add documentation on the layout of the artifacts directory

* Use dag hash in the container name

* Add reproducer options to improve local testing

* --use-local-head allows running reproducer with
  the current Spack HEAD commit rather than computing
  a commit for the reproducer

* Add test to verify commits and recreating reproduction environment

* Add test for non-merge commit case

* ci reproduce-build: Drop overwrite option
in favor of throwing an error if the working dir is non-empty
2025-02-21 10:46:43 -06:00
Harmen Stoppels
0da5bafaf2 Repo.packages_with_tags: do not construct a set of all packages (#49141) 2025-02-21 16:23:42 +01:00
Harmen Stoppels
b8ec69112f Extracted changes from 45189 (#49137)
Co-Authored-By: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2025-02-21 10:58:00 +01:00
Massimiliano Culpo
a3645fd372 Make BaseConfiguration pickleable (#47545)
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2025-02-21 09:54:22 +01:00
dependabot[bot]
9bcd86071f build(deps): bump sphinx from 8.1.3 to 8.2.0 in /lib/spack/docs (#49118)
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 8.1.3 to 8.2.0.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES.rst)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v8.1.3...v8.2.0)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-20 16:42:00 -08:00
Harmen Stoppels
3e8cb852b0 spec.py: use json.dumps directly to avoid hash breakage (#48884) 2025-02-20 17:39:07 +01:00
Wouter Deconinck
f9e4d3898a cargo: avoid need to use super().build_args with std_build_args (#49071)
* cargo: avoid need to use super().build_args with std_build_args

* cargo: fix style

* jujutsu: avoid need for super().build_args
2025-02-19 09:01:56 -08:00
Harmen Stoppels
c97be2a9d7 checksum.py tests: extract add_versions_to_pkg fixture (#49100) 2025-02-19 07:33:50 +00:00
Harmen Stoppels
97d66b637f Fix tests modifying package.py files (#49093) 2025-02-18 13:28:07 +01:00
Harmen Stoppels
e4a8d45d86 views: resolve symlinked dir - dir conflict when same file (#49039)
A directory and a symlink to it under the same relative path in a
different prefix

```
/prefix1/dir/
/prefix1/dir/file
/prefix2/dir -> /prefix1/dir/
```

are not a blocker to create a view. The view structure simply looks like
this:

```
/view/dir/
/view/dir/file
```

This should be the case independently of the order in which we visit
prefixes, so we could in principle create views order independently.
2025-02-18 06:53:17 +01:00
Harmen Stoppels
2e372c53ab spec.py: remove Spec.virtual_dependencies (#49079) 2025-02-18 06:17:55 +01:00
dependabot[bot]
a9da160160 build(deps): bump flake8 from 7.1.1 to 7.1.2 in /lib/spack/docs (#49087)
Bumps [flake8](https://github.com/pycqa/flake8) from 7.1.1 to 7.1.2.
- [Commits](https://github.com/pycqa/flake8/compare/7.1.1...7.1.2)

---
updated-dependencies:
- dependency-name: flake8
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-17 20:08:05 -07:00
Harmen Stoppels
545750873e Spec.__contains__: restrict to direct build and transitive runtime deps (#49072)
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2025-02-17 13:57:16 +01:00
dependabot[bot]
7d4523a9fc build(deps): bump isort from 5.13.2 to 6.0.0 in /lib/spack/docs (#48747)
Bumps [isort](https://github.com/pycqa/isort) from 5.13.2 to 6.0.0.
- [Release notes](https://github.com/pycqa/isort/releases)
- [Changelog](https://github.com/PyCQA/isort/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pycqa/isort/compare/5.13.2...6.0.0)

---
updated-dependencies:
- dependency-name: isort
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-17 12:05:41 +01:00
Harmen Stoppels
754a64d1fe apply black v25.1.0 (#49076) 2025-02-17 11:42:12 +01:00
dependabot[bot]
2b52639032 build(deps): bump black from 24.10.0 to 25.1.0 in /lib/spack/docs (#48780)
Bumps [black](https://github.com/psf/black) from 24.10.0 to 25.1.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/24.10.0...25.1.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-16 16:37:36 -07:00
Massimiliano Culpo
ccd205bfeb Allow tuning max_dupes for build dependencies (#48948)
Up to now, Spack was allowing all build-tools that
may appear in the DAG to have 2 max_dupes.

This is not needed in practice for most of them,
and adding them out of caution just increases
grounding and concretization time.

This PR makes the value of max_dupes configurable
per package, and sets only a few known packages to
2 max_dupes by default.

In case user needs different values, they can
tune the configuration for their use case.
2025-02-14 14:25:12 +01:00
Paul Gessinger
114bd5744f views: normalize paths on case insensitive file systems (#47370)
On macOS, prefix_a/file and prefix_b/FILE map to the same file view/file or view/FILE.

This commit ensures that we test whether a view is created on a case insensitive filesystem and handle projection conflicts accordingly.
2025-02-14 09:35:40 +01:00
Harmen Stoppels
8ef5f1027a Spec.__getitem__: restrict to direct deps + transitive runtime deps (#49016)
With this change spec["pkg"] searches only direct dependencies and transitive link/run
dependencies, ordered by depth. This avoids situations where we pick up unwanted 
deps of build/test deps. 

To reach those, you need to do spec["build_dep"]["pkg"] explicitly.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2025-02-14 08:45:50 +01:00
Wouter Deconinck
36bc53ee07 MesonPackage: depends_on pkgconfig (#46955)
meson's `dependency` function often uses pkg-config to locate a dependency, and may even fall back to cmake. The former case is very common, and since packagers often forget to add the tiny pkgconfig package as a build dep, we do it for them.
2025-02-14 08:36:16 +01:00
Tamara Dahlgren
236b8fc009 RepoSplit/tests: update repo tests relying on builtin package repo to only use mock repos (#48926)
* RepoSplit/tests: update repo tests relying on builtin

* test_repo_last_mtime: skip on windows due to mtime issues in CI
2025-02-13 21:49:50 -08:00
Harmen Stoppels
a7b542dd37 spack debug create-db-tarball: remove after test failures (#49025) 2025-02-13 13:10:28 +01:00
John W. Parent
fd59d3e589 Ci: ensure file path comparsion uses posix paths (#47033)
Git always produces posix paths, ensure we're always comparsing apples to apples by normalizing paths compared to git output to posix.
2025-02-12 12:14:16 -07:00
Massimiliano Culpo
0172208c52 solver: add type-hints to OutputConfiguration (#48979) 2025-02-12 20:12:12 +01:00
Harmen Stoppels
a137da1cd5 Spec.validate_detection -> spack.detection.path.validate_detection (#48987) 2025-02-12 09:30:14 -07:00
Harmen Stoppels
03e972314f Remove spack.repo.PATH.is_virtual call from SpecBuildInterface.(#48984)
This PR is effectively a breaking change extracted from #45189, which removes 
support for spec["mpi"] if spec itself is openmpi / mpich that could provide mpi; 
from the Spec instance we don't have any parent it provides it to, 
hence it's a KeyError.
2025-02-12 17:17:45 +01:00
Harmen Stoppels
76f00a3659 Spec.is_virtual -> spack.repo.PATH.is_virtual (#48986) 2025-02-12 16:57:27 +01:00
Harmen Stoppels
cd98781fb4 concretize.lp: don't warn about deprecation when external (#49008) 2025-02-12 16:27:37 +01:00
Harmen Stoppels
9747978c7f Spec.package_class -> spack.repo.PATH.get_pkg_class (#48985) 2025-02-12 11:52:04 +01:00
Harmen Stoppels
fb9d6427e6 package_hash.py: move metadata_attrs inline out of package_base (#48981) 2025-02-12 10:38:35 +01:00
Harmen Stoppels
18cd922aab style.py: fix false negative in redundant import statements (#48980) 2025-02-11 19:30:50 +01:00
Massimiliano Culpo
5518ad9611 Fix performance issue on macOS (#48997)
archspec.cpu.host() is not memoized, so compute
it as less as possible.

---------

Co-authored-by: alalazo <alalazo@users.noreply.github.com>
2025-02-11 18:54:32 +01:00
Harmen Stoppels
3909308d5c spack.util.elf: catch seek errors (#48972) 2025-02-11 08:52:52 +01:00
Massimiliano Culpo
54210270c8 concretizer: reduce search space with static analysis (#48729)
Currently, when we setup the ASP problem for `clingo`, we don't take into account the configuration. This results in setting up ASP problems that are larger than necessary, with possibly redundant information, and higher concretization times. 

This PR tries to improve things by adding an opt-in feature that computes the _possible dependencies_ of a solve taking also into account the current configuration, and avoids adding possible dependencies that we are certain can't be in the final solution.

The feature can be activated with:
```yaml
concretizer:
  static_analysis: true
```

Examples of simple rules to discard dependencies are:
- Dependencies that are not buildable, and for which no binary is present (e.g. `cray-mpich` etc. on non Cray systems)
- Dependencies that are not for the current platform (e.g. `msmpi` on non Windows platforms)
- Conditional dependencies that cannot be activated, because of some user requirement (e.g. `cuda` etc. if the user requires `~cuda` in configuration)
- Virtual providers that cannot be used, because of a requirement on a virtual

The speed-up these rules seem to give depends on the use case at hand, but if the configuration is updated properly, they are noticeable. 

Since in cases where there is no rule to exclude packages upfront, reuse is active, and this option is activated, it's possible to see some minor slow down, the feature has been added as opt-in, so it's turned off by default.
2025-02-11 08:44:20 +01:00
Harmen Stoppels
6d608a9664 log.py: remove setenv calls (#48933) 2025-02-10 09:53:36 +01:00
Harmen Stoppels
afe7d6c39e package_base.py: remove use_cray_compiler_names (#48932) 2025-02-07 12:47:17 +01:00
psakievich
1f428c4188 Add a message for CMake incremental build (#48905)
* Add a message for CMake incremental build

Requested message to explain CMake phase is getting skipped.

* [@spackbot] updating style on behalf of psakievich

* Update import

---------

Co-authored-by: psakievich <psakievich@users.noreply.github.com>
2025-02-07 01:32:06 +00:00
Harmen Stoppels
1c93fef160 spec.py: ensure == is false if true modulo precomputed dag hash (#48889) 2025-02-06 09:55:27 +01:00
Harmen Stoppels
e0e6f29584 relocate.py: don't warn about symlinks (#48904)
`relocate_links` warns when the target is absolute and not matched by
any prefix from the prefix to prefix map.

This can lead to false positives, cause the prefix to prefix map does
not contain trivial/identity entries whenever a package is installed to
its original location.

Since relocate_links is the odd one out there (we don't warn about
similar issues with rpaths, etc), just remove the warning.
2025-02-05 12:33:44 -07:00
Harmen Stoppels
4487598d60 spec.py: ensure spec.extra_attributes is {} if is null in json (#48896) 2025-02-05 17:55:53 +01:00