Commit Graph

36056 Commits

Author SHA1 Message Date
Harmen Stoppels
d88cfbd839 compilers.yaml: require list of strings for modules (#47197) 2024-11-18 12:47:11 +01:00
Harmen Stoppels
4127a93a91 ensure write_fd.close() isn't called when sys.std* cannot be redirected 2024-11-18 12:47:11 +01:00
Harmen Stoppels
5df2189e43 Avoid a socket to communicate effectively a bit 2024-11-18 12:47:11 +01:00
Harmen Stoppels
667c1960d0 Replace MultiProcessFd with Connection objects
Connection objects are Python version, platform and multiprocessing
start method independent, so better to use those than a mix of plain
file descriptors and inadequate guesses in the child process whether it
was forked or not.

This also allows us to delete the now redundant MultiProcessFd class,
hopefully making things a bit easier to follow.
2024-11-18 12:47:11 +01:00
Harmen Stoppels
4449058257 fix use of traceback.format_exception (#47080)
Co-authored-by: Peter Scheibel <scheibel1@llnl.gov>
2024-11-18 12:47:11 +01:00
Todd Gamblin
1b03f1d5bc Better docs and typing for _setup_pkg_and_run (#46709)
There was a bit of mystery surrounding the arguments for `_setup_pkg_and_run`. It passes
two file descriptors for handling the `gmake`'s job server in child processes, but they are
unsed in the method.

It turns out that there are good reasons to do this -- depending on the multiprocessing
backend, these file descriptors may be closed in the child if they're not passed
directly to it.

- [x] Document all args to `_setup_pkg_and_run`.
- [x] Document all arguments to `_setup_pkg_and_run`.
- [x] Add type hints for `_setup_pkg_and_run`.
- [x] Refactor exception handling in `_setup_pkg_and_run` so it's easier to add type
      hints. `exc_info()` was problematic because it *can* return `None` (just not
      in the context where it's used).  `mypy` was too dumb to notice this.

Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
2024-11-18 12:47:11 +01:00
Harmen Stoppels
cf4e9cb3b3 installer.py: handle external roots the same (#44917)
There was logic not to enqueue build requests for externals if they occur as
roots. That's unnecessary.
2024-11-18 12:47:11 +01:00
Harmen Stoppels
d189b12050 installer: improve init signature and explicits (#44374)
Change the installer to take `([pkg], args)` in the constructor instead
of `[(pkg, args)]`. The reason is that certain arguments are global
settings, and the new API ensures that those arguments cannot be
different across different "build requests".

The `explicit` install arg is now a list of hashes, and the installer is
no longer responsible for determining what package is installed
explicitly. This way environment installs can simply pass the list of
environment roots, without them necessarily being explicit build
requests. For example an env with two roots [a, b], where b depends on
a, would not always cause spack install to mark b as explicit.

Notice that `overwrite` already took a list of hashes, this makes
`explicit` consistent.

`package.do_install(explicit=True)` continues to take a boolean.
2024-11-18 12:47:11 +01:00
Massimiliano Culpo
5134504fd8 Fix broken spack find -u (#47102)
fixes #47101

The bug was introduced in #33495, where `spack find was not updated,
and wasn't caught by unit tests.

Now a Database can accept a custom predicate to select the installation
records. A unit test is added to prevent regressions. The weird convention
of having `any` as a default value has been replaced by the more commonly
used `None`.

Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2024-11-18 12:47:11 +01:00
Harmen Stoppels
ae5018ee09 python: drop build-tools tag (#46980)
Remove the `build-tools` tag of python, otherwise these types of
concretizations are possible:

```
py-root
  ^py-pip
    ^python@3.12
  ^python@3.13
```

So, a package would be configured with py-pip using python 3.12, but
installed for 3.13, which does not work.
2024-11-18 12:47:11 +01:00
Harmen Stoppels
106ebeb502 Show underlying errors on fetch failure (#45714)
- unwrap/flatten nested exceptions
- improve tests
- unify curl lookup
2024-11-18 12:47:11 +01:00
Harmen Stoppels
ba89754ee1 buildcache: recognize . and .. as paths instead of names (#47105) 2024-11-18 12:47:11 +01:00
Harmen Stoppels
e733eb0fd9 docs: do not promote build_systems/* at all (#47111) 2024-11-18 12:47:11 +01:00
Harmen Stoppels
07b344bf10 docs: tune ranking further (#47110)
promote hand-written docs, demote generated "docs" for sources, modules, packages.
2024-11-18 12:47:11 +01:00
Harmen Stoppels
a71c65399e docs search: rank api lowest and generated commands low (#47107) 2024-11-18 12:47:11 +01:00
Harmen Stoppels
d8b73331f6 bootstrap: remove all system gnupg/patchelf executables (#47165) 2024-11-18 12:47:11 +01:00
Peter Scheibel
4741ea683c avoid double closing of fd in sub-processes (#47035)
Both `multiprocessing.connection.Connection.__del__` and `io.IOBase.__del__` called `os.close` on the same file descriptor. As of Python 3.13, this is an explicit warning. Ensure we close once by usef `os.fdopen(..., closefd=False)`
2024-11-18 12:47:11 +01:00
Harmen Stoppels
f33c18290b tests: fix wrong install name (#46711) 2024-11-18 12:47:11 +01:00
Massimiliano Culpo
5ea67e8882 archspec: update to v0.2.5 (#46958)
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2024-11-18 12:47:11 +01:00
Massimiliano Culpo
8ade071253 Bump archspec 0.2.5-dev (#46503)
Use commit bceb39528ac49dd0c876b2e9bf3e7482e9c2be4a
2024-11-18 12:47:11 +01:00
Tobias Ribizel
dca09e6e0f env depfile: generate Makefile with absolute script path (#46966)
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2024-11-18 12:47:11 +01:00
Harmen Stoppels
2776402c90 Update release documentation (#46991) 2024-11-18 12:47:11 +01:00
Harmen Stoppels
3961a86f86 support python 3.13 bootstrapping from sources (#46983) 2024-11-18 12:47:11 +01:00
Harmen Stoppels
39e594d096 clingo: fix build with Python 3.13+ (#46775)
Python 3.9 deprecated `PyEval_InitThreads` and kept it as a no-op, and
Python 3.13 deleted the function.

The patch removes calls to it.
2024-11-18 12:47:11 +01:00
Harmen Stoppels
7a91bed5c9 Set version to v0.22.3.dev0 2024-11-18 12:47:11 +01:00
Harmen Stoppels
594a376c52 Set version to v0.22.2 2024-09-21 12:39:55 +02:00
Harmen Stoppels
1538c48616 run-unit-tests: no xdist if coverage (#46480)
xdist only slows down unit tests under coverage
2024-09-21 12:39:55 +02:00
Massimiliano Culpo
683e50b8d9 Run unit test in parallel again in CI (#45793)
The --trace-config option was failing for linux unit-tests,
so we were running serial.
2024-09-21 12:39:55 +02:00
Harmen Stoppels
9b32fb0beb Revert "Change environment modifications to escape with double quotes (#36789)" (#42780)
This reverts commit 690394fabc, as it causes arbitrary code execution.
2024-09-21 12:39:55 +02:00
Harmen Stoppels
2c6df0d491 deal with TimeoutError from ssl.py (#45683) 2024-09-21 12:39:55 +02:00
Harmen Stoppels
ce7218acae buildcache: fix hard-coded, outdated layout version (#45645) 2024-09-21 12:39:55 +02:00
Dominic Hofer
246eeb2b69 Remove execution permission from setup-env.sh (#45641)
`setup-env.sh` is meant to be sourced, not executed directly.
By revoking execution permissions, users who accidentally execute
the script will receive an error instead of seeing no effect.

* Remove execution permission from `setup-env.sh` and friends
* Don't make output file executable in `spack commands --update-completion`

---------

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
2024-09-21 12:39:55 +02:00
Harmen Stoppels
cc47ee3984 unparser.py: remove print statements (#45235) 2024-09-21 12:39:55 +02:00
Harmen Stoppels
7b644719c1 Avoid duplicate detectable tag (#45160)
in case of inheritance the static tags prop may be updated multiple
times, and it turns out builder classes magically inherit from
traditional package classes
2024-09-21 12:39:55 +02:00
Harmen Stoppels
d8a6aa551e build_environment: explicitly disable ccache if disabled (#45275) 2024-09-21 12:39:55 +02:00
Massimiliano Culpo
ac7b18483a Bump archspec to latest commit (#46445)
This should fix an issue with Neoverse XX detection

Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2024-09-21 12:39:55 +02:00
Massimiliano Culpo
39f37de4ce Update archspec to v0.2.5-dev (7e6740012b897ae4a950f0bba7e9726b767e921f) (#45721) 2024-09-21 12:39:55 +02:00
Harmen Stoppels
703e153404 require spec in develop entry (#46485) 2024-09-21 12:39:55 +02:00
Harmen Stoppels
aa013611bc url join: fix oci scheme (#46483)
* url.py: also special case oci scheme in join

* avoid fetching keys from oci mirror
2024-09-21 12:39:55 +02:00
Harmen Stoppels
6a7ccd4e46 docs: refer to upstreams.yaml in chain.rst title (#46475) 2024-09-21 12:39:55 +02:00
Harmen Stoppels
1c6c4b4690 spack.util.url: fix join breakage in python 3.12.6 (#46453) 2024-09-21 12:39:55 +02:00
arezaii
68558b3dd0 Chapel package: updates post release (#45304)
* Fix +rocm variant, to ensure correct dependencies on ROCm packages
  and use of AMD LLVM
* Add a +pshm variant for comm=gasnet to enable fast shared-memory
  comms between co-locales
* Add logic to ensure we get the native CXI libfabric network provider
  on Cray EX
* Expand dependency type for package modules to encompass runtime
  dependencies
* Factor logic for setting (LD_)LIBRARY_PATH and PKG_CONFIG_PATH of
  runtime dependencies
* Workaround issue #44746 that causes a transitive dependency on lua
  to break SLURM
* Disable nonfunctional checkChplDoc test
* Annotate some variants as conditional, to improve spack info output
  and reduce confusion

---------

Co-authored-by: Dan Bonachea <dobonachea@lbl.gov>
2024-09-21 12:39:55 +02:00
arezaii
5440fe09cd update chapel package for v2.1 (#44931) 2024-09-21 12:39:55 +02:00
arezaii
03c22f403f Chapel package: major update (#42197)
* add cray detection taken from upcxx
* add CUDA/ROCm support
* add numerous pass-through options to Chapel build,
  like gpu_mem_strategy, comm_substrate, etc.; all variants are
  translated to analogous CHPL_* environment variables. As a side
  effect, this defines a number of environment variables that are
  not actually used by Chapel.
* Define LD_LIBRARY_PATH, LIBRARY_PATH, and PKG_CONFIG_PATH to
  help programs built with Chapel properly locate needed runtime
  dependencies

---------

Co-authored-by: bonachea <dobonachea@lbl.gov>
2024-09-21 12:39:55 +02:00
Greg Becker
f339225d22 include_concrete: read from older env formats properly (#45766)
* include_concrete: read from older env formats properly
* spack env rm: fix logic for checking env includes
* regression test

Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
2024-09-21 12:39:55 +02:00
Massimiliano Culpo
22c815f3d4 Do not halt concretization on unknown variants in externals (#45326)
* Do not halt concretization on unkwnown variants in externals
2024-09-21 12:39:55 +02:00
Massimiliano Culpo
4354288e44 Run minimization of weights only on known targets (#45269)
This prevents excessive output from clingo of the kind:

.../spack/lib/spack/spack/solver/concretize.lp:1640:5-11: info: tuple ignored:
  #sup@2
2024-09-21 12:39:55 +02:00
Massimiliano Culpo
ea2d43b4a6 Do not initialize previous store state in "use_store" (#45268)
The "use_store" context manager is used to swap the value
of a global variable (spack.store.STORE), while keeping
another global variable consistent (spack.config.CONFIG).

When doing that it tries to evaluate the previous value
of the store, if that was not done already. This is wrong,
since the configuration might be in an "intermediate" state
that was never meant to trigger side effects.

Remove that operation, and add a unit test to
prevent regressions.
2024-09-21 12:39:55 +02:00
Massimiliano Culpo
85e67d60a0 Add compatibility of sequoia with previous macOS versions (#45127)
* Add compatibility of sequoia with previous macOS versions

* Add compatibility of sequoia with previous macOS versions
2024-09-21 12:39:55 +02:00
Adam J. Stewart
bf6a9ff5ed Add support for macOS Sequoia (#45018) 2024-09-21 12:39:55 +02:00