Commit Graph

7540 Commits

Author SHA1 Message Date
Joe
c764400338
Allowing environment variables to be set in a spack.yaml (#47587)
This adds a new configuration section called `env_vars:` that can be set in an environment.

It looks very similar to the existing `environment:` section that can be added to `modules.yaml`,
but it is global for an entire spack environment. It's called `env_vars:` to deconflate it with spack
environments (the term was too overloaded).

The syntax looks like this:

```yaml
spack:
  specs:
    - cmake%gcc
  env_vars:
    set:
      ENVAR_SET_IN_ENV_LOAD: "True"
```

Any of our standard environment modifications can be added to the `env_vars` section, e.g.
`prepend_path:`, `unset:`, `append_path:`, etc.  Operations in `env_vars:` are performed
on `spack env activate` and undone on `spack env deactivate`.
2025-01-27 12:20:22 -08:00
Alec Scott
ebc9f03dda
Go build system: reduce resulting installation sizes (#47943)
* Reduce the size of outputted go built binaries
* Remove unused import from go package
* go: remove comment from setup dependents build env
* Add back missing imports after rebase
2025-01-27 11:32:56 -08:00
Alec Scott
8ac0bd2825
GoPackage: respect -j concurrency (#48421) 2025-01-27 12:58:29 -05:00
Tara Drwenski
cc9e0137df
Initialize deque with path string (#48734) 2025-01-27 17:42:29 +01:00
Harmen Stoppels
c6202842ed
Improve error handling in urlopen / socket read (#48707)
* Backward compat with Python 3.9 for socket.timeout
* Forward compat with Python [unknown] as HTTPResponse.geturl is deprecated
* Catch timeout etc from .read()
* Some minor simplifications: json.load(...) takes file object in binary mode.
* Fix CDash code which does error handling wrong: non-2XX responses raise.
2025-01-27 16:59:05 +01:00
Greg Becker
0ec00a9c9a
rewiring.py: eliminate code duplication from bindist (#48723) 2025-01-27 10:09:04 +01:00
psakievich
5e3020ad02
filesystem.py: fix recursive_mtime_greater_than (#48718) 2025-01-26 08:29:29 +01:00
Harmen Stoppels
25a3e8ba59
Remove unused Tokenizer.full_match (#48650) 2025-01-24 15:53:42 +01:00
Juan Miguel Carceller
f95e27a159
SIPPackage: add gmake as a build dependency to be able to build py-pyqt5 (#48660) 2025-01-23 15:51:59 +01:00
Massimiliano Culpo
2db654bf5a
Define the DB index file name in a single place (#48681) 2025-01-23 12:17:34 +01:00
Harmen Stoppels
9992b563db
Improve spack.build_systems typing (#48590) 2025-01-23 12:15:43 +01:00
Harmen Stoppels
daba1a805e
build_environment.py: clear CONFIG_SITE for configure scripts (#48690)
Should be sufficient to set CONFIG_SITE to /dev/null before running configure to 
prevent any config site file to be loaded, which currently causes non-determinism in builds.

Setting the variable CONFIG_SITE prevents configure files to check $prefix/share/config.site, 
$prefix/etc/config.site, $ac_default_prefix/share/config.site, $ac_default_prefix/etc/config.site 
so we don't have to patch a block of code.
2025-01-23 11:45:52 +01:00
Tara Drwenski
518572e710
Use gcc toolchain when using hip and gcc host compiler (#48632) 2025-01-22 23:37:24 -08:00
jnhealy2
aba0a740c2
Address quoting issue impacting dev_paths containing @ symbols (#48555)
* Address quoting issue that casuses dev_paths containing @ symbols to parse as versions

* Fix additional location were dev_path was imporperly constructed

The dev_path must be quoted to avoid parsing issues when
paths contain '@' symbols. Also add tests to catch
regression of this behavior

* Format to fix line length

* fix failing tests

* Fix whitespace error

* Add binary_compatibility fixture to test

* Change string formatting to avoid multiline f string

* Update lib/spack/spack/test/concretization/core.py

* Add tmate debug session

* Revert "Add tmate debug session"

This reverts commit 24e2f77e3c.

* Move test and refactor to use env methods

* Update lib/spack/spack/test/cmd/develop.py

---------

Co-authored-by: psakievich <psakiev@sandia.gov>
2025-01-22 18:03:13 +01:00
Harmen Stoppels
0fe8e763c3
bootstrap: do not show disabled sources as errors if enabled sources fail (#48676) 2025-01-22 17:02:23 +01:00
Harmen Stoppels
0e2d261b7e
bootstrap/status.py: remove make (#48677) 2025-01-22 17:00:25 +01:00
Harmen Stoppels
87a83db623
autotools.py: set lt_cv_apple_cc_single_mod=yes (#48671)
Since macOS 15 `ld -single_module` warns with a deprecation message,
which makes configure scripts believe the flag is unsupported. That
in turn triggers a code path where `archive_cmds` is set to

```
$CC -r -keep_private_externs -nostdlib ... -dynamiclib
```

instead of just

```
$CC -dynamiclib ...
```

This code path was meant to trigger only on ancient macOS <= 14.4 where
libtool had to add `-single_module`, which is the default since macos
14.4, and is now apparently deprecated because the flag is a no-op for
more than 15 years.

The wrong `archive_cmds` causes actual problems combined with a bug in
OpenMPI's compiler wrapper (`CC=mpicc`), which appends `-rpath` flags,
which cause an error when combined with the `-r` flag added by the
autotools.

Spack's compiler wrapper doesn't do this, but it's likely there are
other compiler wrappers out there that are not aware that `-r` and
`-rpath` cannot be combined.

The fix is to change defaults: `lt_cv_apple_cc_single_mod=yes`.
2025-01-22 11:29:50 +01:00
Harmen Stoppels
70df460fa7
PackageBase.detect_dev_src_change: speed up (#48618) 2025-01-21 16:48:37 +01:00
Harmen Stoppels
31a1b2fd6c
relocate.py, binary_distribution.py: cleanup (#48651) 2025-01-21 15:45:08 +01:00
Harmen Stoppels
ecd14f0ad9
archive.py: fix include_parent_directories=True with path_to_name (#48570) 2025-01-20 10:21:32 +01:00
Massimiliano Culpo
783eccfbd5
jsonschema: use draft7 validator and simplify schemas (#48621) 2025-01-20 09:51:29 +01:00
Harmen Stoppels
7e41288ca6
spack_yaml.py / spec.py: use dict instead of OrderedDict (#48616)
* for config: let `syaml_dict` inherit from `dict` instead of `OrderedDict`. `syaml_dict` now only exists as a mutable wrapper for yaml related metadata.
* for spec serialization / hashing: use `dict` directly

This is possible since we only support cpython 3.6+ in which dicts are ordered.

This improves performance of hash computation a bit. For a larger spec I'm getting 9.22ms instead of 11.9ms, so 22.5% reduction in runtime.
2025-01-20 17:33:44 +09:00
Tara Drwenski
3bb375a47f
Change llvm-amdgpu to a build dependency of rocm (#48612) 2025-01-20 17:28:15 +09:00
Harmen Stoppels
962498095d
compat with mypy 1.14 (#48626)
* OSError.errno apparently is int | None, but already part of __str__ anyway so drop

* fix disambiguate_spec_from_hashes

* List[Spec].sort() complains, ignore it

* fix typing of KnownCompiler, and use it in asp.py
2025-01-18 21:16:05 +01:00
John Gouwar
9f7cff1780
Stable splice-topo order for resolving splicing (#48605)
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2025-01-17 16:27:16 +01:00
Harmen Stoppels
bb43fa5444
spec.py: make hashing of extra_attributes order independent (#48615) 2025-01-17 13:50:36 +01:00
Massimiliano Culpo
59a71959e7
Fix wrong hash in spliced specs, and improve unit-tests (#48574)
Modifications:
* Fix a severe bug where a spliced spec has the same hash as the original build spec
* Removed CacheManager in favor of install_mockery. The latter sets up a temporary store, and removes it at the end 
   of the test.
* Deleted a couple of helper functions e.g. _has_dependencies
* Checked that SolverException is raised, rather than Exception (more specific)
* Extended, and renamed the splicing_setup fixture (now called install_specs)
* Added more specific assertions in each test

One test is currently flaky, due to some instability when sorting multiple specs. It's currently marked xfail
2025-01-15 19:43:55 +01:00
Harmen Stoppels
00e804a94b
package.py: expose types of common globals unconditionally (#48588) 2025-01-15 17:33:38 +01:00
Massimiliano Culpo
6fac041d40
Add type-hints to which and which_string (#48554) 2025-01-15 15:24:18 +01:00
Jordan Galby
c504304d39
Add version attributes up_to_1, up_to_2, up_to_3 (#38410)
Making them also available in spec format string
2025-01-15 13:07:17 +01:00
Harmen Stoppels
976f1c2198
flag_mixing.py: add missing import (#48579) 2025-01-15 10:48:34 +01:00
Greg Becker
e7c591a8b8
Deprecate Spec.concretize/Spec.concretized in favor of spack.concretize.concretize_one (#47971)
The methods spack.spec.Spec.concretize and spack.spec.Spec.concretized
are deprecated in favor of spack.concretize.concretize_one.

This will resolve a circular dependency between the spack.spec and
spack.concretize in the next Spack release.
2025-01-15 10:13:19 +01:00
jclause
335fca7049
Improve support using external modules with zsh (#48115)
* Improve support using external modules with zsh

Spack integrates with external modules by launching a python subprocess
to scrape the path from the module command. In zsh, subshells do not
inheret functions defined in the parent shell (only environment
variables). This breaks the module function in module_cmd.py.

As a workaround, source the module commands using $MODULESHOME prior to
running the module command.

* Fix formatting

* Fix flake error

* Add guard around sourcing module file

* Add improved unit testing to module src command

* Correct style

* Another attempt at style

* formatting again

* formatting again

---------

Co-authored-by: psakievich <psakiev@sandia.gov>
2025-01-14 21:29:15 -07:00
Greg Becker
2447d16e55
bugfix: associate nodes directly with specs for compiler flag reordering (#48496) 2025-01-14 10:15:57 -08:00
Harmen Stoppels
d72b371c8a
relocate_text: fix return value (#48568) 2025-01-14 13:29:06 +01:00
Todd Gamblin
d89ae7bcde
Database: make constructor safe for read-only filesystems (#48454)
`spack find` and other commands that read the DB will fail if the databse is in a
read-only location, because `FailureTracker` and `Database` can try to create their
parent directories when they are constructed, even if the DB root is read-only.

Instead, we should only write to the filesystem when needed.

- [x] don't create parent directories in constructors
- [x] have write operations create parents if needed
- [x] in tests, `chmod` databases to be read-only unless absolutely needed.

---------

Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
2025-01-14 00:10:51 -08:00
Harmen Stoppels
53d1665a8b
traverse: use deque and popleft (#48353)
Avoids a quadratic time operation. In practice it is not really felt, cause it has a tiny constant.
But maybe in very large DAGs it is.
2025-01-14 00:09:57 -08:00
Cody Balos
66e8523e14
add long_spec property to get fully enumerated spec string (#48389) 2025-01-13 16:43:39 -08:00
Tamara Dahlgren
f556ba46d9
bugfix/r: git property needs to be classproperty (#48228) 2025-01-13 12:14:26 -06:00
Harmen Stoppels
064e70990d
binary_distribution.py: stop relocation old /bin/bash sbang shebang (#48502) 2025-01-13 09:30:18 +01:00
Massimiliano Culpo
8ab6f33eb6
Deprecate make, gmake and ninja globals in package API (#48450)
Instead, depends_on("gmake", type="build") should be used, which makes the global available through `setup_dependent_package`.
2025-01-11 22:43:22 +01:00
Massimiliano Culpo
dd69b646ad
Add subscript notation to packages (#48467)
This PR allows using the subscript notation directly on packages. The
intent is to reduce the boilerplate needed to retrieve package
properties from nodes other than root.
2025-01-10 19:00:51 +01:00
Harmen Stoppels
d6d8800466
executable.py: fix overlapping overload set (#48503) 2025-01-10 16:34:04 +00:00
Harmen Stoppels
d46ac9b1e4
spec.py: fix return type of concretized() (#48504) 2025-01-10 13:31:41 +00:00
Harmen Stoppels
7edb525599
binary_distribution: improve deps_to_relocate (#48484) 2025-01-10 11:32:11 +01:00
Harmen Stoppels
93cd216603
binary_distribution: stop relocating tarballs with relative rpaths (#48488) 2025-01-10 11:30:18 +01:00
Harmen Stoppels
c1d385ada2
traverse: use overload for return type when depth=True vs depth=False (#48482) 2025-01-10 09:53:28 +01:00
Massimiliano Culpo
5085f635dd
Add type hints to spack.util.executable.Executable (#48468)
* Add type-hints to `spack.util.executable.Executable`
* Add type-hint to input
* Use overload, and remove assertions at calling sites
* Bump mypy to v1.11.2 (working locally), Python to 3.13
2025-01-09 14:16:24 -08:00
dependabot[bot]
e76677cbd5
build(deps): bump pygments from 2.18.0 to 2.19.1 in /lib/spack/docs (#48431)
Bumps [pygments](https://github.com/pygments/pygments) from 2.18.0 to 2.19.1.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.18.0...2.19.1)

---
updated-dependencies:
- dependency-name: pygments
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-01-09 11:31:42 +01:00
Harmen Stoppels
c424611010
directory_layout.py: remove dependency on global spack.store.STORE (#48460) 2025-01-08 17:44:32 +01:00