This commit adds a layer of indirection to improve build isolation with
and without external Python, as well as usability of environment views.
It adds `python-venv` as a dependency to all packages that `extends("python")`,
which has the following advantages:
1. Build isolation: only `PYTHONPATH` is considered in builds, not
user / system packages
2. Stable install layout: fixes the problem on Debian, RHEL and Fedora where
external / system python produces `bin/local` subdirs in Spack install prefixes.
3. Environment views are Python virtual environments (and if you add
`py-pip` things like `pip list` work)
Views work whether they're symlink, hardlink or copy type.
This commit additionally makes `spec["python"].command` return
`spec["python-venv"].command`. The rationale is that packages in repos we do
not own do not pass the underlying python to the build system, which could still
result in incorrectly computed install layouts.
Other attributes like `libs`, `headers` should be on `python` anyways and need no change.
Currently bootstrapping from source fails because clingo requires gnupg
requires clingo.
This commit stops eager bootstrapping. We don't need `patchelf` nor `gnupg`
generally. They're bootstrapped when needed.
This creates shared infrastructure for compiler packages to implement the
detailed search capabilities from the `spack compiler find` command for the
`spack external find` command.
After this commit, `spack compiler find` can be replaced with
`spack external find --tag compiler`, with the exception of mixed toolchains.
A named env cannot contain `.` and `/`.
So when a user runs `spack env create ./here` do not error but treat it
as `spack env create -d ./here`.
Also fix help string of `spack env create`, which seems to have been
copied from `activate` incorrectly.
Since reuse is the default now, `--reuse-deps` can be confusing, as it
technically does not imply roots are fresh.
So add `--fresh-roots`, which is also easier to discover when running
`spack concretize --fre<tab>`
We recently switched to using the new ReadTheDocs with "addons". That includes its own
analytics, which is nice, but we also want to continue using our GA4 analytics.
Adding GA4 is no longer supported by RTD, so we have to add it manually.
- [x] re-add the gtag to all pages, manually
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Adds a pre-concretization check for the Windows SDK and WGL (Windows
GL) packages as non-buildable externals.
This is a redo of https://github.com/spack/spack/pull/43459, but makes
sure to modify the configuration scope outside of the bootstrap scope:
whichever is highest-precedence in the user's environment at the time
the concretization runs, which should either be an env scope or the
~ scope.
Adds pytest fixture mocking the check for WGL and WSDK as if they were
present.
This PR gives users finer control over which specs are reused during concretization.
The value of the `concretizer:reuse` config option now can take an object with the following properties:
- `roots`: true if reusing roots, false if reusing just dependencies
- `exclude`: list of constraints used to select reusable specs
- `include`: list of constraints used to select reusable specs
- `from`: allows to select the sources of reused specs
### Examples
#### Reuse only specs compiled with GCC
```yaml
concretizer:
reuse:
roots: true
include:
- "%gcc"
```
#### `openmpi` must be used from externals, and it must be the only external used
```yaml
concretizer:
reuse:
roots: true
from:
- type: local
exclude:
- "openmpi"
- type: buildcache
exclude:
- "openmpi"
- type: external
include:
- "openmpi"
```
* PackageStillNeededError: add pkg that needs spec to exception msg
* PackageStillNeededError: f-string with short fmt and hash
* PackageStillNeededError: split long string
The old concretizer creates a cyclic graph when expanding virtuals for
`iconv`, which is a bug. This hack drops glibc and musl as possible
providers for `iconv` in the old concretizer to work around it.
Add debug log for external detection tests. The debug log
is used to print which test is being executed.
Skip version audit on Windows where appropriate
We run `extend spack_flags_list SPACK_LDFLAGS` for `$mode in ld|ccld`.
That's problematic, cause `ccld` needs `-Wl,--flag` whereas `ld` needs
`--flag` directly. Only `-L` and `-l` are common to compiler & linker.
In all build systems `LDFLAGS` is for the compiler not the linker, cause
any linker flag `-x` can be passed as a compiler flag `-Wl,-x`, and there
are many compiler flags that affect the linker invocation, like `-fopenmp`,
`-fuse-ld=`, `-fsanitize=` etc.
So don't pass `LDFLAGS` to the linker directly.
This way users can set `ldflags: -Wl,--allow-shlib-undefined` in compilers.yaml
to work around an issue where the linker tries to resolve the `libcuda.so.1`
stub lib which cannot be located by design in `cuda`.
Currently the concretizer fails if you reuse a git ref version
that has already been installed but modify the spec at all.
See #38484 for futher diagnosis
The issue here is that since there is no established provenance for
these versions the highest weight they are currently assigned is
that of prior install. Re-use checks then fail because the weight of
the version is identical to the solver.
Ironically, these versions are given the highest weights possible when
specified on the CLI for the first time. They should only appear in a
DAG if they are an exact match or if the user specifies them at the CLI.
Therefore it makes sense to preserve their higher ordering.
Getting this right is critical to moving all branch based versions to a pinned
git-ref in the future.
Some packages can't be redistributed in source or binary form. We need an explicit way to say that in a package.
This adds a `redistribute()` directive so that package authors can write, e.g.:
```python
redistribute(source=False, binary=False)
```
You can also do this conditionally with `when=`, as with other directives, e.g.:
```python
# 12.0 and higher are proprietary
redistribute(source=False, binary=False, when="@12.0:")
# can't redistribute when we depend on some proprietary dependency
redistribute(source=False, binary=False, when="^proprietary-dependency")
```
To prevent Spack from adding either their sources or binaries to public mirrors and build caches. You can still unconditionally add things *if* you run either:
* `spack mirror create --private`
* `spack buildcache push --private`
But the default behavior for build caches is not to include non-redistributable packages in either mirrors or build caches. We have previously done this manually for our public buildcache, but with this we can start maintaining redistributability directly in packages.
Caveats: currently the default for `redistribute()` is `True` for both `source` and `binary`, and you can only set either of them to `False` via this directive.
- [x] add `redistribute()` directive
- [x] add `redistribute_source` and `redistribute_binary` class methods to `PackageBase`
- [x] add `--private` option to `spack mirror`
- [x] add `--private` option to `spack buildcache push`
- [x] test exclusion of packages from source mirror (both as a root and as a dependency)
- [x] test exclusion of packages from binary mirror (both as a root and as a dependency)
