Add the ability to include any number of (potentially nested) concrete environments, e.g.:
```yaml
spack:
specs: []
concretizer:
unify: true
include_concrete:
- /path/to/environment1
- /path/to/environment2
```
or, from the CLI:
```console
$ spack env create myenv
$ spack -e myenv add python
$ spack -e myenv concretize
$ spack env create --include-concrete myenv included_env
```
The contents of included concrete environments' spack.lock files are
included in the environment's lock file at creation time. Any changes
to included concrete environments are only reflected after the environment
is re-concretized from the re-concretized included environments.
- [x] Concretize included envs
- [x] Save concrete specs in memory by hash
- [x] Add included envs to combined env's lock file
- [x] Add test
- [x] Update documentation
Co-authored-by: Kayla Butler <<butler59@llnl.gov>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.co
m>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
This commit adds a layer of indirection to improve build isolation with
and without external Python, as well as usability of environment views.
It adds `python-venv` as a dependency to all packages that `extends("python")`,
which has the following advantages:
1. Build isolation: only `PYTHONPATH` is considered in builds, not
user / system packages
2. Stable install layout: fixes the problem on Debian, RHEL and Fedora where
external / system python produces `bin/local` subdirs in Spack install prefixes.
3. Environment views are Python virtual environments (and if you add
`py-pip` things like `pip list` work)
Views work whether they're symlink, hardlink or copy type.
This commit additionally makes `spec["python"].command` return
`spec["python-venv"].command`. The rationale is that packages in repos we do
not own do not pass the underlying python to the build system, which could still
result in incorrectly computed install layouts.
Other attributes like `libs`, `headers` should be on `python` anyways and need no change.
This creates shared infrastructure for compiler packages to implement the
detailed search capabilities from the `spack compiler find` command for the
`spack external find` command.
After this commit, `spack compiler find` can be replaced with
`spack external find --tag compiler`, with the exception of mixed toolchains.
A named env cannot contain `.` and `/`.
So when a user runs `spack env create ./here` do not error but treat it
as `spack env create -d ./here`.
Also fix help string of `spack env create`, which seems to have been
copied from `activate` incorrectly.
We recently switched to using the new ReadTheDocs with "addons". That includes its own
analytics, which is nice, but we also want to continue using our GA4 analytics.
Adding GA4 is no longer supported by RTD, so we have to add it manually.
- [x] re-add the gtag to all pages, manually
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Adds a pre-concretization check for the Windows SDK and WGL (Windows
GL) packages as non-buildable externals.
This is a redo of https://github.com/spack/spack/pull/43459, but makes
sure to modify the configuration scope outside of the bootstrap scope:
whichever is highest-precedence in the user's environment at the time
the concretization runs, which should either be an env scope or the
~ scope.
Adds pytest fixture mocking the check for WGL and WSDK as if they were
present.
This PR gives users finer control over which specs are reused during concretization.
The value of the `concretizer:reuse` config option now can take an object with the following properties:
- `roots`: true if reusing roots, false if reusing just dependencies
- `exclude`: list of constraints used to select reusable specs
- `include`: list of constraints used to select reusable specs
- `from`: allows to select the sources of reused specs
### Examples
#### Reuse only specs compiled with GCC
```yaml
concretizer:
reuse:
roots: true
include:
- "%gcc"
```
#### `openmpi` must be used from externals, and it must be the only external used
```yaml
concretizer:
reuse:
roots: true
from:
- type: local
exclude:
- "openmpi"
- type: buildcache
exclude:
- "openmpi"
- type: external
include:
- "openmpi"
```
On Windows, bootstrapping logic now searches for and adds the win-sdk
and wgl packages to the user's top scope as externals if they are not
present.
These packages are generally required to install most packages with
Spack on Windows, and are only available as externals, so it is
assumed that doing this automatically would be useful and avoid
a mandatory manual step for each new Spack instance.
Note this is the first case of bootstrapping logic modifying
configuration other than the bootstrap configuration.
This commit adds a property `autopush` to mirrors. When true, every source build is immediately followed by a push to the build cache. This is useful in ephemeral environments such as CI / containers.
To enable autopush on existing build caches, use `spack mirror set --autopush <name>`. The same flag can be used in `spack mirror add`.
This PR allows the user to specify a path to a custom cert file (or directory) in
Spack's config:
```yaml
# This is where custom certs for proxy/firewall are stored.
# It can be a path or environment variable. To match ssl env configuration
# the default is the environment variable SSL_CERT_FILE
ssl_certs: $SSL_CERT_FILE
```
`config:ssl_certs` can be a path to a file or a directory, or it can be and environment
variable that resolves to one of those. When it posts to something valid, Spack will
update the ssl context to include custom certs, and fetching via `urllib` and `curl`
will trust the provided certs.
This should resolve many issues with fetching behind corporate firewalls.
---------
Co-authored-by: psakievich <psakievich@users.noreply.github.com>
Co-authored-by: Alec Scott <alec@bcs.sh>
This adds support for prereleases. Alpha, beta and release candidate
suffixes are ordered in the intuitive way:
```
1.2.0-alpha < 1.2.0-alpha.1 < 1.2.0-beta.2 < 1.2.0-rc.3 < 1.2.0 < 1.2.0-xyz
```
Alpha, beta and rc prereleases are defined as follows: split the version
string into components like before (on delimiters and string boundaries).
If there's a string component `alpha`, `beta` or `rc` followed by an optional
numeric component at the end, then the version is prerelease.
So `1.2.0-alpha.1 == 1.2.0alpha1 == 1.2.0.alpha1` are all the same, as usual.
The strings `alpha`, `beta` and `rc` are chosen because they match semver,
they are sufficiently long to be unambiguous, and and all contain at least
one non-hex character so distinguish them from shasum/digest type suffixes.
The comparison key is now stored as `(release_tuple, prerelease_tuple)`, so in
the above example:
```
((1,2,0),(ALPHA,)) < ((1,2,0),(ALPHA,1)) < ((1,2,0),(BETA,2)) < ((1,2,0),(RC,3)) < ((1,2,0),(FINAL,)) < ((1,2,0,"xyz"), (FINAL,))
```
The version ranges `@1.2.0:` and `@:1.1` do *not* include prereleases of
`1.2.0`.
So for packaging, if the `1.2.0alpha` and `1.2.0` versions have the same constraints on
dependencies, it's best to write
```python
depends_on("x@1:", when="@1.2.0alpha:")
```
However, `@1.2:` does include `1.2.0alpha`. This is because Spack considers
`1.2 < 1.2.0` as distinct versions, with `1.2 < 1.2.0alpha < 1.2.0` as a consequence.
Alternatively, the above `depends_on` statement can thus be written
```python
depends_on("x@1:", when="@1.2:")
```
which can be useful too. A short-hand to include prereleases, but you
can still be explicit to exclude the prerelease by specifying the patch version
number.
### Concretization
Concretization uses a different version order than `<`. Prereleases are ordered
between final releases and develop versions. That way, users should not
have to set `preferred=True` on every final release if they add just one
prerelease to a package. The concretizer is unlikely to pick a prerelease when
final releases are possible.
### Limitations
1. You can't express a range that includes all alpha release but excludes all beta
releases. Only alternative is good old repeated nines: `@:1.2.0alpha99`.
2. The Python ecosystem defaults to `a`, `b`, `rc` strings, so translation of Python versions to
Spack versions requires expansion to `alpha`, `beta`, `rc`. It's mildly annoying, because
this means we may need to compute URLs differently (not done in this commit).
### Hash
Care is taken not to break hashes of versions that do not have a prerelease
suffix.
Closes#43052.
Maybe moving the argument to the `find` subcommand is a good idea, but I
just wanted to get the docs fix out.
Co-authored-by: Patrice Peterson <patrice.peterson@itz.uni-halle.de>
This PR adds the ability to load spack extensions through `importlib.metadata` entry
points, in addition to the regular configuration variable.
It requires Python 3.8 or greater to be properly supported.
These 7 hooks were not used.
- Six of them related to install phases were unused after `spack`
`monitor` was removed, and the code seems to have bit rotten as there
were reports they were not (always?) triggered when they should.
- The post environment one was made redundant after spack install for
environment started following the common code path for generating
module files in #42147.
It should not be a breaking change to remove, since users cannot define
hooks in extensions, they would have to fork Spack.
If we ever _were_ to make those hooks extendable outside of core Spack,
it would also be better to start with fewer rather than more, cause
everything you expose gets relied upon...
Removing those also allows us to rethink what hooks we really need, and
in particular it seems like we need a hook that runs post install also when
the spec is inserted into the database.
The section was highly outdated as it referred to old defaults, and
failed to mention `hide_implicits: true`.
This commit restructures it, moves some deeply nested sections a level
up, and promotes `hide_implicits: true` + `autoload: direct` before
talking about `exclude`.
