## Summary
Compilers stop being a *node attribute*, and become a *build-only* dependency.
Packages may declare a dependency on the `c`, `cxx`, or `fortran` languages, which
are now treated as virtuals, and compilers would be *providers* for one or more of
those languages. Compilers can also inject runtime dependency, on the node being
compiled. An example graph for something as simple as `zlib-ng` is the following:
<p align="center">
<img src="https://github.com/user-attachments/assets/ee6471cb-09fd-4127-9f16-b9fe6d1338ac" alt="zlib-ng DAG" width="80%" height="auto">
</p>
Here `gcc` is used for both the `c`, and `cxx` languages. Edges are annotated with
the virtuals they satisfy (`c`, `cxx`, `libc`). `gcc` injects `gcc-runtime` on the nodes
being compiled. `glibc` is also injected for packages that require `c`. The
`compiler-wrapper` is explicitly represented as a node in the DAG, and is included in
the hash.
This change in the model has implications on the semantics of the `%` sigil, as
discussed in #44379, and requires a version bump for our `Specfile`, `Database`,
and `Lockfile` formats.
## Breaking changes
Breaking changes below may impact users of this branch.
### 1. Custom, non-numeric version of compilers are not supported
Currently, users can assign to compilers any custom version they want, and Spack
will try to recover the "real version" whenever the custom version fails some operation.
To deduce the "real version" Spack must run the compiler, which can add needless
overhead to common operations.
Since any information that a version like `gcc@foo` might give to the user, can also
be suffixed while retaining the correct numeric version, e.g. `gcc@10.5.0-foo`, Spack
will **not try** anymore to deduce real versions for compilers.
Said otherwise, users should have no expectation that `gcc@foo` behaves as
`gcc@X.Y.Z` internally.
### 2. The `%` sigil in the spec syntax means "direct build dependency"
The `%` sigil in the spec syntax means *"direct build dependency"*, and is not a node
attribute anymore. This means that:
```python
node.satisfies("%gcc")
```
is true only if `gcc` is a direct build dependency of the node. *Nodes without a compiler
dependency are allowed.*
### `parent["child"]`, and `node in spec`, will now only inspect the link/run sub-DAG
and direct build dependencies
The subscript notation for `Spec`:
```python
parent["child"]
```
will look for a `child` node only in the link/run transitive graph of `parent`, and in its
direct build dependencies. This means that to reach a transitive build dependency,
we must first pass through the node it is associated with.
Assuming `parent` does not depend on `cmake`, but depends on a `CMakePackage`,
e.g. `hdf5`, then we have the following situation:
```python
# This one raises an Exception, since "parent" does not depend on cmake
parent["cmake"]
# This one is ok
cmake = parent["hdf5"]["cmake"]
```
### 3. Externals differing by just the compiler attribute
Externals are nodes where dependencies are trimmed, and that _is not planned to
change_ in this branch. Currently, on `develop` it is ok to write:
```yaml
packages:
hdf5:
externals:
- spec: hdf5@1.12 %gcc
prefix: /prefix/gcc
- spec: hdf5@1.12 %clang
prefix: /prefix/clang
```
and Spack will account for the compiler node attribute when computing the optimal
spec. In this branch, using externals with a compiler specified is allowed only if any
compiler in the dag matches the constraints specified on the external. _The external
will be still represented as a single node without dependencies_.
### 4. Spec matrices enforcing a compiler
Currently we can have matrices of the form:
```yaml
matrix:
- [x, y, z]
- [%gcc, %clang]
```
to get the cross-product of specs and compilers. We can disregard the nature of the
packages in the first row, since the compiler is a node attribute required on each node.
In this branch, instead, we require a spec to depend on `c`, `cxx`, or `fortran` for the
`%` to have any meaning. If any of the specs in the first row doesn't depend on these
languages, there will be a concretization error.
## Deprecations
* The entire `compilers` section in the configuration (i.e., `compilers.yaml`) has been
deprecated, and current entries will be removed in v1.2.0. For the time being, if Spack
finds any `compilers` configuration, it will try to convert it automatically to a set of
external packages.
* The `packages:compiler` soft-preference has been deprecated. It will be removed
in v1.1.0.
## Other notable changes
* The tokens `{compiler}`, `{compiler.version}`, and `{compiler.name}` in `Spec.format`
expand to `"none"` if a Spec does not depend on C, C++, or Fortran.
* The default install tree layout is now
`"{architecture.platform}-{architecture.target}/{name}-{version}-{hash}"`
## Known limitations
The major known limitations of this branch that we intend to fix before v1.0 is that compilers
cannot be bootstrapped directly.
In this branch we can build a new compiler using an existing external compiler, for instance:
```
$ spack install gcc@14 %gcc@10.5.0
```
where `gcc@10.5.0` is external, and `gcc@14` is to be built.
What we can't do at the moment is use a yet to be built compiler, and expect it will be
bootstrapped, e.g. :
```
spack install hdf5 %gcc@14
```
We plan to tackle this issue in a following PR.
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Signed-off-by: Harmen Stoppels <me@harmenstoppels.nl>
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment).
This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention.
As part of this change, the module roots configuration moved from the `config` section to inside each module configuration.
Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec.
TODO:
- [x] code changes to support multiple module sets
- [x] code changes to support modules relative to a view
- [x] Tests for multiple module configurations
- [x] Tests for modules relative to a view
- [x] Backwards compatibility for module roots from config section
- [x] Backwards compatibility for default module set without the name specified
- [x] Tests for backwards compatibility
The YAML config for paths and modules of external packages has
changed: the new format allows a single spec to load multiple
modules. Spack will automatically convert from the old format
when reading the configs (the updates do not add new essential
properties, so this change in Spack is backwards-compatible).
With this update, Spack cannot modify existing configs/environments
without updating them (e.g. “spack config add” will fail if the
configuration is in a format that predates this PR). The user is
prompted to do this explicitly and commands are provided. All
config scopes can be updated at once. Each environment must be
updated one at a time.
* Separate Apple Clang from LLVM Clang
Apple Clang is a compiler of its own. All places
referring to "-apple" suffix have been updated.
* Hack to use a dash in 'apple-clang'
To be able to use autodoc from Sphinx we need
a valid Python name for the module that contains
Apple's Clang code.
* Updated packages to account for the existence of apple-clang
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Added unit test for XCode related functions
Co-authored-by: Gregory Becker <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This commit removes the DYLD_LIBRARY_PATH variable from the default
modules.yaml for darwin. The rationale behind deleting this
environment variable is that paths in this environment variable take
precedence over the default locations of libraries (usually the
install path of the library), which can lead to linking errors in some
circumstances. For example, executables intended to link with Apple's
system BLAS and LAPACK will instead link to a spack-installed
implementation (e.g., OpenBLAS), causing runtime errors.
These errors are resolved by instead relying on paths set in
DYLD_FALLBACK_LIBRARY_PATH, which is lower in precedence than default
locations of libraries.
This commit adds default unwind providers to the default packages.yaml
for darwin. Compiler versions are supplied with the apple-unwind
package so that apple-unwind is only used with Apple's clang fork, and
not with LLVM's clang.
