spack/etc/spack/defaults/config.yaml
Massimiliano Culpo 5b3942a489
Turn compilers into nodes (#45189)
## Summary

Compilers stop being a *node attribute*, and become a *build-only* dependency. 

Packages may declare a dependency on the `c`, `cxx`, or `fortran` languages, which
are now treated as virtuals, and compilers would be *providers* for one or more of
those languages. Compilers can also inject runtime dependency, on the node being
compiled. An example graph for something as simple as `zlib-ng` is the following:

<p align="center">
<img src="https://github.com/user-attachments/assets/ee6471cb-09fd-4127-9f16-b9fe6d1338ac" alt="zlib-ng DAG" width="80%" height="auto">
</p>

Here `gcc` is used for both the `c`, and `cxx` languages. Edges are annotated with
the virtuals they satisfy (`c`, `cxx`, `libc`). `gcc` injects `gcc-runtime` on the nodes
being compiled. `glibc` is also injected for packages that require `c`. The
`compiler-wrapper` is explicitly represented as a node in the DAG, and is included in
the hash.

This change in the model has implications on the semantics of the `%` sigil, as
discussed in #44379, and requires a version bump for our `Specfile`, `Database`,
and `Lockfile` formats.

## Breaking changes

Breaking changes below may impact users of this branch.

### 1. Custom, non-numeric version of compilers are not supported

Currently, users can assign to compilers any custom version they want, and Spack
will try to recover the "real version" whenever the custom version fails some operation.
To deduce the "real version" Spack must run the compiler, which can add needless
overhead to common operations.

Since any information that a version like `gcc@foo` might give to the user, can also
be suffixed while retaining the correct numeric version, e.g. `gcc@10.5.0-foo`, Spack
will **not try** anymore to deduce real versions for compilers.

Said otherwise, users should have no expectation that `gcc@foo` behaves as
`gcc@X.Y.Z` internally.

### 2. The `%` sigil in the spec syntax means "direct build dependency"

The `%` sigil in the spec syntax means *"direct build dependency"*, and is not a node
attribute anymore. This means that:

```python
node.satisfies("%gcc")
``` 
is true only if `gcc` is a direct build dependency of the node. *Nodes without a compiler
dependency are allowed.*

### `parent["child"]`, and `node in spec`, will now only inspect the link/run sub-DAG
and direct build dependencies

The subscript notation for `Spec`:

```python
parent["child"]
```

will look for a `child` node only in the link/run transitive graph of `parent`, and in its
direct build dependencies. This means that to reach a transitive build dependency,
we must first pass through the node it is associated with. 

Assuming `parent` does not depend on `cmake`, but depends on a `CMakePackage`,
e.g. `hdf5`, then we have the following situation:

```python
# This one raises an Exception, since "parent" does not depend on cmake
parent["cmake"]
# This one is ok
cmake = parent["hdf5"]["cmake"]
```

### 3. Externals differing by just the compiler attribute

Externals are nodes where dependencies are trimmed, and that _is not planned to
change_ in this branch. Currently, on `develop` it is ok to write:

```yaml
packages:
  hdf5:
    externals:
    - spec: hdf5@1.12 %gcc
      prefix: /prefix/gcc
    - spec: hdf5@1.12 %clang
      prefix: /prefix/clang
```
and Spack will account for the compiler node attribute when computing the optimal
spec. In this branch, using externals with a compiler specified is allowed only if any
compiler in the dag matches the constraints specified on the external. _The external
will be still represented as a single node without dependencies_.

### 4. Spec matrices enforcing a compiler

Currently we can have matrices of the form:

```yaml
matrix:
- [x, y, z]
- [%gcc, %clang]
```
to get the cross-product of specs and compilers. We can disregard the nature of the
packages in the first row, since the compiler is a node attribute required on each node.

In this branch, instead, we require a spec to depend on `c`, `cxx`, or `fortran` for the
`%` to have any meaning. If any of the specs in the first row doesn't depend on these
languages, there will be a concretization error. 

## Deprecations

* The entire `compilers` section in the configuration (i.e., `compilers.yaml`) has been
  deprecated, and current entries will be removed in v1.2.0. For the time being, if Spack
  finds any `compilers` configuration, it will try to convert it automatically to a set of
  external packages.
* The `packages:compiler` soft-preference has been deprecated. It will be removed
  in v1.1.0.

## Other notable changes

* The tokens `{compiler}`, `{compiler.version}`, and `{compiler.name}` in `Spec.format`
  expand to `"none"` if a Spec does not depend on C, C++, or Fortran.
* The default install tree layout is now
  `"{architecture.platform}-{architecture.target}/{name}-{version}-{hash}"`

## Known limitations

The major known limitations of this branch that we intend to fix before v1.0 is that compilers
cannot be bootstrapped directly. 

In this branch we can build a new compiler using an existing external compiler, for instance:
	
```
$ spack install gcc@14 %gcc@10.5.0
```

where `gcc@10.5.0` is external, and `gcc@14` is to be built.

What we can't do at the moment is use a yet to be built compiler, and expect it will be
bootstrapped, e.g. :

```
spack install hdf5 %gcc@14
```

We plan to tackle this issue in a following PR.

---------

Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Signed-off-by: Harmen Stoppels <me@harmenstoppels.nl>
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2025-03-25 22:32:49 -06:00

229 lines
9.2 KiB
YAML

# -------------------------------------------------------------------------
# This is the default spack configuration file.
#
# Settings here are versioned with Spack and are intended to provide
# sensible defaults out of the box. Spack maintainers should edit this
# file to keep it current.
#
# Users can override these settings by editing the following files.
#
# Per-spack-instance settings (overrides defaults):
# $SPACK_ROOT/etc/spack/config.yaml
#
# Per-user settings (overrides default and site settings):
# ~/.spack/config.yaml
# -------------------------------------------------------------------------
config:
# This is the path to the root of the Spack install tree.
# You can use $spack here to refer to the root of the spack instance.
install_tree:
root: $spack/opt/spack
projections:
all: "{architecture.platform}-{architecture.target}/{name}-{version}-{hash}"
# install_tree can include an optional padded length (int or boolean)
# default is False (do not pad)
# if padded_length is True, Spack will pad as close to the system max path
# length as possible
# if padded_length is an integer, Spack will pad to that many characters,
# assuming it is higher than the length of the install_tree root.
# padded_length: 128
# Locations where templates should be found
template_dirs:
- $spack/share/spack/templates
# Directory where licenses should be located
license_dir: $spack/etc/spack/licenses
# Temporary locations Spack can try to use for builds.
#
# Recommended options are given below.
#
# Builds can be faster in temporary directories on some (e.g., HPC) systems.
# Specifying `$tempdir` will ensure use of the default temporary directory
# (i.e., ``$TMP` or ``$TMPDIR``).
#
# Another option that prevents conflicts and potential permission issues is
# to specify `$user_cache_path/stage`, which ensures each user builds in their
# home directory.
#
# A more traditional path uses the value of `$spack/var/spack/stage`, which
# builds directly inside Spack's instance without staging them in a
# temporary space. Problems with specifying a path inside a Spack instance
# are that it precludes its use as a system package and its ability to be
# pip installable.
#
# In Spack environment files, chaining onto existing system Spack
# installations, the $env variable can be used to download, cache and build
# into user-writable paths that are relative to the currently active
# environment.
#
# In any case, if the username is not already in the path, Spack will append
# the value of `$user` in an attempt to avoid potential conflicts between
# users in shared temporary spaces.
#
# The build stage can be purged with `spack clean --stage` and
# `spack clean -a`, so it is important that the specified directory uniquely
# identifies Spack staging to avoid accidentally wiping out non-Spack work.
build_stage:
- $tempdir/$user/spack-stage
- $user_cache_path/stage
# - $spack/var/spack/stage
# Directory in which to run tests and store test results.
# Tests will be stored in directories named by date/time and package
# name/hash.
test_stage: $user_cache_path/test
# Cache directory for already downloaded source tarballs and archived
# repositories. This can be purged with `spack clean --downloads`.
source_cache: $spack/var/spack/cache
## Directory where spack managed environments are created and stored
# environments_root: $spack/var/spack/environments
# Cache directory for miscellaneous files, like the package index.
# This can be purged with `spack clean --misc-cache`
misc_cache: $user_cache_path/cache
# Timeout in seconds used for downloading sources etc. This only applies
# to the connection phase and can be increased for slow connections or
# servers. 0 means no timeout.
connect_timeout: 10
# If this is false, tools like curl that use SSL will not verify
# certifiates. (e.g., curl will use use the -k option)
verify_ssl: true
# This is where custom certs for proxy/firewall are stored.
# It can be a path or environment variable. To match ssl env configuration
# the default is the environment variable SSL_CERT_FILE
ssl_certs: $SSL_CERT_FILE
# Suppress gpg warnings from binary package verification
# Only suppresses warnings, gpg failure will still fail the install
# Potential rationale to set True: users have already explicitly trusted the
# gpg key they are using, and may not want to see repeated warnings that it
# is self-signed or something of the sort.
suppress_gpg_warnings: false
# If set to true, Spack will always check checksums after downloading
# archives. If false, Spack skips the checksum step.
checksum: true
# If set to true, Spack will fetch deprecated versions without warning.
# If false, Spack will raise an error when trying to install a deprecated version.
deprecated: false
# If set to true, `spack install` and friends will NOT clean
# potentially harmful variables from the build environment. Use wisely.
dirty: false
# The language the build environment will use. This will produce English
# compiler messages by default, so the log parser can highlight errors.
# If set to C, it will use English (see man locale).
# If set to the empty string (''), it will use the language from the
# user's environment.
build_language: C
# When set to true, concurrent instances of Spack will use locks to
# avoid modifying the install tree, database file, etc. If false, Spack
# will disable all locking, but you must NOT run concurrent instances
# of Spack. For filesystems that don't support locking, you should set
# this to false and run one Spack at a time, but otherwise we recommend
# enabling locks.
locks: true
# The default url fetch method to use.
# If set to 'curl', Spack will require curl on the user's system
# If set to 'urllib', Spack will use python built-in libs to fetch
url_fetch_method: urllib
# The maximum number of jobs to use for the build system (e.g. `make`), when
# the -j flag is not given on the command line. Defaults to 16 when not set.
# Note that the maximum number of jobs is limited by the number of cores
# available, taking thread affinity into account when supported. For instance:
# - With `build_jobs: 16` and 4 cores available `spack install` will run `make -j4`
# - With `build_jobs: 16` and 32 cores available `spack install` will run `make -j16`
# - With `build_jobs: 2` and 4 cores available `spack install -j6` will run `make -j6`
# build_jobs: 16
# If set to true, Spack will use ccache to cache C compiles.
ccache: false
# How long to wait to lock the Spack installation database. This lock is used
# when Spack needs to manage its own package metadata and all operations are
# expected to complete within the default time limit. The timeout should
# therefore generally be left untouched.
db_lock_timeout: 60
# How long to wait when attempting to modify a package (e.g. to install it).
# This value should typically be 'null' (never time out) unless the Spack
# instance only ever has a single user at a time, and only if the user
# anticipates that a significant delay indicates that the lock attempt will
# never succeed.
package_lock_timeout: null
# Control how shared libraries are located at runtime on Linux. See the
# the Spack documentation for details.
shared_linking:
# Spack automatically embeds runtime search paths in ELF binaries for their
# dependencies. Their type can either be "rpath" or "runpath". For glibc, rpath is
# inherited and has precedence over LD_LIBRARY_PATH; runpath is not inherited
# and of lower precedence. DO NOT MIX these within the same install tree.
type: rpath
# (Experimental) Embed absolute paths of dependent libraries directly in ELF
# binaries to avoid runtime search. This can improve startup time of
# executables with many dependencies, in particular on slow filesystems.
bind: false
# Controls the handling of missing dynamic libraries after installation.
# Options are ignore (default), warn, or error. If set to error, the
# installation fails if installed binaries reference dynamic libraries that
# are not found in their specified rpaths.
missing_library_policy: ignore
# Set to 'false' to allow installation on filesystems that doesn't allow setgid bit
# manipulation by unprivileged user (e.g. AFS)
allow_sgid: true
# Whether to show status information during building and installing packages.
# This gives information about Spack's current progress as well as the current
# and total number of packages. Information is shown both in the terminal
# title and inline.
install_status: true
# Number of seconds a buildcache's index.json is cached locally before probing
# for updates, within a single Spack invocation. Defaults to 10 minutes.
binary_index_ttl: 600
flags:
# Whether to keep -Werror flags active in package builds.
keep_werror: 'none'
# A mapping of aliases that can be used to define new commands. For instance,
# `sp: spec -I` will define a new command `sp` that will execute `spec` with
# the `-I` argument. Aliases cannot override existing commands.
aliases:
concretise: concretize
containerise: containerize
rm: remove