Support independent includes with conditional, optional, and remote entries (#48784)
Supersedes #46792.
Closes #40018.
Closes #31026.
Closes #2700.
There were a number of feature requests for os-specific config. This enables os-specific
config without adding a lot of special sub-scopes.
Support `include:` as an independent configuration schema, allowing users to include
configuration scopes from files or directories. Includes can be:
* conditional (similar to definitions in environments), and/or
* optional (i.e., the include will be skipped if it does not exist).
Includes can be paths or URLs (`ftp`, `https`, `http` or `file`). Paths can be absolute or
relative . Environments can include configuration files using the same schema. Remote includes
must be checked by `sha256`.
Includes can also be recursive, and this modifies the config system accordingly so that
we push included configuration scopes on the stack *before* their including scopes, and
we remove configuration scopes from the stack when their including scopes are removed.
For example, you could have an `include.yaml` file (e.g., under `$HOME/.spack`) to specify
global includes:
```
include:
- ./enable_debug.yaml
- path: https://github.com/spack/spack-configs/blob/main/NREL/configs/mac/config.yaml
sha256: 37f982915b03de18cc4e722c42c5267bf04e46b6a6d6e0ef3a67871fcb1d258b
```
Or an environment `spack.yaml`:
```
spack:
include:
- path: "/path/to/a/config-dir-or-file"
when: os == "ventura"
- ./path/relative/to/containing/file/that/is/required
- path: "/path/with/spack/variables/$os/$target"
optional: true
- path: https://raw.githubusercontent.com/spack/spack-configs/refs/heads/main/path/to/required/raw/config.yaml
sha256: 26e871804a92cd07bb3d611b31b4156ae93d35b6a6d6e0ef3a67871fcb1d258b
```
Updated TODO:
- [x] Get existing unit tests to pass with Todd's changes
- [x] Resolve new (or old) circular imports
- [x] Ensure remote includes (global) work
- [x] Ensure remote includes for environments work (note: caches remote
files under user cache root)
- [x] add sha256 field to include paths, validate, and require for remote includes
- [x] add sha256 remote file unit tests
- [x] revisit how diamond includes should work
- [x] support recursive includes
- [x] add recursive include unit tests
- [x] update docs and unit test to indicate ordering of recursive includes with
conflicting options is deferred to follow-on work
---------
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Co-authored-by: Peter Scheibel <scheibel1@llnl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2025-03-09 19:33:44 -07:00
|
|
|
.. Copyright Spack Project Developers. See COPYRIGHT file for details.
|
|
|
|
|
|
|
|
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
|
|
|
|
|
|
|
.. _include-yaml:
|
|
|
|
|
|
|
|
===============================
|
|
|
|
Include Settings (include.yaml)
|
|
|
|
===============================
|
|
|
|
|
|
|
|
Spack allows you to include configuration files through ``include.yaml``.
|
|
|
|
Using the ``include:`` heading results in pulling in external configuration
|
|
|
|
information to be used by any Spack command.
|
|
|
|
|
|
|
|
Included configuration files are required *unless* they are explicitly optional
|
|
|
|
or the entry's condition evaluates to ``false``. Optional includes are specified
|
|
|
|
with the ``optional`` clause and conditional with the ``when`` clause. For
|
|
|
|
example,
|
|
|
|
|
|
|
|
.. code-block:: yaml
|
|
|
|
|
|
|
|
include:
|
|
|
|
- /path/to/a/required/config.yaml
|
|
|
|
- path: /path/to/$os/$target/config
|
|
|
|
optional: true
|
|
|
|
- path: /path/to/os-specific/config-dir
|
|
|
|
when: os == "ventura"
|
|
|
|
|
|
|
|
shows all three. The first entry, ``/path/to/a/required/config.yaml``,
|
|
|
|
indicates that included ``config.yaml`` file is required (so must exist).
|
|
|
|
Use of ``optional: true`` for ``/path/to/$os/$target/config`` means
|
|
|
|
the path is only included if it exists. The condition ``os == "ventura"``
|
|
|
|
in the ``when`` clause for ``/path/to/os-specific/config-dir`` means the
|
|
|
|
path is only included when the operating system (``os``) is ``ventura``.
|
|
|
|
|
|
|
|
The same conditions and variables in `Spec List References
|
|
|
|
<https://spack.readthedocs.io/en/latest/environments.html#spec-list-references>`_
|
|
|
|
can be used for conditional activation in the ``when`` clauses.
|
|
|
|
|
|
|
|
Included files can be specified by path or by their parent directory.
|
|
|
|
Paths may be absolute, relative (to the configuration file including the path),
|
|
|
|
or specified as URLs. Only the ``file``, ``ftp``, ``http`` and ``https`` protocols (or
|
|
|
|
schemes) are supported. Spack-specific, environment and user path variables
|
|
|
|
can be used. (See :ref:`config-file-variables` for more information.)
|
|
|
|
|
2025-04-01 00:02:50 -07:00
|
|
|
A ``sha256`` is required for remote file URLs and must be specified as follows:
|
|
|
|
|
|
|
|
.. code-block:: yaml
|
|
|
|
|
|
|
|
include:
|
|
|
|
- path: https://github.com/path/to/raw/config/compilers.yaml
|
|
|
|
sha256: 26e871804a92cd07bb3d611b31b4156ae93d35b6a6d6e0ef3a67871fcb1d258b
|
|
|
|
|
|
|
|
Additionally, remote file URLs must link to the **raw** form of the file's
|
|
|
|
contents (e.g., `GitHub
|
|
|
|
<https://docs.github.com/en/repositories/working-with-files/using-files/viewing-and-understanding-files#viewing-or-copying-the-raw-file-content>`_
|
|
|
|
or `GitLab
|
|
|
|
<https://docs.gitlab.com/ee/api/repository_files.html#get-raw-file-from-repository>`_).
|
|
|
|
|
Support independent includes with conditional, optional, and remote entries (#48784)
Supersedes #46792.
Closes #40018.
Closes #31026.
Closes #2700.
There were a number of feature requests for os-specific config. This enables os-specific
config without adding a lot of special sub-scopes.
Support `include:` as an independent configuration schema, allowing users to include
configuration scopes from files or directories. Includes can be:
* conditional (similar to definitions in environments), and/or
* optional (i.e., the include will be skipped if it does not exist).
Includes can be paths or URLs (`ftp`, `https`, `http` or `file`). Paths can be absolute or
relative . Environments can include configuration files using the same schema. Remote includes
must be checked by `sha256`.
Includes can also be recursive, and this modifies the config system accordingly so that
we push included configuration scopes on the stack *before* their including scopes, and
we remove configuration scopes from the stack when their including scopes are removed.
For example, you could have an `include.yaml` file (e.g., under `$HOME/.spack`) to specify
global includes:
```
include:
- ./enable_debug.yaml
- path: https://github.com/spack/spack-configs/blob/main/NREL/configs/mac/config.yaml
sha256: 37f982915b03de18cc4e722c42c5267bf04e46b6a6d6e0ef3a67871fcb1d258b
```
Or an environment `spack.yaml`:
```
spack:
include:
- path: "/path/to/a/config-dir-or-file"
when: os == "ventura"
- ./path/relative/to/containing/file/that/is/required
- path: "/path/with/spack/variables/$os/$target"
optional: true
- path: https://raw.githubusercontent.com/spack/spack-configs/refs/heads/main/path/to/required/raw/config.yaml
sha256: 26e871804a92cd07bb3d611b31b4156ae93d35b6a6d6e0ef3a67871fcb1d258b
```
Updated TODO:
- [x] Get existing unit tests to pass with Todd's changes
- [x] Resolve new (or old) circular imports
- [x] Ensure remote includes (global) work
- [x] Ensure remote includes for environments work (note: caches remote
files under user cache root)
- [x] add sha256 field to include paths, validate, and require for remote includes
- [x] add sha256 remote file unit tests
- [x] revisit how diamond includes should work
- [x] support recursive includes
- [x] add recursive include unit tests
- [x] update docs and unit test to indicate ordering of recursive includes with
conflicting options is deferred to follow-on work
---------
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Co-authored-by: Peter Scheibel <scheibel1@llnl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2025-03-09 19:33:44 -07:00
|
|
|
.. warning::
|
|
|
|
|
|
|
|
Recursive includes are not currently processed in a breadth-first manner
|
|
|
|
so the value of a configuration option that is altered by multiple included
|
|
|
|
files may not be what you expect. This will be addressed in a future
|
|
|
|
update.
|