2022-01-18 01:54:00 +08:00
|
|
|
# -------------------------------------------------------------------------
|
|
|
|
|
# This is the default spack configuration file.
|
|
|
|
|
#
|
|
|
|
|
# Settings here are versioned with Spack and are intended to provide
|
|
|
|
|
# sensible defaults out of the box. Spack maintainers should edit this
|
|
|
|
|
# file to keep it current.
|
|
|
|
|
#
|
|
|
|
|
# Users can override these settings by editing
|
|
|
|
|
# `$SPACK_ROOT/etc/spack/concretizer.yaml`, `~/.spack/concretizer.yaml`,
|
|
|
|
|
# or by adding a `concretizer:` section to an environment.
|
|
|
|
|
# -------------------------------------------------------------------------
|
|
|
|
|
concretizer:
|
|
|
|
|
# Whether to consider installed packages or packages from buildcaches when
|
|
|
|
|
# concretizing specs. If `true`, we'll try to use as many installs/binaries
|
|
|
|
|
# as possible, rather than building. If `false`, we'll always give you a fresh
|
2023-03-14 16:22:20 +08:00
|
|
|
# concretization. If `dependencies`, we'll only reuse dependencies but
|
|
|
|
|
# give you a fresh concretization for your root specs.
|
2024-04-23 23:42:14 +08:00
|
|
|
reuse: true
|
2022-04-26 08:19:51 +08:00
|
|
|
# Options that tune which targets are considered for concretization. The
|
|
|
|
|
# concretization process is very sensitive to the number targets, and the time
|
|
|
|
|
# needed to reach a solution increases noticeably with the number of targets
|
|
|
|
|
# considered.
|
|
|
|
|
targets:
|
2023-03-10 02:51:11 +08:00
|
|
|
# Determine whether we want to target specific or generic
|
|
|
|
|
# microarchitectures. Valid values are: "microarchitectures" or "generic".
|
|
|
|
|
# An example of "microarchitectures" would be "skylake" or "bulldozer",
|
|
|
|
|
# while an example of "generic" would be "aarch64" or "x86_64_v4".
|
2022-04-26 08:19:51 +08:00
|
|
|
granularity: microarchitectures
|
|
|
|
|
# If "false" allow targets that are incompatible with the current host (for
|
|
|
|
|
# instance concretize with target "icelake" while running on "haswell").
|
|
|
|
|
# If "true" only allow targets that are compatible with the host.
|
2022-05-14 00:11:10 +08:00
|
|
|
host_compatible: true
|
2022-05-24 04:20:34 +08:00
|
|
|
# When "true" concretize root specs of environments together, so that each unique
|
|
|
|
|
# package in an environment corresponds to one concrete spec. This ensures
|
|
|
|
|
# environments can always be activated. When "false" perform concretization separately
|
|
|
|
|
# on each root spec, allowing different versions and variants of the same package in
|
|
|
|
|
# an environment.
|
2023-03-10 02:51:11 +08:00
|
|
|
unify: true
|
2023-06-29 17:26:25 +08:00
|
|
|
# Option to deal with possible duplicate nodes (i.e. different nodes from the same package) in the DAG.
|
|
|
|
|
duplicates:
|
|
|
|
|
# "none": allows a single node for any package in the DAG.
|
Add solver capability for synthesizing splices of ABI compatible packages. (#46729)
This PR provides complementary 2 features:
1. An augmentation to the package language to express ABI compatibility relationships among packages.
2. An extension to the concretizer that can synthesize splices between ABI compatible packages.
1. The `can_splice` directive and ABI compatibility
We augment the package language with a single directive: `can_splice`. Here is an example of a package `Foo` exercising the `can_splice` directive:
class Foo(Package):
version("1.0")
version("1.1")
variant("compat", default=True)
variant("json", default=False)
variant("pic", default=False)
can_splice("foo@1.0", when="@1.1")
can_splice("bar@1.0", when="@1.0+compat")
can_splice("baz@1.0+compat", when="@1.0+compat", match_variants="*")
can_splice("quux@1.0", when=@1.1~compat", match_variants="json")
Explanations of the uses of each directive:
- `can_splice("foo@1.0", when="@1.1")`: If `foo@1.0` is the dependency of an already installed spec and `foo@1.1` could be a valid dependency for the parent spec, then `foo@1.1` can be spliced in for `foo@1.0` in the parent spec.
- `can_splice("bar@1.0", when="@1.0+compat")`: If `bar@1.0` is the dependency of an already installed spec and `foo@1.0+compat` could be a valid dependency for the parent spec, then `foo@1.0+compat` can be spliced in for `bar@1.0+compat` in the parent spec
- `can_splice("baz@1.0", when="@1.0+compat", match_variants="*")`: If `baz@1.0+compat` is the dependency of an already installed spec and `foo@1.0+compat` could be a valid dependency for the parent spec, then `foo@1.0+compat` can be spliced in for `baz@1.0+compat` in the parent spec, provided that they have the same value for all other variants (regardless of what those values are).
- `can_splice("quux@1.0", when=@1.1~compat", match_variants="json")`:If `quux@1.0` is the dependency of an already installed spec and `foo@1.1~compat` could be a valid dependency for the parent spec, then `foo@1.0~compat` can be spliced in for `quux@1.0` in the parent spec, provided that they have the same value for their `json` variant.
2. Augmenting the solver to synthesize splices
### Changes to the hash encoding in `asp.py`
Previously, when including concrete specs in the solve, they would have the following form:
installed_hash("foo", "xxxyyy")
imposed_constraint("xxxyyy", "foo", "attr1", ...)
imposed_constraint("xxxyyy", "foo", "attr2", ...)
% etc.
Concrete specs now have the following form:
installed_hash("foo", "xxxyyy")
hash_attr("xxxyyy", "foo", "attr1", ...)
hash_attr("xxxyyy", "foo", "attr2", ...)
This transformation allows us to control which constraints are imposed when we select a hash, to facilitate the splicing of dependencies.
2.1 Compiling `can_splice` directives in `asp.py`
Consider the concrete spec:
foo@2.72%gcc@11.4 arch=linux-ubuntu22.04-icelake build_system=autotools ^bar ...
It will emit the following facts for reuse (below is a subset)
installed_hash("foo", "xxxyyy")
hash_attr("xxxyyy", "hash", "foo", "xxxyyy")
hash_attr("xxxyyy", "version", "foo", "2.72")
hash_attr("xxxyyy", "node_os", "ubuntu22.04")
hash_attr("xxxyyy", "hash", "bar", "zzzqqq")
hash_attr("xxxyyy", "depends_on", "foo", "bar", "link")
Rules that derive abi_splice_conditions_hold will be generated from
use of the `can_splice` directive. They will have the following form:
can_splice("foo@1.0.0+a", when="@1.0.1+a", match_variants=["b"]) --->
abi_splice_conditions_hold(0, node(SID, "foo"), "foo", BaseHash) :-
installed_hash("foo", BaseHash),
attr("node", node(SID, SpliceName)),
attr("node_version_satisfies", node(SID, "foo"), "1.0.1"),
hash_attr("hash", "node_version_satisfies", "foo", "1.0.1"),
attr("variant_value", node(SID, "foo"), "a", "True"),
hash_attr("hash", "variant_value", "foo", "a", "True"),
attr("variant_value", node(SID, "foo"), "b", VariVar0),
hash_attr("hash", "variant_value", "foo", "b", VariVar0).
2.2 Synthesizing splices in `concretize.lp` and `splices.lp`
The ASP solver generates "splice_at_hash" attrs to indicate that a particular node has a splice in one of its immediate dependencies.
Splices can be introduced in the dependencies of concrete specs when `splices.lp` is conditionally loaded (based on the config option `concretizer:splice:True`.
2.3 Constructing spliced specs in `asp.py`
The method `SpecBuilder._resolve_splices` implements a top-down memoized implementation of hybrid splicing. This is an optimization over the more general `Spec.splice`, since the solver gives a global view of exactly which specs can be shared, to ensure the minimal number of splicing operations.
Misc changes to facilitate configuration and benchmarking
- Added the method `Solver.solve_with_stats` to expose timers from the public interface for easier benchmarking
- Added the boolean config option `concretizer:splice` to conditionally load splicing behavior
Co-authored-by: Greg Becker <becker33@llnl.gov>
2024-11-13 12:51:19 +08:00
|
|
|
# "minimal": allows the duplication of 'build-tools' nodes only
|
|
|
|
|
# (e.g. py-setuptools, cmake etc.)
|
2023-06-29 17:26:25 +08:00
|
|
|
# "full" (experimental): allows separation of the entire build-tool stack (e.g. the entire "cmake" subDAG)
|
2023-10-06 16:24:21 +08:00
|
|
|
strategy: minimal
|
2025-02-14 21:25:12 +08:00
|
|
|
# Maximum number of duplicates in a DAG, when using a strategy that allows duplicates. "default" is the
|
|
|
|
|
# number used if there isn't a more specific alternative
|
|
|
|
|
max_dupes:
|
|
|
|
|
default: 1
|
|
|
|
|
# Virtuals
|
|
|
|
|
c: 2
|
|
|
|
|
cxx: 2
|
|
|
|
|
fortran: 1
|
|
|
|
|
# Regular packages
|
|
|
|
|
cmake: 2
|
|
|
|
|
gmake: 2
|
2025-02-27 07:04:31 +08:00
|
|
|
python: 2
|
|
|
|
|
python-venv: 2
|
2025-02-14 21:25:12 +08:00
|
|
|
py-cython: 2
|
|
|
|
|
py-flit-core: 2
|
2025-02-27 07:04:31 +08:00
|
|
|
py-pip: 2
|
2025-02-14 21:25:12 +08:00
|
|
|
py-setuptools: 2
|
2025-02-27 07:04:31 +08:00
|
|
|
py-wheel: 2
|
|
|
|
|
xcb-proto: 2
|
|
|
|
|
# Compilers
|
2025-02-14 21:25:12 +08:00
|
|
|
gcc: 2
|
|
|
|
|
llvm: 2
|
2024-10-30 16:35:30 +08:00
|
|
|
# Option to specify compatibility between operating systems for reuse of compilers and packages
|
2024-10-31 05:32:54 +08:00
|
|
|
# Specified as a key: [list] where the key is the os that is being targeted, and the list contains the OS's
|
|
|
|
|
# it can reuse. Note this is a directional compatibility so mutual compatibility between two OS's
|
2024-03-27 23:39:07 +08:00
|
|
|
# requires two entries i.e. os_compatible: {sonoma: [monterey], monterey: [sonoma]}
|
|
|
|
|
os_compatible: {}
|
Add solver capability for synthesizing splices of ABI compatible packages. (#46729)
This PR provides complementary 2 features:
1. An augmentation to the package language to express ABI compatibility relationships among packages.
2. An extension to the concretizer that can synthesize splices between ABI compatible packages.
1. The `can_splice` directive and ABI compatibility
We augment the package language with a single directive: `can_splice`. Here is an example of a package `Foo` exercising the `can_splice` directive:
class Foo(Package):
version("1.0")
version("1.1")
variant("compat", default=True)
variant("json", default=False)
variant("pic", default=False)
can_splice("foo@1.0", when="@1.1")
can_splice("bar@1.0", when="@1.0+compat")
can_splice("baz@1.0+compat", when="@1.0+compat", match_variants="*")
can_splice("quux@1.0", when=@1.1~compat", match_variants="json")
Explanations of the uses of each directive:
- `can_splice("foo@1.0", when="@1.1")`: If `foo@1.0` is the dependency of an already installed spec and `foo@1.1` could be a valid dependency for the parent spec, then `foo@1.1` can be spliced in for `foo@1.0` in the parent spec.
- `can_splice("bar@1.0", when="@1.0+compat")`: If `bar@1.0` is the dependency of an already installed spec and `foo@1.0+compat` could be a valid dependency for the parent spec, then `foo@1.0+compat` can be spliced in for `bar@1.0+compat` in the parent spec
- `can_splice("baz@1.0", when="@1.0+compat", match_variants="*")`: If `baz@1.0+compat` is the dependency of an already installed spec and `foo@1.0+compat` could be a valid dependency for the parent spec, then `foo@1.0+compat` can be spliced in for `baz@1.0+compat` in the parent spec, provided that they have the same value for all other variants (regardless of what those values are).
- `can_splice("quux@1.0", when=@1.1~compat", match_variants="json")`:If `quux@1.0` is the dependency of an already installed spec and `foo@1.1~compat` could be a valid dependency for the parent spec, then `foo@1.0~compat` can be spliced in for `quux@1.0` in the parent spec, provided that they have the same value for their `json` variant.
2. Augmenting the solver to synthesize splices
### Changes to the hash encoding in `asp.py`
Previously, when including concrete specs in the solve, they would have the following form:
installed_hash("foo", "xxxyyy")
imposed_constraint("xxxyyy", "foo", "attr1", ...)
imposed_constraint("xxxyyy", "foo", "attr2", ...)
% etc.
Concrete specs now have the following form:
installed_hash("foo", "xxxyyy")
hash_attr("xxxyyy", "foo", "attr1", ...)
hash_attr("xxxyyy", "foo", "attr2", ...)
This transformation allows us to control which constraints are imposed when we select a hash, to facilitate the splicing of dependencies.
2.1 Compiling `can_splice` directives in `asp.py`
Consider the concrete spec:
foo@2.72%gcc@11.4 arch=linux-ubuntu22.04-icelake build_system=autotools ^bar ...
It will emit the following facts for reuse (below is a subset)
installed_hash("foo", "xxxyyy")
hash_attr("xxxyyy", "hash", "foo", "xxxyyy")
hash_attr("xxxyyy", "version", "foo", "2.72")
hash_attr("xxxyyy", "node_os", "ubuntu22.04")
hash_attr("xxxyyy", "hash", "bar", "zzzqqq")
hash_attr("xxxyyy", "depends_on", "foo", "bar", "link")
Rules that derive abi_splice_conditions_hold will be generated from
use of the `can_splice` directive. They will have the following form:
can_splice("foo@1.0.0+a", when="@1.0.1+a", match_variants=["b"]) --->
abi_splice_conditions_hold(0, node(SID, "foo"), "foo", BaseHash) :-
installed_hash("foo", BaseHash),
attr("node", node(SID, SpliceName)),
attr("node_version_satisfies", node(SID, "foo"), "1.0.1"),
hash_attr("hash", "node_version_satisfies", "foo", "1.0.1"),
attr("variant_value", node(SID, "foo"), "a", "True"),
hash_attr("hash", "variant_value", "foo", "a", "True"),
attr("variant_value", node(SID, "foo"), "b", VariVar0),
hash_attr("hash", "variant_value", "foo", "b", VariVar0).
2.2 Synthesizing splices in `concretize.lp` and `splices.lp`
The ASP solver generates "splice_at_hash" attrs to indicate that a particular node has a splice in one of its immediate dependencies.
Splices can be introduced in the dependencies of concrete specs when `splices.lp` is conditionally loaded (based on the config option `concretizer:splice:True`.
2.3 Constructing spliced specs in `asp.py`
The method `SpecBuilder._resolve_splices` implements a top-down memoized implementation of hybrid splicing. This is an optimization over the more general `Spec.splice`, since the solver gives a global view of exactly which specs can be shared, to ensure the minimal number of splicing operations.
Misc changes to facilitate configuration and benchmarking
- Added the method `Solver.solve_with_stats` to expose timers from the public interface for easier benchmarking
- Added the boolean config option `concretizer:splice` to conditionally load splicing behavior
Co-authored-by: Greg Becker <becker33@llnl.gov>
2024-11-13 12:51:19 +08:00
|
|
|
|
|
|
|
|
# Option to specify whether to support splicing. Splicing allows for
|
|
|
|
|
# the relinking of concrete package dependencies in order to better
|
|
|
|
|
# reuse already built packages with ABI compatible dependencies
|
|
|
|
|
splice:
|
|
|
|
|
explicit: []
|
|
|
|
|
automatic: false
|
2024-11-19 22:00:26 +08:00
|
|
|
# Maximum time, in seconds, allowed for the 'solve' phase. If set to 0, there is no time limit.
|
|
|
|
|
timeout: 0
|
|
|
|
|
# If set to true, exceeding the timeout will always result in a concretization error. If false,
|
|
|
|
|
# the best (suboptimal) model computed before the timeout is used.
|
|
|
|
|
#
|
|
|
|
|
# Setting this to false yields unreproducible results, so we advise to use that value only
|
|
|
|
|
# for debugging purposes (e.g. check which constraints can help Spack concretize faster).
|
|
|
|
|
error_on_timeout: true
|
concretizer: reduce search space with static analysis (#48729)
Currently, when we setup the ASP problem for `clingo`, we don't take into account the configuration. This results in setting up ASP problems that are larger than necessary, with possibly redundant information, and higher concretization times.
This PR tries to improve things by adding an opt-in feature that computes the _possible dependencies_ of a solve taking also into account the current configuration, and avoids adding possible dependencies that we are certain can't be in the final solution.
The feature can be activated with:
```yaml
concretizer:
static_analysis: true
```
Examples of simple rules to discard dependencies are:
- Dependencies that are not buildable, and for which no binary is present (e.g. `cray-mpich` etc. on non Cray systems)
- Dependencies that are not for the current platform (e.g. `msmpi` on non Windows platforms)
- Conditional dependencies that cannot be activated, because of some user requirement (e.g. `cuda` etc. if the user requires `~cuda` in configuration)
- Virtual providers that cannot be used, because of a requirement on a virtual
The speed-up these rules seem to give depends on the use case at hand, but if the configuration is updated properly, they are noticeable.
Since in cases where there is no rule to exclude packages upfront, reuse is active, and this option is activated, it's possible to see some minor slow down, the feature has been added as opt-in, so it's turned off by default.
2025-02-11 15:44:20 +08:00
|
|
|
|
|
|
|
|
# Static analysis may reduce the concretization time by generating smaller ASP problems, in
|
|
|
|
|
# cases where there are requirements that prevent part of the search space to be explored.
|
|
|
|
|
static_analysis: false
|