spack solve: add a --profile option

We often want to know what the solver is thinking about in order to optimize it. We can
generate a profile of which atoms are most frequently propagated and/or undone by the
solver using a custom propagator class.

This adds a `spack.solver.profiler` module with a simple propagator that counts `undo`
and `propagate` operations and associates them with atoms in the solve. To simplify
things, rather than enumerating *every* atom in the logic program, we coarsen to show
only function names for everything but `attr()`, and we show the first argument of any
`attr()` atom (e.g. `attr("node")`).

`spack solve --profile` will print the top 40 most propagated atoms in the program,
along with how many times they are undone (these are usually but not always similar).
The list can give people working with the concretizer a rough idea of where the solver
is spending its time, and hopefully point to areas of the program to optimize.

This is based on some code that Tom Scogland wrote a year or so ago. I did the work of
integrating it as an option so that I do not have to keep looking it up and running this
from the CLI :).

Co-authored-by: Tom Scogland <scogland1@llnl.gov>
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>

CHANGELOG.md

@@ -1,3 +1,365 @@
+# v0.23.0 (2024-11-13)
+
+`v0.23.0` is a major feature release.
+
+We are planning to make this the last major release before Spack `v1.0`
+in June 2025. Alongside `v0.23`, we will be making pre-releases (alpha,
+beta, etc.)  of `v1.0`, and we encourage users to try them and send us
+feedback, either on GitHub or on Slack. You can track the road to
+`v1.0` here:
+
+  * https://github.com/spack/spack/releases
+  * https://github.com/spack/spack/discussions/30634
+
+## Features in this Release
+
+1. **Language virtuals**
+
+   Your packages can now explicitly depend on the languages they require.
+   Historically, Spack has considered C, C++, and Fortran compiler
+   dependencies to be implicit. In `v0.23`, you should ensure that
+   new packages add relevant C, C++, and Fortran dependencies like this:
+
+   ```python
+   depends_on("c", type="build")
+   depends_on("cxx", type="build")
+   depends_on("fortran", type="build")
+   ```
+
+   We encourage you to add these annotations to your packages now, to prepare
+   for Spack `v1.0.0`. In `v1.0.0`, these annotations will be necessary for
+   your package to use C, C++, and Fortran compilers. Note that you should
+   *not* add language dependencies to packages that don't need them, e.g.,
+   pure python packages.
+
+   We have already auto-generated these dependencies for packages in the
+   `builtin` repository (see #45217), based on the types of source files
+   present in each package's source code. We *may* have added too many or too
+   few language dependencies, so please submit pull requests to correct
+   packages if you find that the language dependencies are incorrect.
+
+   Note that we have also backported support for these dependencies to
+   `v0.21.3` and `v0.22.2`, to make all of them forward-compatible with
+   `v0.23`. This should allow you to move easily between older and newer Spack
+   releases without breaking your packages.
+
+2. **Spec splicing**
+
+   We are working to make binary installation more seamless in Spack. `v0.23`
+   introduces "splicing", which allows users to deploy binaries using local,
+   optimized versions of a binary interface, even if they were not built with
+   that interface. For example, this would allow you to build binaries in the
+   cloud using `mpich` and install them on a system using a local, optimized
+   version of `mvapich2` *without rebuilding*. Spack preserves full provenance
+   for the installed packages and knows that they were built one way but
+   deployed another.
+
+   Our intent is to leverage this across many key HPC binary packages,
+   e.g. MPI, CUDA, ROCm, and libfabric.
+
+   Fundamentally, splicing allows Spack to redeploy an existing spec with
+   different dependencies than how it was built. There are two interfaces to
+   splicing.
+
+   a. Explicit Splicing
+
+      #39136 introduced the explicit splicing interface. In the
+      concretizer config, you can specify a target spec and a replacement
+      by hash.
+
+      ```yaml
+      concretizer:
+        splice:
+          explicit:
+          - target: mpi
+            replacement: mpich/abcdef
+      ```
+
+      Here, every installation that would normally use the target spec will
+      instead use its replacement. Above, any spec using *any* `mpi` will be
+      spliced to depend on the specific `mpich` installation requested. This
+      *can* go wrong if you try to replace something built with, e.g.,
+      `openmpi` with `mpich`, and it is on the user to ensure ABI
+      compatibility between target and replacement specs. This currently
+      requires some expertise to use, but it will allow users to reuse the
+      binaries they create across more machines and environments.
+
+   b. Automatic Splicing (experimental)
+
+      #46729 introduced automatic splicing. In the concretizer config, enable
+      automatic splicing:
+
+      ```yaml
+      concretizer:
+        splice:
+          automatic: true
+      ```
+
+      or run:
+
+      ```console
+      spack config add concretizer:splice:automatic:true
+      ```
+
+      The concretizer will select splices for ABI compatibility to maximize
+      package reuse. Packages can denote ABI compatibility using the
+      `can_splice` directive. No packages in Spack yet use this directive, so
+      if you want to use this feature you will need to add `can_splice`
+      annotations to your packages. We are working on ways to add more ABI
+      compatibility information to the Spack package repository, and this
+      directive may change in the future.
+
+   See the documentation for more details:
+   * https://spack.readthedocs.io/en/latest/build_settings.html#splicing
+   * https://spack.readthedocs.io/en/latest/packaging_guide.html#specifying-abi-compatibility
+
+3. Broader variant propagation
+
+   Since #42931, you can specify propagated variants like `hdf5
+   build_type==RelWithDebInfo` or `trilinos ++openmp` to propagate a variant
+   to all dependencies for which it is relevant. This is valid *even* if the
+   variant does not exist on the package or its dependencies.
+
+   See https://spack.readthedocs.io/en/latest/basic_usage.html#variants.
+
+4. Query specs by namespace
+
+   #45416 allows a package's namespace (indicating the repository it came from)
+   to be treated like a variant. You can request packages from particular repos
+   like this:
+
+   ```console
+   spack find zlib namespace=builtin
+   spack find zlib namespace=myrepo
+   ```
+
+   Previously, the spec syntax only allowed namespaces to be prefixes of spec
+   names, e.g. `builtin.zlib`. The previous syntax still works.
+
+5. `spack spec` respects environment settings and `unify:true`
+
+   `spack spec` did not previously respect environment lockfiles or
+   unification settings, which made it difficult to see exactly how a spec
+   would concretize within an environment. Now it does, so the output you get
+   with `spack spec` will be *the same* as what your environment will
+   concretize to when you run `spack concretize`. Similarly, if you provide
+   multiple specs on the command line with `spack spec`, it will concretize
+   them together if `unify:true` is set.
+
+   See #47556 and #44843.
+
+6. Less noisy `spack spec` output
+
+   `spack spec` previously showed output like this:
+
+   ```console
+    > spack spec /v5fn6xo
+    Input spec
+    --------------------------------
+     -   /v5fn6xo
+
+    Concretized
+    --------------------------------
+    [+]  openssl@3.3.1%apple-clang@16.0.0~docs+shared arch=darwin-sequoia-m1
+    ...
+   ```
+
+   But the input spec is redundant, and we know we run `spack spec` to concretize
+   the input spec. `spack spec` now *only* shows the concretized spec. See #47574.
+
+7. Better output for `spack find -c`
+
+   In an environmnet, `spack find -c` lets you search the concretized, but not
+   yet installed, specs, just as you would the installed ones. As with `spack
+   spec`, this should make it easier for you to see what *will* be built
+   before building and installing it. See #44713.
+
+8. `spack -C <env>`: use an environment's configuration without activation
+
+   Spack environments allow you to associate:
+   1. a set of (possibly concretized) specs, and
+   2. configuration
+
+   When you activate an environment, you're using both of these. Previously, we
+   supported:
+   * `spack -e <env>` to run spack in the context of a specific environment, and
+   * `spack -C <directory>` to run spack using a directory with configuration files.
+
+   You can now also pass an environment to `spack -C` to use *only* the environment's
+   configuration, but not the specs or lockfile. See #45046.
+
+## New commands, options, and directives
+
+* The new `spack env track` command (#41897) takes a non-managed Spack
+  environment and adds a symlink to Spack's `$environments_root` directory, so
+  that it will be included for reference counting for commands like `spack
+  uninstall` and `spack gc`. If you use free-standing directory environments,
+  this is useful for preventing Spack from removing things required by your
+  environments. You can undo this tracking with the `spack env untrack`
+  command.
+
+* Add `-t` short option for `spack --backtrace` (#47227)
+
+  `spack -d / --debug` enables backtraces on error, but it can be very
+  verbose, and sometimes you just want the backtrace. `spack -t / --backtrace`
+  provides that option.
+
+* `gc`: restrict to specific specs (#46790)
+
+  If you only want to garbage-collect specific packages, you can now provide
+  them on the command line. This gives users finer-grained control over what
+  is uninstalled.
+
+* oci buildcaches now support `--only=package`. You can now push *just* a
+  package and not its dependencies to an OCI registry. This allows dependents
+  of non-redistributable specs to be stored in OCI registries without an
+  error. See #45775.
+
+## Notable refactors
+* Variants are now fully conditional
+
+  The `variants` dictionary on packages was previously keyed by variant name,
+  and allowed only one definition of any given variant. Spack is now smart
+  enough to understand that variants may have different values and defaults
+  for different versions. For example, `warpx` prior to `23.06` only supported
+  builds for one dimensionality, and newer `warpx` versions could be built
+  with support for many different dimensions:
+
+  ```python
+  variant(
+      "dims",
+      default="3",
+      values=("1", "2", "3", "rz"),
+      multi=False,
+      description="Number of spatial dimensions",
+      when="@:23.05",
+  )
+  variant(
+      "dims",
+      default="1,2,rz,3",
+      values=("1", "2", "3", "rz"),
+      multi=True,
+      description="Number of spatial dimensions",
+      when="@23.06:",
+  )
+  ```
+
+  Previously, the default for the old version of `warpx` was not respected and
+  had to be specified manually. Now, Spack will select the right variant
+  definition for each version at concretization time. This allows variants to
+  evolve more smoothly over time. See #44425 for details.
+
+## Highlighted bugfixes
+
+1. Externals no longer override the preferred provider (#45025).
+
+   External definitions could interfere with package preferences. Now, if
+   `openmpi` is the preferred `mpi`, and an external `mpich` is defined, a new
+   `openmpi` *will* be built if building it is possible. Previously we would
+   prefer `mpich` despite the preference.
+
+2. Composable `cflags` (#41049).
+
+   This release fixes a longstanding bug that concretization would fail if
+   there were different `cflags` specified in `packages.yaml`,
+   `compilers.yaml`, or on `the` CLI. Flags and their ordering are now tracked
+   in the concretizer and flags from multiple sources will be merged.
+
+3. Fix concretizer Unification for included environments (#45139).
+
+## Deprecations, removals, and syntax changes
+
+1. The old concretizer has been removed from Spack, along with the
+   `config:concretizer` config option. Spack will emit a warning if the option
+   is present in user configuration, since it now has no effect. Spack now
+   uses a simpler bootstrapping mechanism, where a JSON prototype is tweaked
+   slightly to get an initial concrete spec to download. See #45215.
+
+2. Best-effort expansion of spec matrices has been removed. This feature did
+   not work with the "new" ASP-based concretizer, and did not work with
+   `unify: True` or `unify: when_possible`. Use the
+   [exclude key](https://spack.readthedocs.io/en/latest/environments.html#spec-matrices)
+   for the environment to exclude invalid components, or use multiple spec
+   matrices to combine the list of specs for which the constraint is valid and
+   the list of specs for which it is not. See #40792.
+
+3. The old Cray `platform` (based on Cray PE modules) has been removed, and
+   `platform=cray` is no longer supported. Since `v0.19`, Spack has handled
+   Cray machines like Linux clusters with extra packages, and we have
+   encouraged using this option to support Cray. The new approach allows us to
+   correctly handle Cray machines with non-SLES operating systems, and it is
+   much more reliable than making assumptions about Cray modules. See the
+   `v0.19` release notes and #43796 for more details.
+
+4. The `config:install_missing_compilers` config option has been deprecated,
+   and it is a no-op when set in `v0.23`. Our new compiler dependency model
+   will replace it with a much more reliable and robust mechanism in `v1.0`.
+   See #46237.
+
+5. Config options that deprecated in `v0.21` have been removed in `v0.23`. You
+   can now only specify preferences for `compilers`, `targets`, and
+   `providers` globally via the `packages:all:` section. Similarly, you can
+   only specify `versions:` locally for a specific package. See #44061 and
+   #31261 for details.
+
+6. Spack's old test interface has been removed (#45752), having been
+   deprecated in `v0.22.0` (#34236). All `builtin` packages have been updated
+   to use the new interface. See the [stand-alone test documentation](
+   https://spack.readthedocs.io/en/latest/packaging_guide.html#stand-alone-tests)
+
+7. The `spack versions --safe-only` option, deprecated since `v0.21.0`, has
+   been removed. See #45765.
+
+* The `--dependencies` and `--optimize` arguments to `spack ci` have been
+  deprecated. See #45005.
+
+## Binary caches
+1. Public binary caches now include an ML stack for Linux/aarch64 (#39666)We
+   now build an ML stack for Linux/aarch64 for all pull requests and on
+   develop. The ML stack includes both CPU-only and CUDA builds for Horovod,
+   Hugging Face, JAX, Keras, PyTorch,scikit-learn, TensorBoard, and
+   TensorFlow, and related packages. The CPU-only stack also includes XGBoost.
+   See https://cache.spack.io/tag/develop/?stack=ml-linux-aarch64-cuda.
+
+2. There is also now an stack of developer tools for macOS (#46910), which is
+   analogous to the Linux devtools stack. You can use this to avoid building
+   many common build dependencies. See
+   https://cache.spack.io/tag/develop/?stack=developer-tools-darwin.
+
+## Architecture support
+* archspec has been updated to `v0.2.5`, with support for `zen5`
+* Spack's CUDA package now supports the Grace Hopper `9.0a` compute capability (#45540)
+
+## Windows
+* Windows bootstrapping: `file` and `gpg` (#41810)
+* `scripts` directory added to PATH on Windows for python extensions (#45427)
+* Fix `spack load --list` and `spack unload` on Windows (#35720)
+
+## Other notable changes
+* Bugfix: `spack find -x` in environments (#46798)
+* Spec splices are now robust to duplicate nodes with the same name in a spec (#46382)
+* Cache per-compiler libc calculations for performance (#47213)
+* Fixed a bug in external detection for openmpi (#47541)
+* Mirror configuration allows username/password as environment variables (#46549)
+* Default library search caps maximum depth (#41945)
+* Unify interface for `spack spec` and `spack solve` commands (#47182)
+* Spack no longer RPATHs directories in the default library search path (#44686)
+* Improved performance of Spack database (#46554)
+* Enable package reuse for packages with versions from git refs (#43859)
+* Improved handling for `uuid` virtual on macos (#43002)
+* Improved tracking of task queueing/requeueing in the installer (#46293)
+
+## Spack community stats
+
+* Over 2,000 pull requests updated package recipes
+* 8,307 total packages, 329 new since `v0.22.0`
+    * 140 new Python packages
+    * 14 new R packages
+* 373 people contributed to this release
+    * 357 committers to packages
+    * 60 committers to core
+
+
 # v0.22.2 (2024-09-21)
 
 ## Bugfixes
@@ -419,7 +781,7 @@
 - spack graph: fix coloring with environments (#41240)
 - spack info: sort variants in --variants-by-name (#41389)
 - Spec.format: error on old style format strings (#41934)
-- ASP-based solver: 
+- ASP-based solver:
   - fix infinite recursion when computing concretization errors (#41061)
   - don't error for type mismatch on preferences (#41138)
   - don't emit spurious debug output (#41218)

lib/spack/spack/__init__.py

@@ -11,7 +11,7 @@
 import spack.util.git
 
 #: PEP440 canonical <major>.<minor>.<micro>.<devN> string
-__version__ = "0.23.0.dev0"
+__version__ = "0.23.0"
 spack_version = __version__
 
 

lib/spack/spack/cmd/__init__.py

@@ -25,6 +25,7 @@
 import spack.extensions
 import spack.parser
 import spack.paths
+import spack.repo
 import spack.spec
 import spack.store
 import spack.traverse as traverse

lib/spack/spack/cmd/solve.py

@@ -3,6 +3,7 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
+import argparse
 import re
 import sys
 
@@ -48,6 +49,13 @@ def setup_parser(subparser):
     subparser.add_argument(
         "--stats", action="store_true", default=False, help="print out statistics from clingo"
     )
+    subparser.add_argument(
+        "--profile",
+        action="store_true",
+        default=False,
+        help="profile the solve phase and print out statistics on atoms",
+    )
+    subparser.add_argument("specs", nargs=argparse.REMAINDER, help="specs of packages")
 
     spack.cmd.spec.setup_parser(subparser)
 
@@ -147,6 +155,7 @@ def solve(parser, args):
             stats=args.stats,
             setup_only=setup_only,
             allow_deprecated=allow_deprecated,
+            profile=args.profile,
         )
         if not setup_only:
             _process_result(result, show, required_format, kwargs)

lib/spack/spack/cmd/spec.py

@@ -82,14 +82,6 @@ def spec(parser, args):
     if args.namespaces:
         fmt = "{namespace}." + fmt
 
-    tree_kwargs = {
-        "cover": args.cover,
-        "format": fmt,
-        "hashlen": None if args.very_long else 7,
-        "show_types": args.types,
-        "status_fn": install_status_fn if args.install_status else None,
-    }
-
     # use a read transaction if we are getting install status for every
     # spec in the DAG.  This avoids repeatedly querying the DB.
     tree_context = lang.nullcontext
@@ -99,46 +91,35 @@ def spec(parser, args):
     env = ev.active_environment()
 
     if args.specs:
-        input_specs = spack.cmd.parse_specs(args.specs)
-        concretized_specs = spack.cmd.parse_specs(args.specs, concretize=True)
-        specs = list(zip(input_specs, concretized_specs))
+        concrete_specs = spack.cmd.parse_specs(args.specs, concretize=True)
     elif env:
         env.concretize()
-        specs = env.concretized_specs()
-
-        if not args.format:
-            # environments are printed together in a combined tree() invocation,
-            # except when using --yaml or --json, which we print spec by spec below.
-            tree_kwargs["key"] = spack.traverse.by_dag_hash
-            tree_kwargs["hashes"] = args.long or args.very_long
-            print(spack.spec.tree([concrete for _, concrete in specs], **tree_kwargs))
-            return
+        concrete_specs = env.concrete_roots()
     else:
         tty.die("spack spec requires at least one spec or an active environment")
 
-    for input, output in specs:
-        # With --yaml or --json, just print the raw specs to output
-        if args.format:
+    # With --yaml, --json, or --format, just print the raw specs to output
+    if args.format:
+        for spec in concrete_specs:
             if args.format == "yaml":
                 # use write because to_yaml already has a newline.
-                sys.stdout.write(output.to_yaml(hash=ht.dag_hash))
+                sys.stdout.write(spec.to_yaml(hash=ht.dag_hash))
             elif args.format == "json":
-                print(output.to_json(hash=ht.dag_hash))
+                print(spec.to_json(hash=ht.dag_hash))
             else:
-                print(output.format(args.format))
-            continue
+                print(spec.format(args.format))
+        return
 
-        with tree_context():
-            # Only show the headers for input specs that are not concrete to avoid
-            # repeated output. This happens because parse_specs outputs concrete
-            # specs for `/hash` inputs.
-            if not input.concrete:
-                tree_kwargs["hashes"] = False  # Always False for input spec
-                print("Input spec")
-                print("--------------------------------")
-                print(input.tree(**tree_kwargs))
-                print("Concretized")
-                print("--------------------------------")
-
-            tree_kwargs["hashes"] = args.long or args.very_long
-            print(output.tree(**tree_kwargs))
+    with tree_context():
+        print(
+            spack.spec.tree(
+                concrete_specs,
+                cover=args.cover,
+                format=fmt,
+                hashlen=None if args.very_long else 7,
+                show_types=args.types,
+                status_fn=install_status_fn if args.install_status else None,
+                hashes=args.long or args.very_long,
+                key=spack.traverse.by_dag_hash,
+            )
+        )

lib/spack/spack/cmd/tutorial.py

@@ -24,7 +24,7 @@
 
 
 # tutorial configuration parameters
-tutorial_branch = "releases/v0.22"
+tutorial_branch = "releases/v0.23"
 tutorial_mirror = "file:///mirror"
 tutorial_key = os.path.join(spack.paths.share_path, "keys", "tutorial.pub")
 

lib/spack/spack/mirror.py

@@ -180,7 +180,7 @@ def ensure_mirror_usable(self, direction: str = "push"):
         if errors:
             msg = f"invalid {direction} configuration for mirror {self.name}: "
             msg += "\n    ".join(errors)
-            raise spack.mirror.MirrorError(msg)
+            raise MirrorError(msg)
 
     def _update_connection_dict(self, current_data: dict, new_data: dict, top_level: bool):
         # Only allow one to exist in the config

lib/spack/spack/solver/asp.py

@@ -64,6 +64,7 @@
     parse_term,
 )
 from .counter import FullDuplicatesCounter, MinimalDuplicatesCounter, NoDuplicatesCounter
+from .profiler import ProfilePropagator
 from .version_order import concretization_version_order
 
 GitOrStandardVersion = Union[spack.version.GitVersion, spack.version.StandardVersion]
@@ -800,7 +801,16 @@ def __init__(self, cores=True):
         # This attribute will be reset at each call to solve
         self.control = None
 
-    def solve(self, setup, specs, reuse=None, output=None, control=None, allow_deprecated=False):
+    def solve(
+        self,
+        setup,
+        specs,
+        reuse=None,
+        output=None,
+        control=None,
+        allow_deprecated=False,
+        profile=False,
+    ):
         """Set up the input and solve for dependencies of ``specs``.
 
         Arguments:
@@ -826,6 +836,11 @@ def solve(self, setup, specs, reuse=None, output=None, control=None, allow_depre
         # Initialize the control object for the solver
         self.control = control or default_clingo_control()
 
+        # if profiling is enabled, register a profiling propagator
+        if profile:
+            propagator = ProfilePropagator()
+            self.control.register_propagator(propagator)
+
         # ensure core deps are present on Windows
         # needs to modify active config scope, so cannot be run within
         # bootstrap config scope
@@ -925,13 +940,18 @@ def on_model(model):
             result.cores.extend(cores)
 
         if output.timers:
+            tty.msg("Timers:")
             timer.write_tty()
             print()
 
         if output.stats:
-            print("Statistics:")
+            tty.msg("Statistics:")
             pprint.pprint(self.control.statistics)
 
+        if profile:
+            tty.msg("Profile:")
+            propagator.print_profile(40)
+
         result.raise_if_unsat()
 
         if result.satisfiable and result.unsolved_specs and setup.concretize_everything:
@@ -3839,12 +3859,21 @@ def splice_at_hash(
         self._splices.setdefault(parent_node, []).append(splice)
 
     def _resolve_automatic_splices(self):
-        """After all of the specs have been concretized, apply all immediate
-        splices in size order. This ensures that all dependencies are resolved
+        """After all of the specs have been concretized, apply all immediate splices.
+
+        Use reverse topological order to ensure that all dependencies are resolved
         before their parents, allowing for maximal sharing and minimal copying.
+
         """
         fixed_specs = {}
-        for node, spec in sorted(self._specs.items(), key=lambda x: len(x[1])):
+
+        # create a mapping from dag hash to an integer representing position in reverse topo order.
+        specs = self._specs.values()
+        topo_order = list(traverse.traverse_nodes(specs, order="topo", key=traverse.by_dag_hash))
+        topo_lookup = {spec.dag_hash(): index for index, spec in enumerate(reversed(topo_order))}
+
+        # iterate over specs, children before parents
+        for node, spec in sorted(self._specs.items(), key=lambda x: topo_lookup[x[1].dag_hash()]):
             immediate = self._splices.get(node, [])
             if not immediate and not any(
                 edge.spec in fixed_specs for edge in spec.edges_to_dependencies()
@@ -4352,6 +4381,7 @@ def solve_with_stats(
         tests=False,
         setup_only=False,
         allow_deprecated=False,
+        profile=False,
     ):
         """
         Concretize a set of specs and track the timing and statistics for the solve
@@ -4373,7 +4403,12 @@ def solve_with_stats(
         setup = SpackSolverSetup(tests=tests)
         output = OutputConfiguration(timers=timers, stats=stats, out=out, setup_only=setup_only)
         return self.driver.solve(
-            setup, specs, reuse=reusable_specs, output=output, allow_deprecated=allow_deprecated
+            setup,
+            specs,
+            reuse=reusable_specs,
+            output=output,
+            allow_deprecated=allow_deprecated,
+            profile=profile,
         )
 
     def solve(self, specs, **kwargs):

lib/spack/spack/solver/concretize.lp

@@ -1455,7 +1455,7 @@ attr("node_flag", PackageNode, NodeFlag) :- attr("node_flag_set", PackageNode, N
 #defined installed_hash/2.
 #defined abi_splice_conditions_hold/4.
 
-% These are the previously concretized attributes of the installed package as 
+% These are the previously concretized attributes of the installed package as
 % a hash. It has the general form:
 % hash_attr(Hash, Attribute, PackageName, Args*)
 #defined hash_attr/3.
@@ -1479,12 +1479,12 @@ hash_attr(Hash, "node_version_satisfies", PackageName, Constraint) :-
 
 % This recovers the exact semantics for hash reuse hash and depends_on are where
 % splices are decided, and virtual_on_edge can result in name-changes, which is
-% why they are all treated separately. 
+% why they are all treated separately.
 imposed_constraint(Hash, Attr, PackageName) :-
   hash_attr(Hash, Attr, PackageName).
 imposed_constraint(Hash, Attr, PackageName, A1) :-
   hash_attr(Hash, Attr, PackageName, A1), Attr != "hash".
-imposed_constraint(Hash, Attr, PackageName, Arg1, Arg2) :- 
+imposed_constraint(Hash, Attr, PackageName, Arg1, Arg2) :-
   hash_attr(Hash, Attr, PackageName, Arg1, Arg2),
   Attr != "depends_on",
   Attr != "virtual_on_edge".
@@ -1492,16 +1492,16 @@ imposed_constraint(Hash, Attr, PackageName, A1, A2, A3) :-
   hash_attr(Hash, Attr, PackageName, A1, A2, A3).
 imposed_constraint(Hash, "hash", PackageName, Hash) :- installed_hash(PackageName, Hash).
 % Without splicing, we simply recover the exact semantics
-imposed_constraint(ParentHash, "hash", ChildName, ChildHash) :- 
+imposed_constraint(ParentHash, "hash", ChildName, ChildHash) :-
   hash_attr(ParentHash, "hash", ChildName, ChildHash),
   ChildHash != ParentHash,
   not abi_splice_conditions_hold(_, _, ChildName, ChildHash).
-  
+
 imposed_constraint(Hash, "depends_on", PackageName, DepName, Type) :-
   hash_attr(Hash, "depends_on", PackageName, DepName, Type),
   hash_attr(Hash, "hash", DepName, DepHash),
   not attr("splice_at_hash", _, _, DepName, DepHash).
-  
+
 imposed_constraint(Hash, "virtual_on_edge", PackageName, DepName, VirtName) :-
   hash_attr(Hash, "virtual_on_edge", PackageName, DepName, VirtName),
   not attr("splice_at_hash", _, _, DepName,_).
@@ -1511,7 +1511,7 @@ imposed_constraint(Hash, "virtual_on_edge", PackageName, DepName, VirtName) :-
 
 impose(Hash, PackageNode) :- attr("hash", PackageNode, Hash), attr("node", PackageNode).
 
-% If there is not a hash for a package, we build it. 
+% If there is not a hash for a package, we build it.
 build(PackageNode) :- attr("node", PackageNode), not concrete(PackageNode).
 
 

lib/spack/spack/solver/profiler.py

@@ -0,0 +1,111 @@
+# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+"""Profiling propagator for clingo solves."""
+
+import re
+from typing import Any, Dict, List
+
+import llnl.util.tty as tty
+
+
+class Data:
+    """Counters for propagations and undos of an atom."""
+
+    __slots__ = ["atom", "literal", "prop", "undo"]
+
+    # currently we use Any for clingo types because clingo has a bunch of import
+    # wrappers around it that make typing difficult (see spack.solver.core for details)
+    def __init__(self, atom: Any, literal: int, prop: int, undo: int):
+        self.atom = atom
+        self.literal = literal
+        self.prop = prop
+        self.undo = undo
+
+
+class AggregatedData:
+    """Aggregated data for a profile, constructed from ``Data``.
+
+    We coarsen from atom granularity to string keys when aggregating.
+    """
+
+    __slots__ = ["name", "prop", "undo"]
+
+    def __init__(self, name: str, prop: int, undo: int):
+        self.name = name
+        self.prop = prop
+        self.undo = undo
+
+
+class ProfilePropagator:
+    """Profiling propagator for `spack solve --profile`.
+
+    Register this with the ``clingo.Control`` object to profile a solve.
+    """
+
+    _literal_to_atom: Dict
+    _profile: Dict[int, Data]
+
+    def init(self, init) -> None:
+        self._literal_to_atom = {}
+        self._profile = {}
+        for atom in init.symbolic_atoms:
+            solver_literal = init.solver_literal(atom.literal)
+            self._profile[solver_literal] = Data(atom, solver_literal, 0, 0)
+            init.add_watch(solver_literal)
+
+    def propagate(self, ctl, changes: List[int]) -> bool:
+        """Record a propagation in the solve."""
+        for literal in changes:
+            data = self._profile[literal]
+            data.prop += 1
+        return True
+
+    def undo(self, solver_id: int, assign, undo: List[int]) -> None:
+        """Record an undo in the solve."""
+        for literal in undo:
+            data = self._profile[literal]
+            data.undo += 1
+
+    def color_sym(self, string: str) -> str:
+        """Colorize a symbol for profile output"""
+        string = re.sub(r"^(\w+)", r"@C{\1}", string)
+        string = re.sub(r'("[^"]*")', r"@G{\1}", string)
+        string = re.sub(r"([\(\)])", r"@b{\1}", string)
+        return tty.color.colorize(string)
+
+    def key(self, atom) -> str:
+        """Convert an atom into an aggregate key for our profile.
+
+        Currently this compresses most things to their function name, and expands
+        ``attr("name", ...)`` to ``attr("name")`` so we can see which attributes affect
+        the solve most.
+
+        """
+        sym = atom.symbol
+        return f"attr({sym.arguments[0]})" if sym.name == "attr" else sym.name
+
+    def print_profile(self, n_atoms: int) -> None:
+        """Aggregate and print nicely formatted profile data."""
+        aggregated = {}
+        for data in self._profile.values():
+            name = self.key(data.atom)
+            if name not in aggregated:
+                aggregated[name] = AggregatedData(name, data.prop, data.undo)
+            else:
+                agg = aggregated[name]
+                agg.prop += data.prop
+                agg.undo += data.undo
+
+        values = sorted(
+            (x for x in aggregated.values() if x.prop), key=lambda x: x.prop, reverse=True
+        )
+
+        # format the output nicely
+        w = 10  # width for number fields
+        print(tty.color.colorize(f"  @*{{{'Prop':<{w}}{'Undo':<{w}}{'Symbol'}}}"))
+        for a in values[:n_atoms]:
+            print(f"  {a.prop:<{w}}{a.undo:<{w}}{self.color_sym(a.name)}")
+        if len(values) > n_atoms:
+            print("  ...")

lib/spack/spack/spec.py

@@ -1431,8 +1431,6 @@ def tree(
 class Spec:
     #: Cache for spec's prefix, computed lazily in the corresponding property
     _prefix = None
-    #: Cache for spec's length, computed lazily in the corresponding property
-    _length = None
     abstract_hash = None
 
     @staticmethod
@@ -3702,18 +3700,6 @@ def __getitem__(self, name: str):
 
         return child
 
-    def __len__(self):
-        if not self.concrete:
-            raise spack.error.SpecError(f"Cannot get length of abstract spec: {self}")
-
-        if not self._length:
-            self._length = 1 + sum(len(dep) for dep in self.dependencies())
-        return self._length
-
-    def __bool__(self):
-        # Need to define this so __len__ isn't used by default
-        return True
-
     def __contains__(self, spec):
         """True if this spec or some dependency satisfies the spec.
 
@@ -4472,7 +4458,7 @@ def clear_caches(self, ignore=()):
             if h.attr not in ignore:
                 if hasattr(self, h.attr):
                     setattr(self, h.attr, None)
-        for attr in ("_dunder_hash", "_prefix", "_length"):
+        for attr in ("_dunder_hash", "_prefix"):
             if attr not in ignore:
                 setattr(self, attr, None)
 

lib/spack/spack/test/abi_splicing.py

@@ -10,9 +10,6 @@
 
 import spack.config
 import spack.deptypes as dt
-import spack.package_base
-import spack.paths
-import spack.repo
 import spack.solver.asp
 from spack.installer import PackageInstaller
 from spack.spec import Spec
@@ -232,3 +229,19 @@ def test_spliced_build_deps_only_in_build_spec(splicing_setup):
         assert _has_build_dependency(build_spec, "splice-z")
         # Spliced build dependencies are removed
         assert len(concr_goal.dependencies(None, dt.BUILD)) == 0
+
+
+def test_spliced_transitive_dependency(splicing_setup):
+    cache = ["splice-depends-on-t@1.0 ^splice-h@1.0.1"]
+    goal_spec = Spec("splice-depends-on-t^splice-h@1.0.2")
+
+    with CacheManager(cache):
+        spack.config.set("packages", _make_specs_non_buildable(["splice-depends-on-t"]))
+        _enable_splicing()
+        concr_goal = goal_spec.concretized()
+        # Spec has been spliced
+        assert concr_goal._build_spec is not None
+        assert concr_goal["splice-t"]._build_spec is not None
+        assert concr_goal.satisfies(goal_spec)
+        # Spliced build dependencies are removed
+        assert len(concr_goal.dependencies(None, dt.BUILD)) == 0

lib/spack/spack/test/cmd/init_py_functions.py

@@ -4,9 +4,11 @@
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 import pytest
 
+import spack.config
 import spack.environment as ev
 import spack.error
 import spack.solver.asp as asp
+import spack.store
 from spack.cmd import (
     CommandNameError,
     PythonNameError,

lib/spack/spack/test/cmd/pkg.py

@@ -315,6 +315,7 @@ def test_pkg_grep(mock_packages, capfd):
             "depends-on-manyvariants",
             "manyvariants",
             "splice-a",
+            "splice-depends-on-t",
             "splice-h",
             "splice-t",
             "splice-vh",

lib/spack/spack/test/cmd/spec.py

@@ -7,6 +7,7 @@
 
 import pytest
 
+import spack.config
 import spack.environment as ev
 import spack.error
 import spack.spec

lib/spack/spack/test/concretization/core.py

@@ -2871,6 +2871,18 @@ def test_specifying_different_versions_build_deps(self):
         assert any(x.satisfies(hdf5_str) for x in result.specs)
         assert any(x.satisfies(pinned_str) for x in result.specs)
 
+    def test_solve_with_profile(self, capsys):
+        """For now, just ensure that the profiler runs."""
+        solver = spack.solver.asp.Solver()
+        solver.solve([Spec("hdf5")], profile=True)
+
+        out, _ = capsys.readouterr()
+        assert "Profile:" in out
+        assert "Symbol" in out
+        assert "Prop" in out
+        assert "Undo" in out
+        assert "internal_error" in out  # symbol is always in small solves
+
 
 @pytest.mark.parametrize(
     "v_str,v_opts,checksummed",

share/spack/spack-completion.bash

@@ -1849,7 +1849,7 @@ _spack_restage() {
 _spack_solve() {
     if $list_options
     then
-        SPACK_COMPREPLY="-h --help --show --timers --stats -l --long -L --very-long -N --namespaces -I --install-status --no-install-status -y --yaml -j --json --format -c --cover -t --types -U --fresh --reuse --fresh-roots --reuse-deps --deprecated"
+        SPACK_COMPREPLY="-h --help --show --timers --stats --profile -l --long -L --very-long -N --namespaces -I --install-status --no-install-status -y --yaml -j --json --format -c --cover -t --types -U --fresh --reuse --fresh-roots --reuse-deps --deprecated"
     else
         _all_packages
     fi

share/spack/spack-completion.fish

@@ -2799,8 +2799,9 @@ complete -c spack -n '__fish_spack_using_command restage' -s h -l help -f -a hel
 complete -c spack -n '__fish_spack_using_command restage' -s h -l help -d 'show this help message and exit'
 
 # spack solve
-set -g __fish_spack_optspecs_spack_solve h/help show= timers stats l/long L/very-long N/namespaces I/install-status no-install-status y/yaml j/json format= c/cover= t/types U/fresh reuse fresh-roots deprecated
+set -g __fish_spack_optspecs_spack_solve h/help show= timers stats profile l/long L/very-long N/namespaces I/install-status no-install-status y/yaml j/json format= c/cover= t/types U/fresh reuse fresh-roots deprecated
 complete -c spack -n '__fish_spack_using_command_pos_remainder 0 solve' -f -k -a '(__fish_spack_specs_or_id)'
+complete -c spack -n '__fish_spack_using_command_pos_remainder 1 solve' -f -k -a '(__fish_spack_specs_or_id)'
 complete -c spack -n '__fish_spack_using_command solve' -s h -l help -f -a help
 complete -c spack -n '__fish_spack_using_command solve' -s h -l help -d 'show this help message and exit'
 complete -c spack -n '__fish_spack_using_command solve' -l show -r -f -a show
@@ -2809,6 +2810,8 @@ complete -c spack -n '__fish_spack_using_command solve' -l timers -f -a timers
 complete -c spack -n '__fish_spack_using_command solve' -l timers -d 'print out timers for different solve phases'
 complete -c spack -n '__fish_spack_using_command solve' -l stats -f -a stats
 complete -c spack -n '__fish_spack_using_command solve' -l stats -d 'print out statistics from clingo'
+complete -c spack -n '__fish_spack_using_command solve' -l profile -f -a profile
+complete -c spack -n '__fish_spack_using_command solve' -l profile -d 'profile the solve phase and print out statistics on atoms'
 complete -c spack -n '__fish_spack_using_command solve' -s l -l long -f -a long
 complete -c spack -n '__fish_spack_using_command solve' -s l -l long -d 'show dependency hashes as well as versions'
 complete -c spack -n '__fish_spack_using_command solve' -s L -l very-long -f -a very_long

var/spack/repos/builtin.mock/packages/splice-depends-on-t/package.py

@@ -0,0 +1,22 @@
+# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+from spack.package import *
+
+
+class SpliceDependsOnT(Package):
+    """Package that depends on splice-t"""
+
+    homepage = "http://www.example.com"
+    url = "http://www.example.com/splice-depends-on-t-1.0.tar.gz"
+
+    version("1.0", md5="0123456789abcdef0123456789abcdef")
+
+    depends_on("splice-t")
+
+    def install(self, spec, prefix):
+        with open(prefix.join("splice-depends-on-t"), "w") as f:
+            f.write("splice-depends-on-t: {0}".format(prefix))
+            f.write("splice-t: {0}".format(spec["splice-t"].prefix))