foo

* Add initial version of verible to spack
* Update to use explict url path for each release, as the release tagh includes extra data, also added the bottom most point of gcc, gcc9

.github/workflows/build-containers.yml

@@ -57,7 +57,7 @@ jobs:
       - name: Checkout
         uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
 
-      - uses: docker/metadata-action@96383f45573cb7f253c731d3b3ab81c87ef81934
+      - uses: docker/metadata-action@e6428a5c4e294a61438ed7f43155db912025b6b3
         id: docker_meta
         with:
           images: |
@@ -113,7 +113,7 @@ jobs:
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
       - name: Build & Deploy ${{ matrix.dockerfile[0] }}
-        uses: docker/build-push-action@0565240e2d4ab88bba5387d719585280857ece09
+        uses: docker/build-push-action@4a13e500e55cf31b7a5d59a38ab2040ab0f42f56
         with:
           context: dockerfiles/${{ matrix.dockerfile[0] }}
           platforms: ${{ matrix.dockerfile[1] }}

.github/workflows/style/requirements.txt

@@ -1,7 +1,7 @@
-black==23.10.1
+black==23.11.0
 clingo==5.6.2
 flake8==6.1.0
 isort==5.12.0
-mypy==1.6.1
+mypy==1.7.1
 types-six==1.16.21.9
-vermin==1.5.2
+vermin==1.6.0

.github/workflows/unit_tests.yaml

@@ -215,7 +215,7 @@ jobs:
         $(which spack) bootstrap disable spack-install
         $(which spack) solve zlib
         common_args=(--dist loadfile --tx '4*popen//python=./bin/spack-tmpconfig python -u ./bin/spack python' -x)
-        $(which spack) unit-test --cov --cov-config=pyproject.toml --cov-report=xml:coverage.xml "${common_args[@]}"
+        $(which spack) unit-test --verbose --cov --cov-config=pyproject.toml --cov-report=xml:coverage.xml "${common_args[@]}"
     - uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d
       with:
         flags: unittests,macos

CHANGELOG.md

@@ -1,3 +1,290 @@
+# v0.21.0 (2023-11-11)
+
+`v0.21.0` is a major feature release.
+
+## Features in this release
+
+1. **Better error messages with condition chaining**
+
+   In v0.18, we added better error messages that could tell you what problem happened,
+   but they couldn't tell you *why* it happened. `0.21` adds *condition chaining* to the
+   solver, and Spack can now trace back through the conditions that led to an error and
+   build a tree of causes potential causes and where they came from. For example:
+
+   ```console
+   $ spack solve hdf5 ^cmake@3.0.1
+   ==> Error: concretization failed for the following reasons:
+
+      1. Cannot satisfy 'cmake@3.0.1'
+      2. Cannot satisfy 'cmake@3.0.1'
+           required because hdf5 ^cmake@3.0.1 requested from CLI
+      3. Cannot satisfy 'cmake@3.18:' and 'cmake@3.0.1
+           required because hdf5 ^cmake@3.0.1 requested from CLI
+           required because hdf5 depends on cmake@3.18: when @1.13:
+             required because hdf5 ^cmake@3.0.1 requested from CLI
+      4. Cannot satisfy 'cmake@3.12:' and 'cmake@3.0.1
+           required because hdf5 depends on cmake@3.12:
+             required because hdf5 ^cmake@3.0.1 requested from CLI
+           required because hdf5 ^cmake@3.0.1 requested from CLI
+   ```
+
+   More details in #40173.
+
+2. **OCI build caches**
+
+   You can now use an arbitrary [OCI](https://opencontainers.org) registry as a build
+   cache:
+
+   ```console
+   $ spack mirror add my_registry oci://user/image # Dockerhub
+   $ spack mirror add my_registry oci://ghcr.io/haampie/spack-test # GHCR
+   $ spack mirror set --push --oci-username ... --oci-password ... my_registry  # set login creds
+   $ spack buildcache push my_registry [specs...]
+   ```
+
+   And you can optionally add a base image to get *runnable* images:
+
+   ```console
+   $ spack buildcache push --base-image ubuntu:23.04 my_registry python
+   Pushed ... as [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack
+
+   $ docker run --rm -it [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack
+   ```
+
+   This creates a container image from the Spack installations on the host system,
+   without the need to run `spack install` from a `Dockerfile` or `sif` file. It also
+   addresses the inconvenience of losing binaries of dependencies when `RUN spack
+   install` fails inside `docker build`.
+
+   Further, the container image layers and build cache tarballs are the same files. This
+   means that `spack install` and `docker pull` use the exact same underlying binaries.
+   If you previously used `spack install` inside of `docker build`, this feature helps
+   you save storage by a factor two.
+
+   More details in #38358.
+
+3. **Multiple versions of build dependencies**
+
+   Increasingly, complex package builds require multiple versions of some build
+   dependencies. For example, Python packages frequently require very specific versions
+   of `setuptools`, `cython`, and sometimes different physics packages require different
+   versions of Python to build. The concretizer enforced that every solve was *unified*,
+   i.e., that there only be one version of every package. The concretizer now supports
+   "duplicate" nodes for *build dependencies*, but enforces unification through
+   transitive link and run dependencies. This will allow it to better resolve complex
+   dependency graphs in ecosystems like Python, and it also gets us very close to
+   modeling compilers as proper dependencies.
+
+   This change required a major overhaul of the concretizer, as well as a number of
+   performance optimizations. See #38447, #39621.
+
+4. **Cherry-picking virtual dependencies**
+
+   You can now select only a subset of virtual dependencies from a spec that may provide
+   more. For example, if you want `mpich` to be your `mpi` provider, you can be explicit
+   by writing:
+
+   ```
+   hdf5 ^[virtuals=mpi] mpich
+   ```
+
+   Or, if you want to use, e.g., `intel-parallel-studio` for `blas` along with an external
+   `lapack` like `openblas`, you could write:
+
+   ```
+   strumpack ^[virtuals=mpi] intel-parallel-studio+mkl ^[virtuals=lapack] openblas
+   ```
+
+   The `virtuals=mpi` is an edge attribute, and dependency edges in Spack graphs now
+   track which virtuals they satisfied. More details in #17229 and #35322.
+
+   Note for packaging: in Spack 0.21 `spec.satisfies("^virtual")` is true if and only if
+   the package specifies `depends_on("virtual")`. This is different from Spack 0.20,
+   where depending on a provider implied depending on the virtual provided. See #41002
+   for an example where `^mkl` was being used to test for several `mkl` providers in a
+   package that did not depend on `mkl`.
+
+5. **License directive**
+
+   Spack packages can now have license metadata, with the new `license()` directive:
+
+   ```python
+       license("Apache-2.0")
+   ```
+
+   Licenses use [SPDX identifiers](https://spdx.org/licenses), and you can use SPDX
+   expressions to combine them:
+
+   ```python
+       license("Apache-2.0 OR MIT")
+   ```
+
+   Like other directives in Spack, it's conditional, so you can handle complex cases like
+   Spack itself:
+
+   ```python
+      license("LGPL-2.1", when="@:0.11")
+      license("Apache-2.0 OR MIT", when="@0.12:")
+   ```
+
+   More details in #39346, #40598.
+
+6. **`spack deconcretize` command**
+
+   We are getting close to having a `spack update` command for environments, but we're
+   not quite there yet. This is the next best thing. `spack deconcretize` gives you
+   control over what you want to update in an already concrete environment. If you have
+   an environment built with, say, `meson`, and you want to update your `meson` version,
+   you can run:
+
+   ```console
+   spack deconcretize meson
+   ```
+
+   and have everything that depends on `meson` rebuilt the next time you run `spack
+   concretize`. In a future Spack version, we'll handle all of this in a single command,
+   but for now you can use this to drop bits of your lockfile and resolve your
+   dependencies again. More in #38803.
+
+7. **UI Improvements**
+
+   The venerable `spack info` command was looking shabby compared to the rest of Spack's
+   UI, so we reworked it to have a bit more flair. `spack info` now makes much better
+   use of terminal space and shows variants, their values, and their descriptions much
+   more clearly. Conditional variants are grouped separately so you can more easily
+   understand how packages are structured. More in #40998.
+
+   `spack checksum` now allows you to filter versions from your editor, or by version
+   range. It also notifies you about potential download URL changes. See #40403.
+
+8. **Environments can include definitions**
+
+   Spack did not previously support using `include:` with The
+   [definitions](https://spack.readthedocs.io/en/latest/environments.html#spec-list-references)
+   section of an environment, but now it does. You can use this to curate lists of specs
+   and more easily reuse them across environments. See #33960.
+
+9. **Aliases**
+
+   You can now add aliases to Spack commands in `config.yaml`, e.g. this might enshrine
+   your favorite args to `spack find` as `spack f`:
+
+   ```yaml
+   config:
+     aliases:
+       f: find -lv
+   ```
+
+   See #17229.
+
+10. **Improved autoloading of modules**
+
+    Spack 0.20 was the first release to enable autoloading of direct dependencies in
+    module files.
+
+    The downside of this was that `module avail` and `module load` tab completion would
+    show users too many modules to choose from, and many users disabled generating
+    modules for dependencies through `exclude_implicits: true`. Further, it was
+    necessary to keep hashes in module names to avoid file name clashes.
+
+    In this release, you can start using `hide_implicits: true` instead, which exposes
+    only explicitly installed packages to the user, while still autoloading
+    dependencies. On top of that, you can safely use `hash_length: 0`, as this config
+    now only applies to the modules exposed to the user -- you don't have to worry about
+    file name clashes for hidden dependencies.
+
+   Note: for `tcl` this feature requires Modules 4.7 or higher
+
+11. **Updated container labeling**
+
+    Nightly Docker images from the `develop` branch will now be tagged as `:develop` and
+    `:nightly`. The `:latest` tag is no longer associated with `:develop`, but with the
+    latest stable release. Releases will be tagged with `:{major}`, `:{major}.{minor}`
+    and `:{major}.{minor}.{patch}`. `ubuntu:18.04` has also been removed from the list of
+    generated Docker images, as it is no longer supported. See #40593.
+
+## Other new commands and directives
+
+* `spack env activate` without arguments now loads a `default` environment that you do
+  not have to create (#40756).
+* `spack find -H` / `--hashes`: a new shortcut for piping `spack find` output to
+  other commands (#38663)
+* Add `spack checksum --verify`, fix `--add` (#38458)
+* New `default_args` context manager factors out common args for directives (#39964)
+* `spack compiler find --[no]-mixed-toolchain` lets you easily mix `clang` and
+  `gfortran` on Linux (#40902)
+
+## Performance improvements
+
+* `spack external find` execution is now much faster (#39843)
+* `spack location -i` now much faster on success (#40898)
+* Drop redundant rpaths post install (#38976)
+* ASP-based solver: avoid cycles in clingo using hidden directive (#40720)
+* Fix multiple quadratic complexity issues in environments (#38771)
+
+## Other new features of note
+
+* archspec: update to v0.2.2, support for Sapphire Rapids, Power10, Neoverse V2 (#40917)
+* Propagate variants across nodes that don't have that variant (#38512)
+* Implement fish completion (#29549)
+* Can now distinguish between source/binary mirror; don't ping mirror.spack.io as much (#34523)
+* Improve status reporting on install (add [n/total] display) (#37903)
+
+## Windows
+
+This release has the best Windows support of any Spack release yet, with numerous
+improvements and much larger swaths of tests passing:
+
+* MSVC and SDK improvements (#37711, #37930, #38500, #39823, #39180)
+* Windows external finding: update default paths; treat .bat as executable on Windows (#39850)
+* Windows decompression: fix removal of intermediate file (#38958)
+* Windows: executable/path handling (#37762)
+* Windows build systems: use ninja and enable tests (#33589)
+* Windows testing (#36970, #36972, #36973, #36840, #36977, #36792, #36834, #34696, #36971)
+* Windows PowerShell support (#39118, #37951)
+* Windows symlinking and libraries (#39933, #38599, #34701, #38578, #34701)
+
+## Notable refactors
+* User-specified flags take precedence over others in Spack compiler wrappers (#37376)
+* Improve setup of build, run, and test environments (#35737, #40916)
+* `make` is no longer a required system dependency of Spack (#40380)
+* Support Python 3.12 (#40404, #40155, #40153)
+* docs: Replace package list with packages.spack.io (#40251)
+* Drop Python 2 constructs in Spack (#38720, #38718, #38703)
+
+## Binary cache and stack updates
+* e4s arm stack: duplicate and target neoverse v1 (#40369)
+* Add macOS ML CI stacks (#36586)
+* E4S Cray CI Stack (#37837)
+* e4s cray: expand spec list (#38947)
+* e4s cray sles ci: expand spec list (#39081)
+
+## Removals, deprecations, and syntax changes
+* ASP: targets, compilers and providers soft-preferences are only global (#31261)
+* Parser: fix ambiguity with whitespace in version ranges (#40344)
+* Module file generation is disabled by default; you'll need to enable it to use it (#37258)
+* Remove deprecated "extra_instructions" option for containers (#40365)
+* Stand-alone test feature deprecation postponed to v0.22 (#40600)
+* buildcache push: make `--allow-root` the default and deprecate the option (#38878)
+
+## Notable Bugfixes
+* Bugfix: propagation of multivalued variants (#39833)
+* Allow `/` in git versions (#39398)
+* Fetch & patch: actually acquire stage lock, and many more issues (#38903)
+* Environment/depfile: better escaping of targets with Git versions (#37560)
+* Prevent "spack external find" to error out on wrong permissions (#38755)
+* lmod: allow core compiler to be specified with a version range (#37789)
+
+## Spack community stats
+
+* 7,469 total packages, 303 new since `v0.20.0`
+    * 150 new Python packages
+    * 34 new R packages
+* 353 people contributed to this release
+    * 336 committers to packages
+    * 65 committers to core
+
+
 # v0.20.3 (2023-10-31)
 
 ## Bugfixes

etc/spack/defaults/darwin/packages.yaml

@@ -50,4 +50,4 @@ packages:
     # Apple bundles libuuid in libsystem_c version 1353.100.2,
     # although the version number used here isn't critical
     - spec: apple-libuuid@1353.100.2
-      prefix: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
+      prefix: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

etc/spack/defaults/modules.yaml

@@ -46,12 +46,10 @@ modules:
     tcl:
       all:
         autoload: direct
-      hide_implicits: true
 
     # Default configurations if lmod is enabled
     lmod:
       all:
         autoload: direct
-      hide_implicits: true
       hierarchy:
         - mpi

lib/spack/docs/binary_caches.rst

@@ -182,6 +182,7 @@ section of the configuration:
        padded_length: 128
 
 
+.. _binary_caches_oci:
 
 -----------------------------------------
 OCI / Docker V2 registries as build cache

lib/spack/docs/build_settings.rst

@@ -37,7 +37,11 @@ to enable reuse for a single installation, and you can use:
    spack install --fresh <spec>
 
 to do a fresh install if ``reuse`` is enabled by default.
-``reuse: true`` is the default.
+``reuse: dependencies`` is the default.
+
+.. seealso::
+
+   FAQ: :ref:`Why does Spack pick particular versions and variants? <faq-concretizer-precedence>`
 
 ------------------------------------------
 Selection of the target microarchitectures
@@ -99,547 +103,3 @@ while `py-numpy` still needs an older version:
 
 Up to Spack v0.20 ``duplicates:strategy:none`` was the default (and only) behavior. From Spack v0.21 the
 default behavior is ``duplicates:strategy:minimal``.
-
-.. _build-settings:
-
-================================
-Package Settings (packages.yaml)
-================================
-
-Spack allows you to customize how your software is built through the
-``packages.yaml`` file.  Using it, you can make Spack prefer particular
-implementations of virtual dependencies (e.g., MPI or BLAS/LAPACK),
-or you can make it prefer to build with particular compilers.  You can
-also tell Spack to use *external* software installations already
-present on your system.
-
-At a high level, the ``packages.yaml`` file is structured like this:
-
-.. code-block:: yaml
-
-   packages:
-     package1:
-       # settings for package1
-     package2:
-       # settings for package2
-     # ...
-     all:
-       # settings that apply to all packages.
-
-So you can either set build preferences specifically for *one* package,
-or you can specify that certain settings should apply to *all* packages.
-The types of settings you can customize are described in detail below.
-
-Spack's build defaults are in the default
-``etc/spack/defaults/packages.yaml`` file.  You can override them in
-``~/.spack/packages.yaml`` or ``etc/spack/packages.yaml``. For more
-details on how this works, see :ref:`configuration-scopes`.
-
-.. _sec-external-packages:
-
------------------
-External Packages
------------------
-
-Spack can be configured to use externally-installed
-packages rather than building its own packages. This may be desirable
-if machines ship with system packages, such as a customized MPI
-that should be used instead of Spack building its own MPI.
-
-External packages are configured through the ``packages.yaml`` file.
-Here's an example of an external configuration:
-
-.. code-block:: yaml
-
-   packages:
-     openmpi:
-       externals:
-       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
-         prefix: /opt/openmpi-1.4.3
-       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
-         prefix: /opt/openmpi-1.4.3-debug
-       - spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
-         prefix: /opt/openmpi-1.6.5-intel
-
-This example lists three installations of OpenMPI, one built with GCC,
-one built with GCC and debug information, and another built with Intel.
-If Spack is asked to build a package that uses one of these MPIs as a
-dependency, it will use the pre-installed OpenMPI in
-the given directory. Note that the specified path is the top-level
-install prefix, not the ``bin`` subdirectory.
-
-``packages.yaml`` can also be used to specify modules to load instead
-of the installation prefixes.  The following example says that module
-``CMake/3.7.2`` provides cmake version 3.7.2.
-
-.. code-block:: yaml
-
-   cmake:
-     externals:
-     - spec: cmake@3.7.2
-       modules:
-       - CMake/3.7.2
-
-Each ``packages.yaml`` begins with a ``packages:`` attribute, followed
-by a list of package names.  To specify externals, add an ``externals:``
-attribute under the package name, which lists externals.
-Each external should specify a ``spec:`` string that should be as
-well-defined as reasonably possible.  If a
-package lacks a spec component, such as missing a compiler or
-package version, then Spack will guess the missing component based
-on its most-favored packages, and it may guess incorrectly.
-
-Each package version and compiler listed in an external should
-have entries in Spack's packages and compiler configuration, even
-though the package and compiler may not ever be built.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Prevent packages from being built from sources
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Adding an external spec in ``packages.yaml`` allows Spack to use an external location,
-but it does not prevent Spack from building packages from sources. In the above example,
-Spack might choose for many valid reasons to start building and linking with the
-latest version of OpenMPI rather than continue using the pre-installed OpenMPI versions.
-
-To prevent this, the ``packages.yaml`` configuration also allows packages
-to be flagged as non-buildable.  The previous example could be modified to
-be:
-
-.. code-block:: yaml
-
-   packages:
-     openmpi:
-       externals:
-       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
-         prefix: /opt/openmpi-1.4.3
-       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
-         prefix: /opt/openmpi-1.4.3-debug
-       - spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
-         prefix: /opt/openmpi-1.6.5-intel
-       buildable: False
-
-The addition of the ``buildable`` flag tells Spack that it should never build
-its own version of OpenMPI from sources, and it will instead always rely on a pre-built
-OpenMPI.
-
-.. note::
-
-   If ``concretizer:reuse`` is on (see :ref:`concretizer-options` for more information on that flag)
-   pre-built specs include specs already available from a local store, an upstream store, a registered
-   buildcache or specs marked as externals in ``packages.yaml``. If ``concretizer:reuse`` is off, only
-   external specs in ``packages.yaml`` are included in the list of pre-built specs.
-
-If an external module is specified as not buildable, then Spack will load the
-external module into the build environment which can be used for linking.
-
-The ``buildable`` does not need to be paired with external packages.
-It could also be used alone to forbid packages that may be
-buggy or otherwise undesirable.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Non-buildable virtual packages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Virtual packages in Spack can also be specified as not buildable, and
-external implementations can be provided. In the example above,
-OpenMPI is configured as not buildable, but Spack will often prefer
-other MPI implementations over the externally available OpenMPI. Spack
-can be configured with every MPI provider not buildable individually,
-but more conveniently:
-
-.. code-block:: yaml
-
-   packages:
-     mpi:
-       buildable: False
-     openmpi:
-       externals:
-       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
-         prefix: /opt/openmpi-1.4.3
-       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
-         prefix: /opt/openmpi-1.4.3-debug
-       - spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
-         prefix: /opt/openmpi-1.6.5-intel
-
-Spack can then use any of the listed external implementations of MPI
-to satisfy a dependency, and will choose depending on the compiler and
-architecture.
-
-In cases where the concretizer is configured to reuse specs, and other ``mpi`` providers
-(available via stores or buildcaches) are not wanted, Spack can be configured to require
-specs matching only the available externals:
-
-.. code-block:: yaml
-
-   packages:
-     mpi:
-       buildable: False
-       require:
-       - one_of: [
-           "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64",
-           "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug",
-           "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
-         ]
-     openmpi:
-       externals:
-       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
-         prefix: /opt/openmpi-1.4.3
-       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
-         prefix: /opt/openmpi-1.4.3-debug
-       - spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
-         prefix: /opt/openmpi-1.6.5-intel
-
-This configuration prevents any spec using MPI and originating from stores or buildcaches to be reused,
-unless it matches the requirements under ``packages:mpi:require``. For more information on requirements see
-:ref:`package-requirements`.
-
-.. _cmd-spack-external-find:
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Automatically Find External Packages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-You can run the :ref:`spack external find <spack-external-find>` command
-to search for system-provided packages and add them to ``packages.yaml``.
-After running this command your ``packages.yaml`` may include new entries:
-
-.. code-block:: yaml
-
-   packages:
-     cmake:
-       externals:
-       - spec: cmake@3.17.2
-         prefix: /usr
-
-Generally this is useful for detecting a small set of commonly-used packages;
-for now this is generally limited to finding build-only dependencies.
-Specific limitations include:
-
-* Packages are not discoverable by default: For a package to be
-  discoverable with ``spack external find``, it needs to add special
-  logic. See :ref:`here <make-package-findable>` for more details.
-* The logic does not search through module files, it can only detect
-  packages with executables defined in ``PATH``; you can help Spack locate
-  externals which use module files by loading any associated modules for
-  packages that you want Spack to know about before running
-  ``spack external find``.
-* Spack does not overwrite existing entries in the package configuration:
-  If there is an external defined for a spec at any configuration scope,
-  then Spack will not add a new external entry (``spack config blame packages``
-  can help locate all external entries).
-
-.. _package-requirements:
-
---------------------
-Package Requirements
---------------------
-
-Spack can be configured to always use certain compilers, package
-versions, and variants during concretization through package
-requirements.
-
-Package requirements are useful when you find yourself repeatedly
-specifying the same constraints on the command line, and wish that
-Spack respects these constraints whether you mention them explicitly
-or not. Another use case is specifying constraints that should apply
-to all root specs in an environment, without having to repeat the
-constraint everywhere.
-
-Apart from that, requirements config is more flexible than constraints
-on the command line, because it can specify constraints on packages
-*when they occur* as a dependency. In contrast, on the command line it
-is not possible to specify constraints on dependencies while also keeping
-those dependencies optional.
-
-^^^^^^^^^^^^^^^^^^^
-Requirements syntax
-^^^^^^^^^^^^^^^^^^^
-
-The package requirements configuration is specified in ``packages.yaml``,
-keyed by package name and expressed using the Spec syntax. In the simplest
-case you can specify attributes that you always want the package to have
-by providing a single spec string to ``require``:
-
-.. code-block:: yaml
-
-   packages:
-     libfabric:
-       require: "@1.13.2"
-
-In the above example, ``libfabric`` will always build with version 1.13.2. If you
-need to compose multiple configuration scopes ``require`` accepts a list of
-strings:
-
-.. code-block:: yaml
-
-   packages:
-     libfabric:
-       require:
-       - "@1.13.2"
-       - "%gcc"
-
-In this case ``libfabric`` will always build with version 1.13.2 **and** using GCC
-as a compiler.
-
-For more complex use cases, require accepts also a list of objects. These objects
-must have either a ``any_of`` or a ``one_of`` field, containing a list of spec strings,
-and they can optionally have a ``when`` and a ``message`` attribute:
-
-.. code-block:: yaml
-
-   packages:
-     openmpi:
-       require:
-       - any_of: ["@4.1.5", "%gcc"]
-         message: "in this example only 4.1.5 can build with other compilers"
-
-``any_of`` is a list of specs. One of those specs must be satisfied
-and it is also allowed for the concretized spec to match more than one.
-In the above example, that means you could build ``openmpi@4.1.5%gcc``,
-``openmpi@4.1.5%clang`` or ``openmpi@3.9%gcc``, but
-not ``openmpi@3.9%clang``.
-
-If a custom message is provided, and the requirement is not satisfiable,
-Spack will print the custom error message:
-
-.. code-block:: console
-
-   $ spack spec openmpi@3.9%clang
-   ==> Error: in this example only 4.1.5 can build with other compilers
-
-We could express a similar requirement using the ``when`` attribute:
-
-.. code-block:: yaml
-
-   packages:
-     openmpi:
-       require:
-       - any_of: ["%gcc"]
-         when: "@:4.1.4"
-         message: "in this example only 4.1.5 can build with other compilers"
-
-In the example above, if the version turns out to be 4.1.4 or less, we require the compiler to be GCC.
-For readability, Spack also allows a ``spec`` key accepting a string when there is only a single
-constraint:
-
-.. code-block:: yaml
-
-   packages:
-     openmpi:
-       require:
-       - spec: "%gcc"
-         when: "@:4.1.4"
-         message: "in this example only 4.1.5 can build with other compilers"
-
-This code snippet and the one before it are semantically equivalent.
-
-Finally, instead of ``any_of`` you can use ``one_of`` which also takes a list of specs. The final
-concretized spec must match one and only one of them:
-
-.. code-block:: yaml
-
-   packages:
-     mpich:
-       require:
-       - one_of: ["+cuda", "+rocm"]
-
-In the example above, that means you could build ``mpich+cuda`` or ``mpich+rocm`` but not ``mpich+cuda+rocm``.
-
-.. note::
-
-   For ``any_of`` and ``one_of``, the order of specs indicates a
-   preference: items that appear earlier in the list are preferred
-   (note that these preferences can be ignored in favor of others).
-
-.. note::
-
-   When using a conditional requirement, Spack is allowed to actively avoid the triggering
-   condition (the ``when=...`` spec) if that leads to a concrete spec with better scores in
-   the optimization criteria. To check the current optimization criteria and their
-   priorities you can run ``spack solve zlib``.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Setting default requirements
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-You can also set default requirements for all packages under ``all``
-like this:
-
-.. code-block:: yaml
-
-   packages:
-     all:
-       require: '%clang'
-
-which means every spec will be required to use ``clang`` as a compiler.
-
-Note that in this case ``all`` represents a *default set of requirements* -
-if there are specific package requirements, then the default requirements
-under ``all`` are disregarded. For example, with a configuration like this:
-
-.. code-block:: yaml
-
-   packages:
-     all:
-       require: '%clang'
-     cmake:
-       require: '%gcc'
-
-Spack requires ``cmake`` to use ``gcc`` and all other nodes (including ``cmake``
-dependencies) to use ``clang``.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Setting requirements on virtual specs
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A requirement on a virtual spec applies whenever that virtual is present in the DAG.
-This can be useful for fixing which virtual provider you want to use:
-
-.. code-block:: yaml
-
-   packages:
-     mpi:
-       require: 'mvapich2 %gcc'
-
-With the configuration above the only allowed ``mpi`` provider is ``mvapich2 %gcc``.
-
-Requirements on the virtual spec and on the specific provider are both applied, if
-present. For instance with a configuration like:
-
-.. code-block:: yaml
-
-   packages:
-     mpi:
-       require: 'mvapich2 %gcc'
-     mvapich2:
-       require: '~cuda'
-
-you will use ``mvapich2~cuda %gcc`` as an ``mpi`` provider.
-
-.. _package-preferences:
-
--------------------
-Package Preferences
--------------------
-
-In some cases package requirements can be too strong, and package
-preferences are the better option. Package preferences do not impose
-constraints on packages for particular versions or variants values,
-they rather only set defaults. The concretizer is free to change
-them if it must, due to other constraints, and also prefers reusing
-installed packages over building new ones that are a better match for
-preferences.
-
-Most package preferences (``compilers``, ``target`` and ``providers``)
-can only be set globally under the ``all`` section of ``packages.yaml``:
-
-.. code-block:: yaml
-
-   packages:
-     all:
-       compiler: [gcc@12.2.0, clang@12:, oneapi@2023:]
-       target: [x86_64_v3]
-       providers:
-         mpi: [mvapich2, mpich, openmpi]
-
-These preferences override Spack's default and effectively reorder priorities
-when looking for the best compiler, target or virtual package provider. Each
-preference takes an ordered list of spec constraints, with earlier entries in
-the list being preferred over later entries.
-
-In the example above all packages prefer to be compiled with ``gcc@12.2.0``,
-to target the ``x86_64_v3`` microarchitecture and to use ``mvapich2`` if they
-depend on ``mpi``.
-
-The ``variants`` and ``version`` preferences can be set under
-package specific sections of the ``packages.yaml`` file:
-
-.. code-block:: yaml
-
-   packages:
-     opencv:
-       variants: +debug
-     gperftools:
-       version: [2.2, 2.4, 2.3]
-
-In this case, the preference for ``opencv`` is to build with debug options, while
-``gperftools`` prefers version 2.2 over 2.4.
-
-Any preference can be overwritten on the command line if explicitly requested.
-
-Preferences cannot overcome explicit constraints, as they only set a preferred
-ordering among homogeneous attribute values. Going back to the example, if
-``gperftools@2.3:`` was requested, then Spack will install version 2.4
-since the most preferred version 2.2 is prohibited by the version constraint.
-
-.. _package_permissions:
-
--------------------
-Package Permissions
--------------------
-
-Spack can be configured to assign permissions to the files installed
-by a package.
-
-In the ``packages.yaml`` file under ``permissions``, the attributes
-``read``, ``write``, and ``group`` control the package
-permissions. These attributes can be set per-package, or for all
-packages under ``all``. If permissions are set under ``all`` and for a
-specific package, the package-specific settings take precedence.
-
-The ``read`` and ``write`` attributes take one of ``user``, ``group``,
-and ``world``.
-
-.. code-block:: yaml
-
-  packages:
-    all:
-      permissions:
-        write: group
-        group: spack
-    my_app:
-      permissions:
-        read: group
-        group: my_team
-
-The permissions settings describe the broadest level of access to
-installations of the specified packages. The execute permissions of
-the file are set to the same level as read permissions for those files
-that are executable. The default setting for ``read`` is ``world``,
-and for ``write`` is ``user``. In the example above, installations of
-``my_app`` will be installed with user and group permissions but no
-world permissions, and owned by the group ``my_team``. All other
-packages will be installed with user and group write privileges, and
-world read privileges. Those packages will be owned by the group
-``spack``.
-
-The ``group`` attribute assigns a Unix-style group to a package. All
-files installed by the package will be owned by the assigned group,
-and the sticky group bit will be set on the install prefix and all
-directories inside the install prefix. This will ensure that even
-manually placed files within the install prefix are owned by the
-assigned group. If no group is assigned, Spack will allow the OS
-default behavior to go as expected.
-
-----------------------------
-Assigning Package Attributes
-----------------------------
-
-You can assign class-level attributes in the configuration:
-
-.. code-block:: yaml
-
-  packages:
-    mpileaks:
-      # Override existing attributes
-      url: http://www.somewhereelse.com/mpileaks-1.0.tar.gz
-      # ... or add new ones
-      x: 1
-
-Attributes set this way will be accessible to any method executed
-in the package.py file (e.g. the ``install()`` method). Values for these
-attributes may be any value parseable by yaml.
-
-These can only be applied to specific packages, not "all" or
-virtual packages.

lib/spack/docs/build_systems/cmakepackage.rst

@@ -82,7 +82,7 @@ class already contains:
 
 .. code-block:: python
 
-   depends_on('cmake', type='build')
+   depends_on("cmake", type="build")
 
 
 If you need to specify a particular version requirement, you can
@@ -90,7 +90,7 @@ override this in your package:
 
 .. code-block:: python
 
-   depends_on('cmake@2.8.12:', type='build')
+   depends_on("cmake@2.8.12:", type="build")
 
 
 ^^^^^^^^^^^^^^^^^^^
@@ -137,10 +137,10 @@ and without the :meth:`~spack.build_systems.cmake.CMakeBuilder.define` and
 
    def cmake_args(self):
        args = [
-           '-DWHATEVER:STRING=somevalue',
-           self.define('ENABLE_BROKEN_FEATURE', False),
-           self.define_from_variant('DETECT_HDF5', 'hdf5'),
-           self.define_from_variant('THREADS'), # True if +threads
+           "-DWHATEVER:STRING=somevalue",
+           self.define("ENABLE_BROKEN_FEATURE", False),
+           self.define_from_variant("DETECT_HDF5", "hdf5"),
+           self.define_from_variant("THREADS"), # True if +threads
        ]
 
        return args
@@ -151,10 +151,10 @@ and CMake simply ignores the empty command line argument. For example the follow
 
 .. code-block:: python
 
-   variant('example', default=True, when='@2.0:')
+   variant("example", default=True, when="@2.0:")
 
    def cmake_args(self):
-      return [self.define_from_variant('EXAMPLE', 'example')]
+      return [self.define_from_variant("EXAMPLE", "example")]
 
 will generate ``'cmake' '-DEXAMPLE=ON' ...`` when `@2.0: +example` is met, but will
 result in ``'cmake' '' ...`` when the spec version is below ``2.0``.
@@ -193,9 +193,9 @@ a variant to control this:
 
 .. code-block:: python
 
-   variant('build_type', default='RelWithDebInfo',
-           description='CMake build type',
-           values=('Debug', 'Release', 'RelWithDebInfo', 'MinSizeRel'))
+   variant("build_type", default="RelWithDebInfo",
+           description="CMake build type",
+           values=("Debug", "Release", "RelWithDebInfo", "MinSizeRel"))
 
 However, not every CMake package accepts all four of these options.
 Grep the ``CMakeLists.txt`` file to see if the default values are
@@ -205,9 +205,9 @@ package overrides the default variant with:
 
 .. code-block:: python
 
-   variant('build_type', default='DebugRelease',
-           description='The build type to build',
-           values=('Debug', 'Release', 'DebugRelease'))
+   variant("build_type", default="DebugRelease",
+           description="The build type to build",
+           values=("Debug", "Release", "DebugRelease"))
 
 For more information on ``CMAKE_BUILD_TYPE``, see:
 https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html
@@ -250,7 +250,7 @@ generator is Ninja. To switch to the Ninja generator, simply add:
 
 .. code-block:: python
 
-   generator = 'Ninja'
+   generator = "Ninja"
 
 
 ``CMakePackage`` defaults to "Unix Makefiles". If you switch to the
@@ -258,7 +258,7 @@ Ninja generator, make sure to add:
 
 .. code-block:: python
 
-   depends_on('ninja', type='build')
+   depends_on("ninja", type="build")
 
 to the package as well. Aside from that, you shouldn't need to do
 anything else. Spack will automatically detect that you are using
@@ -288,7 +288,7 @@ like so:
 
 .. code-block:: python
 
-   root_cmakelists_dir = 'src'
+   root_cmakelists_dir = "src"
 
 
 Note that this path is relative to the root of the extracted tarball,
@@ -304,7 +304,7 @@ different sub-directory, simply override ``build_directory`` like so:
 
 .. code-block:: python
 
-   build_directory = 'my-build'
+   build_directory = "my-build"
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 Build and install targets
@@ -324,8 +324,8 @@ library or build the documentation, you can add these like so:
 
 .. code-block:: python
 
-   build_targets = ['all', 'docs']
-   install_targets = ['install', 'docs']
+   build_targets = ["all", "docs"]
+   install_targets = ["install", "docs"]
 
 ^^^^^^^
 Testing

lib/spack/docs/build_systems/inteloneapipackage.rst

@@ -53,18 +53,24 @@ Install the oneAPI compilers::
 
 Add the compilers to your ``compilers.yaml`` so spack can use them::
 
-  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin/intel64
-  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin
+  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin
 
 Verify that the compilers are available::
 
   spack compiler list
 
+Note that 2024 and later releases do not include ``icc``. Before 2024,
+the package layout was different::
+  
+  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin/intel64
+  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin
+
 The ``intel-oneapi-compilers`` package includes 2 families of
 compilers:
 
 * ``intel``: ``icc``, ``icpc``, ``ifort``. Intel's *classic*
-  compilers.
+  compilers. 2024 and later releases contain ``ifort``, but not
+  ``icc`` and ``icpc``.
 * ``oneapi``: ``icx``, ``icpx``, ``ifx``. Intel's new generation of
   compilers based on LLVM.
 
@@ -89,8 +95,8 @@ Install the oneAPI compilers::
 
 Add the compilers to your ``compilers.yaml`` so Spack can use them::
 
-  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin/intel64
-  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin
+  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin
+  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin
 
 Verify that the compilers are available::
 
@@ -146,8 +152,7 @@ Compilers
 To use the compilers, add some information about the installation to
 ``compilers.yaml``. For most users, it is sufficient to do::
 
-  spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin/intel64
-  spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin
+  spack compiler add /opt/intel/oneapi/compiler/latest/bin
 
 Adapt the paths above if you did not install the tools in the default
 location. After adding the compilers, using them is the same
@@ -156,6 +161,12 @@ Another option is to manually add the configuration to
 ``compilers.yaml`` as described in :ref:`Compiler configuration
 <compiler-config>`.
 
+Before 2024, the directory structure was different::
+  
+  spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin/intel64
+  spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin
+
+
 Libraries
 ---------
 

lib/spack/docs/build_systems/intelpackage.rst

@@ -392,7 +392,7 @@ See section
 :ref:`Configuration Scopes <configuration-scopes>`
 for an explanation about the different files
 and section
-:ref:`Build customization <build-settings>`
+:ref:`Build customization <packages-config>`
 for specifics and examples for ``packages.yaml`` files.
 
 .. If your system administrator did not provide modules for pre-installed Intel
@@ -934,9 +934,9 @@ a *virtual* ``mkl`` package is declared in Spack.
   .. code-block:: python
 
      # Examples for absolute and conditional dependencies:
-     depends_on('mkl')
-     depends_on('mkl', when='+mkl')
-     depends_on('mkl', when='fftw=mkl')
+     depends_on("mkl")
+     depends_on("mkl", when="+mkl")
+     depends_on("mkl", when="fftw=mkl")
 
   The ``MKLROOT`` environment variable (part of the documented API) will be set
   during all stages of client package installation, and is available to both
@@ -972,8 +972,8 @@ a *virtual* ``mkl`` package is declared in Spack.
       def configure_args(self):
           args = []
           ...
-          args.append('--with-blas=%s' % self.spec['blas'].libs.ld_flags)
-          args.append('--with-lapack=%s' % self.spec['lapack'].libs.ld_flags)
+          args.append("--with-blas=%s" % self.spec["blas"].libs.ld_flags)
+          args.append("--with-lapack=%s" % self.spec["lapack"].libs.ld_flags)
           ...
 
   .. tip::
@@ -989,13 +989,13 @@ a *virtual* ``mkl`` package is declared in Spack.
 
   .. code-block:: python
 
-    self.spec['blas'].headers.include_flags
+    self.spec["blas"].headers.include_flags
 
   and to generate linker options (``-L<dir> -llibname ...``), use the same as above,
 
   .. code-block:: python
 
-    self.spec['blas'].libs.ld_flags
+    self.spec["blas"].libs.ld_flags
 
   See
   :ref:`MakefilePackage <makefilepackage>`

lib/spack/docs/build_systems/luapackage.rst

@@ -88,7 +88,7 @@ override the ``luarocks_args`` method like so:
 .. code-block:: python
 
     def luarocks_args(self):
-        return ['flag1', 'flag2']
+        return ["flag1", "flag2"]
 
 One common use of this is to override warnings or flags for newer compilers, as in:
 

lib/spack/docs/build_systems/mavenpackage.rst

@@ -48,8 +48,8 @@ class automatically adds the following dependencies:
 
 .. code-block:: python
 
-   depends_on('java', type=('build', 'run'))
-   depends_on('maven', type='build')
+   depends_on("java", type=("build", "run"))
+   depends_on("maven", type="build")
 
 
 In the ``pom.xml`` file, you may see sections like:
@@ -72,8 +72,8 @@ should add:
 
 .. code-block:: python
 
-   depends_on('java@7:', type='build')
-   depends_on('maven@3.5.4:', type='build')
+   depends_on("java@7:", type="build")
+   depends_on("maven@3.5.4:", type="build")
 
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -88,9 +88,9 @@ the build phase. For example:
 
    def build_args(self):
        return [
-           '-Pdist,native',
-           '-Dtar',
-           '-Dmaven.javadoc.skip=true'
+           "-Pdist,native",
+           "-Dtar",
+           "-Dmaven.javadoc.skip=true"
        ]
 
 

lib/spack/docs/build_systems/mesonpackage.rst

@@ -86,8 +86,8 @@ the ``MesonPackage`` base class already contains:
 
 .. code-block:: python
 
-   depends_on('meson', type='build')
-   depends_on('ninja', type='build')
+   depends_on("meson", type="build")
+   depends_on("ninja", type="build")
 
 
 If you need to specify a particular version requirement, you can
@@ -95,8 +95,8 @@ override this in your package:
 
 .. code-block:: python
 
-   depends_on('meson@0.43.0:', type='build')
-   depends_on('ninja', type='build')
+   depends_on("meson@0.43.0:", type="build")
+   depends_on("ninja", type="build")
 
 
 ^^^^^^^^^^^^^^^^^^^
@@ -121,7 +121,7 @@ override the ``meson_args`` method like so:
 .. code-block:: python
 
    def meson_args(self):
-       return ['--warnlevel=3']
+       return ["--warnlevel=3"]
 
 
 This method can be used to pass flags as well as variables.

lib/spack/docs/build_systems/perlpackage.rst

@@ -118,7 +118,7 @@ so ``PerlPackage`` contains:
 
 .. code-block:: python
 
-   extends('perl')
+   extends("perl")
 
 
 If your package requires a specific version of Perl, you should
@@ -132,14 +132,14 @@ properly. If your package uses ``Makefile.PL`` to build, add:
 
 .. code-block:: python
 
-   depends_on('perl-extutils-makemaker', type='build')
+   depends_on("perl-extutils-makemaker", type="build")
 
 
 If your package uses ``Build.PL`` to build, add:
 
 .. code-block:: python
 
-   depends_on('perl-module-build', type='build')
+   depends_on("perl-module-build", type="build")
 
 
 ^^^^^^^^^^^^^^^^^
@@ -165,11 +165,11 @@ arguments to ``Makefile.PL`` or ``Build.PL`` by overriding
 .. code-block:: python
 
    def configure_args(self):
-       expat = self.spec['expat'].prefix
+       expat = self.spec["expat"].prefix
 
        return [
-           'EXPATLIBPATH={0}'.format(expat.lib),
-           'EXPATINCPATH={0}'.format(expat.include),
+           "EXPATLIBPATH={0}".format(expat.lib),
+           "EXPATINCPATH={0}".format(expat.include),
        ]
 
 

lib/spack/docs/build_systems/qmakepackage.rst

@@ -83,7 +83,7 @@ base class already contains:
 
 .. code-block:: python
 
-   depends_on('qt', type='build')
+   depends_on("qt", type="build")
 
 
 If you want to specify a particular version requirement, or need to
@@ -91,7 +91,7 @@ link to the ``qt`` libraries, you can override this in your package:
 
 .. code-block:: python
 
-   depends_on('qt@5.6.0:')
+   depends_on("qt@5.6.0:")
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 Passing arguments to qmake
@@ -103,7 +103,7 @@ override the ``qmake_args`` method like so:
 .. code-block:: python
 
    def qmake_args(self):
-       return ['-recursive']
+       return ["-recursive"]
 
 
 This method can be used to pass flags as well as variables.
@@ -118,7 +118,7 @@ sub-directory by adding the following to the package:
 
 .. code-block:: python
 
-   build_directory = 'src'
+   build_directory = "src"
 
 
 ^^^^^^^^^^^^^^^^^^^^^^

lib/spack/docs/build_systems/rpackage.rst

@@ -163,28 +163,28 @@ attributes that can be used to set ``homepage``, ``url``, ``list_url``, and
 
 .. code-block:: python
 
-   cran = 'caret'
+   cran = "caret"
 
 is equivalent to:
 
 .. code-block:: python
 
-   homepage = 'https://cloud.r-project.org/package=caret'
-   url      = 'https://cloud.r-project.org/src/contrib/caret_6.0-86.tar.gz'
-   list_url = 'https://cloud.r-project.org/src/contrib/Archive/caret'
+   homepage = "https://cloud.r-project.org/package=caret"
+   url      = "https://cloud.r-project.org/src/contrib/caret_6.0-86.tar.gz"
+   list_url = "https://cloud.r-project.org/src/contrib/Archive/caret"
 
 Likewise, the following ``bioc`` attribute:
 
 .. code-block:: python
 
-   bioc = 'BiocVersion'
+   bioc = "BiocVersion"
 
 is equivalent to:
 
 .. code-block:: python
 
-   homepage = 'https://bioconductor.org/packages/BiocVersion/'
-   git      = 'https://git.bioconductor.org/packages/BiocVersion'
+   homepage = "https://bioconductor.org/packages/BiocVersion/"
+   git      = "https://git.bioconductor.org/packages/BiocVersion"
 
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -200,7 +200,7 @@ base class contains:
 
 .. code-block:: python
 
-   extends('r')
+   extends("r")
 
 
 Take a close look at the homepage for ``caret``. If you look at the
@@ -209,7 +209,7 @@ You should add this to your package like so:
 
 .. code-block:: python
 
-   depends_on('r@3.2.0:', type=('build', 'run'))
+   depends_on("r@3.2.0:", type=("build", "run"))
 
 
 ^^^^^^^^^^^^^^
@@ -227,7 +227,7 @@ and list all of their dependencies in the following sections:
 * LinkingTo
 
 As far as Spack is concerned, all 3 of these dependency types
-correspond to ``type=('build', 'run')``, so you don't have to worry
+correspond to ``type=("build", "run")``, so you don't have to worry
 about the details. If you are curious what they mean,
 https://github.com/spack/spack/issues/2951 has a pretty good summary:
 
@@ -330,7 +330,7 @@ the dependency:
 
 .. code-block:: python
 
-   depends_on('r-lattice@0.20:', type=('build', 'run'))
+   depends_on("r-lattice@0.20:", type=("build", "run"))
 
 
 ^^^^^^^^^^^^^^^^^^
@@ -361,20 +361,20 @@ like so:
 .. code-block:: python
 
    def configure_args(self):
-       mpi_name = self.spec['mpi'].name
+       mpi_name = self.spec["mpi"].name
 
        # The type of MPI. Supported values are:
        # OPENMPI, LAM, MPICH, MPICH2, or CRAY
-       if mpi_name == 'openmpi':
-           Rmpi_type = 'OPENMPI'
-       elif mpi_name == 'mpich':
-           Rmpi_type = 'MPICH2'
+       if mpi_name == "openmpi":
+           Rmpi_type = "OPENMPI"
+       elif mpi_name == "mpich":
+           Rmpi_type = "MPICH2"
        else:
-           raise InstallError('Unsupported MPI type')
+           raise InstallError("Unsupported MPI type")
 
        return [
-           '--with-Rmpi-type={0}'.format(Rmpi_type),
-           '--with-mpi={0}'.format(spec['mpi'].prefix),
+           "--with-Rmpi-type={0}".format(Rmpi_type),
+           "--with-mpi={0}".format(spec["mpi"].prefix),
        ]
 
 

lib/spack/docs/build_systems/rubypackage.rst

@@ -84,8 +84,8 @@ The ``*.gemspec`` file may contain something like:
 
 .. code-block:: ruby
 
-   summary = 'An implementation of the AsciiDoc text processor and publishing toolchain'
-   description = 'A fast, open source text processor and publishing toolchain for converting AsciiDoc content to HTML 5, DocBook 5, and other formats.'
+   summary = "An implementation of the AsciiDoc text processor and publishing toolchain"
+   description = "A fast, open source text processor and publishing toolchain for converting AsciiDoc content to HTML 5, DocBook 5, and other formats."
 
 
 Either of these can be used for the description of the Spack package.
@@ -98,7 +98,7 @@ The ``*.gemspec`` file may contain something like:
 
 .. code-block:: ruby
 
-   homepage = 'https://asciidoctor.org'
+   homepage = "https://asciidoctor.org"
 
 
 This should be used as the official homepage of the Spack package.
@@ -112,21 +112,21 @@ the base class contains:
 
 .. code-block:: python
 
-   extends('ruby')
+   extends("ruby")
 
 
 The ``*.gemspec`` file may contain something like:
 
 .. code-block:: ruby
 
-   required_ruby_version = '>= 2.3.0'
+   required_ruby_version = ">= 2.3.0"
 
 
 This can be added to the Spack package using:
 
 .. code-block:: python
 
-   depends_on('ruby@2.3.0:', type=('build', 'run'))
+   depends_on("ruby@2.3.0:", type=("build", "run"))
 
 
 ^^^^^^^^^^^^^^^^^

lib/spack/docs/build_systems/sippackage.rst

@@ -124,7 +124,7 @@ are wrong, you can provide the names yourself by overriding
 
 .. code-block:: python
 
-   import_modules = ['PyQt5']
+   import_modules = ["PyQt5"]
 
 
 These tests often catch missing dependencies and non-RPATHed

lib/spack/docs/build_systems/wafpackage.rst

@@ -63,8 +63,8 @@ run package-specific unit tests.
 .. code-block:: python
 
    def installtest(self):
-       with working_dir('test'):
-           pytest = which('py.test')
+       with working_dir("test"):
+           pytest = which("py.test")
            pytest()
 
 
@@ -93,7 +93,7 @@ the following dependency automatically:
 
 .. code-block:: python
 
-   depends_on('python@2.5:', type='build')
+   depends_on("python@2.5:", type="build")
 
 
 Waf only supports Python 2.5 and up.
@@ -113,7 +113,7 @@ phase, you can use:
        args = []
 
        if self.run_tests:
-           args.append('--test')
+           args.append("--test")
 
        return args
 

lib/spack/docs/configuration.rst

@@ -17,7 +17,7 @@ case you want to skip directly to specific docs:
 * :ref:`config.yaml <config-yaml>`
 * :ref:`mirrors.yaml <mirrors>`
 * :ref:`modules.yaml <modules>`
-* :ref:`packages.yaml <build-settings>`
+* :ref:`packages.yaml <packages-config>`
 * :ref:`repos.yaml <repositories>`
 
 You can also add any of these as inline configuration in the YAML
@@ -243,9 +243,11 @@ lower-precedence settings. Completely ignoring higher-level configuration
 options is supported with the ``::`` notation for keys (see
 :ref:`config-overrides` below).
 
-There are also special notations for string concatenation and precendense override.
-Using the ``+:`` notation  can be used to force *prepending* strings or lists. For lists, this is identical
-to the default behavior. Using the ``-:`` works similarly, but for *appending* values.
+There are also special notations for string concatenation and precendense override:
+
+* ``+:`` will force *prepending* strings or lists. For lists, this is the default behavior.
+* ``-:`` works similarly, but for *appending* values.
+
 :ref:`config-prepend-append`
 
 ^^^^^^^^^^^

lib/spack/docs/containers.rst

@@ -24,6 +24,16 @@ image, or to set up a proper entrypoint to run the image. These tasks are
 usually both necessary and repetitive, so Spack comes with a command
 to generate recipes for container images starting from a ``spack.yaml``.
 
+.. seealso::
+
+  This page is a reference for generating recipes to build container images.
+  It means that your environment is built from scratch inside the container
+  runtime.
+
+  Since v0.21, Spack can also create container images from existing package installations
+  on your host system. See :ref:`binary_caches_oci` for more information on
+  that topic.
+
 --------------------
 A Quick Introduction
 --------------------

lib/spack/docs/frequently_asked_questions.rst

@@ -0,0 +1,77 @@
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+   SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+==========================
+Frequently Asked Questions
+==========================
+
+This page contains answers to frequently asked questions about Spack.
+If you have questions that are not answered here, feel free to ask on
+`Slack <https://slack.spack.io>`_ or `GitHub Discussions
+<https://github.com/spack/spack/discussions>`_. If you've learned the
+answer to a question that you think should be here, please consider
+contributing to this page.
+
+.. _faq-concretizer-precedence:
+
+-----------------------------------------------------
+Why does Spack pick particular versions and variants?
+-----------------------------------------------------
+
+This question comes up in a variety of forms:
+
+ 1. Why does Spack seem to ignore my package preferences from ``packages.yaml`` config?
+ 2. Why does Spack toggle a variant instead of using the default from the ``package.py`` file?
+
+The short answer is that Spack always picks an optimal configuration
+based on a complex set of criteria\ [#f1]_. These criteria are more nuanced
+than always choosing the latest versions or default variants.
+
+.. note::
+
+    As a rule of thumb: requirements + constraints > reuse > preferences > defaults.
+
+The following set of criteria (from lowest to highest precedence) explain
+common cases where concretization output may seem surprising at first.
+
+1. :ref:`Package preferences <package-preferences>` configured in ``packages.yaml``
+   override variant defaults from ``package.py`` files, and influence the optimal
+   ordering of versions. Preferences are specified as follows:
+
+   .. code-block:: yaml
+
+      packages:
+        foo:
+          version: [1.0, 1.1]
+          variants: ~mpi
+
+2. :ref:`Reuse concretization <concretizer-options>` configured in ``concretizer.yaml``
+   overrides preferences, since it's typically faster to reuse an existing spec than to
+   build a preferred one from sources. When build caches are enabled, specs may be reused
+   from a remote location too. Reuse concretization is configured as follows:
+
+   .. code-block:: yaml
+
+      concretizer:
+        reuse: dependencies  # other options are 'true' and 'false'
+
+3. :ref:`Package requirements <package-requirements>` configured in ``packages.yaml``,
+   and constraints from the command line as well as ``package.py`` files override all
+   of the above. Requirements are specified as follows:
+
+   .. code-block:: yaml
+
+      packages:
+        foo:
+          require:
+          - "@1.2: +mpi"
+
+Requirements and constraints restrict the set of possible solutions, while reuse
+behavior and preferences influence what an optimal solution looks like.
+
+
+.. rubric:: Footnotes
+
+.. [#f1] The exact list of criteria can be retrieved with the ``spack solve`` command

lib/spack/docs/index.rst

@@ -55,6 +55,7 @@ or refer to the full manual below.
    getting_started
    basic_usage
    replace_conda_homebrew
+   frequently_asked_questions
 
 .. toctree::
    :maxdepth: 2
@@ -70,7 +71,7 @@ or refer to the full manual below.
 
    configuration
    config_yaml
-   bootstrapping
+   packages_yaml
    build_settings
    environments
    containers
@@ -78,6 +79,7 @@ or refer to the full manual below.
    module_file_support
    repositories
    binary_caches
+   bootstrapping
    command_index
    chain
    extensions

lib/spack/docs/packages_yaml.rst

@@ -0,0 +1,591 @@
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+   SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+
+.. _packages-config:
+
+================================
+Package Settings (packages.yaml)
+================================
+
+Spack allows you to customize how your software is built through the
+``packages.yaml`` file.  Using it, you can make Spack prefer particular
+implementations of virtual dependencies (e.g., MPI or BLAS/LAPACK),
+or you can make it prefer to build with particular compilers.  You can
+also tell Spack to use *external* software installations already
+present on your system.
+
+At a high level, the ``packages.yaml`` file is structured like this:
+
+.. code-block:: yaml
+
+   packages:
+     package1:
+       # settings for package1
+     package2:
+       # settings for package2
+     # ...
+     all:
+       # settings that apply to all packages.
+
+So you can either set build preferences specifically for *one* package,
+or you can specify that certain settings should apply to *all* packages.
+The types of settings you can customize are described in detail below.
+
+Spack's build defaults are in the default
+``etc/spack/defaults/packages.yaml`` file.  You can override them in
+``~/.spack/packages.yaml`` or ``etc/spack/packages.yaml``. For more
+details on how this works, see :ref:`configuration-scopes`.
+
+.. _sec-external-packages:
+
+-----------------
+External Packages
+-----------------
+
+Spack can be configured to use externally-installed
+packages rather than building its own packages. This may be desirable
+if machines ship with system packages, such as a customized MPI
+that should be used instead of Spack building its own MPI.
+
+External packages are configured through the ``packages.yaml`` file.
+Here's an example of an external configuration:
+
+.. code-block:: yaml
+
+   packages:
+     openmpi:
+       externals:
+       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
+         prefix: /opt/openmpi-1.4.3
+       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
+         prefix: /opt/openmpi-1.4.3-debug
+       - spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
+         prefix: /opt/openmpi-1.6.5-intel
+
+This example lists three installations of OpenMPI, one built with GCC,
+one built with GCC and debug information, and another built with Intel.
+If Spack is asked to build a package that uses one of these MPIs as a
+dependency, it will use the pre-installed OpenMPI in
+the given directory. Note that the specified path is the top-level
+install prefix, not the ``bin`` subdirectory.
+
+``packages.yaml`` can also be used to specify modules to load instead
+of the installation prefixes.  The following example says that module
+``CMake/3.7.2`` provides cmake version 3.7.2.
+
+.. code-block:: yaml
+
+   cmake:
+     externals:
+     - spec: cmake@3.7.2
+       modules:
+       - CMake/3.7.2
+
+Each ``packages.yaml`` begins with a ``packages:`` attribute, followed
+by a list of package names.  To specify externals, add an ``externals:``
+attribute under the package name, which lists externals.
+Each external should specify a ``spec:`` string that should be as
+well-defined as reasonably possible.  If a
+package lacks a spec component, such as missing a compiler or
+package version, then Spack will guess the missing component based
+on its most-favored packages, and it may guess incorrectly.
+
+Each package version and compiler listed in an external should
+have entries in Spack's packages and compiler configuration, even
+though the package and compiler may not ever be built.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Prevent packages from being built from sources
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Adding an external spec in ``packages.yaml`` allows Spack to use an external location,
+but it does not prevent Spack from building packages from sources. In the above example,
+Spack might choose for many valid reasons to start building and linking with the
+latest version of OpenMPI rather than continue using the pre-installed OpenMPI versions.
+
+To prevent this, the ``packages.yaml`` configuration also allows packages
+to be flagged as non-buildable.  The previous example could be modified to
+be:
+
+.. code-block:: yaml
+
+   packages:
+     openmpi:
+       externals:
+       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
+         prefix: /opt/openmpi-1.4.3
+       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
+         prefix: /opt/openmpi-1.4.3-debug
+       - spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
+         prefix: /opt/openmpi-1.6.5-intel
+       buildable: False
+
+The addition of the ``buildable`` flag tells Spack that it should never build
+its own version of OpenMPI from sources, and it will instead always rely on a pre-built
+OpenMPI.
+
+.. note::
+
+   If ``concretizer:reuse`` is on (see :ref:`concretizer-options` for more information on that flag)
+   pre-built specs include specs already available from a local store, an upstream store, a registered
+   buildcache or specs marked as externals in ``packages.yaml``. If ``concretizer:reuse`` is off, only
+   external specs in ``packages.yaml`` are included in the list of pre-built specs.
+
+If an external module is specified as not buildable, then Spack will load the
+external module into the build environment which can be used for linking.
+
+The ``buildable`` does not need to be paired with external packages.
+It could also be used alone to forbid packages that may be
+buggy or otherwise undesirable.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Non-buildable virtual packages
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Virtual packages in Spack can also be specified as not buildable, and
+external implementations can be provided. In the example above,
+OpenMPI is configured as not buildable, but Spack will often prefer
+other MPI implementations over the externally available OpenMPI. Spack
+can be configured with every MPI provider not buildable individually,
+but more conveniently:
+
+.. code-block:: yaml
+
+   packages:
+     mpi:
+       buildable: False
+     openmpi:
+       externals:
+       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
+         prefix: /opt/openmpi-1.4.3
+       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
+         prefix: /opt/openmpi-1.4.3-debug
+       - spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
+         prefix: /opt/openmpi-1.6.5-intel
+
+Spack can then use any of the listed external implementations of MPI
+to satisfy a dependency, and will choose depending on the compiler and
+architecture.
+
+In cases where the concretizer is configured to reuse specs, and other ``mpi`` providers
+(available via stores or buildcaches) are not wanted, Spack can be configured to require
+specs matching only the available externals:
+
+.. code-block:: yaml
+
+   packages:
+     mpi:
+       buildable: False
+       require:
+       - one_of: [
+           "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64",
+           "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug",
+           "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
+         ]
+     openmpi:
+       externals:
+       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64"
+         prefix: /opt/openmpi-1.4.3
+       - spec: "openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug"
+         prefix: /opt/openmpi-1.4.3-debug
+       - spec: "openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64"
+         prefix: /opt/openmpi-1.6.5-intel
+
+This configuration prevents any spec using MPI and originating from stores or buildcaches to be reused,
+unless it matches the requirements under ``packages:mpi:require``. For more information on requirements see
+:ref:`package-requirements`.
+
+.. _cmd-spack-external-find:
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Automatically Find External Packages
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can run the :ref:`spack external find <spack-external-find>` command
+to search for system-provided packages and add them to ``packages.yaml``.
+After running this command your ``packages.yaml`` may include new entries:
+
+.. code-block:: yaml
+
+   packages:
+     cmake:
+       externals:
+       - spec: cmake@3.17.2
+         prefix: /usr
+
+Generally this is useful for detecting a small set of commonly-used packages;
+for now this is generally limited to finding build-only dependencies.
+Specific limitations include:
+
+* Packages are not discoverable by default: For a package to be
+  discoverable with ``spack external find``, it needs to add special
+  logic. See :ref:`here <make-package-findable>` for more details.
+* The logic does not search through module files, it can only detect
+  packages with executables defined in ``PATH``; you can help Spack locate
+  externals which use module files by loading any associated modules for
+  packages that you want Spack to know about before running
+  ``spack external find``.
+* Spack does not overwrite existing entries in the package configuration:
+  If there is an external defined for a spec at any configuration scope,
+  then Spack will not add a new external entry (``spack config blame packages``
+  can help locate all external entries).
+
+.. _package-requirements:
+
+--------------------
+Package Requirements
+--------------------
+
+Spack can be configured to always use certain compilers, package
+versions, and variants during concretization through package
+requirements.
+
+Package requirements are useful when you find yourself repeatedly
+specifying the same constraints on the command line, and wish that
+Spack respects these constraints whether you mention them explicitly
+or not. Another use case is specifying constraints that should apply
+to all root specs in an environment, without having to repeat the
+constraint everywhere.
+
+Apart from that, requirements config is more flexible than constraints
+on the command line, because it can specify constraints on packages
+*when they occur* as a dependency. In contrast, on the command line it
+is not possible to specify constraints on dependencies while also keeping
+those dependencies optional.
+
+.. seealso::
+
+   FAQ: :ref:`Why does Spack pick particular versions and variants? <faq-concretizer-precedence>`
+
+
+^^^^^^^^^^^^^^^^^^^
+Requirements syntax
+^^^^^^^^^^^^^^^^^^^
+
+The package requirements configuration is specified in ``packages.yaml``,
+keyed by package name and expressed using the Spec syntax. In the simplest
+case you can specify attributes that you always want the package to have
+by providing a single spec string to ``require``:
+
+.. code-block:: yaml
+
+   packages:
+     libfabric:
+       require: "@1.13.2"
+
+In the above example, ``libfabric`` will always build with version 1.13.2. If you
+need to compose multiple configuration scopes ``require`` accepts a list of
+strings:
+
+.. code-block:: yaml
+
+   packages:
+     libfabric:
+       require:
+       - "@1.13.2"
+       - "%gcc"
+
+In this case ``libfabric`` will always build with version 1.13.2 **and** using GCC
+as a compiler.
+
+For more complex use cases, require accepts also a list of objects. These objects
+must have either a ``any_of`` or a ``one_of`` field, containing a list of spec strings,
+and they can optionally have a ``when`` and a ``message`` attribute:
+
+.. code-block:: yaml
+
+   packages:
+     openmpi:
+       require:
+       - any_of: ["@4.1.5", "%gcc"]
+         message: "in this example only 4.1.5 can build with other compilers"
+
+``any_of`` is a list of specs. One of those specs must be satisfied
+and it is also allowed for the concretized spec to match more than one.
+In the above example, that means you could build ``openmpi@4.1.5%gcc``,
+``openmpi@4.1.5%clang`` or ``openmpi@3.9%gcc``, but
+not ``openmpi@3.9%clang``.
+
+If a custom message is provided, and the requirement is not satisfiable,
+Spack will print the custom error message:
+
+.. code-block:: console
+
+   $ spack spec openmpi@3.9%clang
+   ==> Error: in this example only 4.1.5 can build with other compilers
+
+We could express a similar requirement using the ``when`` attribute:
+
+.. code-block:: yaml
+
+   packages:
+     openmpi:
+       require:
+       - any_of: ["%gcc"]
+         when: "@:4.1.4"
+         message: "in this example only 4.1.5 can build with other compilers"
+
+In the example above, if the version turns out to be 4.1.4 or less, we require the compiler to be GCC.
+For readability, Spack also allows a ``spec`` key accepting a string when there is only a single
+constraint:
+
+.. code-block:: yaml
+
+   packages:
+     openmpi:
+       require:
+       - spec: "%gcc"
+         when: "@:4.1.4"
+         message: "in this example only 4.1.5 can build with other compilers"
+
+This code snippet and the one before it are semantically equivalent.
+
+Finally, instead of ``any_of`` you can use ``one_of`` which also takes a list of specs. The final
+concretized spec must match one and only one of them:
+
+.. code-block:: yaml
+
+   packages:
+     mpich:
+       require:
+       - one_of: ["+cuda", "+rocm"]
+
+In the example above, that means you could build ``mpich+cuda`` or ``mpich+rocm`` but not ``mpich+cuda+rocm``.
+
+.. note::
+
+   For ``any_of`` and ``one_of``, the order of specs indicates a
+   preference: items that appear earlier in the list are preferred
+   (note that these preferences can be ignored in favor of others).
+
+.. note::
+
+   When using a conditional requirement, Spack is allowed to actively avoid the triggering
+   condition (the ``when=...`` spec) if that leads to a concrete spec with better scores in
+   the optimization criteria. To check the current optimization criteria and their
+   priorities you can run ``spack solve zlib``.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Setting default requirements
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can also set default requirements for all packages under ``all``
+like this:
+
+.. code-block:: yaml
+
+   packages:
+     all:
+       require: '%clang'
+
+which means every spec will be required to use ``clang`` as a compiler.
+
+Requirements on variants for all packages are possible too, but note that they
+are only enforced for those packages that define these variants, otherwise they
+are disregarded. For example:
+
+.. code-block:: yaml
+
+   packages:
+     all:
+       require:
+       - "+shared"
+       - "+cuda"
+
+will just enforce ``+shared`` on ``zlib``, which has a boolean ``shared`` variant but
+no ``cuda`` variant.
+
+Constraints in a single spec literal are always considered as a whole, so in a case like:
+
+.. code-block:: yaml
+
+   packages:
+     all:
+       require: "+shared +cuda"
+
+the default requirement will be enforced only if a package has both a ``cuda`` and
+a ``shared`` variant, and will never be partially enforced.
+
+Finally, ``all`` represents a *default set of requirements* -
+if there are specific package requirements, then the default requirements
+under ``all`` are disregarded. For example, with a configuration like this:
+
+.. code-block:: yaml
+
+   packages:
+     all:
+       require:
+       - 'build_type=Debug'
+       - '%clang'
+     cmake:
+       require:
+       - 'build_type=Debug'
+       - '%gcc'
+
+Spack requires ``cmake`` to use ``gcc`` and all other nodes (including ``cmake``
+dependencies) to use ``clang``. If enforcing ``build_type=Debug`` is needed also
+on ``cmake``, it must be repeated in the specific ``cmake`` requirements.
+
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Setting requirements on virtual specs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A requirement on a virtual spec applies whenever that virtual is present in the DAG.
+This can be useful for fixing which virtual provider you want to use:
+
+.. code-block:: yaml
+
+   packages:
+     mpi:
+       require: 'mvapich2 %gcc'
+
+With the configuration above the only allowed ``mpi`` provider is ``mvapich2 %gcc``.
+
+Requirements on the virtual spec and on the specific provider are both applied, if
+present. For instance with a configuration like:
+
+.. code-block:: yaml
+
+   packages:
+     mpi:
+       require: 'mvapich2 %gcc'
+     mvapich2:
+       require: '~cuda'
+
+you will use ``mvapich2~cuda %gcc`` as an ``mpi`` provider.
+
+.. _package-preferences:
+
+-------------------
+Package Preferences
+-------------------
+
+In some cases package requirements can be too strong, and package
+preferences are the better option. Package preferences do not impose
+constraints on packages for particular versions or variants values,
+they rather only set defaults. The concretizer is free to change
+them if it must, due to other constraints, and also prefers reusing
+installed packages over building new ones that are a better match for
+preferences.
+
+.. seealso::
+
+   FAQ: :ref:`Why does Spack pick particular versions and variants? <faq-concretizer-precedence>`
+
+
+Most package preferences (``compilers``, ``target`` and ``providers``)
+can only be set globally under the ``all`` section of ``packages.yaml``:
+
+.. code-block:: yaml
+
+   packages:
+     all:
+       compiler: [gcc@12.2.0, clang@12:, oneapi@2023:]
+       target: [x86_64_v3]
+       providers:
+         mpi: [mvapich2, mpich, openmpi]
+
+These preferences override Spack's default and effectively reorder priorities
+when looking for the best compiler, target or virtual package provider. Each
+preference takes an ordered list of spec constraints, with earlier entries in
+the list being preferred over later entries.
+
+In the example above all packages prefer to be compiled with ``gcc@12.2.0``,
+to target the ``x86_64_v3`` microarchitecture and to use ``mvapich2`` if they
+depend on ``mpi``.
+
+The ``variants`` and ``version`` preferences can be set under
+package specific sections of the ``packages.yaml`` file:
+
+.. code-block:: yaml
+
+   packages:
+     opencv:
+       variants: +debug
+     gperftools:
+       version: [2.2, 2.4, 2.3]
+
+In this case, the preference for ``opencv`` is to build with debug options, while
+``gperftools`` prefers version 2.2 over 2.4.
+
+Any preference can be overwritten on the command line if explicitly requested.
+
+Preferences cannot overcome explicit constraints, as they only set a preferred
+ordering among homogeneous attribute values. Going back to the example, if
+``gperftools@2.3:`` was requested, then Spack will install version 2.4
+since the most preferred version 2.2 is prohibited by the version constraint.
+
+.. _package_permissions:
+
+-------------------
+Package Permissions
+-------------------
+
+Spack can be configured to assign permissions to the files installed
+by a package.
+
+In the ``packages.yaml`` file under ``permissions``, the attributes
+``read``, ``write``, and ``group`` control the package
+permissions. These attributes can be set per-package, or for all
+packages under ``all``. If permissions are set under ``all`` and for a
+specific package, the package-specific settings take precedence.
+
+The ``read`` and ``write`` attributes take one of ``user``, ``group``,
+and ``world``.
+
+.. code-block:: yaml
+
+  packages:
+    all:
+      permissions:
+        write: group
+        group: spack
+    my_app:
+      permissions:
+        read: group
+        group: my_team
+
+The permissions settings describe the broadest level of access to
+installations of the specified packages. The execute permissions of
+the file are set to the same level as read permissions for those files
+that are executable. The default setting for ``read`` is ``world``,
+and for ``write`` is ``user``. In the example above, installations of
+``my_app`` will be installed with user and group permissions but no
+world permissions, and owned by the group ``my_team``. All other
+packages will be installed with user and group write privileges, and
+world read privileges. Those packages will be owned by the group
+``spack``.
+
+The ``group`` attribute assigns a Unix-style group to a package. All
+files installed by the package will be owned by the assigned group,
+and the sticky group bit will be set on the install prefix and all
+directories inside the install prefix. This will ensure that even
+manually placed files within the install prefix are owned by the
+assigned group. If no group is assigned, Spack will allow the OS
+default behavior to go as expected.
+
+----------------------------
+Assigning Package Attributes
+----------------------------
+
+You can assign class-level attributes in the configuration:
+
+.. code-block:: yaml
+
+  packages:
+    mpileaks:
+      # Override existing attributes
+      url: http://www.somewhereelse.com/mpileaks-1.0.tar.gz
+      # ... or add new ones
+      x: 1
+
+Attributes set this way will be accessible to any method executed
+in the package.py file (e.g. the ``install()`` method). Values for these
+attributes may be any value parseable by yaml.
+
+These can only be applied to specific packages, not "all" or
+virtual packages.

lib/spack/docs/packaging_guide.rst

@@ -2337,7 +2337,7 @@ window while a batch job is running ``spack install`` on the same or
 overlapping dependencies without any process trying to re-do the work of
 another.
 
-For example, if you are using SLURM, you could launch an installation
+For example, if you are using Slurm, you could launch an installation
 of ``mpich`` using the following command:
 
 .. code-block:: console

lib/spack/docs/requirements.txt

@@ -1,13 +1,13 @@
 sphinx==7.2.6
 sphinxcontrib-programoutput==0.17
 sphinx_design==0.5.0
-sphinx-rtd-theme==1.3.0
+sphinx-rtd-theme==2.0.0
 python-levenshtein==0.23.0
-docutils==0.18.1
-pygments==2.16.1
-urllib3==2.0.7
+docutils==0.20.1
+pygments==2.17.2
+urllib3==2.1.0
 pytest==7.4.3
 isort==5.12.0
-black==23.10.1
+black==23.11.0
 flake8==6.1.0
-mypy==1.6.1
+mypy==1.7.1

lib/spack/spack/__init__.py

@@ -4,7 +4,7 @@
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
 #: PEP440 canonical <major>.<minor>.<micro>.<devN> string
-__version__ = "0.21.0.dev0"
+__version__ = "0.22.0.dev0"
 spack_version = __version__
 
 

lib/spack/spack/audit.py

@@ -40,6 +40,7 @@ def _search_duplicate_compilers(error_cls):
 import collections.abc
 import glob
 import inspect
+import io
 import itertools
 import pathlib
 import pickle
@@ -54,6 +55,7 @@ def _search_duplicate_compilers(error_cls):
 import spack.repo
 import spack.spec
 import spack.util.crypto
+import spack.util.spack_yaml as syaml
 import spack.variant
 
 #: Map an audit tag to a list of callables implementing checks
@@ -250,6 +252,88 @@ def _search_duplicate_specs_in_externals(error_cls):
     return errors
 
 
+@config_packages
+def _deprecated_preferences(error_cls):
+    """Search package preferences deprecated in v0.21 (and slated for removal in v0.22)"""
+    # TODO (v0.22): remove this audit as the attributes will not be allowed in config
+    errors = []
+    packages_yaml = spack.config.CONFIG.get_config("packages")
+
+    def make_error(attribute_name, config_data, summary):
+        s = io.StringIO()
+        s.write("Occurring in the following file:\n")
+        dict_view = syaml.syaml_dict((k, v) for k, v in config_data.items() if k == attribute_name)
+        syaml.dump_config(dict_view, stream=s, blame=True)
+        return error_cls(summary=summary, details=[s.getvalue()])
+
+    if "all" in packages_yaml and "version" in packages_yaml["all"]:
+        summary = "Using the deprecated 'version' attribute under 'packages:all'"
+        errors.append(make_error("version", packages_yaml["all"], summary))
+
+    for package_name in packages_yaml:
+        if package_name == "all":
+            continue
+
+        package_conf = packages_yaml[package_name]
+        for attribute in ("compiler", "providers", "target"):
+            if attribute not in package_conf:
+                continue
+            summary = (
+                f"Using the deprecated '{attribute}' attribute " f"under 'packages:{package_name}'"
+            )
+            errors.append(make_error(attribute, package_conf, summary))
+
+    return errors
+
+
+@config_packages
+def _avoid_mismatched_variants(error_cls):
+    """Warns if variant preferences have mismatched types or names."""
+    errors = []
+    packages_yaml = spack.config.CONFIG.get_config("packages")
+
+    def make_error(config_data, summary):
+        s = io.StringIO()
+        s.write("Occurring in the following file:\n")
+        syaml.dump_config(config_data, stream=s, blame=True)
+        return error_cls(summary=summary, details=[s.getvalue()])
+
+    for pkg_name in packages_yaml:
+        # 'all:' must be more forgiving, since it is setting defaults for everything
+        if pkg_name == "all" or "variants" not in packages_yaml[pkg_name]:
+            continue
+
+        preferences = packages_yaml[pkg_name]["variants"]
+        if not isinstance(preferences, list):
+            preferences = [preferences]
+
+        for variants in preferences:
+            current_spec = spack.spec.Spec(variants)
+            pkg_cls = spack.repo.PATH.get_pkg_class(pkg_name)
+            for variant in current_spec.variants.values():
+                # Variant does not exist at all
+                if variant.name not in pkg_cls.variants:
+                    summary = (
+                        f"Setting a preference for the '{pkg_name}' package to the "
+                        f"non-existing variant '{variant.name}'"
+                    )
+                    errors.append(make_error(preferences, summary))
+                    continue
+
+                # Variant cannot accept this value
+                s = spack.spec.Spec(pkg_name)
+                try:
+                    s.update_variant_validate(variant.name, variant.value)
+                except Exception:
+                    summary = (
+                        f"Setting the variant '{variant.name}' of the '{pkg_name}' package "
+                        f"to the invalid value '{str(variant)}'"
+                    )
+                    errors.append(make_error(preferences, summary))
+
+    return errors
+
+
 #: Sanity checks on package directives
 package_directives = AuditClass(
     group="packages",
@@ -415,7 +499,7 @@ def _ensure_packages_are_pickeleable(pkgs, error_cls):
 @package_properties
 def _ensure_packages_are_unparseable(pkgs, error_cls):
     """Ensure that all packages can unparse and that unparsed code is valid Python"""
-    import spack.util.package_hash as ph
+    import spack.package_hash as ph
 
     errors = []
     for pkg_name in pkgs:
@@ -562,7 +646,11 @@ def _linting_package_file(pkgs, error_cls):
         if pkg_cls.homepage.startswith("http://"):
             https = re.sub("http", "https", pkg_cls.homepage, 1)
             try:
-                response = urlopen(https)
+                response = urlopen(
+                    https,
+                    verify_ssl=spack.config.get("config:verify_ssl", True),
+                    timeout=spack.config.get("config:connect_timeout", 10),
+                )
             except Exception as e:
                 msg = 'Error with attempting https for "{0}": '
                 errors.append(error_cls(msg.format(pkg_cls.name), [str(e)]))
@@ -776,7 +864,7 @@ def _version_constraints_are_satisfiable_by_some_version_in_repo(pkgs, error_cls
                 )
             except Exception:
                 summary = (
-                    "{0}: dependency on {1} cannot be satisfied " "by known versions of {1.name}"
+                    "{0}: dependency on {1} cannot be satisfied by known versions of {1.name}"
                 ).format(pkg_name, s)
                 details = ["happening in " + filename]
                 if dependency_pkg_cls is not None:
@@ -818,6 +906,53 @@ def _analyze_variants_in_directive(pkg, constraint, directive, error_cls):
     return errors
 
 
+@package_directives
+def _named_specs_in_when_arguments(pkgs, error_cls):
+    """Reports named specs in the 'when=' attribute of a directive.
+
+    Note that 'conflicts' is the only directive allowing that.
+    """
+    errors = []
+    for pkg_name in pkgs:
+        pkg_cls = spack.repo.PATH.get_pkg_class(pkg_name)
+
+        def _extracts_errors(triggers, summary):
+            _errors = []
+            for trigger in list(triggers):
+                when_spec = spack.spec.Spec(trigger)
+                if when_spec.name is not None and when_spec.name != pkg_name:
+                    details = [f"using '{trigger}', should be '^{trigger}'"]
+                    _errors.append(error_cls(summary=summary, details=details))
+            return _errors
+
+        for dname, triggers in pkg_cls.dependencies.items():
+            summary = f"{pkg_name}: wrong 'when=' condition for the '{dname}' dependency"
+            errors.extend(_extracts_errors(triggers, summary))
+
+        for vname, (variant, triggers) in pkg_cls.variants.items():
+            summary = f"{pkg_name}: wrong 'when=' condition for the '{vname}' variant"
+            errors.extend(_extracts_errors(triggers, summary))
+
+        for provided, triggers in pkg_cls.provided.items():
+            summary = f"{pkg_name}: wrong 'when=' condition for the '{provided}' virtual"
+            errors.extend(_extracts_errors(triggers, summary))
+
+        for _, triggers in pkg_cls.requirements.items():
+            triggers = [when_spec for when_spec, _, _ in triggers]
+            summary = f"{pkg_name}: wrong 'when=' condition in 'requires' directive"
+            errors.extend(_extracts_errors(triggers, summary))
+
+        triggers = list(pkg_cls.patches)
+        summary = f"{pkg_name}: wrong 'when=' condition in 'patch' directives"
+        errors.extend(_extracts_errors(triggers, summary))
+
+        triggers = list(pkg_cls.resources)
+        summary = f"{pkg_name}: wrong 'when=' condition in 'resource' directives"
+        errors.extend(_extracts_errors(triggers, summary))
+
+    return llnl.util.lang.dedupe(errors)
+
+
 #: Sanity checks on package directives
 external_detection = AuditClass(
     group="externals",

lib/spack/spack/binary_distribution.py

@@ -38,12 +38,14 @@
 import spack.config as config
 import spack.database as spack_db
 import spack.error
+import spack.gpg
 import spack.hooks
 import spack.hooks.sbang
 import spack.mirror
 import spack.oci.image
 import spack.oci.oci
 import spack.oci.opener
+import spack.paths
 import spack.platforms
 import spack.relocate as relocate
 import spack.repo
@@ -52,7 +54,6 @@
 import spack.traverse as traverse
 import spack.util.crypto
 import spack.util.file_cache as file_cache
-import spack.util.gpg
 import spack.util.path
 import spack.util.spack_json as sjson
 import spack.util.spack_yaml as syaml
@@ -66,8 +67,9 @@
 from spack.stage import Stage
 from spack.util.executable import which
 
-_build_cache_relative_path = "build_cache"
-_build_cache_keys_relative_path = "_pgp"
+BUILD_CACHE_RELATIVE_PATH = "build_cache"
+BUILD_CACHE_KEYS_RELATIVE_PATH = "_pgp"
+CURRENT_BUILD_CACHE_LAYOUT_VERSION = 1
 
 
 class BuildCacheDatabase(spack_db.Database):
@@ -229,7 +231,11 @@ def _associate_built_specs_with_mirror(self, cache_key, mirror_url):
                 )
                 return
 
-            spec_list = db.query_local(installed=False, in_buildcache=True)
+            spec_list = [
+                s
+                for s in db.query_local(installed=any, in_buildcache=any)
+                if s.external or db.query_local_by_spec_hash(s.dag_hash()).in_buildcache
+            ]
 
             for indexed_spec in spec_list:
                 dag_hash = indexed_spec.dag_hash()
@@ -481,7 +487,10 @@ def _fetch_and_cache_index(self, mirror_url, cache_entry={}):
         scheme = urllib.parse.urlparse(mirror_url).scheme
 
         if scheme != "oci" and not web_util.url_exists(
-            url_util.join(mirror_url, _build_cache_relative_path, "index.json")
+            url_util.join(mirror_url, BUILD_CACHE_RELATIVE_PATH, "index.json"),
+            fetch_method=spack.config.get("config:url_fetch_method", "urllib"),
+            verify_ssl=spack.config.get("config:verify_ssl"),
+            timeout=spack.config.get("config:connect_timeout", 10),
         ):
             return False
 
@@ -527,7 +536,9 @@ def _fetch_and_cache_index(self, mirror_url, cache_entry={}):
 def binary_index_location():
     """Set up a BinaryCacheIndex for remote buildcache dbs in the user's homedir."""
     cache_root = os.path.join(misc_cache_location(), "indices")
-    return spack.util.path.canonicalize_path(cache_root)
+    return spack.util.path.canonicalize_path(
+        cache_root, replacements=spack.paths.path_replacements()
+    )
 
 
 #: Default binary cache index instance
@@ -600,6 +611,10 @@ def __init__(self, msg):
         super().__init__(msg)
 
 
+class InvalidMetadataFile(spack.error.SpackError):
+    pass
+
+
 class UnsignedPackageException(spack.error.SpackError):
     """
     Raised if installation of unsigned package is attempted without
@@ -614,11 +629,11 @@ def compute_hash(data):
 
 
 def build_cache_relative_path():
-    return _build_cache_relative_path
+    return BUILD_CACHE_RELATIVE_PATH
 
 
 def build_cache_keys_relative_path():
-    return _build_cache_keys_relative_path
+    return BUILD_CACHE_KEYS_RELATIVE_PATH
 
 
 def build_cache_prefix(prefix):
@@ -818,7 +833,7 @@ def tarball_path_name(spec, ext):
 
 def select_signing_key(key=None):
     if key is None:
-        keys = spack.util.gpg.signing_keys()
+        keys = spack.gpg.signing_keys()
         if len(keys) == 1:
             key = keys[0]
 
@@ -843,7 +858,7 @@ def sign_specfile(key, force, specfile_path):
             raise NoOverwriteException(signed_specfile_path)
 
     key = select_signing_key(key)
-    spack.util.gpg.sign(key, specfile_path, signed_specfile_path, clearsign=True)
+    spack.gpg.sign(key, specfile_path, signed_specfile_path, clearsign=True)
 
 
 def _read_specs_and_push_index(file_list, read_method, cache_prefix, db, temp_dir, concurrency):
@@ -895,6 +910,7 @@ def _read_specs_and_push_index(file_list, read_method, cache_prefix, db, temp_di
         url_util.join(cache_prefix, "index.json"),
         keep_original=False,
         extra_args={"ContentType": "application/json", "CacheControl": "no-cache"},
+        verify_ssl=spack.config.get("config:verify_ssl", True),
     )
 
     # Push the hash
@@ -903,6 +919,7 @@ def _read_specs_and_push_index(file_list, read_method, cache_prefix, db, temp_di
         url_util.join(cache_prefix, "index.json.hash"),
         keep_original=False,
         extra_args={"ContentType": "text/plain", "CacheControl": "no-cache"},
+        verify_ssl=spack.config.get("config:verify_ssl", True),
     )
 
 
@@ -967,9 +984,13 @@ def _specs_from_cache_fallback(cache_prefix):
     def url_read_method(url):
         contents = None
         try:
-            _, _, spec_file = web_util.read_from_url(url)
+            _, _, spec_file = web_util.read_from_url(
+                url,
+                verify_ssl=spack.config.get("config:verify_ssl", True),
+                timeout=spack.config.get("config:connect_timeout", 10),
+            )
             contents = codecs.getreader("utf-8")(spec_file).read()
-        except (URLError, web_util.SpackWebError) as url_err:
+        except (URLError, web_util.WebError) as url_err:
             tty.error("Error reading specfile: {0}".format(url))
             tty.error(url_err)
         return contents
@@ -977,7 +998,9 @@ def url_read_method(url):
     try:
         file_list = [
             url_util.join(cache_prefix, entry)
-            for entry in web_util.list_url(cache_prefix)
+            for entry in web_util.list_url(
+                cache_prefix, verify_ssl=spack.config.get("config:verify_ssl", True)
+            )
             if entry.endswith("spec.json") or entry.endswith("spec.json.sig")
         ]
         read_fn = url_read_method
@@ -1075,7 +1098,9 @@ def generate_key_index(key_prefix, tmpdir=None):
     try:
         fingerprints = (
             entry[:-4]
-            for entry in web_util.list_url(key_prefix, recursive=False)
+            for entry in web_util.list_url(
+                key_prefix, recursive=False, verify_ssl=spack.config.get("config:verify_ssl", True)
+            )
             if entry.endswith(".pub")
         )
     except KeyError as inst:
@@ -1112,6 +1137,7 @@ def generate_key_index(key_prefix, tmpdir=None):
                 url_util.join(key_prefix, "index.json"),
                 keep_original=False,
                 extra_args={"ContentType": "application/json"},
+                verify_ssl=spack.config.get("config:verify_ssl", True),
             )
         except Exception as err:
             msg = "Encountered problem pushing key index to {0}: {1}".format(key_prefix, err)
@@ -1355,10 +1381,18 @@ def _build_tarball_in_stage_dir(spec: Spec, out_url: str, stage_dir: str, option
     spackfile_path = os.path.join(cache_prefix, tarball_path_name(spec, ".spack"))
     remote_spackfile_path = url_util.join(out_url, os.path.relpath(spackfile_path, stage_dir))
 
+    fetch_method = (spack.config.get("config:url_fetch_method", "urllib"),)
+    verify_ssl = (spack.config.get("config:verify_ssl"),)
+    timeout = spack.config.get("config:connect_timeout", 10)
+
+    url_args = {"fetch_method": fetch_method, "verify_ssl": verify_ssl, "timeout": timeout}
+
     mkdirp(tarfile_dir)
-    if web_util.url_exists(remote_spackfile_path):
+    if web_util.url_exists(remote_spackfile_path, **url_args):
         if options.force:
-            web_util.remove_url(remote_spackfile_path)
+            web_util.remove_url(
+                remote_spackfile_path, verify_ssl=spack.config.get("config:verify_ssl", True)
+            )
         else:
             raise NoOverwriteException(url_util.format(remote_spackfile_path))
 
@@ -1378,12 +1412,13 @@ def _build_tarball_in_stage_dir(spec: Spec, out_url: str, stage_dir: str, option
 
     # If force and exists, overwrite. Otherwise raise exception on collision.
     if options.force:
-        if web_util.url_exists(remote_specfile_path):
-            web_util.remove_url(remote_specfile_path)
-        if web_util.url_exists(remote_signed_specfile_path):
-            web_util.remove_url(remote_signed_specfile_path)
-    elif web_util.url_exists(remote_specfile_path) or web_util.url_exists(
-        remote_signed_specfile_path
+        verify_ssl = spack.config.get("config:verify_ssl", True)
+        if web_util.url_exists(remote_specfile_path, **url_args):
+            web_util.remove_url(remote_specfile_path, verify_ssl=verify_ssl)
+        if web_util.url_exists(remote_signed_specfile_path, **url_args):
+            web_util.remove_url(remote_signed_specfile_path, verify_ssl=verify_ssl)
+    elif web_util.url_exists(remote_specfile_path, **url_args) or web_util.url_exists(
+        remote_signed_specfile_path, **url_args
     ):
         raise NoOverwriteException(url_util.format(remote_specfile_path))
 
@@ -1401,7 +1436,7 @@ def _build_tarball_in_stage_dir(spec: Spec, out_url: str, stage_dir: str, option
             spec_dict = sjson.load(content)
         else:
             raise ValueError("{0} not a valid spec file type".format(spec_file))
-    spec_dict["buildcache_layout_version"] = 1
+    spec_dict["buildcache_layout_version"] = CURRENT_BUILD_CACHE_LAYOUT_VERSION
     spec_dict["binary_cache_checksum"] = {"hash_algorithm": "sha256", "hash": checksum}
 
     with open(specfile_path, "w") as outfile:
@@ -1417,11 +1452,17 @@ def _build_tarball_in_stage_dir(spec: Spec, out_url: str, stage_dir: str, option
         sign_specfile(key, options.force, specfile_path)
 
     # push tarball and signed spec json to remote mirror
-    web_util.push_to_url(spackfile_path, remote_spackfile_path, keep_original=False)
+    web_util.push_to_url(
+        spackfile_path,
+        remote_spackfile_path,
+        keep_original=False,
+        verify_ssl=spack.config.get("config:verify_ssl", True),
+    )
     web_util.push_to_url(
         signed_specfile_path if not options.unsigned else specfile_path,
         remote_signed_specfile_path if not options.unsigned else remote_specfile_path,
         keep_original=False,
+        verify_ssl=spack.config.get("config:verify_ssl", True),
     )
 
     # push the key to the build cache's _pgp directory so it can be
@@ -1525,7 +1566,7 @@ def try_verify(specfile_path):
     suppress = config.get("config:suppress_gpg_warnings", False)
 
     try:
-        spack.util.gpg.verify(specfile_path, suppress_warnings=suppress)
+        spack.gpg.verify(specfile_path, suppress_warnings=suppress)
     except Exception:
         return False
 
@@ -1560,6 +1601,42 @@ def _delete_staged_downloads(download_result):
     download_result["specfile_stage"].destroy()
 
 
+def _get_valid_spec_file(path: str, max_supported_layout: int) -> Tuple[Dict, int]:
+    """Read and validate a spec file, returning the spec dict with its layout version, or raising
+    InvalidMetadataFile if invalid."""
+    try:
+        with open(path, "rb") as f:
+            binary_content = f.read()
+    except OSError:
+        raise InvalidMetadataFile(f"No such file: {path}")
+
+    # In the future we may support transparently decompressing compressed spec files.
+    if binary_content[:2] == b"\x1f\x8b":
+        raise InvalidMetadataFile("Compressed spec files are not supported")
+
+    try:
+        as_string = binary_content.decode("utf-8")
+        if path.endswith(".json.sig"):
+            spec_dict = Spec.extract_json_from_clearsig(as_string)
+        else:
+            spec_dict = json.loads(as_string)
+    except Exception as e:
+        raise InvalidMetadataFile(f"Could not parse {path} due to: {e}") from e
+
+    # Ensure this version is not too new.
+    try:
+        layout_version = int(spec_dict.get("buildcache_layout_version", 0))
+    except ValueError as e:
+        raise InvalidMetadataFile("Could not parse layout version") from e
+
+    if layout_version > max_supported_layout:
+        raise InvalidMetadataFile(
+            f"Layout version {layout_version} is too new for this version of Spack"
+        )
+
+    return spec_dict, layout_version
+
+
 def download_tarball(spec, unsigned=False, mirrors_for_spec=None):
     """
     Download binary tarball for given package into stage area, returning
@@ -1652,6 +1729,18 @@ def download_tarball(spec, unsigned=False, mirrors_for_spec=None):
                     try:
                         local_specfile_stage.fetch()
                         local_specfile_stage.check()
+                        try:
+                            _get_valid_spec_file(
+                                local_specfile_stage.save_filename,
+                                CURRENT_BUILD_CACHE_LAYOUT_VERSION,
+                            )
+                        except InvalidMetadataFile as e:
+                            tty.warn(
+                                f"Ignoring binary package for {spec.name}/{spec.dag_hash()[:7]} "
+                                f"from {mirror} due to invalid metadata file: {e}"
+                            )
+                            local_specfile_stage.destroy()
+                            continue
                     except Exception:
                         continue
                     local_specfile_stage.cache_local()
@@ -1674,14 +1763,26 @@ def download_tarball(spec, unsigned=False, mirrors_for_spec=None):
 
             else:
                 ext = "json.sig" if try_signed else "json"
-                specfile_path = url_util.join(mirror, _build_cache_relative_path, specfile_prefix)
+                specfile_path = url_util.join(mirror, BUILD_CACHE_RELATIVE_PATH, specfile_prefix)
                 specfile_url = f"{specfile_path}.{ext}"
-                spackfile_url = url_util.join(mirror, _build_cache_relative_path, tarball)
+                spackfile_url = url_util.join(mirror, BUILD_CACHE_RELATIVE_PATH, tarball)
                 local_specfile_stage = try_fetch(specfile_url)
                 if local_specfile_stage:
                     local_specfile_path = local_specfile_stage.save_filename
                     signature_verified = False
 
+                    try:
+                        _get_valid_spec_file(
+                            local_specfile_path, CURRENT_BUILD_CACHE_LAYOUT_VERSION
+                        )
+                    except InvalidMetadataFile as e:
+                        tty.warn(
+                            f"Ignoring binary package for {spec.name}/{spec.dag_hash()[:7]} "
+                            f"from {mirror} due to invalid metadata file: {e}"
+                        )
+                        local_specfile_stage.destroy()
+                        continue
+
                     if try_signed and not unsigned:
                         # If we found a signed specfile at the root, try to verify
                         # the signature immediately.  We will not download the
@@ -1936,7 +2037,7 @@ def _extract_inner_tarball(spec, filename, extract_to, unsigned, remote_checksum
         if os.path.exists("%s.asc" % specfile_path):
             suppress = config.get("config:suppress_gpg_warnings", False)
             try:
-                spack.util.gpg.verify("%s.asc" % specfile_path, specfile_path, suppress)
+                spack.gpg.verify("%s.asc" % specfile_path, specfile_path, suppress)
             except Exception:
                 raise NoVerifyException(
                     "Spack was unable to verify package "
@@ -2001,24 +2102,16 @@ def extract_tarball(spec, download_result, unsigned=False, force=False, timer=ti
     )
 
     specfile_path = download_result["specfile_stage"].save_filename
-
-    with open(specfile_path, "r") as inputfile:
-        content = inputfile.read()
-        if specfile_path.endswith(".json.sig"):
-            spec_dict = Spec.extract_json_from_clearsig(content)
-        else:
-            spec_dict = sjson.load(content)
-
+    spec_dict, layout_version = _get_valid_spec_file(
+        specfile_path, CURRENT_BUILD_CACHE_LAYOUT_VERSION
+    )
     bchecksum = spec_dict["binary_cache_checksum"]
 
     filename = download_result["tarball_stage"].save_filename
     signature_verified = download_result["signature_verified"]
     tmpdir = None
 
-    if (
-        "buildcache_layout_version" not in spec_dict
-        or int(spec_dict["buildcache_layout_version"]) < 1
-    ):
+    if layout_version == 0:
         # Handle the older buildcache layout where the .spack file
         # contains a spec json, maybe an .asc file (signature),
         # and another tarball containing the actual install tree.
@@ -2029,7 +2122,7 @@ def extract_tarball(spec, download_result, unsigned=False, force=False, timer=ti
             _delete_staged_downloads(download_result)
             shutil.rmtree(tmpdir)
             raise e
-    else:
+    elif layout_version == 1:
         # Newer buildcache layout: the .spack file contains just
         # in the install tree, the signature, if it exists, is
         # wrapped around the spec.json at the root.  If sig verify
@@ -2053,7 +2146,6 @@ def extract_tarball(spec, download_result, unsigned=False, force=False, timer=ti
             raise NoChecksumException(
                 tarfile_path, size, contents, "sha256", expected, local_checksum
             )
-
     try:
         with closing(tarfile.open(tarfile_path, "r")) as tar:
             # Remove install prefix from tarfil to extract directly into spec.prefix
@@ -2151,7 +2243,8 @@ def install_root_node(spec, unsigned=False, force=False, sha256=None):
         tty.debug("Verified SHA256 checksum of the build cache")
 
     # don't print long padded paths while extracting/relocating binaries
-    with spack.util.path.filter_padding():
+    padding = spack.config.get("config:install_tree:padded_length", None)
+    with spack.util.path.filter_padding(padding=padding):
         tty.msg('Installing "{0}" from a buildcache'.format(spec.format()))
         extract_tarball(spec, download_result, unsigned, force)
         spack.hooks.post_install(spec, False)
@@ -2184,18 +2277,26 @@ def try_direct_fetch(spec, mirrors=None):
 
     for mirror in binary_mirrors:
         buildcache_fetch_url_json = url_util.join(
-            mirror.fetch_url, _build_cache_relative_path, specfile_name
+            mirror.fetch_url, BUILD_CACHE_RELATIVE_PATH, specfile_name
         )
         buildcache_fetch_url_signed_json = url_util.join(
-            mirror.fetch_url, _build_cache_relative_path, signed_specfile_name
+            mirror.fetch_url, BUILD_CACHE_RELATIVE_PATH, signed_specfile_name
         )
         try:
-            _, _, fs = web_util.read_from_url(buildcache_fetch_url_signed_json)
+            _, _, fs = web_util.read_from_url(
+                buildcache_fetch_url_signed_json,
+                verify_ssl=spack.config.get("config:verify_ssl", True),
+                timeout=spack.config.get("config:connect_timeout", 10),
+            )
             specfile_is_signed = True
-        except (URLError, web_util.SpackWebError, HTTPError) as url_err:
+        except (URLError, web_util.WebError, HTTPError) as url_err:
             try:
-                _, _, fs = web_util.read_from_url(buildcache_fetch_url_json)
-            except (URLError, web_util.SpackWebError, HTTPError) as url_err_x:
+                _, _, fs = web_util.read_from_url(
+                    buildcache_fetch_url_json,
+                    verify_ssl=spack.config.get("config:verify_ssl", True),
+                    timeout=spack.config.get("config:connect_timeout", 10),
+                )
+            except (URLError, web_util.WebError, HTTPError) as url_err_x:
                 tty.debug(
                     "Did not find {0} on {1}".format(
                         specfile_name, buildcache_fetch_url_signed_json
@@ -2292,17 +2393,26 @@ def get_keys(install=False, trust=False, force=False, mirrors=None):
     for mirror in mirror_collection.values():
         fetch_url = mirror.fetch_url
         keys_url = url_util.join(
-            fetch_url, _build_cache_relative_path, _build_cache_keys_relative_path
+            fetch_url, BUILD_CACHE_RELATIVE_PATH, BUILD_CACHE_KEYS_RELATIVE_PATH
         )
         keys_index = url_util.join(keys_url, "index.json")
 
         tty.debug("Finding public keys in {0}".format(url_util.format(fetch_url)))
 
         try:
-            _, _, json_file = web_util.read_from_url(keys_index)
+            _, _, json_file = web_util.read_from_url(
+                keys_index,
+                verify_ssl=spack.config.get("config:verify_ssl", True),
+                timeout=spack.config.get("config:connect_timeout", 10),
+            )
             json_index = sjson.load(codecs.getreader("utf-8")(json_file))
-        except (URLError, web_util.SpackWebError) as url_err:
-            if web_util.url_exists(keys_index):
+        except (URLError, web_util.WebError) as url_err:
+            if web_util.url_exists(
+                keys_index,
+                fetch_method=spack.config.get("config:url_fetch_method", "urllib"),
+                verify_ssl=spack.config.get("config:verify_ssl"),
+                timeout=spack.config.get("config:connect_timeout", 10),
+            ):
                 err_msg = [
                     "Unable to find public keys in {0},",
                     " caught exception attempting to read from {1}.",
@@ -2333,7 +2443,7 @@ def get_keys(install=False, trust=False, force=False, mirrors=None):
             tty.debug("Found key {0}".format(fingerprint))
             if install:
                 if trust:
-                    spack.util.gpg.trust(stage.save_filename)
+                    spack.gpg.trust(stage.save_filename)
                     tty.debug("Added this key to trusted keys.")
                 else:
                     tty.debug(
@@ -2351,13 +2461,13 @@ def push_keys(*mirrors, **kwargs):
     tmpdir = kwargs.get("tmpdir")
     remove_tmpdir = False
 
-    keys = spack.util.gpg.public_keys(*(keys or []))
+    keys = spack.gpg.public_keys(*(keys or []))
 
     try:
         for mirror in mirrors:
             push_url = getattr(mirror, "push_url", mirror)
             keys_url = url_util.join(
-                push_url, _build_cache_relative_path, _build_cache_keys_relative_path
+                push_url, BUILD_CACHE_RELATIVE_PATH, BUILD_CACHE_KEYS_RELATIVE_PATH
             )
             keys_local = url_util.local_file_path(keys_url)
 
@@ -2383,7 +2493,7 @@ def push_keys(*mirrors, **kwargs):
                 export_target = os.path.join(prefix, filename)
 
                 # Export public keys (private is set to False)
-                spack.util.gpg.export_keys(export_target, [fingerprint])
+                spack.gpg.export_keys(export_target, [fingerprint])
 
                 # If mirror is local, the above export writes directly to the
                 # mirror (export_target points directly to the mirror).
@@ -2392,7 +2502,10 @@ def push_keys(*mirrors, **kwargs):
                 # uploaded to the mirror.
                 if not keys_local:
                     spack.util.web.push_to_url(
-                        export_target, url_util.join(keys_url, filename), keep_original=False
+                        export_target,
+                        url_util.join(keys_url, filename),
+                        keep_original=False,
+                        verify_ssl=spack.config.get("config:verify_ssl", True),
                     )
 
             if regenerate_index:
@@ -2426,7 +2539,12 @@ def needs_rebuild(spec, mirror_url):
     # Only check for the presence of the json version of the spec.  If the
     # mirror only has the json version, or doesn't have the spec at all, we
     # need to rebuild.
-    return not web_util.url_exists(specfile_path)
+    return not web_util.url_exists(
+        specfile_path,
+        fetch_method=spack.config.get("config:url_fetch_method", "urllib"),
+        verify_ssl=spack.config.get("config:verify_ssl"),
+        timeout=spack.config.get("config:connect_timeout", 10),
+    )
 
 
 def check_specs_against_mirrors(mirrors, specs, output_file=None):
@@ -2495,11 +2613,11 @@ def download_buildcache_entry(file_descriptions, mirror_url=None):
         )
 
     if mirror_url:
-        mirror_root = os.path.join(mirror_url, _build_cache_relative_path)
+        mirror_root = os.path.join(mirror_url, BUILD_CACHE_RELATIVE_PATH)
         return _download_buildcache_entry(mirror_root, file_descriptions)
 
     for mirror in spack.mirror.MirrorCollection(binary=True).values():
-        mirror_root = os.path.join(mirror.fetch_url, _build_cache_relative_path)
+        mirror_root = os.path.join(mirror.fetch_url, BUILD_CACHE_RELATIVE_PATH)
 
         if _download_buildcache_entry(mirror_root, file_descriptions):
             return True
@@ -2590,9 +2708,13 @@ def __init__(self, url, local_hash, urlopen=web_util.urlopen):
 
     def get_remote_hash(self):
         # Failure to fetch index.json.hash is not fatal
-        url_index_hash = url_util.join(self.url, _build_cache_relative_path, "index.json.hash")
+        url_index_hash = url_util.join(self.url, BUILD_CACHE_RELATIVE_PATH, "index.json.hash")
         try:
-            response = self.urlopen(urllib.request.Request(url_index_hash, headers=self.headers))
+            response = self.urlopen(
+                urllib.request.Request(url_index_hash, headers=self.headers),
+                verify_ssl=spack.config.get("config:verify_ssl", True),
+                timeout=spack.config.get("config:connect_timeout", 10),
+            )
         except urllib.error.URLError:
             return None
 
@@ -2611,10 +2733,14 @@ def conditional_fetch(self) -> FetchIndexResult:
             return FetchIndexResult(etag=None, hash=None, data=None, fresh=True)
 
         # Otherwise, download index.json
-        url_index = url_util.join(self.url, _build_cache_relative_path, "index.json")
+        url_index = url_util.join(self.url, BUILD_CACHE_RELATIVE_PATH, "index.json")
 
         try:
-            response = self.urlopen(urllib.request.Request(url_index, headers=self.headers))
+            response = self.urlopen(
+                urllib.request.Request(url_index, headers=self.headers),
+                verify_ssl=spack.config.get("config:verify_ssl", True),
+                timeout=spack.config.get("config:connect_timeout", 10),
+            )
         except urllib.error.URLError as e:
             raise FetchIndexError("Could not fetch index from {}".format(url_index), e) from e
 
@@ -2655,14 +2781,18 @@ def __init__(self, url, etag, urlopen=web_util.urlopen):
 
     def conditional_fetch(self) -> FetchIndexResult:
         # Just do a conditional fetch immediately
-        url = url_util.join(self.url, _build_cache_relative_path, "index.json")
+        url = url_util.join(self.url, BUILD_CACHE_RELATIVE_PATH, "index.json")
         headers = {
             "User-Agent": web_util.SPACK_USER_AGENT,
             "If-None-Match": '"{}"'.format(self.etag),
         }
 
         try:
-            response = self.urlopen(urllib.request.Request(url, headers=headers))
+            response = self.urlopen(
+                urllib.request.Request(url, headers=headers),
+                verify_ssl=spack.config.get("config:verify_ssl", True),
+                timeout=spack.config.get("config:connect_timeout", 10),
+            )
         except urllib.error.HTTPError as e:
             if e.getcode() == 304:
                 # Not modified; that means fresh.

lib/spack/spack/bootstrap/_common.py

@@ -213,7 +213,8 @@ def _root_spec(spec_str: str) -> str:
     if str(spack.platforms.host()) == "darwin":
         spec_str += " %apple-clang"
     elif str(spack.platforms.host()) == "windows":
-        spec_str += " %msvc"
+        # TODO (johnwparent): Remove version constraint when clingo patch is up
+        spec_str += " %msvc@:19.37"
     else:
         spec_str += " %gcc"
 

lib/spack/spack/bootstrap/config.py

@@ -45,7 +45,8 @@ def spec_for_current_python() -> str:
 def root_path() -> str:
     """Root of all the bootstrap related folders"""
     return spack.util.path.canonicalize_path(
-        spack.config.get("bootstrap:root", spack.paths.default_user_bootstrap_path)
+        spack.config.get("bootstrap:root", spack.paths.default_user_bootstrap_path),
+        replacements=spack.paths.path_replacements(),
     )
 
 
@@ -79,12 +80,16 @@ def spack_python_interpreter() -> Generator:
 
 def _store_path() -> str:
     bootstrap_root_path = root_path()
-    return spack.util.path.canonicalize_path(os.path.join(bootstrap_root_path, "store"))
+    return spack.util.path.canonicalize_path(
+        os.path.join(bootstrap_root_path, "store"), replacements=spack.paths.path_replacements()
+    )
 
 
 def _config_path() -> str:
     bootstrap_root_path = root_path()
-    return spack.util.path.canonicalize_path(os.path.join(bootstrap_root_path, "config"))
+    return spack.util.path.canonicalize_path(
+        os.path.join(bootstrap_root_path, "config"), replacements=spack.paths.path_replacements()
+    )
 
 
 @contextlib.contextmanager

lib/spack/spack/bootstrap/core.py

@@ -92,7 +92,9 @@ class Bootstrapper:
     def __init__(self, conf: ConfigDictionary) -> None:
         self.conf = conf
         self.name = conf["name"]
-        self.metadata_dir = spack.util.path.canonicalize_path(conf["metadata"])
+        self.metadata_dir = spack.util.path.canonicalize_path(
+            conf["metadata"], replacements=spack.paths.path_replacements()
+        )
 
         # Promote (relative) paths to file urls
         url = conf["info"]["url"]
@@ -585,7 +587,9 @@ def bootstrapping_sources(scope: Optional[str] = None):
     list_of_sources = []
     for entry in source_configs:
         current = copy.copy(entry)
-        metadata_dir = spack.util.path.canonicalize_path(entry["metadata"])
+        metadata_dir = spack.util.path.canonicalize_path(
+            entry["metadata"], replacements=spack.paths.path_replacements()
+        )
         metadata_yaml = os.path.join(metadata_dir, METADATA_YAML_FILENAME)
         with open(metadata_yaml, encoding="utf-8") as stream:
             current.update(spack.util.spack_yaml.load(stream))

lib/spack/spack/bootstrap/environment.py

@@ -16,6 +16,7 @@
 from llnl.util import tty
 
 import spack.environment
+import spack.paths
 import spack.tengine
 import spack.util.cpus
 import spack.util.executable
@@ -50,7 +51,8 @@ def environment_root(cls) -> pathlib.Path:
         environment_dir = f"{python_part}-{arch_part}-{interpreter_part}"
         return pathlib.Path(
             spack.util.path.canonicalize_path(
-                os.path.join(bootstrap_root_path, "environments", environment_dir)
+                os.path.join(bootstrap_root_path, "environments", environment_dir),
+                replacements=spack.paths.path_replacements(),
             )
         )
 
@@ -137,7 +139,9 @@ def _install_with_depfile(self) -> None:
             "-C",
             str(self.environment_root()),
             "-j",
-            str(spack.util.cpus.determine_number_of_jobs(parallel=True)),
+            str(
+                spack.util.cpus.determine_number_of_jobs(parallel=True, config=spack.config.CONFIG)
+            ),
             **kwargs,
         )
 
@@ -161,7 +165,7 @@ def _write_spack_yaml_file(self) -> None:
 
 def isort_root_spec() -> str:
     """Return the root spec used to bootstrap isort"""
-    return _root_spec("py-isort@4.3.5:")
+    return _root_spec("py-isort@5")
 
 
 def mypy_root_spec() -> str:

lib/spack/spack/build_environment.py

@@ -324,19 +324,29 @@ def set_compiler_environment_variables(pkg, env):
     # ttyout, ttyerr, etc.
     link_dir = spack.paths.build_env_path
 
-    # Set SPACK compiler variables so that our wrapper knows what to call
+    # Set SPACK compiler variables so that our wrapper knows what to
+    # call.  If there is no compiler configured then use a default
+    # wrapper which will emit an error if it is used.
     if compiler.cc:
         env.set("SPACK_CC", compiler.cc)
         env.set("CC", os.path.join(link_dir, compiler.link_paths["cc"]))
+    else:
+        env.set("CC", os.path.join(link_dir, "cc"))
     if compiler.cxx:
         env.set("SPACK_CXX", compiler.cxx)
         env.set("CXX", os.path.join(link_dir, compiler.link_paths["cxx"]))
+    else:
+        env.set("CC", os.path.join(link_dir, "c++"))
     if compiler.f77:
         env.set("SPACK_F77", compiler.f77)
         env.set("F77", os.path.join(link_dir, compiler.link_paths["f77"]))
+    else:
+        env.set("F77", os.path.join(link_dir, "f77"))
     if compiler.fc:
         env.set("SPACK_FC", compiler.fc)
         env.set("FC", os.path.join(link_dir, compiler.link_paths["fc"]))
+    else:
+        env.set("FC", os.path.join(link_dir, "fc"))
 
     # Set SPACK compiler rpath flags so that our wrapper knows what to use
     env.set("SPACK_CC_RPATH_ARG", compiler.cc_rpath_arg)
@@ -743,15 +753,16 @@ def setup_package(pkg, dirty, context: Context = Context.BUILD):
         set_compiler_environment_variables(pkg, env_mods)
         set_wrapper_variables(pkg, env_mods)
 
-    tty.debug("setup_package: grabbing modifications from dependencies")
-    env_mods.extend(setup_context.get_env_modifications())
-    tty.debug("setup_package: collected all modifications from dependencies")
-
-    # architecture specific setup
+    # Platform specific setup goes before package specific setup. This is for setting
+    # defaults like MACOSX_DEPLOYMENT_TARGET on macOS.
     platform = spack.platforms.by_name(pkg.spec.architecture.platform)
     target = platform.target(pkg.spec.architecture.target)
     platform.setup_platform_environment(pkg, env_mods)
 
+    tty.debug("setup_package: grabbing modifications from dependencies")
+    env_mods.extend(setup_context.get_env_modifications())
+    tty.debug("setup_package: collected all modifications from dependencies")
+
     if context == Context.TEST:
         env_mods.prepend_path("PATH", ".")
     elif context == Context.BUILD and not dirty and not env_mods.is_unset("CPATH"):

lib/spack/spack/build_systems/cached_cmake.py

@@ -34,6 +34,11 @@ def cmake_cache_option(name, boolean_value, comment="", force=False):
     return 'set({0} {1} CACHE BOOL "{2}"{3})\n'.format(name, value, comment, force_str)
 
 
+def cmake_cache_filepath(name, value, comment=""):
+    """Generate a string for a cmake cache variable of type FILEPATH"""
+    return 'set({0} "{1}" CACHE FILEPATH "{2}")\n'.format(name, value, comment)
+
+
 class CachedCMakeBuilder(CMakeBuilder):
     #: Phases of a Cached CMake package
     #: Note: the initconfig phase is used for developer builds as a final phase to stop on
@@ -257,6 +262,15 @@ def initconfig_hardware_entries(self):
             entries.append(
                 cmake_cache_path("HIP_CXX_COMPILER", "{0}".format(self.spec["hip"].hipcc))
             )
+            llvm_bin = spec["llvm-amdgpu"].prefix.bin
+            llvm_prefix = spec["llvm-amdgpu"].prefix
+            # Some ROCm systems seem to point to /<path>/rocm-<ver>/ and
+            # others point to /<path>/rocm-<ver>/llvm
+            if os.path.basename(os.path.normpath(llvm_prefix)) != "llvm":
+                llvm_bin = os.path.join(llvm_prefix, "llvm/bin/")
+            entries.append(
+                cmake_cache_filepath("CMAKE_HIP_COMPILER", os.path.join(llvm_bin, "clang++"))
+            )
             archs = self.spec.variants["amdgpu_target"].value
             if archs[0] != "none":
                 arch_str = ";".join(archs)
@@ -277,7 +291,7 @@ def std_initconfig_entries(self):
             "#------------------{0}".format("-" * 60),
             "# CMake executable path: {0}".format(self.pkg.spec["cmake"].command.path),
             "#------------------{0}\n".format("-" * 60),
-            cmake_cache_path("CMAKE_PREFIX_PATH", cmake_prefix_path),
+            cmake_cache_string("CMAKE_PREFIX_PATH", cmake_prefix_path),
             self.define_cmake_cache_from_variant("CMAKE_BUILD_TYPE", "build_type"),
         ]
 

lib/spack/spack/build_systems/cargo.py

@@ -0,0 +1,89 @@
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import inspect
+
+import llnl.util.filesystem as fs
+
+import spack.builder
+import spack.package_base
+from spack.directives import build_system, depends_on
+from spack.multimethod import when
+
+from ._checks import BaseBuilder, execute_install_time_tests
+
+
+class CargoPackage(spack.package_base.PackageBase):
+    """Specialized class for packages built using a Makefiles."""
+
+    #: This attribute is used in UI queries that need to know the build
+    #: system base class
+    build_system_class = "CargoPackage"
+
+    build_system("cargo")
+
+    with when("build_system=cargo"):
+        depends_on("rust", type="build")
+
+
+@spack.builder.builder("cargo")
+class CargoBuilder(BaseBuilder):
+    """The Cargo builder encodes the most common way of building software with
+    a rust Cargo.toml file. It has two phases that can be overridden, if need be:
+
+            1. :py:meth:`~.CargoBuilder.build`
+            2. :py:meth:`~.CargoBuilder.install`
+
+    For a finer tuning you may override:
+
+        +-----------------------------------------------+----------------------+
+        | **Method**                                    | **Purpose**          |
+        +===============================================+======================+
+        | :py:meth:`~.CargoBuilder.build_args`          | Specify arguments    |
+        |                                               | to ``cargo install`` |
+        +-----------------------------------------------+----------------------+
+        | :py:meth:`~.CargoBuilder.check_args`          | Specify arguments    |
+        |                                               | to ``cargo test``    |
+        +-----------------------------------------------+----------------------+
+    """
+
+    phases = ("build", "install")
+
+    #: Callback names for install-time test
+    install_time_test_callbacks = ["check"]
+
+    @property
+    def build_directory(self):
+        """Return the directory containing the main Cargo.toml."""
+        return self.pkg.stage.source_path
+
+    @property
+    def build_args(self):
+        """Arguments for ``cargo build``."""
+        return []
+
+    @property
+    def check_args(self):
+        """Argument for ``cargo test`` during check phase"""
+        return []
+
+    def build(self, pkg, spec, prefix):
+        """Runs ``cargo install`` in the source directory"""
+        with fs.working_dir(self.build_directory):
+            inspect.getmodule(pkg).cargo(
+                "install", "--root", "out", "--path", ".", *self.build_args
+            )
+
+    def install(self, pkg, spec, prefix):
+        """Copy build files into package prefix."""
+        with fs.working_dir(self.build_directory):
+            fs.install_tree("out", prefix)
+
+    spack.builder.run_after("install")(execute_install_time_tests)
+
+    def check(self):
+        """Run "cargo test"."""
+        with fs.working_dir(self.build_directory):
+            inspect.getmodule(self.pkg).cargo("test", *self.check_args)

lib/spack/spack/build_systems/cuda.py

@@ -136,12 +136,12 @@ def cuda_flags(arch_list):
         conflicts("%gcc@11:", when="+cuda ^cuda@:11.4.0")
         conflicts("%gcc@11.2:", when="+cuda ^cuda@:11.5")
         conflicts("%gcc@12:", when="+cuda ^cuda@:11.8")
-        conflicts("%gcc@13:", when="+cuda ^cuda@:12.1")
+        conflicts("%gcc@13:", when="+cuda ^cuda@:12.3")
         conflicts("%clang@12:", when="+cuda ^cuda@:11.4.0")
         conflicts("%clang@13:", when="+cuda ^cuda@:11.5")
         conflicts("%clang@14:", when="+cuda ^cuda@:11.7")
         conflicts("%clang@15:", when="+cuda ^cuda@:12.0")
-        conflicts("%clang@16:", when="+cuda ^cuda@:12.1")
+        conflicts("%clang@16:", when="+cuda ^cuda@:12.3")
 
         # https://gist.github.com/ax3l/9489132#gistcomment-3860114
         conflicts("%gcc@10", when="+cuda ^cuda@:11.4.0")

lib/spack/spack/build_systems/go.py

@@ -0,0 +1,98 @@
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import inspect
+
+import llnl.util.filesystem as fs
+
+import spack.builder
+import spack.package_base
+from spack.directives import build_system, extends
+from spack.multimethod import when
+
+from ._checks import BaseBuilder, execute_install_time_tests
+
+
+class GoPackage(spack.package_base.PackageBase):
+    """Specialized class for packages built using the Go toolchain."""
+
+    #: This attribute is used in UI queries that need to know the build
+    #: system base class
+    build_system_class = "GoPackage"
+
+    #: Legacy buildsystem attribute used to deserialize and install old specs
+    legacy_buildsystem = "go"
+
+    build_system("go")
+
+    with when("build_system=go"):
+        # TODO: this seems like it should be depends_on, see
+        # setup_dependent_build_environment in go for why I kept it like this
+        extends("go@1.14:", type="build")
+
+
+@spack.builder.builder("go")
+class GoBuilder(BaseBuilder):
+    """The Go builder encodes the most common way of building software with
+    a golang go.mod file. It has two phases that can be overridden, if need be:
+
+            1. :py:meth:`~.GoBuilder.build`
+            2. :py:meth:`~.GoBuilder.install`
+
+    For a finer tuning you may override:
+
+        +-----------------------------------------------+--------------------+
+        | **Method**                                    | **Purpose**        |
+        +===============================================+====================+
+        | :py:meth:`~.GoBuilder.build_args`             | Specify arguments  |
+        |                                               | to ``go build``    |
+        +-----------------------------------------------+--------------------+
+        | :py:meth:`~.GoBuilder.check_args`             | Specify arguments  |
+        |                                               | to ``go test``     |
+        +-----------------------------------------------+--------------------+
+    """
+
+    phases = ("build", "install")
+
+    #: Callback names for install-time test
+    install_time_test_callbacks = ["check"]
+
+    def setup_build_environment(self, env):
+        env.set("GO111MODULE", "on")
+        env.set("GOTOOLCHAIN", "local")
+
+    @property
+    def build_directory(self):
+        """Return the directory containing the main go.mod."""
+        return self.pkg.stage.source_path
+
+    @property
+    def build_args(self):
+        """Arguments for ``go build``."""
+        # Pass ldflags -s = --strip-all and -w = --no-warnings by default
+        return ["-ldflags", "-s -w", "-o", f"{self.pkg.name}"]
+
+    @property
+    def check_args(self):
+        """Argument for ``go test`` during check phase"""
+        return []
+
+    def build(self, pkg, spec, prefix):
+        """Runs ``go build`` in the source directory"""
+        with fs.working_dir(self.build_directory):
+            inspect.getmodule(pkg).go("build", *self.build_args)
+
+    def install(self, pkg, spec, prefix):
+        """Install built binaries into prefix bin."""
+        with fs.working_dir(self.build_directory):
+            fs.mkdirp(prefix.bin)
+            fs.install(pkg.name, prefix.bin)
+
+    spack.builder.run_after("install")(execute_install_time_tests)
+
+    def check(self):
+        """Run ``go test .`` in the source directory"""
+        with fs.working_dir(self.build_directory):
+            inspect.getmodule(self.pkg).go("test", *self.check_args)

lib/spack/spack/build_systems/oneapi.py

@@ -7,13 +7,12 @@
 import os
 import platform
 import shutil
-from os.path import basename, dirname, isdir
+from os.path import basename, isdir
 
-from llnl.util.filesystem import find_headers, find_libraries, join_path
+from llnl.util.filesystem import HeaderList, find_libraries, join_path, mkdirp
 from llnl.util.link_tree import LinkTree
 
 from spack.directives import conflicts, variant
-from spack.package import mkdirp
 from spack.util.environment import EnvironmentModifications
 from spack.util.executable import Executable
 
@@ -56,10 +55,21 @@ def component_dir(self):
         """Subdirectory for this component in the install prefix."""
         raise NotImplementedError
 
+    @property
+    def v2_layout_versions(self):
+        """Version that implements the v2 directory layout."""
+        raise NotImplementedError
+
+    @property
+    def v2_layout(self):
+        """Returns true if this version implements the v2 directory layout."""
+        return self.spec.satisfies(self.v2_layout_versions)
+
     @property
     def component_prefix(self):
         """Path to component <prefix>/<component>/<version>."""
-        return self.prefix.join(join_path(self.component_dir, self.spec.version))
+        v = self.spec.version.up_to(2) if self.v2_layout else self.spec.version
+        return self.prefix.join(self.component_dir).join(str(v))
 
     @property
     def env_script_args(self):
@@ -113,8 +123,9 @@ def install_component(self, installer_path):
                 shutil.rmtree("/var/intel/installercache", ignore_errors=True)
 
         # Some installers have a bug and do not return an error code when failing
-        if not isdir(join_path(self.prefix, self.component_dir)):
-            raise RuntimeError("install failed")
+        install_dir = self.component_prefix
+        if not isdir(install_dir):
+            raise RuntimeError("install failed to directory: {0}".format(install_dir))
 
     def setup_run_environment(self, env):
         """Adds environment variables to the generated module file.
@@ -129,7 +140,7 @@ def setup_run_environment(self, env):
         if "~envmods" not in self.spec:
             env.extend(
                 EnvironmentModifications.from_sourcing_file(
-                    join_path(self.component_prefix, "env", "vars.sh"), *self.env_script_args
+                    self.component_prefix.env.join("vars.sh"), *self.env_script_args
                 )
             )
 
@@ -168,16 +179,40 @@ class IntelOneApiLibraryPackage(IntelOneApiPackage):
 
     """
 
+    def header_directories(self, dirs):
+        h = HeaderList([])
+        h.directories = dirs
+        return h
+
     @property
     def headers(self):
-        include_path = join_path(self.component_prefix, "include")
-        return find_headers("*", include_path, recursive=True)
+        return self.header_directories(
+            [self.component_prefix.include, self.component_prefix.include.join(self.component_dir)]
+        )
 
     @property
     def libs(self):
-        lib_path = join_path(self.component_prefix, "lib", "intel64")
-        lib_path = lib_path if isdir(lib_path) else dirname(lib_path)
-        return find_libraries("*", root=lib_path, shared=True, recursive=True)
+        # for v2_layout all libraries are in the top level, v1 sometimes put them in intel64
+        return find_libraries("*", root=self.component_prefix.lib, recursive=not self.v2_layout)
+
+
+class IntelOneApiLibraryPackageWithSdk(IntelOneApiPackage):
+    """Base class for Intel oneAPI library packages with SDK components.
+
+    Contains some convenient default implementations for libraries
+    that expose functionality in sdk subdirectories.
+    Implement the method directly in the package if something
+    different is needed.
+
+    """
+
+    @property
+    def headers(self):
+        return self.header_directories([self.component_prefix.sdk.include])
+
+    @property
+    def libs(self):
+        return find_libraries("*", self.component_prefix.sdk.lib64)
 
 
 class IntelOneApiStaticLibraryList:
@@ -212,3 +247,7 @@ def link_flags(self):
     @property
     def ld_flags(self):
         return "{0} {1}".format(self.search_flags, self.link_flags)
+
+
+#: Tuple of Intel math libraries, exported to packages
+INTEL_MATH_LIBRARIES = ("intel-mkl", "intel-oneapi-mkl", "intel-parallel-studio")

lib/spack/spack/build_systems/racket.py

@@ -10,6 +10,7 @@
 import llnl.util.tty as tty
 
 import spack.builder
+import spack.config
 from spack.build_environment import SPACK_NO_PARALLEL_MAKE
 from spack.directives import build_system, extends, maintainers
 from spack.package_base import PackageBase
@@ -93,7 +94,7 @@ def install(self, pkg, spec, prefix):
                 "--copy",
                 "-i",
                 "-j",
-                str(determine_number_of_jobs(parallel=parallel)),
+                str(determine_number_of_jobs(parallel=parallel, config=spack.config.CONFIG)),
                 "--",
                 os.getcwd(),
             ]

lib/spack/spack/caches.py

@@ -26,7 +26,7 @@ def misc_cache_location():
     providers and for which packages provide which tags.
     """
     path = spack.config.get("config:misc_cache", spack.paths.default_misc_cache_path)
-    return spack.util.path.canonicalize_path(path)
+    return spack.util.path.canonicalize_path(path, replacements=spack.paths.path_replacements())
 
 
 def _misc_cache():
@@ -49,7 +49,7 @@ def fetch_cache_location():
     path = spack.config.get("config:source_cache")
     if not path:
         path = spack.paths.default_fetch_cache_path
-    path = spack.util.path.canonicalize_path(path)
+    path = spack.util.path.canonicalize_path(path, replacements=spack.paths.path_replacements())
     return path
 
 

lib/spack/spack/ci.py

@@ -31,13 +31,13 @@
 import spack.binary_distribution as bindist
 import spack.config as cfg
 import spack.environment as ev
+import spack.gpg
 import spack.main
 import spack.mirror
 import spack.paths
 import spack.repo
 import spack.spec
 import spack.util.git
-import spack.util.gpg as gpg_util
 import spack.util.spack_yaml as syaml
 import spack.util.url as url_util
 import spack.util.web as web_util
@@ -46,7 +46,22 @@
 from spack.reporters import CDash, CDashConfiguration
 from spack.reporters.cdash import build_stamp as cdash_build_stamp
 
-JOB_RETRY_CONDITIONS = ["always"]
+# See https://docs.gitlab.com/ee/ci/yaml/#retry for descriptions of conditions
+JOB_RETRY_CONDITIONS = [
+    # "always",
+    "unknown_failure",
+    "script_failure",
+    "api_failure",
+    "stuck_or_timeout_failure",
+    "runner_system_failure",
+    "runner_unsupported",
+    "stale_schedule",
+    # "job_execution_timeout",
+    "archived_failure",
+    "unmet_prerequisites",
+    "scheduler_failure",
+    "data_integrity_failure",
+]
 
 TEMP_STORAGE_MIRROR_NAME = "ci_temporary_mirror"
 SPACK_RESERVED_TAGS = ["public", "protected", "notary"]
@@ -1439,13 +1454,13 @@ def can_sign_binaries():
     """Utility method to determine if this spack instance is capable of
     signing binary packages.  This is currently only possible if the
     spack gpg keystore contains exactly one secret key."""
-    return len(gpg_util.signing_keys()) == 1
+    return len(spack.gpg.signing_keys()) == 1
 
 
 def can_verify_binaries():
     """Utility method to determin if this spack instance is capable (at
     least in theory) of verifying signed binaries."""
-    return len(gpg_util.public_keys()) >= 1
+    return len(spack.gpg.public_keys()) >= 1
 
 
 def _push_mirror_contents(input_spec, sign_binaries, mirror_url):
@@ -1741,7 +1756,11 @@ def reproduce_ci_job(url, work_dir, autostart, gpg_url, runtime):
 
     gpg_path = None
     if gpg_url:
-        gpg_path = web_util.fetch_url_text(gpg_url, dest_dir=os.path.join(work_dir, "_pgp"))
+        gpg_path = web_util.fetch_url_text(
+            gpg_url,
+            dest_dir=os.path.join(work_dir, "_pgp"),
+            fetch_method=spack.config.get("config:url_fetch_method"),
+        )
         rel_gpg_path = gpg_path.replace(work_dir, "").lstrip(os.path.sep)
 
     lock_file = fs.find(work_dir, "spack.lock")[0]
@@ -2098,7 +2117,11 @@ def write_broken_spec(url, pkg_name, stack_name, job_url, pipeline_url, spec_dic
         with open(file_path, "w") as fd:
             fd.write(syaml.dump(broken_spec_details))
         web_util.push_to_url(
-            file_path, url, keep_original=False, extra_args={"ContentType": "text/plain"}
+            file_path,
+            url,
+            keep_original=False,
+            extra_args={"ContentType": "text/plain"},
+            verify_ssl=spack.config.get("config:verify_ssl", True),
         )
     except Exception as err:
         # If there is an S3 error (e.g., access denied or connection
@@ -2115,8 +2138,12 @@ def read_broken_spec(broken_spec_url):
     object.
     """
     try:
-        _, _, fs = web_util.read_from_url(broken_spec_url)
-    except (URLError, web_util.SpackWebError, HTTPError):
+        _, _, fs = web_util.read_from_url(
+            broken_spec_url,
+            verify_ssl=cfg.get("config:verify_ssl", True),
+            timeout=cfg.get("config:connect_timeout", 10),
+        )
+    except (URLError, web_util.WebError, HTTPError):
         tty.warn("Unable to read broken spec from {0}".format(broken_spec_url))
         return None
 

lib/spack/spack/cmd/add.py

@@ -6,7 +6,7 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
+from spack.cmd.common import arguments
 
 description = "add a spec to an environment"
 section = "environments"

lib/spack/spack/cmd/audit.py

@@ -2,6 +2,8 @@
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
+import warnings
+
 import llnl.util.tty as tty
 import llnl.util.tty.colify
 import llnl.util.tty.color as cl
@@ -52,8 +54,10 @@ def setup_parser(subparser):
 
 
 def configs(parser, args):
-    reports = spack.audit.run_group(args.subcommand)
-    _process_reports(reports)
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore")
+        reports = spack.audit.run_group(args.subcommand)
+        _process_reports(reports)
 
 
 def packages(parser, args):

lib/spack/spack/cmd/bootstrap.py

@@ -15,13 +15,14 @@
 import spack.bootstrap
 import spack.bootstrap.config
 import spack.bootstrap.core
-import spack.cmd.common.arguments
 import spack.config
 import spack.main
 import spack.mirror
+import spack.paths
 import spack.spec
 import spack.stage
 import spack.util.path
+from spack.cmd.common import arguments
 
 description = "manage bootstrap configuration"
 section = "system"
@@ -68,12 +69,8 @@
 
 
 def _add_scope_option(parser):
-    scopes = spack.config.scopes()
     parser.add_argument(
-        "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        help="configuration scope to read/modify",
+        "--scope", action=arguments.ConfigScope, help="configuration scope to read/modify"
     )
 
 
@@ -106,7 +103,7 @@ def setup_parser(subparser):
     disable.add_argument("name", help="name of the source to be disabled", nargs="?", default=None)
 
     reset = sp.add_parser("reset", help="reset bootstrapping configuration to Spack defaults")
-    spack.cmd.common.arguments.add_common_arguments(reset, ["yes_to_all"])
+    arguments.add_common_arguments(reset, ["yes_to_all"])
 
     root = sp.add_parser("root", help="get/set the root bootstrap directory")
     _add_scope_option(root)
@@ -195,7 +192,9 @@ def _root(args):
 
     root = spack.config.get("bootstrap:root", default=None, scope=args.scope)
     if root:
-        root = spack.util.path.canonicalize_path(root)
+        root = spack.util.path.canonicalize_path(
+            root, replacements=spack.paths.path_replacements()
+        )
     print(root)
 
 
@@ -339,7 +338,9 @@ def _add(args):
         raise RuntimeError(msg.format(args.name))
 
     # Check that the metadata file exists
-    metadata_dir = spack.util.path.canonicalize_path(args.metadata_dir)
+    metadata_dir = spack.util.path.canonicalize_path(
+        args.metadata_dir, replacements=spack.paths.path_replacements()
+    )
     if not os.path.exists(metadata_dir) or not os.path.isdir(metadata_dir):
         raise RuntimeError('the directory "{0}" does not exist'.format(args.metadata_dir))
 
@@ -388,7 +389,9 @@ def _remove(args):
 
 
 def _mirror(args):
-    mirror_dir = spack.util.path.canonicalize_path(os.path.join(args.root_dir, LOCAL_MIRROR_DIR))
+    mirror_dir = spack.util.path.canonicalize_path(
+        os.path.join(args.root_dir, LOCAL_MIRROR_DIR), replacements=spack.paths.path_replacements()
+    )
 
     # TODO: Here we are adding gnuconfig manually, but this can be fixed
     # TODO: as soon as we have an option to add to a mirror all the possible
@@ -437,9 +440,24 @@ def write_metadata(subdir, metadata):
     instructions += cmd.format("local-sources", rel_directory)
     if args.binary_packages:
         abs_directory, rel_directory = write_metadata(subdir="binaries", metadata=BINARY_METADATA)
-        shutil.copy(spack.util.path.canonicalize_path(CLINGO_JSON), abs_directory)
-        shutil.copy(spack.util.path.canonicalize_path(GNUPG_JSON), abs_directory)
-        shutil.copy(spack.util.path.canonicalize_path(PATCHELF_JSON), abs_directory)
+        shutil.copy(
+            spack.util.path.canonicalize_path(
+                CLINGO_JSON, replacements=spack.paths.path_replacements()
+            ),
+            abs_directory,
+        )
+        shutil.copy(
+            spack.util.path.canonicalize_path(
+                GNUPG_JSON, replacements=spack.paths.path_replacements()
+            ),
+            abs_directory,
+        )
+        shutil.copy(
+            spack.util.path.canonicalize_path(
+                PATCHELF_JSON, replacements=spack.paths.path_replacements()
+            ),
+            abs_directory,
+        )
         instructions += cmd.format("local-binaries", rel_directory)
     print(instructions)
 

lib/spack/spack/cmd/buildcache.py

@@ -21,7 +21,6 @@
 
 import spack.binary_distribution as bindist
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.config
 import spack.environment as ev
 import spack.error
@@ -40,6 +39,7 @@
 import spack.util.web as web_util
 from spack.build_environment import determine_number_of_jobs
 from spack.cmd import display_specs
+from spack.cmd.common import arguments
 from spack.oci.image import (
     Digest,
     ImageReference,
@@ -182,13 +182,10 @@ def setup_parser(subparser: argparse.ArgumentParser):
     )
 
     # used to construct scope arguments below
-    scopes = spack.config.scopes()
-
     check.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_modify_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_modify_scope(),
         help="configuration scope containing mirrors to check",
     )
     check_spec_or_specfile = check.add_mutually_exclusive_group(required=True)
@@ -921,7 +918,12 @@ def copy_buildcache_file(src_url, dest_url, local_path=None):
         try:
             temp_stage.create()
             temp_stage.fetch()
-            web_util.push_to_url(local_path, dest_url, keep_original=True)
+            web_util.push_to_url(
+                local_path,
+                dest_url,
+                keep_original=True,
+                verify_ssl=spack.config.get("config:verify_ssl", True),
+            )
         except spack.error.FetchError as e:
             # Expected, since we have to try all the possible extensions
             tty.debug("no such file: {0}".format(src_url))

lib/spack/spack/cmd/change.py

@@ -4,7 +4,7 @@
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
+from spack.cmd.common import arguments
 
 description = "change an existing spec in an environment"
 section = "environments"

lib/spack/spack/cmd/checksum.py

@@ -11,6 +11,7 @@
 from llnl.util import tty
 
 import spack.cmd
+import spack.config
 import spack.repo
 import spack.spec
 import spack.stage
@@ -275,4 +276,4 @@ def add_versions_to_package(pkg: PackageBase, version_lines: str):
         tty.msg(f"Open {filename} to review the additions.")
 
     if sys.stdout.isatty():
-        editor(filename)
+        editor(filename, debug=spack.config.get("config:debug"))

lib/spack/spack/cmd/ci.py

@@ -16,9 +16,9 @@
 import spack.cmd.buildcache as buildcache
 import spack.config as cfg
 import spack.environment as ev
+import spack.gpg
 import spack.hash_types as ht
 import spack.mirror
-import spack.util.gpg as gpg_util
 import spack.util.timer as timer
 import spack.util.url as url_util
 import spack.util.web as web_util
@@ -305,7 +305,7 @@ def ci_rebuild(args):
     # Fail early if signing is required but we don't have a signing key
     sign_binaries = require_signing is not None and require_signing.lower() == "true"
     if sign_binaries and not spack_ci.can_sign_binaries():
-        gpg_util.list(False, True)
+        spack.gpg.list(False, True)
         tty.die("SPACK_REQUIRE_SIGNING=True => spack must have exactly one signing key")
 
     # Construct absolute paths relative to current $CI_PROJECT_DIR
@@ -730,10 +730,17 @@ def ci_rebuild(args):
             broken_specs_url = ci_config["broken-specs-url"]
             just_built_hash = job_spec.dag_hash()
             broken_spec_path = url_util.join(broken_specs_url, just_built_hash)
-            if web_util.url_exists(broken_spec_path):
+            if web_util.url_exists(
+                broken_spec_path,
+                fetch_method=cfg.get("config:url_fetch_method", "urllib"),
+                verify_ssl=cfg.get("config:verify_ssl"),
+                timeout=cfg.get("config:connect_timeout", 10),
+            ):
                 tty.msg("Removing {0} from the list of broken specs".format(broken_spec_path))
                 try:
-                    web_util.remove_url(broken_spec_path)
+                    web_util.remove_url(
+                        broken_spec_path, verify_ssl=cfg.get("config:verify_ssl", True)
+                    )
                 except Exception as err:
                     # If there is an S3 error (e.g., access denied or connection
                     # error), the first non boto-specific class in the exception

lib/spack/spack/cmd/clean.py

@@ -12,13 +12,14 @@
 
 import spack.bootstrap
 import spack.caches
-import spack.cmd.common.arguments as arguments
 import spack.cmd.test
 import spack.config
+import spack.paths
 import spack.repo
 import spack.stage
 import spack.store
 import spack.util.path
+from spack.cmd.common import arguments
 from spack.paths import lib_path, var_path
 
 description = "remove temporary build files and/or downloaded archives"
@@ -133,7 +134,9 @@ def clean(parser, args):
         remove_python_cache()
 
     if args.bootstrap:
-        bootstrap_prefix = spack.util.path.canonicalize_path(spack.config.get("bootstrap:root"))
+        bootstrap_prefix = spack.util.path.canonicalize_path(
+            spack.config.get("bootstrap:root"), replacements=spack.paths.path_replacements()
+        )
         msg = 'Removing bootstrapped software and configuration in "{0}"'
         tty.msg(msg.format(bootstrap_prefix))
         llnl.util.filesystem.remove_directory_contents(bootstrap_prefix)

lib/spack/spack/cmd/common/arguments.py

@@ -124,6 +124,33 @@ def __call__(self, parser, namespace, values, option_string=None):
         setattr(namespace, self.dest, deptype)
 
 
+class ConfigScope(argparse.Action):
+    """Pick the currently configured config scopes."""
+
+    def __init__(self, *args, **kwargs) -> None:
+        kwargs.setdefault("metavar", spack.config.SCOPES_METAVAR)
+        super().__init__(*args, **kwargs)
+
+    @property
+    def default(self):
+        return self._default() if callable(self._default) else self._default
+
+    @default.setter
+    def default(self, value):
+        self._default = value
+
+    @property
+    def choices(self):
+        return spack.config.scopes().keys()
+
+    @choices.setter
+    def choices(self, value):
+        pass
+
+    def __call__(self, parser, namespace, values, option_string=None):
+        setattr(namespace, self.dest, values)
+
+
 def _cdash_reporter(namespace):
     """Helper function to create a CDash reporter. This function gets an early reference to the
     argparse namespace under construction, so it can later use it to create the object.

lib/spack/spack/cmd/common/confirmation.py

@@ -0,0 +1,30 @@
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import sys
+from typing import List
+
+import llnl.util.tty as tty
+
+import spack.cmd
+
+display_args = {"long": True, "show_flags": False, "variants": False, "indent": 4}
+
+
+def confirm_action(specs: List[spack.spec.Spec], participle: str, noun: str):
+    """Display the list of specs to be acted on and ask for confirmation.
+
+    Args:
+        specs: specs to be removed
+        participle: action expressed as a participle, e.g. "uninstalled"
+        noun: action expressed as a noun, e.g. "uninstallation"
+    """
+    tty.msg(f"The following {len(specs)} packages will be {participle}:\n")
+    spack.cmd.display_specs(specs, **display_args)
+    print("")
+    answer = tty.get_yes_or_no("Do you want to proceed?", default=False)
+    if not answer:
+        tty.msg(f"Aborting {noun}")
+        sys.exit(0)

lib/spack/spack/cmd/common/env_utility.py

@@ -8,13 +8,13 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.deptypes as dt
 import spack.error
 import spack.paths
 import spack.spec
 import spack.store
 from spack import build_environment, traverse
+from spack.cmd.common import arguments
 from spack.context import Context
 from spack.util.environment import dump_environment, pickle_environment
 

lib/spack/spack/cmd/compiler.py

@@ -14,6 +14,7 @@
 import spack.compilers
 import spack.config
 import spack.spec
+from spack.cmd.common import arguments
 
 description = "manage compilers"
 section = "system"
@@ -23,8 +24,6 @@
 def setup_parser(subparser):
     sp = subparser.add_subparsers(metavar="SUBCOMMAND", dest="compiler_command")
 
-    scopes = spack.config.scopes()
-
     # Find
     find_parser = sp.add_parser(
         "find",
@@ -47,9 +46,8 @@ def setup_parser(subparser):
     find_parser.add_argument("add_paths", nargs=argparse.REMAINDER)
     find_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_modify_scope("compilers"),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_modify_scope("compilers"),
         help="configuration scope to modify",
     )
 
@@ -60,20 +58,15 @@ def setup_parser(subparser):
     )
     remove_parser.add_argument("compiler_spec")
     remove_parser.add_argument(
-        "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=None,
-        help="configuration scope to modify",
+        "--scope", action=arguments.ConfigScope, default=None, help="configuration scope to modify"
     )
 
     # List
     list_parser = sp.add_parser("list", help="list available compilers")
     list_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_list_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_list_scope(),
         help="configuration scope to read from",
     )
 
@@ -82,9 +75,8 @@ def setup_parser(subparser):
     info_parser.add_argument("compiler_spec")
     info_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_list_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_list_scope(),
         help="configuration scope to read from",
     )
 

lib/spack/spack/cmd/compilers.py

@@ -3,7 +3,7 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
-import spack.config
+from spack.cmd.common import arguments
 from spack.cmd.compiler import compiler_list
 
 description = "list available compilers"
@@ -12,13 +12,8 @@
 
 
 def setup_parser(subparser):
-    scopes = spack.config.scopes()
-
     subparser.add_argument(
-        "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        help="configuration scope to read/modify",
+        "--scope", action=arguments.ConfigScope, help="configuration scope to read/modify"
     )
 
 

lib/spack/spack/cmd/config.py

@@ -10,7 +10,6 @@
 import llnl.util.filesystem as fs
 import llnl.util.tty as tty
 
-import spack.cmd.common.arguments
 import spack.config
 import spack.environment as ev
 import spack.repo
@@ -18,6 +17,7 @@
 import spack.schema.packages
 import spack.store
 import spack.util.spack_yaml as syaml
+from spack.cmd.common import arguments
 from spack.util.editor import editor
 
 description = "get and set configuration options"
@@ -26,14 +26,9 @@
 
 
 def setup_parser(subparser):
-    scopes = spack.config.scopes()
-
     # User can only choose one
     subparser.add_argument(
-        "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        help="configuration scope to read/modify",
+        "--scope", action=arguments.ConfigScope, help="configuration scope to read/modify"
     )
 
     sp = subparser.add_subparsers(metavar="SUBCOMMAND", dest="config_command")
@@ -101,13 +96,13 @@ def setup_parser(subparser):
     setup_parser.add_parser = add_parser
 
     update = sp.add_parser("update", help="update configuration files to the latest format")
-    spack.cmd.common.arguments.add_common_arguments(update, ["yes_to_all"])
+    arguments.add_common_arguments(update, ["yes_to_all"])
     update.add_argument("section", help="section to update")
 
     revert = sp.add_parser(
         "revert", help="revert configuration files to their state before update"
     )
-    spack.cmd.common.arguments.add_common_arguments(revert, ["yes_to_all"])
+    arguments.add_common_arguments(revert, ["yes_to_all"])
     revert.add_argument("section", help="section to update")
 
 
@@ -185,7 +180,7 @@ def config_edit(args):
     if args.print_file:
         print(config_file)
     else:
-        editor(config_file)
+        editor(config_file, debug=spack.config.get("config:debug"))
 
 
 def config_list(args):

lib/spack/spack/cmd/create.py

@@ -11,6 +11,7 @@
 import llnl.util.tty as tty
 from llnl.util.filesystem import mkdirp
 
+import spack.config
 import spack.repo
 import spack.stage
 import spack.util.web
@@ -172,6 +173,14 @@ def configure_args(self):
         return args"""
 
 
+class CargoPackageTemplate(PackageTemplate):
+    """Provides appropriate overrides for cargo-based packages"""
+
+    base_class_name = "CargoPackage"
+
+    body_def = ""
+
+
 class CMakePackageTemplate(PackageTemplate):
     """Provides appropriate overrides for CMake-based packages"""
 
@@ -186,6 +195,14 @@ def cmake_args(self):
         return args"""
 
 
+class GoPackageTemplate(PackageTemplate):
+    """Provides appropriate overrides for Go-module-based packages"""
+
+    base_class_name = "GoPackage"
+
+    body_def = ""
+
+
 class LuaPackageTemplate(PackageTemplate):
     """Provides appropriate overrides for LuaRocks-based packages"""
 
@@ -575,28 +592,30 @@ def __init__(self, name, *args, **kwargs):
 
 
 templates = {
-    "autotools": AutotoolsPackageTemplate,
     "autoreconf": AutoreconfPackageTemplate,
-    "cmake": CMakePackageTemplate,
-    "bundle": BundlePackageTemplate,
-    "qmake": QMakePackageTemplate,
-    "maven": MavenPackageTemplate,
-    "scons": SconsPackageTemplate,
-    "waf": WafPackageTemplate,
+    "autotools": AutotoolsPackageTemplate,
     "bazel": BazelPackageTemplate,
+    "bundle": BundlePackageTemplate,
+    "cargo": CargoPackageTemplate,
+    "cmake": CMakePackageTemplate,
+    "generic": PackageTemplate,
+    "go": GoPackageTemplate,
+    "intel": IntelPackageTemplate,
+    "lua": LuaPackageTemplate,
+    "makefile": MakefilePackageTemplate,
+    "maven": MavenPackageTemplate,
+    "meson": MesonPackageTemplate,
+    "octave": OctavePackageTemplate,
+    "perlbuild": PerlbuildPackageTemplate,
+    "perlmake": PerlmakePackageTemplate,
     "python": PythonPackageTemplate,
+    "qmake": QMakePackageTemplate,
     "r": RPackageTemplate,
     "racket": RacketPackageTemplate,
-    "perlmake": PerlmakePackageTemplate,
-    "perlbuild": PerlbuildPackageTemplate,
-    "octave": OctavePackageTemplate,
     "ruby": RubyPackageTemplate,
-    "makefile": MakefilePackageTemplate,
-    "intel": IntelPackageTemplate,
-    "meson": MesonPackageTemplate,
-    "lua": LuaPackageTemplate,
+    "scons": SconsPackageTemplate,
     "sip": SIPPackageTemplate,
-    "generic": PackageTemplate,
+    "waf": WafPackageTemplate,
 }
 
 
@@ -679,6 +698,8 @@ def __call__(self, stage, url):
         clues = [
             (r"/CMakeLists\.txt$", "cmake"),
             (r"/NAMESPACE$", "r"),
+            (r"/Cargo\.toml$", "cargo"),
+            (r"/go\.mod$", "go"),
             (r"/configure$", "autotools"),
             (r"/configure\.(in|ac)$", "autoreconf"),
             (r"/Makefile\.am$", "autoreconf"),
@@ -966,4 +987,4 @@ def create(parser, args):
 
     # Optionally open up the new package file in your $EDITOR
     if not args.skip_editor:
-        editor(pkg_path)
+        editor(pkg_path, debug=spack.config.get("config:debug"))

lib/spack/spack/cmd/deconcretize.py

@@ -0,0 +1,103 @@
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
+#
+# SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+import argparse
+import sys
+from typing import List
+
+import llnl.util.tty as tty
+
+import spack.cmd
+import spack.cmd.common.confirmation as confirmation
+import spack.environment as ev
+import spack.spec
+from spack.cmd.common import arguments
+
+description = "remove specs from the concretized lockfile of an environment"
+section = "environments"
+level = "long"
+
+# Arguments for display_specs when we find ambiguity
+display_args = {"long": True, "show_flags": False, "variants": False, "indent": 4}
+
+
+def setup_parser(subparser):
+    subparser.add_argument(
+        "--root", action="store_true", help="deconcretize only specific environment roots"
+    )
+    arguments.add_common_arguments(subparser, ["yes_to_all", "specs"])
+    subparser.add_argument(
+        "-a",
+        "--all",
+        action="store_true",
+        dest="all",
+        help="deconcretize ALL specs that match each supplied spec",
+    )
+
+
+def get_deconcretize_list(
+    args: argparse.Namespace, specs: List[spack.spec.Spec], env: ev.Environment
+) -> List[spack.spec.Spec]:
+    """
+    Get list of environment roots to deconcretize
+    """
+    env_specs = [s for _, s in env.concretized_specs()]
+    to_deconcretize = []
+    errors = []
+
+    for s in specs:
+        if args.root:
+            # find all roots matching given spec
+            to_deconc = [e for e in env_specs if e.satisfies(s)]
+        else:
+            # find all roots matching or depending on a matching spec
+            to_deconc = [e for e in env_specs if any(d.satisfies(s) for d in e.traverse())]
+
+        if len(to_deconc) < 1:
+            tty.warn(f"No matching specs to deconcretize for {s}")
+
+        elif len(to_deconc) > 1 and not args.all:
+            errors.append((s, to_deconc))
+
+        to_deconcretize.extend(to_deconc)
+
+    if errors:
+        for spec, matching in errors:
+            tty.error(f"{spec} matches multiple concrete specs:")
+            sys.stderr.write("\n")
+            spack.cmd.display_specs(matching, output=sys.stderr, **display_args)
+            sys.stderr.write("\n")
+            sys.stderr.flush()
+        tty.die("Use '--all' to deconcretize all matching specs, or be more specific")
+
+    return to_deconcretize
+
+
+def deconcretize_specs(args, specs):
+    env = spack.cmd.require_active_env(cmd_name="deconcretize")
+
+    if args.specs:
+        deconcretize_list = get_deconcretize_list(args, specs, env)
+    else:
+        deconcretize_list = [s for _, s in env.concretized_specs()]
+
+    if not args.yes_to_all:
+        confirmation.confirm_action(deconcretize_list, "deconcretized", "deconcretization")
+
+    with env.write_transaction():
+        for spec in deconcretize_list:
+            env.deconcretize(spec)
+        env.write()
+
+
+def deconcretize(parser, args):
+    if not args.specs and not args.all:
+        tty.die(
+            "deconcretize requires at least one spec argument.",
+            " Use `spack deconcretize --all` to deconcretize ALL specs.",
+        )
+
+    specs = spack.cmd.parse_specs(args.specs) if args.specs else [any]
+    deconcretize_specs(args, specs)

lib/spack/spack/cmd/dependencies.py

@@ -9,11 +9,11 @@
 from llnl.util.tty.colify import colify
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.environment as ev
 import spack.package_base
 import spack.repo
 import spack.store
+from spack.cmd.common import arguments
 
 description = "show dependencies of a package"
 section = "basic"

lib/spack/spack/cmd/dependents.py

@@ -9,10 +9,10 @@
 from llnl.util.tty.colify import colify
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.environment as ev
 import spack.repo
 import spack.store
+from spack.cmd.common import arguments
 
 description = "show packages that depend on another"
 section = "basic"

lib/spack/spack/cmd/deprecate.py

@@ -20,9 +20,9 @@
 from llnl.util.symlink import symlink
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.environment as ev
 import spack.store
+from spack.cmd.common import arguments
 from spack.database import InstallStatuses
 from spack.error import SpackError
 

lib/spack/spack/cmd/dev_build.py

@@ -9,9 +9,9 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.config
 import spack.repo
+from spack.cmd.common import arguments
 
 description = "developer build: build from code in current working directory"
 section = "build"

lib/spack/spack/cmd/develop.py

@@ -8,10 +8,11 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
+import spack.paths
 import spack.spec
 import spack.util.path
 import spack.version
+from spack.cmd.common import arguments
 from spack.error import SpackError
 
 description = "add a spec to an environment's dev-build information"
@@ -55,7 +56,9 @@ def develop(parser, args):
         # download all dev specs
         for name, entry in env.dev_specs.items():
             path = entry.get("path", name)
-            abspath = spack.util.path.canonicalize_path(path, default_wd=env.path)
+            abspath = spack.util.path.canonicalize_path(
+                path, default_wd=env.path, replacements=spack.paths.path_replacements()
+            )
 
             if os.path.exists(abspath):
                 msg = "Skipping developer download of %s" % entry["spec"]
@@ -86,7 +89,9 @@ def develop(parser, args):
 
     # default path is relative path to spec.name
     path = args.path or spec.name
-    abspath = spack.util.path.canonicalize_path(path, default_wd=env.path)
+    abspath = spack.util.path.canonicalize_path(
+        path, default_wd=env.path, replacements=spack.paths.path_replacements()
+    )
 
     # clone default: only if the path doesn't exist
     clone = args.clone

lib/spack/spack/cmd/diff.py

@@ -10,11 +10,11 @@
 from llnl.util.tty.color import cprint, get_color_when
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.environment as ev
 import spack.solver.asp as asp
 import spack.util.environment
 import spack.util.spack_json as sjson
+from spack.cmd.common import arguments
 
 description = "compare two specs"
 section = "basic"
@@ -200,6 +200,8 @@ def diff(parser, args):
 
     specs = []
     for spec in spack.cmd.parse_specs(args.specs):
+        # If the spec has a hash, check it before disambiguating
+        spec.replace_hash()
         if spec.concrete:
             specs.append(spec)
         else:

lib/spack/spack/cmd/edit.py

@@ -9,6 +9,7 @@
 import llnl.util.tty as tty
 
 import spack.cmd
+import spack.config
 import spack.paths
 import spack.repo
 from spack.spec import Spec
@@ -45,7 +46,7 @@ def edit_package(name, repo_path, namespace):
     else:
         raise spack.repo.UnknownPackageError(spec.name)
 
-    editor(path)
+    editor(path, debug=spack.config.get("config:debug"))
 
 
 def setup_parser(subparser):

lib/spack/spack/cmd/env.py

@@ -20,7 +20,6 @@
 import spack.cmd
 import spack.cmd.common
 import spack.cmd.common.arguments
-import spack.cmd.common.arguments as arguments
 import spack.cmd.install
 import spack.cmd.modules
 import spack.cmd.uninstall
@@ -31,6 +30,7 @@
 import spack.schema.env
 import spack.spec
 import spack.tengine
+from spack.cmd.common import arguments
 from spack.util.environment import EnvironmentModifications
 
 description = "manage virtual environments"

lib/spack/spack/cmd/extensions.py

@@ -10,10 +10,10 @@
 from llnl.util.tty.colify import colify
 
 import spack.cmd as cmd
-import spack.cmd.common.arguments as arguments
 import spack.environment as ev
 import spack.repo
 import spack.store
+from spack.cmd.common import arguments
 
 description = "list extensions for package"
 section = "extensions"

lib/spack/spack/cmd/external.py

@@ -14,12 +14,12 @@
 
 import spack
 import spack.cmd
-import spack.cmd.common.arguments
 import spack.config
 import spack.cray_manifest as cray_manifest
 import spack.detection
 import spack.error
 import spack.util.environment
+from spack.cmd.common import arguments
 
 description = "manage external packages in Spack configuration"
 section = "config"
@@ -29,8 +29,6 @@
 def setup_parser(subparser):
     sp = subparser.add_subparsers(metavar="SUBCOMMAND", dest="external_command")
 
-    scopes = spack.config.scopes()
-
     find_parser = sp.add_parser("find", help="add external packages to packages.yaml")
     find_parser.add_argument(
         "--not-buildable",
@@ -48,15 +46,14 @@ def setup_parser(subparser):
     )
     find_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_modify_scope("packages"),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_modify_scope("packages"),
         help="configuration scope to modify",
     )
     find_parser.add_argument(
         "--all", action="store_true", help="search for all packages that Spack knows about"
     )
-    spack.cmd.common.arguments.add_common_arguments(find_parser, ["tags", "jobs"])
+    arguments.add_common_arguments(find_parser, ["tags", "jobs"])
     find_parser.add_argument("packages", nargs=argparse.REMAINDER)
     find_parser.epilog = (
         'The search is by default on packages tagged with the "build-tools" or '

lib/spack/spack/cmd/fetch.py

@@ -6,11 +6,11 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.config
 import spack.environment as ev
 import spack.repo
 import spack.traverse
+from spack.cmd.common import arguments
 
 description = "fetch archives for packages"
 section = "build"

lib/spack/spack/cmd/find.py

@@ -12,9 +12,9 @@
 
 import spack.bootstrap
 import spack.cmd as cmd
-import spack.cmd.common.arguments as arguments
 import spack.environment as ev
 import spack.repo
+from spack.cmd.common import arguments
 from spack.database import InstallStatuses
 
 description = "list and search installed packages"

lib/spack/spack/cmd/gc.py

@@ -6,6 +6,7 @@
 import llnl.util.tty as tty
 
 import spack.cmd.common.arguments
+import spack.cmd.common.confirmation
 import spack.cmd.uninstall
 import spack.environment as ev
 import spack.store
@@ -41,6 +42,6 @@ def gc(parser, args):
         return
 
     if not args.yes_to_all:
-        spack.cmd.uninstall.confirm_removal(specs)
+        spack.cmd.common.confirmation.confirm_action(specs, "uninstalled", "uninstallation")
 
     spack.cmd.uninstall.do_uninstall(specs, force=False)

lib/spack/spack/cmd/gpg.py

@@ -7,11 +7,11 @@
 import os
 
 import spack.binary_distribution
-import spack.cmd.common.arguments as arguments
+import spack.gpg
 import spack.mirror
 import spack.paths
-import spack.util.gpg
 import spack.util.url
+from spack.cmd.common import arguments
 
 description = "handle GPG actions for spack"
 section = "packaging"
@@ -129,40 +129,38 @@ def setup_parser(subparser):
 def gpg_create(args):
     """create a new key"""
     if args.export or args.secret:
-        old_sec_keys = spack.util.gpg.signing_keys()
+        old_sec_keys = spack.gpg.signing_keys()
 
     # Create the new key
-    spack.util.gpg.create(
-        name=args.name, email=args.email, comment=args.comment, expires=args.expires
-    )
+    spack.gpg.create(name=args.name, email=args.email, comment=args.comment, expires=args.expires)
     if args.export or args.secret:
-        new_sec_keys = set(spack.util.gpg.signing_keys())
+        new_sec_keys = set(spack.gpg.signing_keys())
         new_keys = new_sec_keys.difference(old_sec_keys)
 
     if args.export:
-        spack.util.gpg.export_keys(args.export, new_keys)
+        spack.gpg.export_keys(args.export, new_keys)
     if args.secret:
-        spack.util.gpg.export_keys(args.secret, new_keys, secret=True)
+        spack.gpg.export_keys(args.secret, new_keys, secret=True)
 
 
 def gpg_export(args):
     """export a gpg key, optionally including secret key"""
     keys = args.keys
     if not keys:
-        keys = spack.util.gpg.signing_keys()
-    spack.util.gpg.export_keys(args.location, keys, args.secret)
+        keys = spack.gpg.signing_keys()
+    spack.gpg.export_keys(args.location, keys, args.secret)
 
 
 def gpg_list(args):
     """list keys available in the keyring"""
-    spack.util.gpg.list(args.trusted, args.signing)
+    spack.gpg.list(args.trusted, args.signing)
 
 
 def gpg_sign(args):
     """sign a package"""
     key = args.key
     if key is None:
-        keys = spack.util.gpg.signing_keys()
+        keys = spack.gpg.signing_keys()
         if len(keys) == 1:
             key = keys[0]
         elif not keys:
@@ -173,12 +171,12 @@ def gpg_sign(args):
     if not output:
         output = args.spec[0] + ".asc"
     # TODO: Support the package format Spack creates.
-    spack.util.gpg.sign(key, " ".join(args.spec), output, args.clearsign)
+    spack.gpg.sign(key, " ".join(args.spec), output, args.clearsign)
 
 
 def gpg_trust(args):
     """add a key to the keyring"""
-    spack.util.gpg.trust(args.keyfile)
+    spack.gpg.trust(args.keyfile)
 
 
 def gpg_init(args):
@@ -191,12 +189,12 @@ def gpg_init(args):
         for filename in filenames:
             if not filename.endswith(".key"):
                 continue
-            spack.util.gpg.trust(os.path.join(root, filename))
+            spack.gpg.trust(os.path.join(root, filename))
 
 
 def gpg_untrust(args):
     """remove a key from the keyring"""
-    spack.util.gpg.untrust(args.signing, *args.keys)
+    spack.gpg.untrust(args.signing, *args.keys)
 
 
 def gpg_verify(args):
@@ -205,7 +203,7 @@ def gpg_verify(args):
     signature = args.signature
     if signature is None:
         signature = args.spec[0] + ".asc"
-    spack.util.gpg.verify(signature, " ".join(args.spec))
+    spack.gpg.verify(signature, " ".join(args.spec))
 
 
 def gpg_publish(args):

lib/spack/spack/cmd/graph.py

@@ -5,10 +5,10 @@
 from llnl.util import tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.config
 import spack.environment as ev
 import spack.store
+from spack.cmd.common import arguments
 from spack.graph import DAGWithDependencyTypes, SimpleDAG, graph_ascii, graph_dot, static_graph_dot
 
 description = "generate graphs of package dependency relationships"
@@ -61,7 +61,7 @@ def graph(parser, args):
         args.dot = True
         env = ev.active_environment()
         if env:
-            specs = env.all_specs()
+            specs = env.concrete_roots()
         else:
             specs = spack.store.STORE.db.query()
 

lib/spack/spack/cmd/info.py

@@ -3,6 +3,7 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
+import sys
 import textwrap
 from itertools import zip_longest
 
@@ -10,12 +11,13 @@
 import llnl.util.tty.color as color
 from llnl.util.tty.colify import colify
 
-import spack.cmd.common.arguments as arguments
 import spack.deptypes as dt
 import spack.fetch_strategy as fs
 import spack.install_test
 import spack.repo
 import spack.spec
+import spack.version
+from spack.cmd.common import arguments
 from spack.package_base import preferred_version
 
 description = "get detailed information on a particular package"
@@ -53,6 +55,7 @@ def setup_parser(subparser):
         ("--tags", print_tags.__doc__),
         ("--tests", print_tests.__doc__),
         ("--virtuals", print_virtuals.__doc__),
+        ("--variants-by-name", "list variants in strict name order; don't group by condition"),
     ]
     for opt, help_comment in options:
         subparser.add_argument(opt, action="store_true", help=help_comment)
@@ -77,35 +80,10 @@ def license(s):
 
 
 class VariantFormatter:
-    def __init__(self, variants):
-        self.variants = variants
+    def __init__(self, pkg):
+        self.variants = pkg.variants
         self.headers = ("Name [Default]", "When", "Allowed values", "Description")
 
-        # Formats
-        fmt_name = "{0} [{1}]"
-
-        # Initialize column widths with the length of the
-        # corresponding headers, as they cannot be shorter
-        # than that
-        self.column_widths = [len(x) for x in self.headers]
-
-        # Expand columns based on max line lengths
-        for k, e in variants.items():
-            v, w = e
-            candidate_max_widths = (
-                len(fmt_name.format(k, self.default(v))),  # Name [Default]
-                len(str(w)),
-                len(v.allowed_values),  # Allowed values
-                len(v.description),  # Description
-            )
-
-            self.column_widths = (
-                max(self.column_widths[0], candidate_max_widths[0]),
-                max(self.column_widths[1], candidate_max_widths[1]),
-                max(self.column_widths[2], candidate_max_widths[2]),
-                max(self.column_widths[3], candidate_max_widths[3]),
-            )
-
         # Don't let name or possible values be less than max widths
         _, cols = tty.terminal_size()
         max_name = min(self.column_widths[0], 30)
@@ -137,6 +115,8 @@ def default(self, v):
     def lines(self):
         if not self.variants:
             yield "    None"
+            return
+
         else:
             yield "    " + self.fmt % self.headers
             underline = tuple([w * "=" for w in self.column_widths])
@@ -159,7 +139,7 @@ def lines(self):
                     yield "    " + self.fmt % t
 
 
-def print_dependencies(pkg):
+def print_dependencies(pkg, args):
     """output build, link, and run package dependencies"""
 
     for deptype in ("build", "link", "run"):
@@ -172,7 +152,7 @@ def print_dependencies(pkg):
             color.cprint("    None")
 
 
-def print_detectable(pkg):
+def print_detectable(pkg, args):
     """output information on external detection"""
 
     color.cprint("")
@@ -200,7 +180,7 @@ def print_detectable(pkg):
         color.cprint("    False")
 
 
-def print_maintainers(pkg):
+def print_maintainers(pkg, args):
     """output package maintainers"""
 
     if len(pkg.maintainers) > 0:
@@ -209,7 +189,7 @@ def print_maintainers(pkg):
         color.cprint(section_title("Maintainers: ") + mnt)
 
 
-def print_phases(pkg):
+def print_phases(pkg, args):
     """output installation phases"""
 
     if hasattr(pkg.builder, "phases") and pkg.builder.phases:
@@ -221,7 +201,7 @@ def print_phases(pkg):
         color.cprint(phase_str)
 
 
-def print_tags(pkg):
+def print_tags(pkg, args):
     """output package tags"""
 
     color.cprint("")
@@ -233,7 +213,7 @@ def print_tags(pkg):
         color.cprint("    None")
 
 
-def print_tests(pkg):
+def print_tests(pkg, args):
     """output relevant build-time and stand-alone tests"""
 
     # Some built-in base packages (e.g., Autotools) define callback (e.g.,
@@ -271,18 +251,171 @@ def print_tests(pkg):
         color.cprint("    None")
 
 
-def print_variants(pkg):
+def _fmt_value(v):
+    if v is None or isinstance(v, bool):
+        return str(v).lower()
+    else:
+        return str(v)
+
+
+def _fmt_name_and_default(variant):
+    """Print colorized name [default] for a variant."""
+    return color.colorize(f"@c{{{variant.name}}} @C{{[{_fmt_value(variant.default)}]}}")
+
+
+def _fmt_when(when, indent):
+    return color.colorize(f"{indent * ' '}@B{{when}} {color.cescape(when)}")
+
+
+def _fmt_variant_description(variant, width, indent):
+    """Format a variant's description, preserving explicit line breaks."""
+    return "\n".join(
+        textwrap.fill(
+            line, width=width, initial_indent=indent * " ", subsequent_indent=indent * " "
+        )
+        for line in variant.description.split("\n")
+    )
+
+
+def _fmt_variant(variant, max_name_default_len, indent, when=None, out=None):
+    out = out or sys.stdout
+
+    _, cols = tty.terminal_size()
+
+    name_and_default = _fmt_name_and_default(variant)
+    name_default_len = color.clen(name_and_default)
+
+    values = variant.values
+    if not isinstance(variant.values, (tuple, list, spack.variant.DisjointSetsOfValues)):
+        values = [variant.values]
+
+    # put 'none' first, sort the rest by value
+    sorted_values = sorted(values, key=lambda v: (v != "none", v))
+
+    pad = 4  # min padding between 'name [default]' and values
+    value_indent = (indent + max_name_default_len + pad) * " "  # left edge of values
+
+    # This preserves any formatting (i.e., newlines) from how the description was
+    # written in package.py, but still wraps long lines for small terminals.
+    # This allows some packages to provide detailed help on their variants (see, e.g., gasnet).
+    formatted_values = "\n".join(
+        textwrap.wrap(
+            f"{', '.join(_fmt_value(v) for v in sorted_values)}",
+            width=cols - 2,
+            initial_indent=value_indent,
+            subsequent_indent=value_indent,
+        )
+    )
+    formatted_values = formatted_values[indent + name_default_len + pad :]
+
+    # name [default]   value1, value2, value3, ...
+    padding = pad * " "
+    color.cprint(f"{indent * ' '}{name_and_default}{padding}@c{{{formatted_values}}}", stream=out)
+
+    # when <spec>
+    description_indent = indent + 4
+    if when is not None and when != spack.spec.Spec():
+        out.write(_fmt_when(when, description_indent - 2))
+        out.write("\n")
+
+    # description, preserving explicit line breaks from the way it's written in the package file
+    out.write(_fmt_variant_description(variant, cols - 2, description_indent))
+    out.write("\n")
+
+
+def _variants_by_name_when(pkg):
+    """Adaptor to get variants keyed by { name: { when: { [Variant...] } }."""
+    # TODO: replace with pkg.variants_by_name(when=True) when unified directive dicts are merged.
+    variants = {}
+    for name, (variant, whens) in pkg.variants.items():
+        for when in whens:
+            variants.setdefault(name, {}).setdefault(when, []).append(variant)
+    return variants
+
+
+def _variants_by_when_name(pkg):
+    """Adaptor to get variants keyed by { when: { name: Variant } }"""
+    # TODO: replace with pkg.variants when unified directive dicts are merged.
+    variants = {}
+    for name, (variant, whens) in pkg.variants.items():
+        for when in whens:
+            variants.setdefault(when, {})[name] = variant
+    return variants
+
+
+def _print_variants_header(pkg):
     """output variants"""
 
+    if not pkg.variants:
+        print("    None")
+        return
+
     color.cprint("")
     color.cprint(section_title("Variants:"))
 
-    formatter = VariantFormatter(pkg.variants)
-    for line in formatter.lines:
-        color.cprint(color.cescape(line))
+    variants_by_name = _variants_by_name_when(pkg)
+
+    # Calculate the max length of the "name [default]" part of the variant display
+    # This lets us know where to print variant values.
+    max_name_default_len = max(
+        color.clen(_fmt_name_and_default(variant))
+        for name, when_variants in variants_by_name.items()
+        for variants in when_variants.values()
+        for variant in variants
+    )
+
+    return max_name_default_len, variants_by_name
 
 
-def print_versions(pkg):
+def _unconstrained_ver_first(item):
+    """sort key that puts specs with open version ranges first"""
+    spec, _ = item
+    return (spack.version.any_version not in spec.versions, spec)
+
+
+def print_variants_grouped_by_when(pkg):
+    max_name_default_len, _ = _print_variants_header(pkg)
+
+    indent = 4
+    variants = _variants_by_when_name(pkg)
+    for when, variants_by_name in sorted(variants.items(), key=_unconstrained_ver_first):
+        padded_values = max_name_default_len + 4
+        start_indent = indent
+
+        if when != spack.spec.Spec():
+            sys.stdout.write("\n")
+            sys.stdout.write(_fmt_when(when, indent))
+            sys.stdout.write("\n")
+
+            # indent names slightly inside 'when', but line up values
+            padded_values -= 2
+            start_indent += 2
+
+        for name, variant in sorted(variants_by_name.items()):
+            _fmt_variant(variant, padded_values, start_indent, None, out=sys.stdout)
+
+
+def print_variants_by_name(pkg):
+    max_name_default_len, variants_by_name = _print_variants_header(pkg)
+    max_name_default_len += 4
+
+    indent = 4
+    for name, when_variants in variants_by_name.items():
+        for when, variants in sorted(when_variants.items(), key=_unconstrained_ver_first):
+            for variant in variants:
+                _fmt_variant(variant, max_name_default_len, indent, when, out=sys.stdout)
+                sys.stdout.write("\n")
+
+
+def print_variants(pkg, args):
+    """output variants"""
+    if args.variants_by_name:
+        print_variants_by_name(pkg)
+    else:
+        print_variants_grouped_by_when(pkg)
+
+
+def print_versions(pkg, args):
     """output versions"""
 
     color.cprint("")
@@ -300,18 +433,24 @@ def print_versions(pkg):
         pad = padder(pkg.versions, 4)
 
         preferred = preferred_version(pkg)
-        url = ""
-        if pkg.has_code:
-            url = fs.for_package_version(pkg, preferred)
 
+        def get_url(version):
+            try:
+                return fs.for_package_version(pkg, version)
+            except spack.fetch_strategy.InvalidArgsError:
+                return "No URL"
+
+        url = get_url(preferred) if pkg.has_code else ""
         line = version("    {0}".format(pad(preferred))) + color.cescape(url)
-        color.cprint(line)
+        color.cwrite(line)
+
+        print()
 
         safe = []
         deprecated = []
         for v in reversed(sorted(pkg.versions)):
             if pkg.has_code:
-                url = fs.for_package_version(pkg, v)
+                url = get_url(v)
             if pkg.versions[v].get("deprecated", False):
                 deprecated.append((v, url))
             else:
@@ -329,7 +468,7 @@ def print_versions(pkg):
                 color.cprint(line)
 
 
-def print_virtuals(pkg):
+def print_virtuals(pkg, args):
     """output virtual packages"""
 
     color.cprint("")
@@ -352,7 +491,7 @@ def print_virtuals(pkg):
         color.cprint("    None")
 
 
-def print_licenses(pkg):
+def print_licenses(pkg, args):
     """Output the licenses of the project."""
 
     color.cprint("")
@@ -384,7 +523,8 @@ def info(parser, args):
     else:
         color.cprint("    None")
 
-    color.cprint(section_title("Homepage: ") + pkg.homepage)
+    if getattr(pkg, "homepage"):
+        color.cprint(section_title("Homepage: ") + pkg.homepage)
 
     # Now output optional information in expected order
     sections = [
@@ -401,6 +541,6 @@ def info(parser, args):
     ]
     for print_it, func in sections:
         if print_it:
-            func(pkg)
+            func(pkg, args)
 
     color.cprint("")

lib/spack/spack/cmd/install.py

@@ -14,7 +14,6 @@
 
 import spack.build_environment
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.config
 import spack.environment as ev
 import spack.fetch_strategy
@@ -23,6 +22,7 @@
 import spack.report
 import spack.spec
 import spack.store
+from spack.cmd.common import arguments
 from spack.error import SpackError
 from spack.installer import PackageInstaller
 

lib/spack/spack/cmd/list.py

@@ -15,9 +15,9 @@
 import llnl.util.tty as tty
 from llnl.util.tty.colify import colify
 
-import spack.cmd.common.arguments as arguments
 import spack.deptypes as dt
 import spack.repo
+from spack.cmd.common import arguments
 from spack.version import VersionList
 
 description = "list and search available packages"

lib/spack/spack/cmd/load.py

@@ -8,12 +8,12 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.cmd.find
 import spack.environment as ev
 import spack.store
 import spack.user_environment as uenv
 import spack.util.environment
+from spack.cmd.common import arguments
 
 description = "add package to the user environment"
 section = "user environment"

lib/spack/spack/cmd/location.py

@@ -9,11 +9,11 @@
 
 import spack.builder
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.environment as ev
 import spack.paths
 import spack.repo
 import spack.stage
+from spack.cmd.common import arguments
 
 description = "print out locations of packages and spack directories"
 section = "basic"

lib/spack/spack/cmd/mark.py

@@ -8,11 +8,11 @@
 from llnl.util import tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.error
 import spack.package_base
 import spack.repo
 import spack.store
+from spack.cmd.common import arguments
 from spack.database import InstallStatuses
 
 description = "mark packages as explicitly or implicitly installed"

lib/spack/spack/cmd/mirror.py

@@ -11,7 +11,6 @@
 
 import spack.caches
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.concretize
 import spack.config
 import spack.environment as ev
@@ -20,6 +19,7 @@
 import spack.spec
 import spack.util.path
 import spack.util.web as web_util
+from spack.cmd.common import arguments
 from spack.error import SpackError
 
 description = "manage mirrors (source and binary)"
@@ -88,18 +88,14 @@ def setup_parser(subparser):
         "--mirror-url", metavar="mirror_url", type=str, help="find mirror to destroy by url"
     )
 
-    # used to construct scope arguments below
-    scopes = spack.config.scopes()
-
     # Add
     add_parser = sp.add_parser("add", help=mirror_add.__doc__)
     add_parser.add_argument("name", help="mnemonic name for mirror", metavar="mirror")
     add_parser.add_argument("url", help="url of mirror directory from 'spack mirror create'")
     add_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_modify_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_modify_scope(),
         help="configuration scope to modify",
     )
     add_parser.add_argument(
@@ -117,9 +113,8 @@ def setup_parser(subparser):
     remove_parser.add_argument("name", help="mnemonic name for mirror", metavar="mirror")
     remove_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_modify_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_modify_scope(),
         help="configuration scope to modify",
     )
 
@@ -136,9 +131,8 @@ def setup_parser(subparser):
     )
     set_url_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_modify_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_modify_scope(),
         help="configuration scope to modify",
     )
     arguments.add_connection_args(set_url_parser, False)
@@ -165,9 +159,8 @@ def setup_parser(subparser):
     set_parser.add_argument("--url", help="url of mirror directory from 'spack mirror create'")
     set_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_modify_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_modify_scope(),
         help="configuration scope to modify",
     )
     arguments.add_connection_args(set_parser, False)
@@ -176,9 +169,8 @@ def setup_parser(subparser):
     list_parser = sp.add_parser("list", help=mirror_list.__doc__)
     list_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_list_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_list_scope(),
         help="configuration scope to read from",
     )
 
@@ -503,7 +495,9 @@ def mirror_destroy(args):
     elif args.mirror_url:
         mirror_url = args.mirror_url
 
-    web_util.remove_url(mirror_url, recursive=True)
+    web_util.remove_url(
+        mirror_url, recursive=True, verify_ssl=spack.config.get("config:verify_ssl", True)
+    )
 
 
 def mirror(parser, args):

lib/spack/spack/cmd/modules/__init__.py

@@ -14,11 +14,11 @@
 from llnl.util.tty import color
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.config
 import spack.modules
 import spack.modules.common
 import spack.repo
+from spack.cmd.common import arguments
 
 description = "manipulate module files"
 section = "environment"

lib/spack/spack/cmd/patch.py

@@ -6,12 +6,12 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.config
 import spack.environment as ev
 import spack.package_base
 import spack.repo
 import spack.traverse
+from spack.cmd.common import arguments
 
 description = "patch expanded archive sources in preparation for install"
 section = "build"

lib/spack/spack/cmd/pkg.py

@@ -12,11 +12,11 @@
 from llnl.util.tty.colify import colify
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
+import spack.package_hash as ph
 import spack.paths
 import spack.repo
 import spack.util.executable as exe
-import spack.util.package_hash as ph
+from spack.cmd.common import arguments
 
 description = "query packages associated with particular git revisions"
 section = "developer"

lib/spack/spack/cmd/remove.py

@@ -6,7 +6,7 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
+from spack.cmd.common import arguments
 
 description = "remove specs from an environment"
 section = "environments"

lib/spack/spack/cmd/repo.py

@@ -9,8 +9,10 @@
 import llnl.util.tty as tty
 
 import spack.config
+import spack.paths
 import spack.repo
 import spack.util.path
+from spack.cmd.common import arguments
 
 description = "manage package source repositories"
 section = "config"
@@ -19,7 +21,6 @@
 
 def setup_parser(subparser):
     sp = subparser.add_subparsers(metavar="SUBCOMMAND", dest="repo_command")
-    scopes = spack.config.scopes()
 
     # Create
     create_parser = sp.add_parser("create", help=repo_create.__doc__)
@@ -43,9 +44,8 @@ def setup_parser(subparser):
     list_parser = sp.add_parser("list", help=repo_list.__doc__)
     list_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_list_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_list_scope(),
         help="configuration scope to read from",
     )
 
@@ -54,9 +54,8 @@ def setup_parser(subparser):
     add_parser.add_argument("path", help="path to a Spack package repository directory")
     add_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_modify_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_modify_scope(),
         help="configuration scope to modify",
     )
 
@@ -67,9 +66,8 @@ def setup_parser(subparser):
     )
     remove_parser.add_argument(
         "--scope",
-        choices=scopes,
-        metavar=spack.config.SCOPES_METAVAR,
-        default=spack.config.default_modify_scope(),
+        action=arguments.ConfigScope,
+        default=lambda: spack.config.default_modify_scope(),
         help="configuration scope to modify",
     )
 
@@ -86,7 +84,9 @@ def repo_add(args):
     path = args.path
 
     # real_path is absolute and handles substitution.
-    canon_path = spack.util.path.canonicalize_path(path)
+    canon_path = spack.util.path.canonicalize_path(
+        path, replacements=spack.paths.path_replacements()
+    )
 
     # check if the path exists
     if not os.path.exists(canon_path):
@@ -118,9 +118,13 @@ def repo_remove(args):
     namespace_or_path = args.namespace_or_path
 
     # If the argument is a path, remove that repository from config.
-    canon_path = spack.util.path.canonicalize_path(namespace_or_path)
+    canon_path = spack.util.path.canonicalize_path(
+        namespace_or_path, replacements=spack.paths.path_replacements()
+    )
     for repo_path in repos:
-        repo_canon_path = spack.util.path.canonicalize_path(repo_path)
+        repo_canon_path = spack.util.path.canonicalize_path(
+            repo_path, replacements=spack.paths.path_replacements()
+        )
         if canon_path == repo_canon_path:
             repos.remove(repo_path)
             spack.config.set("repos", repos, args.scope)

lib/spack/spack/cmd/restage.py

@@ -6,8 +6,8 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.repo
+from spack.cmd.common import arguments
 
 description = "revert checked out package source code"
 section = "build"

lib/spack/spack/cmd/solve.py

@@ -12,12 +12,12 @@
 
 import spack
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.config
 import spack.environment
 import spack.hash_types as ht
 import spack.package_base
 import spack.solver.asp as asp
+from spack.cmd.common import arguments
 
 description = "concretize a specs using an ASP solver"
 section = "developer"

lib/spack/spack/cmd/spec.py

@@ -10,11 +10,11 @@
 
 import spack
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.environment as ev
 import spack.hash_types as ht
 import spack.spec
 import spack.store
+from spack.cmd.common import arguments
 
 description = "show what would be installed, given a spec"
 section = "build"

lib/spack/spack/cmd/stage.py

@@ -8,13 +8,13 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.config
 import spack.environment as ev
 import spack.package_base
 import spack.repo
 import spack.stage
 import spack.traverse
+from spack.cmd.common import arguments
 
 description = "expand downloaded archive in preparation for install"
 section = "build"

lib/spack/spack/cmd/test.py

@@ -15,12 +15,12 @@
 from llnl.util.tty import colify
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.environment as ev
 import spack.install_test
 import spack.package_base
 import spack.repo
 import spack.report
+from spack.cmd.common import arguments
 
 description = "run spack's tests for an install"
 section = "admin"

lib/spack/spack/cmd/tutorial.py

@@ -10,11 +10,11 @@
 from llnl.util.filesystem import working_dir
 
 import spack
-import spack.cmd.common.arguments as arguments
 import spack.config
+import spack.gpg
 import spack.paths
 import spack.util.git
-import spack.util.gpg
+from spack.cmd.common import arguments
 from spack.util.spack_yaml import syaml_dict
 
 description = "set up spack for our tutorial (WARNING: modifies config!)"
@@ -23,7 +23,7 @@
 
 
 # tutorial configuration parameters
-tutorial_branch = "releases/v0.20"
+tutorial_branch = "releases/v0.21"
 tutorial_mirror = "file:///mirror"
 tutorial_key = os.path.join(spack.paths.share_path, "keys", "tutorial.pub")
 
@@ -76,7 +76,7 @@ def tutorial(parser, args):
     spack.config.set("mirrors", mirror_config, scope="user")
 
     tty.msg("Ensuring that we trust tutorial binaries", f"spack gpg trust {tutorial_key}")
-    spack.util.gpg.trust(tutorial_key)
+    spack.gpg.trust(tutorial_key)
 
     # Note that checkout MUST be last. It changes Spack under our feet.
     # If you don't put this last, you'll get import errors for the code

lib/spack/spack/cmd/undevelop.py

@@ -6,7 +6,7 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
+from spack.cmd.common import arguments
 
 description = "remove specs from an environment"
 section = "environments"

lib/spack/spack/cmd/uninstall.py

@@ -10,14 +10,13 @@
 from llnl.util.tty.colify import colify
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
+import spack.cmd.common.confirmation as confirmation
 import spack.environment as ev
-import spack.error
 import spack.package_base
-import spack.repo
 import spack.spec
 import spack.store
 import spack.traverse as traverse
+from spack.cmd.common import arguments
 from spack.database import InstallStatuses
 
 description = "remove installed packages"
@@ -278,7 +277,7 @@ def uninstall_specs(args, specs):
         return
 
     if not args.yes_to_all:
-        confirm_removal(uninstall_list)
+        confirmation.confirm_action(uninstall_list, "uninstalled", "uninstallation")
 
     # Uninstall everything on the list
     do_uninstall(uninstall_list, args.force)
@@ -292,21 +291,6 @@ def uninstall_specs(args, specs):
         env.regenerate_views()
 
 
-def confirm_removal(specs: List[spack.spec.Spec]):
-    """Display the list of specs to be removed and ask for confirmation.
-
-    Args:
-        specs: specs to be removed
-    """
-    tty.msg("The following {} packages will be uninstalled:\n".format(len(specs)))
-    spack.cmd.display_specs(specs, **display_args)
-    print("")
-    answer = tty.get_yes_or_no("Do you want to proceed?", default=False)
-    if not answer:
-        tty.msg("Aborting uninstallation")
-        sys.exit(0)
-
-
 def uninstall(parser, args):
     if not args.specs and not args.all:
         tty.die(

lib/spack/spack/cmd/unload.py

@@ -7,10 +7,10 @@
 import sys
 
 import spack.cmd
-import spack.cmd.common.arguments as arguments
 import spack.error
 import spack.user_environment as uenv
 import spack.util.environment
+from spack.cmd.common import arguments
 
 description = "remove package from the user environment"
 section = "user environment"

lib/spack/spack/cmd/versions.py

@@ -8,9 +8,9 @@
 import llnl.util.tty as tty
 from llnl.util.tty.colify import colify
 
-import spack.cmd.common.arguments as arguments
 import spack.repo
 import spack.spec
+from spack.cmd.common import arguments
 from spack.version import infinity_versions, ver
 
 description = "list available versions of a package"

lib/spack/spack/compilers/__init__.py

@@ -154,6 +154,14 @@ def add_compilers_to_config(compilers, scope=None, init_config=True):
     """
     compiler_config = get_compiler_config(scope, init_config)
     for compiler in compilers:
+        if not compiler.cc:
+            tty.debug(f"{compiler.spec} does not have a C compiler")
+        if not compiler.cxx:
+            tty.debug(f"{compiler.spec} does not have a C++ compiler")
+        if not compiler.f77:
+            tty.debug(f"{compiler.spec} does not have a Fortran77 compiler")
+        if not compiler.fc:
+            tty.debug(f"{compiler.spec} does not have a Fortran compiler")
         compiler_config.append(_to_dict(compiler))
     spack.config.set("compilers", compiler_config, scope=scope)
 

lib/spack/spack/compilers/aocc.py

@@ -40,7 +40,6 @@ def debug_flags(self):
             "-gdwarf-5",
             "-gline-tables-only",
             "-gmodules",
-            "-gz",
             "-g",
         ]
 

lib/spack/spack/compilers/clang.py

@@ -55,7 +55,6 @@ def debug_flags(self):
             "-gdwarf-5",
             "-gline-tables-only",
             "-gmodules",
-            "-gz",
             "-g",
         ]
 

lib/spack/spack/compilers/intel.py

@@ -85,6 +85,14 @@ def cxx14_flag(self):
         else:
             return "-std=c++14"
 
+    @property
+    def cxx17_flag(self):
+        # https://www.intel.com/content/www/us/en/developer/articles/news/c17-features-supported-by-c-compiler.html
+        if self.real_version < Version("19"):
+            raise UnsupportedCompilerFlag(self, "the C++17 standard", "cxx17_flag", "< 19")
+        else:
+            return "-std=c++17"
+
     @property
     def c99_flag(self):
         if self.real_version < Version("12"):