avoid double closing of fd in sub-processes (#47035)

Both `multiprocessing.connection.Connection.__del__` and `io.IOBase.__del__` called `os.close` on the same file descriptor. As of Python 3.13, this is an explicit warning. Ensure we close once by usef `os.fdopen(..., closefd=False)`

.codecov.yml

@@ -5,7 +5,7 @@ coverage:
   status:
     project:
       default:
-        threshold: 2.0%
+        threshold: 0.2%
 
 ignore:
   - lib/spack/spack/test/.*

.devcontainer/postCreateCommand.sh

@@ -1,20 +0,0 @@
-#!/bin/bash
-
-# Load spack environment at terminal startup
-cat <<EOF >> /root/.bashrc
-. /workspaces/spack/share/spack/setup-env.sh
-EOF
-
-# Load spack environment in this script
-. /workspaces/spack/share/spack/setup-env.sh
-
-# Ensure generic targets for maximum matching with buildcaches
-spack config --scope site add "packages:all:require:[target=x86_64_v3]"
-spack config --scope site add "concretizer:targets:granularity:generic"
-
-# Find compiler and install gcc-runtime
-spack compiler find --scope site
-
-# Setup buildcaches
-spack mirror add --scope site develop https://binaries.spack.io/develop
-spack buildcache keys --install --trust

.devcontainer/ubuntu20.04/devcontainer.json

@@ -1,5 +0,0 @@
-{
-  "name": "Ubuntu 20.04",
-  "image": "ghcr.io/spack/ubuntu20.04-runner-amd64-gcc-11.4:2023.08.01",
-  "postCreateCommand": "./.devcontainer/postCreateCommand.sh"
-}

.devcontainer/ubuntu22.04/devcontainer.json

@@ -1,5 +0,0 @@
-{
-  "name": "Ubuntu 22.04",
-  "image": "ghcr.io/spack/ubuntu-22.04:v2024-05-07",
-  "postCreateCommand": "./.devcontainer/postCreateCommand.sh"
-}

.github/dependabot.yml

@@ -5,10 +5,13 @@ updates:
     directory: "/"
     schedule:
       interval: "daily"
-  # Requirements to run style checks and build documentation
+  # Requirements to build documentation
   - package-ecosystem: "pip"
-    directories:
-     - "/.github/workflows/requirements/style/*"
-     - "/lib/spack/docs"
+    directory: "/lib/spack/docs"
+    schedule:
+      interval: "daily"
+  # Requirements to run style checks
+  - package-ecosystem: "pip"
+    directory: "/.github/workflows/style"
     schedule:
       interval: "daily"

.github/pull_request_template.md

@@ -1,6 +0,0 @@
-<!--  
-Remember that `spackbot` can help with your PR in multiple ways:
-- `@spackbot help` shows all the commands that are currently available
-- `@spackbot fix style` tries to push a commit to fix style issues in this PR
-- `@spackbot re-run pipeline` runs the pipelines again, if you have write access to the repository 
--->

.github/workflows/audit.yaml

@@ -17,58 +17,33 @@ concurrency:
 jobs:
   # Run audits on all the packages in the built-in repository
   package-audits:
-    runs-on: ${{ matrix.system.os }}
+    runs-on: ${{ matrix.operating_system }}
     strategy:
       matrix:
-        system:
-        - { os: windows-latest, shell: 'powershell Invoke-Expression -Command "./share/spack/qa/windows_test_setup.ps1"; {0}' }
-        - { os: ubuntu-latest, shell: bash }
-        - { os: macos-latest, shell: bash }
-    defaults:
-      run:
-        shell: ${{ matrix.system.shell }}
+        operating_system: ["ubuntu-latest", "macos-latest"]
     steps:
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-    - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236 # @v2
       with:
         python-version: ${{inputs.python_version}}
     - name: Install Python packages
       run: |
         pip install --upgrade pip setuptools pytest coverage[toml]
-    - name: Setup for Windows run
-      if: runner.os == 'Windows'
-      run: |
-        python -m pip install --upgrade pywin32
     - name: Package audits (with coverage)
-      env:
-          COVERAGE_FILE: coverage/.coverage-audits-${{ matrix.system.os }}
-      if: ${{ inputs.with_coverage == 'true' && runner.os != 'Windows' }}
+      if: ${{ inputs.with_coverage == 'true' }}
       run: |
           . share/spack/setup-env.sh
           coverage run $(which spack) audit packages
-          coverage run $(which spack) audit configs
-          coverage run $(which spack) -d audit externals
+          coverage run $(which spack) audit externals        
           coverage combine
+          coverage xml
     - name: Package audits (without coverage)
-      if: ${{ inputs.with_coverage == 'false' && runner.os != 'Windows' }}
+      if: ${{ inputs.with_coverage == 'false' }}
       run: |
           . share/spack/setup-env.sh
-          spack -d audit packages
-          spack -d audit configs
-          spack -d audit externals
-    - name: Package audits (without coverage)
-      if: ${{ runner.os == 'Windows' }}
-      run: |
-          . share/spack/setup-env.sh
-          spack -d audit packages
-          ./share/spack/qa/validate_last_exit.ps1
-          spack -d audit configs
-          ./share/spack/qa/validate_last_exit.ps1
-          spack -d audit externals
-          ./share/spack/qa/validate_last_exit.ps1
-    - uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b
-      if: ${{ inputs.with_coverage == 'true' && runner.os != 'Windows' }}
+          $(which spack) audit packages
+          $(which spack) audit externals
+    - uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d # @v2.1.0
+      if: ${{ inputs.with_coverage == 'true' }}
       with:
-        name: coverage-audits-${{ matrix.system.os }}
-        path: coverage
-        include-hidden-files: true
+        flags: unittests,audits

.github/workflows/bootstrap-test.sh

@@ -1,8 +1,7 @@
 #!/bin/bash
-set -e
+set -ex
 source share/spack/setup-env.sh
-$PYTHON bin/spack bootstrap disable github-actions-v0.5
 $PYTHON bin/spack bootstrap disable spack-install
-$PYTHON bin/spack $SPACK_FLAGS solve zlib
+$PYTHON bin/spack -d solve zlib
 tree $BOOTSTRAP/store
 exit 0

.github/workflows/bootstrap.yml

@@ -13,22 +13,118 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
-  distros-clingo-sources:
+  fedora-clingo-sources:
     runs-on: ubuntu-latest
-    container: ${{ matrix.image }}
-    strategy:
-      matrix:
-        image: ["fedora:latest", "opensuse/leap:latest"]
+    container: "fedora:latest"
     steps:
-      - name: Setup Fedora
-        if: ${{ matrix.image == 'fedora:latest' }}
+      - name: Install dependencies
         run: |
           dnf install -y \
-              bzip2 curl file gcc-c++ gcc gcc-gfortran git gzip \
+              bzip2 curl file gcc-c++ gcc gcc-gfortran git gnupg2 gzip \
               make patch unzip which xz python3 python3-devel tree \
               cmake bison bison-devel libstdc++-static
-      - name: Setup OpenSUSE
-        if: ${{ matrix.image == 'opensuse/leap:latest' }}
+      - name: Checkout
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+        with:
+          fetch-depth: 0
+      - name: Setup non-root user
+        run: |
+          # See [1] below
+          git config --global --add safe.directory /__w/spack/spack
+          useradd spack-test && mkdir -p ~spack-test
+          chown -R spack-test . ~spack-test
+      - name: Setup repo
+        shell: runuser -u spack-test -- bash {0}
+        run: |
+          git --version
+          . .github/workflows/setup_git.sh
+      - name: Bootstrap clingo
+        shell: runuser -u spack-test -- bash {0}
+        run: |
+          source share/spack/setup-env.sh
+          spack bootstrap disable github-actions-v0.5
+          spack bootstrap disable github-actions-v0.4
+          spack external find cmake bison
+          spack -d solve zlib
+          tree ~/.spack/bootstrap/store/
+
+  ubuntu-clingo-sources:
+    runs-on: ubuntu-latest
+    container: "ubuntu:latest"
+    steps:
+      - name: Install dependencies
+        env:
+          DEBIAN_FRONTEND: noninteractive
+        run: |
+          apt-get update -y && apt-get upgrade -y
+          apt-get install -y \
+              bzip2 curl file g++ gcc gfortran git gnupg2 gzip \
+              make patch unzip xz-utils python3 python3-dev tree \
+              cmake bison
+      - name: Checkout
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+        with:
+          fetch-depth: 0
+      - name: Setup non-root user
+        run: |
+          # See [1] below
+          git config --global --add safe.directory /__w/spack/spack
+          useradd spack-test && mkdir -p ~spack-test
+          chown -R spack-test . ~spack-test
+      - name: Setup repo
+        shell: runuser -u spack-test -- bash {0}
+        run: |
+          git --version
+          . .github/workflows/setup_git.sh
+      - name: Bootstrap clingo
+        shell: runuser -u spack-test -- bash {0}
+        run: |
+          source share/spack/setup-env.sh
+          spack bootstrap disable github-actions-v0.5
+          spack bootstrap disable github-actions-v0.4
+          spack external find cmake bison
+          spack -d solve zlib
+          tree ~/.spack/bootstrap/store/
+
+  ubuntu-clingo-binaries-and-patchelf:
+    runs-on: ubuntu-latest
+    container: "ubuntu:latest"
+    steps:
+      - name: Install dependencies
+        env:
+          DEBIAN_FRONTEND: noninteractive
+        run: |
+          apt-get update -y && apt-get upgrade -y
+          apt-get install -y \
+              bzip2 curl file g++ gcc gfortran git gnupg2 gzip \
+              make patch unzip xz-utils python3 python3-dev tree
+      - name: Checkout
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+        with:
+          fetch-depth: 0
+      - name: Setup non-root user
+        run: |
+          # See [1] below
+          git config --global --add safe.directory /__w/spack/spack
+          useradd spack-test && mkdir -p ~spack-test
+          chown -R spack-test . ~spack-test
+      - name: Setup repo
+        shell: runuser -u spack-test -- bash {0}
+        run: |
+          git --version
+          . .github/workflows/setup_git.sh
+      - name: Bootstrap clingo
+        shell: runuser -u spack-test -- bash {0}
+        run: |
+          source share/spack/setup-env.sh
+          spack -d solve zlib
+          tree ~/.spack/bootstrap/store/
+
+  opensuse-clingo-sources:
+    runs-on: ubuntu-latest
+    container: "opensuse/leap:latest"
+    steps:
+      - name: Install dependencies
         run: |
           # Harden CI by applying the workaround described here: https://www.suse.com/support/kb/doc/?id=000019505
           zypper update -y || zypper update -y
@@ -37,117 +133,101 @@ jobs:
               make patch unzip which xz python3 python3-devel tree \
               cmake bison
       - name: Checkout
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
         with:
           fetch-depth: 0
+      - name: Setup repo
+        run: |
+          # See [1] below
+          git config --global --add safe.directory /__w/spack/spack
+          git --version
+          . .github/workflows/setup_git.sh
       - name: Bootstrap clingo
         run: |
           source share/spack/setup-env.sh
-          spack bootstrap disable github-actions-v0.6
           spack bootstrap disable github-actions-v0.5
+          spack bootstrap disable github-actions-v0.4
           spack external find cmake bison
           spack -d solve zlib
           tree ~/.spack/bootstrap/store/
 
-  clingo-sources:
-    runs-on: ${{ matrix.runner }}
-    strategy:
-      matrix:
-        runner: ['macos-13', 'macos-14', "ubuntu-latest"]
+  macos-clingo-sources:
+    runs-on: macos-latest
     steps:
-      - name: Setup macOS
-        if: ${{ matrix.runner != 'ubuntu-latest' }}
+      - name: Install dependencies
         run: |
-          brew install cmake bison tree
+          brew install cmake bison@2.7 tree
       - name: Checkout
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-        with:
-          fetch-depth: 0
-      - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+      - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236 # @v2
         with:
           python-version: "3.12"
       - name: Bootstrap clingo
         run: |
           source share/spack/setup-env.sh
-          spack bootstrap disable github-actions-v0.6
+          export PATH=/usr/local/opt/bison@2.7/bin:$PATH
           spack bootstrap disable github-actions-v0.5
+          spack bootstrap disable github-actions-v0.4
           spack external find --not-buildable cmake bison
           spack -d solve zlib
-          tree $HOME/.spack/bootstrap/store/
-
-  gnupg-sources:
-    runs-on: ${{ matrix.runner }}
-    strategy:
-      matrix:
-        runner: [ 'macos-13', 'macos-14', "ubuntu-latest" ]
-    steps:
-      - name: Setup macOS
-        if: ${{ matrix.runner != 'ubuntu-latest' }}
-        run: brew install tree gawk
-      - name: Remove system executables
-        run: |
-          while [ -n "$(command -v gpg gpg2 patchelf)" ]; do
-            sudo rm $(command -v gpg gpg2 patchelf)
-          done
-      - name: Checkout
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-        with:
-          fetch-depth: 0
-      - name: Bootstrap GnuPG
-        run: |
-          source share/spack/setup-env.sh
-          spack solve zlib
-          spack bootstrap disable github-actions-v0.6
-          spack bootstrap disable github-actions-v0.5
-          spack -d gpg list
           tree ~/.spack/bootstrap/store/
 
-  from-binaries:
-    runs-on: ${{ matrix.runner }}
+  macos-clingo-binaries:
+    runs-on: ${{ matrix.macos-version }}
     strategy:
       matrix:
-        runner: ['macos-13', 'macos-14', "ubuntu-latest"]
+        macos-version: ['macos-12']
     steps:
-      - name: Setup macOS
-        if: ${{ matrix.runner != 'ubuntu-latest' }}
-        run: brew install tree
-      - name: Remove system executables
+      - name: Install dependencies
         run: |
-          while [ -n "$(command -v gpg gpg2 patchelf)" ]; do
-            sudo rm $(command -v gpg gpg2 patchelf)
-          done
+          brew install tree
       - name: Checkout
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-        with:
-          fetch-depth: 0
-      - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
-        with:
-          python-version: |
-            3.8
-            3.9
-            3.10
-            3.11
-            3.12
-            3.13
-      - name: Set bootstrap sources
-        run: |
-          source share/spack/setup-env.sh
-          spack bootstrap disable github-actions-v0.5
-          spack bootstrap disable spack-install
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - name: Bootstrap clingo
         run: |
-          set -e
-          for ver in '3.8' '3.9' '3.10' '3.11' '3.12' '3.13'; do
+          set -ex
+          for ver in '3.7' '3.8' '3.9' '3.10' '3.11' ; do
             not_found=1
             ver_dir="$(find $RUNNER_TOOL_CACHE/Python -wholename "*/${ver}.*/*/bin" | grep . || true)"
+            echo "Testing $ver_dir"
             if [[ -d "$ver_dir" ]] ; then
-              echo "Testing $ver_dir"
               if $ver_dir/python --version ; then
                 export PYTHON="$ver_dir/python"
                 not_found=0
                 old_path="$PATH"
                 export PATH="$ver_dir:$PATH"
-                ./bin/spack-tmpconfig -b ./.github/workflows/bin/bootstrap-test.sh
+                ./bin/spack-tmpconfig -b ./.github/workflows/bootstrap-test.sh
+                export PATH="$old_path"
+              fi
+            fi
+            # NOTE: test all pythons that exist, not all do on 12
+          done
+
+  ubuntu-clingo-binaries:
+    runs-on: ubuntu-20.04
+    steps:
+      - name: Checkout
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+        with:
+          fetch-depth: 0
+      - name: Setup repo
+        run: |
+          git --version
+          . .github/workflows/setup_git.sh
+      - name: Bootstrap clingo
+        run: |
+          set -ex
+          for ver in '3.7' '3.8' '3.9' '3.10' '3.11' ; do
+            not_found=1
+            ver_dir="$(find $RUNNER_TOOL_CACHE/Python -wholename "*/${ver}.*/*/bin" | grep . || true)"
+            echo "Testing $ver_dir"
+            if [[ -d "$ver_dir" ]] ; then
+              if $ver_dir/python --version ; then
+                export PYTHON="$ver_dir/python"
+                not_found=0
+                old_path="$PATH"
+                export PATH="$ver_dir:$PATH"
+                ./bin/spack-tmpconfig -b ./.github/workflows/bootstrap-test.sh
                 export PATH="$old_path"
               fi
             fi
@@ -156,39 +236,122 @@ jobs:
               exit 1
             fi
           done
+
+  ubuntu-gnupg-binaries:
+    runs-on: ubuntu-latest
+    container: "ubuntu:latest"
+    steps:
+      - name: Install dependencies
+        env:
+          DEBIAN_FRONTEND: noninteractive
+        run: |
+          apt-get update -y && apt-get upgrade -y
+          apt-get install -y \
+              bzip2 curl file g++ gcc patchelf gfortran git gzip \
+              make patch unzip xz-utils python3 python3-dev tree
+      - name: Checkout
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+        with:
+          fetch-depth: 0
+      - name: Setup non-root user
+        run: |
+          # See [1] below
+          git config --global --add safe.directory /__w/spack/spack
+          useradd spack-test && mkdir -p ~spack-test
+          chown -R spack-test . ~spack-test
+      - name: Setup repo
+        shell: runuser -u spack-test -- bash {0}
+        run: |
+          git --version
+          . .github/workflows/setup_git.sh
+      - name: Bootstrap GnuPG
+        shell: runuser -u spack-test -- bash {0}
+        run: |
+          source share/spack/setup-env.sh
+          spack bootstrap disable github-actions-v0.4
+          spack bootstrap disable spack-install
+          spack -d gpg list
+          tree ~/.spack/bootstrap/store/
+
+  ubuntu-gnupg-sources:
+    runs-on: ubuntu-latest
+    container: "ubuntu:latest"
+    steps:
+      - name: Install dependencies
+        env:
+          DEBIAN_FRONTEND: noninteractive
+        run: |
+          apt-get update -y && apt-get upgrade -y
+          apt-get install -y \
+              bzip2 curl file g++ gcc patchelf gfortran git gzip \
+              make patch unzip xz-utils python3 python3-dev tree \
+              gawk
+      - name: Checkout
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+        with:
+          fetch-depth: 0
+      - name: Setup non-root user
+        run: |
+          # See [1] below
+          git config --global --add safe.directory /__w/spack/spack
+          useradd spack-test && mkdir -p ~spack-test
+          chown -R spack-test . ~spack-test
+      - name: Setup repo
+        shell: runuser -u spack-test -- bash {0}
+        run: |
+          git --version
+          . .github/workflows/setup_git.sh
+      - name: Bootstrap GnuPG
+        shell: runuser -u spack-test -- bash {0}
+        run: |
+          source share/spack/setup-env.sh
+          spack solve zlib
+          spack bootstrap disable github-actions-v0.5
+          spack bootstrap disable github-actions-v0.4
+          spack -d gpg list
+          tree ~/.spack/bootstrap/store/
+
+  macos-gnupg-binaries:
+    runs-on: macos-latest
+    steps:
+      - name: Install dependencies
+        run: |
+          brew install tree
+          # Remove GnuPG since we want to bootstrap it
+          sudo rm -rf /usr/local/bin/gpg
+      - name: Checkout
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - name: Bootstrap GnuPG
         run: |
           source share/spack/setup-env.sh
+          spack bootstrap disable github-actions-v0.4
+          spack bootstrap disable spack-install
           spack -d gpg list
-          tree $HOME/.spack/bootstrap/store/
+          tree ~/.spack/bootstrap/store/
 
-
-  windows:
-    runs-on: "windows-latest"
+  macos-gnupg-sources:
+    runs-on: macos-latest
     steps:
+      - name: Install dependencies
+        run: |
+          brew install gawk tree
+          # Remove GnuPG since we want to bootstrap it
+          sudo rm -rf /usr/local/bin/gpg
       - name: Checkout
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-        with:
-          fetch-depth: 0
-      - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
-        with:
-          python-version: "3.12"
-      - name: Setup Windows
-        run: |
-          Remove-Item -Path (Get-Command gpg).Path
-          Remove-Item -Path (Get-Command file).Path
-      - name: Bootstrap clingo
-        run: |
-          ./share/spack/setup-env.ps1
-          spack bootstrap disable github-actions-v0.6
-          spack bootstrap disable github-actions-v0.5
-          spack external find --not-buildable cmake bison
-          spack -d solve zlib
-          ./share/spack/qa/validate_last_exit.ps1
-          tree $env:userprofile/.spack/bootstrap/store/
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       - name: Bootstrap GnuPG
         run: |
-          ./share/spack/setup-env.ps1
+          source share/spack/setup-env.sh
+          spack solve zlib
+          spack bootstrap disable github-actions-v0.5
+          spack bootstrap disable github-actions-v0.4
           spack -d gpg list
-          ./share/spack/qa/validate_last_exit.ps1
-          tree $env:userprofile/.spack/bootstrap/store/
+          tree ~/.spack/bootstrap/store/
+
+
+# [1] Distros that have patched git to resolve CVE-2022-24765 (e.g. Ubuntu patching v2.25.1)
+#     introduce breaking behaviorso we have to set `safe.directory` in gitconfig ourselves.
+#     See:
+#     - https://github.blog/2022-04-12-git-security-vulnerability-announced/
+#     - https://github.com/actions/checkout/issues/760
+#     - http://changelogs.ubuntu.com/changelogs/pool/main/g/git/git_2.25.1-1ubuntu3.3/changelog

.github/workflows/build-containers.yml

@@ -40,30 +40,24 @@ jobs:
         # 1: Platforms to build for
         # 2: Base image (e.g. ubuntu:22.04)
         dockerfile: [[amazon-linux, 'linux/amd64,linux/arm64', 'amazonlinux:2'],
-                     [centos-stream9, 'linux/amd64,linux/arm64', 'centos:stream9'],
-                     [leap15, 'linux/amd64,linux/arm64', 'opensuse/leap:15'],
-                     [ubuntu-focal, 'linux/amd64,linux/arm64', 'ubuntu:20.04'],
-                     [ubuntu-jammy, 'linux/amd64,linux/arm64', 'ubuntu:22.04'],
-                     [ubuntu-noble, 'linux/amd64,linux/arm64', 'ubuntu:24.04'],
-                     [almalinux8, 'linux/amd64,linux/arm64', 'almalinux:8'],
-                     [almalinux9, 'linux/amd64,linux/arm64', 'almalinux:9'],
+                     [centos7, 'linux/amd64,linux/arm64,linux/ppc64le', 'centos:7'],
+                     [centos-stream, 'linux/amd64,linux/arm64,linux/ppc64le', 'centos:stream'],
+                     [leap15, 'linux/amd64,linux/arm64,linux/ppc64le', 'opensuse/leap:15'],
+                     [ubuntu-focal, 'linux/amd64,linux/arm64,linux/ppc64le', 'ubuntu:20.04'],
+                     [ubuntu-jammy, 'linux/amd64,linux/arm64,linux/ppc64le', 'ubuntu:22.04'],
+                     [almalinux8, 'linux/amd64,linux/arm64,linux/ppc64le', 'almalinux:8'],
+                     [almalinux9, 'linux/amd64,linux/arm64,linux/ppc64le', 'almalinux:9'],
                      [rockylinux8, 'linux/amd64,linux/arm64', 'rockylinux:8'],
                      [rockylinux9, 'linux/amd64,linux/arm64', 'rockylinux:9'],
-                     [fedora39, 'linux/amd64,linux/arm64', 'fedora:39'],
-                     [fedora40, 'linux/amd64,linux/arm64', 'fedora:40']]
+                     [fedora37, 'linux/amd64,linux/arm64,linux/ppc64le', 'fedora:37'],
+                     [fedora38, 'linux/amd64,linux/arm64,linux/ppc64le', 'fedora:38']]
     name: Build ${{ matrix.dockerfile[0] }}
     if: github.repository == 'spack/spack'
     steps:
       - name: Checkout
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+        uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
 
-      - name: Determine latest release tag
-        id: latest
-        run: |
-          git fetch --quiet --tags
-          echo "tag=$(git tag --list --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -n 1)" | tee -a $GITHUB_OUTPUT
-
-      - uses: docker/metadata-action@369eb591f429131d6889c46b94e711f089e6ca96
+      - uses: docker/metadata-action@96383f45573cb7f253c731d3b3ab81c87ef81934
         id: docker_meta
         with:
           images: |
@@ -77,13 +71,12 @@ jobs:
               type=semver,pattern={{major}}
               type=ref,event=branch
               type=ref,event=pr
-              type=raw,value=latest,enable=${{ github.ref == format('refs/tags/{0}', steps.latest.outputs.tag) }}
 
       - name: Generate the Dockerfile
         env:
           SPACK_YAML_OS: "${{ matrix.dockerfile[2] }}"
         run: |
-          .github/workflows/bin/generate_spack_yaml_containerize.sh
+          .github/workflows/generate_spack_yaml_containerize.sh
           . share/spack/setup-env.sh
           mkdir -p dockerfiles/${{ matrix.dockerfile[0] }}
           spack containerize --last-stage=bootstrap | tee dockerfiles/${{ matrix.dockerfile[0] }}/Dockerfile
@@ -94,19 +87,19 @@ jobs:
           fi
 
       - name: Upload Dockerfile
-        uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b
+        uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
         with:
-          name: dockerfiles_${{ matrix.dockerfile[0] }}
+          name: dockerfiles
           path: dockerfiles
 
       - name: Set up QEMU
-        uses: docker/setup-qemu-action@49b3bc8e6bdd4a60e6116a5414239cba5943d3cf
+        uses: docker/setup-qemu-action@68827325e0b33c7199eb31dd4e31fbe9023e06e3
 
       - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@6524bf65af31da8d45b59e8c27de4bd072b392f5
+        uses: docker/setup-buildx-action@f95db51fddba0c2d1ec667646a06c2ce06100226
 
       - name: Log in to GitHub Container Registry
-        uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567
+        uses: docker/login-action@343f7c4344506bcbf9b4de18042ae17996df046d
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
@@ -114,27 +107,18 @@ jobs:
 
       - name: Log in to DockerHub
         if: github.event_name != 'pull_request'
-        uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567
+        uses: docker/login-action@343f7c4344506bcbf9b4de18042ae17996df046d
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
       - name: Build & Deploy ${{ matrix.dockerfile[0] }}
-        uses: docker/build-push-action@48aba3b46d1b1fec4febb7c5d0c644b249a11355
+        uses: docker/build-push-action@0565240e2d4ab88bba5387d719585280857ece09
         with:
           context: dockerfiles/${{ matrix.dockerfile[0] }}
           platforms: ${{ matrix.dockerfile[1] }}
           push: ${{ github.event_name != 'pull_request' }}
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
           tags: ${{ steps.docker_meta.outputs.tags }}
           labels: ${{ steps.docker_meta.outputs.labels }}
-
-  merge-dockerfiles:
-    runs-on: ubuntu-latest
-    needs: deploy-images
-    steps:
-      - name: Merge Artifacts
-        uses: actions/upload-artifact/merge@6f51ac03b9356f520e9adb1b1b7802705f340c2b
-        with:
-          name: dockerfiles
-          pattern: dockerfiles_*
-          delete-merged: true

.github/workflows/ci.yaml

@@ -15,6 +15,17 @@ concurrency:
   cancel-in-progress: true
 
 jobs:
+  prechecks:
+    needs: [ changes ]
+    uses: ./.github/workflows/valid-style.yml
+    with:
+      with_coverage: ${{ needs.changes.outputs.core }}
+  all-prechecks:
+    needs: [ prechecks ]
+    runs-on: ubuntu-latest
+    steps:
+    - name: Success
+      run: "true"
   # Check which files have been updated by the PR
   changes:
     runs-on: ubuntu-latest
@@ -24,12 +35,12 @@ jobs:
       core: ${{ steps.filter.outputs.core }}
       packages: ${{ steps.filter.outputs.packages }}
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
         if: ${{ github.event_name == 'push' }}
         with:
           fetch-depth: 0
             # For pull requests it's not necessary to checkout the code
-      - uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36
+      - uses: dorny/paths-filter@4512585405083f25c027a35db413c2b3b9006d50
         id: filter
         with:
           # See https://github.com/dorny/paths-filter/issues/56 for the syntax used below
@@ -41,13 +52,6 @@ jobs:
             - 'var/spack/repos/builtin/packages/clingo/**'
             - 'var/spack/repos/builtin/packages/python/**'
             - 'var/spack/repos/builtin/packages/re2c/**'
-            - 'var/spack/repos/builtin/packages/gnupg/**'
-            - 'var/spack/repos/builtin/packages/libassuan/**'
-            - 'var/spack/repos/builtin/packages/libgcrypt/**'
-            - 'var/spack/repos/builtin/packages/libgpg-error/**'
-            - 'var/spack/repos/builtin/packages/libksba/**'
-            - 'var/spack/repos/builtin/packages/npth/**'
-            - 'var/spack/repos/builtin/packages/pinentry/**'
             - 'lib/spack/**'
             - 'share/spack/**'
             - '.github/workflows/bootstrap.yml'
@@ -66,58 +70,17 @@ jobs:
     if: ${{ github.repository == 'spack/spack' && needs.changes.outputs.bootstrap == 'true' }}
     needs: [ prechecks, changes ]
     uses: ./.github/workflows/bootstrap.yml
-    secrets: inherit
-
   unit-tests:
     if: ${{ github.repository == 'spack/spack' && needs.changes.outputs.core == 'true' }}
     needs: [ prechecks, changes ]
     uses: ./.github/workflows/unit_tests.yaml
-    secrets: inherit
-
-  prechecks:
-    needs: [ changes ]
-    uses: ./.github/workflows/valid-style.yml
-    secrets: inherit
-    with:
-      with_coverage: ${{ needs.changes.outputs.core }}
-
-  import-check:
-    needs: [ changes ]
-    uses: ./.github/workflows/import-check.yaml
-
-  all-prechecks:
+  windows:
+    if: ${{ github.repository == 'spack/spack' && needs.changes.outputs.core == 'true' }}
     needs: [ prechecks ]
-    if: ${{ always() }}
+    uses: ./.github/workflows/windows_python.yml
+  all:
+    needs: [ windows, unit-tests, bootstrap ]
     runs-on: ubuntu-latest
     steps:
     - name: Success
-      run: |
-        if [ "${{ needs.prechecks.result }}" == "failure" ] || [ "${{ needs.prechecks.result }}" == "canceled" ]; then
-          echo "Unit tests failed." 
-          exit 1
-        else
-          exit 0
-        fi
-
-  coverage:
-    needs: [ unit-tests, prechecks ]
-    uses: ./.github/workflows/coverage.yml
-    secrets: inherit
-
-  all:
-    needs: [ unit-tests, coverage, bootstrap ]
-    if: ${{ always() }}
-    runs-on: ubuntu-latest
-    # See https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/accessing-contextual-information-about-workflow-runs#needs-context
-    steps:
-    - name: Status summary
-      run: |
-        if [ "${{ needs.unit-tests.result }}" == "failure" ] || [ "${{ needs.unit-tests.result }}" == "canceled" ]; then
-          echo "Unit tests failed." 
-          exit 1
-        elif [ "${{ needs.bootstrap.result }}" == "failure" ] || [ "${{ needs.bootstrap.result }}" == "canceled" ]; then
-          echo "Bootstrap tests failed." 
-          exit 1
-        else
-          exit 0
-        fi
+      run: "true"

.github/workflows/coverage.yml

@@ -1,36 +0,0 @@
-name: coverage
-
-on:
-  workflow_call:
-
-jobs:
-  # Upload coverage reports to codecov once as a single bundle
-  upload:
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-    - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
-      with:
-        python-version: '3.11'
-        cache: 'pip'
-
-    - name: Install python dependencies
-      run: pip install -r .github/workflows/requirements/coverage/requirements.txt
-
-    - name: Download coverage artifact files
-      uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16
-      with:
-        pattern: coverage-*
-        path: coverage
-        merge-multiple: true
-
-    - run: ls -la coverage
-    - run: coverage combine -a coverage/.coverage*
-    - run: coverage xml
-
-    - name: "Upload coverage report to CodeCov"
-      uses: codecov/codecov-action@1e68e06f1dbfde0e4cefc87efeba9e4643565303
-      with:
-        verbose: true
-        fail_ci_if_error: false
-        token: ${{ secrets.CODECOV_TOKEN }}

.github/workflows/import-check.yaml

@@ -1,49 +0,0 @@
-name: import-check
-
-on:
-  workflow_call:
-
-jobs:
-  # Check we don't make the situation with circular imports worse
-  import-check:
-    runs-on: ubuntu-latest
-    steps:
-    - uses: julia-actions/setup-julia@v2
-      with:
-        version: '1.10'
-    - uses: julia-actions/cache@v2
-
-    # PR: use the base of the PR as the old commit
-    - name: Checkout PR base commit
-      if: github.event_name == 'pull_request'
-      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-      with:
-        ref: ${{ github.event.pull_request.base.sha }}
-        path: old
-    # not a PR: use the previous commit as the old commit
-    - name: Checkout previous commit
-      if: github.event_name != 'pull_request'
-      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-      with:
-        fetch-depth: 2
-        path: old
-    - name: Checkout previous commit
-      if: github.event_name != 'pull_request'
-      run: git -C old reset --hard HEAD^
-
-    - name: Checkout new commit
-      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-      with:
-        path: new
-    - name: Install circular import checker
-      uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-      with:
-        repository: haampie/circular-import-fighter
-        ref: e38bcd0aa46368e30648b61b7f0d8c1ca68aadff
-        path: circular-import-fighter
-    - name: Install dependencies
-      working-directory: circular-import-fighter
-      run: make -j dependencies
-    - name: Circular import check
-      working-directory: circular-import-fighter
-      run: make -j compare "SPACK_ROOT=../old ../new"

.github/workflows/install_spack.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env sh
+. share/spack/setup-env.sh
+echo -e "config:\n  build_jobs: 2" > etc/spack/config.yaml
+spack config add "packages:all:target:[x86_64]"
+spack compiler find
+spack compiler info apple-clang
+spack debug report
+spack solve zlib

.github/workflows/nightly-win-builds.yml

@@ -14,10 +14,10 @@ jobs:
   build-paraview-deps:
     runs-on: windows-latest
     steps:
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       with:
         fetch-depth: 0
-    - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236
       with:
         python-version: 3.9
     - name: Install Python packages

.github/workflows/requirements/coverage/requirements.txt

@@ -1 +0,0 @@
-coverage==7.6.1

.github/workflows/requirements/style/requirements.txt

@@ -1,7 +0,0 @@
-black==24.10.0
-clingo==5.7.1
-flake8==7.1.1
-isort==5.13.2
-mypy==1.11.2
-types-six==1.17.0.20241205
-vermin==1.6.0

.github/workflows/setup_git.ps1

@@ -1,3 +1,5 @@
+# (c) 2022 Lawrence Livermore National Laboratory
+
 git config --global user.email "spack@example.com"
 git config --global user.name "Test User"
 git config --global core.longpaths true

.github/workflows/style/requirements.txt

@@ -0,0 +1,7 @@
+black==23.10.1
+clingo==5.6.2
+flake8==6.1.0
+isort==5.12.0
+mypy==1.6.1
+types-six==1.16.21.9
+vermin==1.5.2

.github/workflows/unit_tests.yaml

@@ -15,35 +15,46 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest]
-        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']
+        python-version: ['3.7', '3.8', '3.9', '3.10', '3.11', '3.12']
+        concretizer: ['clingo']
         on_develop:
         - ${{ github.ref == 'refs/heads/develop' }}
         include:
+        - python-version: '3.11'
+          os: ubuntu-latest
+          concretizer: original
+          on_develop: ${{ github.ref == 'refs/heads/develop' }}
         - python-version: '3.6'
           os: ubuntu-20.04
-          on_develop: ${{ github.ref == 'refs/heads/develop' }}
-        - python-version: '3.7'
-          os: ubuntu-22.04
+          concretizer: clingo
           on_develop: ${{ github.ref == 'refs/heads/develop' }}
         exclude:
+        - python-version: '3.7'
+          os: ubuntu-latest
+          concretizer: 'clingo'
+          on_develop: false
         - python-version: '3.8'
           os: ubuntu-latest
+          concretizer: 'clingo'
           on_develop: false
         - python-version: '3.9'
           os: ubuntu-latest
+          concretizer: 'clingo'
           on_develop: false
         - python-version: '3.10'
           os: ubuntu-latest
+          concretizer: 'clingo'
           on_develop: false
         - python-version: '3.11'
           os: ubuntu-latest
+          concretizer: 'clingo'
           on_develop: false
 
     steps:
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
       with:
         fetch-depth: 0
-    - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236 # @v2
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install System packages
@@ -52,13 +63,7 @@ jobs:
           # Needed for unit tests
           sudo apt-get -y install \
               coreutils cvs gfortran graphviz gnupg2 mercurial ninja-build \
-              cmake bison libbison-dev subversion
-    # On ubuntu 24.04, kcov was removed. It may come back in some future Ubuntu
-    - name: Set up Homebrew
-      id: set-up-homebrew
-      uses: Homebrew/actions/setup-homebrew@40e9946c182a64b3db1bf51be0dcb915f7802aa9
-    - name: Install kcov with brew
-      run: "brew install kcov"
+              cmake bison libbison-dev kcov
     - name: Install Python packages
       run: |
           pip install --upgrade pip setuptools pytest pytest-xdist pytest-cov
@@ -67,7 +72,7 @@ jobs:
       run: |
           # Need this for the git tests to succeed.
           git --version
-          . .github/workflows/bin/setup_git.sh
+          . .github/workflows/setup_git.sh
     - name: Bootstrap clingo
       if: ${{ matrix.concretizer == 'clingo' }}
       env:
@@ -80,38 +85,30 @@ jobs:
     - name: Run unit tests
       env:
           SPACK_PYTHON: python
+          SPACK_TEST_SOLVER: ${{ matrix.concretizer }}
           SPACK_TEST_PARALLEL: 2
           COVERAGE: true
-          COVERAGE_FILE: coverage/.coverage-${{ matrix.os }}-python${{ matrix.python-version }}
           UNIT_TEST_COVERAGE: ${{ matrix.python-version == '3.11' }}
       run: |
           share/spack/qa/run-unit-tests
-    - uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b
+    - uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d
       with:
-        name: coverage-${{ matrix.os }}-python${{ matrix.python-version }}
-        path: coverage
-        include-hidden-files: true
+        flags: unittests,linux,${{ matrix.concretizer }}
   # Test shell integration
   shell:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
       with:
         fetch-depth: 0
-    - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236 # @v2
       with:
         python-version: '3.11'
     - name: Install System packages
       run: |
           sudo apt-get -y update
           # Needed for shell tests
-          sudo apt-get install -y coreutils csh zsh tcsh fish dash bash subversion
-    # On ubuntu 24.04, kcov was removed. It may come back in some future Ubuntu
-    - name: Set up Homebrew
-      id: set-up-homebrew
-      uses: Homebrew/actions/setup-homebrew@40e9946c182a64b3db1bf51be0dcb915f7802aa9
-    - name: Install kcov with brew
-      run: "brew install kcov"
+          sudo apt-get install -y coreutils kcov csh zsh tcsh fish dash bash
     - name: Install Python packages
       run: |
           pip install --upgrade pip setuptools pytest coverage[toml] pytest-xdist
@@ -119,17 +116,15 @@ jobs:
       run: |
           # Need this for the git tests to succeed.
           git --version
-          . .github/workflows/bin/setup_git.sh
+          . .github/workflows/setup_git.sh
     - name: Run shell tests
       env:
           COVERAGE: true
       run: |
           share/spack/qa/run-shell-tests
-    - uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b
+    - uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d
       with:
-        name: coverage-shell
-        path: coverage
-        include-hidden-files: true
+        flags: shelltests,linux
 
   # Test RHEL8 UBI with platform Python. This job is run
   # only on PRs modifying core Spack
@@ -140,15 +135,15 @@ jobs:
     - name: Install dependencies
       run: |
           dnf install -y \
-              bzip2 curl gcc-c++ gcc gcc-gfortran git gnupg2 gzip \
+              bzip2 curl file gcc-c++ gcc gcc-gfortran git gnupg2 gzip \
               make patch tcl unzip which xz
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
     - name: Setup repo and non-root user
       run: |
           git --version
-          git config --global --add safe.directory '*'
+          git config --global --add safe.directory /__w/spack/spack
           git fetch --unshallow
-          . .github/workflows/bin/setup_git.sh
+          . .github/workflows/setup_git.sh
           useradd spack-test
           chown -R spack-test .
     - name: Run unit tests
@@ -161,49 +156,45 @@ jobs:
   clingo-cffi:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
       with:
         fetch-depth: 0
-    - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236 # @v2
       with:
-        python-version: '3.13'
+        python-version: '3.11'
     - name: Install System packages
       run: |
           sudo apt-get -y update
-          sudo apt-get -y install coreutils gfortran graphviz gnupg2
+          sudo apt-get -y install coreutils cvs gfortran graphviz gnupg2 mercurial ninja-build kcov
     - name: Install Python packages
       run: |
-          pip install --upgrade pip setuptools pytest coverage[toml] pytest-cov clingo
+          pip install --upgrade pip setuptools pytest coverage[toml] pytest-cov clingo pytest-xdist
           pip install --upgrade flake8 "isort>=4.3.5" "mypy>=0.900" "click" "black"
+    - name: Setup git configuration
+      run: |
+          # Need this for the git tests to succeed.
+          git --version
+          . .github/workflows/setup_git.sh
     - name: Run unit tests (full suite with coverage)
       env:
           COVERAGE: true
-          COVERAGE_FILE: coverage/.coverage-clingo-cffi
+          SPACK_TEST_SOLVER: clingo
       run: |
-        . share/spack/setup-env.sh
-        spack bootstrap disable spack-install
-        spack bootstrap disable github-actions-v0.5
-        spack bootstrap disable github-actions-v0.6
-        spack bootstrap status
-        spack solve zlib
-        spack unit-test --verbose --cov --cov-config=pyproject.toml --cov-report=xml:coverage.xml lib/spack/spack/test/concretization/core.py
-    - uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b
+          share/spack/qa/run-unit-tests
+    - uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d # @v2.1.0
       with:
-        name: coverage-clingo-cffi
-        path: coverage
-        include-hidden-files: true
+        flags: unittests,linux,clingo
   # Run unit tests on MacOS
   macos:
-    runs-on: ${{ matrix.os }}
+    runs-on: macos-latest
     strategy:
       matrix:
-        os: [macos-13, macos-14]
         python-version: ["3.11"]
     steps:
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
       with:
         fetch-depth: 0
-    - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236 # @v2
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install Python packages
@@ -212,52 +203,19 @@ jobs:
           pip install --upgrade pytest coverage[toml] pytest-xdist pytest-cov
     - name: Setup Homebrew packages
       run: |
-        brew install dash fish gcc gnupg kcov
+        brew install dash fish gcc gnupg2 kcov
     - name: Run unit tests
       env:
+        SPACK_TEST_SOLVER: clingo
         SPACK_TEST_PARALLEL: 4
-        COVERAGE_FILE: coverage/.coverage-${{ matrix.os }}-python${{ matrix.python-version }}
       run: |
         git --version
-        . .github/workflows/bin/setup_git.sh
+        . .github/workflows/setup_git.sh
         . share/spack/setup-env.sh
         $(which spack) bootstrap disable spack-install
         $(which spack) solve zlib
         common_args=(--dist loadfile --tx '4*popen//python=./bin/spack-tmpconfig python -u ./bin/spack python' -x)
-        $(which spack) unit-test --verbose --cov --cov-config=pyproject.toml --cov-report=xml:coverage.xml "${common_args[@]}"
-    - uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b
+        $(which spack) unit-test --cov --cov-config=pyproject.toml --cov-report=xml:coverage.xml "${common_args[@]}"
+    - uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d
       with:
-        name: coverage-${{ matrix.os }}-python${{ matrix.python-version }}
-        path: coverage
-        include-hidden-files: true
-  # Run unit tests on Windows
-  windows:
-    defaults:
-      run:
-        shell:
-          powershell Invoke-Expression -Command "./share/spack/qa/windows_test_setup.ps1"; {0}
-    runs-on: windows-latest
-    steps:
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-      with:
-        fetch-depth: 0
-    - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
-      with:
-        python-version: 3.9
-    - name: Install Python packages
-      run: |
-          python -m pip install --upgrade pip pywin32 setuptools pytest-cov clingo
-    - name: Create local develop
-      run: |
-        ./.github/workflows/bin/setup_git.ps1
-    - name: Unit Test
-      env:
-        COVERAGE_FILE: coverage/.coverage-windows
-      run: |
-        spack unit-test -x --verbose --cov --cov-config=pyproject.toml
-        ./share/spack/qa/validate_last_exit.ps1
-    - uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b
-      with:
-        name: coverage-windows
-        path: coverage
-        include-hidden-files: true
+        flags: unittests,macos

.github/workflows/valid-style.yml

@@ -13,19 +13,20 @@ concurrency:
 
 
 jobs:
-  # Validate that the code can be run on all the Python versions supported by Spack
+  # Validate that the code can be run on all the Python versions
+  # supported by Spack
   validate:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-    - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236
       with:
-        python-version: '3.13'
+        python-version: '3.11'
         cache: 'pip'
     - name: Install Python Packages
       run: |
         pip install --upgrade pip setuptools
-        pip install -r .github/workflows/requirements/style/requirements.txt
+        pip install -r .github/workflows/style/requirements.txt
     - name: vermin (Spack's Core)
       run: vermin --backport importlib --backport argparse --violations --backport typing -t=3.6- -vvv lib/spack/spack/ lib/spack/llnl/ bin/
     - name: vermin (Repositories)
@@ -34,31 +35,30 @@ jobs:
   style:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
       with:
         fetch-depth: 0
-    - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236
       with:
-        python-version: '3.13'
+        python-version: '3.11'
         cache: 'pip'
     - name: Install Python packages
       run: |
         pip install --upgrade pip setuptools
-        pip install -r .github/workflows/requirements/style/requirements.txt
+        pip install -r .github/workflows/style/requirements.txt
     - name: Setup git configuration
       run: |
         # Need this for the git tests to succeed.
         git --version
-        . .github/workflows/bin/setup_git.sh
+        . .github/workflows/setup_git.sh
     - name: Run style tests
       run: |
           share/spack/qa/run-style-tests
   audit:
     uses: ./.github/workflows/audit.yaml
-    secrets: inherit
     with:
       with_coverage: ${{ inputs.with_coverage }}
-      python_version: '3.13'
+      python_version: '3.11'
   # Check that spack can bootstrap the development environment on Python 3.6 - RHEL8
   bootstrap-dev-rhel8:
     runs-on: ubuntu-latest
@@ -69,13 +69,13 @@ jobs:
           dnf install -y \
               bzip2 curl file gcc-c++ gcc gcc-gfortran git gnupg2 gzip \
               make patch tcl unzip which xz
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
+      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
       - name: Setup repo and non-root user
         run: |
           git --version
-          git config --global --add safe.directory '*'
+          git config --global --add safe.directory /__w/spack/spack
           git fetch --unshallow
-          . .github/workflows/bin/setup_git.sh
+          . .github/workflows/setup_git.sh
           useradd spack-test
           chown -R spack-test .
       - name: Bootstrap Spack development environment
@@ -84,23 +84,5 @@ jobs:
           source share/spack/setup-env.sh
           spack debug report
           spack -d bootstrap now --dev
-          spack -d style -t black
+          spack style -t black
           spack unit-test -V
-
-  # Further style checks from pylint
-  pylint:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-        with:
-          fetch-depth: 0
-      - uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b
-        with:
-          python-version: '3.13'
-          cache: 'pip'
-      - name: Install Python packages
-        run: |
-          pip install --upgrade pip setuptools pylint
-      - name: Pylint (Spack Core)
-        run: |
-          pylint -j 4 --disable=all --enable=unspecified-encoding --ignore-paths=lib/spack/external lib

.github/workflows/windows_python.yml

@@ -0,0 +1,79 @@
+name: windows
+
+on:
+  workflow_call:
+
+concurrency:
+  group: windows-${{github.ref}}-${{github.event.pull_request.number || github.run_number}}
+  cancel-in-progress: true
+
+defaults:
+  run:
+    shell:
+     powershell Invoke-Expression -Command "./share/spack/qa/windows_test_setup.ps1"; {0}
+jobs:
+  unit-tests:
+    runs-on: windows-latest
+    steps:
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+      with:
+        fetch-depth: 0
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236
+      with:
+        python-version: 3.9
+    - name: Install Python packages
+      run: |
+          python -m pip install --upgrade pip pywin32 setuptools pytest-cov clingo
+    - name: Create local develop
+      run: |
+        ./.github/workflows/setup_git.ps1
+    - name: Unit Test
+      run: |
+        spack unit-test -x --verbose --cov --cov-config=pyproject.toml --ignore=lib/spack/spack/test/cmd
+        ./share/spack/qa/validate_last_exit.ps1
+        coverage combine -a
+        coverage xml
+    - uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d
+      with:
+        flags: unittests,windows
+  unit-tests-cmd:
+    runs-on: windows-latest
+    steps:
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+      with:
+        fetch-depth: 0
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236
+      with:
+        python-version: 3.9
+    - name: Install Python packages
+      run: |
+          python -m pip install --upgrade pip pywin32 setuptools coverage pytest-cov clingo
+    - name: Create local develop
+      run: |
+        ./.github/workflows/setup_git.ps1
+    - name: Command Unit Test
+      run: |
+        spack unit-test -x --verbose --cov --cov-config=pyproject.toml lib/spack/spack/test/cmd
+        ./share/spack/qa/validate_last_exit.ps1
+        coverage combine -a
+        coverage xml
+    - uses: codecov/codecov-action@eaaf4bedf32dbdc6b720b63067d99c4d77d6047d
+      with:
+        flags: unittests,windows
+  build-abseil:
+    runs-on: windows-latest
+    steps:
+    - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
+      with:
+        fetch-depth: 0
+    - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236
+      with:
+        python-version: 3.9
+    - name: Install Python packages
+      run: |
+          python -m pip install --upgrade pip pywin32 setuptools coverage
+    - name: Build Test
+      run: |
+        spack compiler find
+        spack -d external find cmake ninja
+        spack -d install abseil-cpp

.readthedocs.yml

@@ -14,26 +14,3 @@ sphinx:
 python:
   install:
     - requirements: lib/spack/docs/requirements.txt
-
-search:
-  ranking:
-    spack.html: -10
-    spack.*.html: -10
-    llnl.html: -10
-    llnl.*.html: -10
-    _modules/*: -10
-    command_index.html: -9
-    basic_usage.html: 5
-    configuration.html: 5
-    config_yaml.html: 5
-    packages_yaml.html: 5
-    build_settings.html: 5
-    environments.html: 5
-    containers.html: 5
-    mirrors.html: 5
-    module_file_support.html: 5
-    repositories.html: 5
-    binary_caches.html: 5
-    chain.html: 5
-    pipelines.html: 5
-    packaging_guide.html: 5

CHANGELOG.md

@@ -1,385 +1,3 @@
-# v0.22.2 (2024-09-21)
-
-## Bugfixes
-- Forward compatibility with Spack 0.23 packages with language dependencies (#45205, #45191)
-- Forward compatibility with `urllib` from Python 3.12.6+ (#46453, #46483)
-- Bump vendored `archspec` for better aarch64 support (#45721, #46445)
-- Support macOS Sequoia (#45018, #45127)
-- Fix regression in `{variants.X}` and `{variants.X.value}` format strings (#46206)
-- Ensure shell escaping of environment variable values in load and activate commands (#42780)
-- Fix an issue where `spec[pkg]` considers specs outside the current DAG (#45090)
-- Do not halt concretization on unknown variants in externals (#45326)
-- Improve validation of `develop` config section (#46485)
-- Explicitly disable `ccache` if turned off in config, to avoid cache pollution (#45275)
-- Improve backwards compatibility in `include_concrete` (#45766)
-- Fix issue where package tags were sometimes repeated (#45160)
-- Make `setup-env.sh` "sourced only" by dropping execution bits (#45641)
-- Make certain source/binary fetch errors recoverable instead of a hard error (#45683)
-- Remove debug statements in package hash computation (#45235)
-- Remove redundant clingo warnings (#45269)
-- Remove hard-coded layout version (#45645)
-- Do not initialize previous store state in `use_store` (#45268)
-- Docs improvements (#46475)
-
-## Package updates
-- `chapel` major update (#42197, #44931, #45304)
-
-# v0.22.1 (2024-07-04)
-
-## Bugfixes
-- Fix reuse of externals on Linux (#44316)
-- Ensure parent gcc-runtime version >= child (#44834, #44870)
-- Ensure the latest gcc-runtime is rpath'ed when multiple exist among link deps (#44219)
-- Improve version detection of glibc (#44154)
-- Improve heuristics for solver (#44893, #44976, #45023)
-- Make strong preferences override reuse (#44373)
-- Reduce verbosity when C compiler is missing (#44182)
-- Make missing ccache executable an error when required (#44740)
-- Make every environment view containing `python` a `venv` (#44382)
-- Fix external detection for compilers with os but no target (#44156)
-- Fix version optimization for roots (#44272)
-- Handle common implementations of pagination of tags in OCI build caches (#43136)
-- Apply fetched patches to develop specs (#44950)
-- Avoid Windows wrappers for filesystem utilities on non-Windows (#44126)
-- Fix issue with long filenames in build caches on Windows (#43851)
-- Fix formatting issue in `spack audit` (#45045)
-- CI fixes (#44582, #43965, #43967, #44279, #44213)
-
-## Package updates
-- protobuf: fix 3.4:3.21 patch checksum (#44443)
-- protobuf: update hash for patch needed when="@3.4:3.21" (#44210)
-- git: bump v2.39 to 2.45; deprecate unsafe versions (#44248)
-- gcc: use -rpath {rpath_dir} not -rpath={rpath dir} (#44315)
-- Remove mesa18 and libosmesa (#44264)
-- Enforce consistency of `gl` providers (#44307)
-- Require libiconv for iconv (#44335, #45026).
-  Notice that glibc/musl also provide iconv, but are not guaranteed to be
-  complete. Set `packages:iconv:require:[glibc]` to restore the old behavior.
-- py-matplotlib: qualify when to do a post install (#44191)
-- rust: fix v1.78.0 instructions (#44127)
-- suite-sparse: improve setting of the `libs` property (#44214)
-- netlib-lapack: provide blas and lapack together (#44981)
-
-# v0.22.0 (2024-05-12)
-
-`v0.22.0` is a major feature release.
-
-## Features in this release
-
-1. **Compiler dependencies**
-
-    We are in the process of making compilers proper dependencies in Spack, and a number
-    of changes in `v0.22` support that effort. You may notice nodes in your dependency
-    graphs for compiler runtime libraries like `gcc-runtime` or `libgfortran`, and you
-    may notice that Spack graphs now include `libc`. We've also begun moving compiler
-    configuration from `compilers.yaml` to `packages.yaml` to make it consistent with
-    other externals. We are trying to do this with the least disruption possible, so
-    your existing `compilers.yaml` files should still work. We expect to be done with
-    this transition by the `v0.23` release in November.
-
-    * #41104: Packages compiled with `%gcc` on Linux, macOS and FreeBSD now depend on a
-      new package `gcc-runtime`, which contains a copy of the shared compiler runtime
-      libraries. This enables gcc runtime libraries to be installed and relocated when
-      using a build cache. When building minimal Spack-generated container images it is
-      no longer necessary to install libgfortran, libgomp etc. using the system package
-      manager.
-
-    * #42062: Packages compiled with `%oneapi` now depend on a new package
-      `intel-oneapi-runtime`. This is similar to `gcc-runtime`, and the runtimes can
-      provide virtuals and compilers can inject dependencies on virtuals into compiled
-      packages. This allows us to model library soname compatibility and allows
-      compilers like `%oneapi` to provide virtuals like `sycl` (which can also be
-      provided by standalone libraries). Note that until we have an agreement in place
-      with intel, Intel packages are marked `redistribute(source=False, binary=False)`
-      and must be downloaded outside of Spack.
-
-    * #43272: changes to the optimization criteria of the solver improve the hit-rate of
-      buildcaches by a fair amount. The solver more relaxed compatibility rules and will
-      not try to strictly match compilers or targets of reused specs. Users can still
-      enforce the previous strict behavior with `require:` sections in `packages.yaml`.
-      Note that to enforce correct linking, Spack will *not* reuse old `%gcc` and
-      `%oneapi` specs that do not have the runtime libraries as a dependency.
-
-    * #43539: Spack will reuse specs built with compilers that are *not* explicitly
-      configured in `compilers.yaml`. Because we can now keep runtime libraries in build
-      cache, we do not require you to also have a local configured compiler to *use* the
-      runtime libraries. This improves reuse in buildcaches and avoids conflicts with OS
-      updates that happen underneath Spack.
-
-    * #43190: binary compatibility on `linux` is now based on the `libc` version,
-      instead of on the `os` tag. Spack builds now detect the host `libc` (`glibc` or
-      `musl`) and add it as an implicit external node in the dependency graph. Binaries
-      with a `libc` with the same name and a version less than or equal to that of the
-      detected `libc` can be reused. This is only on `linux`, not `macos` or `Windows`.
-
-    * #43464: each package that can provide a compiler is now detectable using `spack
-      external find`. External packages defining compiler paths are effectively used as
-      compilers, and `spack external find -t compiler` can be used as a substitute for
-      `spack compiler find`. More details on this transition are in
-      [the docs](https://spack.readthedocs.io/en/latest/getting_started.html#manual-compiler-configuration)
-
-2. **Improved `spack find` UI for Environments**
-
-   If you're working in an enviroment, you likely care about:
-
-   * What are the roots
-   * Which ones are installed / not installed
-   * What's been added that still needs to be concretized
-
-    We've tweaked `spack find` in environments to show this information much more
-    clearly. Installation status is shown next to each root, so you can see what is
-    installed. Roots are also shown in bold in the list of installed packages. There is
-    also a new option for `spack find -r` / `--only-roots` that will only show env
-    roots, if you don't want to look at all the installed specs.
-
-    More details in #42334.
-
-3. **Improved command-line string quoting**
-
-   We are making some breaking changes to how Spack parses specs on the CLI in order to
-   respect shell quoting instead of trying to fight it. If you (sadly) had to write
-   something like this on the command line:
-
-    ```
-    spack install zlib cflags=\"-O2 -g\"
-    ```
-
-    That will now result in an error, but you can now write what you probably expected
-    to work in the first place:
-
-    ```
-    spack install zlib cflags="-O2 -g"
-    ```
-
-    Quoted can also now include special characters, so you can supply flags like:
-
-    ```
-    spack intall zlib ldflags='-Wl,-rpath=$ORIGIN/_libs'
-    ```
-
-    To reduce ambiguity in parsing, we now require that you *not* put spaces around `=`
-    and `==` when for flags or variants. This would not have broken before but will now
-    result in an error:
-
-    ```
-    spack install zlib cflags = "-O2 -g"
-    ```
-
-    More details and discussion in #30634.
-
-4. **Revert default `spack install` behavior to `--reuse`**
-
-   We changed the default concretizer behavior from `--reuse` to `--reuse-deps` in
-   #30990 (in `v0.20`), which meant that *every* `spack install` invocation would
-   attempt to build a new version of the requested package / any environment roots.
-   While this is a common ask for *upgrading* and for *developer* workflows, we don't
-   think it should be the default for a package manager.
-
-   We are going to try to stick to this policy:
-   1. Prioritize reuse and build as little as possible by default.
-   2. Only upgrade or install duplicates if they are explicitly asked for, or if there
-      is a known security issue that necessitates an upgrade.
-
-   With the install command you now have three options:
-
-   * `--reuse` (default): reuse as many existing installations as possible.
-   * `--reuse-deps` / `--fresh-roots`: upgrade (freshen) roots but reuse dependencies if possible.
-   * `--fresh`: install fresh versions of requested packages (roots) and their dependencies.
-
-   We've also introduced `--fresh-roots` as an alias for `--reuse-deps` to make it more clear
-   that it may give you fresh versions. More details in #41302 and #43988.
-
-5. **More control over reused specs**
-
-   You can now control which packages to reuse and how. There is a new
-   `concretizer:reuse` config option, which accepts the following properties:
-
-   - `roots`: `true` to reuse roots, `false` to reuse just dependencies
-   - `exclude`: list of constraints used to select which specs *not* to reuse
-   - `include`: list of constraints used to select which specs *to* reuse
-   - `from`: list of sources for reused specs (some combination of `local`,
-     `buildcache`, or `external`)
-
-   For example, to reuse only specs compiled with GCC, you could write:
-
-   ```yaml
-   concretizer:
-      reuse:
-        roots: true
-        include:
-        - "%gcc"
-   ```
-
-   Or, if `openmpi` must be used from externals, and it must be the only external used:
-
-   ```yaml
-   concretizer:
-     reuse:
-       roots: true
-       from:
-       - type: local
-         exclude: ["openmpi"]
-       - type: buildcache
-         exclude: ["openmpi"]
-       - type: external
-         include: ["openmpi"]
-   ```
-
-6. **New `redistribute()` directive**
-
-   Some packages can't be redistributed in source or binary form. We need an explicit
-   way to say that in a package.
-
-   Now there is a `redistribute()` directive so that package authors can write:
-
-   ```python
-   class MyPackage(Package):
-       redistribute(source=False, binary=False)
-   ```
-
-   Like other directives, this works with `when=`:
-
-   ```python
-   class MyPackage(Package):
-       # 12.0 and higher are proprietary
-       redistribute(source=False, binary=False, when="@12.0:")
-
-       # can't redistribute when we depend on some proprietary dependency
-       redistribute(source=False, binary=False, when="^proprietary-dependency")
-   ```
-
-    More in #20185.
-
-7. **New `conflict:` and `prefer:` syntax for package preferences**
-
-   Previously, you could express conflicts and preferences in `packages.yaml` through
-   some contortions with `require:`:
-
-    ```yaml
-    packages:
-      zlib-ng:
-        require:
-        - one_of: ["%clang", "@:"]   # conflict on %clang
-        - any_of: ["+shared", "@:"]  # strong preference for +shared
-    ```
-
-    You can now use `require:` and `prefer:` for a much more readable configuration:
-
-    ```yaml
-    packages:
-      zlib-ng:
-        conflict:
-        - "%clang"
-        prefer:
-        - "+shared"
-    ```
-
-    See [the documentation](https://spack.readthedocs.io/en/latest/packages_yaml.html#conflicts-and-strong-preferences)
-    and #41832 for more details.
-
-8. **`include_concrete` in environments**
-
-   You may want to build on the *concrete* contents of another environment without
-   changing that environment.  You can now include the concrete specs from another
-   environment's `spack.lock` with `include_concrete`:
-
-   ```yaml
-      spack:
-        specs: []
-        concretizer:
-            unify: true
-        include_concrete:
-        - /path/to/environment1
-        - /path/to/environment2
-   ```
-
-   Now, when *this* environment is concretized, it will bring in the already concrete
-   specs from `environment1` and `environment2`, and build on top of them without
-   changing them. This is useful if you have phased deployments, where old deployments
-   should not be modified but you want to use as many of them as possible. More details
-   in #33768.
-
-9. **`python-venv` isolation**
-
-   Spack has unique requirements for Python because it:
-    1. installs every package in its own independent directory, and
-    2. allows users to register *external* python installations.
-
-   External installations may contain their own installed packages that can interfere
-   with Spack installations, and some distributions (Debian and Ubuntu) even change the
-   `sysconfig` in ways that alter the installation layout of installed Python packages
-   (e.g., with the addition of a `/local` prefix on Debian or Ubuntu). To isolate Spack
-   from these and other issues, we now insert a small `python-venv` package in between
-   `python` and packages that need to install Python code. This isolates Spack's build
-   environment, isolates Spack from any issues with an external python, and resolves a
-   large number of issues we've had with Python installations.
-
-   See #40773 for further details.
-
-## New commands, options, and directives
-
-* Allow packages to be pushed to build cache after install from source (#42423)
-* `spack develop`: stage build artifacts in same root as non-dev builds #41373
-  * Don't delete `spack develop` build artifacts after install (#43424)
-* `spack find`: add options for local/upstream only (#42999)
-* `spack logs`: print log files for packages (either partially built or installed) (#42202)
-* `patch`: support reversing patches (#43040)
-* `develop`: Add -b/--build-directory option to set build_directory package attribute (#39606)
-* `spack list`: add `--namesapce` / `--repo` option (#41948)
-* directives: add `checked_by` field to `license()`, add some license checks
-* `spack gc`: add options for environments and build dependencies (#41731)
-* Add `--create` to `spack env activate` (#40896)
-
-## Performance improvements
-
-* environment.py: fix excessive re-reads (#43746)
-* ruamel yaml: fix quadratic complexity bug  (#43745)
-* Refactor to improve `spec format` speed (#43712)
-* Do not acquire a write lock on the env post install if no views (#43505)
-* asp.py: fewer calls to `spec.copy()` (#43715)
-* spec.py: early return in `__str__`
-* avoid `jinja2` import at startup unless needed (#43237)
-
-## Other new features of note
-
-* `archspec`: update to `v0.2.4`: support for Windows, bugfixes for `neoverse-v1` and
-  `neoverse-v2` detection.
-* `spack config get`/`blame`: with no args, show entire config
-* `spack env create <env>`: dir if dir-like (#44024)
-* ASP-based solver: update os compatibility for macOS (#43862)
-* Add handling of custom ssl certs in urllib ops (#42953)
-* Add ability to rename environments (#43296)
-* Add config option and compiler support to reuse across OS's (#42693)
-* Support for prereleases (#43140)
-* Only reuse externals when configured (#41707)
-* Environments: Add support for including views (#42250)
-
-## Binary caches
-* Build cache: make signed/unsigned a mirror property (#41507)
-* tools stack
-
-## Removals, deprecations, and syntax changes
-* remove `dpcpp` compiler and package (#43418)
-* spack load: remove --only argument (#42120)
-
-## Notable Bugfixes
-* repo.py: drop deleted packages from provider cache (#43779)
-* Allow `+` in module file names (#41999)
-* `cmd/python`: use runpy to allow multiprocessing in scripts (#41789)
-* Show extension commands with spack -h (#41726)
-* Support environment variable expansion inside module projections (#42917)
-* Alert user to failed concretizations (#42655)
-* shell: fix zsh color formatting for PS1 in environments (#39497)
-* spack mirror create --all: include patches (#41579)
-
-## Spack community stats
-
-* 7,994 total packages; 525 since `v0.21.0`
-    * 178 new Python packages, 5 new R packages
-* 358 people contributed to this release
-    * 344 committers to packages
-    * 45 committers to core
-
 # v0.21.3 (2024-10-02)
 
 ## Bugfixes

CITATION.cff

@@ -31,17 +31,13 @@ type: software
 message: "If you are referencing Spack in a publication, please cite the paper below."
 title: "The Spack Package Manager: Bringing Order to HPC Software Chaos"
 abstract: >-
-  Large HPC centers spend considerable time supporting software for thousands of users, but the
-  complexity of HPC software is quickly outpacing the capabilities of existing software management
-  tools. Scientific applications require specific versions of compilers, MPI, and other dependency
-  libraries, so using a single, standard software stack is infeasible. However, managing many
-  configurations is difficult because the configuration space is combinatorial in size. We
-  introduce Spack, a tool used at Lawrence Livermore National Laboratory to manage this complexity.
-  Spack provides a novel, re- cursive specification syntax to invoke parametric builds of packages
-  and dependencies. It allows any number of builds to coexist on the same system, and it ensures
-  that installed packages can find their dependencies, regardless of the environment. We show
-  through real-world use cases that Spack supports diverse and demanding applications, bringing
-  order to HPC software chaos.
+  Large HPC centers spend considerable time supporting software for thousands of users, but the complexity of HPC software is quickly outpacing the capabilities of existing software management tools.
+  Scientific applications require specific versions of compilers, MPI, and other dependency libraries, so using a single, standard software stack is infeasible.
+  However, managing many configurations is difficult because the configuration space is combinatorial in size.
+  We introduce Spack, a tool used at Lawrence Livermore National Laboratory to manage this complexity.
+  Spack provides a novel, re- cursive specification syntax to invoke parametric builds of packages and dependencies.
+  It allows any number of builds to coexist on the same system, and it ensures that installed packages can find their dependencies, regardless of the environment.
+  We show through real-world use cases that Spack supports diverse and demanding applications, bringing order to HPC software chaos.
 preferred-citation:
   title: "The Spack Package Manager: Bringing Order to HPC Software Chaos"
   type: conference-paper
@@ -75,7 +71,7 @@ preferred-citation:
       type: doi
       value: 10.1145/2807591.2807623
     - description: "The DOE Document Release Number of the work"
-      type: other
+      type: other 
       value: "LLNL-CONF-669890"
 authors:
   - family-names: "Gamblin"

COPYRIGHT

@@ -8,9 +8,8 @@ or http://www.apache.org/licenses/LICENSE-2.0) or the MIT license,
 Copyrights and patents in the Spack project are retained by contributors.
 No copyright assignment is required to contribute to Spack.
 
-Spack was originally developed in 2013 by Lawrence Livermore National
-Security, LLC. It was originally distributed under the LGPL-2.1 license.
-Consent from contributors to relicense to Apache-2.0/MIT is documented at
+Spack was originally distributed under the LGPL-2.1 license. Consent from
+contributors to relicense to Apache-2.0/MIT is documented at
 https://github.com/spack/spack/issues/9137.
 
 
@@ -103,6 +102,6 @@ PackageName: sbang
 PackageHomePage: https://github.com/spack/sbang
 PackageLicenseDeclared: Apache-2.0 OR MIT
 
-PackageName: typing_extensions
-PackageHomePage: https://pypi.org/project/typing-extensions/
-PackageLicenseDeclared: Python-2.0
+PackageName: six
+PackageHomePage: https://pypi.python.org/pypi/six
+PackageLicenseDeclared: MIT

LICENSE-MIT

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) Spack Project Developers.
+Copyright (c) 2013-2023 LLNS, LLC and other Spack Project Developers.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,38 +1,17 @@
-<div align="left">
+# <img src="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo.svg" width="64" valign="middle" alt="Spack"/> Spack
 
-<h2>
-<picture>
-  <source media="(prefers-color-scheme: dark)" srcset="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo-white-text.svg" width="250">
-  <source media="(prefers-color-scheme: light)" srcset="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo-text.svg" width="250">
-  <img alt="Spack" src="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo-text.svg" width="250">
-</picture>
-
-<br>
-<br clear="all">
-
-<a href="https://github.com/spack/spack/actions/workflows/ci.yml"><img src="https://github.com/spack/spack/workflows/ci/badge.svg" alt="CI Status"></a>
-<a href="https://github.com/spack/spack/actions/workflows/bootstrapping.yml"><img src="https://github.com/spack/spack/actions/workflows/bootstrap.yml/badge.svg" alt="Bootstrap Status"></a>
-<a href="https://github.com/spack/spack/actions/workflows/build-containers.yml"><img src="https://github.com/spack/spack/actions/workflows/build-containers.yml/badge.svg" alt="Containers Status"></a>
-<a href="https://spack.readthedocs.io"><img src="https://readthedocs.org/projects/spack/badge/?version=latest" alt="Documentation Status"></a>
-<a href="https://codecov.io/gh/spack/spack"><img src="https://codecov.io/gh/spack/spack/branch/develop/graph/badge.svg" alt="Code coverage"/></a>
-<a href="https://slack.spack.io"><img src="https://slack.spack.io/badge.svg" alt="Slack"/></a>
-<a href="https://matrix.to/#/#spack-space:matrix.org"><img src="https://img.shields.io/matrix/spack-space%3Amatrix.org?label=matrix" alt="Matrix"/></a>
-
-</h2>
-
-**[Getting Started] &nbsp; • &nbsp; [Config] &nbsp; • &nbsp; [Community] &nbsp; • &nbsp; [Contributing] &nbsp; • &nbsp; [Packaging Guide]**
-
-[Getting Started]: https://spack.readthedocs.io/en/latest/getting_started.html
-[Config]: https://spack.readthedocs.io/en/latest/configuration.html
-[Community]: #community
-[Contributing]: https://spack.readthedocs.io/en/latest/contribution_guide.html
-[Packaging Guide]: https://spack.readthedocs.io/en/latest/packaging_guide.html
-
-</div>
+[![Unit Tests](https://github.com/spack/spack/workflows/linux%20tests/badge.svg)](https://github.com/spack/spack/actions)
+[![Bootstrapping](https://github.com/spack/spack/actions/workflows/bootstrap.yml/badge.svg)](https://github.com/spack/spack/actions/workflows/bootstrap.yml)
+[![codecov](https://codecov.io/gh/spack/spack/branch/develop/graph/badge.svg)](https://codecov.io/gh/spack/spack)
+[![Containers](https://github.com/spack/spack/actions/workflows/build-containers.yml/badge.svg)](https://github.com/spack/spack/actions/workflows/build-containers.yml)
+[![Read the Docs](https://readthedocs.org/projects/spack/badge/?version=latest)](https://spack.readthedocs.io)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Slack](https://slack.spack.io/badge.svg)](https://slack.spack.io)
+[![Matrix](https://img.shields.io/matrix/spack-space%3Amatrix.org?label=Matrix)](https://matrix.to/#/#spack-space:matrix.org)
 
 Spack is a multi-platform package manager that builds and installs
 multiple versions and configurations of software. It works on Linux,
-macOS, Windows, and many supercomputers. Spack is non-destructive: installing a
+macOS, and many supercomputers. Spack is non-destructive: installing a
 new version of a package does not break existing installations, so many
 configurations of the same package can coexist.
 
@@ -46,18 +25,13 @@ See the
 [Feature Overview](https://spack.readthedocs.io/en/latest/features.html)
 for examples and highlights.
 
-To install spack and your first package, make sure you have Python & Git.
+To install spack and your first package, make sure you have Python.
 Then:
 
-    $ git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git
+    $ git clone -c feature.manyFiles=true https://github.com/spack/spack.git
     $ cd spack/bin
     $ ./spack install zlib
 
-> [!TIP]
-> `-c feature.manyFiles=true` improves git's performance on repositories with 1,000+ files.
->
-> `--depth=2` prunes the git history to reduce the size of the Spack installation.
-
 Documentation
 ----------------
 
@@ -70,7 +44,7 @@ Tutorial
 ----------------
 
 We maintain a
-[**hands-on tutorial**](https://spack-tutorial.readthedocs.io/).
+[**hands-on tutorial**](https://spack.readthedocs.io/en/latest/tutorial.html).
 It covers basic to advanced usage, packaging, developer features, and large HPC
 deployments.  You can do all of the exercises on your own laptop using a
 Docker container.
@@ -92,11 +66,10 @@ Resources:
 * **Matrix space**: [#spack-space:matrix.org](https://matrix.to/#/#spack-space:matrix.org):
   [bridged](https://github.com/matrix-org/matrix-appservice-slack#matrix-appservice-slack) to Slack.
 * [**Github Discussions**](https://github.com/spack/spack/discussions):
-  for Q&A and discussions. Note the pinned discussions for announcements.
-* **X**: [@spackpm](https://twitter.com/spackpm). Be sure to
+  not just for discussions, but also Q&A.
+* **Mailing list**: [groups.google.com/d/forum/spack](https://groups.google.com/d/forum/spack)
+* **Twitter**: [@spackpm](https://twitter.com/spackpm). Be sure to
   `@mention` us!
-* **Mailing list**: [groups.google.com/d/forum/spack](https://groups.google.com/d/forum/spack):
-  only for announcements. Please use other venues for discussions.
 
 Contributing
 ------------------------

bin/haspywin.py

@@ -1,4 +1,5 @@
-# Copyright Spack Project Developers. See COPYRIGHT file for details.
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 import subprocess

bin/sbang

@@ -1,6 +1,7 @@
 #!/bin/sh
 #
-# Copyright sbang project developers. See COPYRIGHT file for details.
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# sbang project developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

bin/spack

@@ -1,7 +1,8 @@
 #!/bin/sh
 # -*- python -*-
 #
-# Copyright Spack Project Developers. See COPYRIGHT file for details.
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -25,6 +26,7 @@ exit 1
 # The code above runs this file with our preferred python interpreter.
 
 import os
+import os.path
 import sys
 
 min_python3 = (3, 6)

bin/spack-python

@@ -1,6 +1,7 @@
 #!/bin/sh
 #
-# Copyright Spack Project Developers. See COPYRIGHT file for details.
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -21,4 +22,4 @@
 #
 # This is compatible across platforms.
 #
-exec spack python "$@"
+exec /usr/bin/env spack python "$@"

bin/spack.bat

@@ -1,4 +1,5 @@
-:: Copyright Spack Project Developers. See COPYRIGHT file for details.
+:: Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+:: Spack Project Developers. See the top-level COPYRIGHT file for details.
 ::
 :: SPDX-License-Identifier: (Apache-2.0 OR MIT)
 ::#######################################################################
@@ -187,27 +188,25 @@ if NOT "%_sp_args%"=="%_sp_args:--help=%" (
 goto :end_switch
 
 :case_load
-if NOT defined _sp_args (
-   exit /B 0
-)
-
-:: If args contain --bat, or -h/--help: just execute.
-if NOT "%_sp_args%"=="%_sp_args:--help=%" (
-    goto :default_case
-) else if NOT "%_sp_args%"=="%_sp_args:-h=%" (
-    goto :default_case
-) else if NOT "%_sp_args%"=="%_sp_args:--bat=%" (
-    goto :default_case
-) else if NOT "%_sp_args%"=="%_sp_args:--list=%" (
-    goto :default_case
+:: If args contain --sh, --csh, or -h/--help: just execute.
+if defined _sp_args (
+    if NOT "%_sp_args%"=="%_sp_args:--help=%" (
+        goto :default_case
+    ) else if NOT "%_sp_args%"=="%_sp_args:-h=%" (
+        goto :default_case
+    ) else if NOT "%_sp_args%"=="%_sp_args:--bat=%" (
+        goto :default_case
+    )
 )
 
 for /f "tokens=* USEBACKQ" %%I in (
-    `python "%spack%" %_sp_flags% %_sp_subcommand% --bat %_sp_args%`
-    ) do %%I
+    `python "%spack%" %_sp_flags% %_sp_subcommand% --bat %_sp_args%`) do %%I
 
 goto :end_switch
 
+:case_unload
+goto :case_load
+
 :default_case
 python "%spack%" %_sp_flags% %_sp_subcommand% %_sp_args%
 goto :end_switch

bin/spack.ps1

@@ -1,4 +1,5 @@
-#  Copyright Spack Project Developers. See COPYRIGHT file for details.
+#  Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+#  Spack Project Developers. See the top-level COPYRIGHT file for details.
 
 #  SPDX-License-Identifier: (Apache-2.0 OR MIT)
 # #######################################################################
@@ -143,5 +144,3 @@ switch($SpackSubCommand)
     "unload" {Invoke-SpackLoad}
     default  {python "$Env:SPACK_ROOT/bin/spack" $SpackCMD_params $SpackSubCommand $SpackSubCommandArgs}
 }
-
-exit $LASTEXITCODE

bin/spack_cmd.bat

@@ -1,11 +1,71 @@
 @ECHO OFF
+setlocal EnableDelayedExpansion
 :: (c) 2021 Lawrence Livermore National Laboratory
 :: To use this file independently of Spack's installer, execute this script in its directory, or add the
 :: associated bin directory to your PATH. Invoke to launch Spack Shell.
 ::
 :: source_dir/spack/bin/spack_cmd.bat
 ::
+pushd %~dp0..
+set SPACK_ROOT=%CD%
+pushd %CD%\..
+set spackinstdir=%CD%
+popd
 
-call "%~dp0..\share\spack\setup-env.bat"
-pushd %SPACK_ROOT%
-%comspec% /K
+
+:: Check if Python is on the PATH
+if not defined python_pf_ver (
+(for /f "delims=" %%F in ('where python.exe') do (
+                                                    set "python_pf_ver=%%F"
+                                                    goto :found_python
+                                                  ) ) 2> NUL
+)
+:found_python
+if not defined python_pf_ver (
+    :: If not, look for Python from the Spack installer
+    :get_builtin
+    (for /f "tokens=*" %%g in ('dir /b /a:d "!spackinstdir!\Python*"') do (
+        set "python_ver=%%g")) 2> NUL
+
+    if not defined python_ver (
+        echo Python was not found on your system.
+        echo Please install Python or add Python to your PATH.
+    ) else (
+        set "py_path=!spackinstdir!\!python_ver!"
+        set "py_exe=!py_path!\python.exe"
+    )
+    goto :exitpoint
+) else (
+    :: Python is already on the path
+    set "py_exe=!python_pf_ver!"
+    (for /F "tokens=* USEBACKQ" %%F in (
+        `"!py_exe!" --version`) do (set "output=%%F")) 2>NUL
+    if not "!output:Microsoft Store=!"=="!output!" goto :get_builtin
+    goto :exitpoint
+)
+:exitpoint
+
+set "PATH=%SPACK_ROOT%\bin\;%PATH%"
+if defined py_path (
+    set "PATH=%py_path%;%PATH%"
+)
+
+if defined py_exe (
+    "%py_exe%" "%SPACK_ROOT%\bin\haspywin.py"
+)
+
+set "EDITOR=notepad"
+
+DOSKEY spacktivate=spack env activate $*
+
+@echo **********************************************************************
+@echo ** Spack Package Manager
+@echo **********************************************************************
+
+IF "%1"=="" GOTO CONTINUE
+set
+GOTO:EOF
+
+:continue
+set PROMPT=[spack] %PROMPT%
+%comspec% /k

bin/spack_pwsh.ps1

@@ -1,4 +1,5 @@
-# Copyright Spack Project Developers. See COPYRIGHT file for details.
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

etc/spack/defaults/bootstrap.yaml

@@ -9,15 +9,15 @@ bootstrap:
   # may not be able to bootstrap all the software that Spack needs,
   # depending on its type.
   sources:
-  - name: github-actions-v0.6
-    metadata: $spack/share/spack/bootstrap/github-actions-v0.6
-  - name: github-actions-v0.5
+  - name: 'github-actions-v0.5'
     metadata: $spack/share/spack/bootstrap/github-actions-v0.5
-  - name: spack-install
+  - name: 'github-actions-v0.4'
+    metadata: $spack/share/spack/bootstrap/github-actions-v0.4
+  - name: 'spack-install'
     metadata: $spack/share/spack/bootstrap/spack-install
   trusted:
     # By default we trust bootstrapping from sources and from binaries
     # produced on Github via the workflow
-    github-actions-v0.6: true
     github-actions-v0.5: true
+    github-actions-v0.4: true
     spack-install: true

etc/spack/defaults/concretizer.yaml

@@ -15,7 +15,7 @@ concretizer:
   # as possible, rather than building. If `false`, we'll always give you a fresh
   # concretization. If `dependencies`, we'll only reuse dependencies but
   # give you a fresh concretization for your root specs.
-  reuse: true
+  reuse: dependencies
   # Options that tune which targets are considered for concretization. The
   # concretization process is very sensitive to the number targets, and the time
   # needed to reach a solution increases noticeably with the number of targets
@@ -39,27 +39,6 @@ concretizer:
   # Option to deal with possible duplicate nodes (i.e. different nodes from the same package) in the DAG.
   duplicates:
     # "none": allows a single node for any package in the DAG.
-    # "minimal": allows the duplication of 'build-tools' nodes only
-    # (e.g. py-setuptools, cmake etc.)
+    # "minimal": allows the duplication of 'build-tools' nodes only (e.g. py-setuptools, cmake etc.)
     # "full" (experimental): allows separation of the entire build-tool stack (e.g. the entire "cmake" subDAG)
     strategy: minimal
-  # Option to specify compatibility between operating systems for reuse of compilers and packages
-  # Specified as a key: [list] where the key is the os that is being targeted, and the list contains the OS's
-  # it can reuse. Note this is a directional compatibility so mutual compatibility between two OS's
-  # requires two entries i.e. os_compatible: {sonoma: [monterey], monterey: [sonoma]}
-  os_compatible: {}
-
-  # Option to specify whether to support splicing. Splicing allows for
-  # the relinking of concrete package dependencies in order to better
-  # reuse already built packages with ABI compatible dependencies
-  splice:
-    explicit: []
-    automatic: false
-  # Maximum time, in seconds, allowed for the 'solve' phase. If set to 0, there is no time limit.
-  timeout: 0
-  # If set to true, exceeding the timeout will always result in a concretization error. If false,
-  # the best (suboptimal) model computed before the timeout is used.
-  #
-  # Setting this to false yields unreproducible results, so we advise to use that value only
-  # for debugging purposes (e.g. check which constraints can help Spack concretize faster).
-  error_on_timeout: true

etc/spack/defaults/config.yaml

@@ -101,12 +101,6 @@ config:
   verify_ssl: true
 
 
-  # This is where custom certs for proxy/firewall are stored.
-  # It can be a path or environment variable. To match ssl env configuration
-  # the default is the environment variable SSL_CERT_FILE
-  ssl_certs: $SSL_CERT_FILE
-
-
   # Suppress gpg warnings from binary package verification
   # Only suppresses warnings, gpg failure will still fail the install
   # Potential rationale to set True: users have already explicitly trusted the
@@ -115,6 +109,12 @@ config:
   suppress_gpg_warnings: false
 
 
+  # If set to true, Spack will attempt to build any compiler on the spec
+  # that is not already available. If set to False, Spack will only use
+  # compilers already configured in compilers.yaml
+  install_missing_compilers: false
+
+
   # If set to true, Spack will always check checksums after downloading
   # archives. If false, Spack skips the checksum step.
   checksum: true
@@ -164,6 +164,23 @@ config:
   # If set to true, Spack will use ccache to cache C compiles.
   ccache: false
 
+
+  # The concretization algorithm to use in Spack. Options are:
+  #
+  #   'clingo': Uses a logic solver under the hood to solve DAGs with full
+  #       backtracking and optimization for user preferences. Spack will
+  #       try to bootstrap the logic solver, if not already available.
+  #
+  #   'original': Spack's original greedy, fixed-point concretizer. This
+  #       algorithm can make decisions too early and will not backtrack
+  #       sufficiently for many specs. This will soon be deprecated in
+  #       favor of clingo.
+  #
+  # See `concretizer.yaml` for more settings you can fine-tune when
+  # using clingo.
+  concretizer: clingo
+
+
   # How long to wait to lock the Spack installation database. This lock is used
   # when Spack needs to manage its own package metadata and all operations are
   # expected to complete within the default time limit. The timeout should
@@ -194,12 +211,6 @@ config:
     # executables with many dependencies, in particular on slow filesystems.
     bind: false
 
-    # Controls the handling of missing dynamic libraries after installation.
-    # Options are ignore (default), warn, or error. If set to error, the
-    # installation fails if installed binaries reference dynamic libraries that
-    # are not found in their specified rpaths.
-    missing_library_policy: ignore
-
 
   # Set to 'false' to allow installation on filesystems that doesn't allow setgid bit
   # manipulation by unprivileged user (e.g. AFS)

etc/spack/defaults/cray/modules.yaml

@@ -0,0 +1,16 @@
+# -------------------------------------------------------------------------
+# This is the default configuration for Spack's module file generation.
+#
+# Settings here are versioned with Spack and are intended to provide
+# sensible defaults out of the box. Spack maintainers should edit this
+# file to keep it current.
+#
+# Users can override these settings by editing the following files.
+#
+# Per-spack-instance settings (overrides defaults):
+#   $SPACK_ROOT/etc/spack/modules.yaml
+#
+# Per-user settings (overrides default and site settings):
+#   ~/.spack/modules.yaml
+# -------------------------------------------------------------------------
+modules: {}

etc/spack/defaults/darwin/packages.yaml

@@ -19,6 +19,7 @@ packages:
     - apple-clang
     - clang
     - gcc
+    - intel
     providers:
       elf: [libelf]
       fuse: [macfuse]
@@ -49,4 +50,4 @@ packages:
     # Apple bundles libuuid in libsystem_c version 1353.100.2,
     # although the version number used here isn't critical
     - spec: apple-libuuid@1353.100.2
-      prefix: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
+      prefix: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

etc/spack/defaults/packages.yaml

@@ -15,20 +15,15 @@
 # -------------------------------------------------------------------------
 packages:
   all:
-    compiler: [gcc, clang, oneapi, xl, nag, fj, aocc]
+    compiler: [gcc, intel, pgi, clang, xl, nag, fj, aocc]
     providers:
       awk: [gawk]
-      armci: [armcimpi]
       blas: [openblas, amdblis]
-      c: [gcc]
-      cxx: [gcc]
       D: [ldc]
       daal: [intel-oneapi-daal]
       elf: [elfutils]
       fftw-api: [fftw, amdfftw]
       flame: [libflame, amdlibflame]
-      fortran: [gcc]
-      fortran-rt: [gcc-runtime, intel-oneapi-runtime]
       fuse: [libfuse]
       gl: [glx, osmesa]
       glu: [mesa-glu, openglu]
@@ -36,14 +31,12 @@ packages:
       go-or-gccgo-bootstrap: [go-bootstrap, gcc]
       iconv: [libiconv]
       ipp: [intel-oneapi-ipp]
-      java: [openjdk, jdk]
+      java: [openjdk, jdk, ibm-java]
       jpeg: [libjpeg-turbo, libjpeg]
       lapack: [openblas, amdlibflame]
-      libc: [glibc, musl]
-      libgfortran: [gcc-runtime]
-      libglx: [mesa+glx]
-      libifcore: [intel-oneapi-runtime]
+      libglx: [mesa+glx, mesa18+glx]
       libllvm: [llvm]
+      libosmesa: [mesa+osmesa, mesa18+osmesa]
       lua-lang: [lua, lua-luajit-openresty, lua-luajit]
       luajit: [lua-luajit-openresty, lua-luajit]
       mariadb-client: [mariadb-c-client, mariadb]
@@ -64,8 +57,6 @@ packages:
       tbb: [intel-tbb]
       unwind: [libunwind]
       uuid: [util-linux-uuid, libuuid]
-      wasi-sdk: [wasi-sdk-prebuilt]
-      xkbdata-api: [xkeyboard-config, xkbdata]
       xxd: [xxd-standalone, vim]
       yacc: [bison, byacc]
       ziglang: [zig]
@@ -73,27 +64,3 @@ packages:
     permissions:
       read: world
       write: user
-  cray-fftw:
-    buildable: false
-  cray-libsci:
-    buildable: false
-  cray-mpich:
-    buildable: false
-  cray-mvapich2:
-    buildable: false
-  cray-pmi:
-    buildable: false
-  egl:
-    buildable: false
-  essl:
-    buildable: false
-  fujitsu-mpi:
-    buildable: false
-  fujitsu-ssl2:
-    buildable: false
-  hpcx-mpi:
-    buildable: false
-  mpt:
-    buildable: false
-  spectrum-mpi:
-    buildable: false

etc/spack/defaults/windows/config.yaml

@@ -1,5 +1,6 @@
 config:
   locks: false
+  concretizer: clingo
   build_stage::
-    - '$user_cache_path/stage'
+    - '$spack/.staging'
   stage_name: '{name}-{version}-{hash:7}'

lib/spack/docs/_pygments/style.py

@@ -1,4 +1,5 @@
-# Copyright Spack Project Developers. See COPYRIGHT file for details.
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/_templates/layout.html

@@ -1,12 +0,0 @@
-{% extends "!layout.html" %}
-
-{%- block extrahead %} 
-  <!-- Google tag (gtag.js) -->
-  <script async src="https://www.googletagmanager.com/gtag/js?id=G-S0PQ7WV75K"></script>
-  <script>
-    window.dataLayer = window.dataLayer || [];
-    function gtag(){dataLayer.push(arguments);}
-    gtag('js', new Date());
-    gtag('config', 'G-S0PQ7WV75K');
-  </script>
-{% endblock %}

lib/spack/docs/basic_usage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -864,7 +865,7 @@ There are several different ways to use Spack packages once you have
 installed them. As you've seen, spack packages are installed into long
 paths with hashes, and you need a way to get them into your path. The
 easiest way is to use :ref:`spack load <cmd-spack-load>`, which is
-described in this section.
+described in the next section.
 
 Some more advanced ways to use Spack packages include:
 
@@ -958,86 +959,7 @@ use ``spack find --loaded``.
 You can also use ``spack load --list`` to get the same output, but it
 does not have the full set of query options that ``spack find`` offers.
 
-We'll learn more about Spack's spec syntax in :ref:`a later section <sec-specs>`.
-
-
-.. _extensions:
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Python packages and virtual environments
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Spack can install a large number of Python packages. Their names are
-typically prefixed with ``py-``. Installing and using them is no
-different from any other package:
-
-.. code-block:: console
-
-   $ spack install py-numpy
-   $ spack load py-numpy
-   $ python3
-   >>> import numpy
-
-The ``spack load`` command sets the ``PATH`` variable so that the right Python
-executable is used, and makes sure that ``numpy`` and its dependencies can be
-located in the ``PYTHONPATH``.
-
-Spack is different from other Python package managers in that it installs
-every package into its *own* prefix. This is in contrast to ``pip``, which
-installs all packages into the same prefix, be it in a virtual environment
-or not.
-
-For many users, **virtual environments** are more convenient than repeated
-``spack load`` commands, particularly when working with multiple Python
-packages. Fortunately Spack supports environments itself, which together
-with a view are no different from Python virtual environments.
-
-The recommended way of working with Python extensions such as ``py-numpy``
-is through :ref:`Environments <environments>`. The following example creates
-a Spack environment with ``numpy`` in the current working directory. It also
-puts a filesystem view in ``./view``, which is a more traditional combined
-prefix for all packages in the environment.
-
-.. code-block:: console
-
-   $ spack env create --with-view view --dir .
-   $ spack -e . add py-numpy
-   $ spack -e . concretize
-   $ spack -e . install
-
-Now you can activate the environment and start using the packages:
-
-.. code-block:: console
-
-   $ spack env activate .
-   $ python3
-   >>> import numpy
-
-The environment view is also a virtual environment, which is useful if you are
-sharing the environment with others who are unfamiliar with Spack. They can
-either use the Python executable directly:
-
-.. code-block:: console
-
-   $ ./view/bin/python3
-   >>> import numpy
-
-or use the activation script:
-
-.. code-block:: console
-
-   $ source ./view/bin/activate
-   $ python3
-   >>> import numpy
-
-In general, there should not be much difference between ``spack env activate``
-and using the virtual environment. The main advantage of ``spack env activate``
-is that it knows about more packages than just Python packages, and it may set
-additional runtime variables that are not covered by the virtual environment
-activation script.
-
-See :ref:`environments` for a more in-depth description of Spack
-environments and customizations to views.
+We'll learn more about Spack's spec syntax in the next section.
 
 
 .. _sec-specs:
@@ -1174,17 +1096,6 @@ unspecified version, but packages can depend on other packages with
 could depend on ``mpich@1.2:`` if it can only build with version
 ``1.2`` or higher of ``mpich``.
 
-.. note:: Windows Spec Syntax Caveats
-   Windows has a few idiosyncrasies when it comes to the Spack spec syntax and the use of certain shells
-   Spack's spec dependency syntax uses the carat (``^``) character, however this is an escape string in CMD
-   so it must be escaped with an additional carat (i.e. ``^^``).
-   CMD also will attempt to interpret strings with ``=`` characters in them. Any spec including this symbol
-   must double quote the string.
-
-   Note: All of these issues are unique to CMD, they can be avoided by using Powershell.
-
-   For more context on these caveats see the related issues: `carat <https://github.com/spack/spack/issues/42833>`_ and `equals <https://github.com/spack/spack/issues/43348>`_
-
 Below are more details about the specifiers that you can add to specs.
 
 .. _version-specifier:
@@ -1208,9 +1119,6 @@ and ``3.4.2``.  Similarly, ``@4.2:`` means any version above and including
 ``4.2``.  As a short-hand, ``@3`` is equivalent to the range ``@3:3`` and
 includes any version with major version ``3``.
 
-Versions are ordered lexicograpically by its components. For more details
-on the order, see :ref:`the packaging guide <version-comparison>`.
-
 Notice that you can distinguish between the specific version ``@=3.2`` and
 the range ``@3.2``. This is useful for packages that follow a versioning
 scheme that omits the zero patch version number: ``3.2``, ``3.2.1``,
@@ -1222,10 +1130,6 @@ A version specifier can also be a list of ranges and specific versions,
 separated by commas.  For example, ``@1.0:1.5,=1.7.1`` matches any version
 in the range ``1.0:1.5`` and the specific version ``1.7.1``.
 
-^^^^^^^^^^^^
-Git versions
-^^^^^^^^^^^^
-
 For packages with a ``git`` attribute, ``git`` references
 may be specified instead of a numerical version i.e. branches, tags
 and commits. Spack will stage and build based off the ``git``
@@ -1358,10 +1262,6 @@ For example, for the ``stackstart`` variant:
     mpileaks stackstart==4   # variant will be propagated to dependencies
     mpileaks stackstart=4    # only mpileaks will have this variant value
 
-Spack also allows variants to be propagated from a package that does
-not have that variant.
-
-
 ^^^^^^^^^^^^^^
 Compiler Flags
 ^^^^^^^^^^^^^^
@@ -1447,12 +1347,22 @@ the reserved keywords ``platform``, ``os`` and ``target``:
    $ spack install libelf os=ubuntu18.04
    $ spack install libelf target=broadwell
 
+or together by using the reserved keyword ``arch``:
+
+.. code-block:: console
+
+   $ spack install libelf arch=cray-CNL10-haswell
+
 Normally users don't have to bother specifying the architecture if they
 are installing software for their current host, as in that case the
 values will be detected automatically.  If you need fine-grained control
 over which packages use which targets (or over *all* packages' default
 target), see :ref:`package-preferences`.
 
+.. admonition:: Cray machines
+
+  The situation is a little bit different for Cray machines and a detailed
+  explanation on how the architecture can be set on them can be found at :ref:`cray-support`
 
 .. _support-for-microarchitectures:
 
@@ -1788,6 +1698,165 @@ check only local packages (as opposed to those used transparently from
 ``upstream`` spack instances) and the ``-j,--json`` option to output
 machine-readable json data for any errors.
 
+
+.. _extensions:
+
+---------------------------
+Extensions & Python support
+---------------------------
+
+Spack's installation model assumes that each package will live in its
+own install prefix.  However, certain packages are typically installed
+*within* the directory hierarchy of other packages.  For example,
+`Python <https://www.python.org>`_ packages are typically installed in the
+``$prefix/lib/python-2.7/site-packages`` directory.
+
+In Spack, installation prefixes are immutable, so this type of installation
+is not directly supported. However, it is possible to create views that
+allow you to merge install prefixes of multiple packages into a single new prefix.
+Views are a convenient way to get a more traditional filesystem structure.
+Using *extensions*, you can ensure that Python packages always share the
+same prefix in the view as Python itself. Suppose you have
+Python installed like so:
+
+.. code-block:: console
+
+   $ spack find python
+   ==> 1 installed packages.
+   -- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
+   python@2.7.8
+
+.. _cmd-spack-extensions:
+
+^^^^^^^^^^^^^^^^^^^^
+``spack extensions``
+^^^^^^^^^^^^^^^^^^^^
+
+You can find extensions for your Python installation like this:
+
+.. code-block:: console
+
+   $ spack extensions python
+   ==> python@2.7.8%gcc@4.4.7 arch=linux-debian7-x86_64-703c7a96
+   ==> 36 extensions:
+   geos          py-ipython     py-pexpect    py-pyside            py-sip
+   py-basemap    py-libxml2     py-pil        py-pytz              py-six
+   py-biopython  py-mako        py-pmw        py-rpy2              py-sympy
+   py-cython     py-matplotlib  py-pychecker  py-scientificpython  py-virtualenv
+   py-dateutil   py-mpi4py      py-pygments   py-scikit-learn
+   py-epydoc     py-mx          py-pylint     py-scipy
+   py-gnuplot    py-nose        py-pyparsing  py-setuptools
+   py-h5py       py-numpy       py-pyqt       py-shiboken
+
+   ==> 12 installed:
+   -- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
+   py-dateutil@2.4.0    py-nose@1.3.4       py-pyside@1.2.2
+   py-dateutil@2.4.0    py-numpy@1.9.1      py-pytz@2014.10
+   py-ipython@2.3.1     py-pygments@2.0.1   py-setuptools@11.3.1
+   py-matplotlib@1.4.2  py-pyparsing@2.0.3  py-six@1.9.0
+
+The extensions are a subset of what's returned by ``spack list``, and
+they are packages like any other.  They are installed into their own
+prefixes, and you can see this with ``spack find --paths``:
+
+.. code-block:: console
+
+   $ spack find --paths py-numpy
+   ==> 1 installed packages.
+   -- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
+       py-numpy@1.9.1  ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/py-numpy@1.9.1-66733244
+
+However, even though this package is installed, you cannot use it
+directly when you run ``python``:
+
+.. code-block:: console
+
+   $ spack load python
+   $ python
+   Python 2.7.8 (default, Feb 17 2015, 01:35:25)
+   [GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux2
+   Type "help", "copyright", "credits" or "license" for more information.
+   >>> import numpy
+   Traceback (most recent call last):
+     File "<stdin>", line 1, in <module>
+   ImportError: No module named numpy
+   >>>
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Using Extensions in Environments
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The recommended way of working with extensions such as ``py-numpy``
+above is through :ref:`Environments <environments>`. For example,
+the following creates an environment in the current working directory
+with a filesystem view in the ``./view`` directory:
+
+.. code-block:: console
+
+   $ spack env create --with-view view --dir .
+   $ spack -e . add py-numpy
+   $ spack -e . concretize
+   $ spack -e . install
+
+We recommend environments for two reasons. Firstly, environments
+can be activated (requires :ref:`shell-support`):
+
+.. code-block:: console
+
+   $ spack env activate .
+
+which sets all the right environment variables such as ``PATH`` and
+``PYTHONPATH``. This ensures that
+
+.. code-block:: console
+
+   $ python
+   >>> import numpy
+
+works. Secondly, even without shell support, the view ensures
+that Python can locate its extensions:
+
+.. code-block:: console
+
+   $ ./view/bin/python
+   >>> import numpy
+
+See :ref:`environments` for a more in-depth description of Spack
+environments and customizations to views.
+
+^^^^^^^^^^^^^^^^^^^^
+Using ``spack load``
+^^^^^^^^^^^^^^^^^^^^
+
+A more traditional way of using Spack and extensions is ``spack load``
+(requires :ref:`shell-support`). This will add the extension to ``PYTHONPATH``
+in your current shell, and Python itself will be available in the ``PATH``:
+
+.. code-block:: console
+
+   $ spack load py-numpy
+   $ python
+   >>> import numpy
+
+The loaded packages can be checked using ``spack find --loaded``
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Loading Extensions via Modules
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Apart from ``spack env activate`` and ``spack load``, you can load numpy
+through your environment modules (using ``environment-modules`` or
+``lmod``). This will also add the extension to the ``PYTHONPATH`` in
+your current shell.
+
+.. code-block:: console
+
+   $ module load <name of numpy module>
+
+If you do not know the name of the specific numpy module you wish to
+load, you can use the ``spack module tcl|lmod loads`` command to get
+the name of the module from the Spack spec.
+
 -----------------------
 Filesystem requirements
 -----------------------

lib/spack/docs/binary_caches.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -152,43 +153,7 @@ keyring, and trusting all downloaded keys.
 List of popular build caches
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-* `Extreme-scale Scientific Software Stack (E4S) <https://e4s-project.github.io/>`_: `build cache <https://oaciss.uoregon.edu/e4s/inventory.html>`_'
-
--------------------
-Build cache signing
--------------------
-
-By default, Spack will add a cryptographic signature to each package pushed to
-a build cache, and verifies the signature when installing from a build cache.
-
-Keys for signing can be managed with the :ref:`spack gpg <cmd-spack-gpg>` command,
-as well as ``spack buildcache keys`` as mentioned above.
-
-You can disable signing when pushing with ``spack buildcache push --unsigned``,
-and disable verification when installing from any build cache with
-``spack install --no-check-signature``.
-
-Alternatively, signing and verification can be enabled or disabled on a per build cache
-basis:
-
-.. code-block:: console
-
-    $ spack mirror add --signed <name> <url>  # enable signing and verification
-    $ spack mirror add --unsigned <name> <url>  # disable signing and verification
-
-    $ spack mirror set --signed <name>  # enable signing and verification for an existing mirror
-    $ spack mirror set --unsigned <name>  # disable signing and verification for an existing mirror
-
-Or you can directly edit the ``mirrors.yaml`` configuration file:
-
-.. code-block:: yaml
-
-    mirrors:
-      <name>:
-        url: <url>
-        signed: false # disable signing and verification
-
-See also :ref:`mirrors`.
+* `Extreme-scale Scientific Software Stack (E4S) <https://e4s-project.github.io/>`_: `build cache <https://oaciss.uoregon.edu/e4s/inventory.html>`_
 
 ----------
 Relocation
@@ -217,41 +182,6 @@ section of the configuration:
        padded_length: 128
 
 
-.. _binary_caches_oci:
-
----------------------------------
-Automatic push to a build cache
----------------------------------
-
-Sometimes it is convenient to push packages to a build cache as soon as they are installed. Spack can do this by setting autopush flag when adding a mirror:
-
-.. code-block:: console
-
-    $ spack mirror add --autopush <name> <url or path>
-
-Or the autopush flag can be set for an existing mirror:
-
-.. code-block:: console
-
-    $ spack mirror set --autopush <name>  # enable automatic push for an existing mirror
-    $ spack mirror set --no-autopush <name>  # disable automatic push for an existing mirror
-
-Then after installing a package it is automatically pushed to all mirrors with ``autopush: true``. The command
-
-.. code-block:: console
-
-    $ spack install <package>
-
-will have the same effect as
-
-.. code-block:: console
-
-    $ spack install <package>
-    $ spack buildcache push <cache> <package>  # for all caches with autopush: true
-
-.. note::
-
-    Packages are automatically pushed to a build cache only if they are built from source.
 
 -----------------------------------------
 OCI / Docker V2 registries as build cache
@@ -264,30 +194,25 @@ infrastructure, or to cache Spack built binaries in Github Actions and
 GitLab CI.
 
 To get started, configure an OCI mirror using ``oci://`` as the scheme,
-and optionally specify variables that hold the username and password (or
-personal access token) for the registry:
+and optionally specify a username and password (or personal access token):
 
 .. code-block:: console
 
-    $ spack mirror add --oci-username-variable REGISTRY_USER \
-                       --oci-password-variable REGISTRY_TOKEN \
-                       my_registry oci://example.com/my_image
+    $ spack mirror add --oci-username username --oci-password password my_registry oci://example.com/my_image
 
 Spack follows the naming conventions of Docker, with Dockerhub as the default
 registry. To use Dockerhub, you can omit the registry domain:
 
 .. code-block:: console
 
-    $ spack mirror add ... my_registry oci://username/my_image
+    $ spack mirror add --oci-username username --oci-password password my_registry oci://username/my_image
 
 From here, you can use the mirror as any other build cache:
 
 .. code-block:: console
 
-    $ export REGISTRY_USER=...
-    $ export REGISTRY_TOKEN=...
     $ spack buildcache push my_registry <specs...>  # push to the registry
-    $ spack install <specs...>  # or install from the registry
+    $ spack install <specs...> # install from the registry
 
 A unique feature of buildcaches on top of OCI registries is that it's incredibly
 easy to generate get a runnable container image with the binaries installed. This
@@ -325,13 +250,87 @@ To significantly speed up Spack in GitHub Actions, binaries can be cached in
 GitHub Packages. This service is an OCI registry that can be linked to a GitHub
 repository.
 
+A typical workflow is to include a ``spack.yaml`` environment in your repository
+that specifies the packages to install, the target architecture, and the build
+cache to use under ``mirrors``:
+
+.. code-block:: yaml
+
+    spack:
+      specs:
+      - python@3.11
+      config:
+        install_tree:
+          root: /opt/spack
+          padded_length: 128
+      packages:
+        all:
+          require: target=x86_64_v2
+      mirrors:
+        local-buildcache: oci://ghcr.io/<organization>/<repository>
+
+A GitHub action can then be used to install the packages and push them to the
+build cache:
+
+.. code-block:: yaml
+
+    name: Install Spack packages
+
+    on: push
+
+    env:
+      SPACK_COLOR: always
+
+    jobs:
+      example:
+        runs-on: ubuntu-22.04
+        permissions:
+          packages: write
+        steps:
+        - name: Checkout
+          uses: actions/checkout@v3
+
+        - name: Checkout Spack
+          uses: actions/checkout@v3
+          with:
+            repository: spack/spack
+            path: spack
+
+        - name: Setup Spack
+          run: echo "$PWD/spack/bin" >> "$GITHUB_PATH"
+
+        - name: Concretize
+          run: spack -e . concretize
+
+        - name: Install
+          run: spack -e . install --no-check-signature
+
+        - name: Run tests
+            run: ./my_view/bin/python3 -c 'print("hello world")'
+
+        - name: Push to buildcache
+          run: |
+            spack -e . mirror set --oci-username ${{ github.actor }} --oci-password "${{ secrets.GITHUB_TOKEN }}" local-buildcache
+            spack -e . buildcache push --base-image ubuntu:22.04 --unsigned --update-index local-buildcache
+          if: ${{ !cancelled() }}
+
+The first time this action runs, it will build the packages from source and
+push them to the build cache. Subsequent runs will pull the binaries from the
+build cache. The concretizer will ensure that prebuilt binaries are favored
+over source builds.
+
+The build cache entries appear in the GitHub Packages section of your repository,
+and contain instructions for pulling and running them with ``docker`` or ``podman``.
+
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Using Spack's public build cache for GitHub Actions
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 Spack offers a public build cache for GitHub Actions with a set of common packages,
 which lets you get started quickly. See the following resources for more information:
 
-* `spack/setup-spack <https://github.com/spack/setup-spack>`_ for setting up Spack in GitHub
-  Actions
-* `spack/github-actions-buildcache <https://github.com/spack/github-actions-buildcache>`_ for
-  more details on the public build cache
+* `spack/github-actions-buildcache <https://github.com/spack/github-actions-buildcache>`_
 
 .. _cmd-spack-buildcache:
 

lib/spack/docs/bootstrapping.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -86,7 +87,7 @@ You can check what is installed in the bootstrapping store at any time using:
 
 .. code-block:: console
 
-   % spack -b find
+   % spack find -b
    ==> Showing internal bootstrap store at "/Users/spack/.spack/bootstrap/store"
    ==> 11 installed packages
    -- darwin-catalina-x86_64 / apple-clang@12.0.0 ------------------
@@ -100,7 +101,7 @@ In case it is needed you can remove all the software in the current bootstrappin
    % spack clean -b
    ==> Removing bootstrapped software and configuration in "/Users/spack/.spack/bootstrap"
 
-   % spack -b find
+   % spack find -b
    ==> Showing internal bootstrap store at "/Users/spack/.spack/bootstrap/store"
    ==> 0 installed packages
 
@@ -170,8 +171,8 @@ bootstrapping.
    To register the mirror on the platform where it's supposed to be used run the following command(s):
      % spack bootstrap add --trust local-sources /opt/bootstrap/metadata/sources
      % spack bootstrap add --trust local-binaries /opt/bootstrap/metadata/binaries
-     % spack buildcache update-index /opt/bootstrap/bootstrap_cache
+
 
 This command needs to be run on a machine with internet access and the resulting folder
 has to be moved over to the air-gapped system. Once the local sources are added using the
-commands suggested at the prompt, they can be used to bootstrap Spack.
+commands suggested at the prompt, they can be used to bootstrap Spack.

lib/spack/docs/build_settings.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -20,89 +21,23 @@ is the following:
 Reuse already installed packages
 --------------------------------
 
-The ``reuse`` attribute controls how aggressively Spack reuses binary packages during concretization. The
-attribute can either be a single value, or an object for more complex configurations.
-
-In the former case ("single value") it allows Spack to:
-
-1. Reuse installed packages and buildcaches for all the specs to be concretized, when ``true``
-2. Reuse installed packages and buildcaches only for the dependencies of the root specs, when ``dependencies``
-3. Disregard reusing installed packages and buildcaches, when ``false``
-
-In case a finer control over which specs are reused is needed, then the value of this attribute can be
-an object, with the following keys:
-
-1. ``roots``: if ``true`` root specs are reused, if ``false`` only dependencies of root specs are reused
-2. ``from``: list of sources from which reused specs are taken
-3. ``namespaces``: list of namespaces from which to reuse specs, or the string ``"any"``.
-
-Each source in ``from`` is itself an object:
-
-.. list-table:: Attributes for a source or reusable specs
-   :header-rows: 1
-
-   * - Attribute name
-     - Description
-   * - type (mandatory, string)
-     - Can be ``local``, ``buildcache``, or ``external``
-   * - include (optional, list of specs)
-     - If present, reusable specs must match at least one of the constraint in the list
-   * - exclude (optional, list of specs)
-     - If present, reusable specs must not match any of the constraint in the list.
-
-For instance, the following configuration:
-
-.. code-block:: yaml
-
-   concretizer:
-     reuse:
-       roots: true
-       namespaces: [builtin]
-       from:
-       - type: local
-         include:
-         - "%gcc"
-         - "%clang"
-
-tells the concretizer to reuse all specs compiled with either ``gcc`` or ``clang``, that are installed
-in the local store. Any spec from remote buildcaches is disregarded. Any spec from a namespace other than
-Spack's builtin repo is disregarded.
-
-To reduce the boilerplate in configuration files, default values for the ``include`` and
-``exclude`` options can be pushed up one level:
-
-.. code-block:: yaml
-
-   concretizer:
-     reuse:
-       roots: true
-       include:
-       - "%gcc"
-       from:
-       - type: local
-       - type: buildcache
-       - type: local
-         include:
-         - "foo %oneapi"
-
-In the example above we reuse all specs compiled with ``gcc`` from the local store
-and remote buildcaches, and we also reuse ``foo %oneapi``. Note that the last source of
-specs override the default ``include`` attribute.
-
-For one-off concretizations, the are command line arguments for each of the simple "single value"
-configurations. This means a user can:
+The ``reuse`` attribute controls whether Spack will prefer to use installed packages (``true``), or
+whether it will do a "fresh" installation and prefer the latest settings from
+``package.py`` files and ``packages.yaml`` (``false``).
+You can use:
 
 .. code-block:: console
 
    % spack install --reuse <spec>
 
-to enable reuse for a single installation, or:
+to enable reuse for a single installation, and you can use:
 
 .. code-block:: console
 
    spack install --fresh <spec>
 
 to do a fresh install if ``reuse`` is enabled by default.
+``reuse: dependencies`` is the default.
 
 .. seealso::
 
@@ -168,106 +103,3 @@ while `py-numpy` still needs an older version:
 
 Up to Spack v0.20 ``duplicates:strategy:none`` was the default (and only) behavior. From Spack v0.21 the
 default behavior is ``duplicates:strategy:minimal``.
-
---------
-Splicing
---------
-
-The ``splice`` key covers config attributes for splicing specs in the solver.
-
-"Splicing" is a method for replacing a dependency with another spec
-that provides the same package or virtual. There are two types of
-splices, referring to different behaviors for shared dependencies
-between the root spec and the new spec replacing a dependency:
-"transitive" and "intransitive". A "transitive" splice is one that
-resolves all conflicts by taking the dependency from the new node. An
-"intransitive" splice is one that resolves all conflicts by taking the
-dependency from the original root. From a theory perspective, hybrid
-splices are possible but are not modeled by Spack.
-
-All spliced specs retain a ``build_spec`` attribute that points to the
-original Spec before any splice occurred. The ``build_spec`` for a
-non-spliced spec is itself.
-
-The figure below shows examples of transitive and intransitive splices:
-
-.. figure:: images/splices.png
-   :align: center
-
-The concretizer can be configured to explicitly splice particular
-replacements for a target spec. Splicing will allow the user to make
-use of generically built public binary caches, while swapping in
-highly optimized local builds for performance critical components
-and/or components that interact closely with the specific hardware
-details of the system. The most prominent candidate for splicing is
-MPI providers. MPI packages have relatively well-understood ABI
-characteristics, and most High Performance Computing facilities deploy
-highly optimized MPI packages tailored to their particular
-hardware. The following config block configures Spack to replace
-whatever MPI provider each spec was concretized to use with the
-particular package of ``mpich`` with the hash that begins ``abcdef``.
-
-.. code-block:: yaml
-
-   concretizer:
-     splice:
-       explicit:
-       - target: mpi
-         replacement: mpich/abcdef
-         transitive: false
-
-.. warning::
-
-   When configuring an explicit splice, you as the user take on the
-   responsibility for ensuring ABI compatibility between the specs
-   matched by the target and the replacement you provide. If they are
-   not compatible, Spack will not warn you and your application will
-   fail to run.
-
-The ``target`` field of an explicit splice can be any abstract
-spec. The ``replacement`` field must be a spec that includes the hash
-of a concrete spec, and the replacement must either be the same
-package as the target, provide the virtual that is the target, or
-provide a virtual that the target provides. The ``transitive`` field
-is optional -- by default, splices will be transitive.
-
-.. note::
-
-   With explicit splices configured, it is possible for Spack to
-   concretize to a spec that does not satisfy the input. For example,
-   with the config above ``hdf5 ^mvapich2`` will concretize to user
-   ``mpich/abcdef`` instead of ``mvapich2`` as the MPI provider. Spack
-   will warn the user in this case, but will not fail the
-   concretization.
-
-.. _automatic_splicing:
-
-^^^^^^^^^^^^^^^^^^
-Automatic Splicing
-^^^^^^^^^^^^^^^^^^
-
-The Spack solver can be configured to do automatic splicing for
-ABI-compatible packages. Automatic splices are enabled in the concretizer
-config section
-
-.. code-block:: yaml
-
-   concretizer:
-     splice:
-       automatic: True
-
-Packages can include ABI-compatibility information using the
-``can_splice`` directive. See :ref:`the packaging
-guide<abi_compatibility>` for instructions on specifying ABI
-compatibility using the ``can_splice`` directive.
-
-.. note::
-
-   The ``can_splice`` directive is experimental and may be changed in
-   future versions.
-
-When automatic splicing is enabled, the concretizer will combine any
-number of ABI-compatible specs if possible to reuse installed packages
-and packages available from binary caches. The end result of these
-specs is equivalent to a series of transitive/intransitive splices,
-but the series may be non-obvious.

lib/spack/docs/build_systems.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/build_systems/autotoolspackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -146,15 +147,6 @@ example, the ``bash`` shell is used to run the ``autogen.sh`` script.
    def autoreconf(self, spec, prefix):
        which("bash")("autogen.sh")
 
-If the ``package.py`` has build instructions in a separate
-:ref:`builder class <multiple_build_systems>`, the signature for a phase changes slightly:
-
-.. code-block:: python
-
-   class AutotoolsBuilder(AutotoolsBuilder):
-      def autoreconf(self, pkg, spec, prefix):
-         which("bash")("autogen.sh")
-
 """""""""""""""""""""""""""""""""""""""
 patching configure or Makefile.in files
 """""""""""""""""""""""""""""""""""""""
@@ -272,9 +264,9 @@ often lists dependencies and the flags needed to locate them. The
 "environment variables" section lists environment variables that the
 build system uses to pass flags to the compiler and linker.
 
-^^^^^^^^^^^^^^^^^^^^^^^^^
-Adding flags to configure
-^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Addings flags to configure
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 For most of the flags you encounter, you will want a variant to
 optionally enable/disable them. You can then optionally pass these
@@ -285,7 +277,7 @@ function like so:
 
    def configure_args(self):
        args = []
-       ...
+
        if self.spec.satisfies("+mpi"):
            args.append("--enable-mpi")
        else:
@@ -299,10 +291,7 @@ Alternatively, you can use the :ref:`enable_or_disable  <autotools_enable_or_dis
 .. code-block:: python
 
    def configure_args(self):
-       args = []
-       ...
-       args.extend(self.enable_or_disable("mpi"))
-       return args
+       return [self.enable_or_disable("mpi")]
 
 
 Note that we are explicitly disabling MPI support if it is not
@@ -347,14 +336,7 @@ typically used to enable or disable some feature within the package.
        default=False,
        description="Memchecker support for debugging [degrades performance]"
    )
-   ...
-
-   def configure_args(self):
-       args = []
-       ...
-       args.extend(self.enable_or_disable("memchecker"))
-
-       return args
+   config_args.extend(self.enable_or_disable("memchecker"))
 
 In this example, specifying the variant ``+memchecker`` will generate
 the following configuration options:

lib/spack/docs/build_systems/bundlepackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/build_systems/cachedcmakepackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/build_systems/cmakepackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -81,7 +82,7 @@ class already contains:
 
 .. code-block:: python
 
-   depends_on("cmake", type="build")
+   depends_on('cmake', type='build')
 
 
 If you need to specify a particular version requirement, you can
@@ -89,7 +90,7 @@ override this in your package:
 
 .. code-block:: python
 
-   depends_on("cmake@2.8.12:", type="build")
+   depends_on('cmake@2.8.12:', type='build')
 
 
 ^^^^^^^^^^^^^^^^^^^
@@ -136,10 +137,10 @@ and without the :meth:`~spack.build_systems.cmake.CMakeBuilder.define` and
 
    def cmake_args(self):
        args = [
-           "-DWHATEVER:STRING=somevalue",
-           self.define("ENABLE_BROKEN_FEATURE", False),
-           self.define_from_variant("DETECT_HDF5", "hdf5"),
-           self.define_from_variant("THREADS"), # True if +threads
+           '-DWHATEVER:STRING=somevalue',
+           self.define('ENABLE_BROKEN_FEATURE', False),
+           self.define_from_variant('DETECT_HDF5', 'hdf5'),
+           self.define_from_variant('THREADS'), # True if +threads
        ]
 
        return args
@@ -150,10 +151,10 @@ and CMake simply ignores the empty command line argument. For example the follow
 
 .. code-block:: python
 
-   variant("example", default=True, when="@2.0:")
+   variant('example', default=True, when='@2.0:')
 
    def cmake_args(self):
-      return [self.define_from_variant("EXAMPLE", "example")]
+      return [self.define_from_variant('EXAMPLE', 'example')]
 
 will generate ``'cmake' '-DEXAMPLE=ON' ...`` when `@2.0: +example` is met, but will
 result in ``'cmake' '' ...`` when the spec version is below ``2.0``.
@@ -192,9 +193,9 @@ a variant to control this:
 
 .. code-block:: python
 
-   variant("build_type", default="RelWithDebInfo",
-           description="CMake build type",
-           values=("Debug", "Release", "RelWithDebInfo", "MinSizeRel"))
+   variant('build_type', default='RelWithDebInfo',
+           description='CMake build type',
+           values=('Debug', 'Release', 'RelWithDebInfo', 'MinSizeRel'))
 
 However, not every CMake package accepts all four of these options.
 Grep the ``CMakeLists.txt`` file to see if the default values are
@@ -204,9 +205,9 @@ package overrides the default variant with:
 
 .. code-block:: python
 
-   variant("build_type", default="DebugRelease",
-           description="The build type to build",
-           values=("Debug", "Release", "DebugRelease"))
+   variant('build_type', default='DebugRelease',
+           description='The build type to build',
+           values=('Debug', 'Release', 'DebugRelease'))
 
 For more information on ``CMAKE_BUILD_TYPE``, see:
 https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html
@@ -249,7 +250,7 @@ generator is Ninja. To switch to the Ninja generator, simply add:
 
 .. code-block:: python
 
-   generator("ninja")
+   generator = 'Ninja'
 
 
 ``CMakePackage`` defaults to "Unix Makefiles". If you switch to the
@@ -257,7 +258,7 @@ Ninja generator, make sure to add:
 
 .. code-block:: python
 
-   depends_on("ninja", type="build")
+   depends_on('ninja', type='build')
 
 to the package as well. Aside from that, you shouldn't need to do
 anything else. Spack will automatically detect that you are using
@@ -287,7 +288,7 @@ like so:
 
 .. code-block:: python
 
-   root_cmakelists_dir = "src"
+   root_cmakelists_dir = 'src'
 
 
 Note that this path is relative to the root of the extracted tarball,
@@ -303,7 +304,7 @@ different sub-directory, simply override ``build_directory`` like so:
 
 .. code-block:: python
 
-   build_directory = "my-build"
+   build_directory = 'my-build'
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 Build and install targets
@@ -323,8 +324,8 @@ library or build the documentation, you can add these like so:
 
 .. code-block:: python
 
-   build_targets = ["all", "docs"]
-   install_targets = ["install", "docs"]
+   build_targets = ['all', 'docs']
+   install_targets = ['install', 'docs']
 
 ^^^^^^^
 Testing

lib/spack/docs/build_systems/cudapackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/build_systems/custompackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -56,13 +57,13 @@ If you look at the ``perl`` package, you'll see:
 
 .. code-block:: python
 
-   phases = ("configure", "build", "install")
+   phases = ["configure", "build", "install"]
 
 Similarly, ``cmake`` defines:
 
 .. code-block:: python
 
-   phases = ("bootstrap", "build", "install")
+   phases = ["bootstrap", "build", "install"]
 
 If we look at the ``cmake`` example, this tells Spack's ``PackageBase``
 class to run the ``bootstrap``, ``build``, and ``install`` functions
@@ -129,19 +130,14 @@ before or after a particular phase. For example, in ``perl``, we see:
 
    @run_after("install")
    def install_cpanm(self):
-        spec = self.spec
-        maker = make
-        cpan_dir = join_path("cpanm", "cpanm")
-        if sys.platform == "win32":
-            maker = nmake
-            cpan_dir = join_path(self.stage.source_path, cpan_dir)
-            cpan_dir = windows_sfn(cpan_dir)
-        if "+cpanm" in spec:
-            with working_dir(cpan_dir):
-                perl = spec["perl"].command
-                perl("Makefile.PL")
-                maker()
-                maker("install")
+       spec = self.spec
+
+       if spec.satisfies("+cpanm"):
+           with working_dir(join_path("cpanm", "cpanm")):
+               perl = spec["perl"].command
+               perl("Makefile.PL")
+               make()
+               make("install")
 
 This extra step automatically installs ``cpanm`` in addition to the
 base Perl installation.
@@ -180,14 +176,8 @@ In the ``perl`` package, we can see:
 
    @run_after("build")
    @on_package_attributes(run_tests=True)
-   def build_test(self):
-        if sys.platform == "win32":
-            win32_dir = os.path.join(self.stage.source_path, "win32")
-            win32_dir = windows_sfn(win32_dir)
-            with working_dir(win32_dir):
-                nmake("test", ignore_quotes=True)
-        else:
-            make("test")
+   def test(self):
+       make("test")
 
 As you can guess, this runs ``make test`` *after* building the package,
 if and only if testing is requested. Again, this is not specific to

lib/spack/docs/build_systems/inteloneapipackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -24,7 +25,7 @@ use Spack to build packages with the tools.
 The Spack Python class ``IntelOneapiPackage`` is a base class that is
 used by ``IntelOneapiCompilers``, ``IntelOneapiMkl``,
 ``IntelOneapiTbb`` and other classes to implement the oneAPI
-packages. Search for ``oneAPI`` at `packages.spack.io <https://packages.spack.io>`_ for the full
+packages. Search for ``oneAPI`` at `<packages.spack.io>`_ for the full
 list of available oneAPI packages, or use::
 
   spack list -d oneAPI
@@ -52,24 +53,18 @@ Install the oneAPI compilers::
 
 Add the compilers to your ``compilers.yaml`` so spack can use them::
 
-  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin
+  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin/intel64
+  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin
 
 Verify that the compilers are available::
 
   spack compiler list
 
-Note that 2024 and later releases do not include ``icc``. Before 2024,
-the package layout was different::
-  
-  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin/intel64
-  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin
-
 The ``intel-oneapi-compilers`` package includes 2 families of
 compilers:
 
 * ``intel``: ``icc``, ``icpc``, ``ifort``. Intel's *classic*
-  compilers. 2024 and later releases contain ``ifort``, but not
-  ``icc`` and ``icpc``.
+  compilers.
 * ``oneapi``: ``icx``, ``icpx``, ``ifx``. Intel's new generation of
   compilers based on LLVM.
 
@@ -94,8 +89,8 @@ Install the oneAPI compilers::
 
 Add the compilers to your ``compilers.yaml`` so Spack can use them::
 
-  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin
-  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/bin
+  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin/intel64
+  spack compiler add `spack location -i intel-oneapi-compilers`/compiler/latest/linux/bin
 
 Verify that the compilers are available::
 
@@ -151,7 +146,8 @@ Compilers
 To use the compilers, add some information about the installation to
 ``compilers.yaml``. For most users, it is sufficient to do::
 
-  spack compiler add /opt/intel/oneapi/compiler/latest/bin
+  spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin/intel64
+  spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin
 
 Adapt the paths above if you did not install the tools in the default
 location. After adding the compilers, using them is the same
@@ -160,12 +156,6 @@ Another option is to manually add the configuration to
 ``compilers.yaml`` as described in :ref:`Compiler configuration
 <compiler-config>`.
 
-Before 2024, the directory structure was different::
-  
-  spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin/intel64
-  spack compiler add /opt/intel/oneapi/compiler/latest/linux/bin
-
-
 Libraries
 ---------
 

lib/spack/docs/build_systems/intelpackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -89,7 +90,7 @@ and optimizers do require a paid license.  In Spack, they are packaged as:
     TODO: Confirm and possible change(!) the scope of MPI components (runtime
     vs. devel) in current (and previous?) *cluster/professional/composer*
     editions, i.e., presence in downloads, possibly subject to license
-    coverage(!); see `discussion in PR #4300
+    coverage(!); see `disussion in PR #4300
     <https://github.com/spack/spack/pull/4300#issuecomment-305582898>`_.  [NB:
     An "mpi" subdirectory is not indicative of the full MPI SDK being present
     (i.e., ``mpicc``, ..., and header files).  The directory may just as well
@@ -933,9 +934,9 @@ a *virtual* ``mkl`` package is declared in Spack.
   .. code-block:: python
 
      # Examples for absolute and conditional dependencies:
-     depends_on("mkl")
-     depends_on("mkl", when="+mkl")
-     depends_on("mkl", when="fftw=mkl")
+     depends_on('mkl')
+     depends_on('mkl', when='+mkl')
+     depends_on('mkl', when='fftw=mkl')
 
   The ``MKLROOT`` environment variable (part of the documented API) will be set
   during all stages of client package installation, and is available to both
@@ -971,8 +972,8 @@ a *virtual* ``mkl`` package is declared in Spack.
       def configure_args(self):
           args = []
           ...
-          args.append("--with-blas=%s" % self.spec["blas"].libs.ld_flags)
-          args.append("--with-lapack=%s" % self.spec["lapack"].libs.ld_flags)
+          args.append('--with-blas=%s' % self.spec['blas'].libs.ld_flags)
+          args.append('--with-lapack=%s' % self.spec['lapack'].libs.ld_flags)
           ...
 
   .. tip::
@@ -988,13 +989,13 @@ a *virtual* ``mkl`` package is declared in Spack.
 
   .. code-block:: python
 
-    self.spec["blas"].headers.include_flags
+    self.spec['blas'].headers.include_flags
 
   and to generate linker options (``-L<dir> -llibname ...``), use the same as above,
 
   .. code-block:: python
 
-    self.spec["blas"].libs.ld_flags
+    self.spec['blas'].libs.ld_flags
 
   See
   :ref:`MakefilePackage <makefilepackage>`

lib/spack/docs/build_systems/luapackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -87,7 +88,7 @@ override the ``luarocks_args`` method like so:
 .. code-block:: python
 
     def luarocks_args(self):
-        return ["flag1", "flag2"]
+        return ['flag1', 'flag2']
 
 One common use of this is to override warnings or flags for newer compilers, as in:
 

lib/spack/docs/build_systems/makefilepackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -87,13 +88,13 @@ command-line. However, Makefiles that use ``?=`` for assignment honor
 environment variables. Since Spack already sets ``CC``, ``CXX``, ``F77``,
 and ``FC``, you won't need to worry about setting these variables. If
 there are any other variables you need to set, you can do this in the
-``setup_build_environment`` method:
+``edit`` method:
 
 .. code-block:: python
 
-   def setup_build_environment(self, env):
-       env.set("PREFIX", prefix)
-       env.set("BLASLIB", spec["blas"].libs.ld_flags)
+   def edit(self, spec, prefix):
+       env["PREFIX"] = prefix
+       env["BLASLIB"] = spec["blas"].libs.ld_flags
 
 
 `cbench <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cbench/package.py>`_
@@ -139,7 +140,7 @@ Edit Makefile
 Some Makefiles are just plain stubborn and will ignore command-line
 variables. The only way to ensure that these packages build correctly
 is to directly edit the Makefile. Spack provides a ``FileFilter`` class
-and a ``filter`` method to help with this. For example:
+and a ``filter_file`` method to help with this. For example:
 
 .. code-block:: python
 

lib/spack/docs/build_systems/mavenpackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -47,8 +48,8 @@ class automatically adds the following dependencies:
 
 .. code-block:: python
 
-   depends_on("java", type=("build", "run"))
-   depends_on("maven", type="build")
+   depends_on('java', type=('build', 'run'))
+   depends_on('maven', type='build')
 
 
 In the ``pom.xml`` file, you may see sections like:
@@ -71,8 +72,8 @@ should add:
 
 .. code-block:: python
 
-   depends_on("java@7:", type="build")
-   depends_on("maven@3.5.4:", type="build")
+   depends_on('java@7:', type='build')
+   depends_on('maven@3.5.4:', type='build')
 
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -87,9 +88,9 @@ the build phase. For example:
 
    def build_args(self):
        return [
-           "-Pdist,native",
-           "-Dtar",
-           "-Dmaven.javadoc.skip=true"
+           '-Pdist,native',
+           '-Dtar',
+           '-Dmaven.javadoc.skip=true'
        ]
 
 

lib/spack/docs/build_systems/mesonpackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -85,8 +86,8 @@ the ``MesonPackage`` base class already contains:
 
 .. code-block:: python
 
-   depends_on("meson", type="build")
-   depends_on("ninja", type="build")
+   depends_on('meson', type='build')
+   depends_on('ninja', type='build')
 
 
 If you need to specify a particular version requirement, you can
@@ -94,8 +95,8 @@ override this in your package:
 
 .. code-block:: python
 
-   depends_on("meson@0.43.0:", type="build")
-   depends_on("ninja", type="build")
+   depends_on('meson@0.43.0:', type='build')
+   depends_on('ninja', type='build')
 
 
 ^^^^^^^^^^^^^^^^^^^
@@ -120,7 +121,7 @@ override the ``meson_args`` method like so:
 .. code-block:: python
 
    def meson_args(self):
-       return ["--warnlevel=3"]
+       return ['--warnlevel=3']
 
 
 This method can be used to pass flags as well as variables.

lib/spack/docs/build_systems/octavepackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/build_systems/perlpackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -117,7 +118,7 @@ so ``PerlPackage`` contains:
 
 .. code-block:: python
 
-   extends("perl")
+   extends('perl')
 
 
 If your package requires a specific version of Perl, you should
@@ -131,14 +132,14 @@ properly. If your package uses ``Makefile.PL`` to build, add:
 
 .. code-block:: python
 
-   depends_on("perl-extutils-makemaker", type="build")
+   depends_on('perl-extutils-makemaker', type='build')
 
 
 If your package uses ``Build.PL`` to build, add:
 
 .. code-block:: python
 
-   depends_on("perl-module-build", type="build")
+   depends_on('perl-module-build', type='build')
 
 
 ^^^^^^^^^^^^^^^^^
@@ -164,80 +165,14 @@ arguments to ``Makefile.PL`` or ``Build.PL`` by overriding
 .. code-block:: python
 
    def configure_args(self):
-       expat = self.spec["expat"].prefix
+       expat = self.spec['expat'].prefix
 
        return [
-           "EXPATLIBPATH={0}".format(expat.lib),
-           "EXPATINCPATH={0}".format(expat.include),
+           'EXPATLIBPATH={0}'.format(expat.lib),
+           'EXPATINCPATH={0}'.format(expat.include),
        ]
 
 
-^^^^^^^
-Testing
-^^^^^^^
-
-``PerlPackage`` provides a simple stand-alone test of the successfully
-installed package to confirm that installed perl module(s) can be used.
-These tests can be performed any time after the installation using
-``spack -v test run``. (For more information on the command, see 
-:ref:`cmd-spack-test-run`.)
-
-The base class automatically detects perl modules based on the presence
-of ``*.pm`` files under the package's library directory. For example,
-the files under ``perl-bignum``'s perl library are:
-
-.. code-block:: console
-
-   $ find . -name "*.pm"
-   ./bigfloat.pm
-   ./bigrat.pm
-   ./Math/BigFloat/Trace.pm
-   ./Math/BigInt/Trace.pm
-   ./Math/BigRat/Trace.pm
-   ./bigint.pm
-   ./bignum.pm
-
-
-which results in the package having the ``use_modules`` property containing:
-
-.. code-block:: python
-
-   use_modules = [
-       "bigfloat",
-       "bigrat",
-       "Math::BigFloat::Trace",
-       "Math::BigInt::Trace",
-       "Math::BigRat::Trace",
-       "bigint",
-       "bignum",
-   ]
-
-.. note::
-
-   This list can often be used to catch missing dependencies.
-
-If the list is somehow wrong, you can provide the names of the modules
-yourself by overriding ``use_modules`` like so:
-
- .. code-block:: python
-
-    use_modules = ["bigfloat", "bigrat", "bigint", "bignum"]
-
-If you only want a subset of the automatically detected modules to be
-tested, you could instead define the ``skip_modules`` property on the
-package. So, instead of overriding ``use_modules`` as shown above, you
-could define the following:
-
- .. code-block:: python
-
-    skip_modules = [
-        "Math::BigFloat::Trace",
-        "Math::BigInt::Trace",
-        "Math::BigRat::Trace",
-    ]
-
-for the same use tests.
-
 ^^^^^^^^^^^^^^^^^^^^^
 Alternatives to Spack
 ^^^^^^^^^^^^^^^^^^^^^

lib/spack/docs/build_systems/pythonpackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -717,45 +718,23 @@ command-line tool, or C/C++/Fortran program with optional Python
 modules? The former should be prepended with ``py-``, while the
 latter should not.
 
-""""""""""""""""""""""""""""""
-``extends`` vs. ``depends_on``
-""""""""""""""""""""""""""""""
+""""""""""""""""""""""
+extends vs. depends_on
+""""""""""""""""""""""
 
+This is very similar to the naming dilemma above, with a slight twist.
 As mentioned in the :ref:`Packaging Guide <packaging_extensions>`,
 ``extends`` and ``depends_on`` are very similar, but ``extends`` ensures
 that the extension and extendee share the same prefix in views.
 This allows the user to import a Python module without
 having to add that module to ``PYTHONPATH``.
 
-Additionally, ``extends("python")`` adds a dependency on the package
-``python-venv``. This improves isolation from the system, whether
-it's during the build or at runtime: user and system site packages
-cannot accidentally be used by any package that ``extends("python")``.
-
-As a rule of thumb: if a package does not install any Python modules
-of its own, and merely puts a Python script in the ``bin`` directory,
-then there is no need for ``extends``. If the package installs modules
-in the ``site-packages`` directory, it requires ``extends``.
-
-"""""""""""""""""""""""""""""""""""""
-Executing ``python`` during the build
-"""""""""""""""""""""""""""""""""""""
-
-Whenever you need to execute a Python command or pass the path of the
-Python interpreter to the build system, it is best to use the global
-variable ``python`` directly. For example:
-
-.. code-block:: python
-
-    @run_before("install")
-    def recythonize(self):
-        python("setup.py", "clean")  # use the `python` global
-
-As mentioned in the previous section, ``extends("python")`` adds an
-automatic dependency on ``python-venv``, which is a virtual environment
-that guarantees build isolation. The ``python`` global always refers to
-the correct Python interpreter, whether the package uses ``extends("python")``
-or ``depends_on("python")``.
+When deciding between ``extends`` and ``depends_on``, the best rule of
+thumb is to check the installation prefix. If Python libraries are
+installed to ``<prefix>/lib/pythonX.Y/site-packages``, then you
+should use ``extends``. If Python libraries are installed elsewhere
+or the only files that get installed reside in ``<prefix>/bin``, then
+don't use ``extends``.
 
 ^^^^^^^^^^^^^^^^^^^^^
 Alternatives to Spack

lib/spack/docs/build_systems/qmakepackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -24,14 +25,6 @@ QMake does not appear to have a standardized way of specifying
 the installation directory, so you may have to set environment
 variables or edit ``*.pro`` files to get things working properly.
 
-QMake packages will depend on the virtual ``qmake`` package which
-is provided by multiple versions of Qt: ``qt`` provides Qt up to
-Qt5, and ``qt-base`` provides Qt from version Qt6 onwards. This
-split was motivated by the desire to split the single Qt package
-into its components to allow for more fine-grained installation.
-To depend on a specific version, refer to the documentation on
-:ref:`virtual-dependencies`.
-
 ^^^^^^
 Phases
 ^^^^^^
@@ -90,7 +83,7 @@ base class already contains:
 
 .. code-block:: python
 
-   depends_on("qt", type="build")
+   depends_on('qt', type='build')
 
 
 If you want to specify a particular version requirement, or need to
@@ -98,7 +91,7 @@ link to the ``qt`` libraries, you can override this in your package:
 
 .. code-block:: python
 
-   depends_on("qt@5.6.0:")
+   depends_on('qt@5.6.0:')
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 Passing arguments to qmake
@@ -110,7 +103,7 @@ override the ``qmake_args`` method like so:
 .. code-block:: python
 
    def qmake_args(self):
-       return ["-recursive"]
+       return ['-recursive']
 
 
 This method can be used to pass flags as well as variables.
@@ -125,7 +118,7 @@ sub-directory by adding the following to the package:
 
 .. code-block:: python
 
-   build_directory = "src"
+   build_directory = 'src'
 
 
 ^^^^^^^^^^^^^^^^^^^^^^

lib/spack/docs/build_systems/racketpackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/build_systems/rocmpackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/build_systems/rpackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -162,28 +163,28 @@ attributes that can be used to set ``homepage``, ``url``, ``list_url``, and
 
 .. code-block:: python
 
-   cran = "caret"
+   cran = 'caret'
 
 is equivalent to:
 
 .. code-block:: python
 
-   homepage = "https://cloud.r-project.org/package=caret"
-   url      = "https://cloud.r-project.org/src/contrib/caret_6.0-86.tar.gz"
-   list_url = "https://cloud.r-project.org/src/contrib/Archive/caret"
+   homepage = 'https://cloud.r-project.org/package=caret'
+   url      = 'https://cloud.r-project.org/src/contrib/caret_6.0-86.tar.gz'
+   list_url = 'https://cloud.r-project.org/src/contrib/Archive/caret'
 
 Likewise, the following ``bioc`` attribute:
 
 .. code-block:: python
 
-   bioc = "BiocVersion"
+   bioc = 'BiocVersion'
 
 is equivalent to:
 
 .. code-block:: python
 
-   homepage = "https://bioconductor.org/packages/BiocVersion/"
-   git      = "https://git.bioconductor.org/packages/BiocVersion"
+   homepage = 'https://bioconductor.org/packages/BiocVersion/'
+   git      = 'https://git.bioconductor.org/packages/BiocVersion'
 
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -199,7 +200,7 @@ base class contains:
 
 .. code-block:: python
 
-   extends("r")
+   extends('r')
 
 
 Take a close look at the homepage for ``caret``. If you look at the
@@ -208,7 +209,7 @@ You should add this to your package like so:
 
 .. code-block:: python
 
-   depends_on("r@3.2.0:", type=("build", "run"))
+   depends_on('r@3.2.0:', type=('build', 'run'))
 
 
 ^^^^^^^^^^^^^^
@@ -226,7 +227,7 @@ and list all of their dependencies in the following sections:
 * LinkingTo
 
 As far as Spack is concerned, all 3 of these dependency types
-correspond to ``type=("build", "run")``, so you don't have to worry
+correspond to ``type=('build', 'run')``, so you don't have to worry
 about the details. If you are curious what they mean,
 https://github.com/spack/spack/issues/2951 has a pretty good summary:
 
@@ -329,7 +330,7 @@ the dependency:
 
 .. code-block:: python
 
-   depends_on("r-lattice@0.20:", type=("build", "run"))
+   depends_on('r-lattice@0.20:', type=('build', 'run'))
 
 
 ^^^^^^^^^^^^^^^^^^
@@ -360,20 +361,20 @@ like so:
 .. code-block:: python
 
    def configure_args(self):
-       mpi_name = self.spec["mpi"].name
+       mpi_name = self.spec['mpi'].name
 
        # The type of MPI. Supported values are:
        # OPENMPI, LAM, MPICH, MPICH2, or CRAY
-       if mpi_name == "openmpi":
-           Rmpi_type = "OPENMPI"
-       elif mpi_name == "mpich":
-           Rmpi_type = "MPICH2"
+       if mpi_name == 'openmpi':
+           Rmpi_type = 'OPENMPI'
+       elif mpi_name == 'mpich':
+           Rmpi_type = 'MPICH2'
        else:
-           raise InstallError("Unsupported MPI type")
+           raise InstallError('Unsupported MPI type')
 
        return [
-           "--with-Rmpi-type={0}".format(Rmpi_type),
-           "--with-mpi={0}".format(spec["mpi"].prefix),
+           '--with-Rmpi-type={0}'.format(Rmpi_type),
+           '--with-mpi={0}'.format(spec['mpi'].prefix),
        ]
 
 

lib/spack/docs/build_systems/rubypackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -83,8 +84,8 @@ The ``*.gemspec`` file may contain something like:
 
 .. code-block:: ruby
 
-   summary = "An implementation of the AsciiDoc text processor and publishing toolchain"
-   description = "A fast, open source text processor and publishing toolchain for converting AsciiDoc content to HTML 5, DocBook 5, and other formats."
+   summary = 'An implementation of the AsciiDoc text processor and publishing toolchain'
+   description = 'A fast, open source text processor and publishing toolchain for converting AsciiDoc content to HTML 5, DocBook 5, and other formats.'
 
 
 Either of these can be used for the description of the Spack package.
@@ -97,7 +98,7 @@ The ``*.gemspec`` file may contain something like:
 
 .. code-block:: ruby
 
-   homepage = "https://asciidoctor.org"
+   homepage = 'https://asciidoctor.org'
 
 
 This should be used as the official homepage of the Spack package.
@@ -111,21 +112,21 @@ the base class contains:
 
 .. code-block:: python
 
-   extends("ruby")
+   extends('ruby')
 
 
 The ``*.gemspec`` file may contain something like:
 
 .. code-block:: ruby
 
-   required_ruby_version = ">= 2.3.0"
+   required_ruby_version = '>= 2.3.0'
 
 
 This can be added to the Spack package using:
 
 .. code-block:: python
 
-   depends_on("ruby@2.3.0:", type=("build", "run"))
+   depends_on('ruby@2.3.0:', type=('build', 'run'))
 
 
 ^^^^^^^^^^^^^^^^^

lib/spack/docs/build_systems/sconspackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -48,14 +49,14 @@ following phases:
 #. ``install`` - install the package
 
 Package developers often add unit tests that can be invoked with
-``scons test`` or ``scons check``. Spack provides a ``build_test`` method
+``scons test`` or ``scons check``. Spack provides a ``test`` method
 to handle this. Since we don't know which one the package developer
-chose, the ``build_test`` method does nothing by default, but can be easily
+chose, the ``test`` method does nothing by default, but can be easily
 overridden like so:
 
 .. code-block:: python
 
-   def build_test(self):
+   def test(self):
        scons("check")
 
 

lib/spack/docs/build_systems/sippackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -123,7 +124,7 @@ are wrong, you can provide the names yourself by overriding
 
 .. code-block:: python
 
-   import_modules = ["PyQt5"]
+   import_modules = ['PyQt5']
 
 
 These tests often catch missing dependencies and non-RPATHed

lib/spack/docs/build_systems/sourceforgepackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/build_systems/wafpackage.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -62,8 +63,8 @@ run package-specific unit tests.
 .. code-block:: python
 
    def installtest(self):
-       with working_dir("test"):
-           pytest = which("py.test")
+       with working_dir('test'):
+           pytest = which('py.test')
            pytest()
 
 
@@ -92,7 +93,7 @@ the following dependency automatically:
 
 .. code-block:: python
 
-   depends_on("python@2.5:", type="build")
+   depends_on('python@2.5:', type='build')
 
 
 Waf only supports Python 2.5 and up.
@@ -112,7 +113,7 @@ phase, you can use:
        args = []
 
        if self.run_tests:
-           args.append("--test")
+           args.append('--test')
 
        return args
 

lib/spack/docs/chain.rst

@@ -1,17 +1,17 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
 .. chain:
 
-=============================================
-Chaining Spack Installations (upstreams.yaml)
-=============================================
+============================
+Chaining Spack Installations
+============================
 
 You can point your Spack installation to another installation to use any
 packages that are installed there. To register the other Spack instance,
-you can add it as an entry to ``upstreams.yaml`` at any of the
-:ref:`configuration-scopes`:
+you can add it as an entry to ``upstreams.yaml``:
 
 .. code-block:: yaml
 
@@ -22,8 +22,7 @@ you can add it as an entry to ``upstreams.yaml`` at any of the
       install_tree: /path/to/another/spack/opt/spack
 
 ``install_tree`` must point to the ``opt/spack`` directory inside of the
-Spack base directory, or the location of the ``install_tree`` defined
-in :ref:`config.yaml <config-yaml>`.
+Spack base directory.
 
 Once the upstream Spack instance has been added, ``spack find`` will
 automatically check the upstream instance when querying installed packages,

lib/spack/docs/conf.py

@@ -1,4 +1,5 @@
-# Copyright Spack Project Developers. See COPYRIGHT file for details.
+# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+# Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -198,31 +199,22 @@ def setup(sphinx):
     ("py:class", "contextlib.contextmanager"),
     ("py:class", "module"),
     ("py:class", "_io.BufferedReader"),
-    ("py:class", "_io.BytesIO"),
     ("py:class", "unittest.case.TestCase"),
     ("py:class", "_frozen_importlib_external.SourceFileLoader"),
     ("py:class", "clingo.Control"),
     ("py:class", "six.moves.urllib.parse.ParseResult"),
     ("py:class", "TextIO"),
     ("py:class", "hashlib._Hash"),
-    ("py:class", "concurrent.futures._base.Executor"),
     # Spack classes that are private and we don't want to expose
     ("py:class", "spack.provider_index._IndexBase"),
     ("py:class", "spack.repo._PrependFileLoader"),
-    ("py:class", "spack.build_systems._checks.BuilderWithDefaults"),
+    ("py:class", "spack.build_systems._checks.BaseBuilder"),
     # Spack classes that intersphinx is unable to resolve
     ("py:class", "spack.version.StandardVersion"),
     ("py:class", "spack.spec.DependencySpec"),
-    ("py:class", "spack.spec.ArchSpec"),
     ("py:class", "spack.spec.InstallStatus"),
     ("py:class", "spack.spec.SpecfileReaderBase"),
     ("py:class", "spack.install_test.Pb"),
-    ("py:class", "spack.filesystem_view.SimpleFilesystemView"),
-    ("py:class", "spack.traverse.EdgeAndDepth"),
-    ("py:class", "archspec.cpu.microarchitecture.Microarchitecture"),
-    ("py:class", "spack.compiler.CompilerCache"),
-    # TypeVar that is not handled correctly
-    ("py:class", "llnl.util.lang.T"),
 ]
 
 # The reST default role (used for this markup: `text`) to use for all documents.

lib/spack/docs/config_yaml.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -25,23 +26,14 @@ These settings can be overridden in ``etc/spack/config.yaml`` or
 The location where Spack will install packages and their dependencies.
 Default is ``$spack/opt/spack``.
 
----------------
-``projections``
----------------
+---------------------------------------------------
+``install_hash_length`` and ``install_path_scheme``
+---------------------------------------------------
 
-.. warning::
-
-   Modifying projections of the install tree is strongly discouraged.
-
-By default Spack installs all packages into a unique directory relative to the install
-tree root with the following layout:
-
-.. code-block::
-
-   {architecture}/{compiler.name}-{compiler.version}/{name}-{version}-{hash}
-
-In very rare cases, it may be necessary to reduce the length of this path. For example,
-very old versions of the Intel compiler are known to segfault when input paths are too long:
+The default Spack installation path can be very long and can create problems
+for scripts with hardcoded shebangs. Additionally, when using the Intel
+compiler, and if there is also a long list of dependencies, the compiler may
+segfault. If you see the following:
 
      .. code-block:: console
 
@@ -49,25 +41,36 @@ very old versions of the Intel compiler are known to segfault when input paths a
        ** Segmentation violation signal raised. **
        Access violation or stack overflow. Please contact Intel Support for assistance.
 
-Another case is Python and R packages with many runtime dependencies, which can result
-in very large ``PYTHONPATH`` and ``R_LIBS`` environment variables. This can cause the
-``execve`` system call to fail with ``E2BIG``, preventing processes from starting.
+it may be because variables containing dependency specs may be too long. There
+are two parameters to help with long path names. Firstly, the
+``install_hash_length`` parameter can set the length of the hash in the
+installation path from 1 to 32. The default path uses the full 32 characters.
 
-For this reason, Spack allows users to modify the installation layout through custom
-projections. For example
+Secondly, it is also possible to modify the entire installation
+scheme. By default Spack uses
+``{architecture}/{compiler.name}-{compiler.version}/{name}-{version}-{hash}``
+where the tokens that are available for use in this directive are the
+same as those understood by the :meth:`~spack.spec.Spec.format`
+method. Using this parameter it is possible to use a different package
+layout or reduce the depth of the installation paths. For example
 
      .. code-block:: yaml
 
        config:
-         install_tree:
-           root: $spack/opt/spack
-           projections:
-             all: "{name}/{version}/{hash:16}"
+         install_path_scheme: '{name}/{version}/{hash:7}'
 
-would install packages into sub-directories using only the package name, version and a
-hash length of 16 characters.
+would install packages into sub-directories using only the package
+name, version and a hash length of 7 characters.
 
-Notice that reducing the hash length increases the likelihood of hash collisions.
+When using either parameter to set the hash length it only affects the
+representation of the hash in the installation directory. You
+should be aware that the smaller the hash length the more likely
+naming conflicts will occur. These parameters are independent of those
+used to configure module names.
+
+.. warning:: Modifying the installation hash length or path scheme after
+   packages have been installed will prevent Spack from being
+   able to find the old installation directories.
 
 --------------------
 ``build_stage``
@@ -142,25 +145,6 @@ hosts when making ``ssl`` connections.  Set to ``false`` to disable, and
 tools like ``curl`` will use their ``--insecure`` options.  Disabling
 this can expose you to attacks.  Use at your own risk.
 
---------------------
-``ssl_certs``
---------------------
-
-Path to custom certificats for SSL verification. The value can be a 
-filesytem path, or an environment variable that expands to an absolute file path.
-The default value is set to the environment variable ``SSL_CERT_FILE``
-to use the same syntax used by many other applications that automatically
-detect custom certificates.
-When ``url_fetch_method:curl`` the ``config:ssl_certs`` should resolve to
-a single file.  Spack will then set the environment variable ``CURL_CA_BUNDLE``
-in the subprocess calling ``curl``.
-If ``url_fetch_method:urllib`` then files and directories are supported i.e. 
-``config:ssl_certs:$SSL_CERT_FILE`` or ``config:ssl_certs:$SSL_CERT_DIR``
-will work.
-In all cases the expanded path must be absolute for Spack to use the certificates.
-Certificates relative to an environment can be created by prepending the path variable
-with the Spack configuration variable``$env``.
-
 --------------------
 ``checksum``
 --------------------

lib/spack/docs/configuration.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -72,12 +73,9 @@ are six configuration scopes. From lowest to highest:
    Spack instance per project) or for site-wide settings on a multi-user
    machine (e.g., for a common Spack instance).
 
-#. **plugin**: Read from a Python project's entry points. Settings here affect
-   all instances of Spack running with the same Python installation.  This scope takes higher precedence than site, system, and default scopes.
-
 #. **user**: Stored in the home directory: ``~/.spack/``. These settings
    affect all instances of Spack and take higher precedence than site,
-   system, plugin, or defaults scopes.
+   system, or defaults scopes.
 
 #. **custom**: Stored in a custom directory specified by ``--config-scope``.
    If multiple scopes are listed on the command line, they are ordered
@@ -198,45 +196,6 @@ with MPICH. You can create different configuration scopes for use with
                mpi: [mpich]
 
 
-.. _plugin-scopes:
-
-^^^^^^^^^^^^^
-Plugin scopes
-^^^^^^^^^^^^^
-
-.. note::
-   Python version >= 3.8 is required to enable plugin configuration.
-
-Spack can be made aware of configuration scopes that are installed as part of a python package.  To do so, register a function that returns the scope's path to the ``"spack.config"`` entry point.  Consider the Python package ``my_package`` that includes Spack configurations:
-
-.. code-block:: console
-
-  my-package/
-  ├── src
-  │   ├── my_package
-  │   │   ├── __init__.py
-  │   │   └── spack/
-  │   │   │   └── config.yaml
-  └── pyproject.toml
-
-adding the following to ``my_package``'s ``pyproject.toml`` will make ``my_package``'s ``spack/`` configurations visible to Spack when ``my_package`` is installed:
-
-.. code-block:: toml
-
-   [project.entry_points."spack.config"]
-   my_package = "my_package:get_config_path"
-
-The function ``my_package.get_extension_path`` in ``my_package/__init__.py`` might look like
-
-.. code-block:: python
-
-   import importlib.resources
-
-   def get_config_path():
-       dirname = importlib.resources.files("my_package").joinpath("spack")
-       if dirname.exists():
-           return str(dirname)
-
 .. _platform-scopes:
 
 ------------------------
@@ -280,15 +239,13 @@ When spack queries for configuration parameters, it searches in
 higher-precedence scopes first. So, settings in a higher-precedence file
 can override those with the same key in a lower-precedence one. For
 list-valued settings, Spack *prepends* higher-precedence settings to
-lower-precedence settings. Completely ignoring lower-precedence configuration
+lower-precedence settings. Completely ignoring higher-level configuration
 options is supported with the ``::`` notation for keys (see
 :ref:`config-overrides` below).
 
-There are also special notations for string concatenation and precendense override:
-
-* ``+:`` will force *prepending* strings or lists. For lists, this is the default behavior.
-* ``-:`` works similarly, but for *appending* values.
-
+There are also special notations for string concatenation and precendense override.
+Using the ``+:`` notation  can be used to force *prepending* strings or lists. For lists, this is identical
+to the default behavior. Using the ``-:`` works similarly, but for *appending* values.
 :ref:`config-prepend-append`
 
 ^^^^^^^^^^^
@@ -510,7 +467,6 @@ Spack understands over a dozen special variables. These are:
 * ``$target_family``. The target family for the current host, as
   detected by ArchSpec. E.g. ``x86_64`` or ``aarch64``.
 * ``$date``: the current date in the format YYYY-MM-DD
-* ``$spack_short_version``: the Spack version truncated to the first components.
 
 
 Note that, as with shell variables, you can write these as ``$varname``

lib/spack/docs/containers.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -8,98 +9,24 @@
 Container Images
 ================
 
-Spack :ref:`environments` can easily be turned into container images. This page
-outlines two ways in which this can be done:
-
-1. By installing the environment on the host system, and copying the installations
-   into the container image. This approach does not require any tools like Docker
-   or Singularity to be installed.
-2. By generating a Docker or Singularity recipe that can be used to build the
-   container image. In this approach, Spack builds the software inside the
-   container runtime, not on the host system.
-
-The first approach is easiest if you already have an installed environment,
-the second approach gives more control over the container image.
-
----------------------------
-From existing installations
----------------------------
-
-If you already have a Spack environment installed on your system, you can
-share the binaries as an OCI compatible container image. To get started you
-just have to configure and OCI registry and run ``spack buildcache push``.
-
-.. code-block:: console
-  
-   # Create and install an environment in the current directory
-   spack env create -d .
-   spack -e . add pkg-a pkg-b
-   spack -e . install
-
-   # Configure the registry
-   spack -e . mirror add --oci-username-variable REGISTRY_USER \
-                         --oci-password-variable REGISTRY_TOKEN \
-                        container-registry oci://example.com/name/image
-
-   # Push the image (do set REGISTRY_USER and REGISTRY_TOKEN)
-   spack -e . buildcache push --update-index --base-image ubuntu:22.04 --tag my_env container-registry
-
-The resulting container image can then be run as follows:
-
-.. code-block:: console
-
-   $ docker run -it example.com/name/image:my_env
-
-The image generated by Spack consists of the specified base image with each package from the
-environment as a separate layer on top. The image is minimal by construction, it only contains the
-environment roots and its runtime dependencies.
-
-.. note::
-
-  When using registries like GHCR and Docker Hub, the ``--oci-password`` flag is not
-  the password for your account, but a personal access token you need to generate separately.
-
-The specified ``--base-image`` should have a libc that is compatible with the host system.
-For example if your host system is Ubuntu 20.04, you can use ``ubuntu:20.04``, ``ubuntu:22.04``
-or newer: the libc in the container image must be at least the version of the host system,
-assuming ABI compatibility. It is also perfectly fine to use a completely different
-Linux distribution as long as the libc is compatible.
-
-For convenience, Spack also turns the OCI registry into a :ref:`build cache <binary_caches_oci>`,
-so that future ``spack install`` of the environment will simply pull the binaries from the
-registry instead of doing source builds. The flag ``--update-index`` is needed to make Spack
-take the build cache into account when concretizing.
-
-.. note::
-
-  When generating container images in CI, the approach above is recommended when CI jobs
-  already run in a sandboxed environment. You can simply use ``spack`` directly
-  in the CI job and push the resulting image to a registry. Subsequent CI jobs should
-  run faster because Spack can install from the same registry instead of rebuilding from
-  sources.
-
----------------------------------------------
-Generating recipes for Docker and Singularity
----------------------------------------------
-
-Apart from copying existing installations into container images, Spack can also
-generate recipes for container images. This is useful if you want to run Spack
-itself in a sandboxed environment instead of on the host system.
-
-Since recipes need a little bit more boilerplate than
+Spack :ref:`environments` are a great tool to create container images, but
+preparing one that is suitable for production requires some more boilerplate
+than just:
 
 .. code-block:: docker
 
    COPY spack.yaml /environment
    RUN spack -e /environment install
 
-Spack provides a command to generate customizable recipes for container images. Customizations
-include minimizing the size of the image, installing packages in the base image using the system
-package manager, and setting up a proper entrypoint to run the image.
+Additional actions may be needed to minimize the size of the
+container, or to update the system software that is installed in the base
+image, or to set up a proper entrypoint to run the image. These tasks are
+usually both necessary and repetitive, so Spack comes with a command
+to generate recipes for container images starting from a ``spack.yaml``.
 
-~~~~~~~~~~~~~~~~~~~~
+--------------------
 A Quick Introduction
-~~~~~~~~~~~~~~~~~~~~
+--------------------
 
 Consider having a Spack environment like the following:
 
@@ -110,8 +37,8 @@ Consider having a Spack environment like the following:
      - gromacs+mpi
      - mpich
 
-Producing a ``Dockerfile`` from it is as simple as changing directories to
-where the ``spack.yaml`` file is stored and running the following command:
+Producing a ``Dockerfile`` from it is as simple as moving to the directory
+where the ``spack.yaml`` file is stored and giving the following command:
 
 .. code-block:: console
 
@@ -177,9 +104,9 @@ configuration are discussed in details in the sections below.
 
 .. _container_spack_images:
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+--------------------------
 Spack Images on Docker Hub
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+--------------------------
 
 Docker images with Spack preinstalled and ready to be used are
 built when a release is tagged, or nightly on ``develop``. The images
@@ -195,18 +122,21 @@ The OS that are currently supported are summarized in the table below:
    * - Operating System
      - Base Image
      - Spack Image
+   * - Ubuntu 18.04
+     - ``ubuntu:18.04``
+     - ``spack/ubuntu-bionic``
    * - Ubuntu 20.04
      - ``ubuntu:20.04``
      - ``spack/ubuntu-focal``
    * - Ubuntu 22.04
      - ``ubuntu:22.04``
      - ``spack/ubuntu-jammy``
-   * - Ubuntu 24.04
-     - ``ubuntu:24.04``
-     - ``spack/ubuntu-noble``
-   * - CentOS Stream9
-     - ``quay.io/centos/centos:stream9``
-     - ``spack/centos-stream9``
+   * - CentOS 7
+     - ``centos:7``
+     - ``spack/centos7``
+   * - CentOS Stream
+     - ``quay.io/centos/centos:stream``
+     - ``spack/centos-stream``
    * - openSUSE Leap
      - ``opensuse/leap``
      - ``spack/leap15``
@@ -225,12 +155,12 @@ The OS that are currently supported are summarized in the table below:
    * - Rocky Linux 9
      - ``rockylinux:9``
      - ``spack/rockylinux9``
-   * - Fedora Linux 39
-     - ``fedora:39``
-     - ``spack/fedora39``
-   * - Fedora Linux 40
-     - ``fedora:40``
-     - ``spack/fedora40``
+   * - Fedora Linux 37
+     - ``fedora:37``
+     - ``spack/fedora37``
+   * - Fedora Linux 38
+     - ``fedora:38``
+     - ``spack/fedora38``
 
 
 
@@ -246,9 +176,9 @@ by Spack use them as default base images for their ``build`` stage,
 even though handles to use custom base images provided by users are
 available to accommodate complex use cases.
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Configuring the Container Recipe
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+---------------------------------
+Creating Images From Environments
+---------------------------------
 
 Any Spack Environment can be used for the automatic generation of container
 recipes. Sensible defaults are provided for things like the base image or the
@@ -289,18 +219,18 @@ under the ``container`` attribute of environments:
 
 A detailed description of the options available can be found in the :ref:`container_config_options` section.
 
-~~~~~~~~~~~~~~~~~~~
+-------------------
 Setting Base Images
-~~~~~~~~~~~~~~~~~~~
+-------------------
 
 The ``images`` subsection is used to select both the image where
 Spack builds the software and the image where the built software
 is installed. This attribute can be set in different ways and
 which one to use depends on the use case at hand.
 
-""""""""""""""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Use Official Spack Images From Dockerhub
-""""""""""""""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 To generate a recipe that uses an official Docker image from the
 Spack organization to build the software and the corresponding official OS image
@@ -505,9 +435,9 @@ responsibility to ensure that:
 Therefore we don't recommend its use in cases that can be otherwise
 covered by the simplified mode shown first.
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+----------------------------
 Singularity Definition Files
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+----------------------------
 
 In addition to producing recipes in ``Dockerfile`` format Spack can produce
 Singularity Definition Files by just changing the value of the ``format``
@@ -528,9 +458,9 @@ attribute:
 The minimum version of Singularity required to build a SIF (Singularity Image Format)
 image from the recipes generated by Spack is ``3.5.3``.
 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+------------------------------
 Extending the Jinja2 Templates
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+------------------------------
 
 The Dockerfile and the Singularity definition file that Spack can generate are based on
 a few Jinja2 templates that are rendered according to the environment being containerized.
@@ -651,9 +581,9 @@ The recipe that gets generated contains the two extra instruction that we added
 
 .. _container_config_options:
 
-~~~~~~~~~~~~~~~~~~~~~~~
+-----------------------
 Configuration Reference
-~~~~~~~~~~~~~~~~~~~~~~~
+-----------------------
 
 The tables below describe all the configuration options that are currently supported
 to customize the generation of container recipes:
@@ -750,13 +680,13 @@ to customize the generation of container recipes:
      - Description string
      - No
 
-~~~~~~~~~~~~~~
+--------------
 Best Practices
-~~~~~~~~~~~~~~
+--------------
 
-"""
+^^^
 MPI
-"""
+^^^
 Due to the dependency on Fortran for OpenMPI, which is the spack default
 implementation, consider adding ``gfortran`` to the ``apt-get install`` list.
 
@@ -767,9 +697,9 @@ For execution on HPC clusters, it can be helpful to import the docker
 image into Singularity in order to start a program with an *external*
 MPI. Otherwise, also add ``openssh-server`` to the ``apt-get install`` list.
 
-""""
+^^^^
 CUDA
-""""
+^^^^
 Starting from CUDA 9.0, Nvidia provides minimal CUDA images based on
 Ubuntu. Please see `their instructions <https://hub.docker.com/r/nvidia/cuda/>`_.
 Avoid double-installing CUDA by adding, e.g.
@@ -788,9 +718,9 @@ to your ``spack.yaml``.
 Users will either need ``nvidia-docker`` or e.g. Singularity to *execute*
 device kernels.
 
-"""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^
 Docker on Windows and OSX
-"""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
 On Mac OS and Windows, docker runs on a hypervisor that is not allocated much
 memory by default, and some spack packages may fail to build due to lack of

lib/spack/docs/contribution_guide.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -183,7 +184,7 @@ Style Tests
 
 Spack uses `Flake8 <http://flake8.pycqa.org/en/latest/>`_ to test for
 `PEP 8 <https://www.python.org/dev/peps/pep-0008/>`_ conformance and
-`mypy <https://mypy.readthedocs.io/en/stable/>`_ for type checking. PEP 8 is
+`mypy <https://mypy.readthedocs.io/en/stable/>` for type checking. PEP 8 is
 a series of style guides for Python that provide suggestions for everything
 from variable naming to indentation. In order to limit the number of PRs that
 were mostly style changes, we decided to enforce PEP 8 conformance. Your PR
@@ -315,214 +316,6 @@ documentation tests to make sure there are no errors. Documentation changes can
 in some obfuscated warning messages. If you don't understand what they mean, feel free
 to ask when you submit your PR.
 
-.. _spack-builders-and-pipelines:
-
-^^^^^^^^^
-GitLab CI
-^^^^^^^^^
-
-""""""""""""""""""
-Build Cache Stacks
-""""""""""""""""""
-
-Spack welcomes the contribution of software stacks of interest to the community. These
-stacks are used to test package recipes and generate publicly available build caches.
-Spack uses GitLab CI for managing the orchestration of build jobs.
-
-GitLab Entry Point
-~~~~~~~~~~~~~~~~~~
-
-Add stack entrypoint to the ``share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml``. There
-are two stages required for each new stack, the generation stage and the build stage.
-
-The generate stage is defined using the job template ``.generate`` configured with
-environment variables defining the name of the stack in ``SPACK_CI_STACK_NAME`` and the
-platform (``SPACK_TARGET_PLATFORM``) and architecture (``SPACK_TARGET_ARCH``) configuration,
-and the tags associated with the class of runners to build on.
-
-.. note::
-
-    The ``SPACK_CI_STACK_NAME`` must match the name of the directory containing the
-    stacks ``spack.yaml``.
-
-
-.. note::
-
-    The platform and architecture variables are specified in order to select the
-    correct configurations from the generic configurations used in Spack CI. The
-    configurations currently available are:
-
-    * ``.cray_rhel_zen4``
-    * ``.cray_sles_zen4``
-    * ``.darwin_aarch64``
-    * ``.darwin_x86_64``
-    * ``.linux_aarch64``
-    * ``.linux_icelake``
-    * ``.linux_neoverse_n1``
-    * ``.linux_neoverse_v1``
-    * ``.linux_neoverse_v2``
-    * ``.linux_skylake``
-    * ``.linux_x86_64``
-    * ``.linux_x86_64_v4``
-
-    New configurations can be added to accommodate new platforms and architectures.
-
-
-The build stage is defined as a trigger job that consumes the GitLab CI pipeline generated in
-the generate stage for this stack. Build stage jobs use the ``.build`` job template which
-handles the basic configuration.
-
-An example entry point for a new stack called ``my-super-cool-stack``
-
-.. code-block:: yaml
-
-    .my-super-cool-stack:
-      extends: [ ".linux_x86_64_v3" ]
-      variables:
-        SPACK_CI_STACK_NAME: my-super-cool-stack
-        tags: [ "all", "tags", "your", "job", "needs"]
-
-    my-super-cool-stack-generate:
-      extends: [ ".generate", ".my-super-cool-stack" ]
-      image: my-super-cool-stack-image:0.0.1
-
-    my-super-cool-stack-build:
-      extends: [ ".build", ".my-super-cool-stack" ]
-      trigger:
-        include:
-          - artifact: jobs_scratch_dir/cloud-ci-pipeline.yml
-            job: my-super-cool-stack-generate
-        strategy: depend
-      needs:
-        - artifacts: True
-          job: my-super-cool-stack-generate
-
-
-Stack Configuration
-~~~~~~~~~~~~~~~~~~~
-
-The stack configuration is a spack environment file with two additional sections added.
-Stack configurations should be located in ``share/spack/gitlab/cloud_pipelines/stacks/<stack_name>/spack.yaml``.
-
-The ``ci`` section is generally used to define stack specific mappings such as image or tags.
-For more information on what can go into the ``ci`` section refer to the docs on pipelines.
-
-The ``cdash`` section is used for defining where to upload the results of builds. Spack configures
-most of the details for posting pipeline results to
-`cdash.spack.io <https://cdash.spack.io/index.php?project=Spack+Testing>`_. The only
-requirement in the stack configuration is to define a ``build-group`` that is unique,
-this is usually the long name of the stack.
-
-An example stack that builds ``zlib``.
-
-.. code-block:: yaml
-
-    spack:
-      view: false
-      packages:
-        all:
-          require: ["%gcc", "target=x86_64_v3"]
-      specs:
-      - zlib
-
-      ci:
-        pipeline-gen
-        - build-job:
-            image: my-super-cool-stack-image:0.0.1
-
-      cdash:
-        build-group: My Super Cool Stack
-
-.. note::
-
-    The ``image`` used in the ``*-generate`` job must match exactly the ``image`` used in the ``build-job``.
-    When the images do not match the build job may fail.
-
-
-"""""""""""""""""""
-Registering Runners
-"""""""""""""""""""
-
-Contributing computational resources to Spack's CI build farm is one way to help expand the
-capabilities and offerings of the public Spack build caches. Currently, Spack utilizes linux runners
-from AWS, Google, and the University of Oregon (UO).
-
-Runners require three key peices:
-* Runner Registration Token
-* Accurate tags
-* OIDC Authentication script
-* GPG keys
-
-
-Minimum GitLab Runner Version: ``16.1.0``
-`Intallation instructions <https://docs.gitlab.com/runner/install/>`_
-
-Registration Token
-~~~~~~~~~~~~~~~~~~
-
-The first step to contribute new runners is to open an issue in the `spack infrastructure <https://github.com/spack/spack-infrastructure/issues/new?assignees=&labels=runner-registration&projects=&template=runner_registration.yml>`_
-project. This will be reported to the spack infrastructure team who will guide users through the process
-of registering new runners for Spack CI.
-
-The information needed to register a runner is the motivation for the new resources, a semi-detailed description of
-the runner, and finallly the point of contact for maintaining the software on the runner.
-
-The point of contact will then work with the infrastruture team to obtain runner registration token(s) for interacting with
-with Spack's GitLab instance. Once the runner is active, this point of contact will also be responsible for updating the
-GitLab runner software to keep pace with Spack's Gitlab.
-
-Tagging
-~~~~~~~
-
-In the initial stages of runner registration it is important to **exclude** the special tag ``spack``. This will prevent
-the new runner(s) from being picked up for production CI jobs while it is configured and evaluated. Once it is determined
-that the runner is ready for production use the ``spack`` tag will be added.
-
-Because gitlab has no concept of tag exclustion, runners that provide specialized resource also require specialized tags.
-For example, a basic CPU only x86_64 runner may have a tag ``x86_64`` associated with it. However, a runner containing an
-CUDA capable GPU may have the tag ``x86_64-cuda`` to denote that it should only be used for packages that will benefit from
-a CUDA capable resource.
-
-OIDC
-~~~~
-
-Spack runners use OIDC authentication for connecting to the appropriate AWS bucket
-which is used for coordinating the communication of binaries between build jobs. In
-order to configure OIDC authentication, Spack CI runners use a python script with minimal
-dependencies. This script can be configured for runners as seen here using the ``pre_build_script``.
-
-.. code-block:: toml
-
-    [[runners]]
-      pre_build_script = """
-      echo 'Executing Spack pre-build setup script'
-
-      for cmd in "${PY3:-}" python3 python; do
-        if command -v > /dev/null "$cmd"; then
-          export PY3="$(command -v "$cmd")"
-          break
-        fi
-      done
-
-      if [ -z "${PY3:-}" ]; then
-        echo "Unable to find python3 executable"
-        exit 1
-      fi
-
-      $PY3 -c "import urllib.request;urllib.request.urlretrieve('https://raw.githubusercontent.com/spack/spack-infrastructure/main/scripts/gitlab_runner_pre_build/pre_build.py', 'pre_build.py')"
-      $PY3 pre_build.py > envvars
-
-      . ./envvars
-      rm -f envvars
-      unset GITLAB_OIDC_TOKEN
-      """
-
-GPG Keys
-~~~~~~~~
-
-Runners that may be utilized for ``protected`` CI require the registration of an intermediate signing key that
-can be used to sign packages. For more information on package signing read :ref:`key_architecture`.
-
 --------
 Coverage
 --------

lib/spack/docs/developer_guide.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -177,8 +178,12 @@ Spec-related modules
   Contains :class:`~spack.spec.Spec`. Also implements most of the logic for concretization
   of specs.
 
-:mod:`spack.spec_parser`
-  Contains :class:`~spack.spec_parser.SpecParser` and functions related to parsing specs.
+:mod:`spack.parser`
+  Contains :class:`~spack.parser.SpecParser` and functions related to parsing specs.
+
+:mod:`spack.concretize`
+  Contains :class:`~spack.concretize.Concretizer` implementation,
+  which allows site administrators to change Spack's :ref:`concretization-policies`.
 
 :mod:`spack.version`
   Implements a simple :class:`~spack.version.Version` class with simple
@@ -332,9 +337,13 @@ inserting them at different places in the spack code base. Whenever a hook
 type triggers by way of a function call, we find all the hooks of that type,
 and run them.
 
-Spack defines hooks by way of a module in the ``lib/spack/spack/hooks`` directory.
-This module has to be registered in ``__init__.py`` so that Spack is aware of it.
-This section will cover the basic kind of hooks, and how to write them.
+Spack defines hooks by way of a module at ``lib/spack/spack/hooks`` where we can define
+types of hooks in the ``__init__.py``, and then python files in that folder
+can use hook functions. The files are automatically parsed, so if you write
+a new file for some integration (e.g., ``lib/spack/spack/hooks/myintegration.py``
+you can then write hook functions in that file that will be automatically detected,
+and run whenever your hook is called. This section will cover the basic kind
+of hooks, and how to write them.
 
 ^^^^^^^^^^^^^^
 Types of Hooks
@@ -348,23 +357,91 @@ If there is a hook that you would like and is missing, you can propose to add a
 ``pre_install(spec)``
 """""""""""""""""""""
 
-A ``pre_install`` hook is run within the install subprocess, directly before the install starts.
-It expects a single argument of a spec.
+A ``pre_install`` hook is run within an install subprocess, directly before
+the install starts. It expects a single argument of a spec, and is run in
+a multiprocessing subprocess. Note that if you see ``pre_install`` functions associated with packages these are not hooks
+as we have defined them here, but rather callback functions associated with
+a package install.
 
 
-"""""""""""""""""""""""""""""""""""""
-``post_install(spec, explicit=None)``
-"""""""""""""""""""""""""""""""""""""
+""""""""""""""""""""""
+``post_install(spec)``
+""""""""""""""""""""""
 
-A ``post_install`` hook is run within the install subprocess, directly after the install finishes,
-but before the build stage is removed and the spec is registered in the database. It expects two
-arguments: spec and an optional boolean indicating whether this spec is being installed explicitly.
+A ``post_install`` hook is run within an install subprocess, directly after
+the install finishes, but before the build stage is removed. If you
+write one of these hooks, you should expect it to accept a spec as the only
+argument. This is run in a multiprocessing subprocess. This ``post_install`` is
+also seen in packages, but in this context not related to the hooks described
+here.
 
-""""""""""""""""""""""""""""""""""""""""""""""""""""
-``pre_uninstall(spec)`` and ``post_uninstall(spec)``
-""""""""""""""""""""""""""""""""""""""""""""""""""""
 
-These hooks are currently used for cleaning up module files after uninstall.
+""""""""""""""""""""""""""
+``on_install_start(spec)``
+""""""""""""""""""""""""""
+
+This hook is run at the beginning of ``lib/spack/spack/installer.py``,
+in the install function of a ``PackageInstaller``,
+and importantly is not part of a build process, but before it. This is when
+we have just newly grabbed the task, and are preparing to install. If you
+write a hook of this type, you should provide the spec to it.
+
+.. code-block:: python
+
+    def on_install_start(spec):
+        """On start of an install, we want to...
+        """
+        print('on_install_start')
+
+
+""""""""""""""""""""""""""""
+``on_install_success(spec)``
+""""""""""""""""""""""""""""
+
+This hook is run on a successful install, and is also run inside the build
+process, akin to ``post_install``. The main difference is that this hook
+is run outside of the context of the stage directory, meaning after the
+build stage has been removed and the user is alerted that the install was
+successful. If you need to write a hook that is run on success of a particular
+phase, you should use ``on_phase_success``.
+
+""""""""""""""""""""""""""""
+``on_install_failure(spec)``
+""""""""""""""""""""""""""""
+
+This hook is run given an install failure that happens outside of the build
+subprocess, but somewhere in ``installer.py`` when something else goes wrong.
+If you need to write a hook that is relevant to a failure within a build
+process, you would want to instead use ``on_phase_failure``.
+
+
+"""""""""""""""""""""""""""
+``on_install_cancel(spec)``
+"""""""""""""""""""""""""""
+
+The same, but triggered if a spec install is cancelled for any reason.
+
+
+"""""""""""""""""""""""""""""""""""""""""""""""
+``on_phase_success(pkg, phase_name, log_file)``
+"""""""""""""""""""""""""""""""""""""""""""""""
+
+This hook is run within the install subprocess, and specifically when a phase
+successfully finishes. Since we are interested in the package, the name of
+the phase, and any output from it, we require:
+
+ - **pkg**: the package variable, which also has the attached spec at ``pkg.spec``
+ - **phase_name**: the name of the phase that was successful (e.g., configure)
+ - **log_file**: the path to the file with output, in case you need to inspect or otherwise interact with it.
+
+"""""""""""""""""""""""""""""""""""""""""""""
+``on_phase_error(pkg, phase_name, log_file)``
+"""""""""""""""""""""""""""""""""""""""""""""
+
+In the case of an error during a phase, we might want to trigger some event
+with a hook, and this is the purpose of this particular hook. Akin to
+``on_phase_success`` we require the same variables - the package that failed,
+the name of the phase, and the log file where we might find errors.
 
 
 ^^^^^^^^^^^^^^^^^^^^^^
@@ -543,11 +620,11 @@ With either interpreter you can run a single command:
 
 .. code-block:: console
 
-   $ spack python -c 'from spack.concretize import concretize_one; concretize_one("python")'
-   ...
+   $ spack python -c 'import distro; distro.linux_distribution()'
+   ('Ubuntu', '18.04', 'Bionic Beaver')
 
-   $ spack python -i ipython -c 'from spack.concretize import concretize_one; concretize_one("python")'
-   Out[1]: ...
+   $ spack python -i ipython -c 'import distro; distro.linux_distribution()'
+   Out[1]: ('Ubuntu', '18.04', 'Bionic Beaver')
 
 or a file:
 
@@ -707,27 +784,27 @@ Release branches
 ^^^^^^^^^^^^^^^^
 
 There are currently two types of Spack releases: :ref:`major releases
-<major-releases>` (``0.21.0``, ``0.22.0``, etc.) and :ref:`patch releases
-<patch-releases>` (``0.22.1``, ``0.22.2``, ``0.22.3``, etc.). Here is a
+<major-releases>` (``0.17.0``, ``0.18.0``, etc.) and :ref:`point releases
+<point-releases>` (``0.17.1``, ``0.17.2``, ``0.17.3``, etc.). Here is a
 diagram of how Spack release branches work::
 
-    o    branch: develop  (latest version, v0.23.0.dev0)
+    o    branch: develop  (latest version, v0.19.0.dev0)
     |
     o
-    | o  branch: releases/v0.22, tag: v0.22.1
+    | o  branch: releases/v0.18, tag: v0.18.1
     o |
-    | o  tag: v0.22.0
+    | o  tag: v0.18.0
     o |
     | o
     |/
     o
     |
     o
-    | o  branch: releases/v0.21, tag: v0.21.2
+    | o  branch: releases/v0.17, tag: v0.17.2
     o |
-    | o  tag: v0.21.1
+    | o  tag: v0.17.1
     o |
-    | o  tag: v0.21.0
+    | o  tag: v0.17.0
     o |
     | o
     |/
@@ -738,8 +815,8 @@ requests target ``develop``. The ``develop`` branch will report that its
 version is that of the next **major** release with a ``.dev0`` suffix.
 
 Each Spack release series also has a corresponding branch, e.g.
-``releases/v0.22`` has ``v0.22.x`` versions of Spack, and
-``releases/v0.21`` has ``v0.21.x`` versions. A major release is the first
+``releases/v0.18`` has ``0.18.x`` versions of Spack, and
+``releases/v0.17`` has ``0.17.x`` versions. A major release is the first
 tagged version on a release branch. Minor releases are back-ported from
 develop onto release branches. This is typically done by cherry-picking
 bugfix commits off of ``develop``.
@@ -769,40 +846,27 @@ for more details.
 Scheduling work for releases
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-We schedule work for **major releases** through `milestones
-<https://github.com/spack/spack/milestones>`_ and `GitHub Projects
-<https://github.com/spack/spack/projects>`_, while **patch releases** use `labels
-<https://github.com/spack/spack/labels>`_.
+We schedule work for releases by creating `GitHub projects
+<https://github.com/spack/spack/projects>`_. At any time, there may be
+several open release projects. For example, below are two releases (from
+some past version of the page linked above):
 
-There is only one milestone open at a time. Its name corresponds to the next major version, for
-example ``v0.23``. Important issues and pull requests should be assigned to this milestone by
-core developers, so that they are not forgotten at the time of release. The milestone is closed
-when the release is made, and a new milestone is created for the next major release.
+.. image:: images/projects.png
 
-Bug reports in GitHub issues are automatically labelled ``bug`` and ``triage``. Spack developers
-assign one of the labels ``impact-low``, ``impact-medium`` or ``impact-high``. This will make the
-issue appear in the `Triaged bugs <https://github.com/orgs/spack/projects/6>`_ project board.
-Important issues should be assigned to the next milestone as well, so they appear at the top of
-the project board.
+This image shows one release in progress for ``0.15.1`` and another for
+``0.16.0``. Each of these releases has a project board containing issues
+and pull requests. GitHub shows a status bar with completed work in
+green, work in progress in purple, and work not started yet in gray, so
+it's fairly easy to see progress.
 
-Spack's milestones are not firm commitments so we move work between releases frequently. If we
-need to make a release and some tasks are not yet done, we will simply move them to the next major
-release milestone, rather than delaying the release to complete them.
+Spack's project boards are not firm commitments so we move work between
+releases frequently. If we need to make a release and some tasks are not
+yet done, we will simply move them to the next minor or major release, rather
+than delaying the release to complete them.
 
-^^^^^^^^^^^^^^^^^^^^^
-Backporting bug fixes
-^^^^^^^^^^^^^^^^^^^^^
+For more on using GitHub project boards, see `GitHub's documentation
+<https://docs.github.com/en/github/managing-your-work-on-github/about-project-boards>`_.
 
-When a bug is fixed in the ``develop`` branch, it is often necessary to backport the fix to one
-(or more) of the ``release/vX.Y`` branches. Only the release manager is responsible for doing
-backports, but Spack maintainers are responsible for labelling pull requests (and issues if no bug
-fix is available yet) with ``vX.Y.Z`` labels. The label should correspond to the next patch version
-that the bug fix should be backported to.
-
-Backports are done publicly by the release manager using a pull request named ``Backports vX.Y.Z``.
-This pull request is opened from the ``backports/vX.Y.Z`` branch, targets the ``releases/vX.Y``
-branch and contains a (growing) list of cherry-picked commits from the ``develop`` branch.
-Typically there are one or two backport pull requests open at any given time.
 
 .. _major-releases:
 
@@ -810,21 +874,25 @@ Typically there are one or two backport pull requests open at any given time.
 Making major releases
 ^^^^^^^^^^^^^^^^^^^^^
 
-Assuming all required work from the milestone is completed, the steps to make the major release
-are:
+Assuming a project board has already been created and all required work
+completed, the steps to make the major release are:
 
-#. `Create a new milestone <https://github.com/spack/spack/milestones>`_ for the next major
-   release.
+#. Create two new project boards:
 
-#. `Create a new label <https://github.com/spack/spack/labels>`_ for the next patch release.
+   * One for the next major release
+   * One for the next point release
 
-#. Move any optional tasks that are not done to the next milestone.
+#. Move any optional tasks that are not done to one of the new project boards.
+
+   In general, small bugfixes should go to the next point release. Major
+   features, refactors, and changes that could affect concretization should
+   go in the next major release.
 
 #. Create a branch for the release, based on ``develop``:
 
    .. code-block:: console
 
-      $ git checkout -b releases/v0.23 develop
+      $ git checkout -b releases/v0.15 develop
 
    For a version ``vX.Y.Z``, the branch's name should be
    ``releases/vX.Y``. That is, you should create a ``releases/vX.Y``
@@ -860,8 +928,8 @@ are:
 
    Create a pull request targeting the ``develop`` branch, bumping the major
    version in ``lib/spack/spack/__init__.py`` with a ``dev0`` release segment.
-   For instance when you have just released ``v0.23.0``, set the version
-   to ``(0, 24, 0, 'dev0')`` on ``develop``.
+   For instance when you have just released ``v0.15.0``, set the version
+   to ``(0, 16, 0, 'dev0')`` on ``develop``.
 
 #. Follow the steps in :ref:`publishing-releases`.
 
@@ -870,52 +938,82 @@ are:
 #. Follow the steps in :ref:`announcing-releases`.
 
 
-.. _patch-releases:
+.. _point-releases:
 
 ^^^^^^^^^^^^^^^^^^^^^
-Making patch releases
+Making point releases
 ^^^^^^^^^^^^^^^^^^^^^
 
-To make the patch release process both efficient and transparent, we use a *backports pull request*
-which contains cherry-picked commits from the ``develop`` branch. The majority of the work is to
-cherry-pick the bug fixes, which ideally should be done as soon as they land on ``develop``:
-this ensures cherry-picking happens in order, and makes conflicts easier to resolve since the
-changes are fresh in the mind of the developer.
+Assuming a project board has already been created and all required work
+completed, the steps to make the point release are:
 
-The backports pull request is always titled ``Backports vX.Y.Z`` and is labelled ``backports``. It
-is opened from a branch named ``backports/vX.Y.Z`` and targets the ``releases/vX.Y`` branch.
+#. Create a new project board for the next point release.
 
-Whenever a pull request labelled ``vX.Y.Z`` is merged, cherry-pick the associated squashed commit
-on ``develop`` to the ``backports/vX.Y.Z`` branch. For pull requests that were rebased (or not
-squashed), cherry-pick each associated commit individually. Never force push to the
-``backports/vX.Y.Z`` branch.
+#. Move any optional tasks that are not done to the next project board.
 
-.. warning::
+#. Check out the release branch (it should already exist).
 
-   Sometimes you may **still** get merge conflicts even if you have
-   cherry-picked all the commits in order. This generally means there
-   is some other intervening pull request that the one you're trying
-   to pick depends on. In these cases, you'll need to make a judgment
-   call regarding those pull requests.  Consider the number of affected
-   files and/or the resulting differences.
+    For the ``X.Y.Z`` release, the release branch is called ``releases/vX.Y``.
+    For ``v0.15.1``, you would check out ``releases/v0.15``:
 
-   1. If the changes are small, you might just cherry-pick it.
+   .. code-block:: console
 
-   2. If the changes are large, then you may decide that this fix is not
-      worth including in a patch release, in which case you should remove
-      the label from the pull request. Remember that large, manual backports
-      are seldom the right choice for a patch release.
+      $ git checkout releases/v0.15
 
-When all commits are cherry-picked in the ``backports/vX.Y.Z`` branch, make the patch
-release as follows:
+#. If a pull request to the release branch named ``Backports vX.Y.Z`` is not already
+   in the project, create it. This pull request ought to be created as early as
+   possible when working on a release project, so that we can build the release
+   commits incrementally, and identify potential conflicts at an early stage.
 
-#. `Create a new label <https://github.com/spack/spack/labels>`_ ``vX.Y.{Z+1}`` for the next patch
-   release.
+#. Cherry-pick each pull request in the ``Done`` column of the release
+   project board onto the ``Backports vX.Y.Z`` pull request.
 
-#. Replace the label ``vX.Y.Z`` with ``vX.Y.{Z+1}`` for all PRs and issues that are not done.
+   This is **usually** fairly simple since we squash the commits from the
+   vast majority of pull requests. That means there is only one commit
+   per pull request to cherry-pick. For example, `this pull request
+   <https://github.com/spack/spack/pull/15777>`_ has three commits, but
+   they were squashed into a single commit on merge. You can see the
+   commit that was created here:
 
-#. Manually push a single commit with commit message ``Set version to vX.Y.Z`` to the
-   ``backports/vX.Y.Z`` branch, that both bumps the Spack version number and updates the changelog:
+   .. image:: images/pr-commit.png
+
+   You can easily cherry pick it like this (assuming you already have the
+   release branch checked out):
+
+   .. code-block:: console
+
+      $ git cherry-pick 7e46da7
+
+   For pull requests that were rebased (or not squashed), you'll need to
+   cherry-pick each associated commit individually.
+
+   .. warning::
+
+      It is important to cherry-pick commits in the order they happened,
+      otherwise you can get conflicts while cherry-picking. When
+      cherry-picking look at the merge date,
+      **not** the number of the pull request or the date it was opened.
+
+      Sometimes you may **still** get merge conflicts even if you have
+      cherry-picked all the commits in order. This generally means there
+      is some other intervening pull request that the one you're trying
+      to pick depends on. In these cases, you'll need to make a judgment
+      call regarding those pull requests.  Consider the number of affected
+      files and or the resulting differences.
+
+      1. If the dependency changes are small, you might just cherry-pick it,
+         too. If you do this, add the task to the release board.
+
+      2. If the changes are large, then you may decide that this fix is not
+         worth including in a point release, in which case you should remove
+         the task from the release project.
+
+      3. You can always decide to manually back-port the fix to the release
+         branch if neither of the above options makes sense, but this can
+         require a lot of work. It's seldom the right choice.
+
+#. When all the commits from the project board are cherry-picked into
+   the ``Backports vX.Y.Z`` pull request, you can push a commit to:
 
    1. Bump the version in ``lib/spack/spack/__init__.py``.
    2. Update ``CHANGELOG.md`` with a list of the changes.
@@ -924,22 +1022,20 @@ release as follows:
    release branch. See `the changelog from 0.14.1
    <https://github.com/spack/spack/commit/ff0abb9838121522321df2a054d18e54b566b44a>`_.
 
-#. Make sure CI passes on the **backports pull request**, including:
+#. Merge the ``Backports vX.Y.Z`` PR with the **Rebase and merge** strategy. This
+   is needed to keep track in the release branch of all the commits that were
+   cherry-picked.
+
+#. Make sure CI passes on the release branch, including:
 
    * Regular unit tests
    * Build tests
    * The E4S pipeline at `gitlab.spack.io <https://gitlab.spack.io>`_
 
-#. Merge the ``Backports vX.Y.Z`` PR with the **Rebase and merge** strategy. This
-   is needed to keep track in the release branch of all the commits that were
-   cherry-picked.
-
-#. Make sure CI passes on the last commit of the **release branch**.
-
-#. In the rare case you need to include additional commits in the patch release after the backports
-   PR is merged, it is best to delete the last commit ``Set version to vX.Y.Z`` from the release
-   branch with a single force push, open a new backports PR named ``Backports vX.Y.Z (2)``, and
-   repeat the process. Avoid repeated force pushes to the release branch.
+   If CI does not pass, you'll need to figure out why, and make changes
+   to the release branch until it does. You can make more commits, modify
+   or remove cherry-picked commits, or cherry-pick **more** from
+   ``develop`` to make this happen.
 
 #. Follow the steps in :ref:`publishing-releases`.
 
@@ -1014,31 +1110,25 @@ Updating `releases/latest`
 
 If the new release is the **highest** Spack release yet, you should
 also tag it as ``releases/latest``. For example, suppose the highest
-release is currently ``0.22.3``:
+release is currently ``0.15.3``:
 
-* If you are releasing ``0.22.4`` or ``0.23.0``, then you should tag
-  it with ``releases/latest``, as these are higher than ``0.22.3``.
+* If you are releasing ``0.15.4`` or ``0.16.0``, then you should tag
+  it with ``releases/latest``, as these are higher than ``0.15.3``.
 
 * If you are making a new release of an **older** major version of
-  Spack, e.g. ``0.21.4``, then you should not tag it as
+  Spack, e.g. ``0.14.4``, then you should not tag it as
   ``releases/latest`` (as there are newer major versions).
 
-To do so, first fetch the latest tag created on GitHub, since you may not have it locally:
+To tag ``releases/latest``, do this:
 
 .. code-block:: console
 
-   $ git fetch --force git@github.com:spack/spack vX.Y.Z
+   $ git checkout releases/vX.Y     # vX.Y is the new release's branch
+   $ git tag --force releases/latest
+   $ git push --force --tags
 
-Then tag ``vX.Y.Z`` as ``releases/latest`` and push the individual tag to GitHub.
-
-.. code-block:: console
-
-   $ git tag --force releases/latest vX.Y.Z
-   $ git push --force git@github.com:spack/spack releases/latest
-
-The ``--force`` argument to ``git tag`` makes ``git`` overwrite the existing ``releases/latest``
-tag with the new one. Do **not** use the ``--tags`` flag when pushing, since this will push *all*
-local tags.
+The ``--force`` argument to ``git tag`` makes ``git`` overwrite the existing
+``releases/latest`` tag with the new one.
 
 
 .. _announcing-releases:
@@ -1049,9 +1139,9 @@ Announcing a release
 
 We announce releases in all of the major Spack communication channels.
 Publishing the release takes care of GitHub. The remaining channels are
-X, Slack, and the mailing list. Here are the steps:
+Twitter, Slack, and the mailing list. Here are the steps:
 
-#. Announce the release on X.
+#. Announce the release on Twitter.
 
    * Compose the tweet on the ``@spackpm`` account per the
      ``spack-twitter`` slack channel.

lib/spack/docs/environments.rst

@@ -1,59 +1,53 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
 .. _environments:
 
-=====================================
-Environments (spack.yaml, spack.lock)
-=====================================
+=========================
+Environments (spack.yaml)
+=========================
 
-An environment is used to group a set of specs intended for some purpose
-to be built, rebuilt, and deployed in a coherent fashion. Environments
-define aspects of the installation of the software, such as:
+An environment is used to group together a set of specs for the
+purpose of building, rebuilding and deploying in a coherent fashion.
+Environments provide a number of advantages over the *à la carte*
+approach of building and loading individual Spack modules:
 
-#. *which* specs to install;
-#. *how* those specs are configured; and
-#. *where* the concretized software will be installed.
-
-Aggregating this information into an environment for processing has advantages
-over the *à la carte* approach of building and loading individual Spack modules.
-
-With environments, you concretize, install, or load (activate) all of the
-specs with a single command. Concretization fully configures the specs
-and dependencies of the environment in preparation for installing the
-software. This is a more robust solution than ad-hoc installation scripts.
-And you can share an environment or even re-use it on a different computer.
-
-Environment definitions, especially *how* specs are configured, allow the
-software to remain stable and repeatable even when Spack packages are upgraded. Changes are only picked up when the environment is explicitly re-concretized.
-
-Defining *where* specs are installed supports a filesystem view of the
-environment. Yet Spack maintains a single installation of the software that
-can be re-used across multiple environments.
-
-Activating an environment determines *when* all of the associated (and
-installed) specs are loaded so limits the software loaded to those specs
-actually needed by the environment. Spack can even generate a script to
-load all modules related to an environment.
+#. Environments separate the steps of (a) choosing what to
+   install, (b) concretizing, and (c) installing.  This allows
+   Environments to remain stable and repeatable, even if Spack packages
+   are upgraded: specs are only re-concretized when the user
+   explicitly asks for it.  It is even possible to reliably
+   transport environments between different computers running
+   different versions of Spack!
+#. Environments allow several specs to be built at once; a more robust
+   solution than ad-hoc scripts making multiple calls to ``spack
+   install``.
+#. An Environment that is built as a whole can be loaded as a whole
+   into the user environment. An Environment can be built to maintain
+   a filesystem view of its packages, and the environment can load
+   that view into the user environment at activation time. Spack can
+   also generate a script to load all modules related to an
+   environment.
 
 Other packaging systems also provide environments that are similar in
 some ways to Spack environments; for example, `Conda environments
 <https://conda.io/docs/user-guide/tasks/manage-environments.html>`_ or
 `Python Virtual Environments
 <https://docs.python.org/3/tutorial/venv.html>`_.  Spack environments
-provide some distinctive features though:
+provide some distinctive features:
 
 #. A spec installed "in" an environment is no different from the same
-   spec installed anywhere else in Spack.
-#. Spack environments may contain more than one spec of the same
+   spec installed anywhere else in Spack.  Environments are assembled
+   simply by collecting together a set of specs.
+#. Spack Environments may contain more than one spec of the same
    package.
 
 Spack uses a "manifest and lock" model similar to `Bundler gemfiles
-<https://bundler.io/man/gemfile.5.html>`_ and other package managers.
-The environment's user input file (or manifest), is named ``spack.yaml``.
-The lock file, which contains the fully configured and concretized specs,
-is named ``spack.lock``.
+<https://bundler.io/man/gemfile.5.html>`_ and other package
+managers. The user input file is named ``spack.yaml`` and the lock
+file is named ``spack.lock``
 
 .. _environments-using:
 
@@ -74,73 +68,55 @@ An environment is created by:
 
    $ spack env create myenv
 
-The directory ``$SPACK_ROOT/var/spack/environments/myenv`` is created
-to manage the environment.
+Spack then creates the directory ``var/spack/environments/myenv``.
 
 .. note::
 
-   All managed environments by default are stored in the
-   ``$SPACK_ROOT/var/spack/environments`` folder. This location can be changed
-   by setting the ``environments_root`` variable in ``config.yaml``.
+   All managed environments by default are stored in the ``var/spack/environments`` folder.
+   This location can be changed by setting the ``environments_root`` variable in ``config.yaml``.
 
-Spack creates the file ``spack.yaml``, hidden directory ``.spack-env``, and
-``spack.lock`` file under ``$SPACK_ROOT/var/spack/environments/myenv``. User
-interaction occurs through the ``spack.yaml`` file and the Spack commands
-that affect it. Metadata and, by default, the view are stored in the
-``.spack-env`` directory. When the environment is concretized, Spack creates
-the ``spack.lock`` file with the fully configured specs and dependencies for
+In the ``var/spack/environments/myenv`` directory, Spack creates the
+file ``spack.yaml`` and the hidden directory ``.spack-env``.
+
+Spack stores metadata in the ``.spack-env`` directory. User
+interaction will occur through the ``spack.yaml`` file and the Spack
+commands that affect it. When the environment is concretized, Spack
+will create a file ``spack.lock`` with the concrete information for
 the environment.
 
-The ``.spack-env`` subdirectory also contains:
+In addition to being the default location for the view associated with
+an Environment, the ``.spack-env`` directory also contains:
 
-  * ``repo/``: A subdirectory acting as the repo consisting of the Spack
-    packages used in the environment. It allows the environment to build
-    the same, in theory, even on different versions of Spack with different
+  * ``repo/``: A repo consisting of the Spack packages used in this
+    environment.  This allows the environment to build the same, in
+    theory, even on different versions of Spack with different
     packages!
-  * ``logs/``: A subdirectory containing the build logs for the packages
-    in this environment.
+  * ``logs/``: A directory containing the build logs for the packages
+    in this Environment.
 
-Spack Environments can also be created from either the user input, or
-manifest, file or the lockfile. Create an environment from a manifest using:
+Spack Environments can also be created from either a manifest file
+(usually but not necessarily named, ``spack.yaml``) or a lockfile.
+To create an Environment from a manifest:
 
 .. code-block:: console
 
    $ spack env create myenv spack.yaml
 
-The resulting environment is guaranteed to have the same root specs as
-the original but may concretize differently in the presence of different
-explicit or default configuration settings (e.g., a different version of
-Spack or for a different user account).
-
-Environments created from a manifest will copy any included configs
-from relative paths inside the environment. Relative paths from
-outside the environment will cause errors, and absolute paths will be
-kept absolute. For example, if ``spack.yaml`` includes:
-
-.. code-block:: yaml
-
-   spack:
-     include: [./config.yaml]
-
-then the created environment will have its own copy of the file
-``config.yaml`` copied from the location in the original environment.
-
-Create an environment from a ``spack.lock`` file using:
+To create an Environment from a ``spack.lock`` lockfile:
 
 .. code-block:: console
 
    $ spack env create myenv spack.lock
 
-The resulting environment, when on the same or a compatible machine, is
-guaranteed to initially have the same concrete specs as the original.
+Either of these commands can also take a full path to the
+initialization file.
 
-.. note::
-
-   Environment creation also accepts a full path to the file.
-
-   If the path is not under the ``$SPACK_ROOT/var/spack/environments``
-   directory then the source is referred to as an
-   :ref:`independent environment <independent_environments>`.
+A Spack Environment created from a ``spack.yaml`` manifest is
+guaranteed to have the same root specs as the original Environment,
+but may concretize differently. A Spack Environment created from a
+``spack.lock`` lockfile is guaranteed to have the same concrete specs
+as the original Environment. Either may obviously then differ as the
+user modifies it.
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 Activating an Environment
@@ -153,7 +129,7 @@ To activate an environment, use the following command:
    $ spack env activate myenv
 
 By default, the ``spack env activate`` will load the view associated
-with the environment into the user environment. The ``-v,
+with the Environment into the user environment. The ``-v,
 --with-view`` argument ensures this behavior, and the ``-V,
 --without-view`` argument activates the environment without changing
 the user environment variables.
@@ -166,20 +142,6 @@ user's prompt to begin with the environment name in brackets.
    $ spack env activate -p myenv
    [myenv] $ ...
 
-The ``activate`` command can also be used to create a new environment, if it is
-not already defined, by adding the ``--create`` flag. Managed and independent
-environments can both be created using the same flags that `spack env create`
-accepts.  If an environment already exists then spack will simply activate it
-and ignore the create-specific flags.
-
-.. code-block:: console
-
-   $ spack env activate --create -p myenv
-   # ...
-   # [creates if myenv does not exist yet]
-   # ...
-   [myenv] $ ...
-
 To deactivate an environment, use the command:
 
 .. code-block:: console
@@ -195,50 +157,34 @@ or the shortcut alias
 If the environment was activated with its view, deactivating the
 environment will remove the view from the user environment.
 
-.. _independent_environments:
+^^^^^^^^^^^^^^^^^^^^^^
+Anonymous Environments
+^^^^^^^^^^^^^^^^^^^^^^
 
-^^^^^^^^^^^^^^^^^^^^^^^^
-Independent Environments
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Independent environments can be located in any directory outside of Spack.
-
-.. note::
-
-   When uninstalling packages, Spack asks the user to confirm the removal of packages
-   that are still used in a managed environment. This is not the case for independent
-   environments.
-
-To create an independent environment, use one of the following commands:
+Any directory can be treated as an environment if it contains a file
+``spack.yaml``. To load an anonymous environment, use:
 
 .. code-block:: console
 
-   $ spack env create --dir my_env
-   $ spack env create ./my_env
+   $ spack env activate -d /path/to/directory
 
-As a shorthand, you can also create an independent environment upon activation if it does not
-already exist:
+Anonymous specs can be created in place using the command:
 
 .. code-block:: console
 
-   $ spack env activate --create ./my_env
+   $ spack env create -d .
 
-For convenience, Spack can also place an independent environment in a temporary directory for you:
+In this case Spack simply creates a ``spack.yaml`` file in the requested
+directory.
 
-.. code-block:: console
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Environment Sensitive Commands
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-   $ spack env activate --temp
-
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-Environment-Aware Commands
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Spack commands are environment-aware. For example, the ``find``
-command shows only the specs in the active environment if an
-environment has been activated. Otherwise it shows all specs in
-the Spack instance. The same rule applies to the ``install`` and
-``uninstall`` commands.
+Spack commands are environment sensitive. For example, the ``find``
+command shows only the specs in the active Environment if an
+Environment has been activated. Similarly, the ``install`` and
+``uninstall`` commands act on the active environment.
 
 .. code-block:: console
 
@@ -283,33 +229,32 @@ the Spack instance. The same rule applies to the ``install`` and
 
 
 Note that when we installed the abstract spec ``zlib@1.2.8``, it was
-presented as a root of the environment. All explicitly installed
-packages will be listed as roots of the environment.
+presented as a root of the Environment. All explicitly installed
+packages will be listed as roots of the Environment.
 
 All of the Spack commands that act on the list of installed specs are
-environment-aware in this way, including ``install``,
-``uninstall``, ``find``, ``extensions``, etcetera. In the
+Environment-sensitive in this way, including ``install``,
+``uninstall``, ``find``, ``extensions``, and more. In the
 :ref:`environment-configuration` section we will discuss
-environment-aware commands further.
+Environment-sensitive commands further.
 
 ^^^^^^^^^^^^^^^^^^^^^
 Adding Abstract Specs
 ^^^^^^^^^^^^^^^^^^^^^
 
-An abstract spec is the user-specified spec before Spack applies
-defaults or dependency information.
+An abstract spec is the user-specified spec before Spack has applied
+any defaults or dependency information.
 
-Users can add abstract specs to an environment using the ``spack add``
-command. The most important component of an environment is a list of
+Users can add abstract specs to an Environment using the ``spack add``
+command. The most important component of an Environment is a list of
 abstract specs.
 
-Adding a spec adds it as a root spec of the environment in the user
-input file (``spack.yaml``). It does not affect the concrete specs
-in the lock file (``spack.lock``) and it does not install the spec.
+Adding a spec adds to the manifest (the ``spack.yaml`` file), which is
+used to define the roots of the Environment, but does not affect the
+concrete specs in the lockfile, nor does it install the spec.
 
-The ``spack add`` command is environment-aware. It adds the spec to the
-currently active environment. An error is generated if there isn't an
-active environment. All environment-aware commands can also
+The ``spack add`` command is environment aware. It adds to the
+currently active environment. All environment aware commands can also
 be called using the ``spack -e`` flag to specify the environment.
 
 .. code-block:: console
@@ -329,11 +274,11 @@ or
 Concretizing
 ^^^^^^^^^^^^
 
-Once user specs have been added to an environment, they can be concretized.
-There are three different modes of operation to concretize an environment,
-explained in detail in :ref:`environments_concretization_config`.
-Regardless of which mode of operation is chosen, the following
-command will ensure all of the root specs are concretized according to the
+Once some user specs have been added to an environment, they can be concretized.
+There are at the moment three different modes of operation to concretize an environment,
+which are explained in details in :ref:`environments_concretization_config`.
+Regardless of which mode of operation has been chosen, the following
+command will ensure all the root specs are concretized according to the
 constraints that are prescribed in the configuration:
 
 .. code-block:: console
@@ -342,15 +287,16 @@ constraints that are prescribed in the configuration:
 
 In the case of specs that are not concretized together, the command
 above will concretize only the specs that were added and not yet
-concretized. Forcing a re-concretization of all of the specs can be done
-by adding the ``-f`` option:
+concretized. Forcing a re-concretization of all the specs can be done
+instead with this command:
 
 .. code-block:: console
 
    [myenv]$ spack concretize -f
 
-Without the option, Spack guarantees that already concretized specs are
-unchanged in the environment.
+When the ``-f`` flag is not used to reconcretize all specs, Spack
+guarantees that already concretized specs are unchanged in the
+environment.
 
 The ``concretize`` command does not install any packages. For packages
 that have already been installed outside of the environment, the
@@ -383,16 +329,16 @@ installed specs using the ``-c`` (``--concretized``) flag.
 Installing an Environment
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
-In addition to adding individual specs to an environment, one
-can install the entire environment at once using the command
+In addition to installing individual specs into an Environment, one
+can install the entire Environment at once using the command
 
 .. code-block:: console
 
    [myenv]$ spack install
 
-If the environment has been concretized, Spack will install the
-concretized specs. Otherwise, ``spack install`` will concretize
-the environment before installing the concretized specs.
+If the Environment has been concretized, Spack will install the
+concretized specs. Otherwise, ``spack install`` will first concretize
+the Environment and then install the concretized specs.
 
 .. note::
 
@@ -413,17 +359,17 @@ the environment before installing the concretized specs.
 
 
 As it installs, ``spack install`` creates symbolic links in the
-``logs/`` directory in the environment, allowing for easy inspection
+``logs/`` directory in the Environment, allowing for easy inspection
 of build logs related to that environment. The ``spack install``
 command also stores a Spack repo containing the ``package.py`` file
 used at install time for each package in the ``repos/`` directory in
-the environment.
+the Environment.
 
 The ``--no-add`` option can be used in a concrete environment to tell
 spack to install specs already present in the environment but not to
 add any new root specs to the environment.  For root specs provided
 to ``spack install`` on the command line, ``--no-add`` is the default,
-while for dependency specs, it is optional.  In other
+while for dependency specs on the other hand, it is optional.  In other
 words, if there is an unambiguous match in the active concrete environment
 for a root spec provided to ``spack install`` on the command line, spack
 does not require you to specify the ``--no-add`` option to prevent the spec
@@ -437,22 +383,12 @@ Developing Packages in a Spack Environment
 
 The ``spack develop`` command allows one to develop Spack packages in
 an environment. It requires a spec containing a concrete version, and
-will configure Spack to install the package from local source.
-If a version is not provided from the command line interface then spack
-will automatically pick the highest version the package has defined.
-This means any infinity versions (``develop``, ``main``, ``stable``) will be
-preferred in this selection process.
-By default, ``spack develop`` will also clone the package to a subdirectory in the
-environment for the local source. This package will have a special variant ``dev_path``
+will configure Spack to install the package from local source. By
+default, it will also clone the package to a subdirectory in the
+environment. This package will have a special variant ``dev_path``
 set, and Spack will ensure the package and its dependents are rebuilt
 any time the environment is installed if the package's local source
-code has been modified. Spack's native implementation to check for modifications
-is to check if ``mtime`` is newer than the installation.
-A custom check can be created by overriding the ``detect_dev_src_change`` method
-in your package class. This is particularly useful for projects using custom spack repo's
-to drive development and want to optimize performance.
-
-Spack ensures that all instances of a
+code has been modified. Spack ensures that all instances of a
 developed package in the environment are concretized to match the
 version (and other constraints) passed as the spec argument to the
 ``spack develop`` command.
@@ -462,26 +398,9 @@ also be used as valid concrete versions (see :ref:`version-specifier`).
 This means that for a package ``foo``, ``spack develop foo@git.main`` will clone
 the ``main`` branch of the package, and ``spack install`` will install from
 that git clone if ``foo`` is in the environment.
-Further development on ``foo`` can be tested by re-installing the environment,
+Further development on ``foo`` can be tested by reinstalling the environment,
 and eventually committed and pushed to the upstream git repo.
 
-If the package being developed supports out-of-source builds then users can use the
-``--build_directory`` flag to control the location and name of the build directory.
-This is a shortcut to set the ``package_attributes:build_directory`` in the
-``packages`` configuration (see :ref:`assigning-package-attributes`).
-The supplied location will become the build-directory for that package in all future builds.
-
-.. warning::
-   Potential pitfalls of setting the build directory
-    Spack does not check for out-of-source build compatibility with the packages and
-    so the onerous of making sure the package supports out-of-source builds is on
-    the user.
-    For example, most ``autotool`` and ``makefile`` packages do not support out-of-source builds
-    while all ``CMake`` packages do.
-    Understanding these nuances are on the software developers and we strongly encourage
-    developers to only redirect the build directory if they understand their package's
-    build-system.
-
 ^^^^^^^
 Loading
 ^^^^^^^
@@ -498,125 +417,6 @@ Sourcing that file in Bash will make the environment available to the
 user; and can be included in ``.bashrc`` files, etc.  The ``loads``
 file may also be copied out of the environment, renamed, etc.
 
-
-.. _environment_include_concrete:
-
-------------------------------
-Included Concrete Environments
-------------------------------
-
-Spack environments can create an environment based off of information in already
-established environments. You can think of it as a combination of existing
-environments. It will gather information from the existing environment's
-``spack.lock`` and use that during the creation of this included concrete
-environment. When an included concrete environment is created it will generate
-a ``spack.lock`` file for the newly created environment.
-
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Creating included environments
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To create a combined concrete environment, you must have at least one existing
-concrete environment. You will use the command ``spack env create`` with the
-argument ``--include-concrete`` followed by the name or path of the environment
-you'd like to include. Here is an example of how to create a combined environment
-from the command line.
-
-.. code-block:: console
-
-   $ spack env create myenv
-   $ spack -e myenv add python
-   $ spack -e myenv concretize
-   $ spack env create --include-concrete myenv included_env
-
-
-You can also include an environment directly in the ``spack.yaml`` file. It
-involves adding the ``include_concrete`` heading in the yaml followed by the
-absolute path to the independent environments.
-
-.. code-block:: yaml
-
-   spack:
-     specs: []
-     concretizer:
-         unify: true
-     include_concrete:
-     - /absolute/path/to/environment1
-     - /absolute/path/to/environment2
-
-
-Once the ``spack.yaml`` has been updated you must concretize the environment to
-get the concrete specs from the included environments.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Updating an included environment
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-If changes were made to the base environment and you want that reflected in the
-included environment you will need to reconcretize both the base environment and the
-included environment for the change to be implemented. For example:
-
-.. code-block:: console
-
-   $ spack env create myenv
-   $ spack -e myenv add python
-   $ spack -e myenv concretize
-   $ spack env create --include-concrete myenv included_env
-
-
-   $ spack -e myenv find
-   ==> In environment myenv
-   ==> Root specs
-   python
-
-   ==> 0 installed packages
-
-
-   $ spack -e included_env find
-   ==> In environment included_env
-   ==> No root specs
-   ==> Included specs
-   python
-
-   ==> 0 installed packages
-
-Here we see that ``included_env`` has access to the python package through
-the ``myenv`` environment. But if we were to add another spec to ``myenv``,
-``included_env`` will not be able to access the new information.
-
-.. code-block:: console
-
-   $ spack -e myenv add perl
-   $ spack -e myenv concretize
-   $ spack -e myenv find
-   ==> In environment myenv
-   ==> Root specs
-   perl  python
-
-   ==> 0 installed packages
-
-
-   $ spack -e included_env find
-   ==> In environment included_env
-   ==> No root specs
-   ==> Included specs
-   python
-
-   ==> 0 installed packages
-
-It isn't until you run the ``spack concretize`` command that the combined
-environment will get the updated information from the reconcretized base environmennt.
-
-.. code-block:: console
-
-   $ spack -e included_env concretize
-   $ spack -e included_env find
-   ==> In environment included_env
-   ==> No root specs
-   ==> Included specs
-   perl  python
-
-   ==> 0 installed packages
-
 .. _environment-configuration:
 
 ------------------------
@@ -647,7 +447,7 @@ manipulate configuration inline in the ``spack.yaml`` file.
 Inline configurations
 ^^^^^^^^^^^^^^^^^^^^^
 
-Inline environment-scope configuration is done using the same yaml
+Inline Environment-scope configuration is done using the same yaml
 format as standard Spack configuration scopes, covered in the
 :ref:`configuration` section. Each section is contained under a
 top-level yaml object with it's name. For example, a ``spack.yaml``
@@ -657,11 +457,11 @@ a ``packages.yaml`` file) could contain:
 .. code-block:: yaml
 
    spack:
-     # ...
+     ...
      packages:
        all:
          compiler: [intel]
-     # ...
+     ...
 
 This configuration sets the default compiler for all packages to
 ``intel``.
@@ -672,7 +472,7 @@ Included configurations
 
 Spack environments allow an ``include`` heading in their yaml
 schema. This heading pulls in external configuration files and applies
-them to the environment.
+them to the Environment.
 
 .. code-block:: yaml
 
@@ -685,9 +485,6 @@ them to the environment.
 Environments can include files or URLs. File paths can be relative or
 absolute. URLs include the path to the text for individual files or
 can be the path to a directory containing configuration files.
-Spack supports ``file``, ``http``, ``https`` and ``ftp`` protocols (or
-schemes). Spack-specific, environment and user path variables may be
-used in these paths. See :ref:`config-file-variables` for more information.
 
 ^^^^^^^^^^^^^^^^^^^^^^^^
 Configuration precedence
@@ -702,7 +499,7 @@ have higher precedence, as the included configs are applied in reverse order.
 Manually Editing the Specs List
 -------------------------------
 
-The list of abstract/root specs in the environment is maintained in
+The list of abstract/root specs in the Environment is maintained in
 the ``spack.yaml`` manifest under the heading ``specs``.
 
 .. code-block:: yaml
@@ -810,7 +607,7 @@ evaluates to the cross-product of those specs. Spec matrices also
 contain an ``excludes`` directive, which eliminates certain
 combinations from the evaluated result.
 
-The following two environment manifests are identical:
+The following two Environment manifests are identical:
 
 .. code-block:: yaml
 
@@ -885,7 +682,7 @@ files are identical.
 In short files like the example, it may be easier to simply list the
 included specs. However for more complicated examples involving many
 packages across many toolchains, separately factored lists make
-environments substantially more manageable.
+Environments substantially more manageable.
 
 Additionally, the ``-l`` option to the ``spack add`` command allows
 one to add to named lists in the definitions section of the manifest
@@ -904,7 +701,7 @@ named list ``compilers`` is ``['%gcc', '%clang', '%intel']`` on
    spack:
      definitions:
        - compilers: ['%gcc', '%clang']
-       - when: arch.satisfies('target=x86_64:')
+       - when: arch.satisfies('x86_64:')
          compilers: ['%intel']
 
 .. note::
@@ -971,90 +768,37 @@ For example, the following environment has three root packages:
 This allows for a much-needed reduction in redundancy between packages
 and constraints.
 
+----------------
+Filesystem Views
+----------------
 
------------------
-Environment Views
------------------
-
-Spack Environments can have an associated filesystem view, which is a directory
-with a more traditional structure ``<view>/bin``, ``<view>/lib``, ``<view>/include``
-in which all files of the installed packages are linked.
-
-By default a view is created for each environment, thanks to the ``view: true``
-option in the ``spack.yaml`` manifest file:
-
-.. code-block:: yaml
-
-   spack:
-     specs: [perl, python]
-     view: true
-
-The view is created in a hidden directory ``.spack-env/view`` relative to the environment.
-If you've used ``spack env activate``, you may have already interacted with this view. Spack
-prepends its ``<view>/bin`` dir to ``PATH`` when the environment is activated, so that
-you can directly run executables from all installed packages in the environment.
-
-Views are highly customizable: you can control where they are put, modify their structure,
-include and exclude specs, change how files are linked, and you can even generate multiple
-views for a single environment.
+Spack Environments can define filesystem views, which provide a direct access point
+for software similar to the directory hierarchy that might exist under ``/usr/local``.
+Filesystem views are updated every time the environment is written out to the lock
+file ``spack.lock``, so the concrete environment and the view are always compatible.
+The files of the view's installed packages are brought into the view by symbolic or
+hard links, referencing the original Spack installation, or by copy.
 
 .. _configuring_environment_views:
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-Minimal view configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Configuration in ``spack.yaml``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The minimal configuration
-
-.. code-block:: yaml
-
-   spack:
-     # ...
-     view: true
-
-lets Spack generate a single view with default settings under the
-``.spack-env/view`` directory of the environment.
-
-Another short way to configure a view is to specify just where to put it:
-
-.. code-block:: yaml
-
-   spack:
-     # ...
-     view: /path/to/view
-
-Views can also be disabled by setting ``view: false``.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Advanced view configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-One or more **view descriptors** can be defined under ``view``, keyed by a name.
-The example from the previous section with ``view: /path/to/view`` is equivalent
-to defining a view descriptor named ``default`` with a ``root`` attribute:
-
-.. code-block:: yaml
-
-   spack:
-     # ...
-     view:
-       default:  # name of the view
-         root: /path/to/view  # view descriptor attribute
-
-The ``default`` view descriptor name is special: when you ``spack env activate`` your
-environment, this view will be used to update (among other things) your ``PATH``
-variable.
-
-View descriptors must contain the root of the view, and optionally projections,
-``select`` and ``exclude`` lists and link information via ``link`` and
+The Spack Environment manifest file has a top-level keyword
+``view``. Each entry under that heading is a **view descriptor**, headed
+by a name. Any number of views may be defined under the ``view`` heading.
+The view descriptor contains the root of the view, and
+optionally the projections for the view, ``select`` and
+``exclude`` lists for the view and link information via ``link`` and
 ``link_type``.
 
-As a more advanced example, in the following manifest
+For example, in the following manifest
 file snippet we define a view named ``mpis``, rooted at
 ``/path/to/view`` in which all projections use the package name,
 version, and compiler name to determine the path for a given
 package. This view selects all packages that depend on MPI, and
-excludes those built with the GCC compiler at version 18.5.
+excludes those built with the PGI compiler at version 18.5.
 The root specs with their (transitive) link and run type dependencies
 will be put in the view due to the  ``link: all`` option,
 and the files in the view will be symlinks to the spack install
@@ -1063,12 +807,12 @@ directories.
 .. code-block:: yaml
 
    spack:
-     # ...
+     ...
      view:
        mpis:
          root: /path/to/view
          select: [^mpi]
-         exclude: ['%gcc@18.5']
+         exclude: ['%pgi@18.5']
          projections:
            all: '{name}/{version}-{compiler.name}'
          link: all
@@ -1094,14 +838,63 @@ of ``hardlink`` or ``copy``.
    when the environment is not activated, and linked libraries will be located
    *outside* of the view thanks to rpaths.
 
+
+There are two shorthands for environments with a single view. If the
+environment at ``/path/to/env`` has a single view, with a root at
+``/path/to/env/.spack-env/view``, with default selection and exclusion
+and the default projection, we can put ``view: True`` in the
+environment manifest. Similarly, if the environment has a view with a
+different root, but default selection, exclusion, and projections, the
+manifest can say ``view: /path/to/view``. These views are
+automatically named ``default``, so that
+
+.. code-block:: yaml
+
+   spack:
+     ...
+     view: True
+
+is equivalent to
+
+.. code-block:: yaml
+
+   spack:
+     ...
+     view:
+       default:
+         root: .spack-env/view
+
+and
+
+.. code-block:: yaml
+
+   spack:
+     ...
+     view: /path/to/view
+
+is equivalent to
+
+.. code-block:: yaml
+
+   spack:
+     ...
+     view:
+       default:
+         root: /path/to/view
+
+By default, Spack environments are configured with ``view: True`` in
+the manifest. Environments can be configured without views using
+``view: False``. For backwards compatibility reasons, environments
+with no ``view`` key are treated the same as ``view: True``.
+
 From the command line, the ``spack env create`` command takes an
 argument ``--with-view [PATH]`` that sets the path for a single, default
 view. If no path is specified, the default path is used (``view:
-true``). The argument ``--without-view`` can be used to create an
+True``). The argument ``--without-view`` can be used to create an
 environment without any view configured.
 
 The ``spack env view`` command can be used to change the manage views
-of an environment. The subcommand ``spack env view enable`` will add a
+of an Environment. The subcommand ``spack env view enable`` will add a
 view named ``default`` to an environment. It takes an optional
 argument to specify the path for the new default view. The subcommand
 ``spack env view disable`` will remove the view named ``default`` from
@@ -1127,17 +920,6 @@ function, as shown in the example below:
      ^mpi: "{name}-{version}/{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version}"
      all: "{name}-{version}/{compiler.name}-{compiler.version}"
 
-Projections also permit environment and spack configuration variable
-expansions as shown below:
-
-.. code-block:: yaml
-
-   projections:
-     all: "{name}-{version}/{compiler.name}-{compiler.version}/$date/$SYSTEM_ENV_VARIBLE"
-
-where ``$date`` is the spack configuration variable that will expand with the ``YYYY-MM-DD``
-format and ``$SYSTEM_ENV_VARIABLE`` is an environment variable defined in the shell.
-
 The entries in the projections configuration file must all be either
 specs or the keyword ``all``. For each spec, the projection used will
 be the first non-``all`` entry that the spec satisfies, or ``all`` if
@@ -1163,18 +945,11 @@ the projection under ``all`` before reaching those entries.
 Activating environment views
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The ``spack env activate <env>`` has two effects:
-
-1. It activates the environment so that further Spack commands such
-   as ``spack install`` will run in the context of the environment.
-2. It activates the view so that environment variables such as
-   ``PATH`` are updated to include the view.
-
-Without further arguments, the ``default`` view of the environment is
-activated. If a view with a different name has to be activated,
-``spack env activate --with-view <name> <env>`` can be
-used instead. You can also activate the environment without modifying
-further environment variables using ``--without-view``.
+The ``spack env activate`` command will put the default view for the
+environment into the user's path, in addition to activating the
+environment for Spack commands. The arguments ``-v,--with-view`` and
+``-V,--without-view`` can be used to tune this behavior. The default
+behavior is to activate with the environment view if there is one.
 
 The environment variables affected by the ``spack env activate``
 command and the paths that are used to update them are determined by
@@ -1197,8 +972,8 @@ relevant variable if the path exists. For this reason, it is not
 recommended to use non-default projections with the default view of an
 environment.
 
-The ``spack env deactivate`` command will remove the active view of
-the Spack environment from the user's environment variables.
+The ``spack env deactivate`` command will remove the default view of
+the environment from the user's path.
 
 
 .. _env-generate-depfile:
@@ -1215,7 +990,7 @@ other targets to depend on the environment installation.
 
 A typical workflow is as follows:
 
-.. code-block:: console
+.. code:: console
 
    spack env create -d .
    spack -e . add perl
@@ -1269,7 +1044,7 @@ gets installed and is available for use in the ``env`` target.
    	$(SPACK) -e . env depfile -o $@ --make-prefix spack
 
    env: spack/env
-   	$(info environment installed!)
+   	$(info Environment installed!)
 
    clean:
    	rm -rf spack.lock env.mk spack/
@@ -1308,7 +1083,7 @@ its dependencies. This can be useful when certain flags should only apply to
 dependencies. Below we show a use case where a spec is installed with verbose
 output (``spack install --verbose``) while its dependencies are installed silently:
 
-.. code-block:: console
+.. code:: console
 
    $ spack env depfile -o Makefile
 
@@ -1330,7 +1105,7 @@ This can be accomplished through the generated ``[<prefix>/]SPACK_PACKAGE_IDS``
 variable. Assuming we have an active and concrete environment, we generate the
 associated ``Makefile`` with a prefix ``example``:
 
-.. code-block:: console
+.. code:: console
 
    $ spack env depfile -o env.mk --make-prefix example
 
@@ -1357,7 +1132,7 @@ index once every package is pushed. Note how this target uses the generated
    example/push/%: example/install/%
    	@mkdir -p $(dir $@)
    	$(info About to push $(SPEC) to a buildcache)
-   	$(SPACK) -e . buildcache push --only=package $(BUILDCACHE_DIR) /$(HASH)
+   	$(SPACK) -e . buildcache push --allow-root --only=package $(BUILDCACHE_DIR) /$(HASH)
    	@touch $@
 
    push: $(addprefix example/push/,$(example/SPACK_PACKAGE_IDS))

lib/spack/docs/extensions.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -8,42 +9,46 @@
 Custom Extensions
 =================
 
-*Spack extensions* allow you to extend Spack capabilities by deploying your
+*Spack extensions* permit you to extend Spack capabilities by deploying your
 own custom commands or logic in an arbitrary location on your filesystem.
 This might be extremely useful e.g. to develop and maintain a command whose purpose is
 too specific to be considered for reintegration into the mainline or to
 evolve a command through its early stages before starting a discussion to merge
 it upstream.
-
 From Spack's point of view an extension is any path in your filesystem which
-respects the following naming and layout for files:
+respects a prescribed naming and layout for files:
 
 .. code-block:: console
 
   spack-scripting/ # The top level directory must match the format 'spack-{extension_name}'
   ├── pytest.ini # Optional file if the extension ships its own tests
   ├── scripting # Folder that may contain modules that are needed for the extension commands
-  │   ├── cmd # Folder containing extension commands
-  │   │   └── filter.py # A new command that will be available
-  │   └── functions.py # Module with internal details
-  └── tests # Tests for this extension
+  │   └── cmd # Folder containing extension commands
+  │       └── filter.py # A new command that will be available
+  ├── tests # Tests for this extension
   │   ├── conftest.py
   │   └── test_filter.py
   └── templates # Templates that may be needed by the extension
 
-In the example above, the extension is named *scripting*. It adds an additional command
-(``spack filter``) and unit tests to verify its behavior.
+In the example above the extension named *scripting* adds an additional command (``filter``)
+and unit tests to verify its behavior. The code for this example can be
+obtained by cloning the corresponding git repository:
 
-The extension can import any core Spack module in its implementation. When loaded by
-the ``spack`` command, the extension itself is imported as a Python package in the
-``spack.extensions`` namespace. In the example above, since the extension is named
-"scripting", the corresponding Python module is ``spack.extensions.scripting``.
-
-The code for this example extension can be obtained by cloning the corresponding git repository:
+.. TODO: write an ad-hoc "hello world" extension and make it part of the spack organization
 
 .. code-block:: console
 
-   $ git -C /tmp clone https://github.com/spack/spack-scripting.git
+   $ cd ~/
+   $ mkdir tmp && cd tmp
+   $ git clone https://github.com/alalazo/spack-scripting.git
+   Cloning into 'spack-scripting'...
+   remote: Counting objects: 11, done.
+   remote: Compressing objects: 100% (7/7), done.
+   remote: Total 11 (delta 0), reused 11 (delta 0), pack-reused 0
+   Receiving objects: 100% (11/11), done.
+
+As you can see by inspecting the sources, Python modules that are part of the extension
+can import any core Spack module.
 
 ---------------------------------
 Configure Spack to Use Extensions
@@ -56,7 +61,7 @@ paths to ``config.yaml``. In the case of our example this means ensuring that:
 
    config:
      extensions:
-     - /tmp/spack-scripting
+     - ~/tmp/spack-scripting
 
 is part of your configuration file. Once this is setup any command that the extension provides
 will be available from the command line:
@@ -81,68 +86,37 @@ will be available from the command line:
      --implicit       select specs that are not installed or were installed implicitly
      --output OUTPUT  where to dump the result
 
-The corresponding unit tests can be run giving the appropriate options to ``spack unit-test``:
+The corresponding unit tests can be run giving the appropriate options
+to ``spack unit-test``:
 
 .. code-block:: console
 
    $ spack unit-test --extension=scripting
-   ========================================== test session starts ===========================================
-   platform linux -- Python 3.11.5, pytest-7.4.3, pluggy-1.3.0
-   rootdir: /home/culpo/github/spack-scripting
-   configfile: pytest.ini
-   testpaths: tests
-   plugins: xdist-3.5.0
+
+   ============================================================== test session starts ===============================================================
+   platform linux2 -- Python 2.7.15rc1, pytest-3.2.5, py-1.4.34, pluggy-0.4.0
+   rootdir: /home/mculpo/tmp/spack-scripting, inifile: pytest.ini
    collected 5 items
 
-   tests/test_filter.py .....                                                                         [100%]
+   tests/test_filter.py ...XX
+   ============================================================ short test summary info =============================================================
+   XPASS tests/test_filter.py::test_filtering_specs[flags3-specs3-expected3]
+   XPASS tests/test_filter.py::test_filtering_specs[flags4-specs4-expected4]
 
-   ========================================== slowest 30 durations ==========================================
-   2.31s setup    tests/test_filter.py::test_filtering_specs[kwargs0-specs0-expected0]
-   0.57s call     tests/test_filter.py::test_filtering_specs[kwargs2-specs2-expected2]
-   0.56s call     tests/test_filter.py::test_filtering_specs[kwargs4-specs4-expected4]
-   0.54s call     tests/test_filter.py::test_filtering_specs[kwargs3-specs3-expected3]
-   0.54s call     tests/test_filter.py::test_filtering_specs[kwargs1-specs1-expected1]
-   0.48s call     tests/test_filter.py::test_filtering_specs[kwargs0-specs0-expected0]
-   0.01s setup    tests/test_filter.py::test_filtering_specs[kwargs4-specs4-expected4]
-   0.01s setup    tests/test_filter.py::test_filtering_specs[kwargs2-specs2-expected2]
-   0.01s setup    tests/test_filter.py::test_filtering_specs[kwargs1-specs1-expected1]
-   0.01s setup    tests/test_filter.py::test_filtering_specs[kwargs3-specs3-expected3]
-
-   (5 durations < 0.005s hidden.  Use -vv to show these durations.)
-   =========================================== 5 passed in 5.06s ============================================
-
----------------------------------------
-Registering Extensions via Entry Points
----------------------------------------
-
-.. note::
-   Python version >= 3.8 is required to register extensions via entry points.
-
-Spack can be made aware of extensions that are installed as part of a python package.  To do so, register a function that returns the extension path, or paths, to the ``"spack.extensions"`` entry point.  Consider the Python package ``my_package`` that includes a Spack extension:
-
-.. code-block:: console
-
-  my-package/
-  ├── src
-  │   ├── my_package
-  │   │   └── __init__.py
-  │   └── spack-scripting/  # the spack extensions
-  └── pyproject.toml
-
-adding the following to ``my_package``'s ``pyproject.toml`` will make the ``spack-scripting`` extension visible to Spack when ``my_package`` is installed:
-
-.. code-block:: toml
-
-   [project.entry_points."spack.extenions"]
-   my_package = "my_package:get_extension_path"
-
-The function ``my_package.get_extension_path`` in ``my_package/__init__.py`` might look like
-
-.. code-block:: python
-
-   import importlib.resources
-
-   def get_extension_path():
-       dirname = importlib.resources.files("my_package").joinpath("spack-scripting")
-       if dirname.exists():
-           return str(dirname)
+   =========================================================== slowest 20 test durations ============================================================
+   3.74s setup    tests/test_filter.py::test_filtering_specs[flags0-specs0-expected0]
+   0.17s call     tests/test_filter.py::test_filtering_specs[flags3-specs3-expected3]
+   0.16s call     tests/test_filter.py::test_filtering_specs[flags2-specs2-expected2]
+   0.15s call     tests/test_filter.py::test_filtering_specs[flags1-specs1-expected1]
+   0.13s call     tests/test_filter.py::test_filtering_specs[flags4-specs4-expected4]
+   0.08s call     tests/test_filter.py::test_filtering_specs[flags0-specs0-expected0]
+   0.04s teardown tests/test_filter.py::test_filtering_specs[flags4-specs4-expected4]
+   0.00s setup    tests/test_filter.py::test_filtering_specs[flags4-specs4-expected4]
+   0.00s setup    tests/test_filter.py::test_filtering_specs[flags3-specs3-expected3]
+   0.00s setup    tests/test_filter.py::test_filtering_specs[flags1-specs1-expected1]
+   0.00s setup    tests/test_filter.py::test_filtering_specs[flags2-specs2-expected2]
+   0.00s teardown tests/test_filter.py::test_filtering_specs[flags2-specs2-expected2]
+   0.00s teardown tests/test_filter.py::test_filtering_specs[flags1-specs1-expected1]
+   0.00s teardown tests/test_filter.py::test_filtering_specs[flags0-specs0-expected0]
+   0.00s teardown tests/test_filter.py::test_filtering_specs[flags3-specs3-expected3]
+   ====================================================== 3 passed, 2 xpassed in 4.51 seconds =======================================================

lib/spack/docs/features.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/frequently_asked_questions.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/getting_started.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -34,7 +35,7 @@ A build matrix showing which packages are working on which systems is shown belo
       .. code-block:: console
 
          apt update
-         apt install bzip2 ca-certificates g++ gcc gfortran git gzip lsb-release patch python3 tar unzip xz-utils zstd
+         apt install build-essential ca-certificates coreutils curl environment-modules gfortran git gpg lsb-release python3 python3-distutils python3-venv unzip zip
 
    .. tab-item:: RHEL
 
@@ -42,14 +43,14 @@ A build matrix showing which packages are working on which systems is shown belo
 
          dnf install epel-release
          dnf group install "Development Tools"
-         dnf install gcc-gfortran redhat-lsb-core python3 unzip
+         dnf install curl findutils gcc-gfortran gnupg2 hostname iproute redhat-lsb-core python3 python3-pip python3-setuptools unzip python3-boto3
 
    .. tab-item:: macOS Brew
 
       .. code-block:: console
 
          brew update
-         brew install gcc git zip
+         brew install curl gcc git gnupg zip
 
 ------------
 Installation
@@ -60,15 +61,10 @@ Getting Spack is easy.  You can clone it from the `github repository
 
 .. code-block:: console
 
-   $ git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git
+   $ git clone -c feature.manyFiles=true https://github.com/spack/spack.git
 
 This will create a directory called ``spack``.
 
-.. note::
-   ``-c feature.manyFiles=true`` improves git's performance on repositories with 1,000+ files.
-
-   ``--depth=2`` prunes the git history to reduce the size of the Spack installation.
-
 .. _shell-support:
 
 ^^^^^^^^^^^^^
@@ -147,22 +143,20 @@ The first time you concretize a spec, Spack will bootstrap automatically:
    --------------------------------
    zlib@1.2.13%gcc@9.4.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu20.04-icelake
 
-The default bootstrap behavior is to use pre-built binaries. You can verify the
-active bootstrap repositories with:
-
-.. command-output:: spack bootstrap list
-
 If for security concerns you cannot bootstrap ``clingo`` from pre-built
 binaries, you have to disable fetching the binaries we generated with Github Actions.
 
 .. code-block:: console
 
-   $ spack bootstrap disable github-actions-v0.6
-   ==> "github-actions-v0.6" is now disabled and will not be used for bootstrapping
-   $ spack bootstrap disable github-actions-v0.5
-   ==> "github-actions-v0.5" is now disabled and will not be used for bootstrapping
+   $ spack bootstrap disable github-actions-v0.4
+   ==> "github-actions-v0.4" is now disabled and will not be used for bootstrapping
+   $ spack bootstrap disable github-actions-v0.3
+   ==> "github-actions-v0.3" is now disabled and will not be used for bootstrapping
+
+You can verify that the new settings are effective with:
+
+.. command-output:: spack bootstrap list
 
-You can verify that the new settings are effective with ``spack bootstrap list``.
 
 .. note::
 
@@ -256,10 +250,9 @@ Compiler configuration
 
 Spack has the ability to build packages with multiple compilers and
 compiler versions. Compilers can be made available to Spack by
-specifying them manually in ``compilers.yaml`` or ``packages.yaml``,
-or automatically by running ``spack compiler find``, but for
-convenience Spack will automatically detect compilers the first time
-it needs them.
+specifying them manually in ``compilers.yaml``, or automatically by
+running ``spack compiler find``, but for convenience Spack will
+automatically detect compilers the first time it needs them.
 
 .. _cmd-spack-compilers:
 
@@ -284,6 +277,10 @@ compilers`` or ``spack compiler list``:
        intel@14.0.1  intel@13.0.1  intel@12.1.2  intel@10.1
    -- clang -------------------------------------------------------
        clang@3.4  clang@3.3  clang@3.2  clang@3.1
+   -- pgi ---------------------------------------------------------
+       pgi@14.3-0   pgi@13.2-0  pgi@12.1-0   pgi@10.9-0  pgi@8.0-1
+       pgi@13.10-0  pgi@13.1-1  pgi@11.10-0  pgi@10.2-0  pgi@7.1-3
+       pgi@13.6-0   pgi@12.8-0  pgi@11.1-0   pgi@9.0-4   pgi@7.0-6
 
 Any of these compilers can be used to build Spack packages.  More on
 how this is done is in :ref:`sec-specs`.
@@ -460,54 +457,6 @@ specification. The operations available to modify the environment are ``set``, `
          prepend_path: # Similar for append|remove_path
            LD_LIBRARY_PATH: /ld/paths/added/by/setvars/sh
 
-.. note::
-
-   Spack is in the process of moving compilers from a separate
-   attribute to be handled like all other packages. As part of this
-   process, the ``compilers.yaml`` section will eventually be replaced
-   by configuration in the ``packages.yaml`` section. This new
-   configuration is now available, although it is not yet the default
-   behavior.
-
-Compilers can also be configured as external packages in the
-``packages.yaml`` config file. Any external package for a compiler
-(e.g. ``gcc`` or ``llvm``) will be treated as a configured compiler
-assuming the paths to the compiler executables are determinable from
-the prefix.
-
-If the paths to the compiler executable are not determinable from the
-prefix, you can add them to the ``extra_attributes`` field. Similarly,
-all other fields from the compilers config can be added to the
-``extra_attributes`` field for an external representing a compiler.
-
-Note that the format for the ``paths`` field in the
-``extra_attributes`` section is different than in the ``compilers``
-config. For compilers configured as external packages, the section is
-named ``compilers`` and the dictionary maps language names (``c``,
-``cxx``, ``fortran``) to paths, rather than using the names ``cc``,
-``fc``, and ``f77``.
-
-.. code-block:: yaml
-
-   packages:
-     gcc:
-       external:
-       - spec: gcc@12.2.0 arch=linux-rhel8-skylake
-         prefix: /usr
-         extra_attributes:
-           environment:
-             set:
-               GCC_ROOT: /usr
-       external:
-       - spec: llvm+clang@15.0.0 arch=linux-rhel8-skylake
-         prefix: /usr
-         extra_attributes:
-           compilers:
-             c: /usr/bin/clang-with-suffix
-             cxx: /usr/bin/clang++-with-extra-info
-             fortran: /usr/bin/gfortran
-           extra_rpaths:
-           - /usr/lib/llvm/
 
 ^^^^^^^^^^^^^^^^^^^^^^^
 Build Your Own Compiler
@@ -674,7 +623,7 @@ Fortran.
 
       compilers:
       - compiler:
-        # ...
+        ...
         paths:
           cc: /usr/bin/clang
           cxx: /usr/bin/clang++
@@ -803,6 +752,65 @@ flags to the ``icc`` command:
            spec: intel@15.0.24.4.9.3
 
 
+^^^
+PGI
+^^^
+
+PGI comes with two sets of compilers for C++ and Fortran,
+distinguishable by their names.  "Old" compilers:
+
+.. code-block:: yaml
+
+    cc:  /soft/pgi/15.10/linux86-64/15.10/bin/pgcc
+    cxx: /soft/pgi/15.10/linux86-64/15.10/bin/pgCC
+    f77: /soft/pgi/15.10/linux86-64/15.10/bin/pgf77
+    fc:  /soft/pgi/15.10/linux86-64/15.10/bin/pgf90
+
+"New" compilers:
+
+.. code-block:: yaml
+
+    cc:  /soft/pgi/15.10/linux86-64/15.10/bin/pgcc
+    cxx: /soft/pgi/15.10/linux86-64/15.10/bin/pgc++
+    f77: /soft/pgi/15.10/linux86-64/15.10/bin/pgfortran
+    fc:  /soft/pgi/15.10/linux86-64/15.10/bin/pgfortran
+
+Older installations of PGI contains just the old compilers; whereas
+newer installations contain the old and the new.  The new compiler is
+considered preferable, as some packages
+(``hdf``) will not build with the old compiler.
+
+When auto-detecting a PGI compiler, there are cases where Spack will
+find the old compilers, when you really want it to find the new
+compilers.  It is best to check this ``compilers.yaml``; and if the old
+compilers are being used, change ``pgf77`` and ``pgf90`` to
+``pgfortran``.
+
+Other issues:
+
+* There are reports that some packages will not build with PGI,
+  including ``libpciaccess`` and ``openssl``.  A workaround is to
+  build these packages with another compiler and then use them as
+  dependencies for PGI-build packages.  For example:
+
+  .. code-block:: console
+
+     $ spack install openmpi%pgi ^libpciaccess%gcc
+
+
+* PGI requires a license to use; see :ref:`licensed-compilers` for more
+  information on installation.
+
+.. note::
+
+   It is believed the problem with HDF 4 is that everything is
+   compiled with the ``F77`` compiler, but at some point some Fortran
+   90 code slipped in there. So compilers that can handle both FORTRAN
+   77 and Fortran 90 (``gfortran``, ``pgfortran``, etc) are fine.  But
+   compilers specific to one or the other (``pgf77``, ``pgf90``) won't
+   work.
+
+
 ^^^
 NAG
 ^^^
@@ -1307,6 +1315,187 @@ This will write the private key to the file `dinosaur.priv`.
     or for help on an issue or the Spack slack.
 
 
+.. _cray-support:
+
+-------------
+Spack on Cray
+-------------
+
+Spack differs slightly when used on a Cray system. The architecture spec
+can differentiate between the front-end and back-end processor and operating system.
+For example, on Edison at NERSC, the back-end target processor
+is "Ivy Bridge", so you can specify to use the back-end this way:
+
+.. code-block:: console
+
+   $ spack install zlib target=ivybridge
+
+You can also use the operating system to build against the back-end:
+
+.. code-block:: console
+
+   $ spack install zlib os=CNL10
+
+Notice that the name includes both the operating system name and the major
+version number concatenated together.
+
+Alternatively, if you want to build something for the front-end,
+you can specify the front-end target processor. The processor for a login node
+on Edison is "Sandy bridge" so we specify on the command line like so:
+
+.. code-block:: console
+
+   $ spack install zlib target=sandybridge
+
+And the front-end operating system is:
+
+.. code-block:: console
+
+   $ spack install zlib os=SuSE11
+
+^^^^^^^^^^^^^^^^^^^^^^^
+Cray compiler detection
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Spack can detect compilers using two methods. For the front-end, we treat
+everything the same. The difference lies in back-end compiler detection.
+Back-end compiler detection is made via the Tcl module avail command.
+Once it detects the compiler it writes the appropriate PrgEnv and compiler
+module name to compilers.yaml and sets the paths to each compiler with Cray\'s
+compiler wrapper names (i.e. cc, CC, ftn). During build time, Spack will load
+the correct PrgEnv and compiler module and will call appropriate wrapper.
+
+The compilers.yaml config file will also differ. There is a
+modules section that is filled with the compiler's Programming Environment
+and module name. On other systems, this field is empty []:
+
+.. code-block:: yaml
+
+   - compiler:
+       modules:
+         - PrgEnv-intel
+         - intel/15.0.109
+
+As mentioned earlier, the compiler paths will look different on a Cray system.
+Since most compilers are invoked using cc, CC and ftn, the paths for each
+compiler are replaced with their respective Cray compiler wrapper names:
+
+.. code-block:: yaml
+
+     paths:
+       cc: cc
+       cxx: CC
+       f77: ftn
+       fc: ftn
+
+As opposed to an explicit path to the compiler executable. This allows Spack
+to call the Cray compiler wrappers during build time.
+
+For more on compiler configuration, check out :ref:`compiler-config`.
+
+Spack sets the default Cray link type to dynamic, to better match other
+other platforms. Individual packages can enable static linking (which is the
+default outside of Spack on cray systems) using the ``-static`` flag.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Setting defaults and using Cray modules
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you want to use default compilers for each PrgEnv and also be able
+to load cray external modules, you will need to set up a ``packages.yaml``.
+
+Here's an example of an external configuration for cray modules:
+
+.. code-block:: yaml
+
+   packages:
+     mpich:
+       externals:
+       - spec: "mpich@7.3.1%gcc@5.2.0 arch=cray_xc-haswell-CNL10"
+         modules:
+         - cray-mpich
+       - spec: "mpich@7.3.1%intel@16.0.0.109 arch=cray_xc-haswell-CNL10"
+         modules:
+         - cray-mpich
+     all:
+       providers:
+         mpi: [mpich]
+
+This tells Spack that for whatever package that depends on mpi, load the
+cray-mpich module into the environment. You can then be able to use whatever
+environment variables, libraries, etc, that are brought into the environment
+via module load.
+
+.. note::
+
+    For Cray-provided packages, it is best to use ``modules:`` instead of ``prefix:``
+    in ``packages.yaml``, because the Cray Programming Environment heavily relies on
+    modules (e.g., loading the ``cray-mpich`` module adds MPI libraries to the
+    compiler wrapper link line).
+
+You can set the default compiler that Spack can use for each compiler type.
+If you want to use the Cray defaults, then set them under ``all:`` in packages.yaml.
+In the compiler field, set the compiler specs in your order of preference.
+Whenever you build with that compiler type, Spack will concretize to that version.
+
+Here is an example of a full packages.yaml used at NERSC
+
+.. code-block:: yaml
+
+   packages:
+     mpich:
+       externals:
+       - spec: "mpich@7.3.1%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge"
+         modules:
+         - cray-mpich
+       - spec: "mpich@7.3.1%intel@16.0.0.109 arch=cray_xc-SuSE11-ivybridge"
+         modules:
+         - cray-mpich
+       buildable: False
+     netcdf:
+       externals:
+       - spec: "netcdf@4.3.3.1%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge"
+         modules:
+         - cray-netcdf
+       - spec: "netcdf@4.3.3.1%intel@16.0.0.109 arch=cray_xc-CNL10-ivybridge"
+         modules:
+         - cray-netcdf
+       buildable: False
+     hdf5:
+       externals:
+       - spec: "hdf5@1.8.14%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge"
+         modules:
+         - cray-hdf5
+       - spec: "hdf5@1.8.14%intel@16.0.0.109 arch=cray_xc-CNL10-ivybridge"
+         modules:
+         - cray-hdf5
+       buildable: False
+     all:
+       compiler: [gcc@5.2.0, intel@16.0.0.109]
+       providers:
+         mpi: [mpich]
+
+Here we tell spack that whenever we want to build with gcc use version 5.2.0 or
+if we want to build with intel compilers, use version 16.0.0.109. We add a spec
+for each compiler type for each cray modules. This ensures that for each
+compiler on our system we can use that external module.
+
+For more on external packages check out the section :ref:`sec-external-packages`.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Using Linux containers on Cray machines
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Spack uses environment variables particular to the Cray programming
+environment to determine which systems are Cray platforms. These
+environment variables may be propagated into containers that are not
+using the Cray programming environment.
+
+To ensure that Spack does not autodetect the Cray programming
+environment, unset the environment variable ``MODULEPATH``. This
+will cause Spack to treat a linux container on a Cray system as a base
+linux distro.
+
 .. _windows_support:
 
 ----------------
@@ -1327,7 +1516,6 @@ Required:
 * Microsoft Visual Studio
 * Python
 * Git
-* 7z
 
 Optional:
 * Intel Fortran (needed for some packages)
@@ -1341,8 +1529,6 @@ Microsoft Visual Studio
 """""""""""""""""""""""
 
 Microsoft Visual Studio provides the only Windows C/C++ compiler that is currently supported by Spack.
-Spack additionally requires that the Windows SDK (including WGL) to be installed as part of your
-visual studio installation as it is required to build many packages from source.
 
 We require several specific components to be included in the Visual Studio installation.
 One is the C/C++ toolset, which can be selected as "Desktop development with C++" or "C++ build tools,"
@@ -1350,7 +1536,6 @@ depending on installation type (Professional, Build Tools, etc.)  The other requ
 "C++ CMake tools for Windows," which can be selected from among the optional packages.
 This provides CMake and Ninja for use during Spack configuration.
 
-
 If you already have Visual Studio installed, you can make sure these components are installed by
 rerunning the installer.  Next to your installation, select "Modify" and look at the
 "Installation details" pane on the right.
@@ -1393,13 +1578,6 @@ as the project providing Git support on Windows. This is additionally the recomm
 for installing Git on Windows, a link to which can be found above. Spack requires the
 utilities vendored by this project.
 
-"""
-7zip
-"""
-
-A tool for extracting ``.xz`` files is required for extracting source tarballs. The latest 7zip
-can be located at https://sourceforge.net/projects/sevenzip/.
-
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Step 2: Install and setup Spack
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -1426,14 +1604,16 @@ in a Windows CMD prompt.
 Step 3: Run and configure Spack
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-On Windows, Spack supports both primary native shells, Powershell and the traditional command prompt.
-To use Spack, pick your favorite shell, and run ``bin\spack_cmd.bat`` or ``share/spack/setup-env.ps1``
-(you may need to Run as Administrator) from the top-level spack
-directory. This will provide a Spack enabled shell. If you receive a warning message that Python is not in your ``PATH``
+To use Spack, run ``bin\spack_cmd.bat`` (you may need to Run as Administrator) from the top-level spack
+directory. This will provide a Windows command prompt with an environment properly set up with Spack
+and its prerequisites. If you receive a warning message that Python is not in your ``PATH``
 (which may happen if you installed Python from the website and not the Windows Store) add the location
 of the Python executable to your ``PATH`` now. You can permanently add Python to your ``PATH`` variable
 by using the ``Edit the system environment variables`` utility in Windows Control Panel.
 
+.. note::
+   Alternatively, Powershell can be used in place of CMD
+
 To configure Spack, first run the following command inside the Spack console:
 
 .. code-block:: console
@@ -1498,7 +1678,7 @@ and not tabs, so ensure that this is the case when editing one directly.
 
 .. note:: Cygwin
    The use of Cygwin is not officially supported by Spack and is not tested.
-   However Spack will not prevent this, so use if choosing to use Spack
+   However Spack will not throw an error, so use if choosing to use Spack
    with Cygwin, know that no functionality is garunteed.
 
 ^^^^^^^^^^^^^^^^^
@@ -1512,12 +1692,21 @@ Spack console via:
 
    spack install cpuinfo
 
-If in the previous step, you did not have CMake or Ninja installed, running the command above should install both packages
+If in the previous step, you did not have CMake or Ninja installed, running the command above should bootstrap both packages
 
-.. note:: Spec Syntax Caveats
-   Windows has a few idiosyncrasies when it comes to the Spack spec syntax and the use of certain shells
-   See the Spack spec syntax doc for more information
+"""""""""""""""""""""""""""
+Windows Compatible Packages
+"""""""""""""""""""""""""""
 
+Not all spack packages currently have Windows support. Some are inherently incompatible with the
+platform, and others simply have yet to be ported. To view the current set of packages with Windows
+support, the list command should be used via `spack list -t windows`. If there's a package you'd like
+to install on Windows but is not in that list, feel free to reach out to request the port or contribute
+the port yourself.
+
+.. note::
+   This is by no means a comprehensive list, some packages may have ports that were not tagged
+   while others may just work out of the box on Windows and have not been tagged as such.
 
 ^^^^^^^^^^^^^^
 For developers
@@ -1527,3 +1716,6 @@ The intent is to provide a Windows installer that will automatically set up
 Python, Git, and Spack, instead of requiring the user to do so manually.
 Instructions for creating the installer are at
 https://github.com/spack/spack/blob/develop/lib/spack/spack/cmd/installer/README.md
+
+Alternatively a pre-built copy of the Windows installer is available as an artifact of Spack's Windows CI
+available at each run of the CI on develop or any PR.

lib/spack/docs/gpu_configuration.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -110,28 +111,3 @@ CUDA is split into fewer components and is simpler to specify:
          prefix: /opt/cuda/cuda-11.0.2/
 
 where ``/opt/cuda/cuda-11.0.2/lib/`` contains ``libcudart.so``.
-
-
-
------------------------------------
-Using an External OpenGL API
------------------------------------
-Depending on whether we have a graphics card or not, we may choose to use OSMesa or GLX to implement the OpenGL API.
-
-If a graphics card is unavailable, OSMesa is recommended and can typically be built with Spack.
-However, if we prefer to utilize the system GLX tailored to our graphics card, we need to declare it as an external. Here's how to do it:
-
-
-.. code-block:: yaml
-
-  packages:
-    libglx:
-      require: [opengl]
-    opengl:
-      buildable: false
-      externals:
-      - prefix: /usr/
-        spec: opengl@4.6
-
-Note that prefix has to be the root of both the libraries and the headers, using is /usr not the path the the lib.
-To know which spec for opengl is available use ``cd /usr/include/GL && grep -Ri gl_version``.

lib/spack/docs/index.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -11,6 +12,10 @@
 Spack
 ===================
 
+.. epigraph::
+
+   `These are docs for the Spack package manager. For sphere packing, see` `pyspack <https://pyspack.readthedocs.io>`_.
+
 Spack is a package management tool designed to support multiple
 versions and configurations of software on a wide variety of platforms
 and environments.  It was designed for large supercomputing centers,
@@ -34,15 +39,10 @@ package:
 
 .. code-block:: console
 
-   $ git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git
+   $ git clone -c feature.manyFiles=true https://github.com/spack/spack.git
    $ cd spack/bin
    $ ./spack install libelf
 
-.. note::
-   ``-c feature.manyFiles=true`` improves git's performance on repositories with 1,000+ files.
-
-   ``--depth=2`` prunes the git history to reduce the size of the Spack installation.
-
 If you're new to spack and want to start using it, see :doc:`getting_started`,
 or refer to the full manual below.
 

lib/spack/docs/mirrors.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 

lib/spack/docs/module_file_support.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -9,7 +10,7 @@ Modules (modules.yaml)
 ======================
 
 The use of module systems to manage user environment in a controlled way
-is a common practice at HPC centers that is sometimes embraced also by
+is a common practice at HPC centers that is often embraced also by
 individual programmers on their development machines. To support this
 common practice Spack integrates with `Environment Modules
 <http://modules.sourceforge.net/>`_ and `Lmod
@@ -20,38 +21,14 @@ Modules are one of several ways you can use Spack packages. For other
 options that may fit your use case better, you should also look at
 :ref:`spack load <spack-load>` and :ref:`environments <environments>`.
 
------------
-Quick start
------------
+----------------------------
+Using module files via Spack
+----------------------------
 
-In the current version of Spack, module files are not generated by default. To get started, you
-can generate module files for all currently installed packages by running either
-
-.. code-block:: console
-
-   $ spack module tcl refresh
-
-or
-
-.. code-block:: console
-
-   $ spack module lmod refresh
-
-Spack can also generate module files for all future installations automatically through the
-following configuration:
-
-.. code-block:: console
-
-   $ spack config add modules:default:enable:[tcl]
-
-or
-
-.. code-block:: console
-
-   $ spack config add modules:default:enable:[lmod]
-
-Assuming you have a module system installed, you should now be able to use the ``module`` command
-to interact with them:
+If you have installed a supported module system you should be able to
+run ``module avail`` to see what module
+files have been installed. Here is sample output of those programs,
+showing lots of installed packages:
 
 .. code-block:: console
 
@@ -88,17 +65,33 @@ scheme used at your site.
 Module file customization
 -------------------------
 
+Module files are generated by post-install hooks after the successful
+installation of a package.
+
+.. note::
+
+   Spack only generates modulefiles when a package is installed. If
+   you attempt to install a package and it is already installed, Spack
+   will not regenerate modulefiles for the package. This may lead to
+   inconsistent modulefiles if the Spack module configuration has
+   changed since the package was installed, either by editing a file
+   or changing scopes or environments.
+
+   Later in this section there is a subsection on :ref:`regenerating
+   modules <cmd-spack-module-refresh>` that will allow you to bring
+   your modules to a consistent state.
+
 The table below summarizes the essential information associated with
 the different file formats that can be generated by Spack:
 
 
-  +-----------+--------------+------------------------------+----------------------------------------------+----------------------+
-  |           | Hierarchical |  **Default root directory**  | **Default template file**                    | **Compatible tools** |
-  +===========+==============+==============================+==============================================+======================+
-  |  ``tcl``  | No           | share/spack/modules          | share/spack/templates/modules/modulefile.tcl | Env. Modules/Lmod    |
-  +-----------+--------------+------------------------------+----------------------------------------------+----------------------+
-  |  ``lmod`` | Yes          | share/spack/lmod             | share/spack/templates/modules/modulefile.lua | Lmod                 |
-  +-----------+--------------+------------------------------+----------------------------------------------+----------------------+
+  +-----------------------------+--------------------+-------------------------------+----------------------------------------------+----------------------+
+  |                             | **Hook name**      |  **Default root directory**   | **Default template file**                    | **Compatible tools** |
+  +=============================+====================+===============================+==============================================+======================+
+  |  **Tcl - Non-Hierarchical** | ``tcl``            |  share/spack/modules          | share/spack/templates/modules/modulefile.tcl | Env. Modules/Lmod    |
+  +-----------------------------+--------------------+-------------------------------+----------------------------------------------+----------------------+
+  |  **Lua - Hierarchical**     | ``lmod``           |  share/spack/lmod             | share/spack/templates/modules/modulefile.lua | Lmod                 |
+  +-----------------------------+--------------------+-------------------------------+----------------------------------------------+----------------------+
 
 
 Spack ships with sensible defaults for the generation of module files, but
@@ -109,7 +102,7 @@ In general you can override or extend the default behavior by:
  2. writing specific rules in the ``modules.yaml`` configuration file
  3. writing your own templates to override or extend the defaults
 
-The former method lets you express changes in the run-time environment
+The former method let you express changes in the run-time environment
 that are needed to use the installed software properly, e.g. injecting variables
 from language interpreters into their extensions. The latter two instead permit to
 fine tune the filesystem layout, content and creation of module files to meet
@@ -117,62 +110,79 @@ site specific conventions.
 
 .. _overide-api-calls-in-package-py:
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Setting environment variables dynamically in ``package.py``
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Override API calls in ``package.py``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-There are two methods that you can implement in any ``package.py`` to dynamically affect the
-content of the module files generated by Spack. The most important one is
-``setup_run_environment``, which can be used to set environment variables in the module file that
-depend on the spec:
+There are two methods that you can override in any ``package.py`` to affect the
+content of the module files generated by Spack. The first one:
 
 .. code-block:: python
 
    def setup_run_environment(self, env):
-       if self.spec.satisfies("+foo"):
-           env.set("FOO", "bar")
+       pass
 
-The second, less commonly used, is ``setup_dependent_run_environment(self, env, dependent_spec)``,
-which allows a dependency to set variables in the module file of its dependents. This is typically
-used in packages like ``python``, ``r``, or ``perl`` to prepend the dependent's prefix to the
-search path of the interpreter (``PYTHONPATH``, ``R_LIBS``, ``PERL5LIB`` resp.), so it can locate
-the packages at runtime.
-
-For example, a simplified version of the ``python`` package could look like this:
+can alter the content of the module file associated with the same package where it is overridden.
+The second method:
 
 .. code-block:: python
 
    def setup_dependent_run_environment(self, env, dependent_spec):
-       if dependent_spec.package.extends(self.spec):
-           env.prepend_path("PYTHONPATH", dependent_spec.prefix.lib.python)
+       pass
 
-and would make any package that ``extends("python")`` have its library directory added to the
-``PYTHONPATH`` environment variable in the module file. It's much more convenient to set this
-variable here, than to repeat it in every Python extension's ``setup_run_environment`` method.
+can instead inject run-time environment modifications in the module files of packages
+that depend on it. In both cases you need to fill ``env`` with the desired
+list of environment modifications.
+
+.. admonition:: The ``r`` package and callback APIs
+
+  An example in which it is crucial to override both methods
+  is given by the ``r`` package. This package installs libraries and headers
+  in non-standard locations and it is possible to prepend the appropriate directory
+  to the corresponding environment variables:
+
+  ================== =================================
+   LD_LIBRARY_PATH    ``self.prefix/rlib/R/lib``
+   PKG_CONFIG_PATH    ``self.prefix/rlib/pkgconfig``
+  ================== =================================
+
+  with the following snippet:
+
+  .. literalinclude:: _spack_root/var/spack/repos/builtin/packages/r/package.py
+     :pyobject: R.setup_run_environment
+
+  The ``r`` package also knows which environment variable should be modified
+  to make language extensions provided by other packages available, and modifies
+  it appropriately in the override of the second method:
+
+  .. literalinclude:: _spack_root/var/spack/repos/builtin/packages/r/package.py
+     :pyobject: R.setup_dependent_run_environment
 
 .. _modules-yaml:
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The ``modules.yaml`` config file and module sets
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+Write a configuration file
+^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The configuration files that control module generation behavior are named ``modules.yaml``. The
-default configuration looks like this:
+The configuration files that control module generation behavior
+are named ``modules.yaml``. The default configuration:
 
 .. literalinclude:: _spack_root/etc/spack/defaults/modules.yaml
    :language: yaml
 
-You can define one or more **module sets**, each of which can be configured separately with regard
-to install location, naming scheme, inclusion and exclusion, autoloading, et cetera.
+activates the hooks to generate ``tcl`` module files and inspects
+the installation folder of each package for the presence of a set of subdirectories
+(``bin``, ``man``, ``share/man``, etc.). If any is found its full path is prepended
+to the environment variables listed below the folder name.
 
-The default module set is aptly named ``default``. All
-:ref:`Spack commands that operate on modules <maintaining-module-files>` apply to the ``default``
-module set, unless another module set is specified explicitly (with the ``--name`` flag).
+Spack modules can be configured for multiple module sets. The default
+module set is named ``default``. All Spack commands which operate on
+modules default to apply the ``default`` module set, but can be
+applied to any module set in the configuration.
 
-
-^^^^^^^^^^^^^^^^^^^^^^^^^
+"""""""""""""""""""""""""
 Changing the modules root
-^^^^^^^^^^^^^^^^^^^^^^^^^
+"""""""""""""""""""""""""
 
 As shown in the table above, the default module root for ``lmod`` is
 ``$spack/share/spack/lmod`` and the default root for ``tcl`` is
@@ -188,7 +198,7 @@ set by changing the ``roots`` key of the configuration.
      my_custom_lmod_modules:
        roots:
          lmod: /path/to/install/custom/lmod/modules
-         # ...
+         ...
 
 This configuration will create two module sets. The default module set
 will install its ``tcl`` modules to ``/path/to/install/tcl/modules``
@@ -214,32 +224,25 @@ location could be confusing to users of your modules. In the next
 section, we will discuss enabling and disabling module types (module
 file generators) for each module set.
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Automatically generating module files
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+""""""""""""""""""""
+Activate other hooks
+""""""""""""""""""""
 
-Spack can be configured to automatically generate module files as part of package installation.
-This is done by adding the desired module systems to the ``enable`` list.
+Any other module file generator shipped with Spack can be activated adding it to the
+list under the ``enable`` key in the module file. Currently the only generator that
+is not active by default is ``lmod``, which produces hierarchical lua module files.
+
+Each module system can then be configured separately. In fact, you should list configuration
+options that affect a particular type of module files under a top level key corresponding
+to the generator being customized:
 
 .. code-block:: yaml
 
    modules:
      default:
        enable:
-       - tcl
-       - lmod
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Configuring ``tcl`` and ``lmod`` modules
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-You can configure the behavior of either module system separately, under a key corresponding to
-the generator being customized:
-
-.. code-block:: yaml
-
-   modules:
-     default:
+         - tcl
+         - lmod
        tcl:
          # contains environment modules specific customizations
        lmod:
@@ -250,82 +253,16 @@ either change the layout of the module files on the filesystem, or they will aff
 their content. For the latter point it is possible to use anonymous specs
 to fine tune the set of packages on which the modifications should be applied.
 
-.. _autoloading-dependencies:
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Autoloading and hiding dependencies
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-A module file should set the variables that are needed for an application to work. But since an
-application often has many dependencies, where should all the environment variables for those be
-set? In Spack the rule is that each package sets the runtime variables that are needed by the
-package itself, and no more. This way, dependencies can be loaded standalone too, and duplication
-of environment variables is avoided.
-
-That means however that if you want to use an application, you need to load the modules for all its
-dependencies. Of course this is not something you would want users to do manually.
-
-Since Spack knows the dependency graph of every package, it can easily generate module files that
-automatically load the modules for its dependencies recursively. It is enabled by default for both
-Lmod and Environment Modules under the ``autoload: direct`` config option. The former system has
-builtin support through the ``depends_on`` function, the latter simply uses a ``module load``
-statement. Both module systems (at least in newer versions) do reference counting, so that if a
-module is loaded by two different modules, it will only be unloaded after the others are.
-
-The ``autoload`` key accepts the values:
-
-  * ``none``: no autoloading
-  * ``run``: autoload direct *run* type dependencies
-  * ``direct``: autoload direct *link and run* type dependencies
-  * ``all``: autoload all dependencies
-
-In case of ``run`` and ``direct``, a ``module load`` triggers a recursive load.
-
-The ``direct`` option is most correct: there are cases where pure link dependencies need to set
-variables for themselves, or need to have variables of their own dependencies set.
-
-In practice however, ``run`` is often sufficient, and may make ``module load`` snappier.
-
-The ``all`` option is discouraged and seldomly used.
-
-A common complaint about autoloading is the large number of modules that are visible to the user.
-Spack has a solution for this as well: ``hide_implicits: true``. This ensures that only those
-packages you've explicitly installed are exposed by ``module avail``, but still allows for
-autoloading of hidden dependencies. Lmod should support hiding implicits in general, while
-Environment Modules requires version 4.7 or higher.
-
-.. note::
-   If supported by your module system, we highly encourage the following configuration that enables
-   autoloading and hiding of implicits. It ensures all runtime variables are set correctly,
-   including those for dependencies, without overwhelming the user with a large number of available
-   modules. Further, it makes it easier to get readable module names without collisions, see the
-   section below on :ref:`modules-projections`.
-
-    .. code-block:: yaml
-  
-        modules:
-          default:
-            tcl:
-              hide_implicits: true
-              all:
-                autoload: direct # or `run`
-            lmod:
-              hide_implicits: true
-              all:
-                autoload: direct # or `run`
-
 .. _anonymous_specs:
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Setting environment variables for selected packages in config
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+""""""""""""""""""""""""""""
+Selection by anonymous specs
+""""""""""""""""""""""""""""
 
-In the configuration file you can filter particular specs, and make further changes to the
-environment variables that go into their module files. This is very powerful when you want to avoid
-:ref:`modifying the package itself <overide-api-calls-in-package-py>`, or when you want to set
-certain variables on multiple selected packages at once.
-
-For instance, in the snippet below:
+In the configuration file you can use *anonymous specs* (i.e. specs
+that **are not required to have a root package** and are thus used just
+to express constraints) to apply certain modifications on a selected set
+of the installed software. For instance, in the snippet below:
 
 .. code-block:: yaml
 
@@ -368,28 +305,12 @@ the variable ``FOOBAR`` will be unset.
 .. note::
    Order does matter
      The modifications associated with the ``all`` keyword are always evaluated
-     first, no matter where they appear in the configuration file. All the other changes to
-     environment variables for matching specs are evaluated from top to bottom.
+     first, no matter where they appear in the configuration file. All the other
+     spec constraints are instead evaluated top to bottom.
 
-.. warning::
-
-     As general advice, it's often better to set as few unnecessary variables as possible. For
-     example, the following seemingly innocent and potentially useful configuration
-
-     .. code-block:: yaml
-
-        all:
-          environment:
-            set:
-              "{name}_ROOT": "{prefix}"
-
-     sets ``BINUTILS_ROOT`` to its prefix in modules for ``binutils``, which happens to break
-     the ``gcc`` compiler: it uses this variable as its default search path for certain object
-     files and libraries, and by merely setting it, everything fails to link.
-
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+""""""""""""""""""""""""""""""""""""""""""""
 Exclude or include specific module files
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+""""""""""""""""""""""""""""""""""""""""""""
 
 You can use anonymous specs also to prevent module files from being written or
 to force them to be written. Consider the case where you want to hide from users
@@ -409,19 +330,14 @@ you will prevent the generation of module files for any package that
 is compiled with ``gcc@4.4.7``, with the only exception of any ``gcc``
 or any ``llvm`` installation.
 
-It is safe to combine ``exclude`` and ``autoload``
-:ref:`mentioned above <autoloading-dependencies>`. When ``exclude`` prevents a module file to be
-generated for a dependency, the ``autoload`` feature will simply not generate a statement to load
-it.
-
 
 .. _modules-projections:
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+"""""""""""""""""""""""""""""""
 Customize the naming of modules
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+"""""""""""""""""""""""""""""""
 
-The names of environment modules generated by Spack are not always easy to
+The names of environment modules generated by spack are not always easy to
 fully comprehend due to the long hash in the name. There are three module
 configuration options to help with that. The first is a global setting to
 adjust the hash length. It can be set anywhere from 0 to 32 and has a default
@@ -437,13 +353,6 @@ shows how to set hash length in the module file names:
        tcl:
          hash_length: 7
 
-.. tip::
-
-   Using ``hide_implicits: true`` (see :ref:`autoloading-dependencies`) vastly reduces the number
-   modules exposed to the user. The hidden modules always contain the hash in their name, and are
-   not influenced by the ``hash_length`` setting. Hidden implicits thus make it easier to use a
-   short hash length or no hash at all, without risking name conflicts.
-
 To help make module names more readable, and to help alleviate name conflicts
 with a short hash, one can use the ``suffixes`` option in the modules
 configuration file. This option will add strings to modules that match a spec.
@@ -456,13 +365,14 @@ For instance, the following config options,
        tcl:
          all:
            suffixes:
-             ^python@3: 'python{^python.version.up_to_2}'
+             ^python@2.7.12: 'python-2.7.12'
              ^openblas: 'openblas'
 
-will add a ``python3.12`` to module names of packages compiled with Python 3.12, and similarly for
-all specs depending on ``python@3``. This is useful to know which version of Python a set of Python
-extensions is associated with. Likewise, the ``openblas`` string is attached to any program that
-has openblas in the spec, most likely via the ``+blas`` variant specification.
+will add a ``python-2.7.12`` version string to any packages compiled with
+python matching the spec, ``python@2.7.12``. This is useful to know which
+version of python a set of python extensions is associated with. Likewise, the
+``openblas`` string is attached to any program that has openblas in the spec,
+most likely via the ``+blas`` variant specification.
 
 The most heavyweight solution to module naming is to change the entire
 naming convention for module files. This uses the projections format
@@ -558,11 +468,41 @@ that are already in the Lmod hierarchy.
    For hierarchies that are deeper than three layers ``lmod spider`` may have some issues.
    See `this discussion on the Lmod project <https://github.com/TACC/Lmod/issues/114>`_.
 
+""""""""""""""""""""""
+Select default modules
+""""""""""""""""""""""
+
+By default, when multiple modules of the same name share a directory,
+the highest version number will be the default module. This behavior
+of the ``module`` command can be overridden with a symlink named
+``default`` to the desired default module. If you wish to configure
+default modules with Spack, add a ``defaults`` key to your modules
+configuration:
+
+.. code-block:: yaml
+
+  modules:
+    my-module-set:
+      tcl:
+        defaults:
+        - gcc@10.2.1
+        - hdf5@1.2.10+mpi+hl%gcc
+
+These defaults may be arbitrarily specific. For any package that
+satisfies a default, Spack will generate the module file in the
+appropriate path, and will generate a default symlink to the module
+file as well.
+
+.. warning::
+  If Spack is configured to generate multiple default packages in the
+  same directory, the last modulefile to be generated will be the
+  default module.
+
 .. _customize-env-modifications:
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+"""""""""""""""""""""""""""""""""""
 Customize environment modifications
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+"""""""""""""""""""""""""""""""""""
 
 You can control which prefixes in a Spack package are added to
 environment variables with the ``prefix_inspections`` section; this
@@ -660,9 +600,9 @@ stack to users who are likely to inspect the modules to find full
 paths to software, when it is desirable to present the users with a
 simpler set of paths than those generated by the Spack install tree.
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+""""""""""""""""""""""""""""""""""""
 Filter out environment modifications
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+""""""""""""""""""""""""""""""""""""
 
 Modifications to certain environment variables in module files are there by
 default, for instance because they are generated by prefix inspections.
@@ -682,37 +622,49 @@ do so by using the ``exclude_env_vars``:
 The configuration above will generate module files that will not contain
 modifications to either ``CPATH`` or ``LIBRARY_PATH``.
 
-^^^^^^^^^^^^^^^^^^^^^^
-Select default modules
-^^^^^^^^^^^^^^^^^^^^^^
 
-By default, when multiple modules of the same name share a directory,
-the highest version number will be the default module. This behavior
-of the ``module`` command can be overridden with a symlink named
-``default`` to the desired default module. If you wish to configure
-default modules with Spack, add a ``defaults`` key to your modules
-configuration:
+.. _autoloading-dependencies:
+
+"""""""""""""""""""""
+Autoload dependencies
+"""""""""""""""""""""
+
+Often it is required for a module to have its (transient) dependencies loaded as well.
+One example where this is useful is when one package needs to use executables provided
+by its dependency; when the dependency is autoloaded, the executable will be in the
+PATH. Similarly for scripting languages such as Python, packages and their dependencies
+have to be loaded together.
+
+Autoloading is enabled by default for Lmod and Environment Modules. The former
+has builtin support for through the ``depends_on`` function. The latter uses
+``module load`` statement to load and track dependencies.
+
+Autoloading can also be enabled conditionally:
 
 .. code-block:: yaml
 
-  modules:
-    my-module-set:
-      tcl:
-        defaults:
-        - gcc@10.2.1
-        - hdf5@1.2.10+mpi+hl%gcc
+    modules:
+      default:
+        tcl:
+          all:
+            autoload: none
+          ^python:
+            autoload: direct
 
-These defaults may be arbitrarily specific. For any package that
-satisfies a default, Spack will generate the module file in the
-appropriate path, and will generate a default symlink to the module
-file as well.
+The configuration file above will produce module files that will
+load their direct dependencies if the package installed depends on ``python``.
+The allowed values for the ``autoload`` statement are either ``none``,
+``direct`` or ``all``.
 
-.. warning::
-  If Spack is configured to generate multiple default packages in the
-  same directory, the last modulefile to be generated will be the
-  default module.
-
-.. _maintaining-module-files:
+.. note::
+  Tcl prerequisites
+     In the ``tcl`` section of the configuration file it is possible to use
+     the ``prerequisites`` directive that accepts the same values as
+     ``autoload``. It will produce module files that have a ``prereq``
+     statement, which autoloads dependencies on Environment Modules when its
+     ``auto_handling`` configuration option is enabled. If Environment Modules
+     is installed with Spack, ``auto_handling`` is enabled by default starting
+     version 4.2. Otherwise it is enabled by default since version 5.0.
 
 ------------------------
 Maintaining Module Files

lib/spack/docs/packages_yaml.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -96,35 +97,6 @@ Each package version and compiler listed in an external should
 have entries in Spack's packages and compiler configuration, even
 though the package and compiler may not ever be built.
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Extra attributes for external packages
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Sometimes external packages require additional attributes to be used
-effectively. This information can be defined on a per-package basis
-and stored in the ``extra_attributes`` section of the external package
-configuration. In addition to per-package information, this section
-can be used to define environment modifications to be performed
-whenever the package is used. For example, if an external package is
-built without ``rpath`` support, it may require ``LD_LIBRARY_PATH``
-settings to find its dependencies. This could be configured as
-follows:
-
-.. code-block:: yaml
-
-   packages:
-     mpich:
-       externals:
-       - spec: mpich@3.3 %clang@12.0.0 +hwloc
-         prefix: /path/to/mpich
-         extra_attributes:
-           environment:
-             prepend_path:
-               LD_LIBRARY_PATH: /path/to/hwloc/lib64
-
-See :ref:`configuration_environment_variables` for more information on
-how to configure environment modifications in Spack config files.
-
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Prevent packages from being built from sources
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -411,33 +383,7 @@ like this:
 
 which means every spec will be required to use ``clang`` as a compiler.
 
-Requirements on variants for all packages are possible too, but note that they
-are only enforced for those packages that define these variants, otherwise they
-are disregarded. For example:
-
-.. code-block:: yaml
-
-   packages:
-     all:
-       require:
-       - "+shared"
-       - "+cuda"
-
-will just enforce ``+shared`` on ``zlib``, which has a boolean ``shared`` variant but
-no ``cuda`` variant.
-
-Constraints in a single spec literal are always considered as a whole, so in a case like:
-
-.. code-block:: yaml
-
-   packages:
-     all:
-       require: "+shared +cuda"
-
-the default requirement will be enforced only if a package has both a ``cuda`` and
-a ``shared`` variant, and will never be partially enforced.
-
-Finally, ``all`` represents a *default set of requirements* -
+Note that in this case ``all`` represents a *default set of requirements* -
 if there are specific package requirements, then the default requirements
 under ``all`` are disregarded. For example, with a configuration like this:
 
@@ -445,18 +391,12 @@ under ``all`` are disregarded. For example, with a configuration like this:
 
    packages:
      all:
-       require:
-       - 'build_type=Debug'
-       - '%clang'
+       require: '%clang'
      cmake:
-       require:
-       - 'build_type=Debug'
-       - '%gcc'
+       require: '%gcc'
 
 Spack requires ``cmake`` to use ``gcc`` and all other nodes (including ``cmake``
-dependencies) to use ``clang``. If enforcing ``build_type=Debug`` is needed also
-on ``cmake``, it must be repeated in the specific ``cmake`` requirements.
-
+dependencies) to use ``clang``.
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Setting requirements on virtual specs
@@ -486,56 +426,6 @@ present. For instance with a configuration like:
 
 you will use ``mvapich2~cuda %gcc`` as an ``mpi`` provider.
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Conflicts and strong preferences
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-If the semantic of requirements is too strong, you can also express "strong preferences" and "conflicts"
-from configuration files:
-
-.. code-block:: yaml
-
-   packages:
-     all:
-       prefer:
-       - '%clang'
-       conflict:
-       - '+shared'
-
-The ``prefer`` and ``conflict`` sections can be used whenever a ``require`` section is allowed.
-The argument is always a list of constraints, and each constraint can be either a simple string,
-or a more complex object:
-
-.. code-block:: yaml
-
-   packages:
-     all:
-       conflict:
-       - spec: '%clang'
-         when: 'target=x86_64_v3'
-         message: 'reason why clang cannot be used'
-
-The ``spec`` attribute is mandatory, while both ``when`` and ``message`` are optional.
-
-.. note::
-
-   Requirements allow for expressing both "strong preferences" and "conflicts".
-   The syntax for doing so, though, may not be immediately clear. For
-   instance, if we want to prevent any package from using ``%clang``, we can set:
-
-   .. code-block:: yaml
-
-      packages:
-        all:
-          require:
-          - one_of: ['%clang', '@:']
-
-   Since only one of the requirements must hold, and ``@:`` is always true, the rule above is
-   equivalent to a conflict. For "strong preferences" we need to substitute the ``one_of`` policy
-   with ``any_of``.
-
-
-
 .. _package-preferences:
 
 -------------------
@@ -646,8 +536,6 @@ manually placed files within the install prefix are owned by the
 assigned group. If no group is assigned, Spack will allow the OS
 default behavior to go as expected.
 
-.. _assigning-package-attributes:
-
 ----------------------------
 Assigning Package Attributes
 ----------------------------

lib/spack/docs/pipelines.rst

@@ -1,4 +1,5 @@
-.. Copyright Spack Project Developers. See COPYRIGHT file for details.
+.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
@@ -58,7 +59,7 @@ Functional Example
 ------------------
 
 The simplest fully functional standalone example of a working pipeline can be
-examined live at this example `project <https://gitlab.com/spack/pipeline-quickstart>`_
+examined live at this example `project <https://gitlab.com/scott.wittenburg/spack-pipeline-demo>`_
 on gitlab.com.
 
 Here's the ``.gitlab-ci.yml`` file from that example that builds and runs the
@@ -66,46 +67,39 @@ pipeline:
 
 .. code-block:: yaml
 
-   stages: [ "generate", "build" ]
+   stages: [generate, build]
 
    variables:
-     SPACK_REPOSITORY: "https://github.com/spack/spack.git"
-     SPACK_REF: "develop-2024-10-06"
-     SPACK_USER_CONFIG_PATH: ${CI_PROJECT_DIR}
-     SPACK_BACKTRACE: 1
+     SPACK_REPO: https://github.com/scottwittenburg/spack.git
+     SPACK_REF: pipelines-reproducible-builds
 
    generate-pipeline:
-     tags:
-       - saas-linux-small-amd64
      stage: generate
+     tags:
+       - docker
      image:
-       name: ghcr.io/spack/ubuntu20.04-runner-x86_64:2023-01-01
-     script:
-       - git clone ${SPACK_REPOSITORY}
-       - cd spack && git checkout ${SPACK_REF} && cd ../
+       name: ghcr.io/scottwittenburg/ecpe4s-ubuntu18.04-runner-x86_64:2020-09-01
+       entrypoint: [""]
+     before_script:
+       - git clone ${SPACK_REPO}
+       - pushd spack && git checkout ${SPACK_REF} && popd
        - . "./spack/share/spack/setup-env.sh"
-       - spack --version
+     script:
        - spack env activate --without-view .
-       - spack -d -v --color=always
-         ci generate
-         --check-index-only
+       - spack -d ci generate
          --artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir"
-         --output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/cloud-ci-pipeline.yml"
+         --output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/pipeline.yml"
      artifacts:
        paths:
          - "${CI_PROJECT_DIR}/jobs_scratch_dir"
 
-   build-pipeline:
+   build-jobs:
      stage: build
      trigger:
        include:
-         - artifact: jobs_scratch_dir/cloud-ci-pipeline.yml
+         - artifact: "jobs_scratch_dir/pipeline.yml"
            job: generate-pipeline
        strategy: depend
-     needs:
-       - artifacts: True
-         job: generate-pipeline
-
 
 The key thing to note above is that there are two jobs: The first job to run,
 ``generate-pipeline``, runs the ``spack ci generate`` command to generate a
@@ -120,93 +114,82 @@ And here's the spack environment built by the pipeline represented as a
    spack:
      view: false
      concretizer:
-       unify: true
-       reuse: false
+       unify: false
 
      definitions:
      - pkgs:
        - zlib
-       - bzip2 ~debug
-     - compiler:
-       - '%gcc'
+       - bzip2
+     - arch:
+       - '%gcc@7.5.0 arch=linux-ubuntu18.04-x86_64'
 
      specs:
      - matrix:
        - - $pkgs
-       - - $compiler
+       - - $arch
+
+     mirrors: { "mirror": "s3://spack-public/mirror" }
 
      ci:
-       target: gitlab
-
+       enable-artifacts-buildcache: True
+       rebuild-index: False
        pipeline-gen:
        - any-job:
-           tags:
-             - saas-linux-small-amd64
-           image:
-             name: ghcr.io/spack/ubuntu20.04-runner-x86_64:2023-01-01
            before_script:
-           - git clone ${SPACK_REPOSITORY}
-           - cd spack && git checkout ${SPACK_REF} && cd ../
-           - . "./spack/share/spack/setup-env.sh"
-           - spack --version
-           - export SPACK_USER_CONFIG_PATH=${CI_PROJECT_DIR}
-           - spack config blame mirrors
+             - git clone ${SPACK_REPO}
+             - pushd spack && git checkout ${SPACK_CHECKOUT_VERSION} && popd
+             - . "./spack/share/spack/setup-env.sh"
+       - build-job:
+           tags: [docker]
+           image:
+             name: ghcr.io/scottwittenburg/ecpe4s-ubuntu18.04-runner-x86_64:2020-09-01
+             entrypoint: [""]
 
 
+The elements of this file important to spack ci pipelines are described in more
+detail below, but there are a couple of things to note about the above working
+example:
+
 .. note::
-   The use of ``reuse: false`` in spack environments used for pipelines is
-   almost always what you want, as without it your pipelines will not rebuild
-   packages even if package hashes have changed. This is due to the concretizer
-   strongly preferring known hashes when ``reuse: true``.
+   There is no ``script`` attribute specified for here. The reason for this is
+   Spack CI will automatically generate reasonable default scripts. More
+   detail on what is in these scripts can be found below.
 
-The ``ci`` section in the above environment file contains the bare minimum
-configuration required for ``spack ci generate`` to create a working pipeline.
-The ``target: gitlab`` tells spack that the desired pipeline output is for
-gitlab.  However, this isn't strictly required, as currently gitlab is the
-only possible output format for pipelines. The ``pipeline-gen`` section
-contains the key information needed to specify attributes for the generated
-jobs.  Notice that it contains a list which has only a single element in
-this case.  In real pipelines it will almost certainly have more elements,
-and in those cases, order is important: spack starts at the bottom of the
-list and works upwards when applying attributes.
+   Also notice the ``before_script`` section. It is required when using any of the
+   default scripts to source the ``setup-env.sh`` script in order to inform
+   the default scripts where to find the ``spack`` executable.
 
-But in this simple case, we use only the special key ``any-job`` to
-indicate that spack should apply the specified attributes (``tags``, ``image``,
-and ``before_script``) to any job it generates.  This includes jobs for
-building/pushing all packages, a ``rebuild-index`` job at the end of the
-pipeline, as well as any ``noop`` jobs that might be needed by gitlab when
-no rebuilds are required.
+Normally ``enable-artifacts-buildcache`` is not recommended in production as it
+results in large binary artifacts getting transferred back and forth between
+gitlab and the runners.  But in this example on gitlab.com where there is no
+shared, persistent file system, and where no secrets are stored for giving
+permission to write to an S3 bucket, ``enabled-buildcache-artifacts`` is the only
+way to propagate binaries from jobs to their dependents.
 
-Something to note is that in this simple case, we rely on spack to
-generate a reasonable script for the package build jobs (it just creates
-a script that invokes ``spack ci rebuild``).
+Also, it is usually a good idea to let the pipeline generate a final "rebuild the
+buildcache index" job, so that subsequent pipeline generation can quickly determine
+which specs are up to date and which need to be rebuilt (it's a good idea for other
+reasons as well, but those are out of scope for this discussion).  In this case we
+have disabled it (using ``rebuild-index: False``) because the index would only be
+generated in the artifacts mirror anyway, and consequently would not be available
+during subsequent pipeline runs.
 
-Another thing to note is the use of the ``SPACK_USER_CONFIG_DIR`` environment
-variable in any generated jobs.  The purpose of this is to make spack
-aware of one final file in the example, the one that contains the mirror
-configuration.  This file, ``mirrors.yaml`` looks like this:
+.. note::
+   With the addition of reproducible builds (#22887) a previously working
+   pipeline will require some changes:
 
-.. code-block:: yaml
+   * In the build-jobs, the environment location changed.
+     This will typically show as a ``KeyError`` in the failing job. Be sure to
+     point to ``${SPACK_CONCRETE_ENV_DIR}``.
 
-   mirrors:
-     buildcache-destination:
-       url: oci://registry.gitlab.com/spack/pipeline-quickstart
-       binary: true
-       access_pair:
-         id_variable: CI_REGISTRY_USER
-         secret_variable: CI_REGISTRY_PASSWORD
+   * When using ``include`` in your environment, be sure to make the included
+     files available in the build jobs. This means adding those files to the
+     artifact directory. Those files will also be missing in the reproducibility
+     artifact.
 
-
-Note the name of the mirror is ``buildcache-destination``, which is required
-as of Spack 0.23 (see below for more information).  The mirror url simply
-points to the container registry associated with the project, while
-``id_variable`` and ``secret_variable`` refer to to environment variables
-containing the access credentials for the mirror.
-
-When spack builds packages for this example project, they will be pushed to
-the project container registry, where they will be available for subsequent
-jobs to install as dependencies, or for other pipelines to use to build runnable
-container images.
+   * Because the location of the environment changed, including files with
+     relative path may have to be adapted to work both in the project context
+     (generation job) and in the concrete env dir context (build job).
 
 -----------------------------------
 Spack commands supporting pipelines
@@ -270,6 +253,17 @@ can easily happen if it is not updated frequently, this behavior ensures that
 spack has a way to know for certain about the status of any concrete spec on
 the remote mirror, but can slow down pipeline generation significantly.
 
+The ``--optimize`` argument is experimental and runs the generated pipeline
+document through a series of optimization passes designed to reduce the size
+of the generated file.
+
+The ``--dependencies`` is also experimental and disables what in Gitlab is
+referred to as DAG scheduling, internally using the ``dependencies`` keyword
+rather than ``needs`` to list dependency jobs.  The drawback of using this option
+is that before any job can begin, all jobs in previous stages must first
+complete.  The benefit is that Gitlab allows more dependencies to be listed
+when using ``dependencies`` instead of ``needs``.
+
 The optional ``--output-file`` argument should be an absolute path (including
 file name) to the generated pipeline, and if not given, the default is
 ``./.gitlab-ci.yml``.
@@ -434,6 +428,15 @@ configuration with a ``script`` attribute. Specifying a signing job without a sc
 does not create a signing job and the job configuration attributes will be ignored.
 Signing jobs are always assigned the runner tags ``aws``, ``protected``, and ``notary``.
 
+^^^^^^^^^^^^^^^^^
+Cleanup (cleanup)
+^^^^^^^^^^^^^^^^^
+
+When using ``temporary-storage-url-prefix`` the cleanup job will destroy the mirror
+created for the associated Gitlab pipeline. Cleanup jobs do not allow modifying the
+script, but do expect that the spack command is in the path and require a
+``before_script`` to be specified that sources the ``setup-env.sh`` script.
+
 .. _noop_jobs:
 
 ^^^^^^^^^^^^
@@ -600,77 +603,6 @@ the attributes will be merged starting from the bottom match going up to the top
 
 In the case that no match is found in a submapping section, no additional attributes will be applied.
 
-
-^^^^^^^^^^^^^^^^^^^^^^^^
-Dynamic Mapping Sections
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-For large scale CI where cost optimization is required, dynamic mapping allows for the use of real-time
-mapping schemes served by a web service. This type of mapping does not support the ``-remove`` type
-behavior, but it does follow the rest of the merge rules for configurations.
-
-The dynamic mapping service needs to implement a single REST API interface for getting
-requests ``GET <URL>[:PORT][/PATH]?spec=<pkg_name@pkg_version +variant1+variant2%compiler@compiler_version>``.
-
-example request.
-
-.. code-block::
-
-  https://my-dyn-mapping.spack.io/allocation?spec=zlib-ng@2.1.6 +compat+opt+shared+pic+new_strategies arch=linux-ubuntu20.04-x86_64_v3%gcc@12.0.0
-
-
-With an example response the updates kubernetes request variables, overrides the max retries for gitlab,
-and prepends a note about the modifications made by the my-dyn-mapping.spack.io service.
-
-.. code-block::
-
-  200 OK
-
-  {
-    "variables":
-    {
-      "KUBERNETES_CPU_REQUEST": "500m",
-      "KUBERNETES_MEMORY_REQUEST": "2G",
-    },
-    "retry": { "max:": "1"}
-    "script+:":
-    [
-      "echo \"Job modified by my-dyn-mapping.spack.io\""
-    ]
-  }
-
-
-The ci.yaml configuration section takes the URL endpoint as well as a number of options to configure how responses are handled.
-
-It is possible to specify a list of allowed and ignored configuration attributes under ``allow`` and ``ignore``
-respectively. It is also possible to configure required attributes under ``required`` section.
-
-Options to configure the client timeout and SSL verification using the ``timeout`` and ``verify_ssl`` options.
-By default, the ``timeout`` is set to the option in ``config:timeout`` and ``veryify_ssl`` is set the the option in ``config::verify_ssl``.
-
-Passing header parameters to the request can be achieved through the ``header`` section. The values of the variables passed to the
-header may be environment variables that are expanded at runtime, such as a private token configured on the runner.
-
-Here is an example configuration pointing to ``my-dyn-mapping.spack.io/allocation``.
-
-
-.. code-block:: yaml
-
-  ci:
-  - dynamic-mapping:
-      endpoint: my-dyn-mapping.spack.io/allocation
-      timeout: 10
-      verify_ssl: True
-      header:
-        PRIVATE_TOKEN: ${MY_PRIVATE_TOKEN}
-        MY_CONFIG: "fuzz_allocation:false"
-      allow:
-      - variables
-      ignore:
-      - script
-      require: []
-
-
 ^^^^^^^^^^^^^
 Bootstrapping
 ^^^^^^^^^^^^^
@@ -742,13 +674,26 @@ build the package.
 
 When including a bootstrapping phase as in the example above, the result is that
 the bootstrapped compiler packages will be pushed to the binary mirror (and the
-local artifacts mirror) before the actual release specs are built.
+local artifacts mirror) before the actual release specs are built. In this case,
+the jobs corresponding to subsequent release specs are configured to
+``install_missing_compilers``, so that if spack is asked to install a package
+with a compiler it doesn't know about, it can be quickly installed from the
+binary mirror first.
 
 Since bootstrapping compilers is optional, those items can be left out of the
 environment/stack file, and in that case no bootstrapping will be done (only the
 specs will be staged for building) and the runners will be expected to already
 have all needed compilers installed and configured for spack to use.
 
+^^^^^^^^^^^^^^^^^^^
+Pipeline Buildcache
+^^^^^^^^^^^^^^^^^^^
+
+The ``enable-artifacts-buildcache`` key
+takes a boolean and determines whether the pipeline uses artifacts to store and
+pass along the buildcaches from one stage to the next (the default if you don't
+provide this option is ``False``).
+
 ^^^^^^^^^^^^^^^^
 Broken Specs URL
 ^^^^^^^^^^^^^^^^
@@ -865,7 +810,7 @@ generated by ``spack ci generate``.  You also want your generated rebuild jobs
 .. code-block:: yaml
 
    spack:
-     # ...
+     ...
      ci:
        pipeline-gen:
        - build-job: