Merge tag 'v0.15.0' into features/shared

* Activate environment in container file

This PR will ensure that the container recipes will build the spack
environment by first activating the environment.

* Deactivate environment before environment collection

For Singularity, the environment must be deactivated before running the
command to collect the environment variables. This is because the
environment collection uses `spack env activate`.

.codecov.yml

@@ -11,6 +11,5 @@ ignore:
   - lib/spack/docs/.*
   - lib/spack/external/.*
   - share/spack/qa/.*
-  - share/spack/spack-completion.bash
 
 comment: off

.coveragerc

@@ -9,6 +9,7 @@ omit =
      lib/spack/spack/test/*
      lib/spack/docs/*
      lib/spack/external/*
+     share/spack/qa/*
 
 [report]
 # Regexes for lines to exclude from consideration

.flake8_packages

@@ -9,7 +9,7 @@
 #
 # Note that we also add *per-line* exemptions for certain patterns in the
 # `spack flake8` command.  This is where F403 for `from spack import *`
-# is added (beause we *only* allow that wildcard).
+# is added (because we *only* allow that wildcard).
 #
 # See .flake8 for regular exceptions.
 #

.gitattributes

@@ -0,0 +1 @@
+*.py diff=python

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,14 +1,11 @@
 ---
-name: Bug report 
-about: Report a bug in the core of Spack (command not working as expected, etc.) 
-labels: bug
+name: "\U0001F41E Bug report"
+about: Report a bug in the core of Spack (command not working as expected, etc.)
+labels: "bug,triage"
 ---
 
-
-*Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
-Example: "I ran Spack find to list all the installed packages and..."*
-
-
+<!-- Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
+Example: "I ran `spack find` to list all the installed packages and ..." -->
 
 ### Steps to reproduce the issue
 
@@ -20,30 +17,26 @@ $ spack <command2> <spec>
 
 ### Error Message
 
-If Spack reported an error, provide the error message. If it did not report an error
-but the output appears incorrect, provide the incorrect output. If there was no error
-message and no output but the result is incorrect, describe how it does not match
-what you expect. To provide more information you might re-run the commands with 
-the additional -d/--stacktrace flags:
+<!-- If Spack reported an error, provide the error message. If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect. -->
 ```console
-$ spack -d --stacktrace <command1> <spec>
-$ spack -d --stacktrace <command2> <spec>
-...
+$ spack --debug --stacktrace <command>
 ```
-that activate the full debug output. 
-
 
 ### Information on your system
 
-This includes:
+<!-- Please include the output of `spack debug report` -->
 
- 1. which platform you are using
- 2. any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.)
+<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->
 
------
+### Additional information
 
-We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!
+<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
+- [ ] I have run `spack debug report` and reported the version of Spack/Python/Platform
+- [ ] I have searched the issues of this repo and believe this is not a duplicate
+- [ ] I have run the failing commands in debug mode and reported the output
+
+<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!
 
 If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.
 
-Other than that, thanks for taking the time to contribute to Spack!
+Other than that, thanks for taking the time to contribute to Spack! -->

.github/ISSUE_TEMPLATE/build_error.md

@@ -1,78 +1,43 @@
 ---
-name: Build error 
-about: Some package in Spack didn't build correctly  
+name: "\U0001F4A5 Build error"
+about: Some package in Spack didn't build correctly
+title: "Installation issue: "
 labels: "build-error"
 ---
 
-*Thanks for taking the time to report this build failure. To proceed with the
-report please:*
+<!-- Thanks for taking the time to report this build failure. To proceed with the report please:
+
 1. Title the issue "Installation issue: <name-of-the-package>".
 2. Provide the information required below.
-3. Remove the template instructions before posting the issue.
 
-We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!
-
-
----
+We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively! -->
 
 ### Steps to reproduce the issue
 
+<!-- Fill in the exact spec you are trying to build and the relevant part of the error message -->
 ```console
-$ spack install <spec> # Fill in the exact spec you are using
-... # and the relevant part of the error message
+$ spack install <spec>
+...
 ```
 
-### Platform and user environment
+### Information on your system
 
-Please report your OS here:
-```commandline
-$ uname -a 
-Linux nuvolari 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
-$ lsb_release -d
-Description:	Ubuntu 18.04.1 LTS
-``` 
-and, if relevant, post or attach:
+<!-- Please include the output of `spack debug report` -->
 
-- `packages.yaml`
-- `compilers.yaml`
-
-to the issue
+<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->
 
 ### Additional information
 
-Sometimes the issue benefits from additional details. In these cases there are
-a few things we can suggest doing. First of all, you can post the full output of:
-```console
-$ spack spec --install-status <spec>
-...
-```
-to show people whether Spack installed a faulty software or if it was not able to
-build it at all. 
+<!-- Please upload the following files. They should be present in the stage directory of the failing build. Also upload any config.log or similar file if one exists. -->
+* [spack-build-out.txt]()
+* [spack-build-env.txt]()
 
-If your build didn't make it past the configure stage, Spack as also commands to parse 
-logs and report error and warning messages:
-```console
-$ spack log-parse --show=errors,warnings <file-to-parse>
-```
-You might want to run this command on the `config.log` or any other similar file
-found in the stage directory: 
-```console
-$ spack location -s <spec>
-```
-In case in `config.log` there are other settings that you think might be the cause 
-of the build failure, you can consider attaching the file to this issue.
+<!-- Some packages have maintainers who have volunteered to debug build failures. Run `spack maintainers <name-of-the-package>` and @mention them here if they exist. -->
 
-Rebuilding the package with the following options:
-```console
-$ spack -d install -j 1 <spec>
-...
-```
-will provide additional debug information. After the failure you will find two files in the current directory:
+### General information
 
-1. `spack-cc-<spec>.in`, which contains details on the command given in input 
-    to Spack's compiler wrapper  
-1. `spack-cc-<spec>.out`, which contains the command used to compile / link the 
-    failed object after Spack's compiler wrapper did its processing 
-
-You can post or attach those files to provide maintainers with more information on what
-is causing the failure.
+<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
+- [ ] I have run `spack debug report` and reported the version of Spack/Python/Platform
+- [ ] I have run `spack maintainers <name-of-the-package>` and @mentioned any maintainers
+- [ ] I have uploaded the build log and environment files
+- [ ] I have searched the issues of this repo and believe this is not a duplicate

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,28 +1,33 @@
 ---
-name: Feature request 
+name: "\U0001F38A Feature request" 
 about: Suggest adding a feature that is not yet in Spack  
 labels: feature
 
 ---
 
-*Please add a concise summary of your suggestion here.*
+<!--*Please add a concise summary of your suggestion here.*-->
 
 ### Rationale
 
-*Is your feature request related to a problem? Please describe it!*
+<!--*Is your feature request related to a problem? Please describe it!*-->
 
 ### Description
 
-*Describe the solution you'd like and the alternatives you have considered.*
+<!--*Describe the solution you'd like and the alternatives you have considered.*-->
 
 
 ### Additional information
-*Add any other context about the feature request here.*
+<!--*Add any other context about the feature request here.*-->
 
 
------
+### General information
+
+- [ ] I have run `spack --version` and reported the version of Spack
+- [ ] I have searched the issues of this repo and believe this is not a duplicate
 
 
-If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.
 
-Other than that, thanks for taking the time to contribute to Spack!
+<!--If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.
+
+Other than that, thanks for taking the time to contribute to Spack!
+-->

.github/actions/add-maintainers-as-reviewers/entrypoint.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 #
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)

.github/workflows/install_spack.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env sh
+git clone https://github.com/spack/spack.git
+echo -e "config:\n  build_jobs: 2" > spack/etc/spack/config.yaml
+. spack/share/spack/setup-env.sh
+spack compilers

.github/workflows/linux_build_tests.yaml

@@ -0,0 +1,61 @@
+name: linux builds
+
+on:
+  push:
+    branches:
+    - master
+    - develop
+    - releases/**
+  pull_request:
+    branches:
+    - master
+    - develop
+    paths-ignore:
+      # Don't run if we only modified packages in the built-in repository
+      - 'var/spack/repos/builtin/**'
+      - '!var/spack/repos/builtin/packages/lz4/**'
+      - '!var/spack/repos/builtin/packages/mpich/**'
+      - '!var/spack/repos/builtin/packages/tut/**'
+      - '!var/spack/repos/builtin/packages/py-setuptools/**'
+      - '!var/spack/repos/builtin/packages/openjpeg/**'
+      - '!var/spack/repos/builtin/packages/r-rcpp/**'
+      # Don't run if we only modified documentation
+      - 'lib/spack/docs/**'
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 4
+      matrix:
+        package: [lz4, mpich, tut, py-setuptools, openjpeg, r-rcpp]
+    steps:
+    - uses: actions/checkout@v2
+    - name: Cache ccache's store
+      uses: actions/cache@v1
+      with:
+        path: ~/.ccache
+        key: ccache-build-${{ matrix.package }}
+        restore-keys: |
+          ccache-build-${{ matrix.package }}
+    - name: Setup Python
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.8
+    - name: Install System Packages
+      run: |
+        sudo apt-get update
+        sudo apt-get -yqq install ccache gfortran perl perl-base r-base r-base-core r-base-dev findutils openssl libssl-dev libpciaccess-dev
+        R --version
+        perl --version
+    - name: Copy Configuration
+      run: |
+        ccache -M 300M && ccache -z
+        # Set up external deps for build tests, b/c they take too long to compile
+        cp share/spack/qa/configuration/*.yaml etc/spack/
+    - name: Run the build test
+      run: |
+        . share/spack/setup-env.sh
+        SPEC=${{ matrix.package }} share/spack/qa/run-build-tests
+        ccache -s

.github/workflows/linux_unit_tests.yaml

@@ -0,0 +1,62 @@
+name: linux tests
+
+on:
+  push:
+    branches:
+      - master
+      - develop
+      - releases/**
+  pull_request:
+    branches:
+      - master
+      - develop
+jobs:
+  unittests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [2.7, 3.5, 3.6, 3.7, 3.8]
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Setup Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install System packages
+      run: |
+          sudo apt-get -y update
+          sudo apt-get install -y coreutils gfortran graphviz gnupg2 mercurial ninja-build patchelf
+          # Needed for kcov
+          sudo apt-get -y install cmake binutils-dev libcurl4-openssl-dev zlib1g-dev libdw-dev libiberty-dev
+    - name: Install Python packages
+      run: |
+          pip install --upgrade pip six setuptools codecov coverage
+    - name: Setup git configuration
+      run: |
+          # Need this for the git tests to succeed.
+          git --version
+          git config --global user.email "spack@example.com"
+          git config --global user.name "Test User"
+          git fetch -u origin develop:develop
+    - name: Install kcov for bash script coverage
+      env:
+          KCOV_VERSION: 34
+      run: |
+          KCOV_ROOT=$(mktemp -d)
+          wget --output-document=${KCOV_ROOT}/${KCOV_VERSION}.tar.gz https://github.com/SimonKagstrom/kcov/archive/v${KCOV_VERSION}.tar.gz
+          tar -C ${KCOV_ROOT} -xzvf ${KCOV_ROOT}/${KCOV_VERSION}.tar.gz
+          mkdir -p ${KCOV_ROOT}/build
+          cd ${KCOV_ROOT}/build && cmake -Wno-dev ${KCOV_ROOT}/kcov-${KCOV_VERSION} && cd -
+          make -C ${KCOV_ROOT}/build && sudo  make -C ${KCOV_ROOT}/build install
+    - name: Run unit tests
+      env:
+          COVERAGE: true
+      run: |
+          share/spack/qa/run-unit-tests
+          coverage combine
+          coverage xml
+    - name: Upload to codecov.io
+      uses: codecov/codecov-action@v1
+      with:
+        flags: unittests,linux

.github/workflows/macos_python.yml

@@ -0,0 +1,58 @@
+# These are nightly package tests for macOS
+# focus areas:
+# - initial user experience
+# - scientific python stack
+name: macOS builds nightly
+
+on:
+  schedule:
+    # nightly at 1 AM
+    - cron: '0 1 * * *'
+
+# GitHub Action Limits
+# https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions
+
+jobs:
+  install_gcc:
+    name: gcc with clang
+    runs-on: macos-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: spack install
+      run: |
+        . .github/workflows/install_spack.sh
+        spack install -v gcc
+
+  install_jupyter_clang:
+    name: jupyter
+    runs-on: macos-latest
+    timeout-minutes: 700
+    steps:
+    - uses: actions/checkout@v2
+    - name: spack install
+      run: |
+        . .github/workflows/install_spack.sh
+        spack install -v py-jupyter %clang
+
+  install_scipy_clang:
+    name: scipy, mpl, pd
+    runs-on: macos-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: spack install
+      run: |
+        . .github/workflows/install_spack.sh
+        spack install -v py-scipy %clang
+        spack install -v py-matplotlib %clang
+        spack install -v py-pandas %clang
+
+  install_mpi4py_clang:
+    name: mpi4py, petsc4py
+    runs-on: macos-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: spack install
+      run: |
+        . .github/workflows/install_spack.sh
+        spack install -v py-mpi4py %clang
+        spack install -v py-petsc4py %clang

.github/workflows/macos_unit_tests.yaml

@@ -0,0 +1,49 @@
+name: macos tests
+
+on:
+  push:
+    branches:
+      - master
+      - develop
+      - releases/**
+  pull_request:
+    branches:
+      - master
+      - develop
+jobs:
+  build:
+
+    runs-on: macos-latest
+    strategy:
+      matrix:
+        python-version: [3.7]
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Setup Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install Python packages
+      run: |
+          pip install --upgrade pip six setuptools
+          pip install --upgrade codecov coverage
+          pip install --upgrade flake8 pep8-naming
+    - name: Setup Homebrew packages
+      run: |
+        brew update
+        brew upgrade
+        brew install gcc gnupg2 dash kcov
+    - name: Run unit tests
+      run: |
+        git --version
+        git fetch -u origin develop:develop
+        . share/spack/setup-env.sh
+        coverage run $(which spack) test
+        coverage combine
+        coverage xml
+    - name: Upload to codecov.io
+      uses: codecov/codecov-action@v1
+      with:
+        file: ./coverage.xml
+        flags: unittests,macos

.github/workflows/minimum_python_versions.yaml

@@ -0,0 +1,31 @@
+name: python version check
+
+on:
+  push:
+    branches:
+      - master
+      - develop
+      - releases/**
+  pull_request:
+    branches:
+      - master
+      - develop
+jobs:
+  validate:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Setup Python
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.7
+    - name: Install Python Packages
+      run: |
+        pip install --upgrade pip
+        pip install --upgrade vermin
+    - name: Minimum Version (Spack's Core)
+      run: vermin --backport argparse -t=2.6- -t=3.5- -v lib/spack/spack/ lib/spack/llnl/ bin/
+    - name: Minimum Version (Repositories)
+      run: vermin --backport argparse -t=2.6- -t=3.5- -v var/spack/repos

.gitignore

@@ -4,6 +4,7 @@
 /var/spack/environments
 /var/spack/repos/*/index.yaml
 /var/spack/repos/*/lock
+__pycache__/
 *.pyc
 /opt
 *~
@@ -22,8 +23,18 @@
 .coverage
 \#*
 .#*
+.cache
 lib/spack/spack/test/.cache
 /bin/spackc
 *.in.log
 *.out.log
 *.orig
+
+# Eclipse files
+.project
+.cproject
+.pydevproject
+
+# VSCode files
+.vscode
+.devcontainer

.gitlab-ci.yml

@@ -1,14 +0,0 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
-# Spack Project Developers. See the top-level COPYRIGHT file for details.
-#
-# SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-generate ci jobs:
-  script:
-    - "./bin/generate-gitlab-ci-yml.sh"
-  tags:
-    - "spack-pre-ci"
-  artifacts:
-    paths:
-      - ci-generation
-    when: always

.travis.yml

@@ -12,7 +12,7 @@ branches:
 # Build matrix
 #=============================================================================
 
-dist: xenial
+dist: bionic
 
 jobs:
   fast_finish: true
@@ -22,100 +22,38 @@ jobs:
       os: linux
       language: python
       env: TEST_SUITE=flake8
-# Shell integration with module files
-    - python: '3.8'
-      os: linux
-      language: python
-      env: [ TEST_SUITE=bootstrap ]
     - stage: 'unit tests + documentation'
       python: '2.6'
       dist: trusty
       os: linux
       language: python
+      addons:
+        apt:
+          # Everything but patchelf, that is not available for trusty
+          packages:
+            - ccache
+            - gfortran
+            - graphviz
+            - gnupg2
+            - kcov
+            - mercurial
+            - ninja-build
+            - realpath
+            - zsh
+            - fish
       env: [ TEST_SUITE=unit, COVERAGE=true ]
-    - python: '2.7'
-      os: linux
-      language: python
-      env: [ TEST_SUITE=unit, COVERAGE=true ]
-    - python: '3.5'
-      os: linux
-      language: python
-      env: TEST_SUITE=unit
-    - python: '3.6'
-      os: linux
-      language: python
-      env: TEST_SUITE=unit
-    - python: '3.7'
-      os: linux
-      language: python
-      env: TEST_SUITE=unit
     - python: '3.8'
       os: linux
       language: python
-      env: [ TEST_SUITE=unit, COVERAGE=true ]
+      env: [ TEST_SUITE=shell, COVERAGE=true, KCOV_VERSION=38 ]
     - python: '3.8'
       os: linux
       language: python
       env: TEST_SUITE=doc
-    - os: osx
-      language: generic
-      env: [ TEST_SUITE=unit, PYTHON_VERSION=2.7, COVERAGE=true ]
-      if: type != pull_request
-# mpich (AutotoolsPackage)
-    - stage: 'build tests'
-      python: '2.7'
-      os: linux
-      language: python
-      env: [ TEST_SUITE=build, 'SPEC=mpich' ]
-# astyle (MakefilePackage)
-    - python: '3.8'
-      os: linux
-      language: python
-      env: [ TEST_SUITE=build, 'SPEC=astyle' ]
-# tut (WafPackage)
-    - python: '3.8'
-      os: linux
-      language: python
-      env: [ TEST_SUITE=build, 'SPEC=tut' ]
-# py-setuptools (PythonPackage)
-    - python: '3.8'
-      os: linux
-      language: python
-      env: [ TEST_SUITE=build, 'SPEC=py-setuptools' ]
-# perl-dbi (PerlPackage)
-#    - python: '3.8'
-#      os: linux
-#      language: python
-#      env: [ TEST_SUITE=build, 'SPEC=perl-dbi' ]
-# openjpeg (CMakePackage + external cmake)
-    - python: '3.8'
-      os: linux
-      language: python
-      env: [ TEST_SUITE=build, 'SPEC=openjpeg' ]
-# r-rcpp (RPackage + external R)
-    - python: '3.8'
-      os: linux
-      language: python
-      env: [ TEST_SUITE=build, 'SPEC=r-rcpp' ]
-# mpich (AutotoolsPackage)
-    - python: '3.8'
-      os: linux
-      language: python
-      env: [ TEST_SUITE=build, 'SPEC=mpich' ]
-    - python: '3.8'
-      stage: 'docker build'
-      os: linux
-      language: python
-      env: TEST_SUITE=docker
-  allow_failures:
-    - env: TEST_SUITE=docker
 
 stages:
   - 'style checks'
   - 'unit tests + documentation'
-  - 'build tests'
-  - name: 'docker build'
-    if: type = push AND branch IN (develop, master)
 
 
 #=============================================================================
@@ -128,29 +66,15 @@ addons:
   apt:
     packages:
       - ccache
-      - cmake
+      - coreutils
       - gfortran
       - graphviz
       - gnupg2
-      - kcov
       - mercurial
       - ninja-build
-      - perl
-      - perl-base
-      - realpath
-      - r-base
-      - r-base-core
-      - r-base-dev
+      - patchelf
       - zsh
-  # for Mac builds, we use Homebrew
-  homebrew:
-    packages:
-      - python@2
-      - gcc
-      - gnupg2
-      - ccache
-      - dash
-      - kcov
+      - fish
     update: true
 
 # ~/.ccache needs to be cached directly as Travis is not taking care of it
@@ -159,25 +83,27 @@ cache:
   pip: true
   ccache: true
   directories:
-    - ~/.mirror
     - ~/.ccache
 
-# Work around Travis's lack of support for Python on OSX
 before_install:
-  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
-        pip2 install --upgrade pip;
-        pip2 install virtualenv;
-        virtualenv venv;
-        source venv/bin/activate;
-    fi
   - ccache -M 2G && ccache -z
+  # Install kcov manually, since it's not packaged for bionic beaver
+  - if [[ "$KCOV_VERSION" ]]; then
+        sudo apt-get -y install cmake binutils-dev libcurl4-openssl-dev zlib1g-dev libdw-dev libiberty-dev;
+        KCOV_ROOT=$(mktemp -d);
+        wget --output-document=${KCOV_ROOT}/${KCOV_VERSION}.tar.gz https://github.com/SimonKagstrom/kcov/archive/v${KCOV_VERSION}.tar.gz;
+        tar -C ${KCOV_ROOT} -xzvf ${KCOV_ROOT}/${KCOV_VERSION}.tar.gz;
+        mkdir -p ${KCOV_ROOT}/build;
+        cd ${KCOV_ROOT}/build && cmake -Wno-dev ${KCOV_ROOT}/kcov-${KCOV_VERSION} && cd - ;
+        make -C ${KCOV_ROOT}/build && sudo  make -C ${KCOV_ROOT}/build install;
+    fi
 
 # Install various dependencies
 install:
   - pip install --upgrade pip
   - pip install --upgrade six
   - pip install --upgrade setuptools
-  - pip install --upgrade codecov
+  - pip install --upgrade codecov coverage==4.5.4
   - pip install --upgrade flake8
   - pip install --upgrade pep8-naming
   - if [[ "$TEST_SUITE" == "doc" ]]; then
@@ -192,17 +118,9 @@ before_script:
   # Need this to be able to compute the list of changed files
   - git fetch origin ${TRAVIS_BRANCH}:${TRAVIS_BRANCH}
 
-  # Set up external deps for build tests, b/c they take too long to compile
-  - if [[ "$TEST_SUITE" == "build" ]]; then
-        cp share/spack/qa/configuration/*.yaml etc/spack/;
-    fi
-
 #=============================================================================
 # Building
 #=============================================================================
-services:
-  - docker
-
 script:
   - share/spack/qa/run-$TEST_SUITE-tests
 
@@ -216,6 +134,10 @@ after_success:
                         --flags "${TEST_SUITE}${TRAVIS_OS_NAME}";
             fi
             ;;
+        shell)
+            codecov --env PYTHON_VERSION
+            --required
+            --flags "${TEST_SUITE}${TRAVIS_OS_NAME}";
     esac
 
 #=============================================================================
@@ -223,6 +145,8 @@ after_success:
 #=============================================================================
 notifications:
   email:
-    recipients: tgamblin@llnl.gov
+    recipients:
+    - tgamblin@llnl.gov
+    - massimiliano.culpo@gmail.com
     on_success: change
     on_failure: always

CHANGELOG.md

@@ -1,3 +1,286 @@
+# v0.15.0 (2020-06-28)
+
+`v0.15.0` is a major feature release.
+
+## Major Features in this release
+
+1. **Cray support** Spack will now work properly on Cray "Cluster"
+systems (non XC systems) and after a `module purge` command on Cray
+systems. See #12989
+
+2. **Virtual package configuration** Virtual packages are allowed in
+packages.yaml configuration. This allows users to specify a virtual
+package as non-buildable without needing to specify for each
+implementation. See #14934
+
+3. **New config subcommands** This release adds `spack config add` and
+`spack config remove` commands to add to and remove from yaml
+configuration files from the CLI. See #13920
+
+4. **Environment activation** Anonymous environments are **no longer**
+automatically activated in the current working directory. To activate
+an environment from a `spack.yaml` file in the current directory, use
+the `spack env activate .` command. This removes a concern that users
+were too easily polluting their anonymous environments with accidental
+installations. See #17258
+
+5. **Apple clang compiler** The clang compiler and the apple-clang
+compiler are now separate compilers in Spack. This allows Spack to
+improve support for the apple-clang compiler. See #17110
+
+6. **Finding external packages** Spack packages can now support an API
+for finding external installations. This allows the `spack external
+find` command to automatically add installations of those packages to
+the user's configuration. See #15158
+
+
+## Additional new features of note
+
+* support for using Spack with the fish shell (#9279)
+* `spack load --first` option to load first match (instead of prompting user) (#15622)
+* support the Cray cce compiler both new and classic versions (#17256, #12989)
+* `spack dev-build` command:
+  * supports stopping before a specified phase (#14699)
+  * supports automatically launching a shell in the build environment (#14887)
+* `spack install --fail-fast` allows builds to fail at the first error (rather than best-effort) (#15295)
+* environments: SpecList references can be dereferenced as compiler or dependency constraints (#15245)
+* `spack view` command: new support for a copy/relocate view type (#16480)
+* ci pipelines: see documentation for several improvements
+* `spack mirror -a` command now supports excluding packages (#14154)
+* `spack buildcache create` is now environment-aware (#16580)
+* module generation: more flexible format for specifying naming schemes (#16629)
+* lmod module generation: packages can be configured as core specs for lmod hierarchy (#16517)
+
+## Deprecations and Removals
+
+The following commands were deprecated in v0.13.0, and have now been removed:
+
+* `spack configure`
+* `spack build`
+* `spack diy`
+
+The following commands were deprecated in v0.14.0, and will be removed in the next major release:
+
+* `spack bootstrap`
+
+## Bugfixes
+
+Some of the most notable bugfixes in this release include:
+
+* Spack environments can now contain the string `-h` (#15429)
+* The `spack install` gracefully handles being backgrounded (#15723, #14682)
+* Spack uses `-isystem` instead of `-I` in cases that the underlying build system does as well (#16077)
+* Spack no longer prints any specs that cannot be safely copied into a Spack command (#16462)
+* Incomplete Spack environments containing python no longer cause problems (#16473)
+* Several improvements to binary package relocation
+
+## Package Improvements
+
+The Spack project is constantly engaged in routine maintenance,
+bugfixes, and improvements for the package ecosystem. Of particular
+note in this release are the following:
+
+* Spack now contains 4339 packages. There are 430 newly supported packages in v0.15.0
+* GCC now builds properly on ARM architectures (#17280)
+* Python: patched to support compiling mixed C/C++ python modules through distutils (#16856)
+* improvements to pytorch and py-tensorflow packages
+* improvements to major MPI implementations: mvapich2, mpich, openmpi, and others
+
+## Spack Project Management:
+
+* Much of the Spack CI infrastructure has moved from Travis to GitHub Actions (#16610, #14220, #16345)
+* All merges to the `develop` branch run E4S CI pipeline (#16338)
+* New `spack debug report` command makes reporting bugs easier (#15834)
+
+# v0.14.2 (2020-04-15)
+
+This is a minor release on the `0.14` series. It includes performance
+improvements and bug fixes:
+
+* Improvements to how `spack install` handles foreground/background (#15723)
+* Major performance improvements for reading the package DB (#14693, #15777)
+* No longer check for the old `index.yaml` database file (#15298)
+* Properly activate environments with '-h' in the name (#15429)
+* External packages have correct `.prefix` in environments/views (#15475)
+* Improvements to computing env modifications from sourcing files (#15791)
+* Bugfix on Cray machines when getting `TERM` env variable (#15630)
+* Avoid adding spurious `LMOD` env vars to Intel modules (#15778)
+* Don't output [+] for mock installs run during tests (#15609)
+
+# v0.14.1 (2020-03-20)
+
+This is a bugfix release on top of `v0.14.0`.  Specific fixes include:
+
+* several bugfixes for parallel installation (#15339, #15341, #15220, #15197)
+* `spack load` now works with packages that have been renamed (#14348)
+* bugfix for `suite-sparse` installation (#15326)
+* deduplicate identical suffixes added to module names (#14920)
+* fix issues with `configure_args` during module refresh (#11084)
+* increased test coverage and test fixes (#15237, #15354, #15346)
+* remove some unused code (#15431)
+
+# v0.14.0 (2020-02-23)
+
+`v0.14.0` is a major feature release, with 3 highlighted features:
+
+1. **Distributed builds.** Multiple Spack instances will now coordinate
+   properly with each other through locks. This works on a single node
+   (where you've called `spack` several times) or across multiple nodes
+   with a shared filesystem. For example, with SLURM, you could build
+   `trilinos` and its dependencies on 2 24-core nodes, with 3 Spack
+   instances per node and 8 build jobs per instance, with `srun -N 2 -n 6
+   spack install -j 8 trilinos`. This requires a filesystem with locking
+   enabled, but not MPI or any other library for parallelism.
+
+2.  **Build pipelines.** You can also build in parallel through Gitlab
+   CI. Simply create a Spack environment and push it to Gitlab to build
+   on Gitlab runners. Pipeline support is now integrated into a single
+   `spack ci` command, so setting it up is easier than ever.  See the
+   [Pipelines section](https://spack.readthedocs.io/en/v0.14.0/pipelines.html)
+   in the docs.
+
+3. **Container builds.** The new `spack containerize` command allows you
+   to create a Docker or Singularity recipe from any Spack environment.
+   There are options to customize the build if you need them. See the
+   [Container Images section](https://spack.readthedocs.io/en/latest/containers.html)
+   in the docs.
+
+In addition, there are several other new commands, many bugfixes and
+improvements, and `spack load` no longer requires modules, so you can use
+it the same way on your laptop or on your supercomputer.
+
+Spack grew by over 300 packages since our last release in November 2019,
+and the project grew to over 500 contributors.  Thanks to all of you for
+making yet another great release possible. Detailed notes below.
+
+## Major new core features
+* Distributed builds: spack instances coordinate and build in parallel (#13100)
+* New `spack ci` command to manage CI pipelines (#12854)
+* Generate container recipes from environments: `spack containerize` (#14202)
+* `spack load` now works without using modules (#14062, #14628)
+* Garbage collect old/unused installations with `spack gc` (#13534)
+* Configuration files all set environment modifications the same way (#14372,
+  [docs](https://spack.readthedocs.io/en/v0.14.0/configuration.html#environment-modifications))
+* `spack commands --format=bash` auto-generates completion (#14393, #14607)
+* Packages can specify alternate fetch URLs in case one fails (#13881)
+
+## Improvements
+* Improved locking for concurrency with environments (#14676, #14621, #14692)
+* `spack test` sends args to `pytest`, supports better listing (#14319)
+* Better support for aarch64 and cascadelake microarch (#13825, #13780, #13820)
+* Archspec is now a separate library (see https://github.com/archspec/archspec)
+* Many improvements to the `spack buildcache` command (#14237, #14346,
+  #14466, #14467, #14639, #14642, #14659, #14696, #14698, #14714, #14732,
+  #14929, #15003, #15086, #15134)
+
+## Selected Bugfixes
+* Compilers now require an exact match on version (#8735, #14730, #14752)
+* Bugfix for patches that specified specific versions (#13989)
+* `spack find -p` now works in environments (#10019, #13972)
+* Dependency queries work correctly in `spack find` (#14757)
+* Bugfixes for locking upstream Spack instances chains (#13364)
+* Fixes for PowerPC clang optimization flags (#14196)
+* Fix for issue with compilers and specific microarchitectures (#13733, #14798)
+
+## New commands and options
+* `spack ci` (#12854)
+* `spack containerize` (#14202)
+* `spack gc` (#13534)
+* `spack load` accepts `--only package`, `--only dependencies` (#14062, #14628)
+* `spack commands --format=bash` (#14393)
+* `spack commands --update-completion` (#14607)
+* `spack install --with-cache` has new option: `--no-check-signature` (#11107)
+* `spack test` now has `--list`, `--list-long`, and `--list-names` (#14319)
+* `spack install --help-cdash` moves CDash help out of the main help (#13704)
+
+## Deprecations
+* `spack release-jobs` has been rolled into `spack ci`
+* `spack bootstrap` will be removed in a future version, as it is no longer
+  needed to set up modules (see `spack load` improvements above)
+
+## Documentation
+* New section on building container images with Spack (see
+  [docs](https://spack.readthedocs.io/en/latest/containers.html))
+* New section on using `spack ci` command to build pipelines (see
+  [docs](https://spack.readthedocs.io/en/latest/pipelines.html))
+* Document how to add conditional dependencies (#14694)
+* Document how to use Spack to replace Homebrew/Conda (#13083, see
+  [docs](https://spack.readthedocs.io/en/latest/workflows.html#using-spack-to-replace-homebrew-conda))
+
+## Important package changes
+* 3,908 total packages (345 added since 0.13.0)
+* Added first cut at a TensorFlow package (#13112)
+* We now build R without "recommended" packages, manage them w/Spack (#12015)
+* Elpa and OpenBLAS now leverage microarchitecture support (#13655, #14380)
+* Fix `octave` compiler wrapper usage (#14726)
+* Enforce that packages in `builtin` aren't missing dependencies (#13949)
+
+
+# v0.13.4 (2020-02-07)
+
+This release contains several bugfixes:
+
+* bugfixes for invoking python in various environments (#14349, #14496, #14569)
+* brought tab completion up to date (#14392)
+* bugfix for removing extensions from views in order (#12961)
+* bugfix for nondeterministic hashing for specs with externals (#14390)
+
+# v0.13.3 (2019-12-23)
+
+This release contains more major performance improvements for Spack
+environments, as well as bugfixes for mirrors and a `python` issue with
+RHEL8.
+
+* mirror bugfixes: symlinks, duplicate patches, and exception handling (#13789)
+* don't try to fetch `BundlePackages` (#13908)
+* avoid re-fetching patches already added to a mirror (#13908)
+* avoid re-fetching already added patches (#13908)
+* avoid re-fetching already added patches (#13908)
+* allow repeated invocations of `spack mirror create` on the same dir (#13908)
+* bugfix for RHEL8 when `python` is unavailable (#14252)
+* improve concretization performance in environments (#14190)
+* improve installation performance in environments (#14263)
+
+# v0.13.2 (2019-12-04)
+
+This release contains major performance improvements for Spack environments, as
+well as some bugfixes and minor changes.
+
+* allow missing modules if they are blacklisted (#13540)
+* speed up environment activation (#13557)
+* mirror path works for unknown versions (#13626)
+* environments: don't try to modify run-env if a spec is not installed (#13589)
+* use semicolons instead of newlines in module/python command (#13904)
+* verify.py: os.path.exists exception handling (#13656)
+* Document use of the maintainers field (#13479)
+* bugfix with config caching (#13755)
+* hwloc: added 'master' version pointing at the HEAD of the master branch (#13734)
+* config option to allow gpg warning suppression (#13744)
+* fix for relative symlinks when relocating binary packages (#13727)
+* allow binary relocation of strings in relative binaries (#13724)
+
+# v0.13.1 (2019-11-05)
+
+This is a bugfix release on top of `v0.13.0`.  Specific fixes include:
+
+* `spack find` now displays variants and other spec constraints
+* bugfix: uninstall should find concrete specs by DAG hash (#13598)
+* environments: make shell modifications partially unconditional (#13523)
+* binary distribution: relocate text files properly in relative binaries (#13578)
+* bugfix: fetch prefers to fetch local mirrors over remote resources (#13545)
+* environments: only write when necessary (#13546)
+* bugfix: spack.util.url.join() now handles absolute paths correctly (#13488)
+* sbang: use utf-8 for encoding when patching (#13490)
+* Specs with quoted flags containing spaces are parsed correctly (#13521)
+* targets: print a warning message before downgrading (#13513)
+* Travis CI: Test Python 3.8 (#13347)
+* Documentation: Database.query methods share docstrings (#13515)
+* cuda: fix conflict statements for x86-64 targets (#13472)
+* cpu: fix clang flags for generic x86_64 (#13491)
+* syaml_int type should use int.__repr__ rather than str.__repr__ (#13487)
+* elpa: prefer 2016.05.004 until sse/avx/avx2 issues are resolved (#13530)
+* trilinos: temporarily constrain netcdf@:4.7.1 (#13526)
+
 # v0.13.0 (2019-10-25)
 
 `v0.13.0` is our biggest Spack release yet, with *many* new major features.

COPYRIGHT

@@ -68,10 +68,6 @@ PackageName: py
 PackageHomePage: https://pypi.python.org/pypi/py
 PackageLicenseDeclared: MIT
 
-PackageName: pyqver
-PackageHomePage: https://github.com/ghewgill/pyqver
-PackageLicenseDeclared: BSD-3-Clause
-
 PackageName: pytest
 PackageHomePage: https://pypi.python.org/pypi/pytest
 PackageLicenseDeclared: MIT

LICENSE-MIT

@@ -1,4 +1,4 @@
-Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 Spack Project Developers. See the top-level COPYRIGHT file for details.
 
 Permission is hereby granted, free of charge, to any person obtaining a

README.md

@@ -1,6 +1,10 @@
 # <img src="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo.svg" width="64" valign="middle" alt="Spack"/> Spack
 
-[![Build Status](https://travis-ci.org/spack/spack.svg?branch=develop)](https://travis-ci.org/spack/spack)
+[![MacOS Tests](https://github.com/spack/spack/workflows/macos%20tests/badge.svg)](https://github.com/spack/spack/actions)
+[![Linux Tests](https://github.com/spack/spack/workflows/linux%20tests/badge.svg)](https://github.com/spack/spack/actions)
+[![Linux Builds](https://github.com/spack/spack/workflows/linux%20builds/badge.svg)](https://github.com/spack/spack/actions)
+[![macOS Builds (nightly)](https://github.com/spack/spack/workflows/macOS%20builds%20nightly/badge.svg?branch=develop)](https://github.com/spack/spack/actions?query=workflow%3A%22macOS+builds+nightly%22)
+[![Build Status](https://travis-ci.com/spack/spack.svg?branch=develop)](https://travis-ci.com/spack/spack)
 [![codecov](https://codecov.io/gh/spack/spack/branch/develop/graph/badge.svg)](https://codecov.io/gh/spack/spack)
 [![Read the Docs](https://readthedocs.org/projects/spack/badge/?version=latest)](https://spack.readthedocs.io)
 [![Slack](https://spackpm.herokuapp.com/badge.svg)](https://spackpm.herokuapp.com)
@@ -119,4 +123,4 @@ See [LICENSE-MIT](https://github.com/spack/spack/blob/develop/LICENSE-MIT),
 
 SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
-LLNL-CODE-647188
+LLNL-CODE-811652

bin/generate-gitlab-ci-yml.sh

@@ -1,91 +0,0 @@
-#! /usr/bin/env bash
-
-# Remember where we are initially, it's the repo cloned by gitlab-ci
-original_directory=$(pwd)
-. "${original_directory}/share/spack/setup-env.sh"
-
-# Create a temporary working directory
-temp_dir=$(mktemp -d)
-trap 'rm -rf "$temp_dir"' INT TERM QUIT EXIT
-
-if [ -z "${DOWNSTREAM_CI_REPO}" ] ; then
-    echo "ERROR: missing variable: DOWNSTREAM_CI_REPO" >&2
-    exit 1
-fi
-
-if [ -z "${SPACK_RELEASE_ENVIRONMENT_PATH}" ] ; then
-    echo "ERROR: missing variable: SPACK_RELEASE_ENVIRONMENT_PATH" >&2
-    exit 1
-fi
-
-if [ -z "${CDASH_AUTH_TOKEN}" ] ; then
-    echo "WARNING: missing variable: CDASH_AUTH_TOKEN" >&2
-else
-    token_file="${temp_dir}/cdash_auth_token"
-    echo ${CDASH_AUTH_TOKEN} > ${token_file}
-fi
-
-if [ -z "${SPACK_RELEASE_ENVIRONMENT_REPO}" ] ; then
-    echo "Assuming spack repo contains environment" >&2
-    env_repo_dir=${original_directory}
-else
-    echo "Cloning ${SPACK_RELEASE_ENVIRONMENT_REPO} into ${temp_dir}/envrepo" >&2
-    cd ${temp_dir}
-    git clone ${SPACK_RELEASE_ENVIRONMENT_REPO} envrepo
-    cd envrepo
-    env_repo_dir=$(pwd)
-fi
-
-current_branch="$CI_COMMIT_REF_NAME"
-
-# Because want to see generated gitlab-ci file as an artifact,
-# we need to write it within the spack repo cloned by gitlab-ci.
-gen_ci_dir="${original_directory}/ci-generation"
-gen_ci_file="${gen_ci_dir}/.gitlab-ci.yml"
-mkdir -p ${gen_ci_dir}
-
-env_dir="${env_repo_dir}/${SPACK_RELEASE_ENVIRONMENT_PATH}"
-
-if [ ! -f "${env_dir}/spack.yaml" ] ; then
-    echo "ERROR: Cannot find spack environment file in ${env_dir}"
-    exit 1
-fi
-
-cd $env_dir
-
-# The next commands generates the .gitlab-ci.yml (and optionally creates a
-# buildgroup in cdash)
-RELEASE_JOBS_ARGS=("--output-file" "${gen_ci_file}")
-if [ ! -z "${token_file}" ]; then
-    RELEASE_JOBS_ARGS+=("--cdash-credentials" "${token_file}")
-fi
-
-spack release-jobs "${RELEASE_JOBS_ARGS[@]}"
-
-if [[ $? -ne 0 ]]; then
-    echo "spack release-jobs command failed"
-    exit 1
-fi
-
-cd "$original_directory"
-mv .git "$temp_dir/original-git-dir"
-git init .
-
-git config user.email "robot@spack.io"
-git config user.name "Spack Build Bot"
-
-cp ${gen_ci_file} "${original_directory}/.gitlab-ci.yml"
-git add .
-
-echo "git commit"
-commit_message="Auto-generated commit testing"
-commit_message="${commit_message} ${current_branch} (${CI_COMMIT_SHA})"
-git commit --message="${commit_message}"
-
-echo "git push"
-git remote add origin "$DOWNSTREAM_CI_REPO"
-git push --force origin "master:multi-ci-${current_branch}"
-
-rm -rf .git
-mv "$temp_dir/original-git-dir" .git
-git reset --hard HEAD

bin/rebuild-index.sh

@@ -1,13 +0,0 @@
-#!/bin/bash
-
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
-# Spack Project Developers. See the top-level COPYRIGHT file for details.
-#
-# SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-set -x
-
-SPACK_BIN_DIR="${CI_PROJECT_DIR}/bin"
-export PATH="${SPACK_BIN_DIR}:${PATH}"
-
-spack buildcache update-index -d "$MIRROR_URL"

bin/rebuild-package.sh

@@ -1,399 +0,0 @@
-#!/bin/bash
-
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
-# Spack Project Developers. See the top-level COPYRIGHT file for details.
-#
-# SPDX-License-Identifier: (Apache-2.0 OR MIT)
-
-###
-### This script represents a gitlab-ci job, corresponding to a single release
-### spec.  As such this script must first decide whether or not the spec it
-### has been assigned is up to date on the remote binary mirror.  If it is
-### not (i.e. the source code has changed in a way that caused a change in the
-### full_hash of the spec), this script will build the package, create a
-### binary cache for it, and then push all related files to the remote binary
-### mirror.  This script also optionally communicates with a remote CDash
-### instance to share status on the package build process.
-###
-### The following environment variables are (possibly) used within this script
-### in order for the various elements function properly.
-###
-### First are two defaults we rely on from gitlab:
-###
-### CI_PROJECT_DIR
-### CI_JOB_NAME
-###
-### The following must be set up in the variables section of gitlab:
-###
-### AWS_ACCESS_KEY_ID
-### AWS_SECRET_ACCESS_KEY
-### SPACK_SIGNING_KEY
-###
-### SPACK_S3_UPLOAD_MIRROR_URL         // only required in the short term for the cloud case
-###
-### The following variabes are defined by the ci generation process and are
-### required:
-###
-### SPACK_ENABLE_CDASH
-### SPACK_ROOT_SPEC
-### SPACK_MIRROR_URL
-### SPACK_JOB_SPEC_PKG_NAME
-### SPACK_COMPILER_ACTION
-###
-### Finally, these variables are optionally defined by the ci generation
-### process, and may or may not be present:
-###
-### SPACK_CDASH_BASE_URL
-### SPACK_CDASH_PROJECT
-### SPACK_CDASH_PROJECT_ENC
-### SPACK_CDASH_BUILD_NAME
-### SPACK_CDASH_SITE
-### SPACK_RELATED_BUILDS
-### SPACK_JOB_SPEC_BUILDGROUP
-###
-
-shopt -s expand_aliases
-
-export FORCE_UNSAFE_CONFIGURE=1
-
-TEMP_DIR="${CI_PROJECT_DIR}/jobs_scratch_dir"
-
-JOB_LOG_DIR="${TEMP_DIR}/logs"
-SPEC_DIR="${TEMP_DIR}/specs"
-LOCAL_MIRROR="${CI_PROJECT_DIR}/local_mirror"
-BUILD_CACHE_DIR="${LOCAL_MIRROR}/build_cache"
-SPACK_BIN_DIR="${CI_PROJECT_DIR}/bin"
-
-if [ "${SPACK_ENABLE_CDASH}" == "True" ] ; then
-    CDASH_UPLOAD_URL="${SPACK_CDASH_BASE_URL}/submit.php?project=${SPACK_CDASH_PROJECT_ENC}"
-    DEP_JOB_RELATEBUILDS_URL="${SPACK_CDASH_BASE_URL}/api/v1/relateBuilds.php"
-    declare -a JOB_DEPS_PKG_NAMES
-fi
-
-export SPACK_ROOT=${CI_PROJECT_DIR}
-# export PATH="${SPACK_BIN_DIR}:${PATH}"
-export GNUPGHOME="${CI_PROJECT_DIR}/opt/spack/gpg"
-
-. "${CI_PROJECT_DIR}/share/spack/setup-env.sh"
-
-mkdir -p ${JOB_LOG_DIR}
-mkdir -p ${SPEC_DIR}
-
-cleanup() {
-    set +x
-
-    if [ -z "$exit_code" ] ; then
-
-        exit_code=$1
-        if [ -z "$exit_code" ] ; then
-            exit_code=0
-        fi
-
-        restore_io
-
-        if [ "$( type -t finalize )" '=' 'function' ] ; then
-            finalize "$JOB_LOG_DIR/cdash_log.txt"
-        fi
-
-        # We can clean these out later on, once we have a good sense for
-        # how the logging infrastructure is working
-        # rm -rf "$JOB_LOG_DIR"
-    fi
-
-    \exit $exit_code
-}
-
-alias exit='cleanup'
-
-begin_logging() {
-    trap "cleanup 1; \\exit \$exit_code" INT TERM QUIT
-    trap "cleanup 0; \\exit \$exit_code" EXIT
-
-    rm -rf "$JOB_LOG_DIR/cdash_log.txt"
-
-    # NOTE: Here, some redirects are set up
-    exec 3>&1 # fd 3 is now a dup of stdout
-    exec 4>&2 # fd 4 is now a dup of stderr
-
-    # stdout and stderr are joined and redirected to the log
-    exec &> "$JOB_LOG_DIR/cdash_log.txt"
-
-    set -x
-}
-
-restore_io() {
-    exec  >&-
-    exec 2>&-
-
-    exec  >&3
-    exec 2>&4
-
-    exec 3>&-
-    exec 4>&-
-}
-
-finalize() {
-    # If you define a finalize function:
-    #  - it will always be called at the very end of the script
-    #  - the log file will be passed in as the first argument, and
-    #  - the code in this function will not be logged.
-    echo "The full log file is located at $1"
-    # TODO: send this log data to cdash!
-}
-
-check_error()
-{
-    local last_exit_code=$1
-    local last_cmd=$2
-    if [[ ${last_exit_code} -ne 0 ]]; then
-        echo "${last_cmd} exited with code ${last_exit_code}"
-        echo "TERMINATING JOB"
-        exit 1
-    else
-        echo "${last_cmd} completed successfully"
-    fi
-}
-
-extract_build_id()
-{
-    LINES_TO_SEARCH=$1
-    regex="buildSummary\.php\?buildid=([[:digit:]]+)"
-    SINGLE_LINE_OUTPUT=$(echo ${LINES_TO_SEARCH} | tr -d '\n')
-
-    if [[ ${SINGLE_LINE_OUTPUT} =~ ${regex} ]]; then
-        echo "${BASH_REMATCH[1]}"
-    else
-        echo "NONE"
-    fi
-}
-
-get_relate_builds_post_data()
-{
-  cat <<EOF
-{
-  "project": "${1}",
-  "buildid": ${2},
-  "relatedid": ${3},
-  "relationship": "depends on"
-}
-EOF
-}
-
-gen_full_specs_for_job_and_deps() {
-    SPEC_YAML_PATH="${SPEC_DIR}/${SPACK_JOB_SPEC_PKG_NAME}.yaml"
-    local spec_names_to_save="${SPACK_JOB_SPEC_PKG_NAME}"
-
-    if [ "${SPACK_ENABLE_CDASH}" == "True" ] ; then
-        IFS=';' read -ra DEPS <<< "${SPACK_RELATED_BUILDS}"
-        for i in "${DEPS[@]}"; do
-            depPkgName="${i}"
-            spec_names_to_save="${spec_names_to_save} ${depPkgName}"
-            JOB_DEPS_PKG_NAMES+=("${depPkgName}")
-        done
-    fi
-
-    if [ "${SPACK_COMPILER_ACTION}" == "FIND_ANY" ]; then
-        # This corresponds to a bootstrapping phase where we need to
-        # rely on any available compiler to build the package (i.e. the
-        # compiler needed to be stripped from the spec), and thus we need
-        # to concretize the root spec again.
-        spack -d buildcache save-yaml --specs "${spec_names_to_save}" --root-spec "${SPACK_ROOT_SPEC}" --yaml-dir "${SPEC_DIR}"
-    else
-        # in this case, either we're relying on Spack to install missing compiler
-        # bootstrapped in a previous phase, or else we only had one phase (like a
-        # site which already knows what compilers are available on it's runners),
-        # so we don't want to concretize that root spec again.  The reason we need
-        # this in the first case (bootstrapped compiler), is that we can't concretize
-        # a spec at this point if we're going to ask spack to "install_missing_compilers".
-        tmp_dir=$(mktemp -d)
-        TMP_YAML_PATH="${tmp_dir}/root.yaml"
-        ROOT_SPEC_YAML=$(spack python -c "import base64 ; import zlib ; print(str(zlib.decompress(base64.b64decode('${SPACK_ROOT_SPEC}')).decode('utf-8')))")
-        echo "${ROOT_SPEC_YAML}" > "${TMP_YAML_PATH}"
-        spack -d buildcache save-yaml --specs "${spec_names_to_save}" --root-spec-yaml "${TMP_YAML_PATH}" --yaml-dir "${SPEC_DIR}"
-        rm -rf ${tmp_dir}
-    fi
-}
-
-begin_logging
-
-echo "Running job for spec: ${CI_JOB_NAME}"
-
-# This should create the directory we referred to as GNUPGHOME earlier
-spack gpg list
-
-# Importing the secret key using gpg2 directly should allow to
-# sign and verify both
-set +x
-KEY_IMPORT_RESULT=`echo ${SPACK_SIGNING_KEY} | base64 --decode | gpg2 --import`
-check_error $? "gpg2 --import"
-set -x
-
-spack gpg list --trusted
-spack gpg list --signing
-
-# To have spack install missing compilers, we need to add a custom
-# configuration scope, then we pass that to the package installation
-# command
-CUSTOM_CONFIG_SCOPE_DIR="${TEMP_DIR}/config_scope"
-mkdir -p "${CUSTOM_CONFIG_SCOPE_DIR}"
-CUSTOM_CONFIG_SCOPE_ARG=""
-
-if [ "${SPACK_COMPILER_ACTION}" == "INSTALL_MISSING" ]; then
-    echo "Make sure bootstrapped compiler will be installed"
-    custom_config_file_path="${CUSTOM_CONFIG_SCOPE_DIR}/config.yaml"
-      cat <<CONFIG_STUFF > "${custom_config_file_path}"
-config:
-  install_missing_compilers: true
-CONFIG_STUFF
-    CUSTOM_CONFIG_SCOPE_ARG="-C ${CUSTOM_CONFIG_SCOPE_DIR}"
-    # Configure the binary mirror where, if needed, this jobs compiler
-    # was installed in binary pacakge form, then tell spack to
-    # install_missing_compilers.
-elif [ "${SPACK_COMPILER_ACTION}" == "FIND_ANY" ]; then
-    echo "Just find any available compiler"
-    spack compiler find
-else
-    echo "No compiler action to be taken"
-fi
-
-# Finally, list the compilers spack knows about
-echo "Compiler Configurations:"
-spack config get compilers
-
-# Write full-deps yamls for this job spec and its dependencies
-gen_full_specs_for_job_and_deps
-
-# Make the build_cache directory if it doesn't exist
-mkdir -p "${BUILD_CACHE_DIR}"
-
-# Get buildcache name so we can write a CDash build id file in the right place.
-# If we're unable to get the buildcache name, we may have encountered a problem
-# concretizing the spec, or some other issue that will eventually cause the job
-# to fail.
-JOB_BUILD_CACHE_ENTRY_NAME=`spack -d buildcache get-buildcache-name --spec-yaml "${SPEC_YAML_PATH}"`
-if [[ $? -ne 0 ]]; then
-    echo "ERROR, unable to get buildcache entry name for job ${CI_JOB_NAME}"
-    exit 1
-fi
-
-if [ "${SPACK_ENABLE_CDASH}" == "True" ] ; then
-    # Whether we have to build the spec or download it pre-built, we expect to find
-    # the cdash build id file sitting in this location afterwards.
-    JOB_CDASH_ID_FILE="${BUILD_CACHE_DIR}/${JOB_BUILD_CACHE_ENTRY_NAME}.cdashid"
-fi
-
-# Finally, we can check the spec we have been tasked with build against
-# the built binary on the remote mirror to see if it needs to be rebuilt
-spack -d buildcache check --spec-yaml "${SPEC_YAML_PATH}" --mirror-url "${SPACK_MIRROR_URL}" --rebuild-on-error
-
-if [[ $? -ne 0 ]]; then
-    # Configure mirror
-    spack mirror add local_artifact_mirror "file://${LOCAL_MIRROR}"
-
-    if [ "${SPACK_ENABLE_CDASH}" == "True" ] ; then
-        JOB_CDASH_ID="NONE"
-
-        # Install package, using the buildcache from the local mirror to
-        # satisfy dependencies.
-        BUILD_ID_LINE=`spack -d -k -v "${CUSTOM_CONFIG_SCOPE_ARG}" install --keep-stage --cdash-upload-url "${CDASH_UPLOAD_URL}" --cdash-build "${SPACK_CDASH_BUILD_NAME}" --cdash-site "${SPACK_CDASH_SITE}" --cdash-track "${SPACK_JOB_SPEC_BUILDGROUP}" -f "${SPEC_YAML_PATH}" | grep "buildSummary\\.php"`
-        check_error $? "spack install"
-
-        # By parsing the output of the "spack install" command, we can get the
-        # buildid generated for us by CDash
-        JOB_CDASH_ID=$(extract_build_id "${BUILD_ID_LINE}")
-
-        # Write the .cdashid file to the buildcache as well
-        echo "${JOB_CDASH_ID}" >> ${JOB_CDASH_ID_FILE}
-    else
-        spack -d -k -v "${CUSTOM_CONFIG_SCOPE_ARG}" install --keep-stage -f "${SPEC_YAML_PATH}"
-    fi
-
-    # Copy some log files into an artifact location, once we have a way
-    # to provide a spec.yaml file to more spack commands (e.g. "location")
-    # stage_dir=$(spack location --stage-dir -f "${SPEC_YAML_PATH}")
-    # build_log_file=$(find -L "${stage_dir}" | grep "spack-build\\.out")
-    # config_log_file=$(find -L "${stage_dir}" | grep "config\\.log")
-    # cp "${build_log_file}" "${JOB_LOG_DIR}/"
-    # cp "${config_log_file}" "${JOB_LOG_DIR}/"
-
-    # Create buildcache entry for this package, reading the spec from the yaml
-    # file.
-    spack -d buildcache create --spec-yaml "${SPEC_YAML_PATH}" -a -f -d "${LOCAL_MIRROR}" --no-rebuild-index
-    check_error $? "spack buildcache create"
-
-    # TODO: The upload-s3 command should eventually be replaced with something
-    # like: "spack buildcache put <mirror> <spec>", when that subcommand is
-    # properly implemented.
-    if [ ! -z "${SPACK_S3_UPLOAD_MIRROR_URL}" ] ; then
-        spack -d upload-s3 spec --base-dir "${LOCAL_MIRROR}" --spec-yaml "${SPEC_YAML_PATH}" --endpoint-url "${SPACK_S3_UPLOAD_MIRROR_URL}"
-        check_error $? "spack upload-s3 spec"
-    else
-        spack -d buildcache copy --base-dir "${LOCAL_MIRROR}" --spec-yaml "${SPEC_YAML_PATH}" --destination-url "${SPACK_MIRROR_URL}"
-    fi
-else
-    echo "spec ${CI_JOB_NAME} is already up to date on remote mirror, downloading it"
-
-    # Configure remote mirror so we can download buildcache entry
-    spack mirror add remote_binary_mirror ${SPACK_MIRROR_URL}
-
-    # Now download it
-    BUILDCACHE_DL_ARGS=("--spec-yaml" "${SPEC_YAML_PATH}" "--path" "${BUILD_CACHE_DIR}/" )
-    if [ "${SPACK_ENABLE_CDASH}" == "True" ] ; then
-        BUILDCACHE_DL_ARGS+=( "--require-cdashid" )
-    fi
-    spack -d buildcache download "${BUILDCACHE_DL_ARGS[@]}"
-    check_error $? "spack buildcache download"
-fi
-
-# The next step is to relate this job to the jobs it depends on
-if [ "${SPACK_ENABLE_CDASH}" == "True" ] ; then
-    if [ -f "${JOB_CDASH_ID_FILE}" ]; then
-        JOB_CDASH_BUILD_ID=$(<${JOB_CDASH_ID_FILE})
-
-        if [ "${JOB_CDASH_BUILD_ID}" == "NONE" ]; then
-            echo "ERROR: unable to read this jobs id from ${JOB_CDASH_ID_FILE}"
-            exit 1
-        fi
-
-        # Now get CDash ids for dependencies and "relate" each dependency build
-        # with this jobs build
-        for DEP_PKG_NAME in "${JOB_DEPS_PKG_NAMES[@]}"; do
-            echo "Getting cdash id for dependency --> ${DEP_PKG_NAME} <--"
-            DEP_SPEC_YAML_PATH="${SPEC_DIR}/${DEP_PKG_NAME}.yaml"
-            DEP_JOB_BUILDCACHE_NAME=`spack -d buildcache get-buildcache-name --spec-yaml "${DEP_SPEC_YAML_PATH}"`
-
-            if [[ $? -eq 0 ]]; then
-                DEP_JOB_ID_FILE="${BUILD_CACHE_DIR}/${DEP_JOB_BUILDCACHE_NAME}.cdashid"
-                echo "DEP_JOB_ID_FILE path = ${DEP_JOB_ID_FILE}"
-
-                if [ -f "${DEP_JOB_ID_FILE}" ]; then
-                    DEP_JOB_CDASH_BUILD_ID=$(<${DEP_JOB_ID_FILE})
-                    echo "File ${DEP_JOB_ID_FILE} contained value ${DEP_JOB_CDASH_BUILD_ID}"
-                    echo "Relating builds -> ${SPACK_CDASH_BUILD_NAME} (buildid=${JOB_CDASH_BUILD_ID}) depends on ${DEP_PKG_NAME} (buildid=${DEP_JOB_CDASH_BUILD_ID})"
-                    relateBuildsPostBody="$(get_relate_builds_post_data "${SPACK_CDASH_PROJECT}" ${JOB_CDASH_BUILD_ID} ${DEP_JOB_CDASH_BUILD_ID})"
-                    relateBuildsResult=`curl "${DEP_JOB_RELATEBUILDS_URL}" -H "Content-Type: application/json" -H "Accept: application/json" -d "${relateBuildsPostBody}"`
-                    echo "Result of curl request: ${relateBuildsResult}"
-                else
-                    echo "ERROR: Did not find expected .cdashid file for dependency: ${DEP_JOB_ID_FILE}"
-                    exit 1
-                fi
-            else
-                echo "ERROR: Unable to get buildcache entry name for ${DEP_SPEC_NAME}"
-                exit 1
-            fi
-        done
-    else
-        echo "ERROR: Did not find expected .cdashid file ${JOB_CDASH_ID_FILE}"
-        exit 1
-    fi
-fi
-
-# Show the size of the buildcache and a list of what's in it, directly
-# in the gitlab log output
-(
-    restore_io
-    du -sh ${BUILD_CACHE_DIR}
-    find ${BUILD_CACHE_DIR} -maxdepth 3 -type d -ls
-)
-
-echo "End of rebuild package script"

bin/sbang

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -103,10 +103,10 @@ interpreter_f="${interpreter_v[0]}"
 
 # Invoke any interpreter found, or raise an error if none was found.
 if [[ -n "$interpreter_f" ]]; then
-    if [[ "${interpreter_f##*/}" = "perl" ]]; then
-        exec $interpreter_v -x "$@"
+    if [[ "${interpreter_f##*/}" = "perl"* ]]; then
+        exec $interpreter -x "$@"
     else
-        exec $interpreter_v "$@"
+        exec $interpreter "$@"
     fi
 else
     echo "error: sbang found no interpreter in $script"

bin/spack

@@ -1,7 +1,7 @@
 #!/bin/sh
 # -*- python -*-
 #
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)

bin/spack-python

@@ -1,6 +1,6 @@
 #!/bin/sh
 #
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -22,4 +22,4 @@
 #
 # This is compatible across platforms.
 #
-/usr/bin/env spack python "$@"
+exec /usr/bin/env spack python "$@"

etc/spack/defaults/config.yaml

@@ -16,7 +16,7 @@
 config:
   # This is the path to the root of the Spack install tree.
   # You can use $spack here to refer to the root of the spack instance.
-  install_tree: $spack/opt/spack
+  install_tree: ~/.spack/opt/spack
 
 
   # Locations where templates should be found
@@ -30,8 +30,8 @@ config:
 
   # Locations where different types of modules should be installed.
   module_roots:
-    tcl:    $spack/share/spack/modules
-    lmod:   $spack/share/spack/lmod
+    tcl:    ~/.spack/share/spack/modules
+    lmod:   ~/.spack/share/spack/lmod
 
 
   # Temporary locations Spack can try to use for builds.
@@ -67,7 +67,7 @@ config:
 
   # Cache directory for already downloaded source tarballs and archived
   # repositories. This can be purged with `spack clean --downloads`.
-  source_cache: $spack/var/spack/cache
+  source_cache: ~/.spack/var/spack/cache
 
 
   # Cache directory for miscellaneous files, like the package index.
@@ -75,11 +75,25 @@ config:
   misc_cache: ~/.spack/cache
 
 
+  # Timeout in seconds used for downloading sources etc. This only applies
+  # to the connection phase and can be increased for slow connections or
+  # servers. 0 means no timeout.
+  connect_timeout: 10
+
+
   # If this is false, tools like curl that use SSL will not verify
   # certifiates. (e.g., curl will use use the -k option)
   verify_ssl: true
 
 
+  # Suppress gpg warnings from binary package verification
+  # Only suppresses warnings, gpg failure will still fail the install
+  # Potential rationale to set True: users have already explicitly trusted the
+  # gpg key they are using, and may not want to see repeated warnings that it
+  # is self-signed or something of the sort.
+  suppress_gpg_warnings: false
+
+
   # If set to true, Spack will attempt to build any compiler on the spec
   # that is not already available. If set to False, Spack will only use
   # compilers already configured in compilers.yaml
@@ -129,7 +143,7 @@ config:
   # when Spack needs to manage its own package metadata and all operations are
   # expected to complete within the default time limit. The timeout should
   # therefore generally be left untouched.
-  db_lock_timeout: 120
+  db_lock_timeout: 3
 
 
   # How long to wait when attempting to modify a package (e.g. to install it).
@@ -143,3 +157,7 @@ config:
   # Has no effect on macOS. DO NOT MIX these within the same install tree.
   # See the Spack documentation for details.
   shared_linking: 'rpath'
+
+  # Set to 'false' to allow installation on filesystems that doesn't allow setgid bit
+  # manipulation by unprivileged user (e.g. AFS)
+  allow_sgid: true

etc/spack/defaults/darwin/modules.yaml

@@ -16,8 +16,6 @@
 modules:
   prefix_inspections:
     lib:
-      - DYLD_LIBRARY_PATH
       - DYLD_FALLBACK_LIBRARY_PATH
     lib64:
-      - DYLD_LIBRARY_PATH
       - DYLD_FALLBACK_LIBRARY_PATH

etc/spack/defaults/darwin/packages.yaml

@@ -15,7 +15,11 @@
 # -------------------------------------------------------------------------
 packages:
   all:
-    compiler: [clang, gcc, intel]
+    compiler:
+    - apple-clang
+    - clang
+    - gcc
+    - intel
     providers:
       elf: [libelf]
       unwind: [apple-libunwind]

etc/spack/defaults/modules.yaml

@@ -30,7 +30,11 @@ modules:
     lib64:
       - LIBRARY_PATH
     include:
-      - CPATH
+      - C_INCLUDE_PATH
+      - CPLUS_INCLUDE_PATH
+      # The INCLUDE env variable specifies paths to look for
+      # .mod file for Intel Fortran compilers
+      - INCLUDE
     lib/pkgconfig:
       - PKG_CONFIG_PATH
     lib64/pkgconfig:

etc/spack/defaults/packages.yaml

@@ -27,6 +27,7 @@ packages:
       glx: [mesa+glx, opengl]
       glu: [mesa-glu, openglu]
       golang: [gcc]
+      iconv: [libiconv]
       ipp: [intel-ipp]
       java: [openjdk, jdk, ibm-java]
       jpeg: [libjpeg-turbo, libjpeg]
@@ -39,10 +40,13 @@ packages:
       opencl: [pocl]
       pil: [py-pillow]
       pkgconfig: [pkgconf, pkg-config]
+      rpc: [libtirpc]
       scalapack: [netlib-scalapack]
+      sycl: [hipsycl]
       szip: [libszip, libaec]
       tbb: [intel-tbb]
       unwind: [libunwind]
+      sycl: [hipsycl]
     permissions:
       read: world
       write: user

etc/spack/defaults/upstreams.yaml

@@ -0,0 +1,7 @@
+upstreams:
+  global:
+    install_tree: $spack/opt/spack
+    modules:
+      tcl:    $spack/share/spack/modules
+      lmod:   $spack/share/spack/lmod
+      dotkit: $spack/share/spack/dotkit

lib/spack/docs/basic_usage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -25,6 +25,14 @@ It is recommended that the following be put in your ``.bashrc`` file:
 
     alias less='less -R'
 
+If you do not see colorized output when using ``less -R`` it is because color
+is being disabled in the piped output. In this case, tell spack to force
+colorized output.
+
+.. code-block:: console
+
+    $ spack --color always | less -R 
+
 --------------------------
 Listing available packages
 --------------------------
@@ -45,7 +53,7 @@ can install:
 .. command-output:: spack list
    :ellipsis: 10
 
-There are thosands of them, so we've truncated the output above, but you
+There are thousands of them, so we've truncated the output above, but you
 can find a :ref:`full list here <package-list>`.
 Packages are listed by name in alphabetical order.
 A pattern to match with no wildcards, ``*`` or ``?``,
@@ -232,6 +240,50 @@ remove dependent packages *before* removing their dependencies or use the
 
 .. _nondownloadable:
 
+^^^^^^^^^^^^^^^^^^
+Garbage collection
+^^^^^^^^^^^^^^^^^^
+
+When Spack builds software from sources, if often installs tools that are needed
+just to build or test other software. These are not necessary at runtime.
+To support cases where removing these tools can be a benefit Spack provides
+the ``spack gc`` ("garbage collector") command, which will uninstall all unneeded packages:
+
+.. code-block:: console
+
+   $ spack find
+   ==> 24 installed packages
+   -- linux-ubuntu18.04-broadwell / gcc@9.0.1 ----------------------
+   autoconf@2.69    findutils@4.6.0  libiconv@1.16        libszip@2.1.1  m4@1.4.18    openjpeg@2.3.1  pkgconf@1.6.3  util-macros@1.19.1
+   automake@1.16.1  gdbm@1.18.1      libpciaccess@0.13.5  libtool@2.4.6  mpich@3.3.2  openssl@1.1.1d  readline@8.0   xz@5.2.4
+   cmake@3.16.1     hdf5@1.10.5      libsigsegv@2.12      libxml2@2.9.9  ncurses@6.1  perl@5.30.0     texinfo@6.5    zlib@1.2.11
+
+   $ spack gc
+   ==> The following packages will be uninstalled:
+
+       -- linux-ubuntu18.04-broadwell / gcc@9.0.1 ----------------------
+       vn47edz autoconf@2.69    6m3f2qn findutils@4.6.0  ubl6bgk libtool@2.4.6  pksawhz openssl@1.1.1d  urdw22a readline@8.0
+       ki6nfw5 automake@1.16.1  fklde6b gdbm@1.18.1      b6pswuo m4@1.4.18      k3s2csy perl@5.30.0     lp5ya3t texinfo@6.5
+       ylvgsov cmake@3.16.1     5omotir libsigsegv@2.12  leuzbbh ncurses@6.1    5vmfbrq pkgconf@1.6.3   5bmv4tg util-macros@1.19.1
+
+   ==> Do you want to proceed? [y/N] y
+
+   [ ... ]
+
+   $ spack find
+   ==> 9 installed packages
+   -- linux-ubuntu18.04-broadwell / gcc@9.0.1 ----------------------
+   hdf5@1.10.5  libiconv@1.16  libpciaccess@0.13.5  libszip@2.1.1  libxml2@2.9.9  mpich@3.3.2  openjpeg@2.3.1  xz@5.2.4  zlib@1.2.11
+
+In the example above Spack went through all the packages in the package database
+and removed everything that is not either:
+
+1. A package installed upon explicit request of the user
+2. A ``link`` or ``run`` dependency, even transitive, of one of the packages at point 1.
+
+You can check :ref:`cmd-spack-find-metadata` to see how to query for explicitly installed packages
+or :ref:`dependency-types` for a more thorough treatment of dependency types.
+
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 Non-Downloadable Tarballs
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -414,6 +466,8 @@ Packages are divided into groups according to their architecture and
 compiler.  Within each group, Spack tries to keep the view simple, and
 only shows the version of installed packages.
 
+.. _cmd-spack-find-metadata:
+
 """"""""""""""""""""""""""""""""
 Viewing more metadata
 """"""""""""""""""""""""""""""""
@@ -573,8 +627,8 @@ output metadata on specs and all dependencies as json:
        "target": "x86_64"
       },
       "compiler": {
-       "name": "clang",
-       "version": "10.0.0-apple"
+       "name": "apple-clang",
+       "version": "10.0.0"
       },
       "namespace": "builtin",
       "parameters": {
@@ -808,7 +862,7 @@ Variants are named options associated with a particular package. They are
 optional, as each package must provide default values for each variant it
 makes available. Variants can be specified using
 a flexible parameter syntax ``name=<value>``. For example,
-``spack install libelf debug=True`` will install libelf build with debug
+``spack install libelf debug=True`` will install libelf built with debug
 flags. The names of particular variants available for a package depend on
 what was provided by the package author. ``spack info <package>`` will
 provide information on what build variants are available.
@@ -871,7 +925,7 @@ contains any spaces. Any of ``cppflags=-O3``, ``cppflags="-O3"``,
 ``cppflags='-O3'``, and ``cppflags="-O3 -fPIC"`` are acceptable, but
 ``cppflags=-O3 -fPIC`` is not. Additionally, if the value of the
 compiler flags is not the last thing on the line, it must be followed
-by a space. The commmand ``spack install libelf cppflags="-O3"%intel``
+by a space. The command ``spack install libelf cppflags="-O3"%intel``
 will be interpreted as an attempt to set ``cppflags="-O3%intel"``.
 
 The six compiler flags are injected in the order of implicit make commands
@@ -883,11 +937,13 @@ in GNU Autotools. If all flags are set, the order is
 Compiler environment variables and additional RPATHs
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-In the exceptional case a compiler requires setting special environment
-variables, like an explicit library load path. These can bet set in an
-extra section in the compiler configuration (the supported environment
-modification commands are: ``set``, ``unset``, ``append-path``, and
-``prepend-path``). The user can also specify additional ``RPATHs`` that the
+Sometimes compilers require setting special environment variables to
+operate correctly. Spack handles these cases by allowing custom environment
+modifications in the ``environment`` attribute of the compiler configuration
+section. See also the :ref:`configuration_environment_variables` section
+of the configuration files docs for more information.
+
+It is also possible to specify additional ``RPATHs`` that the
 compiler will add to all executables generated by that compiler.  This is
 useful for forcing certain compilers to RPATH their own runtime libraries, so
 that executables will run without the need to set ``LD_LIBRARY_PATH``.
@@ -904,28 +960,19 @@ that executables will run without the need to set ``LD_LIBRARY_PATH``.
           fc: /opt/gcc/bin/gfortran
         environment:
           unset:
-            BAD_VARIABLE: # The colon is required but the value must be empty
+            - BAD_VARIABLE
           set:
             GOOD_VARIABLE_NUM: 1
             GOOD_VARIABLE_STR: good
-          prepend-path:
+          prepend_path:
             PATH: /path/to/binutils
-          append-path:
+          append_path:
             LD_LIBRARY_PATH: /opt/gcc/lib
         extra_rpaths:
         - /path/to/some/compiler/runtime/directory
         - /path/to/some/other/compiler/runtime/directory
 
 
-.. note::
-
-   The section `environment` is interpreted as an ordered dictionary, which
-   means two things. First, environment modification are applied in the order
-   they are specified in the configuration file. Second, you cannot express
-   environment modifications that require mixing different commands, i.e. you
-   cannot `set` one variable, than `prepend-path` to another one, and than
-   again `set` a third one.
-
 ^^^^^^^^^^^^^^^^^^^^^^^
 Architecture specifiers
 ^^^^^^^^^^^^^^^^^^^^^^^
@@ -1028,13 +1075,13 @@ of failing:
 In the snippet above, for instance, the microarchitecture was demoted to ``haswell`` when
 compiling with ``gcc@4.8`` since support to optimize for ``broadwell`` starts from ``gcc@4.9:``.
 
-Finally if Spack has no information to match compiler and target, it will
+Finally, if Spack has no information to match compiler and target, it will
 proceed with the installation but avoid injecting any microarchitecture
 specific flags.
 
 .. warning::
 
-   Currently Spack doesn't print any warning to the user if it has no information
+   Currently, Spack doesn't print any warning to the user if it has no information
    on which optimization flags should be used for a given compiler. This behavior
    might change in the future.
 
@@ -1044,7 +1091,7 @@ specific flags.
 Virtual dependencies
 --------------------
 
-The dependence graph for ``mpileaks`` we saw above wasn't *quite*
+The dependency graph for ``mpileaks`` we saw above wasn't *quite*
 accurate.  ``mpileaks`` uses MPI, which is an interface that has many
 different implementations.  Above, we showed ``mpileaks`` and
 ``callpath`` depending on ``mpich``, which is one *particular*
@@ -1187,6 +1234,8 @@ add a version specifier to the spec:
 Notice that the package versions that provide insufficient MPI
 versions are now filtered out.
 
+.. _extensions:
+
 ---------------------------
 Extensions & Python support
 ---------------------------
@@ -1194,8 +1243,7 @@ Extensions & Python support
 Spack's installation model assumes that each package will live in its
 own install prefix.  However, certain packages are typically installed
 *within* the directory hierarchy of other packages.  For example,
-modules in interpreted languages like `Python
-<https://www.python.org>`_ are typically installed in the
+`Python <https://www.python.org>`_ packages are typically installed in the
 ``$prefix/lib/python-2.7/site-packages`` directory.
 
 Spack has support for this type of installation as well.  In Spack,
@@ -1271,10 +1319,9 @@ directly when you run ``python``:
 Using Extensions
 ^^^^^^^^^^^^^^^^
 
-There are three ways to get ``numpy`` working in Python.  The first is
-to use :ref:`shell-support`.  You can simply ``load`` the
-module for the extension, and it will be added to the ``PYTHONPATH``
-in your current shell:
+There are four ways to get ``numpy`` working in Python.  The first is
+to use :ref:`shell-support`.  You can simply ``load`` the extension,
+and it will be added to the ``PYTHONPATH`` in your current shell:
 
 .. code-block:: console
 
@@ -1284,11 +1331,29 @@ in your current shell:
 Now ``import numpy`` will succeed for as long as you keep your current
 session open.
 
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Loading Extensions via Modules
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Instead of using Spack's environment modification capabilities through
+the ``spack load`` command, you can load numpy through your
+environment modules (using ``environment-modules`` or ``lmod``). This
+will also add the extension to the ``PYTHONPATH`` in your current
+shell.
+
+.. code-block:: console
+
+   $ module load <name of numpy module>
+
+If you do not know the name of the specific numpy module you wish to
+load, you can use the ``spack module tcl|lmod loads`` command to get
+the name of the module from the Spack spec.
+
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Activating Extensions in a View
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The second way to use extensions is to create a view, which merges the
+Another way to use extensions is to create a view, which merges the
 python installation along with the extensions into a single prefix.
 See :ref:`filesystem-views` for a more in-depth description of views and
 :ref:`cmd-spack-view` for usage of the ``spack view`` command.
@@ -1354,12 +1419,12 @@ packages listed as activated:
    py-nose@1.3.4  py-numpy@1.9.1  py-setuptools@11.3.1
 
 Now, when a user runs python, ``numpy`` will be available for import
-*without* the user having to explicitly loaded.  ``python@2.7.8`` now
+*without* the user having to explicitly load it.  ``python@2.7.8`` now
 acts like a system Python installation with ``numpy`` installed inside
 of it.
 
 Spack accomplishes this by symbolically linking the *entire* prefix of
-the ``py-numpy`` into the prefix of the ``python`` package.  To the
+the ``py-numpy`` package into the prefix of the ``python`` package.  To the
 python interpreter, it looks like ``numpy`` is installed in the
 ``site-packages`` directory.
 

lib/spack/docs/binary_caches.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -57,6 +57,12 @@ Build caches are installed via:
 
    $ spack buildcache install
 
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+List of popular build caches
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* `Extreme-scale Scientific Software Stack (E4S) <https://e4s-project.github.io/>`_: `build cache <https://oaciss.uoregon.edu/e4s/inventory.html>`_
+
 
 ----------
 Relocation

lib/spack/docs/build_settings.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -58,9 +58,9 @@ directory. Here's an example of an external configuration:
    packages:
      openmpi:
        paths:
-         openmpi@1.4.3%gcc@4.4.7 arch=linux-x86_64-debian7: /opt/openmpi-1.4.3
-         openmpi@1.4.3%gcc@4.4.7 arch=linux-x86_64-debian7+debug: /opt/openmpi-1.4.3-debug
-         openmpi@1.6.5%intel@10.1 arch=linux-x86_64-debian7: /opt/openmpi-1.6.5-intel
+         openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64: /opt/openmpi-1.4.3
+         openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug: /opt/openmpi-1.4.3-debug
+         openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64: /opt/openmpi-1.6.5-intel
 
 This example lists three installations of OpenMPI, one built with GCC,
 one built with GCC and debug information, and another built with Intel.
@@ -107,9 +107,9 @@ be:
    packages:
      openmpi:
        paths:
-         openmpi@1.4.3%gcc@4.4.7 arch=linux-x86_64-debian7: /opt/openmpi-1.4.3
-         openmpi@1.4.3%gcc@4.4.7 arch=linux-x86_64-debian7+debug: /opt/openmpi-1.4.3-debug
-         openmpi@1.6.5%intel@10.1 arch=linux-x86_64-debian7: /opt/openmpi-1.6.5-intel
+         openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64: /opt/openmpi-1.4.3
+         openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug: /opt/openmpi-1.4.3-debug
+         openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64: /opt/openmpi-1.6.5-intel
        buildable: False
 
 The addition of the ``buildable`` flag tells Spack that it should never build
@@ -124,6 +124,78 @@ The ``buildable`` does not need to be paired with external packages.
 It could also be used alone to forbid packages that may be
 buggy or otherwise undesirable.
 
+Virtual packages in Spack can also be specified as not buildable, and
+external implementations can be provided. In the example above,
+OpenMPI is configured as not buildable, but Spack will often prefer
+other MPI implementations over the externally available OpenMPI. Spack
+can be configured with every MPI provider not buildable individually,
+but more conveniently:
+
+.. code-block:: yaml
+
+   packages:
+     mpi:
+       buildable: False
+     openmpi:
+       paths:
+         openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64: /opt/openmpi-1.4.3
+         openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug: /opt/openmpi-1.4.3-debug
+         openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64: /opt/openmpi-1.6.5-intel
+
+Implementations can also be listed immediately under the virtual they provide:
+
+.. code-block:: yaml
+
+   packages:
+     mpi:
+       buildable: False
+         openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64: /opt/openmpi-1.4.3
+         openmpi@1.4.3%gcc@4.4.7 arch=linux-debian7-x86_64+debug: /opt/openmpi-1.4.3-debug
+         openmpi@1.6.5%intel@10.1 arch=linux-debian7-x86_64: /opt/openmpi-1.6.5-intel
+         mpich@3.3 %clang@9.0.0 arch=linux-debian7-x86_64: /opt/mpich-3.3-intel
+
+Spack can then use any of the listed external implementations of MPI
+to satisfy a dependency, and will choose depending on the compiler and
+architecture.
+
+.. _cmd-spack-external-find:
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Automatically Find External Packages
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can run the :ref:`spack external find <spack-external-find>` command
+to search for system-provided packages and add them to ``packages.yaml``.
+After running this command your ``packages.yaml`` may include new entries:
+
+.. code-block:: yaml
+
+   packages:
+     cmake:
+       paths:
+         cmake@3.17.2: /usr
+
+Generally this is useful for detecting a small set of commonly-used packages;
+for now this is generally limited to finding build-only dependencies.
+Specific limitations include:
+
+* Packages are not discoverable by default: For a package to be
+  discoverable with ``spack external find``, it needs to add special
+  logic. See :ref:`here <make-package-findable>` for more details.
+* The current implementation only collects and examines executable files,
+  so it is typically only useful for build/run dependencies (in some cases
+  if a library package also provides an executable, it may be possible to
+  extract a meaningful Spec by running the executable - for example the
+  compiler wrappers in MPI implementations).
+* The logic does not search through module files, it can only detect
+  packages with executables defined in ``PATH``; you can help Spack locate
+  externals which use module files by loading any associated modules for
+  packages that you want Spack to know about before running
+  ``spack external find``.
+* Spack does not overwrite existing entries in the package configuration:
+  If there is an external defined for a spec at any configuration scope,
+  then Spack will not add a new external entry (``spack config blame packages``
+  can help locate all external entries).
 
 .. _concretization-preferences:
 

lib/spack/docs/build_systems.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -56,6 +56,7 @@ on these ideas for each distinct build system that Spack supports:
    :maxdepth: 1
    :caption: Other
 
+   build_systems/bundlepackage
    build_systems/cudapackage
    build_systems/intelpackage
    build_systems/custompackage

lib/spack/docs/build_systems/autotoolspackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -233,7 +233,124 @@ You may have noticed that most of the Autotools flags are of the form
 ``--without-baz``. Since these flags are so common, Spack provides a
 couple of helper functions to make your life easier.
 
-TODO: document ``with_or_without`` and ``enable_or_disable``.
+"""""""""""""""""
+enable_or_disable
+"""""""""""""""""
+
+Autotools flags for simple boolean variants can be automatically
+generated by calling the ``enable_or_disable`` method. This is
+typically used to enable or disable some feature within the package.
+
+.. code-block:: python
+
+   variant(
+       'memchecker',
+       default=False,
+       description='Memchecker support for debugging [degrades performance]'
+   )
+   config_args.extend(self.enable_or_disable('memchecker'))
+
+In this example, specifying the variant ``+memchecker`` will generate
+the following configuration options:
+
+.. code-block:: console
+
+   --enable-memchecker
+
+"""""""""""""""
+with_or_without
+"""""""""""""""
+
+Autotools flags for more complex variants, including boolean variants
+and multi-valued variants, can be automatically generated by calling
+the ``with_or_without`` method.
+
+.. code-block:: python
+
+   variant(
+       'schedulers',
+       values=disjoint_sets(
+           ('auto',), ('alps', 'lsf', 'tm', 'slurm', 'sge', 'loadleveler')
+       ).with_non_feature_values('auto', 'none'),
+       description="List of schedulers for which support is enabled; "
+       "'auto' lets openmpi determine",
+   )
+   if 'schedulers=auto' not in spec:
+       config_args.extend(self.with_or_without('schedulers'))
+
+In this example, specifying the variant ``schedulers=slurm,sge`` will
+generate the following configuration options:
+
+.. code-block:: console
+
+   --with-slurm --with-sge
+
+``enable_or_disable`` is actually functionally equivalent with
+``with_or_without``, and accepts the same arguments and variant types;
+but idiomatic autotools packages often follow these naming
+conventions.
+
+""""""""""""""""
+activation_value
+""""""""""""""""
+
+Autotools parameters that require an option can still be automatically
+generated, using the ``activation_value`` argument to
+``with_or_without`` (or, rarely, ``enable_or_disable``).
+
+.. code-block:: python
+
+   variant(
+      'fabrics',
+       values=disjoint_sets(
+           ('auto',), ('psm', 'psm2', 'verbs', 'mxm', 'ucx', 'libfabric')
+       ).with_non_feature_values('auto', 'none'),
+       description="List of fabrics that are enabled; "
+       "'auto' lets openmpi determine",
+   )
+   if 'fabrics=auto' not in spec:
+       config_args.extend(self.with_or_without('fabrics',
+           activation_value='prefix'))
+
+``activation_value`` accepts a callable that generates the configure
+parameter value given the variant value; but the special value
+``prefix`` tells Spack to automatically use the dependenency's
+installation prefix, which is the most common use for such
+parameters. In this example, specifying the variant
+``fabrics=libfabric`` will generate the following configuration
+options:
+
+.. code-block:: console
+
+   --with-libfabric=</path/to/libfabric>
+
+""""""""""""""""""""
+activation overrides
+""""""""""""""""""""
+
+Finally, the behavior of either ``with_or_without`` or
+``enable_or_disable`` can be overridden for specific variant
+values. This is most useful for multi-values variants where some of
+the variant values require atypical behavior.
+
+.. code-block:: python
+
+   def with_or_without_verbs(self, activated):
+       # Up through version 1.6, this option was named --with-openib.
+       # In version 1.7, it was renamed to be --with-verbs.
+       opt = 'verbs' if self.spec.satisfies('@1.7:') else 'openib'
+       if not activated:
+           return '--without-{0}'.format(opt)
+       return '--with-{0}={1}'.format(opt, self.spec['rdma-core'].prefix)
+
+Defining ``with_or_without_verbs`` overrides the behavior of a
+``fabrics=verbs`` variant, changing the configure-time option to
+``--with-openib`` for older versions of the package and specifying an
+alternative dependency name:
+
+.. code-block::
+
+   --with-openib=</path/to/rdma-core>
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Configure script in a sub-directory

lib/spack/docs/build_systems/bundlepackage.rst

@@ -0,0 +1,52 @@
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+   SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+.. _bundlepackage:
+
+-------------
+BundlePackage
+-------------
+
+``BundlePackage`` represents a set of packages that are expected to work well
+together, such as a collection of commonly used software libraries.  The
+associated software is specified as bundle dependencies.
+
+
+^^^^^^^^
+Creation
+^^^^^^^^
+
+Be sure to specify the ``bundle`` template if you are using ``spack create``
+to generate a package from the template.  For example, use the following
+command to create a bundle package whose class name will be ``Mybundle``:
+
+.. code-block:: console
+
+    $ spack create --template bundle --name mybundle
+
+
+
+^^^^^^
+Phases
+^^^^^^
+
+The ``BundlePackage`` base class does not provide any phases by default
+since the bundle does not represent a build system.
+
+
+^^^
+URL
+^^^
+
+The ``url`` property does not have meaning since there is no package-specific
+code to fetch.
+
+
+^^^^^^^
+Version
+^^^^^^^
+
+At least one ``version`` must be specified in order for the package to
+build.

lib/spack/docs/build_systems/cmakepackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -128,17 +128,20 @@ Adding flags to cmake
 ^^^^^^^^^^^^^^^^^^^^^
 
 To add additional flags to the ``cmake`` call, simply override the
-``cmake_args`` function:
+``cmake_args`` function. The following example defines values for the flags
+``WHATEVER``, ``ENABLE_BROKEN_FEATURE``, ``DETECT_HDF5``, and ``THREADS`` with
+and without the :py:meth:`~.CMakePackage.define` and
+:py:meth:`~.CMakePackage.define_from_variant` helper functions:
 
 .. code-block:: python
 
    def cmake_args(self):
-       args = []
-
-       if '+hdf5' in self.spec:
-           args.append('-DDETECT_HDF5=ON')
-       else:
-           args.append('-DDETECT_HDF5=OFF')
+       args = [
+           '-DWHATEVER:STRING=somevalue',
+           self.define('ENABLE_BROKEN_FEATURE', False),
+           self.define_from_variant('DETECT_HDF5', 'hdf5'),
+           self.define_from_variant('THREADS'), # True if +threads
+       ]
 
        return args
 

lib/spack/docs/build_systems/cudapackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -37,7 +37,7 @@ In order to use it, just add another base class to your package, for example:
             if '+cuda' in spec:
                 options.append('-DWITH_CUDA=ON')
                 cuda_arch = spec.variants['cuda_arch'].value
-                if cuda_arch is not None:
+                if cuda_arch != 'none':
                     options.append('-DCUDA_FLAGS=-arch=sm_{0}'.format(cuda_arch[0]))
             else:
                 options.append('-DWITH_CUDA=OFF')

lib/spack/docs/build_systems/custompackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/build_systems/intelpackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -553,7 +553,7 @@ follow `the next section <intel-install-libs_>`_ instead.
                f77:      stub
                fc:       stub
 
-      Replace ``18.0.3`` with the version that you determined in the preceeding
+      Replace ``18.0.3`` with the version that you determined in the preceding
       step. The contents under ``paths:`` do not matter yet.
 
    You are right to ask: "Why on earth is that necessary?" [fn8]_.
@@ -696,7 +696,7 @@ follow `the next section <intel-install-libs_>`_ instead.
              - /home/$user/spack-stage
 
         Do not duplicate the ``config:`` line if it already is present.
-        Adapt the location, which here is the same as in the preceeding example.
+        Adapt the location, which here is the same as in the preceding example.
 
    3. Retry installing the large package.
 
@@ -965,7 +965,7 @@ a *virtual* ``mkl`` package is declared in Spack.
 
   Likewise, in a
   :ref:`MakefilePackage <makefilepackage>`
-  or similiar package that does not use AutoTools you may need to provide include
+  or similar package that does not use AutoTools you may need to provide include
   and link options for use on command lines or in environment variables.
   For example, to generate an option string of the form ``-I<dir>``, use:
 
@@ -1055,6 +1055,6 @@ Footnotes
    2. Set the hash length in ``install-path-scheme``, also in ``config.yaml``
       (:ref:`q.v. <config-yaml>`).
    3. You will want to set the *same* hash length for
-      :ref:`tcl module files <modules-naming-scheme>`
-      if you have Spack produce them for you, under ``naming_scheme`` in
-      ``modules.yaml``.  Other module dialects cannot be altered in this manner.
+      :ref:`module files <modules-projections>`
+      if you have Spack produce them for you, under ``projections`` in
+      ``modules.yaml``.

lib/spack/docs/build_systems/makefilepackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/build_systems/mesonpackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/build_systems/octavepackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/build_systems/perlpackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/build_systems/pythonpackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/build_systems/qmakepackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/build_systems/rpackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/build_systems/rubypackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/build_systems/sconspackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/build_systems/sippackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -51,10 +51,8 @@ Build system dependencies
 ``SIPPackage`` requires several dependencies. Python is needed to run
 the ``configure.py`` build script, and to run the resulting Python
 libraries. Qt is needed to provide the ``qmake`` command. SIP is also
-needed to build the package. SIP is an unusual dependency in that it
-must be installed in the same installation directory as the package,
-so instead of a ``depends_on``, we use a ``resource``. All of these
-dependencies are automatically added via the base class
+needed to build the package. All of these dependencies are automatically
+added via the base class
 
 .. code-block:: python
 
@@ -62,11 +60,7 @@ dependencies are automatically added via the base class
 
    depends_on('qt', type='build')
 
-   resource(name='sip',
-            url='https://www.riverbankcomputing.com/static/Downloads/sip/4.19.18/sip-4.19.18.tar.gz',
-            sha256='c0bd863800ed9b15dcad477c4017cdb73fa805c25908b0240564add74d697e1e',
-            destination='.')
-
+   depends_on('py-sip', type='build')
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Passing arguments to ``configure.py``

lib/spack/docs/build_systems/wafpackage.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -47,8 +47,9 @@ Each phase provides a ``<phase>`` function that runs:
 
 where ``<jobs>`` is the number of parallel jobs to build with. Each phase
 also has a ``<phase_args>`` function that can pass arguments to this call.
-All of these functions are empty except for the ``configure_args``
-function, which passes ``--prefix=/path/to/installation/prefix``.
+All of these functions are empty. The ``configure`` phase
+automatically adds  ``--prefix=/path/to/installation/prefix``, so you
+don't need to add that in the ``configure_args``.
 
 ^^^^^^^
 Testing

lib/spack/docs/chain.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/conf.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -176,7 +176,25 @@ def setup(sphinx):
 #show_authors = False
 
 # The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
+# We use our own extension of the default style with a few modifications
+from pygments.style import Style
+from pygments.styles.default import DefaultStyle
+from pygments.token import Generic, Comment, Text
+
+class SpackStyle(DefaultStyle):
+    styles = DefaultStyle.styles.copy()
+    background_color       = "#f4f4f8"
+    styles[Generic.Output] = "#355"
+    styles[Generic.Prompt] = "bold #346ec9"
+
+import pkg_resources
+dist = pkg_resources.Distribution(__file__)
+sys.path.append('.')  # make 'conf' module findable
+ep = pkg_resources.EntryPoint.parse('spack = conf:SpackStyle', dist=dist)
+dist._ep_map = {'pygments.styles': {'plugin1': ep}}
+pkg_resources.working_set.add(dist)
+
+pygments_style = 'spack'
 
 # A list of ignored prefixes for module index sorting.
 #modindex_common_prefix = []

lib/spack/docs/config_yaml.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -30,11 +30,21 @@ Default is ``$spack/opt/spack``.
 ``install_hash_length`` and ``install_path_scheme``
 ---------------------------------------------------
 
-The default Spack installation path can be very long and can create
-problems for scripts with hardcoded shebangs. There are two parameters
-to help with that. Firstly, the ``install_hash_length`` parameter can
-set the length of the hash in the installation path from 1 to 32. The
-default path uses the full 32 characters.
+The default Spack installation path can be very long and can create problems
+for scripts with hardcoded shebangs. Additionally, when using the Intel
+compiler, and if there is also a long list of dependencies, the compiler may
+segfault. If you see the following:
+
+     .. code-block:: console
+
+       : internal error: ** The compiler has encountered an unexpected problem.
+       ** Segmentation violation signal raised. **
+       Access violation or stack overflow. Please contact Intel Support for assistance.
+
+it may be because variables containing dependency specs may be too long. There
+are two parameters to help with long path names. Firstly, the
+``install_hash_length`` parameter can set the length of the hash in the
+installation path from 1 to 32. The default path uses the full 32 characters.
 
 Secondly, it is also possible to modify the entire installation
 scheme. By default Spack uses

lib/spack/docs/configuration.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -427,6 +427,33 @@ home directory, and ``~user`` will expand to a specified user's home
 directory. The ``~`` must appear at the beginning of the path, or Spack
 will not expand it.
 
+.. _configuration_environment_variables:
+
+-------------------------
+Environment Modifications
+-------------------------
+
+Spack allows to prescribe custom environment modifications in a few places
+within its configuration files. Every time these modifications are allowed
+they are specified as a dictionary, like in the following example:
+
+.. code-block:: yaml
+
+   environment:
+     set:
+       LICENSE_FILE: '/path/to/license'
+     unset:
+     - CPATH
+     - LIBRARY_PATH
+     append_path:
+       PATH: '/new/bin/dir'
+
+The possible actions that are permitted are ``set``, ``unset``, ``append_path``,
+``prepend_path`` and finally ``remove_path``. They all require a dictionary
+of variable names mapped to the values used for the modification.
+The only exception is ``unset`` that requires just a list of variable names.
+No particular order is ensured on the execution of each of these modifications.
+
 ----------------------------
 Seeing Spack's Configuration
 ----------------------------

lib/spack/docs/containers.rst

@@ -0,0 +1,307 @@
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+   SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+.. _containers:
+
+================
+Container Images
+================
+
+Spack can be an ideal tool to setup images for containers since all the
+features discussed in :ref:`environments` can greatly help to manage
+the installation of software during the image build process. Nonetheless,
+building a production image from scratch still requires a lot of
+boilerplate to:
+
+- Get Spack working within the image, possibly running as root
+- Minimize the physical size of the software installed
+- Properly update the system software in the base image
+
+To facilitate users with these tedious tasks, Spack provides a command
+to automatically generate recipes for container images based on
+Environments:
+
+.. code-block:: console
+
+   $ ls
+   spack.yaml
+
+   $ spack containerize
+   # Build stage with Spack pre-installed and ready to be used
+   FROM spack/centos7:latest as builder
+
+   # What we want to install and how we want to install it
+   # is specified in a manifest file (spack.yaml)
+   RUN mkdir /opt/spack-environment \
+   &&  (echo "spack:" \
+   &&   echo "  specs:" \
+   &&   echo "  - gromacs+mpi" \
+   &&   echo "  - mpich" \
+   &&   echo "  concretization: together" \
+   &&   echo "  config:" \
+   &&   echo "    install_tree: /opt/software" \
+   &&   echo "  view: /opt/view") > /opt/spack-environment/spack.yaml
+
+   # Install the software, remove unnecessary deps
+   RUN cd /opt/spack-environment && spack install && spack gc -y
+
+   # Strip all the binaries
+   RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
+       xargs file -i | \
+       grep 'charset=binary' | \
+       grep 'x-executable\|x-archive\|x-sharedlib' | \
+       awk -F: '{print $1}' | xargs strip -s
+
+   # Modifications to the environment that are necessary to run
+   RUN cd /opt/spack-environment && \
+       spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
+
+
+   # Bare OS image to run the installed executables
+   FROM centos:7
+
+   COPY --from=builder /opt/spack-environment /opt/spack-environment
+   COPY --from=builder /opt/software /opt/software
+   COPY --from=builder /opt/view /opt/view
+   COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
+
+   RUN yum update -y && yum install -y epel-release && yum update -y                                   \
+    && yum install -y libgomp \
+    && rm -rf /var/cache/yum  && yum clean all
+
+   RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
+
+
+   LABEL "app"="gromacs"
+   LABEL "mpi"="mpich"
+
+   ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
+
+
+The bits that make this automation possible are discussed in details
+below. All the images generated in this way will be based on
+multi-stage builds with:
+
+- A fat ``build`` stage containing common build tools and Spack itself
+- A minimal ``final`` stage containing only the software requested by the user
+
+-----------------
+Spack Base Images
+-----------------
+
+Docker images with Spack preinstalled and ready to be used are
+built on `Docker Hub <https://hub.docker.com/u/spack>`_
+at every push to ``develop`` or to a release branch. The OS that
+are currently supported are summarized in the table below:
+
+.. _containers-supported-os:
+
+.. list-table:: Supported operating systems
+   :header-rows: 1
+
+   * - Operating System
+     - Base Image
+     - Spack Image
+   * - Ubuntu 16.04
+     - ``ubuntu:16.04``
+     - ``spack/ubuntu-xenial``
+   * - Ubuntu 18.04
+     - ``ubuntu:18.04``
+     - ``spack/ubuntu-bionic``
+   * - CentOS 6
+     - ``centos:6``
+     - ``spack/centos6``
+   * - CentOS 7
+     - ``centos:7``
+     - ``spack/centos7``
+
+All the images are tagged with the corresponding release of Spack:
+
+.. image:: dockerhub_spack.png
+
+with the exception of the ``latest`` tag that points to the HEAD
+of the ``develop`` branch. These images are available for anyone
+to use and take care of all the repetitive tasks that are necessary
+to setup Spack within a container. All the container recipes generated
+automatically by Spack use them as base images for their ``build`` stage.
+
+
+-------------------------
+Environment Configuration
+-------------------------
+
+Any Spack Environment can be used for the automatic generation of container
+recipes. Sensible defaults are provided for things like the base image or the
+version of Spack used in the image. If a finer tuning is needed it can be
+obtained by adding the relevant metadata under the ``container`` attribute
+of environments:
+
+.. code-block:: yaml
+
+   spack:
+     specs:
+     - gromacs+mpi
+     - mpich
+
+     container:
+       # Select the format of the recipe e.g. docker,
+       # singularity or anything else that is currently supported
+       format: docker
+
+       # Select from a valid list of images
+       base:
+         image: "centos:7"
+         spack: develop
+
+       # Whether or not to strip binaries
+       strip: true
+
+       # Additional system packages that are needed at runtime
+       os_packages:
+       - libgomp
+
+       # Extra instructions
+       extra_instructions:
+         final: |
+           RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
+
+       # Labels for the image
+       labels:
+         app: "gromacs"
+         mpi: "mpich"
+
+The tables below describe the configuration options that are currently supported:
+
+.. list-table:: General configuration options for the ``container`` section of ``spack.yaml``
+   :header-rows: 1
+
+   * - Option Name
+     - Description
+     - Allowed Values
+     - Required
+   * - ``format``
+     - The format of the recipe
+     - ``docker`` or ``singularity``
+     - Yes
+   * - ``base:image``
+     - Base image for ``final`` stage
+     - See :ref:`containers-supported-os`
+     - Yes
+   * - ``base:spack``
+     - Version of Spack
+     - Valid tags for ``base:image``
+     - Yes
+   * - ``strip``
+     - Whether to strip binaries
+     - ``true`` (default) or ``false``
+     - No
+   * - ``os_packages``
+     - System packages to be installed
+     - Valid packages for the ``final`` OS
+     - No
+   * - ``extra_instructions:build``
+     - Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``build`` stage
+     - Anything understood by the current ``format``
+     - No
+   * - ``extra_instructions:final``
+     - Extra instructions (e.g. `RUN`, `COPY`, etc.) at the end of the ``final`` stage
+     - Anything understood by the current ``format``
+     - No
+   * - ``labels``
+     - Labels to tag the image
+     - Pairs of key-value strings
+     - No
+
+.. list-table:: Configuration options specific to Singularity
+   :header-rows: 1
+
+   * - Option Name
+     - Description
+     - Allowed Values
+     - Required
+   * - ``singularity:runscript``
+     - Content of ``%runscript``
+     - Any valid script
+     - No
+   * - ``singularity:startscript``
+     - Content of ``%startscript``
+     - Any valid script
+     - No
+   * - ``singularity:test``
+     - Content of ``%test``
+     - Any valid script
+     - No
+   * - ``singularity:help``
+     - Description of the image
+     - Description string
+     - No
+
+Once the Environment is properly configured a recipe for a container
+image can be printed to standard output by issuing the following
+command from the directory where the ``spack.yaml`` resides:
+
+.. code-block:: console
+
+   $ spack containerize
+
+The example ``spack.yaml`` above would produce for instance the
+following ``Dockerfile``:
+
+.. code-block:: docker
+
+   # Build stage with Spack pre-installed and ready to be used
+   FROM spack/centos7:latest as builder
+
+   # What we want to install and how we want to install it
+   # is specified in a manifest file (spack.yaml)
+   RUN mkdir /opt/spack-environment \
+   &&  (echo "spack:" \
+   &&   echo "  specs:" \
+   &&   echo "  - gromacs+mpi" \
+   &&   echo "  - mpich" \
+   &&   echo "  concretization: together" \
+   &&   echo "  config:" \
+   &&   echo "    install_tree: /opt/software" \
+   &&   echo "  view: /opt/view") > /opt/spack-environment/spack.yaml
+
+   # Install the software, remove unnecessary deps
+   RUN cd /opt/spack-environment && spack install && spack gc -y
+
+   # Strip all the binaries
+   RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
+       xargs file -i | \
+       grep 'charset=binary' | \
+       grep 'x-executable\|x-archive\|x-sharedlib' | \
+       awk -F: '{print $1}' | xargs strip -s
+
+   # Modifications to the environment that are necessary to run
+   RUN cd /opt/spack-environment && \
+       spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh
+
+
+   # Bare OS image to run the installed executables
+   FROM centos:7
+
+   COPY --from=builder /opt/spack-environment /opt/spack-environment
+   COPY --from=builder /opt/software /opt/software
+   COPY --from=builder /opt/view /opt/view
+   COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh
+
+   RUN yum update -y && yum install -y epel-release && yum update -y                                   \
+    && yum install -y libgomp \
+    && rm -rf /var/cache/yum  && yum clean all
+
+   RUN echo 'export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][gromacs]\[$(tput setaf 2)\]\u\[$(tput sgr0)\]:\w $ \[$(tput sgr0)\]"' >> ~/.bashrc
+
+
+   LABEL "app"="gromacs"
+   LABEL "mpi"="mpich"
+
+   ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
+
+.. note::
+   Spack can also produce Singularity definition files to build the image. The
+   minimum version of Singularity required to build a SIF (Singularity Image Format)
+   from them is ``3.5.3``.

lib/spack/docs/contribution_guide.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -64,6 +64,8 @@ If you take a look in ``$SPACK_ROOT/.travis.yml``, you'll notice that we test
 against Python 2.6, 2.7, and 3.4-3.7 on both macOS and Linux. We currently
 perform 3 types of tests:
 
+.. _cmd-spack-test:
+
 ^^^^^^^^^^
 Unit Tests
 ^^^^^^^^^^
@@ -86,40 +88,83 @@ To run *all* of the unit tests, use:
 
    $ spack test
 
-These tests may take several minutes to complete. If you know you are only
-modifying a single Spack feature, you can run a single unit test at a time:
+These tests may take several minutes to complete. If you know you are
+only modifying a single Spack feature, you can run subsets of tests at a
+time.  For example, this would run all the tests in
+``lib/spack/spack/test/architecture.py``:
 
 .. code-block:: console
 
-   $ spack test architecture
+   $ spack test architecture.py
 
-This allows you to develop iteratively: make a change, test that change, make
-another change, test that change, etc. To get a list of all available unit
-tests, run:
+And this would run the ``test_platform`` test from that file:
+
+.. code-block:: console
+
+   $ spack test architecture.py::test_platform
+
+This allows you to develop iteratively: make a change, test that change,
+make another change, test that change, etc.  We use `pytest
+<http://pytest.org/>`_ as our tests fromework, and these types of
+arguments are just passed to the ``pytest`` command underneath. See `the
+pytest docs
+<http://doc.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_
+for more details on test selection syntax.
+
+``spack test`` has a few special options that can help you understand
+what tests are available.  To get a list of all available unit test
+files, run:
 
 .. command-output:: spack test --list
+   :ellipsis: 5
 
-A more detailed list of available unit tests can be found by running
-``spack test --long-list``.
+To see a more detailed list of available unit tests, use ``spack test
+--list-long``:
 
-By default, ``pytest`` captures the output of all unit tests. If you add print
-statements to a unit test and want to see the output, simply run:
+.. command-output:: spack test --list-long
+   :ellipsis: 10
+
+And to see the fully qualified names of all tests, use ``--list-names``:
+
+.. command-output:: spack test --list-names
+   :ellipsis: 5
+
+You can combine these with ``pytest`` arguments to restrict which tests
+you want to know about.  For example, to see just the tests in
+``architecture.py``:
+
+.. command-output:: spack test --list-long architecture.py
+
+You can also combine any of these options with a ``pytest`` keyword
+search.  For example, to see the names of all tests that have "spec"
+or "concretize" somewhere in their names:
+
+.. command-output:: spack test --list-names -k "spec and concretize"
+
+By default, ``pytest`` captures the output of all unit tests, and it will
+print any captured output for failed tests. Sometimes it's helpful to see
+your output interactively, while the tests run (e.g., if you add print
+statements to a unit tests).  To see the output *live*, use the ``-s``
+argument to ``pytest``:
 
 .. code-block:: console
 
-   $ spack test -s -k architecture
+   $ spack test -s architecture.py::test_platform
 
-Unit tests are crucial to making sure bugs aren't introduced into Spack. If you
-are modifying core Spack libraries or adding new functionality, please consider
-adding new unit tests or strengthening existing tests.
+Unit tests are crucial to making sure bugs aren't introduced into
+Spack. If you are modifying core Spack libraries or adding new
+functionality, please add new unit tests for your feature, and consider
+strengthening existing tests.  You will likely be asked to do this if you
+submit a pull request to the Spack project on GitHub.  Check out the
+`pytest docs <http://pytest.org/>`_ and feel free to ask for guidance on
+how to write tests!
 
 .. note::
 
-   There is also a ``run-unit-tests`` script in ``share/spack/qa`` that
-   runs the unit tests. Afterwards, it reports back to Codecov with the
-   percentage of Spack that is covered by unit tests. This script is
-   designed for Travis CI. If you want to run the unit tests yourself, we
-   suggest you use ``spack test``.
+   You may notice the ``share/spack/qa/run-unit-tests`` script in the
+   repository.  This script is designed for Travis CI.  It runs the unit
+   tests and reports coverage statistics back to Codecov. If you want to
+   run the unit tests yourself, we suggest you use ``spack test``.
 
 ^^^^^^^^^^^^
 Flake8 Tests
@@ -223,8 +268,7 @@ documentation. In order to prevent things like broken links and missing imports,
 we added documentation tests that build the documentation and fail if there
 are any warning or error messages.
 
-Building the documentation requires several dependencies, all of which can be
-installed with Spack:
+Building the documentation requires several dependencies:
 
 * sphinx
 * sphinxcontrib-programoutput
@@ -234,11 +278,18 @@ installed with Spack:
 * mercurial
 * subversion
 
+All of these can be installed with Spack, e.g.
+
+.. code-block:: console
+
+   $ spack install py-sphinx py-sphinxcontrib-programoutput py-sphinx-rtd-theme graphviz git mercurial subversion
+
 .. warning::
 
    Sphinx has `several required dependencies <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-sphinx/package.py>`_.
-   If you installed ``py-sphinx`` with Spack, make sure to add all of these
-   dependencies to your ``PYTHONPATH``. The easiest way to do this is to run:
+   If you're using a ``python`` from Spack and you installed
+   ``py-sphinx`` and friends, you need to make them available to your
+   ``python``. The easiest way to do this is to run:
 
    .. code-block:: console
 
@@ -246,8 +297,10 @@ installed with Spack:
       $ spack activate py-sphinx-rtd-theme
       $ spack activate py-sphinxcontrib-programoutput
 
-   so that all of the dependencies are symlinked to a central location.
-   If you see an error message like:
+   so that all of the dependencies are symlinked into that Python's
+   tree.  Alternatively, you could arrange for their library
+   directories to be added to PYTHONPATH.  If you see an error message
+   like:
 
    .. code-block:: console
 
@@ -332,8 +385,8 @@ coverage. This helps us tell what percentage of lines of code in Spack are
 covered by unit tests. Although code covered by unit tests can still contain
 bugs, it is much less error prone than code that is not covered by unit tests.
 
-Codecov provides `browser extensions <https://github.com/codecov/browser-extension>`_
-for Google Chrome, Firefox, and Opera. These extensions integrate with GitHub
+Codecov provides `browser extensions <https://github.com/codecov/sourcegraph-codecov>`_
+for Google Chrome and Firefox. These extensions integrate with GitHub
 and allow you to see coverage line-by-line when viewing the Spack repository.
 If you are new to Spack, a great way to get started is to write unit tests to
 increase coverage!

lib/spack/docs/developer_guide.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -363,12 +363,12 @@ Developer commands
 ``spack doc``
 ^^^^^^^^^^^^^
 
-.. _cmd-spack-test:
-
 ^^^^^^^^^^^^^^
 ``spack test``
 ^^^^^^^^^^^^^^
 
+See the :ref:`contributor guide section <cmd-spack-test>` on ``spack test``.
+
 .. _cmd-spack-python:
 
 ^^^^^^^^^^^^^^^^

lib/spack/docs/docker_for_developers.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/environments.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -49,6 +49,8 @@ Spack uses a "manifest and lock" model similar to `Bundler gemfiles
 managers. The user input file is named ``spack.yaml`` and the lock
 file is named ``spack.lock``
 
+.. _environments-using:
+
 ------------------
 Using Environments
 ------------------
@@ -128,7 +130,7 @@ To activate an environment, use the following command:
 By default, the ``spack env activate`` will load the view associated
 with the Environment into the user environment. The ``-v,
 --with-view`` argument ensures this behavior, and the ``-V,
---without-vew`` argument activates the environment without changing
+--without-view`` argument activates the environment without changing
 the user environment variables.
 
 The ``-p`` option to the ``spack env activate`` command modifies the
@@ -165,15 +167,6 @@ Any directory can be treated as an environment if it contains a file
 
    $ spack env activate -d /path/to/directory
 
-Spack commands that are environment sensitive will also act on the
-environment any time the current working directory contains a
-``spack.yaml`` file. Changing working directory to a directory
-containing a ``spack.yaml`` file is equivalent to the command:
-
-.. code-block:: console
-
-   $ spack env activate -d /path/to/dir --without-view
-
 Anonymous specs can be created in place using the command:
 
 .. code-block:: console
@@ -279,18 +272,18 @@ in the lockfile, nor does it install the spec.
 
 The ``spack add`` command is environment aware. It adds to the
 currently active environment. All environment aware commands can also
-be called using the ``spack -E`` flag to specify the environment.
+be called using the ``spack -e`` flag to specify the environment.
 
 .. code-block:: console
 
-   $ spack activate myenv
+   $ spack env activate myenv
    $ spack add mpileaks
 
 or
 
 .. code-block:: console
 
-   $ spack -E myenv add python
+   $ spack -e myenv add python
 
 .. _environments_concretization:
 
@@ -382,11 +375,12 @@ the Environment.
 Loading
 ^^^^^^^
 
-Once an environment has been installed, the following creates a load script for it:
+Once an environment has been installed, the following creates a load
+script for it:
 
 .. code-block:: console
 
-   $ spack env myenv loads -r
+   $ spack env loads -r
 
 This creates a file called ``loads`` in the environment directory.
 Sourcing that file in Bash will make the environment available to the
@@ -599,7 +593,7 @@ files are identical.
    spack:
      definitions:
        - first: [libelf, libdwarf]
-       - compilers: ['%gcc', '^intel']
+       - compilers: ['%gcc', '%intel']
        - second:
            - $first
            - matrix:
@@ -644,7 +638,7 @@ named list ``compilers`` is ``['%gcc', '%clang', '%intel']`` on
    spack:
      definitions:
        - compilers: ['%gcc', '%clang']
-       - when: target == 'x86_64'
+       - when: arch.satisfies('x86_64:')
          compilers: ['%intel']
 
 .. note::
@@ -663,8 +657,12 @@ The valid variables for a ``when`` clause are:
 #. ``target``. The target string of the default Spack
    architecture on the system.
 
-#. ``architecture`` or ``arch``. The full string of the
-   default Spack architecture on the system.
+#. ``architecture`` or ``arch``. A Spack spec satisfying the default Spack
+   architecture on the system. This supports querying via the ``satisfies``
+   method, as shown above.
+
+#. ``arch_str``. The architecture string of the default Spack architecture
+   on the system.
 
 #. ``re``. The standard regex module in Python.
 
@@ -673,6 +671,40 @@ The valid variables for a ``when`` clause are:
 #. ``hostname``. The hostname of the system (if ``hostname`` is an
    executable in the user's PATH).
 
+""""""""""""""""""""""""
+SpecLists as Constraints
+""""""""""""""""""""""""
+
+Dependencies and compilers in Spack can be both packages in an
+environment and constraints on other packages. References to SpecLists
+allow a shorthand to treat packages in a list as either a compiler or
+a dependency using the ``$%`` or ``$^`` syntax respectively.
+
+For example, the following environment has three root packages:
+``gcc@8.1.0``, ``mvapich2@2.3.1 %gcc@8.1.0``, and ``hdf5+mpi
+%gcc@8.1.0 ^mvapich2@2.3.1``.
+
+.. code-block:: yaml
+
+   spack:
+     definitions:
+     - compilers: [gcc@8.1.0]
+     - mpis: [mvapich2@2.3.1]
+     - packages: [hdf5+mpi]
+
+     specs:
+     - $compilers
+     - matrix:
+       - [$mpis]
+       - [$%compilers]
+     - matrix:
+       - [$packages]
+       - [$^mpis]
+       - [$%compilers]
+
+This allows for a much-needed reduction in redundancy between packages
+and constraints.
+
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 Environment-managed Views
 ^^^^^^^^^^^^^^^^^^^^^^^^^

lib/spack/docs/extensions.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -9,12 +9,6 @@
 Custom Extensions
 =================
 
-.. warning::
-
-   The support for extending Spack with custom commands is still experimental.
-   Users should expect APIs or prescribed directory structures to
-   change at any time.
-
 *Spack extensions* permit you to extend Spack capabilities by deploying your
 own custom commands or logic in an arbitrary location on your filesystem.
 This might be extremely useful e.g. to develop and maintain a command whose purpose is

lib/spack/docs/features.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/getting_started.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -71,10 +71,6 @@ This automatically adds Spack to your ``PATH`` and allows the ``spack``
 command to be used to execute spack :ref:`commands <shell-support>` and
 :ref:`useful packaging commands <packaging-shell-support>`.
 
-If :ref:`environment-modules <InstallEnvironmentModules>` is
-installed and available, the ``spack`` command can also load and unload
-:ref:`modules <modules>`.
-
 ^^^^^^^^^^^^^^^^^
 Clean Environment
 ^^^^^^^^^^^^^^^^^
@@ -97,7 +93,7 @@ Check Installation
 With Spack installed, you should be able to run some basic Spack
 commands.  For example:
 
-.. command-output:: spack spec netcdf
+.. command-output:: spack spec netcdf-c
 
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -482,7 +478,7 @@ Fortran.
           cxx: /usr/bin/clang++
           f77: /path/to/bin/gfortran
           fc: /path/to/bin/gfortran
-        spec: clang@11.0.0-apple
+        spec: apple-clang@11.0.0
 
 
    If you used Spack to install GCC, you can get the installation prefix by
@@ -851,7 +847,7 @@ from websites and from git.
 
 .. warning::
 
-   This workaround should be used ONLY as a last resort!  Wihout SSL
+   This workaround should be used ONLY as a last resort!  Without SSL
    certificate verification, spack and git will download from sites you
    wouldn't normally trust.  The code you download and run may then be
    compromised!  While this is not a major issue for archives that will
@@ -900,9 +896,8 @@ Core Spack Utilities
 ^^^^^^^^^^^^^^^^^^^^
 
 Core Spack uses the following packages, mainly to download and unpack
-source code, and to load generated environment modules: ``curl``,
-``env``, ``git``, ``go``, ``hg``, ``svn``, ``tar``, ``unzip``,
-``patch``, ``environment-modules``.
+source code: ``curl``, ``env``, ``git``, ``go``, ``hg``, ``svn``,
+``tar``, ``unzip``, ``patch``
 
 As long as the user's environment is set up to successfully run these
 programs from outside of Spack, they should work inside of Spack as
@@ -910,10 +905,6 @@ well.  They can generally be activated as in the ``curl`` example above;
 or some systems might already have an appropriate hand-built
 environment module that may be loaded.  Either way works.
 
-If you find that you are missing some of these programs, ``spack`` can
-build some of them for you with ``spack bootstrap``. Currently supported
-programs are ``environment-modules``.
-
 A few notes on specific programs in this list:
 
 """"""""""""""""""""""""""
@@ -941,45 +932,6 @@ other programs will also not work, because they also rely on OpenSSL.
 Once ``curl`` has been installed, you can similarly install the others.
 
 
-.. _InstallEnvironmentModules:
-
-"""""""""""""""""""
-Environment Modules
-"""""""""""""""""""
-
-In order to use Spack's generated module files, you must have
-installed ``environment-modules`` or ``lmod``. The simplest way
-to get the latest version of either of these tools is installing
-it as part of Spack's bootstrap procedure:
-
-.. code-block:: console
-
-   $ spack bootstrap
-
-.. warning::
-   At the moment ``spack bootstrap`` is only able to install ``environment-modules``.
-   Extending its capabilities to prefer ``lmod`` where possible is in the roadmap,
-   and likely to happen before the next release.
-
-Alternatively, on many Linux distributions, you can install a pre-built binary
-from the vendor's repository. On Fedora/RHEL/CentOS, for example, this can be
-done with the command:
-
-.. code-block:: console
-
-   $ yum install environment-modules
-
-Once you have the tool installed and available in your path, you can source
-Spack's setup file:
-
-.. code-block:: console
-
-   $ source share/spack/setup-env.sh
-
-This activates :ref:`shell support <shell-support>` and makes commands like
-``spack load`` available for use.
-
-
 ^^^^^^^^^^^^^^^^^
 Package Utilities
 ^^^^^^^^^^^^^^^^^

lib/spack/docs/index.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -66,6 +66,7 @@ or refer to the full manual below.
    config_yaml
    build_settings
    environments
+   containers
    mirrors
    module_file_support
    repositories
@@ -74,6 +75,7 @@ or refer to the full manual below.
    package_list
    chain
    extensions
+   pipelines
 
 .. toctree::
    :maxdepth: 2

lib/spack/docs/known_issues.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/mirrors.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/module_file_support.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -17,22 +17,16 @@ Spack integrates with `Environment Modules
 <http://lmod.readthedocs.io/en/latest/>`_ by
 providing post-install hooks that generate module files and commands to manipulate them.
 
-.. note::
-
-   If your machine does not already have a module system installed,
-   we advise you to use either Environment Modules or LMod. See :ref:`InstallEnvironmentModules`
-   for more details.
-
 .. _shell-support:
 
 ----------------------------
 Using module files via Spack
 ----------------------------
 
-If you have installed a supported module system either manually or through
-``spack bootstrap``, you should be able to run either ``module avail`` or
-``use -l spack`` to see what module files have been installed.  Here is
-sample output of those programs, showing lots of installed packages:
+If you have installed a supported module system you should be able to
+run either ``module avail`` or ``use -l spack`` to see what module
+files have been installed.  Here is sample output of those programs,
+showing lots of installed packages:
 
 .. code-block:: console
 
@@ -93,9 +87,7 @@ Note that in the latter case it is necessary to explicitly set ``SPACK_ROOT``
 before sourcing the setup file (you will get a meaningful error message
 if you don't).
 
-When ``bash`` and ``ksh`` users update their environment with ``setup-env.sh``, it will check for spack-installed environment modules and add the ``module`` command to their environment; This only occurs if the module command is not already available. You can install ``environment-modules`` with ``spack bootstrap`` as described in :ref:`InstallEnvironmentModules`.
-
-Finally, if you want to have Spack's shell support available on the command line at
+If you want to have Spack's shell support available on the command line at
 any login you can put this source line in one of the files that are sourced
 at startup (like ``.profile``, ``.bashrc`` or ``.cshrc``). Be aware though
 that the startup time may be slightly increased because of that.
@@ -119,7 +111,7 @@ For example this will add the ``mpich`` package built with ``gcc`` to your path:
 
    # ... wait for install ...
 
-   $ spack load mpich %gcc@4.4.7    # modules
+   $ spack load mpich %gcc@4.4.7
    $ which mpicc
    ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpich@3.0.4/bin/mpicc
 
@@ -129,27 +121,29 @@ want to use a package, you can type unload or unuse similarly:
 
 .. code-block:: console
 
-   $ spack unload mpich %gcc@4.4.7  # modules
+   $ spack unload mpich %gcc@4.4.7
 
 .. note::
 
-   The ``load`` and ``unload`` subcommands are
-   only available if you have enabled Spack's shell support *and* you
-   have environment-modules installed on your machine.
+   The ``load`` and ``unload`` subcommands are only available if you
+   have enabled Spack's shell support. These command DO NOT use the
+   underlying Spack-generated module files.
 
-^^^^^^^^^^^^^^^^^^^^^^
-Ambiguous module names
-^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^
+Ambiguous specs
+^^^^^^^^^^^^^^^
 
-If a spec used with load/unload or use/unuse is ambiguous (i.e. more
-than one installed package matches it), then Spack will warn you:
+If a spec used with load/unload or is ambiguous (i.e. more than one
+installed package matches it), then Spack will warn you:
 
 .. code-block:: console
 
    $ spack load libelf
-   ==> Error: Multiple matches for spec libelf.  Choose one:
-   libelf@0.8.13%gcc@4.4.7 arch=linux-debian7-x86_64
-   libelf@0.8.13%intel@15.0.0 arch=linux-debian7-x86_64
+   ==> Error: libelf matches multiple packages.
+   Matching packages:
+     libelf@0.8.13%gcc@4.4.7 arch=linux-debian7-x86_64
+     libelf@0.8.13%intel@15.0.0 arch=linux-debian7-x86_64
+   Use a more specific spec
 
 You can either type the ``spack load`` command again with a fully
 qualified argument, or you can add just enough extra constraints to
@@ -163,16 +157,21 @@ used ``gcc``.  You could therefore just type:
 
 To identify just the one built with the Intel compiler.
 
-.. _extensions:
-
 .. _cmd-spack-module-loads:
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 ``spack module tcl loads``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-In some cases, it is desirable to load not just a module, but also all
-the modules it depends on.  This is not required for most modules
+In some cases, it is desirable to use a Spack-generated module, rather
+than relying on Spack's built-in user-environment modification
+capabilities. To translate a spec into a module name, use ``spack
+module tcl loads`` or ``spack module lmod loads`` depending on the
+module system desired.
+
+
+To load not just a module, but also all the modules it depends on, use
+the ``--dependencies`` option. This is not required for most modules
 because Spack builds binaries with RPATH support.  However, not all
 packages use RPATH to find their dependencies: this can be true in
 particular for Python extensions, which are currently *not* built with
@@ -460,14 +459,14 @@ is compiled with ``gcc@4.4.7``, with the only exception of any ``gcc``
 or any ``llvm`` installation.
 
 
-.. _modules-naming-scheme:
+.. _modules-projections:
 
-"""""""""""""""""""""""""""
-Customize the naming scheme
-"""""""""""""""""""""""""""
+"""""""""""""""""""""""""""""""
+Customize the naming of modules
+"""""""""""""""""""""""""""""""
 
 The names of environment modules generated by spack are not always easy to
-fully comprehend due to the long hash in the name. There are two module
+fully comprehend due to the long hash in the name. There are three module
 configuration options to help with that. The first is a global setting to
 adjust the hash length. It can be set anywhere from 0 to 32 and has a default
 length of 7. This is the representation of the hash in the module file name and
@@ -501,20 +500,46 @@ version of python a set of python extensions is associated with. Likewise, the
 ``openblas`` string is attached to any program that has openblas in the spec,
 most likely via the ``+blas`` variant specification.
 
+The most heavyweight solution to module naming is to change the entire
+naming convention for module files. This uses the projections format
+covered in :ref:`adding_projections_to_views`.
+
+.. code-block:: yaml
+
+  modules:
+    tcl:
+      projections:
+        all: '{name}/{version}-{compiler.name}-{compiler.version}-module'
+        ^mpi: '{name}/{version}-{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version}-module'
+
+will create module files that are nested in directories by package
+name, contain the version and compiler name and version, and have the
+word ``module`` before the hash for all specs that do not depend on
+mpi, and will have the same information plus the MPI implementation
+name and version for all packages that depend on mpi.
+
+When specifying module names by projection for Lmod modules, we
+recommend NOT including names of dependencies (e.g., MPI, compilers)
+that are already in the LMod hierarchy.
+
+
+
 .. note::
-   TCL module files
-     A modification that is specific to ``tcl`` module files is the possibility
-     to change the naming scheme of modules.
+   TCL modules
+     TCL modules also allow for explicit conflicts between modulefiles.
 
      .. code-block:: yaml
 
-       modules:
-         tcl:
-           naming_scheme: '{name}/{version}-{compiler.name}-{compiler.version}'
-           all:
-             conflict:
-               - '{name}'
-               - 'intel/14.0.1'
+        modules:
+          enable:
+            - tcl
+          tcl:
+            projections:
+              all: '{name}/{version}-{compiler.name}-{compiler.version}'
+            all:
+              conflict:
+                - '{name}'
+                - 'intel/14.0.1'
 
      will create module files that will conflict with ``intel/14.0.1`` and with the
      base directory of the same module, effectively preventing the possibility to
@@ -539,6 +564,8 @@ most likely via the ``+blas`` variant specification.
          lmod:
            core_compilers:
              - 'gcc@4.8'
+           core_specs:
+             - 'python'
            hierarchy:
              - 'mpi'
              - 'lapack'
@@ -548,6 +575,15 @@ most likely via the ``+blas`` variant specification.
      implementations of ``mpi`` and ``lapack``, and let LMod switch safely from one to the
      other.
 
+     All packages built with a compiler in ``core_compilers`` and all
+     packages that satisfy a spec in ``core_specs`` will be put in the
+     ``Core`` hierarchy of the lua modules.
+
+.. warning::
+  Consistency of Core packages
+   The user is responsible for maintining consistency among core packages, as ``core_specs``
+   bypasses the hierarchy that allows LMod to safely switch between coherent software stacks.
+
 .. warning::
   Deep hierarchies and ``lmod spider``
    For hierarchies that are deeper than three layers ``lmod spider`` may have some issues.

lib/spack/docs/package_list.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/packaging_guide.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -136,6 +136,10 @@ generates a boilerplate template for your package, and opens up the new
        homepage = "http://www.example.com"
        url      = "https://gmplib.org/download/gmp/gmp-6.1.2.tar.bz2"
 
+       # FIXME: Add a list of GitHub accounts to
+       # notify when the package is updated.
+       # maintainers = ['github_user1', 'github_user2']
+
        version('6.1.2', '8ddbb26dc3bd4e2302984debba1406a5')
        version('6.1.1', '4c175f86e11eb32d8bf9872ca3a8e11d')
        version('6.1.0', '86ee6e54ebfc4a90b643a65e402c4048')
@@ -184,6 +188,17 @@ The rest of the tasks you need to do are as follows:
    The ``homepage`` is displayed when users run ``spack info`` so
    that they can learn more about your package.
 
+#. Add a comma-separated list of maintainers.
+
+   The ``maintainers`` field is a list of GitHub accounts of people
+   who want to be notified any time the package is modified. When a
+   pull request is submitted that updates the package, these people
+   will be requested to review the PR. This is useful for developers
+   who maintain a Spack package for their own software, as well as
+   users who rely on a piece of software and want to ensure that the
+   package doesn't break. It also gives users a list of people to
+   contact for help when someone reports a build error with the package.
+
 #. Add ``depends_on()`` calls for the package's dependencies.
 
    ``depends_on`` tells Spack that other packages need to be built
@@ -538,6 +553,34 @@ version. This is useful for packages that have an easy to extrapolate URL, but
 keep changing their URL format every few releases. With this method, you only
 need to specify the ``url`` when the URL changes.
 
+"""""""""""""""""""""""
+Mirrors of the main URL
+"""""""""""""""""""""""
+
+Spack supports listing mirrors of the main URL in a package by defining
+the ``urls`` attribute:
+
+.. code-block:: python
+
+  class Foo(Package):
+
+    urls = [
+        'http://example.com/foo-1.0.tar.gz',
+        'http://mirror.com/foo-1.0.tar.gz'
+    ]
+
+instead of just a single ``url``. This attribute is a list of possible URLs that
+will be tried in order when fetching packages. Notice that either one of ``url``
+or ``urls`` can be present in a package, but not both at the same time.
+
+A well-known case of packages that can be fetched from multiple mirrors is that
+of GNU. For that, Spack goes a step further and defines a mixin class that
+takes care of all of the plumbing and requires packagers to just define a proper
+``gnu_mirror_path`` attribute:
+
+.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/autoconf/package.py
+   :lines: 9-18
+
 ^^^^^^^^^^^^^^^^^^^^^^^^
 Skipping the expand step
 ^^^^^^^^^^^^^^^^^^^^^^^^
@@ -886,6 +929,9 @@ Git fetching supports the following parameters to ``version``:
 * ``tag``: Name of a tag to fetch.
 * ``commit``: SHA hash (or prefix) of a commit to fetch.
 * ``submodules``: Also fetch submodules recursively when checking out this repository.
+* ``submodules_delete``: A list of submodules to forcibly delete from the repository
+  after fetching. Useful if a version in the repository has submodules that
+  have disappeared/are no longer accessible.
 * ``get_full_repo``: Ensure the full git history is checked out with all remote
   branch information. Normally (``get_full_repo=False``, the default), the git
   option ``--depth 1`` will be used if the version of git and the specified
@@ -1464,8 +1510,8 @@ that the same package with different patches applied will have different
 hash identifiers.  To ensure that the hashing scheme is consistent, you
 must use a ``sha256`` checksum for the patch.  Patches will be fetched
 from their URLs, checked, and applied to your source code.  You can use
-the ``spack sha256`` command to generate a checksum for a patch file or
-URL.
+the GNU utils ``sha256sum`` or the macOS ``shasum -a 256`` commands to
+generate a checksum for a patch file.
 
 Spack can also handle compressed patches.  If you use these, Spack needs
 a little more help.  Specifically, it needs *two* checksums: the
@@ -1629,15 +1675,15 @@ can see the patches that would be applied to ``m4``::
 
   Concretized
   --------------------------------
-  m4@1.4.18%clang@9.0.0-apple patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=darwin-highsierra-x86_64
-      ^libsigsegv@2.11%clang@9.0.0-apple arch=darwin-highsierra-x86_64
+  m4@1.4.18%apple-clang@9.0.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=darwin-highsierra-x86_64
+      ^libsigsegv@2.11%apple-clang@9.0.0 arch=darwin-highsierra-x86_64
 
 You can also see patches that have been applied to installed packages
 with ``spack find -v``::
 
   $ spack find -v m4
   ==> 1 installed package
-  -- darwin-highsierra-x86_64 / clang@9.0.0-apple -----------------
+  -- darwin-highsierra-x86_64 / apple-clang@9.0.0 -----------------
   m4@1.4.18 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv
 
 .. _cmd-spack-resource:
@@ -1667,7 +1713,7 @@ wonder where the extra boost patches are coming from::
 
   $ spack spec dealii ^boost@1.68.0 ^hdf5+fortran | grep '\^boost'
       ^boost@1.68.0
-          ^boost@1.68.0%clang@9.0.0-apple+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199,b37164268f34f7133cbc9a4066ae98fda08adf51e1172223f6a969909216870f ~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=darwin-highsierra-x86_64
+          ^boost@1.68.0%apple-clang@9.0.0+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199,b37164268f34f7133cbc9a4066ae98fda08adf51e1172223f6a969909216870f ~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=darwin-highsierra-x86_64
   $ spack resource show b37164268
   b37164268f34f7133cbc9a4066ae98fda08adf51e1172223f6a969909216870f
       path:       /home/spackuser/src/spack/var/spack/repos/builtin/packages/dealii/boost_1.68.0.patch
@@ -1907,6 +1953,8 @@ issues with 1.64.0, 1.65.0, and 1.66.0, you can say:
    depends_on('boost@1.59.0:1.63,1.65.1,1.67.0:')
 
 
+.. _dependency-types:
+
 ^^^^^^^^^^^^^^^^
 Dependency types
 ^^^^^^^^^^^^^^^^
@@ -1944,6 +1992,28 @@ inject the dependency's ``prefix/lib`` directory, but the package needs to
 be in ``PATH`` and ``PYTHONPATH`` during the build process and later when
 a user wants to run the package.
 
+^^^^^^^^^^^^^^^^^^^^^^^^
+Conditional dependencies
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+You may have a package that only requires a dependency under certain
+conditions. For example, you may have a package that has optional MPI support,
+- MPI is only a dependency when you want to enable MPI support for the
+package. In that case, you could say something like:
+
+.. code-block:: python
+
+   variant('mpi', default=False)
+   depends_on('mpi', when='+mpi')
+
+``when`` can include constraints on the variant, version, compiler, etc. and
+the :mod:`syntax<spack.spec>` is the same as for Specs written on the command
+line.
+
+If a dependency/feature of a package isn't typically used, you can save time
+by making it conditional (since Spack will not build the dependency unless it
+is required for the Spec).
+
 .. _dependency_dependency_patching:
 
 ^^^^^^^^^^^^^^^^^^^
@@ -2099,13 +2169,17 @@ Adding the following to a package:
 
 .. code-block:: python
 
-    conflicts('%intel', when='@1.2')
+    conflicts('%intel', when='@:1.2',
+              msg='<myNicePackage> <= v1.2 cannot be built with Intel ICC, '
+                  'please use a newer release.')
 
 we express the fact that the current package *cannot be built* with the Intel
-compiler when we are trying to install version "1.2". The ``when`` argument can
-be omitted, in which case the conflict will always be active.
+compiler when we are trying to install a version "<=1.2". The ``when`` argument
+can be omitted, in which case the conflict will always be active.
 Conflicts are always evaluated after the concretization step has been performed,
 and if any match is found a detailed error message is shown to the user.
+You can add an additional message via the ``msg=`` parameter to a conflict that
+provideds more specific instructions for users.
 
 .. _packaging_extensions:
 
@@ -2127,7 +2201,7 @@ property to ``True``, e.g.:
        extendable = True
        ...
 
-To make a package into an extension, simply add simply add an
+To make a package into an extension, simply add an
 ``extends`` call in the package definition, and pass it the name of an
 extendable package:
 
@@ -2142,6 +2216,10 @@ Now, the ``py-numpy`` package can be used as an argument to ``spack
 activate``.  When it is activated, all the files in its prefix will be
 symbolically linked into the prefix of the python package.
 
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Adding additional constraints
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 Some packages produce a Python extension, but are only compatible with
 Python 3, or with Python 2.  In those cases, a ``depends_on()``
 declaration should be made in addition to the ``extends()``
@@ -2161,8 +2239,7 @@ variant(s) are selected.  This may be accomplished with conditional
 .. code-block:: python
 
    class FooLib(Package):
-       variant('python', default=True, description= \
-           'Build the Python extension Module')
+       variant('python', default=True, description='Build the Python extension Module')
        extends('python', when='+python')
        ...
 
@@ -2843,7 +2920,7 @@ discover its dependencies.
 
 If you want to see the environment that a package will build with, or
 if you want to run commands in that environment to test them out, you
-can use the :ref:`cmd-spack-env` command, documented
+can use the :ref:`cmd-spack-build-env` command, documented
 below.
 
 ^^^^^^^^^^^^^^^^^^^^^
@@ -3537,7 +3614,7 @@ the command line.
     For most compilers, ``$rpath_flag`` is ``-Wl,-rpath,``. However, NAG
     passes its flags to GCC instead of passing them directly to the linker.
     Therefore, its ``$rpath_flag`` is doubly wrapped: ``-Wl,-Wl,,-rpath,``.
-    ``$rpath_flag`` can be overriden on a compiler specific basis in
+    ``$rpath_flag`` can be overridden on a compiler specific basis in
     ``lib/spack/spack/compilers/$compiler.py``.
 
 The compiler wrappers also pass the compiler flags specified by the user from
@@ -3971,6 +4048,70 @@ File functions
 :py:func:`touch(path) <spack.touch>`
   Create an empty file at ``path``.
 
+.. _make-package-findable:
+
+----------------------------------------------------------
+Making a package discoverable with ``spack external find``
+----------------------------------------------------------
+
+To make a package discoverable with
+:ref:`spack external find <cmd-spack-external-find>` you must
+define one or more executables associated with the package and must
+implement a method to generate a Spec when given an executable.
+
+The executables are specified as a package level ``executables``
+attribute which is a list of strings (see example below); each string
+is treated as a regular expression (e.g. 'gcc' would match 'gcc', 'gcc-8.3',
+'my-weird-gcc', etc.).
+
+The method ``determine_spec_details`` has the following signature:
+
+.. code-block:: python
+
+   def determine_spec_details(prefix, exes_in_prefix):
+       # exes_in_prefix = a set of paths, each path is an executable
+       # prefix = a prefix that is common to each path in exes_in_prefix
+
+       # return None or [] if none of the exes represent an instance of
+       # the package. Return one or more Specs for each instance of the
+       # package which is thought to be installed in the provided prefix
+
+``determine_spec_details`` takes as parameters a set of discovered
+executables (which match those specified by the user) as well as a
+common prefix shared by all of those executables. The function must
+return one or more Specs associated with the executables (it can also
+return ``None`` to indicate that no provided executables are associated
+with the package).
+
+Say for example we have a package called ``foo-package`` which
+builds an executable called ``foo``. ``FooPackage`` would appear as
+follows:
+
+.. code-block:: python
+
+   class FooPackage(Package):
+       homepage = "..."
+       url = "..."
+
+       version(...)
+
+       # Each string provided here is treated as a regular expression, and
+       # would match for example 'foo', 'foobar', and 'bazfoo'.
+       executables = ['foo']
+
+       @classmethod
+       def determine_spec_details(cls, prefix, exes_in_prefix):
+           candidates = list(x for x in exes_in_prefix
+                             if os.path.basename(x) == 'foo')
+           if not candidates:
+               return
+           # This implementation is lazy and only checks the first candidate
+           exe_path = candidates[0]
+           exe = spack.util.executable.Executable(exe_path)
+           output = exe('--version')
+           version_str = ...  # parse output for version string
+           return Spec('foo-package@{0}'.format(version_str))
+
 .. _package-lifecycle:
 
 -----------------------------
@@ -4026,16 +4167,23 @@ want to clean up the temporary directory, or if the package isn't
 downloading properly, you might want to run *only* the ``fetch`` stage
 of the build.
 
+Spack performs best-effort installation of package dependencies by default,
+which means it will continue to install as many dependencies as possible
+after detecting failures.  If you are trying to install a package with a
+lot of dependencies where one or more may fail to build, you might want to
+try the ``--fail-fast`` option to stop the installation process on the first
+failure.
+
 A typical package workflow might look like this:
 
 .. code-block:: console
 
    $ spack edit mypackage
-   $ spack install mypackage
+   $ spack install --fail-fast mypackage
    ... build breaks! ...
    $ spack clean mypackage
    $ spack edit mypackage
-   $ spack install mypackage
+   $ spack install --fail-fast mypackage
    ... repeat clean/install until install works ...
 
 Below are some commands that will allow you some finer-grained
@@ -4104,23 +4252,29 @@ Does this in one of two ways:
 ``spack clean``
 ^^^^^^^^^^^^^^^
 
-Cleans up all of Spack's temporary and cached files.  This can be used to
+Cleans up Spack's temporary and cached files.  This command can be used to
 recover disk space if temporary files from interrupted or failed installs
-accumulate in the staging area.
+accumulate.
 
 When called with ``--stage`` or without arguments this removes all staged
 files.
 
-When called with ``--downloads`` this will clear all resources
-:ref:`cached <caching>` during installs.
+The ``--downloads`` option removes cached :ref:`cached <caching>` downloads.
 
-When called with ``--user-cache`` this will remove caches in the user home
-directory, including cached virtual indices.
+You can force the removal of all install failure tracking markers using the
+``--failures`` option.  Note that ``spack install`` will automatically clear
+relevant failure markings prior to performing the requested installation(s).
+
+Long-lived caches, like the virtual package index, are removed using the
+``--misc-cache`` option.
+
+The ``--python-cache`` option removes `.pyc`, `.pyo`, and `__pycache__`
+folders.
 
 To remove all of the above, the command can be called with ``--all``.
 
-When called with positional arguments, cleans up temporary files only
-for a particular package. If ``fetch``, ``stage``, or ``install``
+When called with positional arguments, this command cleans up temporary files
+only for a particular package. If ``fetch``, ``stage``, or ``install``
 are run again after this, Spack's build process will start from scratch.
 
 
@@ -4262,31 +4416,31 @@ directory, install directory, package directory) and others change to
 core spack locations.  For example, ``spack cd --module-dir`` will take you to
 the main python source directory of your spack install.
 
-.. _cmd-spack-env:
+.. _cmd-spack-build-env:
 
-^^^^^^^^^^^^^
-``spack env``
-^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^
+``spack build-env``
+^^^^^^^^^^^^^^^^^^^
 
-``spack env`` functions much like the standard unix ``env`` command,
-but it takes a spec as an argument.  You can use it to see the
+``spack build-env`` functions much like the standard unix ``build-env``
+command, but it takes a spec as an argument.  You can use it to see the
 environment variables that will be set when a particular build runs,
 for example:
 
 .. code-block:: console
 
-   $ spack env mpileaks@1.1%intel
+   $ spack build-env mpileaks@1.1%intel
 
 This will display the entire environment that will be set when the
 ``mpileaks@1.1%intel`` build runs.
 
 To run commands in a package's build environment, you can simply
-provide them after the spec argument to ``spack env``:
+provide them after the spec argument to ``spack build-env``:
 
 .. code-block:: console
 
    $ spack cd mpileaks@1.1%intel
-   $ spack env mpileaks@1.1%intel ./configure
+   $ spack build-env mpileaks@1.1%intel ./configure
 
 This will cd to the build directory and then run ``configure`` in the
 package's build environment.
@@ -4384,7 +4538,7 @@ translate variant flags into CMake definitions.  For example:
 
 .. code-block:: python
 
-   def configure_args(self):
+   def cmake_args(self):
        spec = self.spec
        return [
            '-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'),

lib/spack/docs/pipelines.rst

@@ -0,0 +1,445 @@
+.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+   SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+.. _pipelines:
+
+=========
+Pipelines
+=========
+
+Spack provides commands that support generating and running automated build
+pipelines designed for Gitlab CI.  At the highest level it works like this:
+provide a spack environment describing the set of packages you care about,
+and include within that environment file a description of how those packages
+should be mapped to Gitlab runners.  Spack can then generate a ``.gitlab-ci.yml``
+file containing job descriptions for all your packages that can be run by a
+properly configured Gitlab CI instance.  When run, the generated pipeline will
+build and deploy binaries, and it can optionally report to a CDash instance
+regarding the health of the builds as they evolve over time.
+
+------------------------------
+Getting started with pipelines
+------------------------------
+
+It is fairly straightforward to get started with automated build pipelines.  At
+a minimum, you'll need to set up a Gitlab instance (more about Gitlab CI
+`here <https://about.gitlab.com/product/continuous-integration/>`_) and configure
+at least one `runner <https://docs.gitlab.com/runner/>`_.  Then the basic steps
+for setting up a build pipeline are as follows:
+
+#. Create a repository on your gitlab instance
+#. Add a ``spack.yaml`` at the root containing your pipeline environment (see
+   below for details)
+#. Add a ``.gitlab-ci.yml`` at the root containing two jobs (one to generate
+   the pipeline dynamically, and one to run the generated jobs), similar to
+   this one:
+
+   .. code-block:: yaml
+
+      stages: [generate, build]
+
+      generate-pipeline:
+        stage: generate
+        tags:
+          - <custom-tag>
+        script:
+          - spack env activate .
+          - spack ci generate
+            --output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/pipeline.yml"
+        artifacts:
+          paths:
+            - "${CI_PROJECT_DIR}/jobs_scratch_dir/pipeline.yml"
+
+      build-jobs:
+        stage: build
+        trigger:
+          include:
+            - artifact: "jobs_scratch_dir/pipeline.yml"
+              job: generate-pipeline
+          strategy: depend
+
+
+#. Add any secrets required by the CI process to environment variables using the
+   CI web ui
+#. Push a commit containing the ``spack.yaml`` and ``.gitlab-ci.yml`` mentioned above
+   to the gitlab repository
+
+The ``<custom-tag>``, above, is used to pick one of your configured runners to
+run the pipeline generation phase (this is implemented in the ``spack ci generate``
+command, which assumes the runner has an appropriate version of spack installed
+and configured for use).  Of course, there are many ways to customize the process.
+You can configure CDash reporting on the progress of your builds, set up S3 buckets
+to mirror binaries built by the pipeline, clone a custom spack repository/ref for
+use by the pipeline, and more.
+
+While it is possible to set up pipelines on gitlab.com, the builds there are
+limited to 60 minutes and generic hardware.  It is also possible to
+`hook up <https://about.gitlab.com/blog/2018/04/24/getting-started-gitlab-ci-gcp>`_
+Gitlab to Google Kubernetes Engine (`GKE <https://cloud.google.com/kubernetes-engine/>`_)
+or Amazon Elastic Kubernetes Service (`EKS <https://aws.amazon.com/eks>`_), though those
+topics are outside the scope of this document.
+
+Spack's pipelines are now making use of the
+`trigger <https://docs.gitlab.com/12.9/ee/ci/yaml/README.html#trigger>` syntax to run
+dynamically generated
+`child pipelines <https://docs.gitlab.com/12.9/ee/ci/parent_child_pipelines.html>`.
+Note that the use of dynamic child pipelines requires running Gitlab version
+``>= 12.9``.
+
+-----------------------------------
+Spack commands supporting pipelines
+-----------------------------------
+
+Spack provides a command ``ci`` with two sub-commands: ``spack ci generate`` generates
+a pipeline (a .gitlab-ci.yml file) from a spack environment, and ``spack ci rebuild``
+checks a spec against a remote mirror and possibly rebuilds it from source and updates
+the binary mirror with the latest built package.  Both ``spack ci ...`` commands must
+be run from within the same environment, as each one makes use of the environment for
+different purposes.  Additionally, some options to the commands (or conditions present
+in the spack environment file) may require particular environment variables to be
+set in order to function properly.  Examples of these are typically secrets
+needed for pipeline operation that should not be visible in a spack environment
+file.  These environment variables are described in more detail
+:ref:`ci_environment_variables`.
+
+.. _cmd-spack-ci:
+
+^^^^^^^^^^^^^^^^^^
+``spack ci``
+^^^^^^^^^^^^^^^^^^
+
+Super-command for functionality related to generating pipelines and executing
+pipeline jobs.
+
+.. _cmd-spack-ci-generate:
+
+^^^^^^^^^^^^^^^^^^^^^
+``spack ci generate``
+^^^^^^^^^^^^^^^^^^^^^
+
+Concretizes the specs in the active environment, stages them (as described in
+:ref:`staging_algorithm`), and writes the resulting ``.gitlab-ci.yml`` to disk.
+
+.. _cmd-spack-ci-rebuild:
+
+^^^^^^^^^^^^^^^^^^^^
+``spack ci rebuild``
+^^^^^^^^^^^^^^^^^^^^
+
+This sub-command is responsible for ensuring a single spec from the release
+environment is up to date on the remote mirror configured in the environment,
+and as such, corresponds to a single job in the ``.gitlab-ci.yml`` file.
+
+------------------------------------
+A pipeline-enabled spack environment
+------------------------------------
+
+Here's an example of a spack environment file that has been enhanced with
+sections describing a build pipeline:
+
+.. code-block:: yaml
+
+   spack:
+     definitions:
+     - pkgs:
+       - readline@7.0
+     - compilers:
+       - '%gcc@5.5.0'
+     - oses:
+       - os=ubuntu18.04
+       - os=centos7
+     specs:
+     - matrix:
+       - [$pkgs]
+       - [$compilers]
+       - [$oses]
+     mirrors:
+       cloud_gitlab: https://mirror.spack.io
+     gitlab-ci:
+       mappings:
+         - match:
+             - os=ubuntu18.04
+           runner-attributes:
+             tags:
+               - spack-kube
+             image: spack/ubuntu-bionic
+         - match:
+             - os=centos7
+           runner-attributes:
+             tags:
+               - spack-kube
+             image: spack/centos7
+     cdash:
+       build-group: Release Testing
+       url: https://cdash.spack.io
+       project: Spack
+       site: Spack AWS Gitlab Instance
+
+Hopefully, the ``definitions``, ``specs``, ``mirrors``, etc. sections are already
+familiar, as they are part of spack :ref:`environments`.  So let's take a more
+in-depth look some of the pipeline-related sections in that environment file
+that might not be as familiar.
+
+The ``gitlab-ci`` section is used to configure how the pipeline workload should be
+generated, mainly how the jobs for building specs should be assigned to the
+configured runners on your instance.  Each entry within the list of ``mappings``
+corresponds to a known gitlab runner, where the ``match`` section is used
+in assigning a release spec to one of the runners, and the ``runner-attributes``
+section is used to configure the spec/job for that particular runner.
+
+There are other pipeline options you can configure within the ``gitlab-ci`` section
+as well.  The ``bootstrap`` section allows you to specify lists of specs from
+your ``definitions`` that should be staged ahead of the environment's ``specs`` (this
+section is described in more detail below).  The ``enable-artifacts-buildcache`` key
+takes a boolean and determines whether the pipeline uses artifacts to store and
+pass along the buildcaches from one stage to the next (the default if you don't
+provide this option is ``False``).  The ``enable-debug-messages`` key takes a boolean
+and allows you to choose whether the pipeline build jobs are run as ``spack -d ci rebuild``
+or just ``spack ci rebuild`` (the default is not to enable debug messages).  The
+``final-stage-rebuild-index`` section controls whether an extra job is added to the
+end of your pipeline (in a stage by itself) which will regenerate the mirror's
+buildcache index.  Under normal operation, each pipeline job that rebuilds a package
+will re-generate the mirror's buildcache index after the buildcache entry for that
+job has been created and pushed to the mirror.  Since jobs in the same stage can run in
+parallel, there is the possibility that at the end of some stage, the index may not
+reflect all the binaries in the buildcache.  Adding the ``final-stage-rebuild-index``
+section ensures that at the end of the pipeline, the index will be in sync with the
+binaries on the mirror.  If the mirror lives in an S3 bucket, this job will need to
+run on a machine with the Python ``boto3`` module installed, and consequently the
+``final-stage-rebuild-index`` needs to specify a list of ``tags`` to pick a runner
+satisfying that condition.  It can also take an ``image`` key so Docker executor type
+runners can pick the right image for the index regeneration job.
+
+The optional ``cdash`` section provides information that will be used by the
+``spack ci generate`` command (invoked by ``spack ci start``) for reporting
+to CDash.  All the jobs generated from this environment will belong to a
+"build group" within CDash that can be tracked over time.  As the release
+progresses, this build group may have jobs added or removed. The url, project,
+and site are used to specify the CDash instance to which build results should
+be reported.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Assignment of specs to runners
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``mappings`` section corresponds to a list of runners, and during assignment
+of specs to runners, the list is traversed in order looking for matches, the
+first runner that matches a release spec is assigned to build that spec.  The
+``match`` section within each runner mapping section is a list of specs, and
+if any of those specs match the release spec (the ``spec.satisfies()`` method
+is used), then that runner is considered a match.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Configuration of specs/jobs for a runner
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Once a runner has been chosen to build a release spec, the ``runner-attributes``
+section provides information determining details of the job in the context of
+the runner.  The ``runner-attributes`` section must have a ``tags`` key, which
+is a list containing at least one tag used to select the runner from among the
+runners known to the gitlab instance.  For Docker executor type runners, the
+``image`` key is used to specify the Docker image used to build the release spec
+(and could also appear as a dictionary with a ``name`` specifying the image name,
+as well as an ``entrypoint`` to override whatever the default for that image is).
+For other types of runners the ``variables`` key will be useful to pass any
+information on to the runner that it needs to do its work (e.g. scheduler
+parameters, etc.).
+
+.. _staging_algorithm:
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Summary of ``.gitlab-ci.yml`` generation algorithm
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+All specs yielded by the matrix (or all the specs in the environment) have their
+dependencies computed, and the entire resulting set of specs are staged together
+before being run through the ``gitlab-ci/mappings`` entries, where each staged
+spec is assigned a runner.  "Staging" is the name we have given to the process
+of figuring out in what order the specs should be built, taking into consideration
+Gitlab CI rules about jobs/stages.  In the staging process the goal is to maximize
+the number of jobs in any stage of the pipeline, while ensuring that the jobs in
+any stage only depend on jobs in previous stages (since those jobs are guaranteed
+to have completed already).  As a runner is determined for a job, the information
+in the ``runner-attributes`` is used to populate various parts of the job
+description that will be used by Gitlab CI. Once all the jobs have been assigned
+a runner, the ``.gitlab-ci.yml`` is written to disk.
+
+The short example provided above would result in the ``readline``, ``ncurses``,
+and ``pkgconf`` packages getting staged and built on the runner chosen by the
+``spack-k8s`` tag.  In this example, we assume the runner is a Docker executor
+type runner, and thus certain jobs will be run in the ``centos7`` container,
+and others in the ``ubuntu-18.04`` container.  The resulting ``.gitlab-ci.yml``
+will contain 6 jobs in three stages.  Once the jobs have been generated, the
+presence of a ``SPACK_CDASH_AUTH_TOKEN`` environment variable during the
+``spack ci generate`` command would result in all of the jobs being put in a
+build group on CDash called "Release Testing" (that group will be created if
+it didn't already exist).
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Optional compiler bootstrapping
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Spack pipelines also have support for bootstrapping compilers on systems that
+may not already have the desired compilers installed. The idea here is that
+you can specify a list of things to bootstrap in your ``definitions``, and
+spack will guarantee those will be installed in a phase of the pipeline before
+your release specs, so that you can rely on those packages being available in
+the binary mirror when you need them later on in the pipeline.  At the moment
+the only viable use-case for bootstrapping is to install compilers.
+
+Here's an example of what bootstrapping some compilers might look like:
+
+.. code-block:: yaml
+
+   spack:
+     definitions:
+     - compiler-pkgs:
+       - 'llvm+clang@6.0.1 os=centos7'
+       - 'gcc@6.5.0 os=centos7'
+       - 'llvm+clang@6.0.1 os=ubuntu18.04'
+       - 'gcc@6.5.0 os=ubuntu18.04'
+     - pkgs:
+       - readline@7.0
+     - compilers:
+       - '%gcc@5.5.0'
+       - '%gcc@6.5.0'
+       - '%gcc@7.3.0'
+       - '%clang@6.0.0'
+       - '%clang@6.0.1'
+     - oses:
+       - os=ubuntu18.04
+       - os=centos7
+     specs:
+     - matrix:
+       - [$pkgs]
+       - [$compilers]
+       - [$oses]
+       exclude:
+         - '%gcc@7.3.0 os=centos7'
+         - '%gcc@5.5.0 os=ubuntu18.04'
+     gitlab-ci:
+       bootstrap:
+         - name: compiler-pkgs
+           compiler-agnostic: true
+       mappings:
+         # mappings similar to the example higher up in this description
+         ...
+
+In the example above, we have added a list to the ``definitions`` called
+``compiler-pkgs`` (you can add any number of these), which lists compiler packages
+we want to be staged ahead of the full matrix of release specs (which consists
+only of readline in our example).  Then within the ``gitlab-ci`` section, we
+have added a ``bootstrap`` section, which can contain a list of items, each
+referring to a list in the ``definitions`` section.  These items can either
+be a dictionary or a string.  If you supply a dictionary, it must have a name
+key whose value must match one of the lists in definitions and it can have a
+``compiler-agnostic`` key whose value is a boolean.  If you supply a string,
+then it needs to match one of the lists provided in ``definitions``.  You can
+think of the bootstrap list as an ordered list of pipeline "phases" that will
+be staged before your actual release specs.  While this introduces another
+layer of bottleneck in the pipeline (all jobs in all stages of one phase must
+complete before any jobs in the next phase can begin), it also means you are
+guaranteed your bootstrapped compilers will be available when you need them.
+
+The ``compiler-agnostic`` key can be provided with each item in the
+bootstrap list. It tells the ``spack ci generate`` command that any jobs staged
+from that particular list should have the compiler removed from the spec, so
+that any compiler available on the runner where the job is run can be used to
+build the package.
+
+When including a bootstrapping phase as in the example above, the result is that
+the bootstrapped compiler packages will be pushed to the binary mirror (and the
+local artifacts mirror) before the actual release specs are built. In this case,
+the jobs corresponding to subsequent release specs are configured to
+``install_missing_compilers``, so that if spack is asked to install a package
+with a compiler it doesn't know about, it can be quickly installed from the
+binary mirror first.
+
+Since bootstrapping compilers is optional, those items can be left out of the
+environment/stack file, and in that case no bootstrapping will be done (only the
+specs will be staged for building) and the runners will be expected to already
+have all needed compilers installed and configured for spack to use.
+
+-------------------------------------
+Using a custom spack in your pipeline
+-------------------------------------
+
+If your runners will not have a version of spack ready to invoke, or if for some
+other reason you want to use a custom version of spack to run your pipelines,
+this can be accomplished fairly simply.  First, create CI environment variables
+containing the url and branch/tag you want to clone (calling them, for example,
+``SPACK_REPO`` and ``SPACK_REF``), use them to clone spack in your pre-ci
+``before_script``, and finally pass those same values along to the workload
+generation process via the ``spack-repo`` and ``spack-ref`` cli args.  Here's
+the ``generate-pipeline`` job from the top of this document, updated to clone
+a custom spack and make sure the generated rebuild jobs will clone it too:
+
+.. code-block:: yaml
+
+   generate-pipeline:
+     tags:
+       - <some-other-tag>
+   before_script:
+     - git clone ${SPACK_REPO} --branch ${SPACK_REF}
+     - . ./spack/share/spack/setup-env.sh
+   script:
+     - spack env activate .
+     - spack ci generate
+       --spack-repo ${SPACK_REPO} --spack-ref ${SPACK_REF}
+       --output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/pipeline.yml"
+   after_script:
+     - rm -rf ./spack
+   artifacts:
+     paths:
+       - "${CI_PROJECT_DIR}/jobs_scratch_dir/pipeline.yml"
+
+
+If the ``spack ci generate`` command receives those extra command line arguments,
+then it adds similar ``before_script`` and ``after_script`` sections for each of
+the ``spack ci rebuild`` jobs it generates (cloning and sourcing a custom
+spack in the ``before_script`` and removing it again in the ``after_script``).
+This gives you control over the version of spack used when the rebuild jobs
+are actually run on the gitlab runner.
+
+.. _ci_environment_variables:
+
+--------------------------------------------------
+Environment variables affecting pipeline operation
+--------------------------------------------------
+
+Certain secrets and some other information should be provided to the pipeline
+infrastructure via environment variables, usually for reasons of security, but
+in some cases to support other pipeline use cases such as PR testing.  The
+environment variables used by the pipeline infrastructure are described here.
+
+^^^^^^^^^^^^^^^^^
+AWS_ACCESS_KEY_ID
+^^^^^^^^^^^^^^^^^
+
+Needed when binary mirror is an S3 bucket.
+
+^^^^^^^^^^^^^^^^^^^^^
+AWS_SECRET_ACCESS_KEY
+^^^^^^^^^^^^^^^^^^^^^
+
+Needed when binary mirror is an S3 bucket.
+
+^^^^^^^^^^^^^^^
+S3_ENDPOINT_URL
+^^^^^^^^^^^^^^^
+
+Needed when binary mirror is an S3 bucket that is *not* on AWS.
+
+^^^^^^^^^^^^^^^^^
+CDASH_AUTH_TOKEN
+^^^^^^^^^^^^^^^^^
+
+Needed in order to report build groups to CDash.
+
+^^^^^^^^^^^^^^^^^
+SPACK_SIGNING_KEY
+^^^^^^^^^^^^^^^^^
+
+Needed to sign/verify binary packages from the remote binary mirror.

lib/spack/docs/repositories.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -280,16 +280,16 @@ you install it, you can use ``spack spec -N``:
 
    Concretized
    --------------------------------
-   builtin.hdf5@1.10.0-patch1%clang@7.0.2-apple+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=darwin-elcapitan-x86_64
-       ^builtin.openmpi@2.0.1%clang@7.0.2-apple~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm~verbs+vt arch=darwin-elcapitan-x86_64
-           ^builtin.hwloc@1.11.4%clang@7.0.2-apple arch=darwin-elcapitan-x86_64
-               ^builtin.libpciaccess@0.13.4%clang@7.0.2-apple arch=darwin-elcapitan-x86_64
-                   ^builtin.libtool@2.4.6%clang@7.0.2-apple arch=darwin-elcapitan-x86_64
-                       ^builtin.m4@1.4.17%clang@7.0.2-apple+sigsegv arch=darwin-elcapitan-x86_64
-                           ^builtin.libsigsegv@2.10%clang@7.0.2-apple arch=darwin-elcapitan-x86_64
-                   ^builtin.pkg-config@0.29.1%clang@7.0.2-apple+internal_glib arch=darwin-elcapitan-x86_64
-                   ^builtin.util-macros@1.19.0%clang@7.0.2-apple arch=darwin-elcapitan-x86_64
-       ^builtin.zlib@1.2.8%clang@7.0.2-apple+pic arch=darwin-elcapitan-x86_64
+   builtin.hdf5@1.10.0-patch1%apple-clang@7.0.2+cxx~debug+fortran+mpi+shared~szip~threadsafe arch=darwin-elcapitan-x86_64
+       ^builtin.openmpi@2.0.1%apple-clang@7.0.2~mxm~pmi~psm~psm2~slurm~sqlite3~thread_multiple~tm~verbs+vt arch=darwin-elcapitan-x86_64
+           ^builtin.hwloc@1.11.4%apple-clang@7.0.2 arch=darwin-elcapitan-x86_64
+               ^builtin.libpciaccess@0.13.4%apple-clang@7.0.2 arch=darwin-elcapitan-x86_64
+                   ^builtin.libtool@2.4.6%apple-clang@7.0.2 arch=darwin-elcapitan-x86_64
+                       ^builtin.m4@1.4.17%apple-clang@7.0.2+sigsegv arch=darwin-elcapitan-x86_64
+                           ^builtin.libsigsegv@2.10%apple-clang@7.0.2 arch=darwin-elcapitan-x86_64
+                   ^builtin.pkg-config@0.29.1%apple-clang@7.0.2+internal_glib arch=darwin-elcapitan-x86_64
+                   ^builtin.util-macros@1.19.0%apple-clang@7.0.2 arch=darwin-elcapitan-x86_64
+       ^builtin.zlib@1.2.8%apple-clang@7.0.2+pic arch=darwin-elcapitan-x86_64
 
 .. warning::
 

lib/spack/docs/requirements.txt

@@ -1,7 +1,7 @@
 # These dependencies should be installed using pip in order
 # to build the documentation.
 
-sphinx==2.0.1
-sphinxcontrib-programoutput==0.14
-sphinx-rtd-theme==0.4.3
+sphinx
+sphinxcontrib-programoutput
+sphinx-rtd-theme
 python-levenshtein

lib/spack/docs/spack.yaml

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/docs/workflows.rst

@@ -1,4 +1,4 @@
-.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+.. Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
    Spack Project Developers. See the top-level COPYRIGHT file for details.
 
    SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -253,14 +253,14 @@ However, other more powerful methods are generally preferred for user
 environments.
 
 
-^^^^^^^^^^^^^^^^^^^^^^^
-Spack-Generated Modules
-^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Using ``spack load`` to Manage the User Environment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Suppose that Spack has been used to install a set of command-line
 programs, which users now wish to use.  One can in principle put a
 number of ``spack load`` commands into ``.bashrc``, for example, to
-load a set of Spack-generated modules:
+load a set of Spack packages:
 
 .. code-block:: sh
 
@@ -273,7 +273,7 @@ load a set of Spack-generated modules:
 Although simple load scripts like this are useful in many cases, they
 have some drawbacks:
 
-1. The set of modules loaded by them will in general not be
+1. The set of packages loaded by them will in general not be
    consistent.  They are a decent way to load commands to be called
    from command shells.  See below for better ways to assemble a
    consistent set of packages for building application programs.
@@ -284,20 +284,27 @@ have some drawbacks:
    The ``spack load`` and ``spack module tcl loads`` commands, on the
    other hand, are not very smart: if the user-supplied spec matches
    more than one installed package, then ``spack module tcl loads`` will
-   fail. This may change in the future.  For now, the workaround is to
-   be more specific on any ``spack module tcl loads`` lines that fail.
+   fail. This default behavior may change in the future.  For now,
+   the workaround is to either be more specific on any failing ``spack load``
+   commands or to use ``spack load --first`` to allow spack to load the
+   first matching spec.
 
 
 """"""""""""""""""""""
 Generated Load Scripts
 """"""""""""""""""""""
 
-Another problem with using `spack load` is, it is slow; a typical user
-environment could take several seconds to load, and would not be
-appropriate to put into ``.bashrc`` directly.  It is preferable to use
-a series of ``spack module tcl loads`` commands to pre-compute which
-modules to load.  These can be put in a script that is run whenever
-installed Spack packages change.  For example:
+Another problem with using `spack load` is, it can be slow; a typical
+user environment could take several seconds to load, and would not be
+appropriate to put into ``.bashrc`` directly.  This is because it
+requires the full start-up overhead of python/Spack for each command.
+In some circumstances it is preferable to use a series of ``spack
+module tcl loads`` (or ``spack module lmod loads``) commands to
+pre-compute which modules to load.  This will generate the modulenames
+to load the packages using environment modules, rather than Spack's
+built-in support for environment modifications. These can be put in a
+script that is run whenever installed Spack packages change.  For
+example:
 
 .. code-block:: sh
 
@@ -439,7 +446,7 @@ environment.
 
 A single-prefix filesystem view is a single directory tree that is the
 union of the directory hierarchies of a number of installed packages;
-it is similar to the directory hiearchy that might exist under
+it is similar to the directory hierarchy that might exist under
 ``/usr/local``.  The files of the view's installed packages are
 brought into the view by symbolic or hard links, referencing the
 original Spack installation.
@@ -634,7 +641,7 @@ Global Activations
 Python (and similar systems) packages directly or creating a view.
 If extensions are globally activated, then ``spack load python`` will
 also load all the extensions activated for the given ``python``.
-This reduces the need for users to load a large number of modules.
+This reduces the need for users to load a large number of packages.
 
 However, Spack global activations have two potential drawbacks:
 
@@ -1090,6 +1097,248 @@ or filesystem views.  However, it has some drawbacks:
    integrate Spack explicitly in their workflow.  Not all users are
    willing to do this.
 
+-------------------------------------
+Using Spack to Replace Homebrew/Conda
+-------------------------------------
+
+Spack is an incredibly powerful package manager, designed for supercomputers
+where users have diverse installation needs. But Spack can also be used to
+handle simple single-user installations on your laptop. Most macOS users are
+already familiar with package managers like Homebrew and Conda, where all
+installed packages are symlinked to a single central location like ``/usr/local``.
+In this section, we will show you how to emulate the behavior of Homebrew/Conda
+using :ref:`environments`!
+
+^^^^^
+Setup
+^^^^^
+
+First, let's create a new environment. We'll assume that Spack is already set up
+correctly, and that you've already sourced the setup script for your shell.
+To create a new environment, simply run:
+
+.. code-block:: console
+
+   $ spack env create myenv
+   ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
+   ==> Created environment 'myenv' in /Users/me/spack/var/spack/environments/myenv
+   $ spack env activate myenv
+
+Here, *myenv* can be anything you want to name your environment. Next, we can add
+a list of packages we would like to install into our environment. Let's say we
+want a newer version of Bash than the one that comes with macOS, and we want a
+few Python libraries. We can run:
+
+.. code-block:: console
+
+   $ spack add bash
+   ==> Adding bash to environment myenv
+   ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
+   $ spack add python@3:
+   ==> Adding python@3: to environment myenv
+   ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
+   $ spack add py-numpy py-scipy py-matplotlib
+   ==> Adding py-numpy to environment myenv
+   ==> Adding py-scipy to environment myenv
+   ==> Adding py-matplotlib to environment myenv
+   ==> Updating view at /Users/me/spack/var/spack/environments/myenv/.spack-env/view
+
+Each package can be listed on a separate line, or combined into a single line.
+Notice that we're explicitly asking for Python 3 here. You can use any spec
+you would normally use on the command line with other Spack commands.
+
+Next, we want to manually configure a couple of things. In the ``myenv``
+directory, we can find the ``spack.yaml`` that actually defines our environment.
+
+.. code-block:: console
+
+   $ vim ~/spack/var/spack/environments/myenv/spack.yaml
+
+.. code-block:: yaml
+
+   # This is a Spack Environment file.
+   #
+   # It describes a set of packages to be installed, along with
+   # configuration settings.
+   spack:
+     # add package specs to the `specs` list
+     specs: [bash, 'python@3:', py-numpy, py-scipy, py-matplotlib]
+     view:
+       default:
+         root: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
+         projections: {}
+     config: {}
+     mirrors: {}
+     modules:
+       enable: []
+     packages: {}
+     repos: []
+     upstreams: {}
+     definitions: []
+     concretization: separately
+
+You can see the packages we added earlier in the ``specs:`` section. If you
+ever want to add more packages, you can either use ``spack add`` or manually
+edit this file.
+
+We also need to change the ``concretization:`` option. By default, Spack
+concretizes each spec *separately*, allowing multiple versions of the same
+package to coexist. Since we want a single consistent environment, we want to
+concretize all of the specs *together*.
+
+Here is what your ``spack.yaml`` looks like with these new settings, and with
+some of the sections we don't plan on using removed:
+
+.. code-block:: diff
+
+   spack:
+   -  specs: [bash, 'python@3:', py-numpy, py-scipy, py-matplotlib]
+   +  specs:
+   +  - bash
+   +  - 'python@3:'
+   +  - py-numpy
+   +  - py-scipy
+   +  - py-matplotlib
+   -  view:
+   -    default:
+   -      root: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
+   -      projections: {}
+   +  view: /Users/me/spack/var/spack/environments/myenv/.spack-env/view
+   -  config: {}
+   -  mirrors: {}
+   -  modules:
+   -    enable: []
+   -  packages: {}
+   -  repos: []
+   -  upstreams: {}
+   -  definitions: []
+   +  concretization: together
+   -  concretization: separately
+
+""""""""""""""""
+Symlink location
+""""""""""""""""
+
+In the ``spack.yaml`` file above, you'll notice that by default, Spack symlinks
+all installations to ``/Users/me/spack/var/spack/environments/myenv/.spack-env/view``.
+You can actually change this to any directory you want. For example, Homebrew
+uses ``/usr/local``, while Conda uses ``/Users/me/anaconda``. In order to access
+files in these locations, you need to update ``PATH`` and other environment variables
+to point to them. Activating the Spack environment does this automatically, but
+you can also manually set them in your ``.bashrc``.
+
+.. warning::
+
+   There are several reasons why you shouldn't use ``/usr/local``:
+
+   1. If you are on macOS 10.11+ (El Capitan and newer), Apple makes it hard
+      for you. You may notice permissions issues on ``/usr/local`` due to their
+      `System Integrity Protection <https://support.apple.com/en-us/HT204899>`_.
+      By default, users don't have permissions to install anything in ``/usr/local``,
+      and you can't even change this using ``sudo chown`` or ``sudo chmod``.
+   2. Other package managers like Homebrew will try to install things to the
+      same directory. If you plan on using Homebrew in conjunction with Spack,
+      don't symlink things to ``/usr/local``.
+   3. If you are on a shared workstation, or don't have sudo privileges, you
+      can't do this.
+
+   If you still want to do this anyway, there are several ways around SIP.
+   You could disable SIP by booting into recovery mode and running
+   ``csrutil disable``, but this is not recommended, as it can open up your OS
+   to security vulnerabilities. Another technique is to run ``spack concretize``
+   and ``spack install`` using ``sudo``. This is also not recommended.
+
+   The safest way I've found is to create your installation directories using
+   sudo, then change ownership back to the user like so:
+
+   .. code-block:: bash
+
+      for directory in .spack bin contrib include lib man share
+      do
+          sudo mkdir -p /usr/local/$directory
+          sudo chown $(id -un):$(id -gn) /usr/local/$directory
+      done
+
+   Depending on the packages you install in your environment, the exact list of
+   directories you need to create may vary. You may also find some packages
+   like Java libraries that install a single file to the installation prefix
+   instead of in a subdirectory. In this case, the action is the same, just replace
+   ``mkdir -p`` with ``touch`` in the for-loop above.
+
+   But again, it's safer just to use the default symlink location.
+
+
+^^^^^^^^^^^^
+Installation
+^^^^^^^^^^^^
+
+To actually concretize the environment, run:
+
+.. code-block:: console
+
+   $ spack concretize
+
+This will tell you which if any packages are already installed, and alert you
+to any conflicting specs.
+
+To actually install these packages and symlink them to your ``view:``
+directory, simply run:
+
+.. code-block:: console
+
+   $ spack install
+
+Now, when you type ``which python3``, it should find the one you just installed.
+
+In order to change the default shell to our newer Bash installation, we first
+need to add it to this list of acceptable shells. Run:
+
+.. code-block:: console
+
+   $ sudo vim /etc/shells
+
+and add the absolute path to your bash executable. Then run:
+
+.. code-block:: console
+
+   $ chsh -s /path/to/bash
+
+Now, when you log out and log back in, ``echo $SHELL`` should point to the
+newer version of Bash.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Updating Installed Packages
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Let's say you upgraded to a new version of macOS, or a new version of Python
+was released, and you want to rebuild your entire software stack. To do this,
+simply run the following commands:
+
+.. code-block:: console
+
+   $ spack env activate myenv
+   $ spack concretize --force
+   $ spack install
+
+The ``--force`` flag tells Spack to overwrite its previous concretization
+decisions, allowing you to choose a new version of Python. If any of the new
+packages like Bash are already installed, ``spack install`` won't re-install
+them, it will keep the symlinks in place.
+
+^^^^^^^^^^^^^^
+Uninstallation
+^^^^^^^^^^^^^^
+
+If you decide that Spack isn't right for you, uninstallation is simple.
+Just run:
+
+.. code-block:: console
+
+   $ spack env activate myenv
+   $ spack uninstall --all
+
+This will uninstall all packages in your environment and remove the symlinks.
+
 ------------------------
 Using Spack on Travis-CI
 ------------------------
@@ -1109,6 +1358,14 @@ The main points that are implemented below:
    the spack builds in the config.
    (The Travis yaml parser is a bit buggy on the echo command.)
 
+#. Without control for the user, Travis jobs will run on various
+   ``x86_64`` microarchitectures. If you plan to cache build results,
+   e.g. to accelerate dependency builds, consider building for the
+   generic ``x86_64`` target only.
+   Limiting the microarchitecture will also find more packages when
+   working with the
+   `E4S Spack build cache <https://oaciss.uoregon.edu/e4s/e4s_buildcache_inventory.html>`_.
+
 #. Builds over 10 minutes need to be prefixed with ``travis_wait``.
    Alternatively, generate output once with ``spack install -v``.
 
@@ -1151,7 +1408,9 @@ The main points that are implemented below:
      - if ! which spack >/dev/null; then
          mkdir -p $SPACK_ROOT &&
          git clone --depth 50 https://github.com/spack/spack.git $SPACK_ROOT &&
-         echo -e "config:""\n  build_jobs:"" 2" > $SPACK_ROOT/etc/spack/config.yaml;
+         echo -e "config:""\n  build_jobs:"" 2" > $SPACK_ROOT/etc/spack/config.yaml **
+         echo -e "packages:""\n  all:""\n    target:"" ['x86_64']"
+                 > $SPACK_ROOT/etc/spack/packages.yaml;
        fi
      - travis_wait spack install cmake@3.7.2~openssl~ncurses
      - travis_wait spack install boost@1.62.0~graph~iostream~locale~log~wave
@@ -1183,12 +1442,7 @@ The following functionality is prepared:
 
 #. Base image: the example starts from a minimal ubuntu.
 
-#. Installing as root: docker images are usually set up as root.
-   Since some autotools scripts might complain about this being unsafe, we set
-   ``FORCE_UNSAFE_CONFIGURE=1`` to avoid configure errors.
-
-#. Pre-install the spack dependencies, including modules from the packages.
-   This avoids needing to build those from scratch via ``spack bootstrap``.
+#. Pre-install the spack dependencies.
    Package installs are followed by a clean-up of the system package index,
    to avoid outdated information and it saves space.
 
@@ -1217,10 +1471,9 @@ In order to build and run the image, execute:
 
    # general environment for docker
    ENV        DEBIAN_FRONTEND=noninteractive \
-              SPACK_ROOT=/usr/local \
-              FORCE_UNSAFE_CONFIGURE=1
+              SPACK_ROOT=/usr/local
 
-   # install minimal spack depedencies
+   # install minimal spack dependencies
    RUN        apt-get update \
               && apt-get install -y --no-install-recommends \
                  autoconf \
@@ -1254,7 +1507,7 @@ In order to build and run the image, execute:
    RUN        spack install tar \
               && spack clean -a
 
-   # need the modules already during image build?
+   # need the executables from a package already during image build?
    #RUN        /bin/bash -l -c ' \
    #                spack load tar \
    #                && which tar'

lib/spack/env/cc

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -15,9 +15,9 @@
 # 1. It allows Spack to swap compilers into and out of builds easily.
 # 2. It adds several options to the compile line so that spack
 #    packages can find their dependencies at build time and run time:
-#      -I           arguments for dependency /include directories.
-#      -L           arguments for dependency /lib directories.
-#      -Wl,-rpath   arguments for dependency /lib directories.
+#      -I and/or -isystem arguments for dependency /include directories.
+#      -L                 arguments for dependency /lib directories.
+#      -Wl,-rpath         arguments for dependency /lib directories.
 #
 
 # This is an array of environment variables that need to be set before
@@ -43,7 +43,7 @@ parameters=(
 # The compiler input variables are checked for sanity later:
 #   SPACK_CC, SPACK_CXX, SPACK_F77, SPACK_FC
 # The default compiler flags are passed from these variables:
-#   SPACK_CFLAGS, SPACK_CXXFLAGS, SPACK_FCFLAGS, SPACK_FFLAGS,
+#   SPACK_CFLAGS, SPACK_CXXFLAGS, SPACK_FFLAGS,
 #   SPACK_LDFLAGS, SPACK_LDLIBS
 # Debug env var is optional; set to "TRUE" for debug logging:
 #   SPACK_DEBUG
@@ -251,10 +251,11 @@ input_command="$*"
 #
 # Parse the command line arguments.
 #
-# We extract -L, -I, and -Wl,-rpath arguments from the command line and
-# recombine them with Spack arguments later.  We parse these out so that
-# we can make sure that system paths come last, that package arguments
-# come first, and that Spack arguments are injected properly.
+# We extract -L, -I, -isystem and -Wl,-rpath arguments from the
+# command line and recombine them with Spack arguments later.  We
+# parse these out so that we can make sure that system paths come
+# last, that package arguments come first, and that Spack arguments
+# are injected properly.
 #
 # All other arguments, including -l arguments, are treated as
 # 'other_args' and left in their original order.  This ensures that
@@ -273,12 +274,24 @@ system_libdirs=()
 system_rpaths=()
 libs=()
 other_args=()
+isystem_system_includes=()
+isystem_includes=()
 
 while [ -n "$1" ]; do
     # an RPATH to be added after the case statement.
     rp=""
 
     case "$1" in
+        -isystem*)
+            arg="${1#-isystem}"
+	    isystem_was_used=true
+            if [ -z "$arg" ]; then shift; arg="$1"; fi
+            if system_dir "$arg"; then
+                isystem_system_includes+=("$arg")
+            else
+                isystem_includes+=("$arg")
+            fi
+            ;;
         -I*)
             arg="${1#-I}"
             if [ -z "$arg" ]; then shift; arg="$1"; fi
@@ -425,12 +438,6 @@ then
     esac
 fi
 
-# Prepend include directories
-IFS=':' read -ra include_dirs <<< "$SPACK_INCLUDE_DIRS"
-if [[ $mode == cpp || $mode == cc || $mode == as || $mode == ccld ]]; then
-    includes=("${includes[@]}" "${include_dirs[@]}")
-fi
-
 IFS=':' read -ra rpath_dirs <<< "$SPACK_RPATH_DIRS"
 if [[ $mode == ccld || $mode == ld ]]; then
 
@@ -481,9 +488,22 @@ args=()
 # flags assembled earlier
 args+=("${flags[@]}")
 
-# include directory search paths
+# Insert include directories just prior to any system include directories
+
 for dir in "${includes[@]}";         do args+=("-I$dir"); done
+for dir in "${isystem_includes[@]}";         do args+=("-isystem$dir"); done
+
+IFS=':' read -ra spack_include_dirs <<< "$SPACK_INCLUDE_DIRS"
+if [[ $mode == cpp || $mode == cc || $mode == as || $mode == ccld ]]; then
+    if [[ "$isystem_was_used" == "true" ]] ; then
+	for dir in "${spack_include_dirs[@]}";  do args+=("-isystem$dir"); done
+    else
+	for dir in "${spack_include_dirs[@]}";  do args+=("-I$dir"); done
+    fi
+fi
+
 for dir in "${system_includes[@]}";  do args+=("-I$dir"); done
+for dir in "${isystem_system_includes[@]}";  do args+=("-isystem$dir"); done
 
 # Library search paths
 for dir in "${libdirs[@]}";          do args+=("-L$dir"); done

lib/spack/external/__init__.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -82,24 +82,17 @@
   ini-parsing, io, code, and log facilities.
 * Version: 1.4.34 (last version supporting Python 2.6)
 
-pyqver
-------
-
-* Homepage: https://github.com/ghewgill/pyqver
-* Usage: External script to query required python version of
-  python source code. Used for ensuring 2.6 compatibility.
-* Version: Unversioned
-
 pytest
 ------
 
 * Homepage: https://pypi.python.org/pypi/pytest
 * Usage: Testing framework used by Spack.
 * Version: 3.2.5 (last version supporting Python 2.6)
-* Note: This package has been slightly modified to improve
-  Python 2.6 compatibility. See the following commit if the
-  vendored copy ever needs to be updated again:
-  https://github.com/spack/spack/pull/6801/commits/ff513c39f2c67ff615de5cbc581dd69a8ec96526
+* Note: This package has been slightly modified:
+  * We improve Python 2.6 compatibility. See:
+    https://github.com/spack/spack/pull/6801.
+  * We have patched pytest not to depend on setuptools. See:
+    https://github.com/spack/spack/pull/15612
 
 ruamel.yaml
 ------
@@ -133,4 +126,5 @@
 * Homepage: https://altgraph.readthedocs.io/en/latest/index.html
 * Usage: dependency of macholib
 * Version: 0.16.1
+
 """

lib/spack/external/_pytest/config.py

@@ -1028,34 +1028,13 @@ def _consider_importhook(self, args):
             except SystemError:
                 mode = 'plain'
             else:
-                self._mark_plugins_for_rewrite(hook)
+                # REMOVED FOR SPACK: This routine imports `pkg_resources` from
+                # `setuptools`, but we do not need it for Spack. We have removed
+                # it from Spack to avoid a dependency on setuptools.
+                # self._mark_plugins_for_rewrite(hook)
+                pass
         self._warn_about_missing_assertion(mode)
 
-    def _mark_plugins_for_rewrite(self, hook):
-        """
-        Given an importhook, mark for rewrite any top-level
-        modules or packages in the distribution package for
-        all pytest plugins.
-        """
-        import pkg_resources
-        self.pluginmanager.rewrite_hook = hook
-
-        # 'RECORD' available for plugins installed normally (pip install)
-        # 'SOURCES.txt' available for plugins installed in dev mode (pip install -e)
-        # for installed plugins 'SOURCES.txt' returns an empty list, and vice-versa
-        # so it shouldn't be an issue
-        metadata_files = 'RECORD', 'SOURCES.txt'
-
-        package_files = (
-            entry.split(',')[0]
-            for entrypoint in pkg_resources.iter_entry_points('pytest11')
-            for metadata in metadata_files
-            for entry in entrypoint.dist._get_metadata(metadata)
-        )
-
-        for name in _iter_rewritable_modules(package_files):
-            hook.mark_rewrite(name)
-
     def _warn_about_missing_assertion(self, mode):
         try:
             assert False
@@ -1081,7 +1060,12 @@ def _preparse(self, args, addopts=True):
         self._checkversion()
         self._consider_importhook(args)
         self.pluginmanager.consider_preparse(args)
-        self.pluginmanager.load_setuptools_entrypoints('pytest11')
+
+        # REMOVED FOR SPACK: This routine imports `pkg_resources` from
+        # `setuptools`, but we do not need it for Spack. We have removed
+        # it from Spack to avoid a dependency on setuptools.
+        # self.pluginmanager.load_setuptools_entrypoints('pytest11')
+
         self.pluginmanager.consider_env()
         self.known_args_namespace = ns = self._parser.parse_known_args(args, namespace=self.option.copy())
         if self.known_args_namespace.confcutdir is None and self.inifile:

lib/spack/external/_pytest/pytester.py

@@ -569,7 +569,7 @@ def syspathinsert(self, path=None):
     def _possibly_invalidate_import_caches(self):
         # invalidate caches if we can (py33 and above)
         try:
-            import importlib  # nopyqver
+            import importlib
         except ImportError:
             pass
         else:

lib/spack/external/_pytest/vendored_packages/pluggy.py

@@ -497,26 +497,6 @@ def check_pending(self):
                                 "unknown hook %r in plugin %r" %
                                 (name, hookimpl.plugin))
 
-    def load_setuptools_entrypoints(self, entrypoint_name):
-        """ Load modules from querying the specified setuptools entrypoint name.
-        Return the number of loaded plugins. """
-        from pkg_resources import (iter_entry_points, DistributionNotFound,
-                                   VersionConflict)
-        for ep in iter_entry_points(entrypoint_name):
-            # is the plugin registered or blocked?
-            if self.get_plugin(ep.name) or self.is_blocked(ep.name):
-                continue
-            try:
-                plugin = ep.load()
-            except DistributionNotFound:
-                continue
-            except VersionConflict as e:
-                raise PluginValidationError(
-                    "Plugin %r could not be loaded: %s!" % (ep.name, e))
-            self.register(plugin, name=ep.name)
-            self._plugin_distinfo.append((plugin, ep.dist))
-        return len(self._plugin_distinfo)
-
     def list_plugin_distinfo(self):
         """ return list of distinfo/plugin tuples for all setuptools registered
         plugins. """

lib/spack/external/altgraph/__init__.py

@@ -139,9 +139,12 @@
   @contributor: U{Reka Albert <http://www.phys.psu.edu/~ralbert/>}
 
 '''
-import pkg_resources
-__version__ = pkg_resources.require('altgraph')[0].version
-
+# import pkg_resources
+# __version__ = pkg_resources.require('altgraph')[0].version
+# pkg_resources is not finding the altgraph import despite the fact that it is in sys.path
+# there is no .dist-info or .egg-info for pkg_resources to query the version from
+# so it must be set manually
+__version__ = '0.16.1'
 
 class GraphError(ValueError):
     pass

lib/spack/external/distro.py

@@ -64,6 +64,7 @@
     'enterpriseenterprise': 'oracle',  # Oracle Enterprise Linux
     'redhatenterpriseworkstation': 'rhel',  # RHEL 6, 7 Workstation
     'redhatenterpriseserver': 'rhel',  # RHEL 6, 7 Server
+    'redhatenterprisecomputenode': 'rhel',  # RHEL 6 ComputeNode
 }
 
 #: Translation table for normalizing the distro ID derived from the file name

lib/spack/external/ordereddict_backport.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/external/pyqver2.py

@@ -1,344 +0,0 @@
-#!/usr/bin/env python
-#
-# pyqver2.py
-# by Greg Hewgill
-# https://github.com/ghewgill/pyqver
-#
-# This software is provided 'as-is', without any express or implied
-# warranty.  In no event will the author be held liable for any damages
-# arising from the use of this software.
-#
-# Permission is granted to anyone to use this software for any purpose,
-# including commercial applications, and to alter it and redistribute it
-# freely, subject to the following restrictions:
-#
-# 1. The origin of this software must not be misrepresented; you must not
-#    claim that you wrote the original software. If you use this software
-#    in a product, an acknowledgment in the product documentation would be
-#    appreciated but is not required.
-# 2. Altered source versions must be plainly marked as such, and must not be
-#    misrepresented as being the original software.
-# 3. This notice may not be removed or altered from any source distribution.
-#
-# Copyright (c) 2009-2013 Greg Hewgill http://hewgill.com
-#
-
-import compiler
-import platform
-import sys
-
-StandardModules = {
-    "__future__":       (2, 1),
-    "abc":              (2, 6),
-# skip argparse now that it's in lib/spack/external
-#    "argparse":         (2, 7),
-    "ast":              (2, 6),
-    "atexit":           (2, 0),
-    "bz2":              (2, 3),
-    "cgitb":            (2, 2),
-    "collections":      (2, 4),
-    "contextlib":       (2, 5),
-    "cookielib":        (2, 4),
-    "cProfile":         (2, 5),
-    "csv":              (2, 3),
-    "ctypes":           (2, 5),
-    "datetime":         (2, 3),
-    "decimal":          (2, 4),
-    "difflib":          (2, 1),
-    "DocXMLRPCServer":  (2, 3),
-    "dummy_thread":     (2, 3),
-    "dummy_threading":  (2, 3),
-    "email":            (2, 2),
-    "fractions":        (2, 6),
-    "functools":        (2, 5),
-    "future_builtins":  (2, 6),
-    "hashlib":          (2, 5),
-    "heapq":            (2, 3),
-    "hmac":             (2, 2),
-    "hotshot":          (2, 2),
-    "HTMLParser":       (2, 2),
-    "importlib":        (2, 7),
-    "inspect":          (2, 1),
-    "io":               (2, 6),
-    "itertools":        (2, 3),
-    "json":             (2, 6),
-    "logging":          (2, 3),
-    "modulefinder":     (2, 3),
-    "msilib":           (2, 5),
-    "multiprocessing":  (2, 6),
-    "netrc":            (1, 5, 2),
-    "numbers":          (2, 6),
-    "optparse":         (2, 3),
-    "ossaudiodev":      (2, 3),
-    "pickletools":      (2, 3),
-    "pkgutil":          (2, 3),
-    "platform":         (2, 3),
-    "pydoc":            (2, 1),
-    "runpy":            (2, 5),
-    "sets":             (2, 3),
-    "shlex":            (1, 5, 2),
-    "SimpleXMLRPCServer": (2, 2),
-    "spwd":             (2, 5),
-    "sqlite3":          (2, 5),
-    "ssl":              (2, 6),
-    "stringprep":       (2, 3),
-    "subprocess":       (2, 4),
-    "sysconfig":        (2, 7),
-    "tarfile":          (2, 3),
-    "textwrap":         (2, 3),
-    "timeit":           (2, 3),
-    "unittest":         (2, 1),
-    "uuid":             (2, 5),
-    "warnings":         (2, 1),
-    "weakref":          (2, 1),
-    "winsound":         (1, 5, 2),
-    "wsgiref":          (2, 5),
-    "xml.dom":          (2, 0),
-    "xml.dom.minidom":  (2, 0),
-    "xml.dom.pulldom":  (2, 0),
-    "xml.etree.ElementTree": (2, 5),
-    "xml.parsers.expat":(2, 0),
-    "xml.sax":          (2, 0),
-    "xml.sax.handler":  (2, 0),
-    "xml.sax.saxutils": (2, 0),
-    "xml.sax.xmlreader":(2, 0),
-    "xmlrpclib":        (2, 2),
-    "zipfile":          (1, 6),
-    "zipimport":        (2, 3),
-    "_ast":             (2, 5),
-    "_winreg":          (2, 0),
-}
-
-Functions = {
-    "all":                      (2, 5),
-    "any":                      (2, 5),
-    "collections.Counter":      (2, 7),
-    "collections.defaultdict":  (2, 5),
-    "collections.OrderedDict":  (2, 7),
-    "functools.total_ordering": (2, 7),
-    "enumerate":                (2, 3),
-    "frozenset":                (2, 4),
-    "itertools.compress":       (2, 7),
-    "math.erf":                 (2, 7),
-    "math.erfc":                (2, 7),
-    "math.expm1":               (2, 7),
-    "math.gamma":               (2, 7),
-    "math.lgamma":              (2, 7),
-    "memoryview":               (2, 7),
-    "next":                     (2, 6),
-    "os.getresgid":             (2, 7),
-    "os.getresuid":             (2, 7),
-    "os.initgroups":            (2, 7),
-    "os.setresgid":             (2, 7),
-    "os.setresuid":             (2, 7),
-    "reversed":                 (2, 4),
-    "set":                      (2, 4),
-    "subprocess.check_call":    (2, 5),
-    "subprocess.check_output":  (2, 7),
-    "sum":                      (2, 3),
-    "symtable.is_declared_global": (2, 7),
-    "weakref.WeakSet":          (2, 7),
-}
-
-Identifiers = {
-    "False":        (2, 2),
-    "True":         (2, 2),
-}
-
-def uniq(a):
-    if len(a) == 0:
-        return []
-    else:
-        return [a[0]] + uniq([x for x in a if x != a[0]])
-
-class NodeChecker(object):
-    def __init__(self):
-        self.vers = dict()
-        self.vers[(2,0)] = []
-    def add(self, node, ver, msg):
-        if ver not in self.vers:
-            self.vers[ver] = []
-        self.vers[ver].append((node.lineno, msg))
-    def default(self, node):
-        for child in node.getChildNodes():
-            self.visit(child)
-    def visitCallFunc(self, node):
-        def rollup(n):
-            if isinstance(n, compiler.ast.Name):
-                return n.name
-            elif isinstance(n, compiler.ast.Const):
-                return type(n.value).__name__
-            elif isinstance(n, compiler.ast.Getattr):
-                r = rollup(n.expr)
-                if r:
-                    return r + "." + n.attrname
-        name = rollup(node.node)
-        if name:
-            # Special handling for empty format strings, which aren't
-            # allowed in Python 2.6
-            if name in ('unicode.format', 'str.format'):
-                n = node.node
-                if isinstance(n, compiler.ast.Getattr):
-                    n = n.expr
-                    if isinstance(n, compiler.ast.Const):
-                        if '{}' in n.value:
-                            self.add(node, (2,7), name + ' with {} format string')
-
-            v = Functions.get(name)
-            if v is not None:
-                self.add(node, v, name)
-        self.default(node)
-    def visitClass(self, node):
-        if node.bases:
-            self.add(node, (2,2), "new-style class")
-        if node.decorators:
-            self.add(node, (2,6), "class decorator")
-        self.default(node)
-    def visitDictComp(self, node):
-        self.add(node, (2,7), "dictionary comprehension")
-        self.default(node)
-    def visitFloorDiv(self, node):
-        self.add(node, (2,2), "// operator")
-        self.default(node)
-    def visitFrom(self, node):
-        v = StandardModules.get(node.modname)
-        if v is not None:
-            self.add(node, v, node.modname)
-        for n in node.names:
-            name = node.modname + "." + n[0]
-            v = Functions.get(name)
-            if v is not None:
-                self.add(node, v, name)
-    def visitFunction(self, node):
-        if node.decorators:
-            self.add(node, (2,4), "function decorator")
-        self.default(node)
-    def visitGenExpr(self, node):
-        self.add(node, (2,4), "generator expression")
-        self.default(node)
-    def visitGetattr(self, node):
-        if (isinstance(node.expr, compiler.ast.Const)
-            and isinstance(node.expr.value, str)
-            and node.attrname == "format"):
-            self.add(node, (2,6), "string literal .format()")
-        self.default(node)
-    def visitIfExp(self, node):
-        self.add(node, (2,5), "inline if expression")
-        self.default(node)
-    def visitImport(self, node):
-        for n in node.names:
-            v = StandardModules.get(n[0])
-            if v is not None:
-                self.add(node, v, n[0])
-        self.default(node)
-    def visitName(self, node):
-        v = Identifiers.get(node.name)
-        if v is not None:
-            self.add(node, v, node.name)
-        self.default(node)
-    def visitSet(self, node):
-        self.add(node, (2,7), "set literal")
-        self.default(node)
-    def visitSetComp(self, node):
-        self.add(node, (2,7), "set comprehension")
-        self.default(node)
-    def visitTryFinally(self, node):
-        # try/finally with a suite generates a Stmt node as the body,
-        # but try/except/finally generates a TryExcept as the body
-        if isinstance(node.body, compiler.ast.TryExcept):
-            self.add(node, (2,5), "try/except/finally")
-        self.default(node)
-    def visitWith(self, node):
-        if isinstance(node.body, compiler.ast.With):
-            self.add(node, (2,7), "with statement with multiple contexts")
-        else:
-            self.add(node, (2,5), "with statement")
-        self.default(node)
-    def visitYield(self, node):
-        self.add(node, (2,2), "yield expression")
-        self.default(node)
-
-def get_versions(source, filename=None):
-    """Return information about the Python versions required for specific features.
-
-    The return value is a dictionary with keys as a version number as a tuple
-    (for example Python 2.6 is (2,6)) and the value are a list of features that
-    require the indicated Python version.
-    """
-    tree = compiler.parse(source)
-    checker = compiler.walk(tree, NodeChecker())
-    return checker.vers
-
-def v27(source):
-    if sys.version_info >= (2, 7):
-        return qver(source)
-    else:
-        print >>sys.stderr, "Not all features tested, run --test with Python 2.7"
-        return (2, 7)
-
-def qver(source):
-    """Return the minimum Python version required to run a particular bit of code.
-
-    >>> qver('print "hello world"')
-    (2, 0)
-    >>> qver('class test(object): pass')
-    (2, 2)
-    >>> qver('yield 1')
-    (2, 2)
-    >>> qver('a // b')
-    (2, 2)
-    >>> qver('True')
-    (2, 2)
-    >>> qver('enumerate(a)')
-    (2, 3)
-    >>> qver('total = sum')
-    (2, 0)
-    >>> qver('sum(a)')
-    (2, 3)
-    >>> qver('(x*x for x in range(5))')
-    (2, 4)
-    >>> qver('class C:\\n @classmethod\\n def m(): pass')
-    (2, 4)
-    >>> qver('y if x else z')
-    (2, 5)
-    >>> qver('import hashlib')
-    (2, 5)
-    >>> qver('from hashlib import md5')
-    (2, 5)
-    >>> qver('import xml.etree.ElementTree')
-    (2, 5)
-    >>> qver('try:\\n try: pass;\\n except: pass;\\nfinally: pass')
-    (2, 0)
-    >>> qver('try: pass;\\nexcept: pass;\\nfinally: pass')
-    (2, 5)
-    >>> qver('from __future__ import with_statement\\nwith x: pass')
-    (2, 5)
-    >>> qver('collections.defaultdict(list)')
-    (2, 5)
-    >>> qver('from collections import defaultdict')
-    (2, 5)
-    >>> qver('"{0}".format(0)')
-    (2, 6)
-    >>> qver('memoryview(x)')
-    (2, 7)
-    >>> v27('{1, 2, 3}')
-    (2, 7)
-    >>> v27('{x for x in s}')
-    (2, 7)
-    >>> v27('{x: y for x in s}')
-    (2, 7)
-    >>> qver('from __future__ import with_statement\\nwith x:\\n with y: pass')
-    (2, 5)
-    >>> v27('from __future__ import with_statement\\nwith x, y: pass')
-    (2, 7)
-    >>> qver('@decorator\\ndef f(): pass')
-    (2, 4)
-    >>> qver('@decorator\\nclass test:\\n pass')
-    (2, 6)
-
-    #>>> qver('0o0')
-    #(2, 6)
-    #>>> qver('@foo\\nclass C: pass')
-    #(2, 6)
-    """
-    return max(get_versions(source).keys())

lib/spack/external/pyqver3.py

@@ -1,248 +0,0 @@
-#!/usr/bin/env python3
-#
-# pyqver3.py
-# by Greg Hewgill
-# https://github.com/ghewgill/pyqver
-#
-# This software is provided 'as-is', without any express or implied
-# warranty.  In no event will the author be held liable for any damages
-# arising from the use of this software.
-#
-# Permission is granted to anyone to use this software for any purpose,
-# including commercial applications, and to alter it and redistribute it
-# freely, subject to the following restrictions:
-#
-# 1. The origin of this software must not be misrepresented; you must not
-#    claim that you wrote the original software. If you use this software
-#    in a product, an acknowledgment in the product documentation would be
-#    appreciated but is not required.
-# 2. Altered source versions must be plainly marked as such, and must not be
-#    misrepresented as being the original software.
-# 3. This notice may not be removed or altered from any source distribution.
-#
-# Copyright (c) 2009-2013 Greg Hewgill http://hewgill.com
-#
-import ast
-import platform
-import sys
-
-StandardModules = {
-# skip argparse now that it's in lib/spack/external
-#    "argparse":         (3, 2),
-    "faulthandler":     (3, 3),
-    "importlib":        (3, 1),
-    "ipaddress":        (3, 3),
-    "lzma":             (3, 3),
-    "tkinter.ttk":      (3, 1),
-    "unittest.mock":    (3, 3),
-    "venv":             (3, 3),
-}
-
-Functions = {
-    "bytearray.maketrans":                      (3, 1),
-    "bytes.maketrans":                          (3, 1),
-    "bz2.open":                                 (3, 3),
-    "collections.Counter":                      (3, 1),
-    "collections.OrderedDict":                  (3, 1),
-    "crypt.mksalt":                             (3, 3),
-    "email.generator.BytesGenerator":           (3, 2),
-    "email.message_from_binary_file":           (3, 2),
-    "email.message_from_bytes":                 (3, 2),
-    "functools.lru_cache":                      (3, 2),
-    "gzip.compress":                            (3, 2),
-    "gzip.decompress":                          (3, 2),
-    "inspect.getclosurevars":                   (3, 3),
-    "inspect.getgeneratorlocals":               (3, 3),
-    "inspect.getgeneratorstate":                (3, 2),
-    "itertools.combinations_with_replacement":  (3, 1),
-    "itertools.compress":                       (3, 1),
-    "logging.config.dictConfig":                (3, 2),
-    "logging.NullHandler":                      (3, 1),
-    "math.erf":                                 (3, 2),
-    "math.erfc":                                (3, 2),
-    "math.expm1":                               (3, 2),
-    "math.gamma":                               (3, 2),
-    "math.isfinite":                            (3, 2),
-    "math.lgamma":                              (3, 2),
-    "math.log2":                                (3, 3),
-    "os.environb":                              (3, 2),
-    "os.fsdecode":                              (3, 2),
-    "os.fsencode":                              (3, 2),
-    "os.fwalk":                                 (3, 3),
-    "os.getenvb":                               (3, 2),
-    "os.get_exec_path":                         (3, 2),
-    "os.getgrouplist":                          (3, 3),
-    "os.getpriority":                           (3, 3),
-    "os.getresgid":                             (3, 2),
-    "os.getresuid":                             (3, 2),
-    "os.get_terminal_size":                     (3, 3),
-    "os.getxattr":                              (3, 3),
-    "os.initgroups":                            (3, 2),
-    "os.listxattr":                             (3, 3),
-    "os.lockf":                                 (3, 3),
-    "os.pipe2":                                 (3, 3),
-    "os.posix_fadvise":                         (3, 3),
-    "os.posix_fallocate":                       (3, 3),
-    "os.pread":                                 (3, 3),
-    "os.pwrite":                                (3, 3),
-    "os.readv":                                 (3, 3),
-    "os.removexattr":                           (3, 3),
-    "os.replace":                               (3, 3),
-    "os.sched_get_priority_max":                (3, 3),
-    "os.sched_get_priority_min":                (3, 3),
-    "os.sched_getaffinity":                     (3, 3),
-    "os.sched_getparam":                        (3, 3),
-    "os.sched_getscheduler":                    (3, 3),
-    "os.sched_rr_get_interval":                 (3, 3),
-    "os.sched_setaffinity":                     (3, 3),
-    "os.sched_setparam":                        (3, 3),
-    "os.sched_setscheduler":                    (3, 3),
-    "os.sched_yield":                           (3, 3),
-    "os.sendfile":                              (3, 3),
-    "os.setpriority":                           (3, 3),
-    "os.setresgid":                             (3, 2),
-    "os.setresuid":                             (3, 2),
-    "os.setxattr":                              (3, 3),
-    "os.sync":                                  (3, 3),
-    "os.truncate":                              (3, 3),
-    "os.waitid":                                (3, 3),
-    "os.writev":                                (3, 3),
-    "shutil.chown":                             (3, 3),
-    "shutil.disk_usage":                        (3, 3),
-    "shutil.get_archive_formats":               (3, 3),
-    "shutil.get_terminal_size":                 (3, 3),
-    "shutil.get_unpack_formats":                (3, 3),
-    "shutil.make_archive":                      (3, 3),
-    "shutil.register_archive_format":           (3, 3),
-    "shutil.register_unpack_format":            (3, 3),
-    "shutil.unpack_archive":                    (3, 3),
-    "shutil.unregister_archive_format":         (3, 3),
-    "shutil.unregister_unpack_format":          (3, 3),
-    "shutil.which":                             (3, 3),
-    "signal.pthread_kill":                      (3, 3),
-    "signal.pthread_sigmask":                   (3, 3),
-    "signal.sigpending":                        (3, 3),
-    "signal.sigtimedwait":                      (3, 3),
-    "signal.sigwait":                           (3, 3),
-    "signal.sigwaitinfo":                       (3, 3),
-    "socket.CMSG_LEN":                          (3, 3),
-    "socket.CMSG_SPACE":                        (3, 3),
-    "socket.fromshare":                         (3, 3),
-    "socket.if_indextoname":                    (3, 3),
-    "socket.if_nameindex":                      (3, 3),
-    "socket.if_nametoindex":                    (3, 3),
-    "socket.sethostname":                       (3, 3),
-    "ssl.match_hostname":                       (3, 2),
-    "ssl.RAND_bytes":                           (3, 3),
-    "ssl.RAND_pseudo_bytes":                    (3, 3),
-    "ssl.SSLContext":                           (3, 2),
-    "ssl.SSLEOFError":                          (3, 3),
-    "ssl.SSLSyscallError":                      (3, 3),
-    "ssl.SSLWantReadError":                     (3, 3),
-    "ssl.SSLWantWriteError":                    (3, 3),
-    "ssl.SSLZeroReturnError":                   (3, 3),
-    "stat.filemode":                            (3, 3),
-    "textwrap.indent":                          (3, 3),
-    "threading.get_ident":                      (3, 3),
-    "time.clock_getres":                        (3, 3),
-    "time.clock_gettime":                       (3, 3),
-    "time.clock_settime":                       (3, 3),
-    "time.get_clock_info":                      (3, 3),
-    "time.monotonic":                           (3, 3),
-    "time.perf_counter":                        (3, 3),
-    "time.process_time":                        (3, 3),
-    "types.new_class":                          (3, 3),
-    "types.prepare_class":                      (3, 3),
-}
-
-def uniq(a):
-    if len(a) == 0:
-        return []
-    else:
-        return [a[0]] + uniq([x for x in a if x != a[0]])
-
-class NodeChecker(ast.NodeVisitor):
-    def __init__(self):
-        self.vers = dict()
-        self.vers[(3,0)] = []
-    def add(self, node, ver, msg):
-        if ver not in self.vers:
-            self.vers[ver] = []
-        self.vers[ver].append((node.lineno, msg))
-    def visit_Call(self, node):
-        def rollup(n):
-            if isinstance(n, ast.Name):
-                return n.id
-            elif isinstance(n, ast.Attribute):
-                r = rollup(n.value)
-                if r:
-                    return r + "." + n.attr
-        name = rollup(node.func)
-        if name:
-            v = Functions.get(name)
-            if v is not None:
-                self.add(node, v, name)
-        self.generic_visit(node)
-    def visit_Import(self, node):
-        for n in node.names:
-            v = StandardModules.get(n.name)
-            if v is not None:
-                self.add(node, v, n.name)
-        self.generic_visit(node)
-    def visit_ImportFrom(self, node):
-        v = StandardModules.get(node.module)
-        if v is not None:
-            self.add(node, v, node.module)
-        for n in node.names:
-            name = node.module + "." + n.name
-            v = Functions.get(name)
-            if v is not None:
-                self.add(node, v, name)
-    def visit_Raise(self, node):
-        if isinstance(node.cause, ast.Name) and node.cause.id == "None":
-            self.add(node, (3,3), "raise ... from None")
-    def visit_YieldFrom(self, node):
-        self.add(node, (3,3), "yield from")
-
-def get_versions(source, filename=None):
-    """Return information about the Python versions required for specific features.
-
-    The return value is a dictionary with keys as a version number as a tuple
-    (for example Python 3.1 is (3,1)) and the value are a list of features that
-    require the indicated Python version.
-    """
-    tree = ast.parse(source, filename=filename)
-    checker = NodeChecker()
-    checker.visit(tree)
-    return checker.vers
-
-def v33(source):
-    if sys.version_info >= (3, 3):
-        return qver(source)
-    else:
-        print("Not all features tested, run --test with Python 3.3", file=sys.stderr)
-        return (3, 3)
-
-def qver(source):
-    """Return the minimum Python version required to run a particular bit of code.
-
-    >>> qver('print("hello world")')
-    (3, 0)
-    >>> qver("import importlib")
-    (3, 1)
-    >>> qver("from importlib import x")
-    (3, 1)
-    >>> qver("import tkinter.ttk")
-    (3, 1)
-    >>> qver("from collections import Counter")
-    (3, 1)
-    >>> qver("collections.OrderedDict()")
-    (3, 1)
-    >>> qver("import functools\\n@functools.lru_cache()\\ndef f(x): x*x")
-    (3, 2)
-    >>> v33("yield from x")
-    (3, 3)
-    >>> v33("raise x from None")
-    (3, 3)
-    """
-    return max(get_versions(source).keys())

lib/spack/external/ruamel/yaml/compat.py

@@ -12,7 +12,7 @@
     from ruamel.ordereddict import ordereddict
 except:
     try:
-        from collections import OrderedDict  # nopyqver
+        from collections import OrderedDict
     except ImportError:
         from ordereddict import OrderedDict
     # to get the right name import ... as ordereddict doesn't do that

lib/spack/llnl/__init__.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/llnl/util/__init__.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/llnl/util/argparsewriter.py

@@ -1,136 +1,165 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
 from __future__ import print_function
+
 import re
 import argparse
 import errno
 import sys
 
+from six import StringIO
 
-class ArgparseWriter(object):
+
+class Command(object):
+    """Parsed representation of a command from argparse.
+
+    This is a single command from an argparse parser. ``ArgparseWriter``
+    creates these and returns them from ``parse()``, and it passes one of
+    these to each call to ``format()`` so that we can take an action for
+    a single command.
+
+    Parts of a Command:
+      - prog: command name (str)
+      - description: command description (str)
+      - usage: command usage (str)
+      - positionals: list of positional arguments (list)
+      - optionals: list of optional arguments (list)
+      - subcommands: list of subcommand parsers (list)
+    """
+    def __init__(self, prog, description, usage,
+                 positionals, optionals, subcommands):
+        self.prog = prog
+        self.description = description
+        self.usage = usage
+        self.positionals = positionals
+        self.optionals = optionals
+        self.subcommands = subcommands
+
+
+# NOTE: The only reason we subclass argparse.HelpFormatter is to get access
+# to self._expand_help(), ArgparseWriter is not intended to be used as a
+# formatter_class.
+class ArgparseWriter(argparse.HelpFormatter):
     """Analyzes an argparse ArgumentParser for easy generation of help."""
-    def __init__(self, out=sys.stdout):
+
+    def __init__(self, prog, out=None, aliases=False):
+        """Initializes a new ArgparseWriter instance.
+
+        Parameters:
+            prog (str): the program name
+            out (file object): the file to write to (default sys.stdout)
+            aliases (bool): whether or not to include subparsers for aliases
+        """
+        super(ArgparseWriter, self).__init__(prog)
         self.level = 0
-        self.out = out
+        self.prog = prog
+        self.out = sys.stdout if out is None else out
+        self.aliases = aliases
 
-    def _write(self, parser, root=True, level=0):
+    def parse(self, parser, prog):
+        """Parses the parser object and returns the relavent components.
+
+        Parameters:
+            parser (argparse.ArgumentParser): the parser
+            prog (str): the command name
+
+        Returns:
+            (Command) information about the command from the parser
+        """
         self.parser = parser
-        self.level = level
+
+        split_prog = parser.prog.split(' ')
+        split_prog[-1] = prog
+        prog = ' '.join(split_prog)
+        description = parser.description
+
+        fmt = parser._get_formatter()
         actions = parser._actions
+        groups = parser._mutually_exclusive_groups
+        usage = fmt._format_usage(None, actions, groups, '').strip()
 
-        # allow root level to be flattened with rest of commands
-        if type(root) == int:
-            self.level = root
-            root = True
-
-        # go through actions and split them into optionals, positionals,
+        # Go through actions and split them into optionals, positionals,
         # and subcommands
         optionals = []
         positionals = []
         subcommands = []
         for action in actions:
             if action.option_strings:
-                optionals.append(action)
+                flags = action.option_strings
+                dest_flags = fmt._format_action_invocation(action)
+                help = self._expand_help(action) if action.help else ''
+                help = help.replace('\n', ' ')
+                optionals.append((flags, dest_flags, help))
             elif isinstance(action, argparse._SubParsersAction):
                 for subaction in action._choices_actions:
                     subparser = action._name_parser_map[subaction.dest]
-                    subcommands.append(subparser)
+                    subcommands.append((subparser, subaction.dest))
+
+                    # Look for aliases of the form 'name (alias, ...)'
+                    if self.aliases:
+                        match = re.match(r'(.*) \((.*)\)', subaction.metavar)
+                        if match:
+                            aliases = match.group(2).split(', ')
+                            for alias in aliases:
+                                subparser = action._name_parser_map[alias]
+                                subcommands.append((subparser, alias))
             else:
-                positionals.append(action)
+                args = fmt._format_action_invocation(action)
+                help = self._expand_help(action) if action.help else ''
+                help = help.replace('\n', ' ')
+                positionals.append((args, help))
 
-        groups = parser._mutually_exclusive_groups
-        fmt = parser._get_formatter()
-        description = parser.description
+        return Command(
+            prog, description, usage, positionals, optionals, subcommands)
 
-        def action_group(function, actions):
-            for action in actions:
-                arg = fmt._format_action_invocation(action)
-                help = action.help if action.help else ''
-                function(arg, re.sub('\n', ' ', help))
+    def format(self, cmd):
+        """Returns the string representation of a single node in the
+        parser tree.
 
-        if root:
-            self.begin_command(parser.prog)
+        Override this in subclasses to define how each subcommand
+        should be displayed.
 
-            if description:
-                self.description(parser.description)
+        Parameters:
+            (Command): parsed information about a command or subcommand
 
-            usage = fmt._format_usage(None, actions, groups, '').strip()
-            self.usage(usage)
+        Returns:
+            str: the string representation of this subcommand
+        """
+        raise NotImplementedError
 
-            if positionals:
-                self.begin_positionals()
-                action_group(self.positional, positionals)
-                self.end_positionals()
+    def _write(self, parser, prog, level=0):
+        """Recursively writes a parser.
 
-            if optionals:
-                self.begin_optionals()
-                action_group(self.optional, optionals)
-                self.end_optionals()
+        Parameters:
+            parser (argparse.ArgumentParser): the parser
+            prog (str): the command name
+            level (int): the current level
+        """
+        self.level = level
 
-        if subcommands:
-            self.begin_subcommands(subcommands)
-            for subparser in subcommands:
-                self._write(subparser, root=True, level=level + 1)
-            self.end_subcommands(subcommands)
+        cmd = self.parse(parser, prog)
+        self.out.write(self.format(cmd))
 
-        if root:
-            self.end_command(parser.prog)
+        for subparser, prog in cmd.subcommands:
+            self._write(subparser, prog, level=level + 1)
 
-    def write(self, parser, root=True):
+    def write(self, parser):
         """Write out details about an ArgumentParser.
 
         Args:
-            parser (ArgumentParser): an ``argparse`` parser
-            root (bool or int): if bool, whether to include the root parser;
-                or ``1`` to flatten the root parser with first-level
-                subcommands
+            parser (argparse.ArgumentParser): the parser
         """
         try:
-            self._write(parser, root, level=0)
+            self._write(parser, self.prog)
         except IOError as e:
-            # swallow pipe errors
+            # Swallow pipe errors
+            # Raises IOError in Python 2 and BrokenPipeError in Python 3
             if e.errno != errno.EPIPE:
                 raise
 
-    def begin_command(self, prog):
-        pass
-
-    def end_command(self, prog):
-        pass
-
-    def description(self, description):
-        pass
-
-    def usage(self, usage):
-        pass
-
-    def begin_positionals(self):
-        pass
-
-    def positional(self, name, help):
-        pass
-
-    def end_positionals(self):
-        pass
-
-    def begin_optionals(self):
-        pass
-
-    def optional(self, option, help):
-        pass
-
-    def end_optionals(self):
-        pass
-
-    def begin_subcommands(self, subcommands):
-        pass
-
-    def end_subcommands(self, subcommands):
-        pass
-
 
 _rst_levels = ['=', '-', '^', '~', ':', '`']
 
@@ -138,66 +167,214 @@ def end_subcommands(self, subcommands):
 class ArgparseRstWriter(ArgparseWriter):
     """Write argparse output as rst sections."""
 
-    def __init__(self, out=sys.stdout, rst_levels=_rst_levels,
-                 strip_root_prog=True):
+    def __init__(self, prog, out=None, aliases=False,
+                 rst_levels=_rst_levels):
         """Create a new ArgparseRstWriter.
 
-        Args:
+        Parameters:
+            prog (str): program name
             out (file object): file to write to
+            aliases (bool): whether or not to include subparsers for aliases
             rst_levels (list of str): list of characters
                 for rst section headings
-            strip_root_prog (bool): if ``True``, strip the base command name
-                from subcommands in output
         """
-        super(ArgparseRstWriter, self).__init__(out)
+        out = sys.stdout if out is None else out
+        super(ArgparseRstWriter, self).__init__(prog, out, aliases)
         self.rst_levels = rst_levels
-        self.strip_root_prog = strip_root_prog
 
-    def line(self, string=''):
-        self.out.write('%s\n' % string)
+    def format(self, cmd):
+        string = StringIO()
+        string.write(self.begin_command(cmd.prog))
+
+        if cmd.description:
+            string.write(self.description(cmd.description))
+
+        string.write(self.usage(cmd.usage))
+
+        if cmd.positionals:
+            string.write(self.begin_positionals())
+            for args, help in cmd.positionals:
+                string.write(self.positional(args, help))
+            string.write(self.end_positionals())
+
+        if cmd.optionals:
+            string.write(self.begin_optionals())
+            for flags, dest_flags, help in cmd.optionals:
+                string.write(self.optional(dest_flags, help))
+            string.write(self.end_optionals())
+
+        if cmd.subcommands:
+            string.write(self.begin_subcommands(cmd.subcommands))
+
+        return string.getvalue()
 
     def begin_command(self, prog):
-        self.line()
-        self.line('----')
-        self.line()
-        self.line('.. _%s:\n' % prog.replace(' ', '-'))
-        self.line('%s' % prog)
-        self.line(self.rst_levels[self.level] * len(prog) + '\n')
+        return """
+----
+
+.. _{0}:
+
+{1}
+{2}
+
+""".format(prog.replace(' ', '-'), prog,
+           self.rst_levels[self.level] * len(prog))
 
     def description(self, description):
-        self.line('%s\n' % description)
+        return description + '\n\n'
 
     def usage(self, usage):
-        self.line('.. code-block:: console\n')
-        self.line('    %s\n' % usage)
+        return """\
+.. code-block:: console
+
+    {0}
+
+""".format(usage)
 
     def begin_positionals(self):
-        self.line()
-        self.line('**Positional arguments**\n')
+        return '\n**Positional arguments**\n\n'
 
     def positional(self, name, help):
-        self.line(name)
-        self.line('  %s\n' % help)
+        return """\
+{0}
+  {1}
+
+""".format(name, help)
+
+    def end_positionals(self):
+        return ''
 
     def begin_optionals(self):
-        self.line()
-        self.line('**Optional arguments**\n')
+        return '\n**Optional arguments**\n\n'
 
     def optional(self, opts, help):
-        self.line('``%s``' % opts)
-        self.line('  %s\n' % help)
+        return """\
+``{0}``
+  {1}
+
+""".format(opts, help)
+
+    def end_optionals(self):
+        return ''
 
     def begin_subcommands(self, subcommands):
-        self.line()
-        self.line('**Subcommands**\n')
-        self.line('.. hlist::')
-        self.line('   :columns: 4\n')
+        string = """
+**Subcommands**
 
-        for cmd in subcommands:
-            prog = cmd.prog
-            if self.strip_root_prog:
-                prog = re.sub(r'^[^ ]* ', '', prog)
+.. hlist::
+   :columns: 4
 
-            self.line('   * :ref:`%s <%s>`'
-                      % (prog, cmd.prog.replace(' ', '-')))
-        self.line()
+"""
+
+        for cmd, _ in subcommands:
+            prog = re.sub(r'^[^ ]* ', '', cmd.prog)
+            string += '   * :ref:`{0} <{1}>`\n'.format(
+                prog, cmd.prog.replace(' ', '-'))
+
+        return string + '\n'
+
+
+class ArgparseCompletionWriter(ArgparseWriter):
+    """Write argparse output as shell programmable tab completion functions."""
+
+    def format(self, cmd):
+        """Returns the string representation of a single node in the
+        parser tree.
+
+        Override this in subclasses to define how each subcommand
+        should be displayed.
+
+        Parameters:
+            (Command): parsed information about a command or subcommand
+
+        Returns:
+            str: the string representation of this subcommand
+        """
+
+        assert cmd.optionals  # we should always at least have -h, --help
+        assert not (cmd.positionals and cmd.subcommands)  # one or the other
+
+        # We only care about the arguments/flags, not the help messages
+        positionals = []
+        if cmd.positionals:
+            positionals, _ = zip(*cmd.positionals)
+        optionals, _, _ = zip(*cmd.optionals)
+        subcommands = []
+        if cmd.subcommands:
+            _, subcommands = zip(*cmd.subcommands)
+
+        # Flatten lists of lists
+        optionals = [x for xx in optionals for x in xx]
+
+        return (self.start_function(cmd.prog) +
+                self.body(positionals, optionals, subcommands) +
+                self.end_function(cmd.prog))
+
+    def start_function(self, prog):
+        """Returns the syntax needed to begin a function definition.
+
+        Parameters:
+            prog (str): the command name
+
+        Returns:
+            str: the function definition beginning
+        """
+        name = prog.replace('-', '_').replace(' ', '_')
+        return '\n_{0}() {{'.format(name)
+
+    def end_function(self, prog=None):
+        """Returns the syntax needed to end a function definition.
+
+        Parameters:
+            prog (str, optional): the command name
+
+        Returns:
+            str: the function definition ending
+        """
+        return '}\n'
+
+    def body(self, positionals, optionals, subcommands):
+        """Returns the body of the function.
+
+        Parameters:
+            positionals (list): list of positional arguments
+            optionals (list): list of optional arguments
+            subcommands (list): list of subcommand parsers
+
+        Returns:
+            str: the function body
+        """
+        return ''
+
+    def positionals(self, positionals):
+        """Returns the syntax for reporting positional arguments.
+
+        Parameters:
+            positionals (list): list of positional arguments
+
+        Returns:
+            str: the syntax for positional arguments
+        """
+        return ''
+
+    def optionals(self, optionals):
+        """Returns the syntax for reporting optional flags.
+
+        Parameters:
+            optionals (list): list of optional arguments
+
+        Returns:
+            str: the syntax for optional flags
+        """
+        return ''
+
+    def subcommands(self, subcommands):
+        """Returns the syntax for reporting subcommands.
+
+        Parameters:
+            subcommands (list): list of subcommand parsers
+
+        Returns:
+            str: the syntax for subcommand parsers
+        """
+        return ''

lib/spack/llnl/util/cpu/__init__.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/llnl/util/cpu/alias.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)

lib/spack/llnl/util/cpu/detect.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -13,6 +13,7 @@
 import six
 
 from .microarchitecture import generic_microarchitecture, targets
+from .schema import targets_json
 
 #: Mapping from operating systems to chain of commands
 #: to obtain a dictionary of raw info on the current cpu
@@ -108,23 +109,39 @@ def sysctl(*args):
         'model': sysctl('-n', 'machdep.cpu.model'),
         'model name': sysctl('-n', 'machdep.cpu.brand_string')
     }
-
-    # Super hacky way to deal with slight representation differences
-    # Would be better to somehow consider these "identical"
-    if 'sse4.1' in info['flags']:
-        info['flags'] += ' sse4_1'
-    if 'sse4.2' in info['flags']:
-        info['flags'] += ' sse4_2'
-    if 'avx1.0' in info['flags']:
-        info['flags'] += ' avx'
-    if 'clfsopt' in info['flags']:
-        info['flags'] += ' clflushopt'
-    if 'xsave' in info['flags']:
-        info['flags'] += ' xsavec xsaveopt'
-
     return info
 
 
+def adjust_raw_flags(info):
+    """Adjust the flags detected on the system to homogenize
+    slightly different representations.
+    """
+    # Flags detected on Darwin turned to their linux counterpart
+    flags = info.get('flags', [])
+    d2l = targets_json['conversions']['darwin_flags']
+    for darwin_flag, linux_flag in d2l.items():
+        if darwin_flag in flags:
+            info['flags'] += ' ' + linux_flag
+
+
+def adjust_raw_vendor(info):
+    """Adjust the vendor field to make it human readable"""
+    if 'CPU implementer' not in info:
+        return
+
+    # Mapping numeric codes to vendor (ARM). This list is a merge from
+    # different sources:
+    #
+    # https://github.com/karelzak/util-linux/blob/master/sys-utils/lscpu-arm.c
+    # https://developer.arm.com/docs/ddi0487/latest/arm-architecture-reference-manual-armv8-for-armv8-a-architecture-profile
+    # https://github.com/gcc-mirror/gcc/blob/master/gcc/config/aarch64/aarch64-cores.def
+    # https://patchwork.kernel.org/patch/10524949/
+    arm_vendors = targets_json['conversions']['arm_vendors']
+    arm_code = info['CPU implementer']
+    if arm_code in arm_vendors:
+        info['CPU implementer'] = arm_vendors[arm_code]
+
+
 def raw_info_dictionary():
     """Returns a dictionary with information on the cpu of the current host.
 
@@ -139,6 +156,8 @@ def raw_info_dictionary():
             warnings.warn(str(e))
 
         if info:
+            adjust_raw_flags(info)
+            adjust_raw_vendor(info)
             break
 
     return info
@@ -223,3 +242,15 @@ def compatibility_check_for_x86_64(info, target):
     return (target == arch_root or arch_root in target.ancestors) \
         and (target.vendor == vendor or target.vendor == 'generic') \
         and target.features.issubset(features)
+
+
+@compatibility_check(architecture_family='aarch64')
+def compatibility_check_for_aarch64(info, target):
+    basename = 'aarch64'
+    features = set(info.get('Features', '').split())
+    vendor = info.get('CPU implementer', 'generic')
+
+    arch_root = targets[basename]
+    return (target == arch_root or arch_root in target.ancestors) \
+        and (target.vendor == vendor or target.vendor == 'generic') \
+        and target.features.issubset(features)

lib/spack/llnl/util/cpu/microarchitecture.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -8,7 +8,7 @@
 import warnings
 
 try:
-    from collections.abc import Sequence
+    from collections.abc import Sequence  # novm
 except ImportError:
     from collections import Sequence
 
@@ -204,10 +204,22 @@ def optimization_flags(self, compiler, version):
             compiler (str): name of the compiler to be used
             version (str): version of the compiler to be used
         """
-        # If we don't have information on compiler return an empty string
-        if compiler not in self.compilers:
+        # If we don't have information on compiler at all
+        # return an empty string
+        if compiler not in self.family.compilers:
             return ''
 
+        # If we have information but it stops before this
+        # microarchitecture, fall back to the best known target
+        if compiler not in self.compilers:
+            best_target = [
+                x for x in self.ancestors if compiler in x.compilers
+            ][0]
+            msg = ("'{0}' compiler is known to optimize up to the '{1}'"
+                   " microarchitecture in the '{2}' architecture family")
+            msg = msg.format(compiler, best_target, best_target.family)
+            raise UnsupportedMicroarchitecture(msg)
+
         # If we have information on this compiler we need to check the
         # version being used
         compiler_info = self.compilers[compiler]
@@ -219,15 +231,10 @@ def optimization_flags(self, compiler, version):
         def satisfies_constraint(entry, version):
             min_version, max_version = entry['versions'].split(':')
 
-            # Check version suffixes
-            min_version, min_suffix = version_components(min_version)
-            max_version, max_suffix = version_components(max_version)
-            version, suffix = version_components(version)
-
-            # If the suffixes are not all equal there's no match
-            if ((suffix != min_suffix and min_version) or
-                (suffix != max_suffix and max_version)):
-                return False
+            # Extract numeric part of the version
+            min_version, _ = version_components(min_version)
+            max_version, _ = version_components(max_version)
+            version, _ = version_components(version)
 
             # Assume compiler versions fit into semver
             tuplify = lambda x: tuple(int(y) for y in x.split('.'))

lib/spack/llnl/util/cpu/microarchitectures.json

@@ -61,16 +61,18 @@
             "flags": "-march={name} -mtune={name}"
           }
         ],
-        "clang": [
-          {
-            "versions": "0.0.0-apple:",
-            "family": "x86-64",
-            "flags": "-march={family}"
-          },
+        "apple-clang": [
           {
             "versions": ":",
-            "family": "x86-64",
-            "flags": "-march={family} -mcpu=generic"
+            "name": "x86-64",
+            "flags": "-march={name}"
+          }
+        ],
+        "clang": [
+          {
+            "versions": ":",
+            "name": "x86-64",
+            "flags": "-march={name} -mtune=generic"
           }
         ],
         "intel": {
@@ -96,8 +98,7 @@
         },
         "clang": {
           "versions": "3.9:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -122,8 +123,7 @@
         },
         "clang": {
           "versions": "3.9:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -157,8 +157,7 @@
         ],
         "clang": {
           "versions": "3.9:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -188,8 +187,7 @@
         },
         "clang": {
           "versions": "3.9:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -227,8 +225,7 @@
         ],
         "clang": {
           "versions": "3.9:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": [
           {
@@ -274,8 +271,7 @@
         ],
         "clang": {
           "versions": "3.9:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": [
           {
@@ -326,8 +322,7 @@
         ],
         "clang": {
           "versions": "3.9:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": [
           {
@@ -373,8 +368,7 @@
         },
         "clang": {
           "versions": "3.9:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "18.0:",
@@ -416,8 +410,7 @@
         },
         "clang": {
           "versions": "3.9:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "18.0:",
@@ -463,8 +456,7 @@
         "clang": {
           "versions": "3.9:",
           "name": "knl",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "18.0:",
@@ -515,8 +507,7 @@
         "clang": {
           "versions": "3.9:",
           "name": "skylake-avx512",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "18.0:",
@@ -568,8 +559,7 @@
         },
         "clang": {
           "versions": "3.9:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "18.0:",
@@ -609,7 +599,7 @@
         "avx512bw",
         "avx512dq",
         "avx512cd",
-        "avx512vnni"
+        "avx512_vnni"
       ],
       "compilers": {
         "gcc": {
@@ -618,8 +608,7 @@
         },
         "clang": {
           "versions": "8.0:",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "19.0:",
@@ -685,13 +674,11 @@
           {
             "versions": "7.0:",
             "name": "icelake-client",
-            "family": "x86-64",
-            "flags": "-march={family} -mcpu={name}"
+            "flags": "-march={name} -mtune={name}"
           },
           {
             "versions": "6.0:6.9",
-            "family": "x86-64",
-            "flags": "-march={family} -mcpu={name}"
+            "flags": "-march={name} -mtune={name}"
           }
         ],
         "intel": {
@@ -723,8 +710,7 @@
         "clang": {
           "versions": "3.9:",
           "name": "amdfam10",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -761,8 +747,7 @@
         "clang": {
           "versions": "3.9:",
           "name": "bdver1",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -803,8 +788,7 @@
         "clang": {
           "versions": "3.9:",
           "name": "bdver2",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -846,8 +830,7 @@
         "clang": {
           "versions": "3.9:",
           "name": "bdver3",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -892,8 +875,7 @@
         "clang": {
           "versions": "3.9:",
           "name": "bdver4",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -942,8 +924,7 @@
         "clang": {
           "versions": "4.0:",
           "name": "znver1",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -993,8 +974,7 @@
         "clang": {
           "versions": "9.0:",
           "name": "znver2",
-          "family": "x86-64",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-march={name} -mtune={name}"
         },
         "intel": {
           "versions": "16.0:",
@@ -1016,8 +996,7 @@
         },
         "clang": {
           "versions": ":",
-          "family": "ppc64",
-          "flags": "-march={family} -mcpu=generic"
+          "flags": "-mcpu={name} -mtune={name}"
         }
       }
     },
@@ -1033,9 +1012,7 @@
         },
         "clang": {
           "versions": "3.9:",
-          "family": "ppc64",
-          "name": "pwr7",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-mcpu={name} -mtune={name}"
         }
       }
     },
@@ -1058,9 +1035,7 @@
         ],
         "clang": {
           "versions": "3.9:",
-          "family": "ppc64",
-          "name": "pwr8",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-mcpu={name} -mtune={name}"
         }
       }
     },
@@ -1076,9 +1051,7 @@
         },
         "clang": {
           "versions": "3.9:",
-          "family": "ppc64",
-          "name": "pwr9",
-          "flags": "-march={family} -mcpu={name}"
+          "flags": "-mcpu={name} -mtune={name}"
         }
       }
     },
@@ -1094,8 +1067,7 @@
         },
         "clang": {
           "versions": ":",
-          "family": "ppc64le",
-          "flags": "-march={family} -mcpu=generic"
+          "flags": "-mcpu={name} -mtune={name}"
         }
       }
     },
@@ -1121,8 +1093,8 @@
         "clang": {
           "versions": "3.9:",
           "family": "ppc64le",
-          "name": "pwr8",
-          "flags": "-march={family} -mcpu={name}"
+          "name": "power8",
+          "flags": "-mcpu={name} -mtune={name}"
         }
       }
     },
@@ -1140,8 +1112,8 @@
         "clang": {
           "versions": "3.9:",
           "family": "ppc64le",
-          "name": "pwr9",
-          "flags": "-march={family} -mcpu={name}"
+          "name": "power9",
+          "flags": "-mcpu={name} -mtune={name}"
         }
       }
     },
@@ -1156,11 +1128,113 @@
         },
         "clang": {
           "versions": ":",
-          "family": "aarch64",
-          "flags": "-march={family} -mcpu=generic"
+          "flags": "-march=armv8-a -mtune=generic"
         }
       }
     },
+    "thunderx2": {
+      "from": "aarch64",
+      "vendor": "Cavium",
+      "features": [
+        "fp",
+        "asimd",
+        "evtstrm",
+        "aes",
+        "pmull",
+        "sha1",
+        "sha2",
+        "crc32",
+        "atomics",
+        "cpuid",
+        "asimdrdm"
+      ],
+      "compilers": {
+        "gcc": [
+          {
+            "versions": "4.8:4.8.9",
+            "flags": "-march=armv8-a"
+          },
+          {
+            "versions": "4.9:5.9",
+            "flags": "-march=armv8-a+crc+crypto"
+          },
+          {
+            "versions": "6:6.9",
+            "flags": "-march=armv8.1-a+crc+crypto"
+          },
+          {
+            "versions": "7:",
+            "flags": "-mcpu=thunderx2t99"
+          }
+        ],
+        "clang": [
+          {
+            "versions": "3.9:4.9",
+            "flags": "-march=armv8.1-a+crc+crypto"
+          },
+          {
+            "versions": "5:",
+            "flags": "-mcpu=thunderx2t99"
+          }
+        ]
+      }
+    },
+    "a64fx": {
+      "from": "aarch64",
+      "vendor": "Fujitsu",
+      "features": [
+        "fp",
+        "asimd",
+        "evtstrm",
+        "aes",
+        "pmull",
+        "sha1",
+        "sha2",
+        "crc32",
+        "atomics",
+        "cpuid",
+        "asimdrdm",
+        "fphp",
+        "asimdhp",
+        "fcma",
+        "dcpop",
+        "sve"
+      ],
+      "compilers": {
+        "gcc": [
+          {
+            "versions": "4.8:4.8.9",
+            "flags": "-march=armv8-a"
+          },
+          {
+            "versions": "4.9:5.9",
+            "flags": "-march=armv8-a+crc+crypto"
+          },
+          {
+            "versions": "6:6.9",
+            "flags": "-march=armv8.1-a+crc+crypto"
+          },
+          {
+            "versions": "7:7.9",
+            "flags": "-march=armv8.2-a+crc+crypto+fp16"
+          },
+          {
+            "versions": "8:",
+            "flags": "-march=armv8.2-a+crc+aes+sha2+fp16+sve -msve-vector-bits=512"
+          }
+        ],
+        "clang": [
+          {
+            "versions": "3.9:4.9",
+            "flags": "-march=armv8.2-a+crc+crypto+fp16"
+          },
+          {
+            "versions": "5:",
+            "flags": "-march=armv8.2-a+crc+crypto+fp16+sve"
+          }
+        ]
+      }
+    },
     "arm": {
       "from": null,
       "vendor": "generic",
@@ -1226,6 +1300,20 @@
         "ppc64"
       ]
     },
+    "vsx": {
+      "reason": "VSX alitvec extensions are supported by PowerISA from v2.06 (Power7+), but might not be listed in features",
+      "families": [
+        "ppc64le",
+        "ppc64"
+      ]
+    },
+    "fma": {
+      "reason": "FMA has been supported by PowerISA since Power1, but might not be listed in features",
+      "families": [
+        "ppc64le",
+        "ppc64"
+      ]
+    },
     "sse4.1": {
       "reason": "permits to refer to sse4_1 also as sse4.1",
       "any_of": [
@@ -1244,5 +1332,34 @@
         "aarch64"
       ]
     }
+  },
+  "conversions": {
+    "description": "Conversions that map some platform specific values to canonical values",
+    "arm_vendors": {
+      "0x41": "ARM",
+      "0x42": "Broadcom",
+      "0x43": "Cavium",
+      "0x44": "DEC",
+      "0x46": "Fujitsu",
+      "0x48": "HiSilicon",
+      "0x49": "Infineon Technologies AG",
+      "0x4d": "Motorola",
+      "0x4e": "Nvidia",
+      "0x50": "APM",
+      "0x51": "Qualcomm",
+      "0x53": "Samsung",
+      "0x56": "Marvell",
+      "0x61": "Apple",
+      "0x66": "Faraday",
+      "0x68": "HXT",
+      "0x69": "Intel"
+    },
+    "darwin_flags": {
+      "sse4.1": "sse4_1",
+      "sse4.2": "sse4_2",
+      "avx1.0": "avx",
+      "clfsopt": "clflushopt",
+      "xsave": "xsavec xsaveopt"
+    }
   }
 }

lib/spack/llnl/util/cpu/schema.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -6,7 +6,7 @@
 import os.path
 
 try:
-    from collections.abc import MutableMapping
+    from collections.abc import MutableMapping  # novm
 except ImportError:
     from collections import MutableMapping
 
@@ -72,7 +72,21 @@
                 'additionalProperties': False
             }
         },
-
+    },
+    'conversions': {
+        'type': 'object',
+        'properties': {
+            'description': {
+                'type': 'string'
+            },
+            'arm_vendors': {
+                'type': 'object',
+            },
+            'darwin_flags': {
+                'type': 'object'
+            }
+        },
+        'additionalProperties': False
     }
 }
 

lib/spack/llnl/util/filesystem.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -201,7 +201,6 @@ def groupid_to_group(x):
                         output_file.writelines(input_file.readlines())
 
         except BaseException:
-            os.remove(tmp_filename)
             # clean up the original file on failure.
             shutil.move(backup_filename, filename)
             raise
@@ -457,7 +456,7 @@ def copy_tree(src, dest, symlinks=True, ignore=None, _permissions=False):
             if os.path.isdir(s):
                 mkdirp(d)
             else:
-                shutil.copyfile(s, d)
+                shutil.copy2(s, d)
 
         if _permissions:
             set_install_permissions(d)
@@ -625,9 +624,9 @@ def replace_directory_transaction(directory_name, tmp_root=None):
     # Check the input is indeed a directory with absolute path.
     # Raise before anything is done to avoid moving the wrong directory
     assert os.path.isdir(directory_name), \
-        '"directory_name" must be a valid directory'
+        'Invalid directory: ' + directory_name
     assert os.path.isabs(directory_name), \
-        '"directory_name" must contain an absolute path'
+        '"directory_name" must contain an absolute path: ' + directory_name
 
     directory_basename = os.path.basename(directory_name)
 
@@ -917,10 +916,8 @@ def remove_if_dead_link(path):
     Parameters:
         path (str): The potential dead link
     """
-    if os.path.islink(path):
-        real_path = os.path.realpath(path)
-        if not os.path.exists(real_path):
-            os.unlink(path)
+    if os.path.islink(path) and not os.path.exists(path):
+        os.unlink(path)
 
 
 def remove_linked_tree(path):
@@ -1156,7 +1153,9 @@ class HeaderList(FileList):
 
     # Make sure to only match complete words, otherwise path components such
     # as "xinclude" will cause false matches.
-    include_regex = re.compile(r'(.*)(\binclude\b)(.*)')
+    # Avoid matching paths such as <prefix>/include/something/detail/include,
+    # e.g. in the CUDA Toolkit which ships internal libc++ headers.
+    include_regex = re.compile(r'(.*?)(\binclude\b)(.*)')
 
     def __init__(self, files):
         super(HeaderList, self).__init__(files)

lib/spack/llnl/util/lang.py

@@ -1,4 +1,4 @@
-# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
 # Spack Project Developers. See the top-level COPYRIGHT file for details.
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
@@ -19,12 +19,6 @@
 ignore_modules = [r'^\.#', '~$']
 
 
-class classproperty(property):
-    """classproperty decorator: like property but for classmethods."""
-    def __get__(self, cls, owner):
-        return self.fget.__get__(None, owner)()
-
-
 def index_by(objects, *funcs):
     """Create a hierarchy of dictionaries by splitting the supplied
        set of objects on unique values of the supplied functions.
@@ -612,14 +606,41 @@ def load_module_from_file(module_name, module_path):
     """
     if sys.version_info[0] == 3 and sys.version_info[1] >= 5:
         import importlib.util
-        spec = importlib.util.spec_from_file_location(module_name, module_path)
-        module = importlib.util.module_from_spec(spec)
+        spec = importlib.util.spec_from_file_location(  # novm
+            module_name, module_path)
+        module = importlib.util.module_from_spec(spec)  # novm
         spec.loader.exec_module(module)
     elif sys.version_info[0] == 3 and sys.version_info[1] < 5:
         import importlib.machinery
-        loader = importlib.machinery.SourceFileLoader(module_name, module_path)
+        loader = importlib.machinery.SourceFileLoader(  # novm
+            module_name, module_path)
         module = loader.load_module()
     elif sys.version_info[0] == 2:
         import imp
         module = imp.load_source(module_name, module_path)
     return module
+
+
+def uniq(sequence):
+    """Remove strings of duplicate elements from a list.
+
+    This works like the command-line ``uniq`` tool.  It filters strings
+    of duplicate elements in a list. Adjacent matching elements are
+    merged into the first occurrence.
+
+    For example::
+
+        uniq([1, 1, 1, 1, 2, 2, 2, 3, 3]) == [1, 2, 3]
+        uniq([1, 1, 1, 1, 2, 2, 2, 1, 1]) == [1, 2, 1]
+
+    """
+    if not sequence:
+        return []
+
+    uniq_list = [sequence[0]]
+    last = sequence[0]
+    for element in sequence[1:]:
+        if element != last:
+            uniq_list.append(element)
+            last = element
+    return uniq_list