Use spack/user-specific stage root by default; stage cleaning (#12516)

* When cleaning the stage root, only remove directories that appear
  to be used for staging Spack packages. Previously Spack was clearing
  all directories in the stage root, which could remove content not
  related to Spack if the user chose a staging root which contains
  files/directories not managed by Spack.
* The documentation is updated with warnings about choosing a stage
  directory that is only managed by Spack (although generally the
  check added in this PR for "spack clean" should avoid removing
  content that was not created by Spack)
* The default stage directory (in config.yaml) is now
  $tempdir/$user/spack-stage and the logic is updated to omit the
  $user portion of this path if $tempdir already contains a $user
  directory.
* When creating stage root assign user read/write permissions to all
  directories in the path under $user. Previously Spack was assigning
  the permissions of the first existing parent directory

lib/spack/docs/config_yaml.rst

@@ -86,22 +86,28 @@ Spack is designed to run out of a user home directory, and on many
 systems the home directory is a (slow) network file system.  On most systems,
 building in a temporary file system is faster.  Usually, there is also more
 space available in the temporary location than in the home directory.  If the
-username is not already in the path, Spack will also append the value of
-`$user` to the path.
+username is not already in the path, Spack will append the value of ``$user`` to
+the selected ``build_stage`` path.
 
-.. warning:: We highly recommend appending `spack-stage` to `$tempdir` in order
-   to ensure `spack clean` does not delete everything in `$tempdir`.
+.. warning:: We highly recommend specifying ``build_stage`` paths that
+   distinguish between staging and other activities to ensure 
+   ``spack clean`` does not inadvertently remove unrelated files.
+   This can be accomplished by using a combination of ``spack`` and or
+   ``stage`` in each path as shown in the default settings and documented
+   examples.
 
 By default, Spack's ``build_stage`` is configured like this:
 
 .. code-block:: yaml
 
    build_stage:
-    - $tempdir/spack-stage
+    - $tempdir/$user/spack-stage
+    - ~/.spack/stage
 
 This can be an ordered list of paths that Spack should search when trying to
 find a temporary directory for the build stage.  The list is searched in
 order, and Spack will use the first directory to which it has write access.
+
 Specifying `~/.spack/stage` first will ensure each user builds in their home
 directory.  The historic Spack stage path `$spack/var/spack/stage` will build
 directly inside the Spack instance.  See :ref:`config-file-variables` for more
@@ -116,7 +122,7 @@ deleted, but you can manually purge them with :ref:`spack clean --stage
 .. note::
 
    The build will fail if there is no writable directory in the ``build_stage``
-   list.
+   list, where any user- and site-specific setting will be searched first.
 
 --------------------
 ``source_cache``

lib/spack/docs/configuration.rst

@@ -36,8 +36,8 @@ Here is an example ``config.yaml`` file:
      module_roots:
        lmod: $spack/share/spack/lmod
      build_stage:
-       - $tempdir
-       - /nfs/tmp2/$user
+       - $tempdir/$user/spack-stage
+       - ~/.spack/stage
 
 Each Spack configuration file is nested under a top-level section
 corresponding to its name. So, ``config.yaml`` starts with ``config:``,
@@ -244,8 +244,8 @@ your configurations look like this:
      module_roots:
        lmod: $spack/share/spack/lmod
      build_stage:
-       - $tempdir
-       - /nfs/tmp2/$user
+       - $tempdir/$user/spack-stage
+       - ~/.spack/stage
 
 
 .. code-block:: yaml
@@ -269,8 +269,8 @@ command:
      module_roots:
        lmod: $spack/share/spack/lmod
      build_stage:
-       - $tempdir
-       - /nfs/tmp2/$user
+       - $tempdir/$user/spack-stage
+       - ~/.spack/stage
 
 
 .. _config-overrides:
@@ -312,8 +312,8 @@ Let's revisit the ``config.yaml`` example one more time. The
    :caption: $(prefix)/etc/spack/defaults/config.yaml
 
    build_stage:
-     - $tempdir
-     - /nfs/tmp2/$user
+     - $tempdir/$user/spack-stage
+     - ~/.spack/stage
 
 
 Suppose the user configuration adds its *own* list of ``build_stage``
@@ -323,7 +323,7 @@ paths:
    :caption: ~/.spack/config.yaml
 
    build_stage:
-     - /lustre-scratch/$user
+     - /lustre-scratch/$user/spack
      - ~/mystage
 
 
@@ -341,10 +341,10 @@ get config`` shows the result:
      module_roots:
        lmod: $spack/share/spack/lmod
      build_stage:
-       - /lustre-scratch/$user
+       - /lustre-scratch/$user/spack
        - ~/mystage
-       - $tempdir
-       - /nfs/tmp2/$user
+       - $tempdir/$user/spack-stage
+       - ~/.spack/stage
 
 
 As in :ref:`config-overrides`, the higher-precedence scope can
@@ -356,7 +356,7 @@ user config looked like this:
    :caption: ~/.spack/config.yaml
 
    build_stage::
-     - /lustre-scratch/$user
+     - /lustre-scratch/$user/spack
      - ~/mystage
 
 
@@ -371,7 +371,7 @@ The merged configuration would look like this:
      module_roots:
        lmod: $spack/share/spack/lmod
      build_stage:
-       - /lustre-scratch/$user
+       - /lustre-scratch/$user/spack
        - ~/mystage
 
 
@@ -465,8 +465,8 @@ account all scopes. For example, to see the fully merged
        lmod: $spack/share/spack/lmod
        dotkit: $spack/share/spack/dotkit
      build_stage:
-     - $tempdir
-     - /nfs/tmp2/$user
+     - $tempdir/$user/spack-stage
+     - ~/.spack/stage
      - $spack/var/spack/stage
      source_cache: $spack/var/spack/cache
      misc_cache: ~/.spack/cache
@@ -516,8 +516,8 @@ down the problem:
    /home/myuser/spack/etc/spack/defaults/config.yaml:34      lmod: $spack/share/spack/lmod
    /home/myuser/spack/etc/spack/defaults/config.yaml:35      dotkit: $spack/share/spack/dotkit
    /home/myuser/spack/etc/spack/defaults/config.yaml:49    build_stage:
-   /home/myuser/spack/etc/spack/defaults/config.yaml:50    - $tempdir
-   /home/myuser/spack/etc/spack/defaults/config.yaml:51    - /nfs/tmp2/$user
+   /home/myuser/spack/etc/spack/defaults/config.yaml:50    - $tempdir/$user/spack-stage
+   /home/myuser/spack/etc/spack/defaults/config.yaml:51    - ~/.spack/stage
    /home/myuser/spack/etc/spack/defaults/config.yaml:52    - $spack/var/spack/stage
    /home/myuser/spack/etc/spack/defaults/config.yaml:57    source_cache: $spack/var/spack/cache
    /home/myuser/spack/etc/spack/defaults/config.yaml:62    misc_cache: ~/.spack/cache

lib/spack/docs/packaging_guide.rst

@@ -3915,7 +3915,8 @@ The first step of ``spack install``.  Takes a spec and determines the
 correct download URL to use for the requested package version, then
 downloads the archive, checks it against an MD5 checksum, and stores
 it in a staging directory if the check was successful.  The staging
-directory will be located under ``$SPACK_HOME/var/spack``.
+directory will be located under the first writable directory in the
+``build_stage`` configuration setting.
 
 When run after the archive has already been downloaded, ``spack
 fetch`` is idempotent and will not download the archive again.

lib/spack/docs/tutorial_configuration.rst

@@ -849,7 +849,17 @@ from this file system with the following ``config.yaml``:
 
    config:
      build_stage:
-       - /scratch/$user
+       - /scratch/$user/spack-stage
+
+
+.. note::
+
+   It is important to distinguish the build stage directory from other
+   directories in your scratch space to ensure ``spack clean`` does not
+   inadvertently remove unrelated files.  This can be accomplished by
+   including a combination of ``spack`` and or ``stage`` in each path
+   as shown in the default settings and documented examples.  See
+   :ref:`config-yaml` for details.
 
 
 On systems with compilers that absolutely *require* environment variables