features: Remove stage symlinks (#12072)

Fixes #11163

The goal of this work is to simplify stage directory structures by eliminating use of symbolic links. This means, among other things, that` $spack/var/spack/stage` will no longer be the core staging directory. Instead, the first accessible `config:build_stage` path will be used.

Spack will no longer automatically append `spack-stage` (or the like) to configured build stage directories so the onus of distinguishing the directory from other work -- so the other work is not automatically removed with a `spack clean` operation -- falls on the user.

etc/spack/defaults/config.yaml

@@ -37,18 +37,28 @@ config:
 
   # Temporary locations Spack can try to use for builds.
   #
-  # Spack will use the first one it finds that exists and is writable.
-  # You can use $tempdir to refer to the system default temp directory
-  # (as returned by tempfile.gettempdir()).
+  # Recommended options are given below.
   #
-  # A value of $spack/var/spack/stage indicates that Spack should run
-  # builds directly inside its install directory without staging them in
+  # Builds can be faster in temporary directories on some (e.g., HPC) systems.
+  # Specifying `$tempdir` will ensure use of the system default temporary
+  # directory (as returned by `tempfile.gettempdir()`).  Spack will append
+  # `spack-stage` and, if the username is not already in the path, the value
+  # of `$user` to the path.  The latter is used to avoid conflicts between
+  # users in shared temporary spaces.
+  #
+  # Another option that prevents conflicts and potential permission issues is
+  # to specify `~/.spack/stage`, which ensures each user builds in their home
+  # directory.
+  #
+  # A more traditional path uses the value of `$spack/var/spack/stage`, which
+  # builds directly inside Spack's instance without staging them in a
   # temporary space.
   #
-  # The build stage can be purged with `spack clean --stage`.
+  # The build stage can be purged with `spack clean --stage` and
+  # `spack clean -a`, so it is important that the specified directory uniquely
+  # identifies Spack staging to avoid accidentally wiping out non-Spack work.
   build_stage:
-    - $tempdir
-    - $spack/var/spack/stage
+    - $tempdir/spack-stage
 
 
   # Cache directory for already downloaded source tarballs and archived