Split the workflow section and remove outdated advices (#29344)

This PR removes a few outdated sections from the "Basics" part of the 
documentation. It also makes a few topic under the environment section
more prominent by removing an unneeded spack.yaml subsection and 
promoting everything under it.

lib/spack/docs/basic_usage.rst

@@ -1723,8 +1723,8 @@ Activating Extensions in a View
 
 Another way to use extensions is to create a view, which merges the
 python installation along with the extensions into a single prefix.
-See :ref:`filesystem-views` for a more in-depth description of views and
+See :ref:`configuring_environment_views` for a more in-depth description
-:ref:`cmd-spack-view` for usage of the ``spack view`` command.
+of views.
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Activating Extensions Globally

lib/spack/docs/environments.rst

@@ -384,18 +384,11 @@ Sourcing that file in Bash will make the environment available to the
 user; and can be included in ``.bashrc`` files, etc.  The ``loads``
 file may also be copied out of the environment, renamed, etc.
 
-----------
-spack.yaml
-----------
-
-Spack environments can be customized at finer granularity by editing
-the ``spack.yaml`` manifest file directly.
-
 .. _environment-configuration:
 
-^^^^^^^^^^^^^^^^^^^^^^^^
+------------------------
 Configuring Environments
-^^^^^^^^^^^^^^^^^^^^^^^^
+------------------------
 
 A variety of Spack behaviors are changed through Spack configuration
 files, covered in more detail in the :ref:`configuration`
@@ -417,9 +410,9 @@ environment can be specified by ``env:NAME`` (to affect environment
 ``foo``, set ``--scope env:foo``). These commands will automatically
 manipulate configuration inline in the ``spack.yaml`` file.
 
-"""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^
 Inline configurations
-"""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^
 
 Inline Environment-scope configuration is done using the same yaml
 format as standard Spack configuration scopes, covered in the
@@ -440,9 +433,9 @@ a ``packages.yaml`` file) could contain:
 This configuration sets the default compiler for all packages to
 ``intel``.
 
-"""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^
 Included configurations
-"""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^
 
 Spack environments allow an ``include`` heading in their yaml
 schema. This heading pulls in external configuration files and applies
@@ -462,9 +455,9 @@ to make small changes to an individual Environment. Included configs
 listed earlier will have higher precedence, as the included configs are
 applied in reverse order.
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-------------------------------
 Manually Editing the Specs List
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+-------------------------------
 
 The list of abstract/root specs in the Environment is maintained in
 the ``spack.yaml`` manifest under the heading ``specs``.
@@ -482,9 +475,9 @@ Appending to this list in the yaml is identical to using the ``spack
 add`` command from the command line. However, there is more power
 available from the yaml file.
 
-"""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^
 Spec concretization
-"""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^
 
 Specs can be concretized separately or together, as already
 explained in :ref:`environments_concretization`. The behavior active
@@ -510,9 +503,9 @@ which can currently take either one of the two allowed values ``together`` or ``
    the environment remains consistent. When instead the specs are concretized
    separately only the new specs will be re-concretized after any addition.
 
-"""""""""""""
+^^^^^^^^^^^^^
 Spec Matrices
-"""""""""""""
+^^^^^^^^^^^^^
 
 Entries in the ``specs`` list can be individual abstract specs or a
 spec matrix.
@@ -572,9 +565,9 @@ This allows one to create toolchains out of combinations of
 constraints and apply them somewhat indiscriminately to packages,
 without regard for the applicability of the constraint.
 
-""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^
 Spec List References
-""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^
 
 The last type of possible entry in the specs list is a reference.
 
@@ -674,9 +667,9 @@ The valid variables for a ``when`` clause are:
 #. ``hostname``. The hostname of the system (if ``hostname`` is an
    executable in the user's PATH).
 
-""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^
 SpecLists as Constraints
-""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^
 
 Dependencies and compilers in Spack can be both packages in an
 environment and constraints on other packages. References to SpecLists
@@ -708,33 +701,32 @@ For example, the following environment has three root packages:
 This allows for a much-needed reduction in redundancy between packages
 and constraints.
 
-^^^^^^^^^^^^^^^^^^^^^^^^^
+----------------
-Environment-managed Views
+Filesystem Views
-^^^^^^^^^^^^^^^^^^^^^^^^^
+----------------
 
-Spack Environments can define filesystem views of their software,
+Spack Environments can define filesystem views, which provide a direct access point
-which are maintained as packages and can be installed and uninstalled from
+for software similar to the directory hierarchy that might exist under ``/usr/local``.
-the Environment. Filesystem views provide an access point for packages
+Filesystem views are updated every time the environment is written out to the lock
-from the filesystem for users who want to access those packages
+file ``spack.lock``, so the concrete environment and the view are always compatible.
-directly. For more information on filesystem views, see the section
+The files of the view's installed packages are brought into the view by symbolic or
-:ref:`filesystem-views`.
+hard links, referencing the original Spack installation, or by copy.
-
-Spack Environment managed views are updated every time the environment
-is written out to the lock file ``spack.lock``, so the concrete
-environment and the view are always compatible.
 
 .. _configuring_environment_views:
 
-"""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Configuring environment views
+Configuration in ``spack.yaml``
-"""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 The Spack Environment manifest file has a top-level keyword
-``view``. Each entry under that heading is a view descriptor, headed
+``view``. Each entry under that heading is a **view descriptor**, headed
-by a name. The view descriptor contains the root of the view, and
+by a name. Any number of views may be defined under the ``view`` heading.
+The view descriptor contains the root of the view, and
 optionally the projections for the view, ``select`` and
 ``exclude`` lists for the view and link information via ``link`` and
-``link_type``. For example, in the following manifest
+``link_type``.
+
+For example, in the following manifest
 file snippet we define a view named ``mpis``, rooted at
 ``/path/to/view`` in which all projections use the package name,
 version, and compiler name to determine the path for a given
@@ -759,8 +751,7 @@ directories.
          link: all
          link_type: symlink
 
-For more information on using view projections, see the section on
+The default for the ``select`` and
-:ref:`adding_projections_to_views`. The default for the ``select`` and
 ``exclude`` values is to select everything and exclude nothing. The
 default projection is the default view projection (``{}``). The ``link``
 attribute allows the following values:
@@ -780,8 +771,6 @@ of ``hardlink`` or ``copy``.
    when the environment is not activated, and linked libraries will be located
    *outside* of the view thanks to rpaths.
 
-Any number of views may be defined under the ``view`` heading in a
-Spack Environment.
 
 There are two shorthands for environments with a single view. If the
 environment at ``/path/to/env`` has a single view, with a root at
@@ -847,9 +836,47 @@ regenerate`` will regenerate the views for the environment. This will
 apply any updates in the environment configuration that have not yet
 been applied.
 
-""""""""""""""""""""""""""""
+.. _view_projections:
+
+""""""""""""""""
+View Projections
+""""""""""""""""
+The default projection into a view is to link every package into the
+root of the view. The projections attribute is a mapping of partial specs to
+spec format strings, defined by the :meth:`~spack.spec.Spec.format`
+function, as shown in the example below:
+
+.. code-block:: yaml
+
+   projections:
+     zlib: {name}-{version}
+     ^mpi: {name}-{version}/{^mpi.name}-{^mpi.version}-{compiler.name}-{compiler.version}
+     all: {name}-{version}/{compiler.name}-{compiler.version}
+
+The entries in the projections configuration file must all be either
+specs or the keyword ``all``. For each spec, the projection used will
+be the first non-``all`` entry that the spec satisfies, or ``all`` if
+there is an entry for ``all`` and no other entry is satisfied by the
+spec. Where the keyword ``all`` appears in the file does not
+matter.
+
+Given the example above, the spec ``zlib@1.2.8``
+will be linked into ``/my/view/zlib-1.2.8/``, the spec
+``hdf5@1.8.10+mpi %gcc@4.9.3 ^mvapich2@2.2`` will be linked into
+``/my/view/hdf5-1.8.10/mvapich2-2.2-gcc-4.9.3``, and the spec
+``hdf5@1.8.10~mpi %gcc@4.9.3`` will be linked into
+``/my/view/hdf5-1.8.10/gcc-4.9.3``.
+
+If the keyword ``all`` does not appear in the projections
+configuration file, any spec that does not satisfy any entry in the
+file will be linked into the root of the view as in a single-prefix
+view. Any entries that appear below the keyword ``all`` in the
+projections configuration file will not be used, as all specs will use
+the projection under ``all`` before reaching those entries.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Activating environment views
-""""""""""""""""""""""""""""
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 The ``spack env activate`` command will put the default view for the
 environment into the user's path, in addition to activating the

lib/spack/docs/index.rst

@@ -54,8 +54,8 @@ or refer to the full manual below.
    features
    getting_started
    basic_usage
-   workflows
    Tutorial: Spack 101 <https://spack-tutorial.readthedocs.io>
+   replace_conda_homebrew
    known_issues
 
 .. toctree::

lib/spack/docs/module_file_support.rst

@@ -378,7 +378,7 @@ most likely via the ``+blas`` variant specification.
 
 The most heavyweight solution to module naming is to change the entire
 naming convention for module files. This uses the projections format
-covered in :ref:`adding_projections_to_views`.
+covered in :ref:`view_projections`.
 
 .. code-block:: yaml
 
@@ -540,8 +540,7 @@ configuration:
 
 #. The configuration is for an :ref:`environment <environments>` and
    will never be applied outside the environment,
-#. The environment in question is configured to use a :ref:`view
+#. The environment in question is configured to use a view,
-   <filesystem-views>`,
 #. The :ref:`environment view is configured
    <configuring_environment_views>` with a projection that ensures
    every package is linked to a unique directory,

lib/spack/docs/packaging_guide.rst

@@ -2543,7 +2543,7 @@ from being linked in at activation time.
 Views
 -----
 
-As covered in :ref:`filesystem-views`, the ``spack view`` command can be
+The ``spack view`` command can be
 used to symlink a number of packages into a merged prefix. The methods of
 ``PackageViewMixin`` can be overridden to customize how packages are added
 to views. Generally this can be used to create copies of specific files rather

lib/spack/docs/replace_conda_homebrew.rst

@@ -0,0 +1,206 @@
+.. Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+   SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+=====================================
+Using Spack to Replace Homebrew/Conda
+=====================================
+
+Spack is an incredibly powerful package manager, designed for supercomputers
+where users have diverse installation needs. But Spack can also be used to
+handle simple single-user installations on your laptop. Most macOS users are
+already familiar with package managers like Homebrew and Conda, where all
+installed packages are symlinked to a single central location like ``/usr/local``.
+In this section, we will show you how to emulate the behavior of Homebrew/Conda
+using :ref:`environments`!
+
+-----
+Setup
+-----
+
+First, let's create a new environment. We'll assume that Spack is already set up
+correctly, and that you've already sourced the setup script for your shell.
+To create a new environment, simply run:
+
+.. code-block:: console
+
+   $ spack env create myenv
+
+Here, *myenv* can be anything you want to name your environment. Next, we can add
+a list of packages we would like to install into our environment. Let's say we
+want a newer version of Bash than the one that comes with macOS, and we want a
+few Python libraries. We can run:
+
+.. code-block:: console
+
+   $ spack -e myenv add bash@5 python py-numpy py-scipy py-matplotlib
+
+Each package can be listed on a separate line, or combined into a single line like we did above.
+Notice that we're explicitly asking for Bash 5 here. You can use any spec
+you would normally use on the command line with other Spack commands.
+
+Next, we want to manually configure a couple of things:
+
+.. code-block:: console
+
+   $ spack -e myenv config edit
+
+.. code-block:: yaml
+
+   # This is a Spack Environment file.
+   #
+   # It describes a set of packages to be installed, along with
+   # configuration settings.
+   spack:
+     # add package specs to the `specs` list
+     specs: [bash@5, python, py-numpy, py-scipy, py-matplotlib]
+     view: true
+
+You can see the packages we added earlier in the ``specs:`` section. If you
+ever want to add more packages, you can either use ``spack add`` or manually
+edit this file.
+
+We also need to change the ``concretization:`` option. By default, Spack
+concretizes each spec *separately*, allowing multiple versions of the same
+package to coexist. Since we want a single consistent environment, we want to
+concretize all of the specs *together*.
+
+Here is what your ``spack.yaml`` looks like with this new setting:
+
+.. code-block:: yaml
+
+   # This is a Spack Environment file.
+   #
+   # It describes a set of packages to be installed, along with
+   # configuration settings.
+   spack:
+     # add package specs to the `specs` list
+     specs: [bash@5, python, py-numpy, py-scipy, py-matplotlib]
+     view: true
+     concretization: together
+
+^^^^^^^^^^^^^^^^
+Symlink location
+^^^^^^^^^^^^^^^^
+
+Spack symlinks all installations to ``/Users/me/spack/var/spack/environments/myenv/.spack-env/view``,
+which is the default when ``view: true``.
+You can actually change this to any directory you want. For example, Homebrew
+uses ``/usr/local``, while Conda uses ``/Users/me/anaconda``. In order to access
+files in these locations, you need to update ``PATH`` and other environment variables
+to point to them. Activating the Spack environment does this automatically, but
+you can also manually set them in your ``.bashrc``.
+
+.. warning::
+
+   There are several reasons why you shouldn't use ``/usr/local``:
+
+   1. If you are on macOS 10.11+ (El Capitan and newer), Apple makes it hard
+      for you. You may notice permissions issues on ``/usr/local`` due to their
+      `System Integrity Protection <https://support.apple.com/en-us/HT204899>`_.
+      By default, users don't have permissions to install anything in ``/usr/local``,
+      and you can't even change this using ``sudo chown`` or ``sudo chmod``.
+   2. Other package managers like Homebrew will try to install things to the
+      same directory. If you plan on using Homebrew in conjunction with Spack,
+      don't symlink things to ``/usr/local``.
+   3. If you are on a shared workstation, or don't have sudo privileges, you
+      can't do this.
+
+   If you still want to do this anyway, there are several ways around SIP.
+   You could disable SIP by booting into recovery mode and running
+   ``csrutil disable``, but this is not recommended, as it can open up your OS
+   to security vulnerabilities. Another technique is to run ``spack concretize``
+   and ``spack install`` using ``sudo``. This is also not recommended.
+
+   The safest way I've found is to create your installation directories using
+   sudo, then change ownership back to the user like so:
+
+   .. code-block:: bash
+
+      for directory in .spack bin contrib include lib man share
+      do
+          sudo mkdir -p /usr/local/$directory
+          sudo chown $(id -un):$(id -gn) /usr/local/$directory
+      done
+
+   Depending on the packages you install in your environment, the exact list of
+   directories you need to create may vary. You may also find some packages
+   like Java libraries that install a single file to the installation prefix
+   instead of in a subdirectory. In this case, the action is the same, just replace
+   ``mkdir -p`` with ``touch`` in the for-loop above.
+
+   But again, it's safer just to use the default symlink location.
+
+
+------------
+Installation
+------------
+
+To actually concretize the environment, run:
+
+.. code-block:: console
+
+   $ spack -e myenv concretize
+
+This will tell you which if any packages are already installed, and alert you
+to any conflicting specs.
+
+To actually install these packages and symlink them to your ``view:``
+directory, simply run:
+
+.. code-block:: console
+
+   $ spack -e myenv install
+   $ spack env activate myenv
+
+Now, when you type ``which python3``, it should find the one you just installed.
+
+In order to change the default shell to our newer Bash installation, we first
+need to add it to this list of acceptable shells. Run:
+
+.. code-block:: console
+
+   $ sudo vim /etc/shells
+
+and add the absolute path to your bash executable. Then run:
+
+.. code-block:: console
+
+   $ chsh -s /path/to/bash
+
+Now, when you log out and log back in, ``echo $SHELL`` should point to the
+newer version of Bash.
+
+---------------------------
+Updating Installed Packages
+---------------------------
+
+Let's say you upgraded to a new version of macOS, or a new version of Python
+was released, and you want to rebuild your entire software stack. To do this,
+simply run the following commands:
+
+.. code-block:: console
+
+   $ spack env activate myenv
+   $ spack concretize --force
+   $ spack install
+
+The ``--force`` flag tells Spack to overwrite its previous concretization
+decisions, allowing you to choose a new version of Python. If any of the new
+packages like Bash are already installed, ``spack install`` won't re-install
+them, it will keep the symlinks in place.
+
+--------------
+Uninstallation
+--------------
+
+If you decide that Spack isn't right for you, uninstallation is simple.
+Just run:
+
+.. code-block:: console
+
+   $ spack env activate myenv
+   $ spack uninstall --all
+
+This will uninstall all packages in your environment and remove the symlinks.