zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs: environment update (#44087)

Browse Source 
Updated the terminology for the two types of environments to be
consistent with that used in the tutorial for the last three years.

Additionally:
* changed 'anonymous' to 'independent in environment command+test for consistency.
This commit is contained in:
Tamara Dahlgren 2024-10-01 13:26:31 -07:00 committed by GitHub
parent 486ff2beac
commit c8efea117f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 138 additions and 122 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

252
lib/spack/docs/environments.rst
View File

@ -5,49 +5,56 @@
.. _environments: .. _environments:
========================= =====================================
Environments (spack.yaml) Environments (spack.yaml, spack.lock)
========================= =====================================
An environment is used to group together a set of specs for the An environment is used to group a set of specs intended for some purpose
purpose of building, rebuilding and deploying in a coherent fashion. to be built, rebuilt, and deployed in a coherent fashion. Environments
Environments provide a number of advantages over the *à la carte* define aspects of the installation of the software, such as:
approach of building and loading individual Spack modules:
#. Environments separate the steps of (a) choosing what to #. *which* specs to install;
install, (b) concretizing, and (c) installing. This allows #. *how* those specs are configured; and
Environments to remain stable and repeatable, even if Spack packages #. *where* the concretized software will be installed.
are upgraded: specs are only re-concretized when the user
explicitly asks for it. It is even possible to reliably Aggregating this information into an environment for processing has advantages
transport environments between different computers running over the *à la carte* approach of building and loading individual Spack modules.
different versions of Spack!
#. Environments allow several specs to be built at once; a more robust With environments, you concretize, install, or load (activate) all of the
solution than ad-hoc scripts making multiple calls to ``spack specs with a single command. Concretization fully configures the specs
install``. and dependencies of the environment in preparation for installing the
#. An Environment that is built as a whole can be loaded as a whole software. This is a more robust solution than ad-hoc installation scripts.
into the user environment. An Environment can be built to maintain And you can share an environment or even re-use it on a different computer.
a filesystem view of its packages, and the environment can load
that view into the user environment at activation time. Spack can Environment definitions, especially *how* specs are configured, allow the
also generate a script to load all modules related to an software to remain stable and repeatable even when Spack packages are upgraded. Changes are only picked up when the environment is explicitly re-concretized.
environment.
Defining *where* specs are installed supports a filesystem view of the
environment. Yet Spack maintains a single installation of the software that
can be re-used across multiple environments.
Activating an environment determines *when* all of the associated (and
installed) specs are loaded so limits the software loaded to those specs
actually needed by the environment. Spack can even generate a script to
load all modules related to an environment.
Other packaging systems also provide environments that are similar in Other packaging systems also provide environments that are similar in
some ways to Spack environments; for example, `Conda environments some ways to Spack environments; for example, `Conda environments
<https://conda.io/docs/user-guide/tasks/manage-environments.html>`_ or <https://conda.io/docs/user-guide/tasks/manage-environments.html>`_ or
`Python Virtual Environments `Python Virtual Environments
<https://docs.python.org/3/tutorial/venv.html>`_. Spack environments <https://docs.python.org/3/tutorial/venv.html>`_. Spack environments
provide some distinctive features: provide some distinctive features though:
#. A spec installed "in" an environment is no different from the same #. A spec installed "in" an environment is no different from the same
spec installed anywhere else in Spack. Environments are assembled spec installed anywhere else in Spack.
simply by collecting together a set of specs. #. Spack environments may contain more than one spec of the same
#. Spack Environments may contain more than one spec of the same
package. package.
Spack uses a "manifest and lock" model similar to `Bundler gemfiles Spack uses a "manifest and lock" model similar to `Bundler gemfiles
<https://bundler.io/man/gemfile.5.html>`_ and other package <https://bundler.io/man/gemfile.5.html>`_ and other package managers.
managers. The user input file is named ``spack.yaml`` and the lock The environment's user input file (or manifest), is named ``spack.yaml``.
file is named ``spack.lock`` The lock file, which contains the fully configured and concretized specs,
is named ``spack.lock``.
.. _environments-using: .. _environments-using:
@ -68,55 +75,60 @@ An environment is created by:
$ spack env create myenv $ spack env create myenv
Spack then creates the directory ``var/spack/environments/myenv``. The directory ``$SPACK_ROOT/var/spack/environments/myenv`` is created
to manage the environment.
.. note:: .. note::
All managed environments by default are stored in the ``var/spack/environments`` folder. All managed environments by default are stored in the
This location can be changed by setting the ``environments_root`` variable in ``config.yaml``. ``$SPACK_ROOT/var/spack/environments`` folder. This location can be changed
by setting the ``environments_root`` variable in ``config.yaml``.
In the ``var/spack/environments/myenv`` directory, Spack creates the Spack creates the file ``spack.yaml``, hidden directory ``.spack-env``, and
file ``spack.yaml`` and the hidden directory ``.spack-env``. ``spack.lock`` file under ``$SPACK_ROOT/var/spack/environments/myenv``. User
interaction occurs through the ``spack.yaml`` file and the Spack commands
Spack stores metadata in the ``.spack-env`` directory. User that affect it. Metadata and, by default, the view are stored in the
interaction will occur through the ``spack.yaml`` file and the Spack ``.spack-env`` directory. When the environment is concretized, Spack creates
commands that affect it. When the environment is concretized, Spack the ``spack.lock`` file with the fully configured specs and dependencies for
will create a file ``spack.lock`` with the concrete information for
the environment. the environment.
In addition to being the default location for the view associated with The ``.spack-env`` subdirectory also contains:
an Environment, the ``.spack-env`` directory also contains:
* ``repo/``: A repo consisting of the Spack packages used in this * ``repo/``: A subdirectory acting as the repo consisting of the Spack
environment. This allows the environment to build the same, in packages used in the environment. It allows the environment to build
theory, even on different versions of Spack with different the same, in theory, even on different versions of Spack with different
packages! packages!
* ``logs/``: A directory containing the build logs for the packages * ``logs/``: A subdirectory containing the build logs for the packages
in this Environment. in this environment.
Spack Environments can also be created from either a manifest file Spack Environments can also be created from either the user input, or
(usually but not necessarily named, ``spack.yaml``) or a lockfile. manifest, file or the lockfile. Create an environment from a manifest using:
To create an Environment from a manifest:
.. code-block:: console .. code-block:: console
$ spack env create myenv spack.yaml $ spack env create myenv spack.yaml
To create an Environment from a ``spack.lock`` lockfile: The resulting environment is guaranteed to have the same root specs as
the original but may concretize differently in the presence of different
explicit or default configuration settings (e.g., a different version of
Spack or for a different user account).
Create an environment from a ``spack.lock`` file using:
.. code-block:: console .. code-block:: console
$ spack env create myenv spack.lock $ spack env create myenv spack.lock
Either of these commands can also take a full path to the The resulting environment, when on the same or a compatible machine, is
initialization file. guaranteed to initially have the same concrete specs as the original.
A Spack Environment created from a ``spack.yaml`` manifest is .. note::
guaranteed to have the same root specs as the original Environment,
but may concretize differently. A Spack Environment created from a Environment creation also accepts a full path to the file.
``spack.lock`` lockfile is guaranteed to have the same concrete specs
as the original Environment. Either may obviously then differ as the If the path is not under the ``$SPACK_ROOT/var/spack/environments``
user modifies it. directory then the source is referred to as an
:ref:`independent environment <independent_environments>`.
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
Activating an Environment Activating an Environment
@ -129,7 +141,7 @@ To activate an environment, use the following command:
$ spack env activate myenv $ spack env activate myenv
By default, the ``spack env activate`` will load the view associated By default, the ``spack env activate`` will load the view associated
with the Environment into the user environment. The ``-v, with the environment into the user environment. The ``-v,
--with-view`` argument ensures this behavior, and the ``-V, --with-view`` argument ensures this behavior, and the ``-V,
--without-view`` argument activates the environment without changing --without-view`` argument activates the environment without changing
the user environment variables. the user environment variables.
@ -142,8 +154,11 @@ user's prompt to begin with the environment name in brackets.
$ spack env activate -p myenv $ spack env activate -p myenv
[myenv] $ ... [myenv] $ ...
The ``activate`` command can also be used to create a new environment if it does not already The ``activate`` command can also be used to create a new environment, if it is
exist. not already defined, by adding the ``--create`` flag. Managed and independent
environments can both be created using the same flags that `spack env create`
accepts. If an environment already exists then spack will simply activate it
and ignore the create-specific flags.
.. code-block:: console .. code-block:: console
@ -168,49 +183,50 @@ or the shortcut alias
If the environment was activated with its view, deactivating the If the environment was activated with its view, deactivating the
environment will remove the view from the user environment. environment will remove the view from the user environment.
^^^^^^^^^^^^^^^^^^^^^^ .. _independent_environments:
Anonymous Environments
^^^^^^^^^^^^^^^^^^^^^^
Apart from managed environments, Spack also supports anonymous environments. ^^^^^^^^^^^^^^^^^^^^^^^^
Independent Environments
^^^^^^^^^^^^^^^^^^^^^^^^
Anonymous environments can be placed in any directory of choice. Independent environments can be located in any directory outside of Spack.
.. note:: .. note::
When uninstalling packages, Spack asks the user to confirm the removal of packages When uninstalling packages, Spack asks the user to confirm the removal of packages
that are still used in a managed environment. This is not the case for anonymous that are still used in a managed environment. This is not the case for independent
environments. environments.
To create an anonymous environment, use one of the following commands: To create an independent environment, use one of the following commands:
.. code-block:: console .. code-block:: console
$ spack env create --dir my_env $ spack env create --dir my_env
$ spack env create ./my_env $ spack env create ./my_env
As a shorthand, you can also create an anonymous environment upon activation if it does not As a shorthand, you can also create an independent environment upon activation if it does not
already exist: already exist:
.. code-block:: console .. code-block:: console
$ spack env activate --create ./my_env $ spack env activate --create ./my_env
For convenience, Spack can also place an anonymous environment in a temporary directory for you: For convenience, Spack can also place an independent environment in a temporary directory for you:
.. code-block:: console .. code-block:: console
$ spack env activate --temp $ spack env activate --temp
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^
Environment Sensitive Commands Environment-Aware Commands
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^
Spack commands are environment sensitive. For example, the ``find`` Spack commands are environment-aware. For example, the ``find``
command shows only the specs in the active Environment if an command shows only the specs in the active environment if an
Environment has been activated. Similarly, the ``install`` and environment has been activated. Otherwise it shows all specs in
``uninstall`` commands act on the active environment. the Spack instance. The same rule applies to the ``install`` and
``uninstall`` commands.
.. code-block:: console .. code-block:: console
@ -255,32 +271,33 @@ Environment has been activated. Similarly, the ``install`` and
Note that when we installed the abstract spec ``zlib@1.2.8``, it was Note that when we installed the abstract spec ``zlib@1.2.8``, it was
presented as a root of the Environment. All explicitly installed presented as a root of the environment. All explicitly installed
packages will be listed as roots of the Environment. packages will be listed as roots of the environment.
All of the Spack commands that act on the list of installed specs are All of the Spack commands that act on the list of installed specs are
Environment-sensitive in this way, including ``install``, environment-aware in this way, including ``install``,
``uninstall``, ``find``, ``extensions``, and more. In the ``uninstall``, ``find``, ``extensions``, etcetera. In the
:ref:`environment-configuration` section we will discuss :ref:`environment-configuration` section we will discuss
Environment-sensitive commands further. environment-aware commands further.
^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
Adding Abstract Specs Adding Abstract Specs
^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
An abstract spec is the user-specified spec before Spack has applied An abstract spec is the user-specified spec before Spack applies
any defaults or dependency information. defaults or dependency information.
Users can add abstract specs to an Environment using the ``spack add`` Users can add abstract specs to an environment using the ``spack add``
command. The most important component of an Environment is a list of command. The most important component of an environment is a list of
abstract specs. abstract specs.
Adding a spec adds to the manifest (the ``spack.yaml`` file), which is Adding a spec adds it as a root spec of the environment in the user
used to define the roots of the Environment, but does not affect the input file (``spack.yaml``). It does not affect the concrete specs
concrete specs in the lockfile, nor does it install the spec. in the lock file (``spack.lock``) and it does not install the spec.
The ``spack add`` command is environment aware. It adds to the The ``spack add`` command is environment-aware. It adds the spec to the
currently active environment. All environment aware commands can also currently active environment. An error is generated if there isn't an
active environment. All environment-aware commands can also
be called using the ``spack -e`` flag to specify the environment. be called using the ``spack -e`` flag to specify the environment.
.. code-block:: console .. code-block:: console
@ -300,11 +317,11 @@ or
Concretizing Concretizing
^^^^^^^^^^^^ ^^^^^^^^^^^^
Once some user specs have been added to an environment, they can be concretized. Once user specs have been added to an environment, they can be concretized.
There are at the moment three different modes of operation to concretize an environment, There are three different modes of operation to concretize an environment,
which are explained in details in :ref:`environments_concretization_config`. explained in detail in :ref:`environments_concretization_config`.
Regardless of which mode of operation has been chosen, the following Regardless of which mode of operation is chosen, the following
command will ensure all the root specs are concretized according to the command will ensure all of the root specs are concretized according to the
constraints that are prescribed in the configuration: constraints that are prescribed in the configuration:
.. code-block:: console .. code-block:: console
@ -313,16 +330,15 @@ constraints that are prescribed in the configuration:
In the case of specs that are not concretized together, the command In the case of specs that are not concretized together, the command
above will concretize only the specs that were added and not yet above will concretize only the specs that were added and not yet
concretized. Forcing a re-concretization of all the specs can be done concretized. Forcing a re-concretization of all of the specs can be done
instead with this command: by adding the ``-f`` option:
.. code-block:: console .. code-block:: console
[myenv]$ spack concretize -f [myenv]$ spack concretize -f
When the ``-f`` flag is not used to reconcretize all specs, Spack Without the option, Spack guarantees that already concretized specs are
guarantees that already concretized specs are unchanged in the unchanged in the environment.
environment.
The ``concretize`` command does not install any packages. For packages The ``concretize`` command does not install any packages. For packages
that have already been installed outside of the environment, the that have already been installed outside of the environment, the
@ -355,16 +371,16 @@ installed specs using the ``-c`` (``--concretized``) flag.
Installing an Environment Installing an Environment
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
In addition to installing individual specs into an Environment, one In addition to adding individual specs to an environment, one
can install the entire Environment at once using the command can install the entire environment at once using the command
.. code-block:: console .. code-block:: console
[myenv]$ spack install [myenv]$ spack install
If the Environment has been concretized, Spack will install the If the environment has been concretized, Spack will install the
concretized specs. Otherwise, ``spack install`` will first concretize concretized specs. Otherwise, ``spack install`` will concretize
the Environment and then install the concretized specs. the environment before installing the concretized specs.
.. note:: .. note::
@ -385,17 +401,17 @@ the Environment and then install the concretized specs.
As it installs, ``spack install`` creates symbolic links in the As it installs, ``spack install`` creates symbolic links in the
``logs/`` directory in the Environment, allowing for easy inspection ``logs/`` directory in the environment, allowing for easy inspection
of build logs related to that environment. The ``spack install`` of build logs related to that environment. The ``spack install``
command also stores a Spack repo containing the ``package.py`` file command also stores a Spack repo containing the ``package.py`` file
used at install time for each package in the ``repos/`` directory in used at install time for each package in the ``repos/`` directory in
the Environment. the environment.
The ``--no-add`` option can be used in a concrete environment to tell The ``--no-add`` option can be used in a concrete environment to tell
spack to install specs already present in the environment but not to spack to install specs already present in the environment but not to
add any new root specs to the environment. For root specs provided add any new root specs to the environment. For root specs provided
to ``spack install`` on the command line, ``--no-add`` is the default, to ``spack install`` on the command line, ``--no-add`` is the default,
while for dependency specs on the other hand, it is optional. In other while for dependency specs, it is optional. In other
words, if there is an unambiguous match in the active concrete environment words, if there is an unambiguous match in the active concrete environment
for a root spec provided to ``spack install`` on the command line, spack for a root spec provided to ``spack install`` on the command line, spack
does not require you to specify the ``--no-add`` option to prevent the spec does not require you to specify the ``--no-add`` option to prevent the spec
@ -430,7 +446,7 @@ also be used as valid concrete versions (see :ref:`version-specifier`).
This means that for a package ``foo``, ``spack develop foo@git.main`` will clone This means that for a package ``foo``, ``spack develop foo@git.main`` will clone
the ``main`` branch of the package, and ``spack install`` will install from the ``main`` branch of the package, and ``spack install`` will install from
that git clone if ``foo`` is in the environment. that git clone if ``foo`` is in the environment.
Further development on ``foo`` can be tested by reinstalling the environment, Further development on ``foo`` can be tested by re-installing the environment,
and eventually committed and pushed to the upstream git repo. and eventually committed and pushed to the upstream git repo.
If the package being developed supports out-of-source builds then users can use the If the package being developed supports out-of-source builds then users can use the
@ -615,7 +631,7 @@ manipulate configuration inline in the ``spack.yaml`` file.
Inline configurations Inline configurations
^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
Inline Environment-scope configuration is done using the same yaml Inline environment-scope configuration is done using the same yaml
format as standard Spack configuration scopes, covered in the format as standard Spack configuration scopes, covered in the
:ref:`configuration` section. Each section is contained under a :ref:`configuration` section. Each section is contained under a
top-level yaml object with it's name. For example, a ``spack.yaml`` top-level yaml object with it's name. For example, a ``spack.yaml``
@ -640,7 +656,7 @@ Included configurations
Spack environments allow an ``include`` heading in their yaml Spack environments allow an ``include`` heading in their yaml
schema. This heading pulls in external configuration files and applies schema. This heading pulls in external configuration files and applies
them to the Environment. them to the environment.
.. code-block:: yaml .. code-block:: yaml
@ -667,7 +683,7 @@ have higher precedence, as the included configs are applied in reverse order.
Manually Editing the Specs List Manually Editing the Specs List
------------------------------- -------------------------------
The list of abstract/root specs in the Environment is maintained in The list of abstract/root specs in the environment is maintained in
the ``spack.yaml`` manifest under the heading ``specs``. the ``spack.yaml`` manifest under the heading ``specs``.
.. code-block:: yaml .. code-block:: yaml
@ -775,7 +791,7 @@ evaluates to the cross-product of those specs. Spec matrices also
contain an ``excludes`` directive, which eliminates certain contain an ``excludes`` directive, which eliminates certain
combinations from the evaluated result. combinations from the evaluated result.
The following two Environment manifests are identical: The following two environment manifests are identical:
.. code-block:: yaml .. code-block:: yaml
@ -850,7 +866,7 @@ files are identical.
In short files like the example, it may be easier to simply list the In short files like the example, it may be easier to simply list the
included specs. However for more complicated examples involving many included specs. However for more complicated examples involving many
packages across many toolchains, separately factored lists make packages across many toolchains, separately factored lists make
Environments substantially more manageable. environments substantially more manageable.
Additionally, the ``-l`` option to the ``spack add`` command allows Additionally, the ``-l`` option to the ``spack add`` command allows
one to add to named lists in the definitions section of the manifest one to add to named lists in the definitions section of the manifest
@ -1066,7 +1082,7 @@ true``). The argument ``--without-view`` can be used to create an
environment without any view configured. environment without any view configured.
The ``spack env view`` command can be used to change the manage views The ``spack env view`` command can be used to change the manage views
of an Environment. The subcommand ``spack env view enable`` will add a of an environment. The subcommand ``spack env view enable`` will add a
view named ``default`` to an environment. It takes an optional view named ``default`` to an environment. It takes an optional
argument to specify the path for the new default view. The subcommand argument to specify the path for the new default view. The subcommand
``spack env view disable`` will remove the view named ``default`` from ``spack env view disable`` will remove the view named ``default`` from
@ -1234,7 +1250,7 @@ gets installed and is available for use in the ``env`` target.
$(SPACK) -e . env depfile -o $@ --make-prefix spack $(SPACK) -e . env depfile -o $@ --make-prefix spack
env: spack/env env: spack/env
$(info Environment installed!) $(info environment installed!)
clean: clean:
rm -rf spack.lock env.mk spack/ rm -rf spack.lock env.mk spack/

4
lib/spack/spack/cmd/env.py
View File

@ -270,7 +270,7 @@ def env_activate_setup_parser(subparser):
nargs="?", nargs="?",
default=None, default=None,
help=( help=(
"name of managed environment or directory of the anonymous env" "name of managed environment or directory of the independent env"
" (when using --dir/-d) to activate" " (when using --dir/-d) to activate"
), ),
) )
@ -540,7 +540,7 @@ def env_rename_setup_parser(subparser):
def env_rename(args): def env_rename(args):
"""Rename an environment. """Rename an environment.
This renames a managed environment or moves an anonymous environment. This renames a managed environment or moves an independent environment.
""" """
# Directory option has been specified # Directory option has been specified

4
lib/spack/spack/test/cmd/env.py
View File

@ -278,7 +278,7 @@ def test_env_rename_managed(capfd):
assert "baz" in out assert "baz" in out
def test_env_rename_anonymous(capfd, tmpdir): def test_env_rename_independent(capfd, tmpdir):
# Need real environment # Need real environment
with pytest.raises(spack.main.SpackCommandError): with pytest.raises(spack.main.SpackCommandError):
env("rename", "-d", "./non-existing", "./also-non-existing") env("rename", "-d", "./non-existing", "./also-non-existing")
@ -3547,7 +3547,7 @@ def test_create_and_activate_managed(tmp_path):
env("deactivate") env("deactivate")
def test_create_and_activate_anonymous(tmp_path): def test_create_and_activate_independent(tmp_path):
with fs.working_dir(str(tmp_path)): with fs.working_dir(str(tmp_path)):
env_dir = os.path.join(str(tmp_path), "foo") env_dir = os.path.join(str(tmp_path), "foo")
shell = env("activate", "--without-view", "--create", "--sh", env_dir) shell = env("activate", "--without-view", "--create", "--sh", env_dir)