Support parallel environment builds (#18131)
As of #13100, Spack installs the dependencies of a _single_ spec in parallel. Environments, when installed, can only get parallelism from each individual spec, as they're installed in order. This PR makes entire environments build in parallel by extending Spack's package installer to accept multiple root specs. The install command and Environment class have been updated to use the new parallel install method. The specs and kwargs for each *uninstalled* package (when not force-replacing installations) of an environment are collected, passed to the `PackageInstaller`, and processed using a single build queue. This introduces a `BuildRequest` class to track install arguments, and it significantly cleans up the code used to track package ids during installation. Package ids in the build queue are now just DAG hashes as you would expect, Other tasks: - [x] Finish updating the unit tests based on `PackageInstaller`'s use of `BuildRequest` and the associated changes - [x] Change `environment.py`'s `install_all` to use the `PackageInstaller` directly - [x] Change the `install` command to leverage the new installation process for multiple specs - [x] Change install output messages for external packages, e.g.: `[+] /usr` -> `[+] /usr (external bzip2-1.0.8-<dag-hash>` - [x] Fix incomplete environment install's view setup/update and not confirming all packages are installed (?) - [x] Ensure externally installed package dependencies are properly accounted for in remaining build tasks - [x] Add tests for coverage (if insufficient and can identity the appropriate, uncovered non-comment lines) - [x] Add documentation - [x] Resolve multi-compiler environment install issues - [x] Fix issue with environment installation reporting (restore CDash/JUnit reports)
This commit is contained in:
Tamara Dahlgren
@@ -132,32 +132,28 @@ If ``mpileaks`` depends on other packages, Spack will install the
|
||||
dependencies first. It then fetches the ``mpileaks`` tarball, expands
|
||||
it, verifies that it was downloaded without errors, builds it, and
|
||||
installs it in its own directory under ``$SPACK_ROOT/opt``. You'll see
|
||||
a number of messages from spack, a lot of build output, and a message
|
||||
that the packages is installed:
|
||||
a number of messages from Spack, a lot of build output, and a message
|
||||
that the package is installed. Add one or more debug options (``-d``)
|
||||
to get increasingly detailed output.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack install mpileaks
|
||||
==> Installing mpileaks
|
||||
==> mpich is already installed in ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpich@3.0.4.
|
||||
==> callpath is already installed in ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/callpath@1.0.2-5dce4318.
|
||||
==> adept-utils is already installed in ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/adept-utils@1.0-5adef8da.
|
||||
==> Trying to fetch from https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz
|
||||
######################################################################## 100.0%
|
||||
==> Staging archive: ~/spack/var/spack/stage/mpileaks@1.0%gcc@4.4.7 arch=linux-debian7-x86_64-59f6ad23/mpileaks-1.0.tar.gz
|
||||
==> Created stage in ~/spack/var/spack/stage/mpileaks@1.0%gcc@4.4.7 arch=linux-debian7-x86_64-59f6ad23.
|
||||
==> No patches needed for mpileaks.
|
||||
==> Building mpileaks.
|
||||
|
||||
... build output ...
|
||||
|
||||
==> Successfully installed mpileaks.
|
||||
Fetch: 2.16s. Build: 9.82s. Total: 11.98s.
|
||||
[+] ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpileaks@1.0-59f6ad23
|
||||
... dependency build output ...
|
||||
==> Installing mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2
|
||||
==> No binary for mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2 found: installing from source
|
||||
==> mpileaks: Executing phase: 'autoreconf'
|
||||
==> mpileaks: Executing phase: 'configure'
|
||||
==> mpileaks: Executing phase: 'build'
|
||||
==> mpileaks: Executing phase: 'install'
|
||||
[+] ~/spack/opt/linux-rhel7-broadwell/gcc-8.1.0/mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2
|
||||
|
||||
The last line, with the ``[+]``, indicates where the package is
|
||||
installed.
|
||||
|
||||
Add the debug option -- ``spack install -d mpileaks`` -- to get additional
|
||||
output.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Building a specific version
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
@@ -191,44 +191,24 @@ Environment has been activated. Similarly, the ``install`` and
|
||||
==> 0 installed packages
|
||||
|
||||
$ spack install zlib@1.2.11
|
||||
==> Installing zlib
|
||||
==> Searching for binary cache of zlib
|
||||
==> Warning: No Spack mirrors are currently configured
|
||||
==> No binary for zlib found: installing from source
|
||||
==> Fetching http://zlib.net/fossils/zlib-1.2.11.tar.gz
|
||||
######################################################################## 100.0%
|
||||
==> Staging archive: /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur/zlib-1.2.11.tar.gz
|
||||
==> Created stage in /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur
|
||||
==> No patches needed for zlib
|
||||
==> Building zlib [Package]
|
||||
==> Executing phase: 'install'
|
||||
==> Successfully installed zlib
|
||||
Fetch: 0.36s. Build: 11.58s. Total: 11.93s.
|
||||
[+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur
|
||||
==> Installing zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv
|
||||
==> No binary for zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv found: installing from source
|
||||
==> zlib: Executing phase: 'install'
|
||||
[+] ~/spack/opt/spack/linux-rhel7-broadwell/gcc-8.1.0/zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv
|
||||
|
||||
$ spack env activate myenv
|
||||
|
||||
$ spack find
|
||||
==> In environment myenv
|
||||
==> No root specs
|
||||
|
||||
==> 0 installed packages
|
||||
|
||||
$ spack install zlib@1.2.8
|
||||
==> Installing zlib
|
||||
==> Searching for binary cache of zlib
|
||||
==> Warning: No Spack mirrors are currently configured
|
||||
==> No binary for zlib found: installing from source
|
||||
==> Fetching http://zlib.net/fossils/zlib-1.2.8.tar.gz
|
||||
######################################################################## 100.0%
|
||||
==> Staging archive: /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7/zlib-1.2.8.tar.gz
|
||||
==> Created stage in /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7
|
||||
==> No patches needed for zlib
|
||||
==> Building zlib [Package]
|
||||
==> Executing phase: 'install'
|
||||
==> Successfully installed zlib
|
||||
Fetch: 0.26s. Build: 2.08s. Total: 2.35s.
|
||||
[+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7
|
||||
==> Installing zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x
|
||||
==> No binary for zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x found: installing from source
|
||||
==> zlib: Executing phase: 'install'
|
||||
[+] ~/spack/opt/spack/linux-rhel7-broadwell/gcc-8.1.0/zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x
|
||||
==> Updating view at ~/spack/var/spack/environments/myenv/.spack-env/view
|
||||
|
||||
$ spack find
|
||||
==> In environment myenv
|
||||
@@ -236,15 +216,17 @@ Environment has been activated. Similarly, the ``install`` and
|
||||
zlib@1.2.8
|
||||
|
||||
==> 1 installed package
|
||||
-- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
|
||||
-- linux-rhel7-broadwell / gcc@8.1.0 ----------------------------
|
||||
zlib@1.2.8
|
||||
|
||||
$ despacktivate
|
||||
|
||||
$ spack find
|
||||
==> 2 installed packages
|
||||
-- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
|
||||
-- linux-rhel7-broadwell / gcc@8.1.0 ----------------------------
|
||||
zlib@1.2.8 zlib@1.2.11
|
||||
|
||||
|
||||
Note that when we installed the abstract spec ``zlib@1.2.8``, it was
|
||||
presented as a root of the Environment. All explicitly installed
|
||||
packages will be listed as roots of the Environment.
|
||||
@@ -349,6 +331,9 @@ installed specs using the ``-c`` (``--concretized``) flag.
|
||||
|
||||
==> 0 installed packages
|
||||
|
||||
|
||||
.. _installing-environment:
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Installing an Environment
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
@@ -1778,8 +1778,18 @@ RPATHs in Spack are handled in one of three ways:
|
||||
Parallel builds
|
||||
---------------
|
||||
|
||||
Spack supports parallel builds on an individual package and at the
|
||||
installation level. Package-level parallelism is established by the
|
||||
``--jobs`` option and its configuration and package recipe equivalents.
|
||||
Installation-level parallelism is driven by the DAG(s) of the requested
|
||||
package or packages.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Package-level build parallelism
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
By default, Spack will invoke ``make()``, or any other similar tool,
|
||||
with a ``-j <njobs>`` argument, so that builds run in parallel.
|
||||
with a ``-j <njobs>`` argument, so those builds run in parallel.
|
||||
The parallelism is determined by the value of the ``build_jobs`` entry
|
||||
in ``config.yaml`` (see :ref:`here <build-jobs>` for more details on
|
||||
how this value is computed).
|
||||
@@ -1827,6 +1837,43 @@ you set ``parallel`` to ``False`` at the package level, then each call
|
||||
to ``make()`` will be sequential by default, but packagers can call
|
||||
``make(parallel=True)`` to override it.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Install-level build parallelism
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Spack supports the concurrent installation of packages within a Spack
|
||||
instance across multiple processes using file system locks. This
|
||||
parallelism is separate from the package-level achieved through build
|
||||
systems' use of the ``-j <njobs>`` option. With install-level parallelism,
|
||||
processes coordinate the installation of the dependencies of specs
|
||||
provided on the command line and as part of an environment build with
|
||||
only **one process** being allowed to install a given package at a time.
|
||||
Refer to :ref:`Dependencies` for more information on dependencies and
|
||||
:ref:`installing-environment` for how to install an environment.
|
||||
|
||||
Concurrent processes may be any combination of interactive sessions and
|
||||
batch jobs. Which means a ``spack install`` can be running in a terminal
|
||||
window while a batch job is running ``spack install`` on the same or
|
||||
overlapping dependencies without any process trying to re-do the work of
|
||||
another.
|
||||
|
||||
For example, if you are using SLURM, you could launch an installation
|
||||
of ``mpich`` using the following command:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ srun -N 2 -n 8 spack install -j 4 mpich@3.3.2
|
||||
|
||||
This will create eight concurrent four-job installation on two different
|
||||
nodes.
|
||||
|
||||
.. note::
|
||||
|
||||
The effective parallelism will be based on the maximum number of
|
||||
packages that can be installed at the same time, which will limited
|
||||
by the number of packages with no (remaining) uninstalled dependencies.
|
||||
|
||||
|
||||
.. _dependencies:
|
||||
|
||||
------------
|
||||
|
||||
Reference in New Issue
Block a user