zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Cap the maximum number of build jobs (#11373)

Browse Source 
* config:build_jobs now controls the number of parallel jobs to spawn during
builds, but cannot ever exceed the number of cores on the machine.

* The default is set to 16 or the number of available cores, whatever
is lowest.

* Updated docs to reflect the changes done to limit parallel builds
This commit is contained in:
Massimiliano Culpo
2019-05-28 15:42:04 +02:00
committed by Todd Gamblin
parent 2a51e07fde
commit 01ece824e1
6 changed files with 55 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
lib/spack/docs/config_yaml.rst
View File

@@ -178,16 +178,23 @@ set ``dirty`` to ``true`` to skip the cleaning step and make all builds
"dirty" by default. Be aware that this will reduce the reproducibility
of builds.
.. _build-jobs:
--------------
``build_jobs``
--------------
Unless overridden in a package or on the command line, Spack builds all
packages in parallel. For a build system that uses Makefiles, this means
running ``make -j<build_jobs>``, where ``build_jobs`` is the number of
threads to use.
packages in parallel. The default parallelism is equal to the number of
cores on your machine, up to 16. Parallelism cannot exceed the number of
cores available on the host. For a build system that uses Makefiles, this
means running:
- ``make -j<build_jobs>``, when ``build_jobs`` is less than the number of
cores on the machine
- ``make -j<ncores>``, when ``build_jobs`` is greater or equal to the
number of cores on the machine
The default parallelism is equal to the number of cores on your machine.
If you work on a shared login node or have a strict ulimit, it may be
necessary to set the default to a lower value. By setting ``build_jobs``
to 4, for example, commands like ``spack install`` will run ``make -j4``

11
lib/spack/docs/packaging_guide.rst
View File

@@ -1713,12 +1713,11 @@ RPATHs in Spack are handled in one of three ways:
Parallel builds
---------------
By default, Spack will invoke ``make()`` with a ``-j <njobs>``
argument, so that builds run in parallel. It figures out how many
jobs to run by determining how many cores are on the host machine.
Specifically, it uses the number of CPUs reported by Python's
`multiprocessing.cpu_count()
<http://docs.python.org/library/multiprocessing.html#multiprocessing.cpu_count>`_.
By default, Spack will invoke ``make()``, or any other similar tool,
with a ``-j <njobs>`` argument, so that builds run in parallel.
The parallelism is determined by the value of the ``build_jobs`` entry
in ``config.yaml`` (see :ref:`here <build-jobs>` for more details on
how this value is computed).
If a package does not build properly in parallel, you can override
this setting by adding ``parallel = False`` to your package. For