Distributed builds (#13100)

Fixes #9394
Closes #13217.

## Background
Spack provides the ability to enable/disable parallel builds through two options: package `parallel` and configuration `build_jobs`.  This PR changes the algorithm to allow multiple, simultaneous processes to coordinate the installation of the same spec (and specs with overlapping dependencies.).

The `parallel` (boolean) property sets the default for its package though the value can be overridden in the `install` method.

Spack's current parallel builds are limited to build tools supporting `jobs` arguments (e.g., `Makefiles`).  The number of jobs actually used is calculated as`min(config:build_jobs, # cores, 16)`, which can be overridden in the package or on the command line (i.e., `spack install -j <# jobs>`).

This PR adds support for distributed (single- and multi-node) parallel builds.  The goals of this work include improving the efficiency of installing packages with many dependencies and reducing the repetition associated with concurrent installations of (dependency) packages.

## Approach
### File System Locks
Coordination between concurrent installs of overlapping packages to a Spack instance is accomplished through bottom-up dependency DAG processing and file system locks.  The runs can be a combination of interactive and batch processes affecting the same file system.  Exclusive prefix locks are required to install a package while shared prefix locks are required to check if the package is installed.

Failures are communicated through a separate exclusive prefix failure lock, for concurrent processes, combined with a persistent store, for separate, related build processes.  The resulting file contains the failing spec to facilitate manual debugging.

### Priority Queue
Management of dependency builds changed from reliance on recursion to use of a priority queue where the priority of a spec is based on the number of its remaining uninstalled dependencies.  

Using a queue required a change to dependency build exception handling with the most visible issue being that the `install` method *must* install something in the prefix.  Consequently, packages can no longer get away with an install method consisting of `pass`, for example.

## Caveats
- This still only parallelizes a single-rooted build.  Multi-rooted installs (e.g., for environments) are TBD in a future PR.

Tasks:
- [x] Adjust package lock timeout to correspond to value used in the demo
- [x] Adjust database lock timeout to reduce contention on startup of concurrent
    `spack install <spec>` calls
- [x] Replace (test) package's `install: pass` methods with file creation since post-install 
    `sanity_check_prefix` will otherwise error out with `Install failed .. Nothing was installed!`
- [x] Resolve remaining existing test failures
- [x] Respond to alalazo's initial feedback
- [x] Remove `bin/demo-locks.py`
- [x] Add new tests to address new coverage issues
- [x] Replace built-in package's `def install(..): pass` to "install" something
    (i.e., only `apple-libunwind`)
- [x] Increase code coverage
This commit is contained in:
Tamara Dahlgren
2020-02-19 00:04:22 -08:00
committed by GitHub
parent 2f4881d582
commit f2aca86502
100 changed files with 2954 additions and 963 deletions

View File

@@ -49,4 +49,6 @@ def build(self, spec, prefix):
pass
def install(self, spec, prefix):
pass
# sanity_check_prefix requires something in the install directory
# Test requires overriding the one provided by `AutotoolsPackage`
mkdirp(prefix.bin)

View File

@@ -13,6 +13,3 @@ class B(Package):
url = "http://www.example.com/b-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -59,6 +59,3 @@ class Boost(Package):
description="Build the Boost Graph library")
variant('taggedlayout', default=False,
description="Augment library names with build options")
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class C(Package):
url = "http://www.example.com/c-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -17,6 +17,3 @@ class ConflictingDependent(Package):
version('1.0', '0123456789abcdef0123456789abcdef')
depends_on('dependency-install@:1.0')
def install(self, spec, prefix):
pass

View File

@@ -25,6 +25,3 @@ class DepDiamondPatchMid1(Package):
# single patch file in repo
depends_on('patch', patches='mid1.patch')
def install(self, spec, prefix):
pass

View File

@@ -28,6 +28,3 @@ class DepDiamondPatchMid2(Package):
patch('http://example.com/urlpatch.patch',
sha256='mid21234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234'), # noqa: E501
])
def install(self, spec, prefix):
pass

View File

@@ -27,6 +27,3 @@ class DepDiamondPatchTop(Package):
depends_on('patch', patches='top.patch')
depends_on('dep-diamond-patch-mid1')
depends_on('dep-diamond-patch-mid2')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class DevelopTest(Package):
version('develop', git='https://github.com/dummy/repo.git')
version('0.2.15', 'b1190f3d3471685f17cfd1ec1d252ac9')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class DevelopTest2(Package):
version('0.2.15.develop', git='https://github.com/dummy/repo.git')
version('0.2.15', 'b1190f3d3471685f17cfd1ec1d252ac9')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class DirectMpich(Package):
version('1.0', 'foobarbaz')
depends_on('mpich')
def install(self, spec, prefix):
pass

View File

@@ -12,6 +12,3 @@ class DtDiamondBottom(Package):
url = "http://www.example.com/dt-diamond-bottom-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -14,6 +14,3 @@ class DtDiamondLeft(Package):
version('1.0', '0123456789abcdef0123456789abcdef')
depends_on('dt-diamond-bottom', type='build')
def install(self, spec, prefix):
pass

View File

@@ -14,6 +14,3 @@ class DtDiamondRight(Package):
version('1.0', '0123456789abcdef0123456789abcdef')
depends_on('dt-diamond-bottom', type=('build', 'link', 'run'))
def install(self, spec, prefix):
pass

View File

@@ -15,6 +15,3 @@ class DtDiamond(Package):
depends_on('dt-diamond-left')
depends_on('dt-diamond-right')
def install(self, spec, prefix):
pass

View File

@@ -18,6 +18,3 @@ class Dtbuild1(Package):
depends_on('dtbuild2', type='build')
depends_on('dtlink2')
depends_on('dtrun2', type='run')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class Dtbuild2(Package):
url = "http://www.example.com/dtbuild2-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class Dtbuild3(Package):
url = "http://www.example.com/dtbuild3-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -15,6 +15,3 @@ class Dtlink1(Package):
version('1.0', '0123456789abcdef0123456789abcdef')
depends_on('dtlink3')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class Dtlink2(Package):
url = "http://www.example.com/dtlink2-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -16,6 +16,3 @@ class Dtlink3(Package):
depends_on('dtbuild2', type='build')
depends_on('dtlink4')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class Dtlink4(Package):
url = "http://www.example.com/dtlink4-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class Dtlink5(Package):
url = "http://www.example.com/dtlink5-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -16,6 +16,3 @@ class Dtrun1(Package):
depends_on('dtlink5')
depends_on('dtrun3', type='run')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class Dtrun2(Package):
url = "http://www.example.com/dtrun2-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -15,6 +15,3 @@ class Dtrun3(Package):
version('1.0', '0123456789abcdef0123456789abcdef')
depends_on('dtbuild3', type='build')
def install(self, spec, prefix):
pass

View File

@@ -17,6 +17,3 @@ class Dttop(Package):
depends_on('dtbuild1', type='build')
depends_on('dtlink1')
depends_on('dtrun1', type='run')
def install(self, spec, prefix):
pass

View File

@@ -15,6 +15,3 @@ class Dtuse(Package):
version('1.0', '0123456789abcdef0123456789abcdef')
depends_on('dttop')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class E(Package):
url = "http://www.example.com/e-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class Externalmodule(Package):
version('1.0', '1234567890abcdef1234567890abcdef')
depends_on('externalprereq')
def install(self, spec, prefix):
pass

View File

@@ -11,6 +11,3 @@ class Externalprereq(Package):
url = "http://somewhere.com/prereq-1.0.tar.gz"
version('1.4', 'f1234567890abcdef1234567890abcde')
def install(self, spec, prefix):
pass

View File

@@ -14,6 +14,3 @@ class Externaltool(Package):
version('0.9', '1234567890abcdef1234567890abcdef')
depends_on('externalprereq')
def install(self, spec, prefix):
pass

View File

@@ -16,6 +16,3 @@ class Externalvirtual(Package):
version('2.2', '4567890abcdef1234567890abcdef123')
provides('stuff', when='@1.0:')
def install(self, spec, prefix):
pass

View File

@@ -11,6 +11,3 @@ class Fake(Package):
url = "http://www.fake-spack-example.org/downloads/fake-1.0.tar.gz"
version('1.0', 'foobarbaz')
def install(self, spec, prefix):
pass

View File

@@ -58,7 +58,11 @@ def install(self, spec, prefix):
if 'really-long-if-statement' != 'that-goes-over-the-line-length-limit-and-requires-noqa': # noqa
pass
# sanity_check_prefix requires something in the install directory
mkdirp(prefix.bin)
# '@when' decorated functions are exempt from redefinition errors
@when('@2.0')
def install(self, spec, prefix):
pass
# sanity_check_prefix requires something in the install directory
mkdirp(prefix.bin)

View File

@@ -15,6 +15,3 @@ class GitSvnTopLevel(Package):
svn = 'https://example.com/some/svn/repo'
version('2.0')
def install(self, spec, prefix):
pass

View File

@@ -11,6 +11,3 @@ class GitTest(Package):
homepage = "http://www.git-fetch-example.com"
version('git', git='to-be-filled-in-by-test')
def install(self, spec, prefix):
pass

View File

@@ -12,6 +12,3 @@ class GitTopLevel(Package):
git = 'https://example.com/some/git/repo'
version('1.0')
def install(self, spec, prefix):
pass

View File

@@ -16,6 +16,3 @@ class GitUrlSvnTopLevel(Package):
svn = 'https://example.com/some/svn/repo'
version('2.0')
def install(self, spec, prefix):
pass

View File

@@ -38,6 +38,3 @@ class GitUrlTopLevel(Package):
version('1.2', sha512='abc12', branch='releases/v1.2')
version('1.1', md5='abc11', tag='v1.1')
version('1.0', 'abc11', tag='abc123')
def install(self, spec, prefix):
pass

View File

@@ -3,10 +3,10 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
import os
from spack import *
class HashTest1(Package):
"""Used to test package hashing
@@ -37,10 +37,16 @@ def install(self, spec, prefix):
print("install 1")
os.listdir(os.getcwd())
# sanity_check_prefix requires something in the install directory
mkdirp(prefix.bin)
@when('@1.5:')
def install(self, spec, prefix):
os.listdir(os.getcwd())
# sanity_check_prefix requires something in the install directory
mkdirp(prefix.bin)
@when('@1.5,1.6')
def extra_phase(self, spec, prefix):
pass

View File

@@ -3,10 +3,10 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
import os
from spack import *
class HashTest2(Package):
"""Used to test package hashing
@@ -31,3 +31,6 @@ def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
def install(self, spec, prefix):
print("install 1")
os.listdir(os.getcwd())
# sanity_check_prefix requires something in the install directory
mkdirp(prefix.bin)

View File

@@ -3,10 +3,10 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
import os
from spack import *
class HashTest3(Package):
"""Used to test package hashing
@@ -32,10 +32,16 @@ def install(self, spec, prefix):
print("install 1")
os.listdir(os.getcwd())
# sanity_check_prefix requires something in the install directory
mkdirp(prefix.bin)
@when('@1.5:')
def install(self, spec, prefix):
os.listdir(os.getcwd())
# sanity_check_prefix requires something in the install directory
mkdirp(prefix.bin)
for _version_constraint in ['@1.5', '@1.6']:
@when(_version_constraint)
def extra_phase(self, spec, prefix):

View File

@@ -11,6 +11,3 @@ class HgTest(Package):
homepage = "http://www.hg-fetch-example.com"
version('hg', hg='to-be-filled-in-by-test')
def install(self, spec, prefix):
pass

View File

@@ -12,6 +12,3 @@ class HgTopLevel(Package):
hg = 'https://example.com/some/hg/repo'
version('1.0')
def install(self, spec, prefix):
pass

View File

@@ -16,6 +16,3 @@ class Hypre(Package):
depends_on('lapack')
depends_on('blas')
def install(self, spec, prefix):
pass

View File

@@ -18,6 +18,3 @@ class IndirectMpich(Package):
depends_on('mpi')
depends_on('direct-mpich')
def install(self, spec, prefix):
pass

View File

@@ -15,6 +15,3 @@ class Maintainers1(Package):
maintainers = ['user1', 'user2']
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -15,6 +15,3 @@ class Maintainers2(Package):
maintainers = ['user2', 'user3']
version('1.0', '0123456789abcdef0123456789abcdef')
def install(self, spec, prefix):
pass

View File

@@ -12,6 +12,3 @@ class Mixedversions(Package):
version('2.0.1', 'hashc')
version('2.0', 'hashb')
version('1.0.1', 'hasha')
def install(self, spec, prefix):
pass

View File

@@ -12,9 +12,6 @@ class ModulePathSeparator(Package):
version(1.0, 'foobarbaz')
def install(self, spec, prefix):
pass
def setup_environment(self, senv, renv):
renv.append_path("COLON", "foo")
renv.prepend_path("COLON", "foo")

View File

@@ -27,6 +27,3 @@ class MultiProviderMpi(Package):
provides('mpi@3.0', when='@1.10.0')
provides('mpi@3.0', when='@1.8.8')
provides('mpi@2.2', when='@1.6.5')
def install(self, spec, prefix):
pass

View File

@@ -15,6 +15,3 @@ class MultimoduleInheritance(si.BaseWithDirectives):
version('1.0', '0123456789abcdef0123456789abcdef')
depends_on('openblas', when='+openblas')
def install(self, spec, prefix):
pass

View File

@@ -33,6 +33,3 @@ class MultivalueVariant(Package):
depends_on('callpath')
depends_on('a')
depends_on('a@1.0', when='fee=barbaz')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class NetlibBlas(Package):
version('3.5.0', 'b1d3e3e425b2e44a06760ff173104bdf')
provides('blas')
def install(self, spec, prefix):
pass

View File

@@ -14,6 +14,3 @@ class NetlibLapack(Package):
provides('lapack')
depends_on('blas')
def install(self, spec, prefix):
pass

View File

@@ -3,10 +3,7 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import os
from spack import *
from llnl.util.filesystem import touch
class NosourceInstall(BundlePackage):
@@ -24,8 +21,8 @@ class NosourceInstall(BundlePackage):
# The install method must also be present.
def install(self, spec, prefix):
touch(os.path.join(self.prefix, 'install.txt'))
touch(join_path(self.prefix, 'install.txt'))
@run_after('install')
def post_install(self):
touch(os.path.join(self.prefix, 'post-install.txt'))
touch(join_path(self.prefix, 'post-install.txt'))

View File

@@ -15,6 +15,3 @@ class OpenblasWithLapack(Package):
provides('lapack')
provides('blas')
def install(self, spec, prefix):
pass

View File

@@ -14,6 +14,3 @@ class Openblas(Package):
version('0.2.15', 'b1190f3d3471685f17cfd1ec1d252ac9')
provides('blas')
def install(self, spec, prefix):
pass

View File

@@ -19,6 +19,3 @@ class OptionalDepTest2(Package):
depends_on('optional-dep-test', when='+odt')
depends_on('optional-dep-test+mpi', when='+mpi')
def install(self, spec, prefix):
pass

View File

@@ -18,6 +18,3 @@ class OptionalDepTest3(Package):
depends_on('a', when='~var')
depends_on('b', when='+var')
def install(self, spec, prefix):
pass

View File

@@ -30,6 +30,3 @@ class OptionalDepTest(Package):
depends_on('mpi', when='^g')
depends_on('mpi', when='+mpi')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class Othervirtual(Package):
version('1.0', '67890abcdef1234567890abcdef12345')
provides('stuff')
def install(self, spec, prefix):
pass

View File

@@ -18,6 +18,3 @@ class OverrideContextTemplates(Package):
tcl_template = 'extension.tcl'
tcl_context = {'sentence': "sentence from package"}
def install(self, spec, prefix):
pass

View File

@@ -14,6 +14,3 @@ class OverrideModuleTemplates(Package):
tcl_template = 'override.txt'
lmod_template = 'override.txt'
def install(self, spec, prefix):
pass

View File

@@ -15,6 +15,3 @@ class PatchADependency(Package):
version('1.0', '0123456789abcdef0123456789abcdef')
depends_on('libelf', patches=patch('libelf.patch'))
def install(self, spec, prefix):
pass

View File

@@ -36,6 +36,3 @@ class PatchSeveralDependencies(Package):
archive_sha256='abcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcdabcd',
sha256='1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd1234abcd')
])
def install(self, spec, prefix):
pass

View File

@@ -21,6 +21,3 @@ class Patch(Package):
patch('bar.patch', when='@2:')
patch('baz.patch')
patch('biz.patch', when='@1.0.1:1.0.2')
def install(self, spec, prefix):
pass

View File

@@ -13,6 +13,3 @@ class Perl(Package):
extendable = True
version('0.0.0', 'hash')
def install(self, spec, prefix):
pass

View File

@@ -15,6 +15,3 @@ class PreferredTest(Package):
version('0.2.16', 'b1190f3d3471685f17cfd1ec1d252ac9')
version('0.2.15', 'b1190f3d3471685f17cfd1ec1d252ac9', preferred=True)
version('0.2.14', 'b1190f3d3471685f17cfd1ec1d252ac9')
def install(self, spec, prefix):
pass

View File

@@ -19,6 +19,3 @@ class Python(Package):
version('2.7.10', 'd7547558fd673bd9d38e2108c6b42521')
version('2.7.9', '5eebcaa0030dc4061156d3429657fb83')
version('2.7.8', 'd4bca0159acb0b44a781292b5231936f')
def install(self, spec, prefix):
pass

View File

@@ -30,6 +30,3 @@ class SimpleInheritance(BaseWithDirectives):
depends_on('openblas', when='+openblas')
provides('lapack', when='+openblas')
def install(self, spec, prefix):
pass

View File

@@ -15,6 +15,3 @@ class SinglevalueVariantDependent(Package):
version('1.0', '0123456789abcdef0123456789abcdef')
depends_on('multivalue_variant fee=baz')
def install(self, spec, prefix):
pass

View File

@@ -11,6 +11,3 @@ class SvnTest(Package):
url = "http://www.example.com/svn-test-1.0.tar.gz"
version('svn', svn='to-be-filled-in-by-test')
def install(self, spec, prefix):
pass

View File

@@ -11,6 +11,3 @@ class SvnTopLevel(Package):
svn = 'https://example.com/some/svn/repo'
version('1.0')
def install(self, spec, prefix):
pass

View File

@@ -5,7 +5,6 @@
from spack import *
import os
import spack.paths
@@ -13,7 +12,7 @@ class UrlListTest(Package):
"""Mock package with url_list."""
homepage = "http://www.url-list-example.com"
web_data_path = os.path.join(spack.paths.test_path, 'data', 'web')
web_data_path = join_path(spack.paths.test_path, 'data', 'web')
url = 'file://' + web_data_path + '/foo-0.0.0.tar.gz'
list_url = 'file://' + web_data_path + '/index.html'
list_depth = 3
@@ -25,6 +24,3 @@ class UrlListTest(Package):
version('2.0.0b2', 'abc200b2')
version('3.0a1', 'abc30a1')
version('4.5-rc5', 'abc45rc5')
def install(self, spec, prefix):
pass

View File

@@ -11,6 +11,3 @@ class UrlTest(Package):
homepage = "http://www.url-fetch-example.com"
version('test', url='to-be-filled-in-by-test')
def install(self, spec, prefix):
pass

View File

@@ -23,6 +23,3 @@ class WhenDirectivesFalse(Package):
resource(url="http://www.example.com/example-1.0-resource.tar.gz",
md5='0123456789abcdef0123456789abcdef',
when=False)
def install(self, spec, prefix):
pass

View File

@@ -23,6 +23,3 @@ class WhenDirectivesTrue(Package):
resource(url="http://www.example.com/example-1.0-resource.tar.gz",
md5='0123456789abcdef0123456789abcdef',
when=True)
def install(self, spec, prefix):
pass

View File

@@ -16,6 +16,3 @@ class Zmpi(Package):
provides('mpi@:10.0')
depends_on('fake')
def install(self, spec, prefix):
pass

View File

@@ -43,7 +43,8 @@ def fetcher(self):
raise InstallError(msg)
def install(self, spec, prefix):
pass
# sanity_check_prefix requires something in the install directory
mkdirp(prefix.lib)
@property
def libs(self):