f2aca86502
Fixes #9394 Closes #13217. ## Background Spack provides the ability to enable/disable parallel builds through two options: package `parallel` and configuration `build_jobs`. This PR changes the algorithm to allow multiple, simultaneous processes to coordinate the installation of the same spec (and specs with overlapping dependencies.). The `parallel` (boolean) property sets the default for its package though the value can be overridden in the `install` method. Spack's current parallel builds are limited to build tools supporting `jobs` arguments (e.g., `Makefiles`). The number of jobs actually used is calculated as`min(config:build_jobs, # cores, 16)`, which can be overridden in the package or on the command line (i.e., `spack install -j <# jobs>`). This PR adds support for distributed (single- and multi-node) parallel builds. The goals of this work include improving the efficiency of installing packages with many dependencies and reducing the repetition associated with concurrent installations of (dependency) packages. ## Approach ### File System Locks Coordination between concurrent installs of overlapping packages to a Spack instance is accomplished through bottom-up dependency DAG processing and file system locks. The runs can be a combination of interactive and batch processes affecting the same file system. Exclusive prefix locks are required to install a package while shared prefix locks are required to check if the package is installed. Failures are communicated through a separate exclusive prefix failure lock, for concurrent processes, combined with a persistent store, for separate, related build processes. The resulting file contains the failing spec to facilitate manual debugging. ### Priority Queue Management of dependency builds changed from reliance on recursion to use of a priority queue where the priority of a spec is based on the number of its remaining uninstalled dependencies. Using a queue required a change to dependency build exception handling with the most visible issue being that the `install` method *must* install something in the prefix. Consequently, packages can no longer get away with an install method consisting of `pass`, for example. ## Caveats - This still only parallelizes a single-rooted build. Multi-rooted installs (e.g., for environments) are TBD in a future PR. Tasks: - [x] Adjust package lock timeout to correspond to value used in the demo - [x] Adjust database lock timeout to reduce contention on startup of concurrent `spack install <spec>` calls - [x] Replace (test) package's `install: pass` methods with file creation since post-install `sanity_check_prefix` will otherwise error out with `Install failed .. Nothing was installed!` - [x] Resolve remaining existing test failures - [x] Respond to alalazo's initial feedback - [x] Remove `bin/demo-locks.py` - [x] Add new tests to address new coverage issues - [x] Replace built-in package's `def install(..): pass` to "install" something (i.e., only `apple-libunwind`) - [x] Increase code coverage
81 lines
2.9 KiB
Python
81 lines
2.9 KiB
Python
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
|
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
|
#
|
|
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
|
|
|
from spack import *
|
|
|
|
|
|
class AppleLibunwind(Package):
|
|
"""Placeholder package for Apple's analogue to non-GNU libunwind"""
|
|
|
|
homepage = "https://opensource.apple.com/source/libunwind/libunwind-35.3/"
|
|
|
|
provides('unwind')
|
|
|
|
# The 'conflicts' directive only accepts valid spack specs;
|
|
# platforms cannot be negated -- 'platform!=darwin' is not a valid
|
|
# spec -- so expressing a conflict for any platform that isn't
|
|
# Darwin must be expressed by listing a conflict with every
|
|
# platform that isn't Darwin/macOS
|
|
conflicts('platform=linux')
|
|
conflicts('platform=bgq')
|
|
conflicts('platform=cray')
|
|
|
|
# Override the fetcher method to throw a useful error message;
|
|
# avoids GitHub issue (#7061) in which the opengl placeholder
|
|
# package threw a generic, uninformative error during the `fetch`
|
|
# step,
|
|
@property
|
|
def fetcher(self):
|
|
msg = """This package is intended to be a placeholder for Apple's
|
|
system-provided, non-GNU-compatible libunwind library.
|
|
|
|
Add to your packages.yaml:
|
|
|
|
packages:
|
|
apple-libunwind:
|
|
paths:
|
|
apple-libunwind@35.3: /usr
|
|
buildable: False
|
|
|
|
"""
|
|
raise InstallError(msg)
|
|
|
|
def install(self, spec, prefix):
|
|
# sanity_check_prefix requires something in the install directory
|
|
mkdirp(prefix.lib)
|
|
|
|
@property
|
|
def libs(self):
|
|
"""Export the Apple libunwind library. The Apple libunwind library
|
|
cannot be linked to directly using an absolute path; doing so
|
|
will cause the linker to throw an error 'cannot link directly
|
|
with /usr/lib/system/libunwind.dylib' and the linker will
|
|
suggest linking with System.framework instead. Linking to this
|
|
framework is equivalent to linking with libSystem.dylib, which
|
|
can be confirmed on a macOS system by executing at a terminal
|
|
the command `ls -l
|
|
/System/Library/Frameworks/System.Framework` -- the file
|
|
"System" is a symlink to `/usr/lib/libSystem.B.dylib`, and
|
|
`/usr/lib/libSystem.dylib` also symlinks to this file.
|
|
|
|
Running `otool -L /usr/lib/libSystem.dylib` confirms that
|
|
it will link dynamically to `/usr/lib/system/libunwind.dylib`.
|
|
|
|
"""
|
|
libs = find_libraries('libSystem',
|
|
self.prefix.lib,
|
|
shared=True, recursive=False)
|
|
if libs:
|
|
return libs
|
|
return None
|
|
|
|
@property
|
|
def headers(self):
|
|
""" Export the Apple libunwind header
|
|
"""
|
|
hdrs = HeaderList(find(self.prefix.include, 'libunwind.h',
|
|
recursive=False))
|
|
return hdrs or None
|