6ee2eb21dd
This changes the hash algorithm so that it does much less object allocation and copying, and so that it is correct. The old version of `_cmp_key()` would call `sorted_deps`, which would call `flat_dependencies` to get a list of dependencies so that it could sort them in alphabetical order. This isn't necessary in the `_cmp_key()`, and in fact we want more DAG structure than that to be included in the `_cmp_key()`. The new version constructs a tuple without copying the Spec DAG, and the tuple contains hashes of sub-DAGs that are computed recursively in-place. This is way faster than the previous algorithm and reduces the numebr of copies significantly. It is also a correct DAG hash. Example timing and copy counts for the different hashing algorithms we've tried: Original (wrong) Spec hash: ``` 106,170 copies real 0m5.024s user 0m4.949s sys 0m0.104s ``` Spec hash using YAML `dag_hash()`: ``` 3,794 copies real 0m5.024s user 0m4.949s sys 0m0.104s New no-copy, no-YAML hash: ``` 3,594 copies real 0m2.543s user 0m2.435s sys 0m0.104s ``` So now we have a hash that is correct AND faster. The remaining ~3k copies happen mostly during concretization, and as all packages are initially loaded. I believe this is because Spack currently has to load all packages to figure out virtual dependency information; it could also be becasue there ar a lot of lookups of partial specs in concretize. I can investigate this further. |
||
|---|---|---|
| bin | ||
| lib/spack | ||
| share/spack | ||
| var/spack | ||
| .gitignore | ||
| .mailmap | ||
| .travis.yml | ||
| LICENSE | ||
| README.md | ||
Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share common installations of software on clusters with exotic architectures, using libraries that do not have a standard ABI. Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system.
Most importantly, Spack is simple. It offers a simple spec syntax so that users can specify versions and configuration options concisely. Spack is also simple for package authors: package files are written in pure Python, and specs allow package authors to write a single build script for many different builds of the same package.
See the Feature Overview for examples and highlights.
To install spack and install your first package:
$ git clone https://github.com/llnl/spack.git
$ cd spack/bin
$ ./spack install libelf
Documentation
Full documentation for Spack is the first place to look.
See also:
- Technical paper and slides on Spack's design and implementation.
- Short presentation from the Getting Scientific Software Installed BOF session at Supercomputing 2015.
Get Involved!
Spack is an open source project. Questions, discussion, and contributions are welcome. Contributions can be anything from new packages to bugfixes, or even new core features.
Mailing list
If you are interested in contributing to spack, the first step is to join the mailing list. We're using a Google Group for this, and you can join it here:
Contributions
At the moment, contributing to Spack is relatively simple. Just send us
a pull request.
When you send your request, make develop the destination branch.
Spack is using a rough approximation of the Git
Flow
branching model. The develop branch contains the latest
contributions, and master is always tagged and points to the
latest stable release.
Authors
Many thanks go to Spack's contributors.
Spack was originally written by Todd Gamblin, tgamblin@llnl.gov.
Citing Spack
If you are referencing Spack in a publication, please cite the following paper:
- Todd Gamblin, Matthew P. LeGendre, Michael R. Collette, Gregory L. Lee, Adam Moody, Bronis R. de Supinski, and W. Scott Futral. The Spack Package Manager: Bringing Order to HPC Software Chaos. In Supercomputing 2015 (SC’15), Austin, Texas, November 15-20 2015. LLNL-CONF-669890.
Release
Spack is released under an LGPL license. For more details see the LICENSE file.
LLNL-CODE-647188