## Background
Spec comparison on develop used a somewhat questionable optimization to
get decent spec comparison performance -- instead of comparing entire spec
DAGs, it put a `hash()` call in `_cmp_iter()` and compared specs by their
runtime hashes. This gets us good performance abstract specs, which don't
have complex dependencies and for which hashing is cheap. But it makes
the order of specs unstable and hard to reproduce.
We really need to do a full, consistent traversal over specs to compare
and to get a stable ordering. Simply taking the hash out and yielding
dependencies recursively (i.e. yielding `dep.spec._cmp_iter()` instead
of a hash) goes exponential for concrete specs because it explores all
paths. Traversal tracks visited nodes, but it's expensive to set up
the data structures for that, and it can slow down comparison of simple
abstract specs. Abstract spec comparison performance is important for
concretization (specifically setup), so we don't want to do that.
## New comparison algorithm
We can have (mostly) the best of both worlds -- it's just a bit more
complicated.
This changes Spec comparison to do a proper, stable graph comparison:
1. Spec comparison will now short-circuit whenever possible for concrete
specs, when DAG hashes are known to be equal or not equal. This means
that concrete spec `==` and `!=` comparisons will no longer have
to traverse the whole DAG.
2. Spec comparison now traverses the graph consistently, comparing nodes
and edges in breadth-first order. This means Spec sort order is stable,
and it won't vary arbitrarily from run to run.
3. Traversal can be expensive, so we avoid it for simple specs. Specifically,
if a spec has no dependencies, or if its dependencies have no dependencies,
we avoid calling `traverse_edges()` by doing some special casing.
The `_cmp_iter` method for `Spec` now iterates over the DAG and yields nodes
in BFS order. While it does that, it generates consistent ids for each node,
based on traversal order. It then outputs edges in terms of these ids, along with
their depflags and virtuals, so that all parts of the Spec DAG are included.
The resulting total ordering of specs keys on node attributes first, then
dependency nodes, then any edge differences between graphs.
Optimized cases skip the id generation and traversal, since we know the
order and therefore the ids in advance.
## Performance ramifications
### Abstract specs
This seems to add around 7-8% overhead to concretization setup time. It's
worth the cost, because this enables concretization caching (as input to
concretization was previously not stable) and setup will eventually be
parallelized, at which point it will no longer be a bottleneck for solving.
Together those two optimizations will cut well over 50% of the time (likely
closer to 90+%) off of most solves.
### Concrete specs
Comparison for concrete specs is faster than before, sometimes *way* faster
because comparison is now guaranteed to be linear time w.r.t. DAG size.
Times for comparing concrete Specs:
```python
def compare(json):
a = spack.spec.Spec(json)
b = spack.spec.Spec(json)
print(a == b)
print(timeit.timeit(lambda: a == b, number=1))
compare("./py-black.json")
compare("./hdf5.json")
```
* `develop` (uses our prior hash optimization):
* `py-black`: 7.013e-05s
* `py-hdf5`: 6.445e-05s
* `develop` with full traversal and no hash:
* `py-black`: 3.955s
* `py-hdf5`: 0.0122s
* This branch (full traversal, stable, short-circuiting, no hash)
* `py-black`: 2.208e-06s
* `py-hdf5`: 3.416e-06s
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Add a `_cmp_fast_eq` method to the `lazy_lexicographic_ordering` protocol
so that implementors can short-circuit full comparison if they know (e.g.
with a hash) that objects are equal (or not).
`_cmp_fast_eq` can return True (known true), False (known false), or
None (unknown -- do full comparison).
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
`_EdgeMap` implements `_cmp_iter` for `lazy_lexicographic_ordering` but it's never used
or tested. Remove it.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Builders and package classes refer to packages from the builtin package
repo and are often modified together with packages. That means that
these classes should move into `spack_repo.builtin`.
* move `spack.build_systems` -> `spack_repo.builtin.build_systems`
* Remove the following re-exports from the `spack.package` module:
- `AspellDictPackage` - `LuaPackage`
- `AutotoolsPackage` - `MakefilePackage`
- `BundlePackage` - `MavenPackage`
- `CachedCMakePackage` - `MesonPackage`
- `cmake_cache_filepath` - `MSBuildPackage`
- `cmake_cache_option` - `NMakePackage`
- `cmake_cache_path` - `OctavePackage`
- `cmake_cache_string` - `PerlPackage`
- `CargoPackage` - `PythonExtension`
- `CMakePackage` - `PythonPackage`
- `generator` - `QMakePackage`
- `CompilerPackage` - `RacketPackage`
- `CudaPackage` - `RPackage`
- `Package` - `ROCmPackage`
- `GNUMirrorPackage` - `RubyPackage`
- `GoPackage` - `SConsPackage`
- `IntelPackage` - `SIPPackage`
- `IntelOneApiLibraryPackageWithSdk` - `SourceforgePackage`
- `IntelOneApiLibraryPackage` - `SourcewarePackage`
- `IntelOneApiStaticLibraryList` - `WafPackage`
- `IntelOneApiPackage` - `XorgPackage`
- `INTEL_MATH_LIBRARIES`
* update mock packages to repo v2.0 and add copies of packages/build
systems they use from builtin
* add missing imports to build systems in `package.py` from builtin
and test repos
* update various tests
This PR is breaking because of removal of various names from
`spack.package`, but breakage should be minimal thanks to #50496, which
ensures the above names are always imported in repo v1 packages.
Specifically this PR breaks imports like the following in `package.py` files:
```python
from spack.package import Package
```
but if your repo is v1.0 (see `spack repo list`) and has the following
much more common pattern:
```python
from spack.package import *
```
nothing should break.
Restores ability to build MSMPI from source.
* Modernizes the MSMPI package
* Strips out MSMPI build system that is long deprecated
* Adds patches to re-introduce missing functionality from said build system
* Adds builds + support for dependencies no longer fetched by build system
* cuda: Add 12.9 and new Blackwell targets
Including family-specific architecture features like sm_100f
* tau, caliper: Add cuda v12.9 conflict
Legacy nvtx was dropped.
* petsc: "libnvToolsExt.a" -> "libnvtx3interop.a" for cuda@12.9: and petsc@:3.23.1
* legion: Add upper bound to cuda version range
Avoid build failure due to CUDA driver API incompatibility.
See https://github.com/StanfordLegion/legion/issues/1864
---------
Co-authored-by: Satish Balay <balay@mcs.anl.gov>
When loading packages from a v1.0 repository, inject a line
from spack.build_systems._package_api import *
This allows removal of certain names in `spack.package` as part of api
v2 without breaking backward compatibility in Spack.
* update tfel package
* Update MGIS package
* add support for Version 5.0.1, 4.2.3, 4.1.4, 4.0.5, 3.4.8, 3.3.7, 3.2.12, 3.1.15 and 3.0.15
* add support for Versions 3.0.1 and 2.2.1
* Update crtm from jcsda/spack-stack-dev
* Update grads from jcsda/spack-stack-dev
* Update hdf from jcsda/spack-stack-dev
* Update ip from jcsda/spack-stack-dev
* Update met from jcsda/spack-stack-dev
* Update metplus from jcsda/spack-stack-dev
* Update py-kiwisolver from jcsda/spack-stack-dev
* Update py-pyogrio from jcsda/spack-stack-dev
* Update py-ruamel-yaml-clib from jcsda/spack-stack-dev
* Update wgrib2 from jcsda/spack-stack-dev
This PR modifies the parser, so that `%` is parsed as a `DEPENDENCY`, and all
node properties that follow are associated to the name after the `%`. e.g.,
in `foo %gcc +binutils` the `+binutils` refers to `gcc` and not to `foo`.
`%` is still parsed as a build-type dependency, at the moment.
Environments, config files and `package.py` files from before Spack v1.0 may have
spec strings with package variants, targets, etc. *after* a build dependency, and these
will need to be updated. You can use the `spack style --spec-strings` command to do this.
To see what strings will be parsed differently under Spack v1.0, run:
```
spack style --spec-strings FILES
```
where `FILES` is a list of filenames that may contain old specs. To update these spec
strings so that they parse correctly under both Spack 1.0 and Spack 0.x, you can run:
```
spack style --fix --spec-strings FILES
```
In the example above, `foo %gcc +binutils` would be rewritten as `foo +binutils %gcc`,
which parses the same in any Spack version.
In addition, this PR fixes several issues with `%` dependencies:
- [x] Ensure we can still constrain compilers on reuse
- [x] Ensure we can reuse a compiler by hash
- [x] Add tests
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
