Commit Graph

4469 Commits

Author SHA1 Message Date
Yang Zongze
40d32890d0 Print groups properly for spack find -d (#20028) 2021-02-17 17:07:45 -08:00
eugeneswalker
18022050f2 store sbang_install_path in buildinfo, use for subsequent relocation (#20768) 2021-02-17 17:07:44 -08:00
Nathan Hanford
cdd86bddec [WIP] relocate.py: parallelize test replacement logic (#19690)
* sbang pushed back to callers;
star moved to util.lang

* updated unit test

* sbang test moved; local tests pass

Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2021-02-17 17:07:44 -08:00
Massimiliano Culpo
488a08b464 concretizer: require at least a dependency type to say the dependency holds
fixes #20784

Similarly to the previous bug, here we were deducing
conditions to be imposed on nodes that were not part
of the DAG.
2021-02-17 17:07:43 -08:00
Massimiliano Culpo
7162e15531 concretizer: dependency conditions cannot hold if package is external
fixes #20736

Before this one line fix we were erroneously deducing
that dependency conditions hold even if a package
was external.

This may result in answer sets that contain imposed
conditions on a node without the node being present
in the DAG, hence #20736.
2021-02-17 17:07:42 -08:00
Robert Underwood
c99850dd58 restore ability of dev-build to skip patches (#20351)
At some point in the past, the skip_patch argument was removed
from the call to package.do_install() this broke the --skip-patch
flag on the dev-build command.
2021-02-17 17:07:41 -08:00
Massimiliano Culpo
ddd9c86ce0 concretizer: make rules on virtual packages more linear
fixes #20679

In this refactor we have a single cardinality rule on the
provider, which triggers a rule transforming a dependency
on a virtual package into a dependency on the provider of
the virtual.
2021-02-17 17:07:39 -08:00
Todd Gamblin
8e442d6dc1 concretizer: use consistent naming for compiler predicates (#20677)
Every other predicate in the concretizer uses a `_set` suffix to
implement user- or package-supplied settings, but compiler settings use a
`_hard` suffix for this. There's no difference in how they're used, so
make the names the same.

- [x] change `node_compiler_hard` to `node_compiler_set`
- [x] change `node_compiler_version_hard` to `node_compiler_version_set`
2021-02-17 17:07:38 -08:00
Todd Gamblin
9393d97139 concretizer: simplify handling of virtual version constraints
Previously, the concretizer handled version constraints by comparing all
pairs of constraints and ensuring they satisfied each other. This led to
INCONSISTENT ressults from clingo, due to ambiguous semantics like:

    version_constraint_satisfies("mpi", ":1", ":3")
    version_constraint_satisfies("mpi", ":3", ":1")

To get around this, we introduce possible (fake) versions for virtuals,
based on their constraints. Essentially, we add any Versions,
VersionRange endpoints, and all such Versions and endpoints from
VersionLists to the constraint. Virtuals will have one of these synthetic
versions "picked" by the solver. This also allows us to remove a special
case from handling of `version_satisfies/3` -- virtuals now work just
like regular packages.
2021-02-17 17:07:38 -08:00
Todd Gamblin
8f85ab88c0 concretizer: remove rule generation code from concretizer
Our program only generates facts now, so remove all unused code related
to generating cardinality constraints and rules.
2021-02-17 17:07:37 -08:00
Todd Gamblin
247e73e85a concretizer: convert virtuals to facts; move all rules to concretize.lp
This converts the virtual handling in the new concretizer from
already-ground rules to facts. This is the last thing that needs to be
refactored, and it converts the entire concretizer to just use facts.

The previous way of handling virtuals hinged on rules involving
`single_provider_for` facts that were tied to the virtual and a version
range. The new method uses the condition pattern we've been using for
dependencies, externals, and conflicts.

To handle virtuals as conditions, we impose constraints on "fake" virtual
specs in the logic program. i.e., `version_satisfies("mpi", "2.0:",
"2.0")` is legal whereas before we wouldn't have seen something like
this. Currently, constriants are only handled on versions -- we don't
handle variants or anything else yet, but they key change here is that we
*could*. For a long time, virtual handling in Spack has only dealt with
versions, and we'd like to be able to handle variants as well. We could
easily add an integrity constraint to handle variants like the one we use
for versions.

One issue with the implementation here is that virtual packages don't
actually declare possible versions like regular packages do. To get
around that, we implement an integrity constraint like this:

    :- virtual_node(Virtual),
       version_satisfies(Virtual, V1), version_satisfies(Virtual, V2),
       not version_constraint_satisfies(Virtual, V1, V2).

This requires us to compare every version constraint to every other, both
in program generation and within the concretizer -- so there's a
potentially quadratic evaluation time on virtual constraints because we
don't have a real version to "anchor" things to. We just say that all the
constraints need to agree for the virtual constraint to hold.

We can investigate adding synthetic versions for virtuals in the future,
to speed this up.
2021-02-17 17:07:37 -08:00
Todd Gamblin
6056cb71d3 concretizer: consolidate handling of virtuals into spec_clauses 2021-02-17 17:07:36 -08:00
Todd Gamblin
66376ab971 concretizer: make _condtion_id_counter an iterator 2021-02-17 17:07:36 -08:00
Todd Gamblin
9bb0375443 concretizer: more detailed section headers in concretize.lp 2021-02-17 17:07:35 -08:00
Todd Gamblin
56acb5a0c7 bugfix: infinite loop when building a set from incomplete specs (#20649)
This code in `SpecBuilder.build_specs()` introduced in #20203, can loop
seemingly interminably for very large specs:

```python
set([spec.root for spec in self._specs.values()])
```

It's deceptive, because it seems like there must be an issue with
`spec.root`, but that works fine. It's building the set afterwards that
takes forever, at least on `r-rminer`. Currently if you try running
`spack solve r-rminer`, it loops infinitely and spins up your fan.

The issue (I think) is that the spec is not yet complete when this is
run, and something is going wrong when constructing and comparing so many
values produced by `_cmp_key()`. We can investigate the efficiency of
`_cmp_key()` separately, but for now, the fix is:

```python
roots = [spec.root for spec in self._specs.values()]
roots = dict((id(r), r) for r in roots)
```

We know the specs in `self._specs` are distinct (they just came out of
the solver), so we can just use their `id()` to unique them here. This
gets rid of the infinite loop.
2021-02-17 17:07:35 -08:00
Massimiliano Culpo
ec42016241 concretizer: generate facts for externals
Generate only facts for external specs. Substitute the
use of already grounded rules with non-grounded rules
in concretize.lp
2021-02-17 17:07:35 -08:00
Greg Becker
f7195123d4 bugfix: do not write empty default dicts/lists in envs (#20526)
Environment yaml files should not have default values written to them.

To accomplish this, we change the validator to not add the default values to yaml. We rely on the code to set defaults for all values (and use defaulting getters like dict.get(key, default)).

Includes regression test.
2021-02-17 17:07:34 -08:00
Robert Cohn
290043b72a Add Intel oneAPI packages (#20411)
This creates a set of packages which all use the same script to install
components of Intel oneAPI. This includes:

* An inheritable IntelOneApiPackage which knows how to invoke the
  installation script based on which components are requested
* For components which include headers/libraries, an inheritable
  IntelOneApiLibraryPackage is provided to locate them
* Individual packages for DAL, DNN, TBB, etc.
* A package for the Intel oneAPI compilers (icx/ifx). This also includes
  icc/ifortran but these are not currently detected in this PR
2021-02-17 17:07:33 -08:00
Todd Gamblin
1a1babe185 concretizer: refactor conditional rules to be less repetitious (#20507)
We have to repeat all the spec attributes in a number of places in
`concretize.lp`, and Spack has a fair number of spec attributes. If we
instead add some rules up front that establish equivalencies like this:

```
    node(Package) :- attr("node", Package).
    attr("node", Package) :- node(Package).

    version(Package, Version) :- attr("version", Package, Version).
    attr("version", Package, Version) :- version(Package, Version).
```

We can rewrite most of the repetitive conditions with `attr` and repeat
only for each arity (there are only 3 arities for spec attributes so far)
as opposed to each spec attribute. This makes the logic easier to read
and the rules easier to follow.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-02-17 17:07:32 -08:00
Massimiliano Culpo
acd523c7f3 concretizer: optimize loop on compiler version
Similar to the optimization on platform
2021-02-17 17:07:32 -08:00
Massimiliano Culpo
4bbc6eec51 concretizer: optimized loop on node platforms
We can speed-up the computation by avoiding a
double loop in a cardinality constraint and
enforcing the rule instead as an integrity
constraint.
2021-02-17 17:07:31 -08:00
Massimiliano Culpo
cb76c5a90d concretizer: fix failing unit tests 2021-02-17 17:07:31 -08:00
Massimiliano Culpo
ab3f53d781 concretizer: emit facts for integrity constraints 2021-02-17 17:07:30 -08:00
Massimiliano Culpo
364c5b636c concretizer: emit facts for constraints on imposed dependencies 2021-02-17 17:07:30 -08:00
Massimiliano Culpo
02e0ea6105 concretizer: avoid redundant grounding on dependency types 2021-02-17 17:07:30 -08:00
Todd Gamblin
ff9ee9f243 concretizer: move conditional dependency logic into concretize.lp
Continuing to convert everything in `asp.py` into facts, make the
generation of ground rules for conditional dependencies use facts, and
move the semantics into `concretize.lp`.

This is probably the most complex logic in Spack, as dependencies can be
conditional on anything, and we need conditional ASP rules to accumulate
and map all the dependency conditions to spec attributes.

The logic looks complicated, but essentially it accumulates any
constraints associated with particular conditions into a fact associated
with the condition by id. Then, if *any* condition id's fact is True, we
trigger the dependency.

This simplifies the way `declared_dependency()` works -- the dependency
is now declared regardless of whether it is conditional, and the
conditions are handled by `dependency_condition()` facts.
2021-02-17 17:07:29 -08:00
Todd Gamblin
6c6631cfdc concretizer: spec_clauses should traverse dependencies
There are currently no places where we do not want to traverse
dependencies in `spec_clauses()`, so simplify the logic by consolidating
`spec_traverse_clauses()` with `spec_clauses()`.
2021-02-17 17:07:29 -08:00
Todd Gamblin
ea617f807f concretizer: pull _develop_specs_from_env out of main setup loop 2021-02-17 17:07:28 -08:00
Todd Gamblin
d0dfd3cb82 concretizer: add #defined statements to avoid warnings.
`version_satisfies/2` and `node_compiler_version_satisfies/3` are
generated but need `#defined` directives to avoid " info: atom does not
occur in any rule head:" warnings.
2021-02-17 17:07:28 -08:00
Massimiliano Culpo
a4066a52be asp: memoize the list of all target_specs to speed-up setup phase (#20473)
* asp: memoize the list of all target_specs to speed-up setup phase

* asp: memoize using a cache per solver object
2021-02-17 17:07:28 -08:00
Scott Wittenburg
18c5f10ae7 ci: fixes for compiler bootstrapping (#17563)
This PR addresses a number of issues related to compiler bootstrapping.

Specifically:
1. Collect compilers to be bootstrapped while queueing in installer
Compiler tasks currently have an incomplete list in their task.dependents,
making those packages fail to install as they think they have not all their
dependencies installed. This PR collects the dependents and sets them on
compiler tasks.

2. allow boostrapped compilers to back off target
Bootstrapped compilers may be built with a compiler that doesn't support
the target used by the rest of the spec.  Allow them to build with less
aggressive target optimization settings.

3. Support for target ranges
Backing off the target necessitates computing target ranges, so make Spack
handle those properly.  Notably, this adds an intersection method for target
ranges and fixes the way ranges are satisfied and constrained on Spec objects.

This PR also:
- adds testing
- improves concretizer handling of target ranges

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Gregory Becker <becker33@llnl.gov>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-02-17 17:07:27 -08:00
Massimiliano Culpo
d82d2bb2db unit-tests: ensure that installed packages can be reused (#20307)
refers #20292

Added a unit test that ensures we can reuse installed
packages even if in the repository variants have been
removed or added.
2021-02-17 17:07:27 -08:00
Greg Becker
378af922a2 Fix comparisons for abstract specs (#20341)
bug only relevant for python3
2021-02-17 17:07:27 -08:00
Todd Gamblin
12d035b225 concretizer: don't use one_of_iff for range constraints (#20383)
Currently, version range constraints, compiler version range constraints,
and target range constraints are implemented by generating ground rules
from `asp.py`, via `one_of_iff()`.  The rules look like this:

```
version_satisfies("python", "2.6:") :- 1 { version("python", "2.4"); ... } 1.
1 { version("python", "2.4"); ... } 1. :- version_satisfies("python", "2.6:").
```

So, `version_satisfies(Package, Constraint)` is true if and only if the
package is assigned a version that satisfies the constraint. We
precompute the set of known versions that satisfy the constraint, and
generate the rule in `SpackSolverSetup`.

We shouldn't need to generate already-ground rules for this. Rather, we
should leave it to the grounder to do the grounding, and generate facts
so that the constraint semantics can be defined in `concretize.lp`.

We can replace rules like the ones above with facts like this:

```
version_satisfies("python", "2.6:", "2.4")
```

And ground them in `concretize.lp` with rules like this:

```
1 { version(Package, Version) : version_satisfies(Package, Constraint, Version) } 1
  :- version_satisfies(Package, Constraint).
version_satisfies(Package, Constraint)
  :- version(Package, Version), version_satisfies(Package, Constraint, Version).
```

The top rule is the same as before. It makes conditional dependencies and
other places where version constraints are used work properly. Note that
we do not need the cardinality constraint for the second rule -- we
already have rules saying there can be only one version assigned to a
package, so we can just infer from `version/2` `version_satisfies/3`.
This form is also safe for grounding -- If we used the original form we'd
have unsafe variables like `Constraint` and `Package` -- the original
form only really worked when specified as ground to begin with.

- [x] use facts instead of generating rules for package version constraints
- [x] use facts instead of generating rules for compiler version constraints
- [x] use facts instead of generating rules for target range constraints
- [x] remove `one_of_iff()` and `iff()` as they're no longer needed
2021-02-17 17:07:26 -08:00
Massimiliano Culpo
bf3a873a42 package sanity: ensure all variant defaults are allowed values (#20373) 2021-02-17 17:07:26 -08:00
Todd Gamblin
27f6b3df92 concretizer: remove clingo command-line driver (#20362)
I was keeping the old `clingo` driver code around in case we had to run
using the command line tool instad of through the Python interface.

So far, the command line is faster than running through Python, but I'm
working on fixing that.  I found that if I do this:

```python
control = clingo.Control()
control.load("concretize.lp")
control.load("hdf5.lp")       # code from spack solve --show asp hdf5
control.load("display.lp")

control.ground([("base", [])])
control.solve(...)
```

It's just as fast as the command line tool. So we can always generate the
code and load it manually if we need to -- we don't need two drivers for
clingo. Given that the python interface is also the only way to get unsat
cores, I think we pretty much have to use it.

So, I'm removing the old command line driver and other unused code. We
can dig it up again from the history if it is needed.
2021-02-17 17:07:25 -08:00
Tamara Dahlgren
802c5bdff7 Tests: enable re-use of post-install tests in smoke tests (#20298) 2021-02-17 17:07:25 -08:00
Andrew W Elble
ab3f1b10db concretizer: try hard to obtain all needed variant_possible_value()'s (#20102)
Track all the variant values mentioned when emitting constraints, validate them
and emit a fact that allows them as possible values.

This modification ensures that open-ended variants (variants accepting any string 
or any integer) are projected to the finite set of values that are relevant for this 
concretization.
2021-02-17 17:07:24 -08:00
Massimiliano Culpo
30a9e6462f bugfix: work around issue handling packages not in any repo 2021-02-17 17:07:24 -08:00
Todd Gamblin
0e725f0ab1 concretizer: refactor handling of special variants dev_build and patches
Other parts of the concretizer code build up lists of things we can't
know without traversing all specs and packages, and they output these
list at the very end.

The code for this for variant values from spec literals was intertwined
with the code for traversing the input specs. This only covers the input
specs and misses variant values that might come from directives in
packages.

- [x] move ad-hoc value handling code into spec_clauses so we do it in
  one place for CLI and packages

- [x] move handling of `variant_possible_value`, etc. into
  `concretize.lp`, where we can automatically infer variant existence
  more concisely.

- [x] simplify/clarify some of the code for variants in `spec_clauses()`
2021-02-17 17:07:23 -08:00
Massimiliano Culpo
ecfba13d89 concretizer: each external version is allowed by definition (#20247)
Registering external versions among the lists of allowed ones
generates the correct rules for `version_satisfies`
2021-02-17 17:07:22 -08:00
Massimiliano Culpo
a6d433b937 concretizer: restrict maximizing variant values to MV variants (#20194) 2021-02-17 17:07:22 -08:00
Matthias Wolf
b4f79f3cb7 environment installs: fix reporting. (#20004)
PR #15702 changed the invocation of the report context when installing
specs, do the same when building environments.
2021-02-17 17:07:22 -08:00
Greg Becker
30290acf67 avoid circular import (#20236) 2021-02-17 17:07:21 -08:00
Andrew W Elble
92de484228 concretizer: call inject_patches_variants() on the roots of the specs (#20203)
As was done in the old concretizer. Fixes an issue where conditionally
patched dependencies did not show up in spec (gdal+jasper)
2021-02-17 17:07:21 -08:00
Massimiliano Culpo
b78cc5b43d concretizer: try hard to infer the real version of compilers (#20099)
fixes #20055

Compiler with custom versions like gcc@foo are not currently
matched to the appropriate targets. This is because the
version of spec doesn't match the "real" version of the
compiler.

This PR replicates the strategy used in the original
concretizer to deal with that and tries to detect the real
version of compilers if the version in the spec returns no
results.
2021-02-17 17:07:21 -08:00
Harmen Stoppels
571e36787b Fix hipcc once more (#20095) 2021-02-17 17:07:20 -08:00
Andrew W Elble
598c25b434 concretizer: don't optimize emitting version_satisfies() (#20128)
When all versions were allowed a version_satisfies rule was not emitted,
and this caused conditional directives to fail.
2021-02-17 17:07:19 -08:00
Massimiliano Culpo
0b7a22dea2 spec: return early from concretization if a spec is already concrete (#20196) 2021-02-17 17:07:18 -08:00
Massimiliano Culpo
1e1a6f4e30 concretizer: remove ad-hoc rule for external packages (#20193)
fixes #20040

Matching compilers among nodes has been prioritized
in #20020. Selection of default variants has been
tuned in #20182. With this setup there is no need
to have an ad-hoc rule for external packages. On
the contrary it should be removed to prefer having
default variant values over more external nodes in
the DAG.
2021-02-17 17:07:18 -08:00
Massimiliano Culpo
22d7937c50 concretizer: swap priority of selecting provider and default variant (#20182)
refers #20040

Before this PR optimization rules would have selected default
providers at a higher priority than default variants. Here we
swap this priority and we consider variants that are forced by
any means (root spec or spec in depends_on clause) the same as
if they were with a default value.

This prevents the solver from avoiding expected configurations
just because they contain directives like:

depends_on('pkg+foo')

and `+foo` is not the default variant value for pkg.
2021-02-17 17:07:18 -08:00
George Hartzell
96283867d6 Typos: add missing closing parens (#20174) 2021-02-17 17:07:17 -08:00
Massimiliano Culpo
d7ffdd76f6 concretizer: treat target ranges in directives correctly (#19988)
fixes #19981

This commit adds support for target ranges in directives,
for instance:

conflicts('+foo', when='target=x86_64:,aarch64:')

If any target in a spec body is not a known target the
following clause will be emitted:

node_target_satisfies(Package, TargetConstraint)

when traversing the spec and a definition of
the clause will then be printed at the end similarly
to what is done for package and compiler versions.
2021-02-17 17:07:17 -08:00
Massimiliano Culpo
b326d59e10 concretizer: prioritize matching compilers over newer versions (#20020)
fixes #20019

Before this modification having a newer version of a node came
at higher priority in the optimization than having matching
compilers. This could result in unexpected configurations for
packages with conflict directives on compilers of the type:

conflicts('%gcc@X.Y:', when='@:A.B')

where changing the compiler for just that node is preferred to
lower the node version to less than 'A.B'. Now the priority has
been switched so the solver will try to lower the version of the
nodes in question before changing their compiler.
2021-02-17 17:07:16 -08:00
Massimiliano Culpo
0ae49821e2 concretizer: allow a bool to be passed as argument for tests dependencies (#20082)
refers #20079

Added docstrings to 'concretize' and 'concretized' to
document the format for tests.

Added tests for the activation of test dependencies.
2021-02-17 17:07:16 -08:00
Massimiliano Culpo
8e1b5f734f concretizer: treat conditional providers correctly (#20086)
refers #20040

This modification emits rules like:

provides_virtual("netlib-lapack","blas") :- variant_value("netlib-lapack","external-blas","False").

for packages that provide virtual dependencies conditionally instead
of a fact that doesn't account for the condition.
2021-02-17 17:07:16 -08:00
Adam J. Stewart
ad84351c43 Docs: remove duplication in Command Reference (#20021) 2021-02-17 17:07:15 -08:00
Martin Aumüller
508534aad9 recognize macOS 11.1 as big sur (#20038)
Big Sur versions go 11.0, 11.0.1, 11.1 (vs. prior versions that
only used the minor component)

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-02-17 17:07:15 -08:00
Greg Becker
38cf4f9fc2 fix error handling for spack test results command (#19987) 2021-02-17 17:07:14 -08:00
Danny Taller
c42ce439e4 hip support for umpire, chai, raja, camp (#19715)
* create HipPackage base class and do some refactoring

* comments and added conflict to raja for openmp with hip
2021-02-17 17:07:14 -08:00
Todd Gamblin
82383093ee bump version number to 0.16.0 2020-11-18 04:22:09 -08:00
Michael Kuhn
20367e472d
cmd: add spack mark command (#16662)
This adds a new `mark` command that can be used to mark packages as either
explicitly or implicitly installed. Apart from fixing the package
database after installing a dependency manually, it can be used to
implement upgrade workflows as outlined in #13385.

The following commands demonstrate how the `mark` and `gc` commands can be
used to only keep the current version of a package installed:
```console
$ spack install pkgA
$ spack install pkgB
$ git pull # Imagine new versions for pkgA and/or pkgB are introduced
$ spack mark -i -a
$ spack install pkgA
$ spack install pkgB
$ spack gc
```

If there is no new version for a package, `install` will simply mark it as
explicitly installed and `gc` will not remove it.

Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-11-18 03:20:56 -08:00
Greg Becker
77b2e578ec
spack test (#15702)
Users can add test() methods to their packages to run smoke tests on
installations with the new `spack test` command (the old `spack test` is
now `spack unit-test`). spack test is environment-aware, so you can
`spack install` an environment and then run `spack test run` to run smoke
tests on all of its packages. Historical test logs can be perused with
`spack test results`. Generic smoke tests for MPI implementations, C,
C++, and Fortran compilers as well as specific smoke tests for 18
packages.

Inside the test method, individual tests can be run separately (and
continue to run best-effort after a test failure) using the `run_test`
method. The `run_test` method encapsulates finding test executables,
running and checking return codes, checking output, and error handling.

This handles the following trickier aspects of testing with direct
support in Spack's package API:

- [x] Caching source or intermediate build files at build time for
      use at test time.
- [x] Test dependencies,
- [x] packages that require a compiler for testing (such as library only
      packages).

See the packaging guide for more details on using Spack testing support.
Included is support for package.py files for virtual packages. This does
not change the Spack interface, but is a major change in internals.

Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
Co-authored-by: wspear <wjspear@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-11-18 02:39:02 -08:00
Massimiliano Culpo
89181f253b Improve warning message for deprecated attributes in "packages.yaml"
The deprecatedProperties custom validator now can accept a function
to compute a better error message.

Improve error/warning message for deprecated properties
2020-11-17 17:34:27 -08:00
Peter Scheibel
c9ad2affcc
Documentation: spack load/environments prefix inspections (#19961)
As of #18260, `spack load` and `spack env activate` now use
`prefix_inspections` from the modules configuration to decide
how to modify environment variables.

This updates the modules configuration documentation to describe
how to update environment variables with the `prefix_inspections`
section. This also updates the `spack load` and environments
documentation to refer to the new `prefix_inspections` documentation.
2020-11-17 15:24:00 -08:00
Dr. Christian Tacke
d65f078f66
spack load/environments: allow customization of prefix inspections (#18260)
`spack load` and `spack env activate` now use the prefix inspections
defined in `modules.yaml`. This allows users to customize/override
environment variable modifications if desired.

If no `prefix_inspections` configuration is present, Spack uses the
values in the default configuration.
2020-11-17 14:04:13 -08:00
Massimiliano Culpo
5f636fc317
spack containerize: allow users to customize the base image (#15028)
This PR reworks a few attributes in the container subsection of
spack.yaml to permit the injection of custom base images when
generating containers with Spack. In more detail, users can still
specify the base operating system and Spack version they want to use:

  spack:
    container:
      images:
        os: ubuntu:18.04
        spack: develop

in which case the generated recipe will use one of the Spack images
built on Docker Hub for the build stage and the base OS image in the
final stage. Alternatively, they can specify explicitly the two
base images:

  spack:
    container:
      images:
        build: spack/ubuntu-bionic:latest
        final: ubuntu:18.04

and it will be up to them to ensure their consistency.

Additional changes:

* This commit adds documentation on the two approaches.
* Users can now specify OS packages to install (e.g. with apt or yum)
  prior to the build (previously this was only available for the
  finalized image).
* Handles to avoid an update of the available system packages have been
  added to the configuration to facilitate the generation of recipes
  permitting deterministic builds.
2020-11-17 11:25:13 -08:00
Massimiliano Culpo
7ffad278d3 concretizer: modified weights for providers and matching for externals
This commit address the case of concretizing a root spec with a
transitive conditional dependency on a virtual package, provided
by an external. Before these modifications default variant values
for the dependency bringing in the virtual package were not
respected, and the external package providing the virtual was added
to the DAG.

The issue stems from two facts:
- Selecting a provider has higher precedence than selecting default variants
- To ensure that an external is preferred, we used a negative weight

To solve it we shift all the providers weight so that:
- External providers have a weight of 0
- Non external provider have a weight of 10 or more

Using a weight of zero for external providers is such that having
an external provider, if present, or not having a provider at all
has the same effect on the higher priority minimization.

Also fixed a few minor bugs in concretize.lp, that were causing
spurious entries in the final answer set.

Cleaned concretize.lp from leftover rules.
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
ca31f52be3 concretizer: maximize the number of default values used for a single variant
If a the default of a multi-valued variant is set to
multiple values either in package.py or in packages.yaml
we need to ensure that all the values are present in the
concretized spec.

Since each default value has a weight of 0 and the
variant value is set implicitly by the concretizer
we need to add a rule to maximize on the number of
default values that are used.
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
9a03fd2834 concretizer: don't require a provider for virtual deps if spec is external
This commit introduces a new rule:

real_node(Package) :- not external(Package), node(Package).

that permits to distinguish between an external node and a
real node that shouldn't trim dependency. It solves the
case of concretizing ninja with an external Python.
2020-11-17 10:04:13 -08:00
Todd Gamblin
44aa94a210 concretizer: spec_clauses() shouldn't emit node_compiler_hard for rule bodies.
`node_compiler_hard()` means that something explicitly asked for a node's
compiler to be set -- i.e., it's not inherited, it's required. We're
generating this in spec_clauses even for specs in rule bodies, which
results in conditions like this for optional dependencies:

In py-torch/package.py:

    depends_on('llvm-openmp', when='%apple-clang +openmp')

In the generated ASP:

    declared_dependency("py-torch","llvm-openmp","build")
      :- node("py-torch"),
         variant_value("py-torch","openmp","True"),
         node_compiler("py-torch","apple-clang"),
         node_compiler_hard("py-torch","apple-clang"),
         node_compiler_version_satisfies("py-torch","apple-clang",":").

The `node_compiler_hard` there means we would have to *explicitly* set
py-torch's compiler to trigger the llvm-openmp dependency, rather than
just letting it be set by preferences. This is wrong; the dependency
should be there regardless of how the compiler was set.

- [x] remove fn.node_compiler_hard() call from spec_clauses when
      generating rule body clauses.
2020-11-17 10:04:13 -08:00
Todd Gamblin
0620d954f5 concretizer: don't generate rules for empty version lists
If the version list passed to one_of_iff is empty, it still generates a
rule like this:

    node_compiler_version_satisfies("fujitsu-mpi", "arm", ":") :- 1 {  } 1.
    1 {  } 1 :- node_compiler_version_satisfies("fujitsu-mpi", "arm", ":").

The cardinality rules on the right and left above are never
satisfiale, and these rules do nothing.

- [x] Skip generating any rules at all for empty version lists.
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
2231dfc898 concretizer: add a rule to avoid cycles in the graph of dependencies 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
522be6cadf External packages have a consistent hash across different concretizers 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
5c5a44988e Don't fail if MV variants have a tuple as default value 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
1b0338befb Fixup for target preferences 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
e0ae60edc4 Added unit tests to for regressions on open concretizer bugs 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
63327d1eea Changed clingo options 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
577676106c Reworked optimization rules 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
6baa8157c7 concretizer: set target preference for inheritance from root 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
ccb537479a install: one less concretization when installing from file 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
27bb970a97 Fixed branch after rebase (port to archspec)
TODO: Investigate the need to remove
memoization on Spec.patches (infinite
recursion when testing `__contains__`)
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
930b05fab4 Add unit tests for dependencies being patched by parent 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
e226523aeb concretizer: handle dependencies conditional on other dependencies 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
e7208b1598 tests: verify to handle dependencies conditional on other dependencies 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
d00e8394f8 concretizer: handle conflicts with compiler ranges correctly
As reported, conflicts with compiler ranges were not treated
correctly. This commit adds tests to verify the expected behavior
for the new concretizer.

The new rules to enforce a correct behavior involve:
- Adding a rule to prefer the compiler selected for
  the root package, if no other preference is set
- Give a strong negative weight to compiler preferences
  expressed in packages.yaml
- Maximize on compiler AND compiler version match
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
0a56b7cfd6 Github actions: add CI for ASP based solver 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
7753d58e7e Make all tests pass
Fixed a couple of tests and marked a few xfails
to solve them later.
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
8a855ddac5 concretizer: added handling for dev_path variant
This variant is currently either set from command line, in
which case it enters the concretization, or attached from
environment after concretization.
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
a0e8ad7a8b concretizer: ensure upfront that variants are valid 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
ae1ef85af5 concretizer: account for test dependencies only when required 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
346beedfd4 Fix installer.py unit tests that check output 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
87c87ff767 Compute the correct package name for hierarchies that change class names 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
116f6b30eb concretizer: handle variants defined through validators
Variant of this kind don't have a list of possible
values encoded in the ASP facts. Since all we have
is a validator the list of possible values just includes
just the default value and possibly the value passed
from packages.yaml or cli.
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
ada2fa36a9 concretizer: account for patches variant
This is done after the builder has actually built
the specs, to respect the semantics use with the
old concretizer.

Later we could move this to the solver as
a multivalued variant.
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
a1fe88c95b concretizer: ensure that no deprecated spec is being used
This is done after the builder has actually built
the specs, to respect the semantics use with the
old concretizer.

A better approach is to substitute the spec
directly in concretization.
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
58683b9e56 conftest: hook the new solver in the config fixture 2020-11-17 10:04:13 -08:00
Massimiliano Culpo
3e4fd64169 concretizer: handle "none" value and '*' wildcard
The "none" variant value cannot be combined with
other values.

The '*' wildcard matches anything, including "none".
It's thus relevant in queries, but disregarded in
concretization.
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
8b055ac8d8 Fixed failing unit tests
- The test on concretization of anonymous dependencies
  has been fixed by raising the expected exception.
- The test on compiler bootstrap has been fixed by
  updating the version of GCC used in the test.
  Since gcc@2.0 does not support targets later than
  x86_64, the new concretizer was looking for a
  non-existing spec, i.e. it was correctly trying
  to retrieve 'gcc target=x86_64' instead of
  'gcc target=core2'.
- The test on gitlab CI needed an update of the target
2020-11-17 10:04:13 -08:00
Massimiliano Culpo
c047495981 concretizer: virtual entry in packages.yaml, external modules
This commit adds support for specifying rules in
packages.yaml that refer to virtual packages.

The approach is to normalize in memory each
configuration and turn it into an equivalent
configuration without rules on virtual. This
is possible if the set of packages to be handled
is considered fixed.
2020-11-17 10:04:13 -08:00