Commit Graph

4448 Commits

Author SHA1 Message Date
Harmen Stoppels
c8a10e4910 Fix clearing cache of InternalConfigScope (#22609)
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-05-22 11:51:20 -07:00
Cyrus Harrison
a5213dabb1 bugfix for active when pkg is already active error (#22587)
* bugfix for active when pkg is already active error

Co-authored-by: Greg Becker <becker33@llnl.gov>
2021-05-22 11:51:20 -07:00
Massimiliano Culpo
6e714808fa Enforce uniqueness of the version_weight atom per node
fixes #22565

This change enforces the uniqueness of the version_weight
atom per node(Package) in the DAG. It does so by applying
FTSE and adding an extra layer of indirection with the
possible_version_weight/2 atom.

Before this change it may have happened that for the same
node two different version_weight/2 were in the answer set,
each of which referred to a different spec with the same
version, and their weights would sum up.

This lead to unexpected result like preferring to build a
new version of an external if the external version was
older.
2021-05-22 11:51:20 -07:00
Massimiliano Culpo
9f99e8ad64 Externals are preferred even when they have non-default variant values
fixes #22596

Variants which are specified in an external spec are not
scored negatively if they encode a non-default value.
2021-05-22 11:51:20 -07:00
Massimiliano Culpo
18880a668b clingo: modify recipe for bootstrapping (#22354)
* clingo: modify recipe for bootstrapping

Modifications:
- clingo builds with shared Python only if ^python+shared
- avoid building the clingo app for bootstrapping
- don't link to libpython when bootstrapping

* Remove option that breaks on linux

* Give more hints for the current Python

* Disable CLINGO_BUILD_PY_SHARED for bootstrapping

* bootstrapping: try to detect the current python from std library

This is much faster than calling external executables

* Fix compatibility with Python 2.6

* Give hints on which compiler and OS to use when bootstrapping

This change hints which compiler to use for bootstrapping clingo
(either GCC or Apple Clang on MacOS). On Cray platforms it also
hints to build for the frontend system, where software is meant
to be installed.

* Use spec_for_current_python to constrain module requirement

(cherry picked from commit d5fa509b07)
2021-05-22 11:51:20 -07:00
Massimiliano Culpo
e7494b627b ASP-based solver: model disjoint sets for multivalued variants (#22534)
* ASP-based solver: avoid adding values to variants when they're set

fixes #22533
fixes #21911

Added a rule that prevents any value to slip in a variant when the
variant is set explicitly. This is relevant for multi-valued variants,
in particular for those that have disjoint sets of values.

* Ensure disjoint sets have a clear semantics for external packages
2021-05-22 11:51:20 -07:00
Massimiliano Culpo
b11dd0478a Make SingleFileScope able to repopulate the cache after clearing it (#22559)
fixes #22547

SingleFileScope was not able to repopulate its cache before this
change. This was affecting the configuration seen by environments
using clingo bootstrapped from sources, since the bootstrapping
operation involved a few cache invalidation for config files.
2021-05-22 11:51:20 -07:00
Harmen Stoppels
61e619bb27 spack location: bugfix for out of source build dirs (#22348) 2021-05-22 11:51:20 -07:00
Todd Gamblin
4d148a430e bugfix: allow imposed constraints to be overridden in special cases
In most cases, we want condition_holds(ID) to imply any imposed
constraints associated with the ID. However, the dependency relationship
in Spack is special because it's "extra" conditional -- a dependency
*condition* may hold, but we have decided that externals will not have
dependencies, so we need a way to avoid having imposed constraints appear
for nodes that don't exist.

This introduces a new rule that says that constraints are imposed
*unless* we define `do_not_impose(ID)`. This allows rules like
dependencies, which rely on more than just spec conditions, to cancel
imposed constraints.

We add one special case for this: dependencies of externals.
2021-05-22 11:51:20 -07:00
Todd Gamblin
9717e0244f bugfix: do not generate dep conditions when no dependency
We only consider test dependencies some of the time. Some packages are
*only* test dependencies. Spack's algorithm was previously generating
dependency conditions that could hold, *even* if there was no potential
dependency type.

- [x] change asp.py so that this can't happen -- we now only generate
      dependency types for possible dependencies.
2021-05-22 11:51:20 -07:00
Todd Gamblin
a823cffc40 concretizer: unify logic for spec conditionals
This builds on #20638 by unifying all the places in the concretizer where
things are conditional on specs. Previously, we duplicated a common spec
conditional pattern for dependencies, virtual providers, conflicts, and
externals. That was introduced in #20423 and refined in #20507, and
roughly looked as follows.

Given some directives in a package like:

```python
depends_on("foo@1.0+bar", when="@2.0+variant")
provides("mpi@2:", when="@1.9:")
```

We handled the `@2.0+variant` and `@1.9:` parts by generating generated
`dependency_condition()`, `required_dependency_condition()`, and
`imposed_dependency_condition()` facts to trigger rules like this:

```prolog
dependency_conditions_hold(ID, Parent, Dependency) :-
  attr(Name, Arg1)             : required_dependency_condition(ID, Name, Arg1);
  attr(Name, Arg1, Arg2)       : required_dependency_condition(ID, Name, Arg1, Arg2);
  attr(Name, Arg1, Arg2, Arg3) : required_dependency_condition(ID, Name, Arg1, Arg2, Arg3);
  dependency_condition(ID, Parent, Dependency);
  node(Parent).
```

And we handled `foo@1.0+bar` and `mpi@2:` parts ("imposed constraints")
like this:

```prolog
attr(Name, Arg1, Arg2) :-
  dependency_conditions_hold(ID, Package, Dependency),
  imposed_dependency_condition(ID, Name, Arg1, Arg2).

attr(Name, Arg1, Arg2, Arg3) :-
  dependency_conditions_hold(ID, Package, Dependency),
  imposed_dependency_condition(ID, Name, Arg1, Arg2, Arg3).
```

These rules were repeated with different input predicates for
requirements (e.g., `required_dependency_condition`) and imposed
constraints (e.g., `imposed_dependency_condition`) throughout
`concretize.lp`. In #20638 it got to be a bit confusing, because we used
the same `dependency_condition_holds` predicate to impose constraints on
conditional dependencies and virtual providers. So, even though the
pattern was repeated, some of the conditional rules were conjoined in a
weird way.

Instead of repeating this pattern everywhere, we now have *one* set of
consolidated rules for conditions:

```prolog
condition_holds(ID) :-
  condition(ID);
  attr(Name, A1)         : condition_requirement(ID, Name, A1);
  attr(Name, A1, A2)     : condition_requirement(ID, Name, A1, A2);
  attr(Name, A1, A2, A3) : condition_requirement(ID, Name, A1, A2, A3).

attr(Name, A1)         :- condition_holds(ID), imposed_constraint(ID, Name, A1).
attr(Name, A1, A2)     :- condition_holds(ID), imposed_constraint(ID, Name, A1, A2).
attr(Name, A1, A2, A3) :- condition_holds(ID), imposed_constraint(ID, Name, A1, A2, A3).
```

this allows us to use `condition(ID)` and `condition_holds(ID)` to
encapsulate the conditional logic on specs in all the scenarios where we
need it. Instead of defining predicates for the requirements and imposed
constraints, we generate the condition inputs with generic facts, and
define predicates to associate the condition ID with a particular
scenario. So, now, the generated facts for a condition look like this:

```prolog
condition(121).
condition_requirement(121,"node","cairo").
condition_requirement(121,"variant_value","cairo","fc","True").
imposed_constraint(121,"version_satisfies","fontconfig","2.10.91:").
dependency_condition(121,"cairo","fontconfig").
dependency_type(121,"build").
dependency_type(121,"link").
```

The requirements and imposed constraints are generic, and we associate
them with their meaning via the id. Here, `dependency_condition(121,
"cairo", "fontconfig")` tells us that condition 121 has to do with the
dependency of `cairo` on `fontconfig`, and the conditional dependency
rules just become:

```prolog
dependency_holds(Package, Dependency, Type) :-
  dependency_condition(ID, Package, Dependency),
  dependency_type(ID, Type),
  condition_holds(ID).
```

Dependencies, virtuals, conflicts, and externals all now use similar
patterns, and the logic for generating condition facts is common to all
of them on the python side, as well. The more specific routines like
`package_dependencies_rules` just call `self.condition(...)` to get an id
and generate requirements and imposed constraints, then they generate
their extra facts with the returned id, like this:

```python
    def package_dependencies_rules(self, pkg, tests):
        """Translate 'depends_on' directives into ASP logic."""
        for _, conditions in sorted(pkg.dependencies.items()):
            for cond, dep in sorted(conditions.items()):
                condition_id = self.condition(cond, dep.spec, pkg.name)  # create a condition and get its id
                self.gen.fact(fn.dependency_condition(  # associate specifics about the dependency w/the id
                    condition_id, pkg.name, dep.spec.name
                ))
        # etc.
```

- [x] unify generation and logic for conditions
- [x] use unified logic for dependencies
- [x] use unified logic for virtuals
- [x] use unified logic for conflicts
- [x] use unified logic for externals

LocalWords:  concretizer mpi attr Arg concretize lp cairo fc fontconfig
LocalWords:  virtuals def pkg cond dep fn refactor github py
2021-05-22 11:51:20 -07:00
Massimiliano Culpo
f1c7402c64 bootstrap: account for platform specific configuration scopes (#22489)
This change accounts for platform specific configuration scopes,
like ~/.spack/linux, during bootstrapping. These scopes were
previously not accounted for and that was causing issues e.g.
when searching for compilers.

(cherry picked from commit 413c422e53)
2021-05-22 11:51:20 -07:00
Massimiliano Culpo
16f7a02654 Bootstrap clingo from sources (#21446)
* Allow the bootstrapping of clingo from sources

Allow python builds with system python as external
for MacOS

* Ensure consistent configuration when bootstrapping clingo

This commit uses context managers to ensure we can
bootstrap clingo using a consistent configuration
regardless of the use case being managed.

* Github actions: test clingo with bootstrapping from sources

* Add command to inspect and clean the bootstrap store

 Prevent users to set the install tree root to the bootstrap store

* clingo: documented how to bootstrap from sources

Co-authored-by: Gregory Becker <becker33@llnl.gov>
(cherry picked from commit 10e9e142b7)
2021-05-22 11:50:59 -07:00
Andreas Baumbach
bd9929f9dc make spack fetch work with environments (#19166)
* make `spack fetch` work with environments
* previously: `spack fetch` required the explicit statement of
              the specs to be fetched, even when in an environment
* now: if there is no spec(s) provided to `spack fetch` we check
       if an environment is active and if yes we fetch all
       uninstalled specs.
2021-05-21 19:42:12 +02:00
Massimiliano Culpo
2a5f46d8d3 Added a context manager to swap architectures
This solves a few FIXMEs in conftest.py, where
we were manipulating globals and seeing side
effects prior to registering fixtures.

This commit solves the FIXMEs, but introduces
a performance regression on tests that may need
to be investigated

(cherry picked from commit 4558dc06e2)
2021-05-21 18:06:49 +02:00
Greg Becker
095ace9028 bugfix for target adjustments on target ranges (#20537)
(cherry picked from commit 61c1b71d38)
2021-05-21 18:05:42 +02:00
Massimiliano Culpo
f30fc6cd33 Move context manager to swap the current configuration into spack.config
The context manager can be used to swap the current
configuration temporarily, for any use case that may need it.

(cherry picked from commit 553d37a6d6)
2021-05-21 18:00:05 +02:00
Massimiliano Culpo
4e5e1e8f35 Move context manager to swap the current store into spack.store
The context manager can be used to swap the current
store temporarily, for any use case that may need it.

(cherry picked from commit cb2c233a97)
2021-05-21 17:59:57 +02:00
Massimiliano Culpo
0678d5df90 repo: generalize "swap" context manager to also accept paths
The method is now called "use_repositories" and
makes it clear in the docstring that it accepts
as arguments either Repo objects or paths.

Since there was some duplication between this
contextmanager and "use_repo" in the testing framework,
remove the latter and use spack.repo.use_repositories
across the entire code base.

Make a few adjustment to MockPackageMultiRepo, since it was
stating in the docstring that it was supposed to mock
spack.repo.Repo and was instead mocking spack.repo.RepoPath.

(cherry picked from commit 1a8963b0f4)
2021-05-21 17:59:50 +02:00
Josh Essman
14e179398f Updates to support clingo-cffi (#20657)
* Support clingo when used with cffi

Clingo recently merged in a new Python module option based on cffi.

Compatibility with this module requires a few changes to spack - it does not automatically convert strings/ints/etc to Symbol and clingo.Symbol.string throws on failure.

manually convert str/int to clingo.Symbol types
catch stringify exceptions
add job for clingo-cffi to Spack CI
switch to potassco-vendored wheel for clingo-cffi CI
on_unsat argument when cffi

(cherry picked from commit 93ed1a410c)
2021-05-21 15:09:08 +02:00
Massimiliano Culpo
8d13193434 Improve error message for inconsistencies in package.py (#21811)
* Improve error message for inconsistencies in package.py

Sometimes directives refer to variants that do not exist.
Make it such that:

1. The name of the variant
2. The name of the package which is supposed to have
   such variant
3. The name of the package making this assumption

are all printed in the error message for easier debugging.

* Add unit tests

(cherry picked from commit 7226bd64dc)
2021-05-21 15:09:08 +02:00
Massimiliano Culpo
94bb37c107 concretizer: simplify "fact" method (#21148)
The "fact" method before was dealing with multiple facts
registered per call, which was used when we were emitting
grounded rules from knowledge of the problem instance.

Now that the encoding is changed we can simplify the method
to deal only with a single fact per call.

(cherry picked from commit ba42c36f00)
2021-05-21 15:09:04 +02:00
Adam J. Stewart
a59fcd60f5 Python 3.10 support: collections.abc (#20441)
(cherry picked from commit 40a40e0265)
2021-05-21 14:55:18 +02:00
Greg Becker
8c05387ebc respect -k/verify-ssl-false in _existing_url method (#21864) 2021-05-10 23:11:29 -07:00
Gregory Becker
a3714b3292 updates for new tutorial
update s3 bucket
update tutorial branch
2021-04-15 00:04:29 -07:00
Tamara Dahlgren
8dd2d740b1 Update CHANGELOG and release version 2021-02-19 11:06:33 -08:00
Tamara Dahlgren
d0798160cf Resolve (post-cherry-picking) flake8 errors 2021-02-19 11:05:53 -08:00
Adam J. Stewart
3256f018eb apple-clang: add correct path to compiler wrappers (#21662)
Follow-up to #17110

### Before
```bash
CC=/Users/Adam/spack/lib/spack/env/clang/clang; export CC
SPACK_CC=/usr/bin/clang; export SPACK_CC
PATH=...:/Users/Adam/spack/lib/spack/env/apple-clang:/Users/Adam/spack/lib/spack/env/case-insensitive:/Users/Adam/spack/lib/spack/env:...; export PATH
```

### After
```bash
CC=/Users/Adam/spack/lib/spack/env/clang/clang; export CC
SPACK_CC=/usr/bin/clang; export SPACK_CC
PATH=...:/Users/Adam/spack/lib/spack/env/clang:/Users/Adam/spack/lib/spack/env/case-insensitive:/Users/Adam/spack/lib/spack/env:...; export PATH
```

`CC` and `SPACK_CC` were being set correctly, but `PATH` was using the name of the compiler `apple-clang` instead of `clang`. For most packages, since `CC` was set correctly, nothing broke. But for packages using `Makefiles` that set `CC` based on `which clang`, it was using the system compilers instead of the compiler wrappers. Discovered when working on `py-xgboost@0.90`.

An alternative fix would be to copy the symlinks in `env/clang` to `env/apple-clang`. Let me know if you think there's a better way to do this, or to test this.
2021-02-17 17:07:48 -08:00
Greg Becker
240726a2e1 add intel oneapi to compiler/pkg translations (#21448) 2021-02-17 17:07:46 -08:00
Yang Zongze
40d32890d0 Print groups properly for spack find -d (#20028) 2021-02-17 17:07:45 -08:00
eugeneswalker
18022050f2 store sbang_install_path in buildinfo, use for subsequent relocation (#20768) 2021-02-17 17:07:44 -08:00
Nathan Hanford
cdd86bddec [WIP] relocate.py: parallelize test replacement logic (#19690)
* sbang pushed back to callers;
star moved to util.lang

* updated unit test

* sbang test moved; local tests pass

Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2021-02-17 17:07:44 -08:00
Massimiliano Culpo
488a08b464 concretizer: require at least a dependency type to say the dependency holds
fixes #20784

Similarly to the previous bug, here we were deducing
conditions to be imposed on nodes that were not part
of the DAG.
2021-02-17 17:07:43 -08:00
Massimiliano Culpo
7162e15531 concretizer: dependency conditions cannot hold if package is external
fixes #20736

Before this one line fix we were erroneously deducing
that dependency conditions hold even if a package
was external.

This may result in answer sets that contain imposed
conditions on a node without the node being present
in the DAG, hence #20736.
2021-02-17 17:07:42 -08:00
Robert Underwood
c99850dd58 restore ability of dev-build to skip patches (#20351)
At some point in the past, the skip_patch argument was removed
from the call to package.do_install() this broke the --skip-patch
flag on the dev-build command.
2021-02-17 17:07:41 -08:00
Massimiliano Culpo
ddd9c86ce0 concretizer: make rules on virtual packages more linear
fixes #20679

In this refactor we have a single cardinality rule on the
provider, which triggers a rule transforming a dependency
on a virtual package into a dependency on the provider of
the virtual.
2021-02-17 17:07:39 -08:00
Todd Gamblin
8e442d6dc1 concretizer: use consistent naming for compiler predicates (#20677)
Every other predicate in the concretizer uses a `_set` suffix to
implement user- or package-supplied settings, but compiler settings use a
`_hard` suffix for this. There's no difference in how they're used, so
make the names the same.

- [x] change `node_compiler_hard` to `node_compiler_set`
- [x] change `node_compiler_version_hard` to `node_compiler_version_set`
2021-02-17 17:07:38 -08:00
Todd Gamblin
9393d97139 concretizer: simplify handling of virtual version constraints
Previously, the concretizer handled version constraints by comparing all
pairs of constraints and ensuring they satisfied each other. This led to
INCONSISTENT ressults from clingo, due to ambiguous semantics like:

    version_constraint_satisfies("mpi", ":1", ":3")
    version_constraint_satisfies("mpi", ":3", ":1")

To get around this, we introduce possible (fake) versions for virtuals,
based on their constraints. Essentially, we add any Versions,
VersionRange endpoints, and all such Versions and endpoints from
VersionLists to the constraint. Virtuals will have one of these synthetic
versions "picked" by the solver. This also allows us to remove a special
case from handling of `version_satisfies/3` -- virtuals now work just
like regular packages.
2021-02-17 17:07:38 -08:00
Todd Gamblin
8f85ab88c0 concretizer: remove rule generation code from concretizer
Our program only generates facts now, so remove all unused code related
to generating cardinality constraints and rules.
2021-02-17 17:07:37 -08:00
Todd Gamblin
247e73e85a concretizer: convert virtuals to facts; move all rules to concretize.lp
This converts the virtual handling in the new concretizer from
already-ground rules to facts. This is the last thing that needs to be
refactored, and it converts the entire concretizer to just use facts.

The previous way of handling virtuals hinged on rules involving
`single_provider_for` facts that were tied to the virtual and a version
range. The new method uses the condition pattern we've been using for
dependencies, externals, and conflicts.

To handle virtuals as conditions, we impose constraints on "fake" virtual
specs in the logic program. i.e., `version_satisfies("mpi", "2.0:",
"2.0")` is legal whereas before we wouldn't have seen something like
this. Currently, constriants are only handled on versions -- we don't
handle variants or anything else yet, but they key change here is that we
*could*. For a long time, virtual handling in Spack has only dealt with
versions, and we'd like to be able to handle variants as well. We could
easily add an integrity constraint to handle variants like the one we use
for versions.

One issue with the implementation here is that virtual packages don't
actually declare possible versions like regular packages do. To get
around that, we implement an integrity constraint like this:

    :- virtual_node(Virtual),
       version_satisfies(Virtual, V1), version_satisfies(Virtual, V2),
       not version_constraint_satisfies(Virtual, V1, V2).

This requires us to compare every version constraint to every other, both
in program generation and within the concretizer -- so there's a
potentially quadratic evaluation time on virtual constraints because we
don't have a real version to "anchor" things to. We just say that all the
constraints need to agree for the virtual constraint to hold.

We can investigate adding synthetic versions for virtuals in the future,
to speed this up.
2021-02-17 17:07:37 -08:00
Todd Gamblin
6056cb71d3 concretizer: consolidate handling of virtuals into spec_clauses 2021-02-17 17:07:36 -08:00
Todd Gamblin
66376ab971 concretizer: make _condtion_id_counter an iterator 2021-02-17 17:07:36 -08:00
Todd Gamblin
9bb0375443 concretizer: more detailed section headers in concretize.lp 2021-02-17 17:07:35 -08:00
Todd Gamblin
56acb5a0c7 bugfix: infinite loop when building a set from incomplete specs (#20649)
This code in `SpecBuilder.build_specs()` introduced in #20203, can loop
seemingly interminably for very large specs:

```python
set([spec.root for spec in self._specs.values()])
```

It's deceptive, because it seems like there must be an issue with
`spec.root`, but that works fine. It's building the set afterwards that
takes forever, at least on `r-rminer`. Currently if you try running
`spack solve r-rminer`, it loops infinitely and spins up your fan.

The issue (I think) is that the spec is not yet complete when this is
run, and something is going wrong when constructing and comparing so many
values produced by `_cmp_key()`. We can investigate the efficiency of
`_cmp_key()` separately, but for now, the fix is:

```python
roots = [spec.root for spec in self._specs.values()]
roots = dict((id(r), r) for r in roots)
```

We know the specs in `self._specs` are distinct (they just came out of
the solver), so we can just use their `id()` to unique them here. This
gets rid of the infinite loop.
2021-02-17 17:07:35 -08:00
Massimiliano Culpo
ec42016241 concretizer: generate facts for externals
Generate only facts for external specs. Substitute the
use of already grounded rules with non-grounded rules
in concretize.lp
2021-02-17 17:07:35 -08:00
Greg Becker
f7195123d4 bugfix: do not write empty default dicts/lists in envs (#20526)
Environment yaml files should not have default values written to them.

To accomplish this, we change the validator to not add the default values to yaml. We rely on the code to set defaults for all values (and use defaulting getters like dict.get(key, default)).

Includes regression test.
2021-02-17 17:07:34 -08:00
Robert Cohn
290043b72a Add Intel oneAPI packages (#20411)
This creates a set of packages which all use the same script to install
components of Intel oneAPI. This includes:

* An inheritable IntelOneApiPackage which knows how to invoke the
  installation script based on which components are requested
* For components which include headers/libraries, an inheritable
  IntelOneApiLibraryPackage is provided to locate them
* Individual packages for DAL, DNN, TBB, etc.
* A package for the Intel oneAPI compilers (icx/ifx). This also includes
  icc/ifortran but these are not currently detected in this PR
2021-02-17 17:07:33 -08:00
Todd Gamblin
1a1babe185 concretizer: refactor conditional rules to be less repetitious (#20507)
We have to repeat all the spec attributes in a number of places in
`concretize.lp`, and Spack has a fair number of spec attributes. If we
instead add some rules up front that establish equivalencies like this:

```
    node(Package) :- attr("node", Package).
    attr("node", Package) :- node(Package).

    version(Package, Version) :- attr("version", Package, Version).
    attr("version", Package, Version) :- version(Package, Version).
```

We can rewrite most of the repetitive conditions with `attr` and repeat
only for each arity (there are only 3 arities for spec attributes so far)
as opposed to each spec attribute. This makes the logic easier to read
and the rules easier to follow.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-02-17 17:07:32 -08:00
Massimiliano Culpo
acd523c7f3 concretizer: optimize loop on compiler version
Similar to the optimization on platform
2021-02-17 17:07:32 -08:00
Massimiliano Culpo
4bbc6eec51 concretizer: optimized loop on node platforms
We can speed-up the computation by avoiding a
double loop in a cardinality constraint and
enforcing the rule instead as an integrity
constraint.
2021-02-17 17:07:31 -08:00