Commit Graph

130 Commits

Author SHA1 Message Date
Todd Gamblin
9e54134daf
docs: Replace package list with packages.spack.io (#40251)
For a long time, the docs have generated a huge, static HTML package list. It has some
disadvantages:

* It's slow to load
* It's slow to build
* It's hard to search

We now have a nice website that can tell us about Spack packages, and it's searchable so
users can easily find the one or two packages out of 7400 that they're looking for. We
should link to this instead of including a static package list page in the docs.

- [x] Replace package list link with link to packages.spack.io
- [x] Remove `package_list.html` generation from `conf.py`.
- [x] Add a new section for "Links" to the docs.
- [x] Remove docstring notes from contribution guide (we haven't generated RST
      for package docstrings for a while)
- [x] Remove referencese to `package-list` from docs.
2023-10-01 05:36:22 +02:00
Harmen Stoppels
fa7719a031
Improve version, version range, and version list syntax and behavior (#36273)
## Version types, parsing and printing

- The version classes have changed: `VersionBase` is removed, there is now a
  `ConcreteVersion` base class. `StandardVersion` and `GitVersion` both inherit
  from this.

- The public api (`Version`, `VersionRange`, `ver`) has changed a bit:
  1. `Version` produces either `StandardVersion` or `GitVersion` instances.
  2. `VersionRange` produces a `ClosedOpenRange`, but this shouldn't affect the user.
  3. `ver` produces any of `VersionList`, `ClosedOpenRange`, `StandardVersion`
     or `GitVersion`.

- No unexpected type promotion, so that the following is no longer an identity:
  `Version(x) != VersionRange(x, x)`.

- `VersionList.concrete` now returns a version if it contains only a single element
  subtyping `ConcreteVersion` (i.e. `StandardVersion(...)` or `GitVersion(...)`)

- In version lists, the parser turns `@x` into `VersionRange(x, x)` instead
  of `Version(x)`.

- The above also means that `ver("x")` produces a range, whereas
  `ver("=x")` produces a `StandardVersion`. The `=` is part of _VersionList_
  syntax.

- `VersionList.__str__` now outputs `=x.y.z` for specific version entries,
  and `x.y.z` as a short-hand for ranges `x.y.z:x.y.z`.

- `Spec.format` no longer aliases `{version}` to `{versions}`, but pulls the
  concrete version out of the list and prints that -- except when the list is
  is not concrete, then is falls back to `{versions}` to avoid a pedantic error.
  For projections of concrete specs, `{version}` should be used to render
  `1.2.3` instead of `=1.2.3` (which you would get with `{versions}`).
  The default `Spec` format string used in `Spec.__str__` now uses
  `{versions}` so that `str(Spec(string)) == string` holds.

## Changes to `GitVersion`

- `GitVersion` is a small wrapper around `StandardVersion` which enriches it
   with a git ref. It no longer inherits from it.

- `GitVersion` _always_ needs to be able to look up an associated Spack version
  if it was not assigned (yet). It throws a `VersionLookupError` whenever `ref_version`
  is accessed but it has no means to look up the ref; in the past Spack would
  not error and use the commit sha as a literal version, which was incorrect.
   
- `GitVersion` is never equal to `StandardVersion`, nor is satisfied by it. This
  is such that we don't lose transitivity. This fixes the following bug on `develop`
  where `git_version_a == standard_version == git_version_b` does not imply
  `git_version_a == git_version_b`. It also ensures equality always implies equal
  hash, which is also currently broken on develop; inclusion tests of a set of
  versions + git versions would behave differently from inclusion tests of a
  list of the same objects.

- The above means `ver("ref=1.2.3) != ver("=1.2.3")` could break packages that branch
  on specific versions, but that was brittle already, since the same happens with
  externals: `pkg@1.2.3-external` suffixes wouldn't be exactly equal either. Instead,
  those checks should be `x.satisfies("@1.2.3")` which works both for git versions and
  custom version suffixes.

- `GitVersion` from commit will now print as `<hash>=<version>` once the
  git ref is resolved to a spack version. This is for reliability -- version is frozen
  when added to the database and queried later. It also improves performance
  since there is no need to clone all repos of all git versions after `spack clean -m`
  is run and something queries the database, triggering version comparison, such
  as potentially reuse concretization.

- The "empty VerstionStrComponent trick" for `GitVerison` is dropped since it wasn't
  representable as a version string (by design). Instead, it's replaced by `git`,
  so you get `1.2.3.git.4` (which reads 4 commits after a tag 1.2.3). This means
  that there's an edge case for version schemes `1.1.1`, `1.1.1a`, since the
  generated git version `1.1.1.git.1` (1 commit after `1.1.1`) compares larger
  than `1.1.1a`, since `a < git` are compared as strings. This is currently a
  wont-fix edge case, but if really required, could be fixed by special casing
  the `git` string.

- Saved, concrete specs (database, lock file, ...) that only had a git sha as their
  version, but have no means to look the effective Spack version anymore, will
  now see their version mapped to `hash=develop`. Previously these specs
  would always have their sha literally interpreted as a version string (even when
  it _could_ be looked up). This only applies to databases, lock files and spec.json
  files created before Spack 0.20; after this PR, we always have a Spack version
  associated to the relevant GitVersion).

- Fixes a bug where previously `to_dict` / `from_dict` (de)serialization would not
  reattach the repo to the GitVersion, causing the git hash to be used as a literal
  (bogus) version instead of the resolved version. This was in particularly breaking
  version comparison in the build process on macOS/Windows.


## Installing or matching specific versions

- In the past, `spack install pkg@3.2` would install `pkg@=3.2` if it was a
  known specific version defined in the package, even when newer patch releases
  `3.2.1`, `3.2.2`, `...` were available. This behavior was only there because
  there was no syntax to distinguish between `3.2` and `3.2.1`. Since there is
  syntax for this now through `pkg@=3.2`, the old exact matching behavior is
  removed. This means that `spack install pkg@3.2` constrains the `pkg` version
  to the range `3.2`, and `spack install pkg@=3.2` constrains it to the specific
  version `3.2`.

- Also in directives such as `depends_on("pkg@2.3")` and their when
  conditions `conflicts("...", when="@2.3")` ranges are ranges, and specific
  version matches require `@=2.3.`.

- No matching version: in the case `pkg@3.2` matches nothing, concretization
  errors. However, if you run `spack install pkg@=3.2` and this version
  doesn't exist, Spack will define it; this allows you to install non-registered
  versions.

- For consistency, you can now do `%gcc@10` and let it match a configured
  `10.x.y` compiler. It errors when there is no matching compiler.
  In the past it was interpreted like a specific `gcc@=10` version, which
  would get bootstrapped.

- When compiler _bootstrapping_ is enabled, `%gcc@=10.2.0` can be used to
  bootstrap a specific compiler version.

## Other changes

- Externals, compilers, and develop spec definitions are backwards compatible.
  They are typically defined as `pkg@3.2.1` even though they should be
  saying `pkg@=3.2.1`. Spack now transforms `pkg@3` into `pkg@=3` in those cases.

- Finally, fix strictness of `version(...)` directive/declaration. It just does a simple
  type check, and now requires strings/integers. Floats are not allowed because
  they are ambiguous `str(3.10) == "3.1"`.
2023-05-05 22:04:41 -06:00
Xavier Delaruelle
df97827a7b
Fix case spelling for Lmod and Tcl (#36215) 2023-03-19 01:42:50 +00:00
Harmen Stoppels
fce95e2efb
license year bump (#34921)
* license bump year
* fix black issues of modified files
* mypy
* fix 2021 -> 2023
2023-01-18 14:30:17 -08:00
Michael Kuhn
901b31a7aa
docs: fix typo (#33926) 2022-11-15 16:06:12 -08:00
Harmen Stoppels
c14dc2f56a
docs: updates related to extensions (#33837) 2022-11-11 13:24:17 +01:00
Harmen Stoppels
0f54a63dfd
remove activate/deactivate support in favor of environments (#29317)
Environments and environment views have taken over the role of `spack activate/deactivate`, and we should deprecate these commands for several reasons:

- Global activation is a really poor idea:
   - Install prefixes should be immutable; since they can have multiple, unrelated dependents; see below
   - Added complexity elsewhere: verification of installations, tarballs for build caches, creation of environment views of packages with unrelated extensions "globally activated"... by removing the feature, it gets easier for people to contribute, and we'd end up with fewer bugs due to edge cases.
- Environment accomplish the same thing for non-global "activation" i.e. `spack view`, but better.

Also we write in the docs:

```
However, Spack global activations have two potential drawbacks:

#. Activated packages that involve compiled C extensions may still
   need their dependencies to be loaded manually.  For example,
   ``spack load openblas`` might be required to make ``py-numpy``
   work.

#. Global activations "break" a core feature of Spack, which is that
   multiple versions of a package can co-exist side-by-side.  For example,
   suppose you wish to run a Python package in two different
   environments but the same basic Python --- one with
   ``py-numpy@1.7`` and one with ``py-numpy@1.8``.  Spack extensions
   will not support this potential debugging use case.
```

Now that environments are established and views can take over the role of activation
non-destructively, we can remove global activation/deactivation.
2022-11-11 00:50:07 -08:00
Kayla Butler
bc209c470d flags/variants: Add ++/~~/== syntax for propagation to dependencies
Currently, compiler flags and variants are inconsistent: compiler flags set for a
package are inherited by its dependencies, while variants are not. We should have these
be consistent by allowing for inheritance to be enabled or disabled for both variants
and compiler flags.

- [x] Make new (spec language) operators
- [x] Apply operators to variants and compiler flags
- [x] Conflicts currently result in an unsatisfiable spec
      (i.e., you can't propagate two conflicting values)

What I propose is using two of the currently used sigils to symbolized that the variant
or compiler flag will be inherited:

Example syntax:
- `package ++variant`
      enabled variant that will be propagated to dependencies
- `package +variant`
      enabled variant that will NOT be propagated to dependencies
- `package ~~variant`
      disabled variant that will be propagated to dependencies
- `package ~variant`
      disabled variant that will NOT be propagated to dependencies
- `package cflags==True`
      `cflags` will be propagated to dependencies
- `package cflags=True`
      `cflags` will NOT be propagated to dependencies

Syntax for string-valued variants is similar to compiler flags.
2022-11-02 09:43:57 -07:00
Greg Sjaardema
e0265745bc
Update command option for example (#33321)
The `spack info <package>` command does not show the `Virtual Packages:` output unless the `--virtuals` command option is passed.  Before this changes, the information that the command is supposed to be illustrating is not shown in the example and is confusing.
2022-11-01 15:29:50 -06:00
Tamara Dahlgren
13356f3bfa
Docs: Spack info option updates (#33376) 2022-10-18 21:11:21 +02:00
Harmen Stoppels
ceda5fb46c
Don't set LD_LIBRARY_PATH by default on Linux (#28354)
`LD_LIBRARY_PATH` can break system executables (e.g., when an enviornment is loaded) and isn't necessary thanks to `RPATH`s.  Packages that require `LD_LIBRARY_PATH` can set this in `setup_run_environment`.

- [x] Prefix inspections no longer set `LD_LIBRARY_PATH` by default
- [x] Document changes and workarounds for people who want `LD_LIBRARY_PATH`
2022-08-11 09:33:08 -05:00
psakievich
ddc373b4e1
Fix doc format for code blocks: GitRef versions (#31937)
Rendering was not pretty using `console` from #31914
2022-08-05 02:01:03 +00:00
psakievich
d67ead5978
Add documentation for git refs as versions (#31914)
* document git commit versions

Include documentation for manually specifying associated known version

* document spack develop command

Co-authorerd-by: psakievich <psakiev@sandia.gov>
Co-authored-by: Gregory Becker <becker33@llnl.gov>
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-08-04 13:20:33 -07:00
Massimiliano Culpo
d900ac2003
Reuse concretization by default (#30396)
* Enable reuse by default in Spack
* Update documentation to match new default
* Configure pipelines not to reuse software
2022-05-13 09:11:10 -07:00
Massimiliano Culpo
2fa495154e
Split the workflow section and remove outdated advices (#29344)
This PR removes a few outdated sections from the "Basics" part of the 
documentation. It also makes a few topic under the environment section
more prominent by removing an unneeded spack.yaml subsection and 
promoting everything under it.
2022-03-18 10:41:27 +01:00
Todd Gamblin
d33973df6c docs: add section on concretizer configuration
* Document `concretizer.yaml`, `--reuse`, and `--fresh`.
2022-02-16 10:17:18 -08:00
Todd Gamblin
93377942d1 Update copyright year to 2022 2022-01-14 22:50:21 -08:00
Todd Gamblin
e13e697067
commands: spack load --list alias for spack find --loaded (#27184)
See #25249 and https://github.com/spack/spack/pull/27159#issuecomment-958163679.
This adds `spack load --list` as an alias for `spack find --loaded`.  The new command is
not as powerful as `spack find --loaded`, as you can't combine it with all the queries or
formats that `spack find` provides.  However, it is more intuitively located in the command
structure in that it appears in the output of `spack load --help`.

The idea here is that people can use `spack load --list`  for simple stuff but fall back to
`spack find --loaded` if they need more.

- add help to `spack load --list` that references `spack find`
- factor some parts of `spack find` out to be called from `spack load`
- add shell tests
- update docs

Co-authored-by: Peter Josef Scheibel <scheibel1@llnl.gov>
Co-authored-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>
2021-11-05 00:58:29 -07:00
Todd Gamblin
8e76244266 docs for experimental --reuse argument to spack install
Add docs for `--reuse`, along with a warning that it will likely be
removed and refactored.
2021-11-05 00:15:47 -07:00
Richarda Butler
1a3747b2b3
Update docs how to display loaded modules (#27159)
* Update spack load docs
2021-11-02 22:12:08 -07:00
Alexander Jaust
50a2316a15
Add missing spack command in basic usage tutorial (#26646)
The `find` command was missing for the examples forcing colorized output. Without this (or another suitable) command, spack produces output that is not using any color. Thus, without the `find` command one does not see any difference between forced colorized and non-colorized output.
2021-10-12 19:23:53 +02:00
Harmen Stoppels
b22728d55c
Support older py-pygments (#25456)
`markdown` is only supported since py-pygments@2.8.0:, see
9647d2ae50

Let's allow old versions too again.
2021-08-18 09:49:51 +02:00
Todd Gamblin
ab5954520f
spack diff: make output order deterministic (#25169)
The output order for `spack diff` is nondeterministic for larger diffs -- if you
ran it several times it will not put the fields in the spec in the same order on
successive invocations.

This makes a few fixes to `spack diff`:

- [x] Implement the change discussed in https://github.com/spack/spack/pull/22283#discussion_r598337448
      to make `AspFunction` comparable in and of itself and to eliminate the need for `to_tuple()`

- [x] Sort the lists of diff properties so that the output is always in the same order.

- [x] Make the output for different fields the same as what we use in the solver. Previously, we
      would use `Type(value)` for non-string values and `value` for strings.  Now we just use
      the value.  So the output looks a little cleaner:

      ```
      == Old ==========================        == New ====================
      @@ node_target @@                        @@ node_target @@
      -  gdbm Target(x86_64)                   -  gdbm x86_64
      +  zlib Target(skylake)                  +  zlib skylake
      @@ variant_value @@                      @@ variant_value @@
      -  ncurses symlinks bool(False)          -  ncurses symlinks False
      +  zlib optimize bool(True)              +  zlib optimize True
      @@ version @@                            @@ version @@
      -  gdbm Version(1.18.1)                  -  gdbm 1.18.1
      +  zlib Version(1.2.11)                  +  zlib 1.2.11
      @@ node_os @@                            @@ node_os @@
      -  gdbm catalina                         -  gdbm catalina
      +  zlib catalina                         +  zlib catalina
      ```

I suppose if we want to use `repr()` in the output we could do that and could be
consistent but we don't do that elsewhere -- the types of things in Specs are
all stringifiable so the string and the name of the attribute (`version`, `node_os`,
etc.) are sufficient to know what they are.
2021-08-01 05:15:33 +00:00
Vanessasaurus
54e8e19a60
adding spack diff command (#22283)
A `spack diff` will take two specs, and then use the spack.solver.asp.SpackSolverSetup to generate
lists of facts about each (e.g., nodes, variants, etc.) and then take a set difference between the
two to show the user the differences.

Example output:

    $ spack diff python@2.7.8 python@3.8.11
     ==> Warning: This interface is subject to change.

     --- python@2.7.8/tsxdi6gl4lihp25qrm4d6nys3nypufbf
     +++ python@3.8.11/yjtseru4nbpllbaxb46q7wfkyxbuvzxx
     @@ variant_value @@
     -  python patches a8c52415a8b03c0e5f28b5d52ae498f7a7e602007db2b9554df28cd5685839b8
     +  python patches 0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87
     @@ version @@
     -  openssl Version(1.0.2u)
     +  openssl Version(1.1.1k)
     -  python Version(2.7.8)
     +  python Version(3.8.11)

Currently this uses diff-like output but we will attempt to improve on this in the future.

One use case for `spack diff` is whenever a user has a disambiguate situation and cannot 
remember how two different installs are different. The command can also output `--json` in
the case of a more analysis type use case where we want to save complete data with all
diffs and the intersection. However, the command is really more intended for a command
line use case, and we likely will have an analyzer more suited to saving data

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2021-07-30 00:08:38 -07:00
Massimiliano Culpo
32f1aa607c
Add an audit system to Spack (#23053)
Add a new "spack audit" command. This command can check for issues
with configuration or with packages and is intended to help a
user debug a failed Spack build. 

In some cases the reported issues are always errors but are too
costly to check for (e.g. packages that specify missing variants on
dependencies). In other cases the issues may be legitimate but
uncommon usage of Spack and we want to be sure the user intended the
behavior (e.g. duplicate compiler definitions).

Audits are grouped by theme, and for now the two themes are packages
and configuration. For example you can run all available audits
on packages with "spack audit packages". It is intended that in
the future users will be able to define their own audits.

The package audits are good candidates for running in package_sanity
(i.e. they could catch bugs in user-submitted packages before they
are merged) but that is left for a later PR.
2021-06-18 07:52:08 -06:00
Harmen Stoppels
3f4c9aeca7
Read colorization from environment variable, if command line is not set (#23130) 2021-04-28 13:03:25 +00:00
Vanessasaurus
e6de04d149
docs: spack does not have a variant debug for libelf (#23021)
Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-04-16 09:29:42 +02:00
Todd Gamblin
a8ccb8e116 copyrights: update all files with license headers for 2021
- [x] add `concretize.lp`, `spack.yaml`, etc. to licensed files
- [x] update all licensed files to say 2013-2021 using
      `spack license update-copyright-year`
- [x] appease mypy with some additions to package.py that needed
      for oneapi.py
2021-01-02 12:12:00 -08:00
Greg Becker
3840c0ac45
docs: fix spack install debug arg order (#20428) 2020-12-16 13:57:08 -08:00
Michael Kuhn
20367e472d
cmd: add spack mark command (#16662)
This adds a new `mark` command that can be used to mark packages as either
explicitly or implicitly installed. Apart from fixing the package
database after installing a dependency manually, it can be used to
implement upgrade workflows as outlined in #13385.

The following commands demonstrate how the `mark` and `gc` commands can be
used to only keep the current version of a package installed:
```console
$ spack install pkgA
$ spack install pkgB
$ git pull # Imagine new versions for pkgA and/or pkgB are introduced
$ spack mark -i -a
$ spack install pkgA
$ spack install pkgB
$ spack gc
```

If there is no new version for a package, `install` will simply mark it as
explicitly installed and `gc` will not remove it.

Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-11-18 03:20:56 -08:00
Peter Scheibel
c9ad2affcc
Documentation: spack load/environments prefix inspections (#19961)
As of #18260, `spack load` and `spack env activate` now use
`prefix_inspections` from the modules configuration to decide
how to modify environment variables.

This updates the modules configuration documentation to describe
how to update environment variables with the `prefix_inspections`
section. This also updates the `spack load` and environments
documentation to refer to the new `prefix_inspections` documentation.
2020-11-17 15:24:00 -08:00
Tamara Dahlgren
6fa6af1070
Support parallel environment builds (#18131)
As of #13100, Spack installs the dependencies of a _single_ spec in parallel.
Environments, when installed, can only get parallelism from each individual
spec, as they're installed in order.  This PR makes entire environments build
in parallel by extending Spack's package installer to accept multiple root
specs.  The install command and Environment class have been updated to use
the new parallel install method.

The specs and kwargs for each *uninstalled* package (when not force-replacing
installations) of an environment are collected, passed to the `PackageInstaller`,
and processed using a single build queue.

This introduces a `BuildRequest` class to track install arguments, and it
significantly cleans up the code used to track package ids during installation.
Package ids in the build queue are now just DAG hashes as you would expect,

Other tasks:

- [x] Finish updating the unit tests based on `PackageInstaller`'s use of
      `BuildRequest` and the associated changes
- [x] Change `environment.py`'s `install_all` to use the `PackageInstaller` directly
- [x] Change the `install` command to leverage the new installation process for multiple specs
- [x] Change install output messages for external packages, e.g.:
       `[+] /usr` -> `[+] /usr (external bzip2-1.0.8-<dag-hash>`
- [x] Fix incomplete environment install's view setup/update and not confirming all 
       packages are installed (?)
- [x] Ensure externally installed package dependencies are properly accounted for in 
       remaining build tasks
- [x] Add tests for coverage (if insufficient and can identity the appropriate, uncovered non-comment lines)
- [x] Add documentation
- [x] Resolve multi-compiler environment install issues
- [x] Fix issue with environment installation reporting (restore CDash/JUnit reports)
2020-11-17 02:41:07 -08:00
Todd Gamblin
2893c23e7c
docs: update docs on shell support and using packages (#19486)
Shell integration no longer requires setting `SPACK_ROOT`, so we can
simplify the documentation on it. The docs on shell support and using
packages are getting a bit old, and information on `spack load` (which
seems to be everyone's most common way of using packages) is hard to
find.

This PR simplifies the shell documentation to remove SPACK_ROOT, and also
moves some sections around for clearer organization.

- [x] make docs on sourcing setup scripts clearer and simpler

- [x] introduce `spack load` early in the basic usage guide instead of
      burying it in the module docs

- [x] clean up module docs so that spack module tcl loads comes later

- [x] be clear about the different ways to use packages so that the users
      can find the docs better.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-10-23 22:16:01 -07:00
Massimiliano Culpo
1dba0ce81b Removed references to BlueGene/Q in docs and comments 2020-08-10 17:09:09 -07:00
Massimiliano Culpo
14599f09be
Separate Apple Clang from LLVM Clang (#17110)
* Separate Apple Clang from LLVM Clang

Apple Clang is a compiler of its own. All places
referring to "-apple" suffix have been updated.

* Hack to use a dash in 'apple-clang'

To be able to use autodoc from Sphinx we need
a valid Python name for the module that contains
Apple's Clang code.

* Updated packages to account for the existence of apple-clang

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* Added unit test for XCode related functions

Co-authored-by: Gregory Becker <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-06-25 11:18:48 -05:00
Massimiliano Culpo
a11de1d29d
Package extensions: fixed a link in docs (#16040)
* Moved link to the right place in the docs

* Fixed a few minor issues in extensions docs

Fixed a typo, added a subsubsection for better
navigation, reworded "modules in Python" as
"Python packages"
2020-04-14 12:54:53 -05:00
Greg Sjaardema
c3e8825f8e
Spelling fixes (#15805)
* Spelling fixes

* spelling fixes

* spelling fixes

* spelling fixes

* spelling fix

* spelling fix

* spelling fix

* spelling fixes

* spelling fix
2020-04-01 12:02:26 -05:00
Glenn Johnson
a568db00b8
A few edits for the Basic Usage Doc page (#15215)
This PR corrects a few minor things and adds a note about colorized
output.
2020-02-26 10:03:03 +01:00
Massimiliano Culpo
b9629c36f2 Unified environment modifications in config files (#14372)
* Unified environment modifications in config files

fixes #13357

This commit factors all the code that is involved in
the validation (schema) and parsing of environment modifications
from configuration files in a single place. The factored out
code is then used for module files and compiler configuration.

Attributes were separated by dashes in `compilers.yaml` files and
by underscores in `modules.yaml` files. This PR unifies the syntax
on attributes separated by underscores.

Unit testing of environment modifications in compilers
has been refactored and simplified.
2020-01-27 08:40:47 -08:00
Greg Becker
c9e01ff9d7 shell support: spack load no longer needs modules (#14062)
Previously the `spack load` command was a wrapper around `module load`. This required some bootstrapping of modules to make `spack load` work properly.

With this PR, the `spack` shell function handles the environment modifications necessary to add packages to your user environment. This removes the dependence on environment modules or lmod and removes the requirement to bootstrap spack (beyond using the setup-env scripts).

Included in this PR is support for MacOS when using Apple's System Integrity Protection (SIP), which is enabled by default in modern MacOS versions. SIP clears the `LD_LIBRARY_PATH` and `DYLD_LIBRARY_PATH` variables on process startup for executables that live in `/usr` (but not '/usr/local', `/System`, `/bin`, and `/sbin` among other system locations. Spack cannot know the `LD_LIBRARY_PATH` of the calling process when executed using `/bin/sh` and `/usr/bin/python`. The `spack` shell function now manually forwards these two variables, if they are present, as `SPACK_<VAR>` and recovers those values on startup.

- [x] spack load/unload no longer delegate to modules
- [x] refactor user_environment modification calculations
- [x] update documentation for spack load/unload

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-22 22:36:02 -08:00
Massimiliano Culpo
08d0267c9a Spack can automatically remove unused specs (#13534)
* Spack can uninstall unused specs

fixes #4382

Added an option to spack uninstall that removes all unused specs i.e.
build dependencies or transitive dependencies that are left
in the store after the specs that pulled them in have been removed.

* Moved the functionality to its own command

The command has been named 'spack autoremove' to follow the naming used
for the same functionality by other widely known package managers i.e.
yum and apt.

* Speed-up autoremoving specs by not locking and re-reading the scratch DB

* Make autoremove work directly on Spack's store

* Added unit tests for the new command

* Display a terser output to the user

* Renamed the "autoremove" command "gc"

Following discussion there's more consensus around
the latter name.

* Preserve root specs in env contexts

* Instead of preserving specs, restrict gc to the active environment

* Added docs

* Added a unit test for gc within an environment

* Updated copyright to 2020

* Updated documentation according to review

Rephrased a couple of sentences, added references to
`spack find` and dependency types.

* Updated function naming and docstrings

* Simplified computation of unused specs

Since the new approach uses private attributes of the DB
it has been coded as a method of that class rather than a
freestanding function.
2020-01-07 08:16:54 -08:00
Todd Gamblin
4af6303086
copyright: update copyright dates for 2020 (#14328) 2019-12-30 22:36:56 -08:00
Greg Becker
cd185c3d28 commands: add spack deprecate command (#12933)
`spack deprecate` allows for the removal of insecure packages with minimal impact to their dependents. It allows one package to be symlinked into the prefix of another to provide seamless transition for rpath'd and hard-coded applications using the old version.

Example usage:

    spack deprecate /hash-of-old-openssl /hash-of-new-openssl

The spack deprecate command is designed for use only in extroardinary circumstances.  The spack deprecate command makes no promises about binary compatibility. It is up to the user to ensure the replacement is suitable for the deprecated package.
2019-10-23 13:11:35 -07:00
Greg Becker
94e80933f0 Feature: installed file verification (#12841)
This feature generates a verification manifest for each installed
package and provides a command, "spack verify", which can be used to
compare the current file checksums/permissions with those calculated
at installed time.

Verification includes

* Checksums of files
* File permissions
* Modification time
* File size

Packages installed before this PR will be skipped during verification.
To verify such a package you must reinstall it.

The spack verify command has three modes.

* With the -a,--all option it will check every installed package.
* With the -f,--files option, it will check some specific files,
  determine which package they belong to, and confirm that they have
  not been changed.
* With the -s,--specs option or by default, it will check some
  specific packages that no files havae changed.
2019-10-15 14:24:52 -07:00
Massimiliano Culpo
76b9c56110 Remove support for generating dotkit files (#11986)
Dotkit is being used only at a few sites and has been deprecated on new
machines. This commit removes all the code that provide support for the
generation of dotkit module files.

A new validator named "deprecatedProperties" has been added to the
jsonschema validators. It permits to prompt a warning message or exit
with an error if a property that has been marked as deprecated is
encountered.

* Removed references to dotkit in the docs
* Removed references to dotkit in setup-env-test.sh
* Added a unit test for the 'deprecatedProperties' schema validator
2019-10-02 22:15:01 -07:00
Todd Gamblin
18d63a239f bugfix: use string keys to set preferred targets (#12921)
Preferred targets were failing because we were looking them up by
Microarchitecture object, not by string.

- [x] Add a call to `str()` to fix target lookup.
- [x] Add a test to exercise this part of concretization.
- [x] Add documentation for setting `target` in `packages.yaml`
2019-09-24 10:18:48 +02:00
Massimiliano Culpo
3c4322bf1a targets: Spack targets can now be fine-grained microarchitectures
Spack can now:

- label ppc64, ppc64le, x86_64, etc. builds with specific
  microarchitecture-specific names, like 'haswell', 'skylake' or
  'icelake'.

- detect the host architecture of a machine from /proc/cpuinfo or similar
  tools.

- Understand which microarchitectures are compatible with which (for
  binary reuse)

- Understand which compiler flags are needed (for GCC, so far) to build
  binaries for particular microarchitectures.

All of this is managed through a JSON file (microarchitectures.json) that
contains detailed auto-detection, compiler flag, and compatibility
information for specific microarchitecture targets.  The `llnl.util.cpu`
module implements a library that allows detection and comparison of
microarchitectures based on the data in this file.

The `target` part of Spack specs is now essentially a Microarchitecture
object, and Specs' targets can be compared for compatibility as well.
This allows us to label optimized binary packages at a granularity that
enables them to be reused on compatible machines.  Previously, we only
knew that a package was built for x86_64, NOT which x86_64 machines it
was usable on.

Currently this feature supports Intel, Power, and AMD chips. Support for
ARM is forthcoming.

Specifics:

- Add microarchitectures.json with descriptions of architectures

- Relaxed semantic of compiler's "target" attribute.  Before this change
  the semantic to check if a compiler could be viable for a given target
  was exact match. This made sense as the finest granularity of targets
  was architecture families.  As now we can target micro-architectures,
  this commit changes the semantic by interpreting as the architecture
  family what is stored in the compiler's "target" attribute. A compiler
  is then a viable choice if the target being concretized belongs to the
  same family. Similarly when a new compiler is detected the architecture
  family is stored in the "target" attribute.

- Make Spack's `cc` compiler wrapper inject target-specific flags on the
  command line

- Architecture concretization updated to use the same algorithm as
  compiler concretization

- Micro-architecture features, vendor, generation etc. are included in
  the package hash.  Generic architectures, such as x86_64 or ppc64, are
  still dumped using the name only.

- If the compiler for a target is not supported exit with an intelligible
  error message. If the compiler support is unknown don't try to use
  optimization flags.

- Support and define feature aliases (e.g., sse3 -> ssse3) in
  microarchitectures.json and on Microarchitecture objects. Feature
  aliases are defined in targets.json and map a name (the "alias") to a
  list of rules that must be met for the test to be successful. The rules
  that are available can be extended later using a decorator.

- Implement subset semantics for comparing microarchitectures (treat
  microarchitectures as a partial order, i.e. (a < b), (a == b) and (b <
  a) can all be false.

- Implement logic to automatically demote the default target if the
  compiler being used is too old to optimize for it. Updated docs to make
  this behavior explicit.  This avoids surprising the user if the default
  compiler is older than the host architecture.

This commit adds unit tests to verify the semantics of target ranges and
target lists in constraints. The implementation to allow target ranges
and lists is minimal and doesn't add any new type.  A more careful
refactor that takes into account the type system might be due later.

Co-authored-by: Gregory Becker <becker33.llnl.gov>
2019-09-20 00:51:37 -07:00
Todd Gamblin
987d8cbaaa docs: add docs for spack find --format and spack find --json
- add docs for `spack find --format`
- add docs for `spack find --json`
- update references to Spec.format() docs and add links
2019-09-02 19:24:48 -07:00
Todd Gamblin
ab21b3d194 docs: truncate spack list output in basic usage
`spack list` output is very long and takes up a lot of space in the docs.
Truncate it to just 10 lines and link to the package list page.
2019-05-23 12:40:01 -07:00
Shahzeb Siddiqui
d20b5ce2ec format change and typo in doc (#10848) 2019-03-11 13:19:13 -05:00