Commit Graph

244 Commits

Author SHA1 Message Date
Todd Gamblin
d33973df6c docs: add section on concretizer configuration
* Document `concretizer.yaml`, `--reuse`, and `--fresh`.
2022-02-16 10:17:18 -08:00
Mark W. Krentel
87a3b72ef0
Add 'stable' to the list of infinity version names. (#28772)
* Add 'stable' to the list of infinity version names.
Rename libunwind 1.5-head to 1.5-stable.

* Add stable to the infinite version list in packaging_guide.rst.
2022-02-16 09:08:51 -08:00
Massimiliano Culpo
cd04109e17
Add a "sticky" property to variants (#28630)
* Add sticky variants

* Add unit tests for sticky variants

* Add documentation for sticky variants

* Revert "Revert 19736 because conflicts are avoided by clingo by default (#26721)"

This reverts commit 33ef7d57c1.

* Add stickiness to "allow-unsupported-compiler"
2022-02-02 10:05:24 -08:00
Todd Gamblin
93377942d1 Update copyright year to 2022 2022-01-14 22:50:21 -08:00
Harmen Stoppels
6357de4e61
Fix table formatting (#28037) 2021-12-16 12:13:12 +00:00
Harmen Stoppels
8bb5ed8464
make version docs reflect reality (#27149)
* make version docs reflect reality

* typo and make things

* 2.6 -> 2.7 in example
2021-11-05 09:39:31 +00:00
Greg Becker
67cd92e6a3
Allow conditional variants (#24858)
A common question from users has been how to model variants 
that are new in new versions of a package, or variants that are 
dependent on other variants. Our stock answer so far has been
an unsatisfying combination of "just have it do nothing in the old
version" and "tell Spack it conflicts".

This PR enables conditional variants, on any spec condition. The 
syntax is straightforward, and matches that of previous features.
2021-11-03 08:11:31 +01:00
Massimiliano Culpo
949094544e
Constrain abstract specs rather than concatenating strings in the "when" context manager (#26700)
Using the Spec.constrain method doesn't work since it might
trigger a repository lookup which could break our directives
and triggers a circular import error.

To fix that we introduce a function to merge abstract anonymous
specs, based only on package names, which does not perform any
lookup in the repository.
2021-10-14 12:33:10 +02:00
Harmen Stoppels
b9e72557e8
Remove .99 from version ranges (#26422)
In most cases, .99 can be omitted thanks to #26402 .
2021-10-03 09:09:02 -04:00
Tamara Dahlgren
8664abc178
Remove references to self.install_test_root from packaging guide (#25238) 2021-08-26 19:22:40 +02:00
Adam J. Stewart
1212847eee
Document how to handle changing build systems (#25174) 2021-08-21 11:05:42 -07:00
Adam J. Stewart
b8afc0fd29 API Docs: fix broken reference targets 2021-07-16 08:30:56 -07:00
Massimiliano Culpo
3d11716e54
Add when context manager to group common constraints in packages (#24650)
This PR adds a context manager that permit to group the common part of a `when=` argument and add that to the context:
```python
class Gcc(AutotoolsPackage):
    with when('+nvptx'):
        depends_on('cuda')
        conflicts('@:6', msg='NVPTX only supported in gcc 7 and above')
        conflicts('languages=ada')
        conflicts('languages=brig')
        conflicts('languages=go')
```
The above snippet is equivalent to:
```python
class Gcc(AutotoolsPackage):
    depends_on('cuda', when='+nvptx')
    conflicts('@:6', when='+nvptx', msg='NVPTX only supported in gcc 7 and above')
    conflicts('languages=ada', when='+nvptx')
    conflicts('languages=brig', when='+nvptx')
    conflicts('languages=go', when='+nvptx')
```
which needs a repetition of the `when='+nvptx'` argument. The context manager might help improving readability and permits to group together directives related to the same semantic aspect (e.g. all the directives needed to model the behavior of `gcc` when `+nvptx` is active). 

Modifications:

- [x] Added a `when` context manager to be used with package directives
- [x] Add unit tests and documentation for the new feature
- [x] Modified `cp2k` and `gcc` to show the use of the context manager
2021-07-02 08:43:15 -07:00
Erik Schnetter
e3b220f699
Implement CVS fetcher (#23212)
Spack packages can now fetch versions from CVS repositories. Note
this fetch mechanism is unsafe unless using :extssh:. Most public
CVS repositories use an insecure protocol implemented as part of CVS.
2021-06-22 09:51:31 -07:00
Tamara Dahlgren
929d1de3e5
Stand-alone/Smoke tests: copy cached test sources to test stage (#23713) 2021-05-25 07:24:32 -07:00
Tamara Dahlgren
bed1644d52
Fix packaging guide table's build system links (#23879) 2021-05-25 07:13:00 +02:00
Seth R. Johnson
d8cbd37aaa
Fix makefile filter suggestions (#23856)
Bash has a builtin `fc` that will override the compiler if you use "fc",
so it's better to use the full spack-supplied compiler path.

Additionally, the filter regex in the docs was wrong: it replaced the
entire assignment operation with the RHS.
2021-05-22 18:47:43 +00:00
Tamara Dahlgren
5bd42d1b31
docs/packaging guide: Reference test stage directory (#23707) 2021-05-18 17:56:38 -07:00
Tamara Dahlgren
066d33b4b3
Documentation: Refinement of "Checking an installation" (#22210)
There have been a lot of questions and some confusion recently surrounding Spack installation test capabilities so this PR is intended to clean up and refine the documentation for "Checking an installation".

It aims to better distinguish between checks that are performed during an installation (i.e., build-time tests) and those that can be done days and weeks after the software has been installed (i.e., install (or smoke) tests).
2021-05-11 10:37:48 -07:00
Tamara Dahlgren
851490bd54
Correct the reference to the staged examples files (#21557) 2021-02-13 08:14:38 -08:00
Adam J. Stewart
7ccb9992a6
Procedure to deprecate old versions of software (#20767)
* Procedure to deprecate old versions of software

* Add documentation

* Fix bug in logic

* Update tab completion

* Deprecate legacy packages

* Deprecate old mxnet as well

* More explicit docs
2021-02-09 13:51:18 -05:00
Mark C. Miller
f766585882
Doc default behavior of install tests (#21309)
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2021-01-27 09:08:27 -08:00
Tamara Dahlgren
25bab31bc8
Added ROCmPackage (build system) documentation (#20743) 2021-01-21 10:46:39 -08:00
Todd Gamblin
a8ccb8e116 copyrights: update all files with license headers for 2021
- [x] add `concretize.lp`, `spack.yaml`, etc. to licensed files
- [x] update all licensed files to say 2013-2021 using
      `spack license update-copyright-year`
- [x] appease mypy with some additions to package.py that needed
      for oneapi.py
2021-01-02 12:12:00 -08:00
Tamara Dahlgren
b7d9e269ef
docs: add single node concurrent build example (#20416) 2020-12-17 17:23:55 +01:00
Adam J. Stewart
20752db103
Docs: add more Command Reference links to spack test (#20413) 2020-12-16 12:08:32 -08:00
Tamara Dahlgren
21f30e3074
Bugfix/docs: correct and expand smoke test documentation (#20278) 2020-12-15 08:38:00 -08:00
Adam J. Stewart
3f5f80956e
Command Reference: add link to spack test docs (#20054) 2020-12-08 09:26:03 -08:00
Greg Becker
77b2e578ec
spack test (#15702)
Users can add test() methods to their packages to run smoke tests on
installations with the new `spack test` command (the old `spack test` is
now `spack unit-test`). spack test is environment-aware, so you can
`spack install` an environment and then run `spack test run` to run smoke
tests on all of its packages. Historical test logs can be perused with
`spack test results`. Generic smoke tests for MPI implementations, C,
C++, and Fortran compilers as well as specific smoke tests for 18
packages.

Inside the test method, individual tests can be run separately (and
continue to run best-effort after a test failure) using the `run_test`
method. The `run_test` method encapsulates finding test executables,
running and checking return codes, checking output, and error handling.

This handles the following trickier aspects of testing with direct
support in Spack's package API:

- [x] Caching source or intermediate build files at build time for
      use at test time.
- [x] Test dependencies,
- [x] packages that require a compiler for testing (such as library only
      packages).

See the packaging guide for more details on using Spack testing support.
Included is support for package.py files for virtual packages. This does
not change the Spack interface, but is a major change in internals.

Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
Co-authored-by: wspear <wjspear@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-11-18 02:39:02 -08:00
Tamara Dahlgren
6fa6af1070
Support parallel environment builds (#18131)
As of #13100, Spack installs the dependencies of a _single_ spec in parallel.
Environments, when installed, can only get parallelism from each individual
spec, as they're installed in order.  This PR makes entire environments build
in parallel by extending Spack's package installer to accept multiple root
specs.  The install command and Environment class have been updated to use
the new parallel install method.

The specs and kwargs for each *uninstalled* package (when not force-replacing
installations) of an environment are collected, passed to the `PackageInstaller`,
and processed using a single build queue.

This introduces a `BuildRequest` class to track install arguments, and it
significantly cleans up the code used to track package ids during installation.
Package ids in the build queue are now just DAG hashes as you would expect,

Other tasks:

- [x] Finish updating the unit tests based on `PackageInstaller`'s use of
      `BuildRequest` and the associated changes
- [x] Change `environment.py`'s `install_all` to use the `PackageInstaller` directly
- [x] Change the `install` command to leverage the new installation process for multiple specs
- [x] Change install output messages for external packages, e.g.:
       `[+] /usr` -> `[+] /usr (external bzip2-1.0.8-<dag-hash>`
- [x] Fix incomplete environment install's view setup/update and not confirming all 
       packages are installed (?)
- [x] Ensure externally installed package dependencies are properly accounted for in 
       remaining build tasks
- [x] Add tests for coverage (if insufficient and can identity the appropriate, uncovered non-comment lines)
- [x] Add documentation
- [x] Resolve multi-compiler environment install issues
- [x] Fix issue with environment installation reporting (restore CDash/JUnit reports)
2020-11-17 02:41:07 -08:00
谭九鼎
5e9f4dc982
Use https for links (#19244) 2020-10-09 11:24:09 -05:00
Adam J. Stewart
1992fdf712
Document test dependency type (#18365) 2020-09-02 13:46:52 -07:00
Adam J. Stewart
17f7b23783
Deprecate spack setup (#18240) 2020-09-01 18:07:48 -07:00
Greg Becker
d10dff1b89
docs: add main version to docs for develop-like versions (#18328) 2020-08-27 12:35:11 -07:00
Adam J. Stewart
f0c0cd5c3f
Fix typo in spack external debug msg (#17982) 2020-08-11 09:55:44 +02:00
Massimiliano Culpo
c0d490ffbe Simplify the detection protocol for packages
Packages can implement “detect_version” to support detection
of external instances of a package. This is generally easier
than implementing “determine_spec_details”. The API for
determine_version is similar: for example you can return
“None” to indicate that an executable is not an instance
of a package.

Users may implement a “determine_variants” method for a package.
When doing external detection, executables are grouped by version
and each group results in a single invocation of “determine_variants”
for the associated spec. The method returns a string specifying
the variants for the package. The method may additionally return
a dictionary representing extra attributes for the package.

These will be stored in the spec yaml and can be retrieved
from self.spec.extra_attributes

The Spack GCC package has been updated with an implementation
of “determine_variants” which adds the following extra
attributes to the package: c, cxx, fortran
2020-08-10 11:59:05 -07:00
Massimiliano Culpo
14599f09be
Separate Apple Clang from LLVM Clang (#17110)
* Separate Apple Clang from LLVM Clang

Apple Clang is a compiler of its own. All places
referring to "-apple" suffix have been updated.

* Hack to use a dash in 'apple-clang'

To be able to use autodoc from Sphinx we need
a valid Python name for the module that contains
Apple's Clang code.

* Updated packages to account for the existence of apple-clang

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* Added unit test for XCode related functions

Co-authored-by: Gregory Becker <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-06-25 11:18:48 -05:00
Tamara Dahlgren
48d3e8d350
features: Add install failure tracking removal through spack clean (#15314)
* Add ability to force removal of install failure tracking data through spack clean

* Add clean failures option to packaging guide
2020-06-24 20:28:53 -05:00
Tamara Dahlgren
96932d65a8 Added support for --fail-fast install option to terminate on first failure 2020-06-23 10:22:41 -07:00
Peter Scheibel
701fc1fdb1
Update docs on "spack external find" (#16482)
This improves the documentation for `spack external find` in several ways:

* Provide a code example of implementing `determine_spec_details` for a package
* Explain how to define executables to look for (and also e.g. that they are treated as regular expressions and so can pull in unexpected files).
* Add the "why" for a couple of constraints (i.e. explain that this logic only works for build/run deps because it examines `PATH` for executables)
* Spread the docs between build customization and packaging sections
* Add cross-references
* Add a label so that `spack external find` is linked from the command reference.
2020-05-12 17:08:08 -07:00
Axel Huebl
ca4c0b97ec
Doc: Conflict Msg Pkg Guide (#16201)
Document the nice `msg=` argument in conflicts. Super useful
for less generic conflict guidance.
2020-04-21 16:22:30 -05:00
Massimiliano Culpo
a11de1d29d
Package extensions: fixed a link in docs (#16040)
* Moved link to the right place in the docs

* Fixed a few minor issues in extensions docs

Fixed a typo, added a subsubsection for better
navigation, reworded "modules in Python" as
"Python packages"
2020-04-14 12:54:53 -05:00
Greg Sjaardema
c3e8825f8e
Spelling fixes (#15805)
* Spelling fixes

* spelling fixes

* spelling fixes

* spelling fixes

* spelling fix

* spelling fix

* spelling fix

* spelling fixes

* spelling fix
2020-04-01 12:02:26 -05:00
TZ
d6d095a868
Docs: "spack env" is in the meantime called "spack build-env" (#15233) 2020-02-27 09:13:09 +01:00
Adam J. Stewart
7a60a04402
Docs: configure_args -> cmake_args (#15102) 2020-02-19 22:48:56 -06:00
Andrew W Elble
4accc78409
Git fetching: add option to remove submodules (#14370)
Add an optional 'submodules_delete' field to Git versions in Spack
packages that allows them to remove specific submodules.

For example: the nervanagpu submodule has become unavailable for the
PyTorch project (see issue 19457 at
https://github.com/pytorch/pytorch/issues/). Removing this submodule
allows 0.4.1 to build.
2020-02-03 19:02:45 -08:00
Peter Scheibel
7b2895109c
Document how to add conditional dependencies (#14694)
* add short docs section on conditional dependencies
* add reference to spec syntax
* add note that conditional dependencies can save time
2020-01-30 12:34:54 -08:00
Massimiliano Culpo
08d0267c9a Spack can automatically remove unused specs (#13534)
* Spack can uninstall unused specs

fixes #4382

Added an option to spack uninstall that removes all unused specs i.e.
build dependencies or transitive dependencies that are left
in the store after the specs that pulled them in have been removed.

* Moved the functionality to its own command

The command has been named 'spack autoremove' to follow the naming used
for the same functionality by other widely known package managers i.e.
yum and apt.

* Speed-up autoremoving specs by not locking and re-reading the scratch DB

* Make autoremove work directly on Spack's store

* Added unit tests for the new command

* Display a terser output to the user

* Renamed the "autoremove" command "gc"

Following discussion there's more consensus around
the latter name.

* Preserve root specs in env contexts

* Instead of preserving specs, restrict gc to the active environment

* Added docs

* Added a unit test for gc within an environment

* Updated copyright to 2020

* Updated documentation according to review

Rephrased a couple of sentences, added references to
`spack find` and dependency types.

* Updated function naming and docstrings

* Simplified computation of unused specs

Since the new approach uses private attributes of the DB
it has been coded as a method of that class rather than a
freestanding function.
2020-01-07 08:16:54 -08:00
Todd Gamblin
4af6303086
copyright: update copyright dates for 2020 (#14328) 2019-12-30 22:36:56 -08:00
Massimiliano Culpo
497fddfcb9 Fetching from URLs falls back to mirrors if they exist (#13881)
Users can now list mirrors of the main url in packages.

- [x] Instead of just a single `url` attribute, users can provide a list (`urls`) in the package, and these will be tried by in order by the fetch strategy.

- [x] To handle one of the most common mirror cases, define a `GNUMirrorPackage` mixin to handle all the standard GNU mirrors.  GNU packages can set `gnu_mirror_path` to define the path within a mirror, and the mixin handles setting up all the requisite GNU mirror URLs.

- [x] update all GNU packages in `builtin` to use the `GNUMirrorPackage` mixin.
2019-12-20 14:32:18 -08:00
Adam J. Stewart
5510bba1fd
Document use of the maintainers field (#12270)
* Document use of the maintainers field

* Use fake GitHub usernames

* GitHub action is not automatic yet
2019-11-15 13:58:24 -06:00
William F Godoy
d6274e1a1d Docs update for deprecated spack sha256 (#13701)
* Docs update for deprecated `spack sha256`

* Added macOS shasum

* Update lib/spack/docs/packaging_guide.rst

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
2019-11-13 15:21:27 -06:00
Massimiliano Culpo
9ddc98e46a Separate setting build environment and run environment in packages (#11115)
* Methods setting the environment now do it separately for build and run

Before this commit the `*_environment` methods were setting
modifications to both the build-time and run-time environment
simultaneously. This might cause issues as the two environments
inherently rely on different preconditions:

1. The build-time environment is set before building a package, thus
the package prefix doesn't exist and can't be inspected

2. The run-time environment instead is set assuming the target package
has been already installed

Here we split each of these functions into two: one setting the
build-time environment, one the run-time.

We also adopt a fallback strategy that inspects for old methods and
executes them as before, but prints a deprecation warning to tty. This
permits to port packages to use the new methods in a distributed way,
rather than having to modify all the packages at once.

* Added a test that fails if any package uses the old API

Marked the test xfail for now as we have a lot of packages in that
state.

* Added a test to check that a package modified by a PR is up to date

This test can be used any time we deprecate a method call to ensure
that during the first modification of the package we update also
the deprecated calls.

* Updated documentation
2019-10-17 10:17:21 -07:00
Massimiliano Culpo
7daf860991 targets: adjust packages to use new specific targets semantics
Seamless translation from 'target=<generic>' to either
- target.family == <generic> (in methods)
- 'target=<generic>:' (in directives)

Also updated docs to show ranges in directives.
2019-09-20 00:51:37 -07:00
Massimiliano Culpo
3c4322bf1a targets: Spack targets can now be fine-grained microarchitectures
Spack can now:

- label ppc64, ppc64le, x86_64, etc. builds with specific
  microarchitecture-specific names, like 'haswell', 'skylake' or
  'icelake'.

- detect the host architecture of a machine from /proc/cpuinfo or similar
  tools.

- Understand which microarchitectures are compatible with which (for
  binary reuse)

- Understand which compiler flags are needed (for GCC, so far) to build
  binaries for particular microarchitectures.

All of this is managed through a JSON file (microarchitectures.json) that
contains detailed auto-detection, compiler flag, and compatibility
information for specific microarchitecture targets.  The `llnl.util.cpu`
module implements a library that allows detection and comparison of
microarchitectures based on the data in this file.

The `target` part of Spack specs is now essentially a Microarchitecture
object, and Specs' targets can be compared for compatibility as well.
This allows us to label optimized binary packages at a granularity that
enables them to be reused on compatible machines.  Previously, we only
knew that a package was built for x86_64, NOT which x86_64 machines it
was usable on.

Currently this feature supports Intel, Power, and AMD chips. Support for
ARM is forthcoming.

Specifics:

- Add microarchitectures.json with descriptions of architectures

- Relaxed semantic of compiler's "target" attribute.  Before this change
  the semantic to check if a compiler could be viable for a given target
  was exact match. This made sense as the finest granularity of targets
  was architecture families.  As now we can target micro-architectures,
  this commit changes the semantic by interpreting as the architecture
  family what is stored in the compiler's "target" attribute. A compiler
  is then a viable choice if the target being concretized belongs to the
  same family. Similarly when a new compiler is detected the architecture
  family is stored in the "target" attribute.

- Make Spack's `cc` compiler wrapper inject target-specific flags on the
  command line

- Architecture concretization updated to use the same algorithm as
  compiler concretization

- Micro-architecture features, vendor, generation etc. are included in
  the package hash.  Generic architectures, such as x86_64 or ppc64, are
  still dumped using the name only.

- If the compiler for a target is not supported exit with an intelligible
  error message. If the compiler support is unknown don't try to use
  optimization flags.

- Support and define feature aliases (e.g., sse3 -> ssse3) in
  microarchitectures.json and on Microarchitecture objects. Feature
  aliases are defined in targets.json and map a name (the "alias") to a
  list of rules that must be met for the test to be successful. The rules
  that are available can be extended later using a decorator.

- Implement subset semantics for comparing microarchitectures (treat
  microarchitectures as a partial order, i.e. (a < b), (a == b) and (b <
  a) can all be false.

- Implement logic to automatically demote the default target if the
  compiler being used is too old to optimize for it. Updated docs to make
  this behavior explicit.  This avoids surprising the user if the default
  compiler is older than the host architecture.

This commit adds unit tests to verify the semantics of target ranges and
target lists in constraints. The implementation to allow target ranges
and lists is minimal and doesn't add any new type.  A more careful
refactor that takes into account the type system might be due later.

Co-authored-by: Gregory Becker <becker33.llnl.gov>
2019-09-20 00:51:37 -07:00
Tamara Dahlgren
c141e99e06 Use spack/user-specific stage root by default; stage cleaning (#12516)
* When cleaning the stage root, only remove directories that appear
  to be used for staging Spack packages. Previously Spack was clearing
  all directories in the stage root, which could remove content not
  related to Spack if the user chose a staging root which contains
  files/directories not managed by Spack.
* The documentation is updated with warnings about choosing a stage
  directory that is only managed by Spack (although generally the
  check added in this PR for "spack clean" should avoid removing
  content that was not created by Spack)
* The default stage directory (in config.yaml) is now
  $tempdir/$user/spack-stage and the logic is updated to omit the
  $user portion of this path if $tempdir already contains a $user
  directory.
* When creating stage root assign user read/write permissions to all
  directories in the path under $user. Previously Spack was assigning
  the permissions of the first existing parent directory
2019-09-03 16:31:27 -07:00
Greg Becker
c8b613d261 fix packaging_guide variant table (#12296) 2019-08-29 09:35:48 -06:00
Chris Green
2499c047f0 Emacs directives for fill-column setting. 2019-08-21 18:21:05 -07:00
Chris Green
ad4309b782 Fetch strategies: new global option no_cache, new git option get_full_repo.
* All fetch strategies now accept the Boolean version keyword option `no_cache` in order to allow per-version control of cache-ability.
* New git-specific version keyword option `get_full_repo` (Boolean). When true, disables the default `--depth 1` and `--single-branch` optimizations that are applied if supported by the git version and (in the former case) transport protocol.
* The try / catch blog attempting `--depth 1` and retrying on failure has been removed in favor of more accurately ascertaining when the `--depth` option should work based on git version and protocol choice. Any failure is now treated as a real problem, and the clone is only attempted once.
* Test improvements:
  * `mock_git_repository.checks[type_of_test].args['git']` is now specified as the URL (with leading `file://`) in order to avoid complaints when using `--depth`.
  * New type_of_test `tag-branch`.
  * mock_git_repository now provides `git_exe`.
  * Improved the action of the `git_version` fixture, which was previously hard-wired.
  * New tests of `--single-branch` and `--depth 1` behavior.
* Add documentation of new options to the packaging guide.
2019-08-21 18:21:05 -07:00
Tamara Dahlgren
a551e4329e docs: updates for stage.source_path (#11702) 2019-06-17 11:36:22 +02:00
Massimiliano Culpo
01ece824e1 Cap the maximum number of build jobs (#11373)
* config:build_jobs now controls the number of parallel jobs to spawn during
builds, but cannot ever exceed the number of cores on the machine.

* The default is set to 16 or the number of available cores, whatever
is lowest.

* Updated docs to reflect the changes done to limit parallel builds
2019-05-28 06:42:04 -07:00
Todd Gamblin
f32843528e docs: make docs build relocatable for localized builds
- make all Spack paths relative to a `_spack_root` symlink, so that we
  can easily relocate the docs build *outside* lib/spack/docs

- set some useful defaults for gettext translation variables in conf.py

- update `relativeinclude` and other references to the spack root in the
  RST files to use _spack_root
2019-05-26 18:23:44 -07:00
Todd Gamblin
43aaf8c404 docs: Use HDF5 as the example package for spack graph 2019-05-26 18:23:44 -07:00
Denis Davydov
5b82bf47af extend Version class so that 2.0 > 1.develop > 1.1 and develop > master > head > trunk > 9999 (#1983)
* extend Version class so that 2.0 > 1.develop > 1.1

* add concretization tests, with preferences and preferred version.

* add master, head, trunk as develop-like versions, develop > master > head > trunk

* update documentation on version comparison
2019-05-03 03:32:40 +09:00
Massimiliano Culpo
c4521535e7 Multi-valued variants: better support for combinations (#9481)
This enforces conventions that allow for correct handling of
multi-valued variants where specifying no value is an option,
and adds convenience functionality for specifying multi-valued
variants with conflicting sets of values. This also adds a notion
of "feature values" for variants, which are those that are understood
by the build system (e.g. those that would appear as configure
options). In more detail:

* Add documentation on variants to the packaging guide
* Forbid usage of '' or None as a possible variant value, in
  particular as a default. To indicate choosing no value, the user
  must explicitly define an option like 'none'. Without this,
  multi-valued variants with default set to None were not parsable
  from the command line (Fixes #6314)
* Add "disjoint_sets" function to support the declaration of
  multi-valued variants with conflicting sets of options. For example
  a variant "foo" with possible values "a", "b", and "c" where "c"
  is exclusive of the other values ("foo=a,b" and "foo=c" are
  valid but "foo=a,c" is not).
* Add "any_combination_of" function to support the declaration of
  multi-valued variants where it is valid to choose none of the
  values. This automatically defines "none" as an option (exclusive
  with all other choices); this value does not appear when iterating
  over the variant's values, for example in "with_or_without" (which
  constructs autotools option strings from variant values).
* The "disjoint_sets" and "any_combination_of" methods return an
  object which tracks the possible values. It is also possible to
  indicate that some of these values do not correspond to options
  understood by the package's build system, such that methods like
  "with_or_without" will not define options for those values (this
  occurs automatically for "none")
* Add documentation for usage of new functions for specifying
  multi-valued variants
2019-01-04 19:02:34 -08:00
Todd Gamblin
6f50cd52ed copyright: update license headers for 2013-2019 copyright. 2019-01-01 00:44:28 -08:00
Todd Gamblin
d763e92141 commands: add spack resource command to inspect downloadable files
- currently just looks at patches
  - allows you to find out which package applied a patch to a spec

- intended to work with tarballs and resources in the future.

- add tab completion for `spack resource` and subcommands
2018-12-30 00:19:08 -08:00
Todd Gamblin
eea786f4e8 relicense: replace LGPL headers with Apache-2.0/MIT SPDX headers
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
  - core and most packages are (Apache-2.0 OR MIT)
  - a very small number of remaining packages are LGPL-2.1-only
2018-10-17 14:42:06 -07:00
Greg Becker
ca0d9ae7f0
Make builtin flag handlers available in package scope (#8668)
* Push default flag handlers into module scope

* Preserve backwards compatibility of builtin flag handler names

Ensure Spack continues to work for packages using the `Package.env_flags` idiom and equivalent.

* update docs and tests to match

* Update packages to match new syntax
2018-10-09 15:18:31 -07:00
Michael Sternberg
a86f22d755 Intel prefixes (#7469)
Consolidate prefix calculation logic for intel packages into the
IntelPackage class.

Add documentation on installing Intel packages with Spack an
(alternatively) adding them as external packages in Spack.
2018-08-29 21:09:34 -07:00
Adam J. Stewart
7902f3eb49 Add docs on new top-level VCS attributes 2018-07-25 23:10:10 -07:00
scheibelp
3560f6dbe9 views: packages can customize how they're added to views (#7152)
Functional updates:

- `python` now creates a copy of the `python` binaries when it is added
  to a view

- Python extensions (packages which subclass `PythonPackage`) rewrite
  their shebang lines to refer to python in the view

- Python packages in the same namespace will not generate conflicts if
  both have `...lib/site-packages/namespace-example/__init__.py`

  - These `__init__` files will also remain when removing any package in
    the namespace until the last package in the namespace is removed


Generally (Updated 2/16):

- Any package can define `add_files_to_view` to customize how it is added
  to a view (and at the moment custom definitions are included for
  `python` and `PythonPackage`)

  - Likewise any package can define `remove_files_from_view` to customize
    which files are removed (e.g. you don't always want to remove the
    namespace `__init__`)

- Any package can define `view_file_conflicts` to customize what it
  considers a merge conflict

- Global activations are handled like views (where the view root is the
  spec prefix of the extendee)

  - Benefit: filesystem-management aspects of activating extensions are
    now placed in views (e.g. now one can hardlink a global activation)

  - Benefit: overriding `Package.activate` is more straightforward (see
    `Python.activate`)

  - Complication: extension packages which have special-purpose logic
    *only* when activated outside of the extendee prefix must check for
    this in their `add_files_to_view` method (see `PythonPackage`)

- `LinkTree` is refactored to have separate methods for copying a
  directory structure and for copying files (since it was found that
  generally packages may want to alter how files are copied but still
  wanted to copy directories in the same way)


TODOs (updated 2/20):

- [x] additional testing (there is some unit testing added at this point
  but more would be useful)

- [x] refactor or reorganize `LinkTree` methods: currently there is a
  separate set of methods for replicating just the directory structure
  without the files, and a set for replicating everything

- [x] Right now external views (i.e. those not used for global
  activations) call `view.add_extension`, but global activations do not
  to avoid some extra work that goes into maintaining external views. I'm
  not sure if addressing that needs to be done here but I'd like to
  clarify it in the comments (UPDATE: for now I have added a TODO and in
  my opinion this can be merged now and the refactor handled later)

- [x] Several method descriptions (e.g. for `Package.activate`) are out
  of date and reference a distinction between global activations and
  views, they need to be updated

- [x] Update aspell package activations
2018-06-26 16:14:05 -07:00
Chris Green
15c98fa57c compiler flags: add cxx98 standard support (#7601)
The following improvements are made to cxx standard support
(e.g. compiler.cxxNN_flag functions) in compilers:

* Add cxx98_flag property
* Add support for throwing an exception when a flag is not supported (previously
  if a flag was not supported the application was terminated with tty.die)
* The name of the flag associated with e.g. c++14 standard support changes for
  different compiler versions (e.g. c++1y vs c++14). This makes a few corrections
  on what flag to return for which version.
* Added tests to confirm that versions report expected flags for various c++
  standards (or raise an exception for versions that don't provide a given cxx
  standard)

Note that if a given cxx standard is the default, the associated flag property will
return ""; cxx98 is assumed to be the default standard so this is the behavior for
the associated property in the base compiler class.

Package changes:

* Improvements to the boost spec to take advantage of the improved standard
  flag facility.
* Update the clingo spec to catch the new exception rather than look for an
  empty flag to indicate non-support (which is not part of the compiler flag API)
2018-06-08 13:49:31 -07:00
Denis Davydov
8285a1778f extend Prefix class with join() member to support dynamic directories (#8329)
* extend Prefix class with join() member to support dynamic directories

* add more tests for Prefix.join()

* more tests for Prefix.join()

* add docstring

* add example to docstring of Prefix class

* cleanup Prefix.join() tests

* use Prefix.join() in Packaging Guide
2018-06-01 07:16:09 -05:00
Adam J. Stewart
87223a531b
Document specs with multiple version ranges (#8284) 2018-05-30 10:36:56 -05:00
Denis Davydov
4b5fe75bc3 add OctavePackage (#6746)
* add OctavePackage

1. remove import CudaPackage which is not needed anymore
2. mention CudaPackage and OctavePackage in packaging guide
3. adjust OctavePackageTemplate
4. add clue file for Octave build
5. sanity check on self.prefix

* use setup_environment
2018-01-02 20:03:27 -06:00
becker33
28d8784ab9 Compiler flag handlers (#6415)
This adds the ability for packages to apply compiler flags in one of
three ways: by injecting them into the compiler wrapper calls (the
default in this PR and previously the only automated choice);
exporting environment variable definitions for variables with
corresponding names (e.g. CPPFLAGS=...); providing them as arguments
to the build system (e.g. configure).

When applying compiler flags using build system arguments, a package
must implement the 'flags_to_build_system_args" function. This is
provided for CMake and autotools packages, so for packages which
subclass those build systems, they need only update their flag
handler method specify which compiler flags should be specified as
arguments to the build system.

Convenience methods are provided to specify that all flags be applied
in one of the 3 available ways, so a custom implementation is only
required if more than one method of applying compiler flags is
needed.

This also removes redundant build system definitions from tutorial
examples
2017-12-20 15:40:38 -08:00
Adam J. Stewart
e01a16b992 Remove spack md5 and sha256 commands (#6428) 2017-11-28 13:41:46 -08:00
becker33
04a4ac68d5 Identify the flag handlers feature as beta (#5609)
The flag-handling logic added in #4421 may change semantics, so this
commit adds a warning to the documentation for this feature.
2017-10-04 18:32:13 -07:00
Christoph Junghans
d22ee8f993 patch: add workdir option (#5501)
* patch: add working_dir option
* added documentation
2017-10-02 22:56:43 -07:00
Todd Gamblin
96d2488e0c Documentation for dependency patching. 2017-09-30 02:06:59 -07:00
Denis Davydov
234e00e84c update Blas/Lapack section of packaging guide (#5383) 2017-09-20 15:40:20 -07:00
Adam J. Stewart
db657d938d Refactor IntelInstaller into IntelPackage base class (#4300)
* Refactor IntelInstaller into IntelPackage base class

* Move license attributes from __init__ to class-level

* Flake8 fixes: remove unused imports

* Fix logic that writes the silent.cfg file

* More specific version numbers for Intel MPI

* Rework logic that selects components to install

* Final changes necessary to get intel package working

* Various updates to intel-parallel-studio

* Add latest version of every Intel package

* Add environment variables for Intel packages

* Update env vars for intel package

* Finalize components for intel-parallel-studio package

Adds a +tbb variant to intel-parallel-studio.
The tbb package was renamed to intel-tbb.
Now both intel-tbb and intel-parallel-studio+tbb
provide tbb.

* Overhaul environment variables set by intel-parallel-studio

* Point dependent packages to the correct MPI wrappers

* Never default to intel-parallel-studio

* Gather env vars by sourcing setup scripts

* Use mpiicc instead of mpicc when using Intel compiler

* Undo change to ARCH

* Add changes from intel-mpi to intel-parallel-studio

* Add comment explaining mpicc vs mpiicc

* Prepend env vars containing 'PATH' or separators

* Flake8 fix

* Fix bugs in from_sourcing_file

* Indentation fix

* Prepend, not set if contains separator

* Fix license symlinking broken by changes to intel-parallel-studio

* Use comments instead of docstrings to document attributes

* Flake8 fixes

* Use a set instead of a list to prevent duplicate components

* Fix MKL and MPI library linking directories

* Remove +all variant from intel-parallel-studio

* It is not possible to build with MKL, GCC, and OpenMP at this time

* Found a workaround for locating GCC libraries

* Typos and variable names

* Fix initialization of empty LibraryList
2017-08-16 12:21:07 -05:00
Massimiliano Culpo
faeb1b77b2 Merged 'purge' command with 'clean' and deleted 'purge' (#4970)
* Merged 'purge' command with 'clean'. Deleted 'purge'. fixes #2942

'spack purge' has been merged with 'spack clean'. Documentation has been
updated accordingly. The 'clean' and 'purge' behavior are not mutually
exclusive, and they log brief information to tty while they go.

* Fixed a wrong reference to spack clean in the docs

* Added tests for 'spack clean'. Updated bash completion.
2017-08-09 10:02:38 -07:00
Adam J. Stewart
7eb263effe Add a SConsPackage base class (#4936)
* Add a SConsPackage base class

* Make Matlab extendable

* Most dependencies are actually required

* Cantera requires older version of fmt
2017-08-04 16:52:10 -05:00
Adam J. Stewart
452f382293 Add a QMakePackage base class (#4925)
* Add a QMakePackage base class

* Fix sqlite linking bug in qt-creator

* Add latest version of qt-creator

* Add latest version of qwt

* Use raw strings for regular expressions

* Increase minimum required version of qt

* Add comment about specific version of sqlite required

* Fixes for latest version of qwt and qt-creator

* Older versions of Qwt only work with older versions of Qt
2017-08-04 08:46:07 -07:00
Adam J. Stewart
250ee413e9 Change Version formatting properties and functions to return Version objects (#4834)
* Change version.up_to() to return Version() object
* Add unit tests for Version.up_to()
* Fix packages that expected up_to() to return a string
* Ensure that up_to() preserves separator characters
* Use version indexing instead of up_to
* Make all Version formatting properties return Version objects
* Update docs
* Tests need to test string representation
2017-07-24 13:02:13 -07:00
becker33
f962aba6ce Allow packages to control handling of compiler flags (#4421)
* Initial work on flag trapping using functions called <flag>_handler and default_flag_handler

* Update packages so they do not obliterate flags

* Added append to EnvironmentModifications class

* changed EnvironmentModifications to have append_flags method

* changed flag_val to be a tuple

* Increased test coverage

* added documentation of flag handling
2017-07-19 20:12:00 -07:00
George Hartzell
ff906faf9a Typo: submdoules -> submodules (#4716) 2017-07-12 07:48:31 +02:00
Adam J. Stewart
9933d759ac Update hdfgroup packages to new URL structure (#4643)
* Update hdfgroup packages to new URL structure

* Update docs now that HDF5 URL isn't that complicated
2017-06-29 21:38:47 -05:00
Adam J. Stewart
e5ce7b1639 Allow arbitrary Prefix attributes (#4591)
* Allow arbitrary Prefix attributes
* Test attribute type as well

* Flake8 fixes

* Remove __new__ method

* Fewer uses of join_path in the docs
2017-06-24 22:39:31 -07:00
Adam J. Stewart
2511520b32 Add a WafPackage base class (#3975)
* Add a WafPackage base class

* Correct comment in docstring

* Be more specific about the Python versions supported
2017-05-01 08:00:09 -07:00
Christoph Junghans
0b002c2911 fetch git submodules recursively (#3779)
* fetch git submodules recursively

This is useful if the submodules have submodules themselves. On
the other hand doing a recursive update doesn't hurt if there
is only one level.

* fetch submodules with depth=1 as well (fix #2190)

* use git submodule with depth only for git>=1.8.4
2017-04-21 21:59:30 -07:00
Adam J. Stewart
8e4ada5e99 Make it more clear that version ranges are inclusive (#3759) 2017-04-10 07:25:48 -05:00
Massimiliano Culpo
ffef681377 new directive: conflicts() (#3125)
* Add conflicts(<spec>) directive
* openblas: added conflicts for intel@16 refs #3119
* added brief docs and unit tests
2017-04-02 11:40:09 -07:00
Milton Woods
9e43ff821c Extendable Perl (#3614)
* perl: make extendable and add Module::Build package
* perl: allow 'spack create' to identify perl packages from their contents
* perl-module-build: fix indenting of package docstring
* perl: split install() method for extensions into phases
* perl: auto-detect build method (Makefile.PL vs Build.PL) and define a 'check' method
* PerlPackage: use import statements similar to those in AutotoolsPackage
* PerlModule: fix detection of Build.PL
* PerlPackageTemplate: remove extraneous lines to avoid flake8 warnings
* PerlPackageTemplate: split into separate templates for Makefile.PL and Build.PL
* PerlPackage: add cross-references to docstrings
* AutotoolsPackage: fix ambiguous cross-references to avoid errors in doc tests
* PerlbuildPackageTemplate: depend on perl-module-build if Build.PL exists
2017-03-30 16:38:58 -07:00
Adam J. Stewart
10b767b93f Don't use @system in packages.yaml (#3472) 2017-03-17 07:55:55 -07:00
Adam J. Stewart
dca4d2b15e Consistent docs and usage of env mod methods (#3351) 2017-03-14 22:26:44 -07:00
Henrik Bengtsson
6e729bc333 DOCS: Examplify what file types are automatically extracted (#2955) 2017-01-31 08:26:19 -08:00
Adam J. Stewart
123f057089 Refactor Spack's URL parsing commands (#2938)
* Replace `spack urls` and `spack url-parse` with `spack url`
* Allow spack url list to only list incorrect parsings
* Add spack url test reporting
* Add unit tests for new URL commands
2017-01-31 11:14:52 -05:00