Commit Graph

256 Commits

Author SHA1 Message Date
Adam J. Stewart
7ccb9992a6
Procedure to deprecate old versions of software (#20767)
* Procedure to deprecate old versions of software

* Add documentation

* Fix bug in logic

* Update tab completion

* Deprecate legacy packages

* Deprecate old mxnet as well

* More explicit docs
2021-02-09 13:51:18 -05:00
Massimiliano Culpo
694d633a2c
spack external find: allow to search by tags (#21407)
This commit adds an option to the `external find`
command that allows it to search by tags. In this
way group of executables with common purposes can
be grouped under a single name and a simple command
can be used to detect all of them.

As an example introduce the 'build-tools' tag to
search for common development tools on a system
2021-02-04 13:17:32 +01:00
Adam J. Stewart
aa8e026242
spack setup: remove the command for v0.17.0 (#20277)
spack setup was deprecated in 0.16 and will be removed in 0.17

Follow-up to #18240
2021-01-27 09:24:09 +01:00
Vanessasaurus
67ce1939a3
spack python: allow use of IPython (#20329)
This adds a -i option to "spack python" which allows use of the
IPython interpreter; it can be used with "spack python -i ipython".
This assumes it is available in the Python instance used to run
Spack (i.e. that you can "import IPython").
2021-01-05 16:54:47 -08:00
Todd Gamblin
a8ccb8e116 copyrights: update all files with license headers for 2021
- [x] add `concretize.lp`, `spack.yaml`, etc. to licensed files
- [x] update all licensed files to say 2013-2021 using
      `spack license update-copyright-year`
- [x] appease mypy with some additions to package.py that needed
      for oneapi.py
2021-01-02 12:12:00 -08:00
Todd Gamblin
78f39bdfee commands: add spack license update-copyright-year
This adds a new subcommand to `spack license` that automatically updates
the copyright year in files that should have a license header.

- [x] add `spack license update-copyright-year` command
- [x] add test
2021-01-02 12:12:00 -08:00
Tom Scogland
857749a9ba
add mypy to style checks; rename spack flake8 to spack style (#20384)
I lost my mind a bit after getting the completion stuff working and
decided to get Mypy working for spack as well. This adds a 
`.mypy.ini` that checks all of the spack and llnl modules, though
not yet packages, and fixes all of the identified missing types and
type issues for the spack library.

In addition to these changes, this includes:

* rename `spack flake8` to `spack style`

Aliases flake8 to style, and just runs flake8 as before, but with
a warning.  The style command runs both `flake8` and `mypy`,
in sequence. Added --no-<tool> options to turn off one or the
other, they are on by default.  Fixed two issues caught by the tools.

* stub typing module for python2.x

We don't support typing in Spack for python 2.x. To allow 2.x to
support `import typing` and `from typing import ...` without a
try/except dance to support old versions, this adds a stub module
*just* for python 2.x.  Doing it this way means we can only reliably
use all type hints in python3.7+, and mypi.ini has been updated to
reflect that.

* add non-default black check to spack style

This is a first step to requiring black.  It doesn't enforce it by
default, but it will check it if requested.  Currently enforcing the
line length of 79 since that's what flake8 requires, but it's a bit odd
for a black formatted project to be quite that narrow.  All settings are
in the style command since spack has no pyproject.toml and I don't
want to add one until more discussion happens. Also re-format
`style.py` since it no longer passed the black style check
with the new length.

* use style check in github action

Update the style and docs action to use `spack style`, adding in mypy
and black to the action even if it isn't running black right now.
2020-12-22 21:39:10 -08:00
Tom Scogland
c1e4f3e131
Refactor flake8 handling and tool compatibility (#20376)
This PR does three related things to try to improve developer tooling quality of life:

1. Adds new options to `.flake8` so it applies the rules of both `.flake8` and `.flake_package` based on paths in the repository.
2. Adds a re-factoring of the `spack flake8` logic into a flake8 plugin so using flake8 directly, or through editor or language server integration, only reports errors that `spack flake8` would.
3. Allows star import of `spack.pkgkit` in packages, since this is now the thing that needs to be imported for completion to work correctly in package files, it's nice to be able to do that.

I'm sorely tempted to sed over the whole repository and put `from spack.pkgkit import *` in every package, but at least being allowed to do it on a per-package basis helps.

As an example of what the result of this is:

```
~/Workspace/Projects/spack/spack develop* ⇣
❯ flake8 --format=pylint ./var/spack/repos/builtin/packages/kripke/package.py
./var/spack/repos/builtin/packages/kripke/package.py:6: [F403] 'from spack.pkgkit import *' used; unable to detect undefined names
./var/spack/repos/builtin/packages/kripke/package.py:25: [E501] line too long (88 > 79 characters)

~/Workspace/Projects/spack/spack refactor-flake8*
1 ❯ flake8 --format=spack ./var/spack/repos/builtin/packages/kripke/package.py

~/Workspace/Projects/spack/spack refactor-flake8*
❯ flake8 ./var/spack/repos/builtin/packages/kripke/package.py
```

* qa/flake8: update .flake8, spack formatter plugin

Adds:
* Modern flake8 settings for per-path/glob error ignores, allows
  packages to use the same `.flake8` as the rest of spack
* A spack formatter plugin to flake8 that implements the behavior of
  `spack flake8` for direct invocations.  Makes integration with
  developer tooling nicer, linting with flake8 reports only errors that
  `spack flake8` would report.  Using pyls and pyls-flake8, or any other
  non-format-dependent flake8 integration, now works with spack's rules.

* qa/flake8: allow star import of spack.pkgkit

To get working completion of directives and spack components it's
necessary to import the contents of spack.pkgkit.  At the moment doing
this makes flake8 displeased.  For now, allow spack.pkgkit and spack
both, next step is to ban spack * and require spack.pkgkit *.

* first cut at refactoring spack flake8

This version still copies all of the files to be checked as befire, and
some other things that probably aren't necessary, but it relies on the
spack formatter plugin to implement the ignore logic.

* keep flake8 from rejecting itself

* remove separate packages flake8 config

* fix failures from too many files

I ran into this in the PR converting pkgkit to std.  The solution in
that branch does not work in all cases as it turns out, and all the
workarounds I tried to use generated configs to get a single invocation
of flake8 with a filename optoion to work failed.  It's an astonishingly
frustrating config option.

Regardless, this removes all temporary file creation from the command
and relies on the plugin instead.  To work around the huge number of
files in spack and still allow the command to control what gets checked,
it scans files in batches of 100.  This is a completely arbitrary number
but was chosen to be safely under common line-length limits.  One
side-effect of this is that every 100 files the command will produce
output, rather than only at the end, which doesn't seem like a terrible
thing.
2020-12-22 09:28:46 -08:00
Tom Scogland
71c77fa8fa
minimal zsh completion (#20253)
Since zsh can load bash completion files natively, seems reasonable to just turn this on.
The only changes are to switch from `type -t` which zsh doesn't support to using `type`
with a regex and adding a new arm to the sourcing of the completions to allow it to work
for zsh as well as bash.

Could use more bash/dash/etc testing probably, but everything I've thought to try has
worked so far.

Notes:
* unit-test zsh support, fix issues
Specifically fixed word splitting in completion-test, use a different
method to apply sh emulation to zsh loaded bash completion, and fixed
an incompatibility in regex operator quoting requirements.

* compinit now ignores insecure directories
Completion isn't meant to be enabled in non-interactive environments, so
by default compinit will ask the user if they want to ignore insecure
directories or load them anyway.  To pass the spack unit tests in GH
actions, this prompt must be disabled, so ignore explicitly until a
better solution can be found.

* debug functions test also requires bash emulation
COMP_WORDS is a bash-ism that zsh doesn't natively support, turn on
emulation for just that section of tests to allow the comparison to
work.  Does not change the behavior of the functions themselves since
they are already pinned to sh emulation elsewhere.

* propagate change to .in file

* fix comment and update script based on .in
2020-12-18 17:26:15 -08:00
vvolkl
ed258ca9e9
Add "spack versions --new" flag to only show new versions (#20030)
* [cmd versions] add spack versions --new flag to only fetch new versions

format

[cmd versions] rename --latest to --newest and add --remote-only

[cmd versions] add tests for --remote-only and --new

format

[cmd versions] update shell tab completion

[cmd versions] remove test for --remote-only --new which gives empty output

[cmd versions] final rename

format

* add brillig mock package

* add test for spack versions --new

* [brillig] format

* [versions] increase test coverage

* Update lib/spack/spack/cmd/versions.py

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>

* Update lib/spack/spack/cmd/versions.py

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-12-07 09:29:10 -06:00
eugeneswalker
badf3368ad
allow install of build-deps from cache via --include-build-deps switch (#19955)
* allow install of build-deps from cache via --include-build-deps switch

* make clear that --include-build-deps is useful for CI pipeline troubleshooting
2020-12-03 15:27:01 -08:00
Michael Kuhn
20367e472d
cmd: add spack mark command (#16662)
This adds a new `mark` command that can be used to mark packages as either
explicitly or implicitly installed. Apart from fixing the package
database after installing a dependency manually, it can be used to
implement upgrade workflows as outlined in #13385.

The following commands demonstrate how the `mark` and `gc` commands can be
used to only keep the current version of a package installed:
```console
$ spack install pkgA
$ spack install pkgB
$ git pull # Imagine new versions for pkgA and/or pkgB are introduced
$ spack mark -i -a
$ spack install pkgA
$ spack install pkgB
$ spack gc
```

If there is no new version for a package, `install` will simply mark it as
explicitly installed and `gc` will not remove it.

Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-11-18 03:20:56 -08:00
Greg Becker
77b2e578ec
spack test (#15702)
Users can add test() methods to their packages to run smoke tests on
installations with the new `spack test` command (the old `spack test` is
now `spack unit-test`). spack test is environment-aware, so you can
`spack install` an environment and then run `spack test run` to run smoke
tests on all of its packages. Historical test logs can be perused with
`spack test results`. Generic smoke tests for MPI implementations, C,
C++, and Fortran compilers as well as specific smoke tests for 18
packages.

Inside the test method, individual tests can be run separately (and
continue to run best-effort after a test failure) using the `run_test`
method. The `run_test` method encapsulates finding test executables,
running and checking return codes, checking output, and error handling.

This handles the following trickier aspects of testing with direct
support in Spack's package API:

- [x] Caching source or intermediate build files at build time for
      use at test time.
- [x] Test dependencies,
- [x] packages that require a compiler for testing (such as library only
      packages).

See the packaging guide for more details on using Spack testing support.
Included is support for package.py files for virtual packages. This does
not change the Spack interface, but is a major change in internals.

Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
Co-authored-by: wspear <wjspear@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-11-18 02:39:02 -08:00
Todd Gamblin
0ed019d4ef concretizer: first working version with pyclingo interface
- [x] Solver now uses the Python interface to clingo
- [x] can extract unsatisfiable cores from problems when things go wrong
- [x] use Python callbacks for versions instead of choice rules (this may
      ultimately hurt performance)
2020-11-17 10:04:13 -08:00
Todd Gamblin
4092c90b57
commands: add spack tutorial command (#19808)
Added a command to set up Spack for our tutorial at
https://spack-tutorial.readthedocs.io.

The command does some common operations we need first-time users to do.
Specifically:

- checks out a particular branch of Spack
- deletes spurious configuration in `~/.spack` that might be
  left over from prior parts of the tutorial
- adds a mirror and trusts its public key
2020-11-09 12:47:08 +01:00
Scott Wittenburg
31f57e56bb
Binary caching: use full hashes (#19209)
* "spack install" now has a "--require-full-hash-match" option, which
  forces Spack to skip an available binary package when the full hash
  doesn't match. Normally only a DAG-hash match is required, which
  ensures equivalent Specs, but does not account for changing logic
  inside the associated package.
* Add a local binary cache index which tracks specs that have a binary
  install available in a remote binary cache. It is updated with
  "spack buildcache list" or for a given spec when a binary package
  is retrieved for that Spec.
2020-10-30 12:53:33 -07:00
elsagermann
4750d479a0
Add testing option to dev-build command (#17293)
* ADD: testing to dev-build command

* RM: mutally exclusive group for testing in parser

* FIX: test option to subparser and not testing

* ADD: spack-completion.bash

* RM: local devbuildcosmo cmd

* FIX: bad merge --drop-in -b --before options forgotten

* FIX: --test place in spack-completion.bash

* FIX: typo

* FIX: blank line removing

* FIX: trailing white space

Co-authored-by: Elsa Germann <egermann@tsa-ln002.cm.cluster>
2020-10-18 23:17:07 -05:00
Greg Becker
7a6268593c
Environments: specify packages for developer builds (#15256)
* allow environments to specify dev-build packages

* spack develop and spack undevelop commands

* never pull dev-build packges from bincache

* reinstall dev_specs when code has changed; reinstall dependents too

* preserve dev info paths and versions in concretization as special variant

* move install overwrite transaction into installer

* move dev-build argument handling to package.do_install

now that specs are dev-aware, package.do_install can add
necessary args (keep_stage=True, use_cache=False) to dev
builds. This simplifies driving logic in cmd and env._install

* allow 'any' as wildcard for variants

* spec: allow anonymous dependencies

raise an error when constraining by or normalizing an anonymous dep
refactor concretize_develop to remove dev_build variant
refactor tests to check for ^dev_path=any instead of +dev_build

* fix variant class hierarchy
2020-10-15 17:23:16 -07:00
Scott Wittenburg
438f80d19e
Revert binary distribution cache manager (#19158)
This reverts #18359 and follow-on PRs intended to address issues with
#18359 because that PR changes the hash of all specs. A future PR will
reintroduce the changes.

* Revert "Fix location in spec.yaml where we look for full_hash (#19132)"
* Revert "Fix fetch of spec.yaml files from buildcache (#19101)"
* Revert "Merge pull request #18359 from scottwittenburg/add-binary-distribution-cache-manager"
2020-10-05 16:02:37 -07:00
Scott Wittenburg
a44135dccf
Update buildcache key index when we update the package index (#19117)
This changes makes sure that when we run the pipeline job that updates
the buildcache package index on the remote mirror, we also update the
key index.  The public keys corresponding to the signing keys used to
sign the package was pushed to the mirror as a part of creating the
buildcache index, so this is just ensuring those keys are reflected
in the key index.

Also, this change makes sure the "spack buildcache update-index"
job runs even when there may have been pipeline failures, since we
would like the index always to reflect the true state of the mirror.
2020-10-02 11:00:42 -06:00
Scott Wittenburg
075c3e0d92
Merge pull request #18359 from scottwittenburg/add-binary-distribution-cache-manager
Add binary distribution cache manager
2020-09-30 16:37:35 -06:00
Omar Padron
2d93154119
Streamline key management for build caches (#17792)
* Rework spack.util.web.list_url()

list_url() now accepts an optional recursive argument (default: False)
for controlling whether to only return files within the prefix url or to
return all files whose path starts with the prefix url.  Allows for the
most effecient implementation for the given prefix url scheme.  For
example, only recursive queries are supported for S3 prefixes, so the
returned list is trimmed down if recursive == False, but the native
search is returned as-is when recursive == True.  Suitable
implementations for each case are also used for file system URLs.

* Switch to using an explicit index for public keys

Switches to maintaining a build cache's keys under build_cache/_pgp.
Within this directory is an index.json file listing all the available
keys and a <fingerprint>.pub file for each such key.

 - Adds spack.binary_distribution.generate_key_index()
   - (re)generates a build cache's key index

 - Modifies spack.binary_distribution.build_tarball()
   - if tarball is signed, automatically pushes the key used for signing
     along with the tarball
   - if regenerate_index == True, automatically (re)generates the build
     cache's key index along with the build cache's package index; as in
     spack.binary_distribution.generate_key_index()

 - Modifies spack.binary_distribution.get_keys()
   - a build cache's key index is now used instead of programmatic
     listing

 - Adds spack.binary_distribution.push_keys()
   - publishes keys from Spack's keyring to a given list of mirrors

 - Adds new spack subcommand: spack gpg publish
   - publishes keys from Spack's keyring to a given list of mirrors

 - Modifies spack.util.gpg.Gpg.signing_keys()
   - Accepts optional positional arguments for filtering the set of keys
     returned

 - Adds spack.util.gpg.Gpg.public_keys()
   - As spack.util.gpg.Gpg.signing_keys(), except public keys are
     returned

 - Modifies spack.util.gpg.Gpg.export_keys()
   - Fixes an issue where GnuPG would prompt for user input if trying to
     overwrite an existing file

 - Modifies spack.util.gpg.Gpg.untrust()
   - Fixes an issue where GnuPG would fail for input that were not key
     fingerprints

 - Modifies spack.util.web.url_exists()
   - Fixes an issue where url_exists() would throw instead of returning
     False

* rework gpg module/fix error with very long GNUPGHOME dir

* add a shim for functools.cached_property

* handle permission denied error in gpg util

* fix tests/make gpgconf optional if no socket dir is available
2020-09-25 12:54:24 -04:00
Scott Wittenburg
ace52bd476 Provide your own script, before_script, and after_script 2020-09-14 10:37:42 -06:00
Richarda Butler
d721bd8070
commands: update help for spack install --yes-to-all (#18367)
`spack install --yes-to-all` doesn't actually make the build non-interactive,
but that is why people typically use it. This documents that you must also
specify `--no-checksum` for a fully non-interactive build.
2020-09-08 13:18:25 -07:00
Robert Blake
ea57171712
Make spack environment configurations writable from spack external and spack compiler find (#18165)
* spack config: default modification scope can be an environment

The previous model was that environments are the highest priority config
scope for config reading operations, but were not considered for config
writing operations. Now, the active environment is the highest priority
config scope for both reading and writing operations.

Now spack config add, spack external find and spack compiler set environment 
configuration in the environment by default if an environment is active. This is a
change in default behavior for these routines, but better matches the mental
model for an environment taking precedence over the user's default config file.

* add scope argument to 'spack external find' to choose non-default scope

* Increase testing for config modifications on environments

Co-authored-by: Gregory Becker <becker33@llnl.gov>
2020-09-05 01:12:26 -07:00
Massimiliano Culpo
c0d490ffbe Simplify the detection protocol for packages
Packages can implement “detect_version” to support detection
of external instances of a package. This is generally easier
than implementing “determine_spec_details”. The API for
determine_version is similar: for example you can return
“None” to indicate that an executable is not an instance
of a package.

Users may implement a “determine_variants” method for a package.
When doing external detection, executables are grouped by version
and each group results in a single invocation of “determine_variants”
for the associated spec. The method returns a string specifying
the variants for the package. The method may additionally return
a dictionary representing extra attributes for the package.

These will be stored in the spec yaml and can be retrieved
from self.spec.extra_attributes

The Spack GCC package has been updated with an implementation
of “determine_variants” which adds the following extra
attributes to the package: c, cxx, fortran
2020-08-10 11:59:05 -07:00
Massimiliano Culpo
193e8333fa Update packages.yaml format and support configuration updates
The YAML config for paths and modules of external packages has
changed: the new format allows a single spec to load multiple
modules. Spack will automatically convert from the old format
when reading the configs (the updates do not add new essential
properties, so this change in Spack is backwards-compatible).

With this update, Spack cannot modify existing configs/environments
without updating them (e.g. “spack config add” will fail if the
configuration is in a format that predates this PR). The user is
prompted to do this explicitly and commands are provided. All
config scopes can be updated at once. Each environment must be
updated one at a time.
2020-08-10 11:59:05 -07:00
Adam J. Stewart
207e496162
spack create: ask how many to download (#17373) 2020-07-08 09:38:42 +02:00
Johannes Blaschke
1d55adfd2b
Add fish shell support (#9279)
* share/spack/setup-env.fish file to setup environment in fish shell

* setup-env.fish testing script

* Update share/spack/setup-env.fish

Co-Authored-By: Elsa Gonsiorowski, PhD <gonsie@me.com>

* Update share/spack/qa/setup-env-test.fish

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>

* updates completions using `spack commands --update-completion`

* added stderr-nocaret warning

* added fish shell tests to CI system


Co-authored-by: becker33 <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Elsa Gonsiorowski, PhD <gonsie@me.com>
2020-06-30 14:26:27 -05:00
Greg Becker
d71fdc9719
remove three commands that have been deprecated since v0.13.0 (#17291)
* remove three commands that have been deprecated since v0.13.0
2020-06-29 11:15:56 -05:00
Scott Wittenburg
dfac09eade
Use json for buildcache index (#15002)
* Start moving toward a json buildcache index

* Add spec and database index schemas

* Add a schema for buildcache spec.yaml files

* Provide a mode for database class to generate buildcache index

* Update db and ci tests to validate object w/ new schema

* Remove unused temporary upload-s3 command

* Use database class to generate buildcache index

* Do not generate index with each buildcache creation

* Make buildcache index mode into a couple of constructor args to Database class

* Use keyword args for  _createtarball 

* Parse new json index when we get specs from buildcache

Now that only one index file per mirror needs to be fetched in
order to have all the concrete specs for binaries available on the
mirror, we can just fetch and refresh the cached specs every time
instead of needing to use the '-f' flag to force re-reading.
2020-06-26 17:05:56 -05:00
Omar Padron
7c54aa2eb0
add workaround for gitlab ci needs limit (#17219)
* add workaround for gitlab ci needs limit

* fix style/address review comments

* convert filter obj to list

* update command completion

* remove dict comprehension

* add workaround tests

* fix sorting issue between disparate types

* add indeces to format
2020-06-25 14:27:20 -04:00
Greg Becker
b26e93af3d
spack config: new subcommands add/remove (#13920)
spack config add <value>: add nested value value to the configuration scope specified
spack config remove/rm: remove specified configuration from the relevant scope
2020-06-25 09:38:01 +02:00
Tamara Dahlgren
48d3e8d350
features: Add install failure tracking removal through spack clean (#15314)
* Add ability to force removal of install failure tracking data through spack clean

* Add clean failures option to packaging guide
2020-06-24 20:28:53 -05:00
Tamara Dahlgren
96932d65a8 Added support for --fail-fast install option to terminate on first failure 2020-06-23 10:22:41 -07:00
Omar Padron
224dc95159
Pre ci optimization (#16372)
* add initial optimization script

* integrate optimization in spack ci

* make optimization opt-in

* fix import error

* flake8 fixes

* update command completion

* work around vermin errors

* fix sphynx errors
2020-06-22 13:19:47 -04:00
Massimiliano Culpo
5b272e3ff3
commands: use a single ThreadPool for spack versions (#16749)
This fixes a fork bomb in `spack versions`. Recursive generation of pools
to scrape URLs in `_spider` was creating large numbers of processes.
Instead of recursively creating process pools, we now use a single
`ThreadPool` with a concurrency limit.

More on the issue: having ~10 users running at the same time spack
versions on front-end nodes caused kernel lockup due to the high number
of sockets opened (sys-admin reports ~210k distributed over 3 nodes).
Users were internal, so they had ulimit -n set to ~70k.

The forking behavior could be observed by just running:

    $ spack versions boost

and checking the number of processes spawned. Number of processes
per se was not the issue, but each one of them opens a socket
which can stress `iptables`.

In the original issue the kernel watchdog was reporting:

    Message from syslogd@login03 at May 19 12:01:30 ...
    kernel:Watchdog CPU:110 Hard LOCKUP
    Message from syslogd@login03 at May 19 12:01:31 ...
    kernel:watchdog: BUG: soft lockup - CPU#110 stuck for 23s! [python3:2756]
    Message from syslogd@login03 at May 19 12:01:31 ...
    kernel:watchdog: BUG: soft lockup - CPU#94 stuck for 22s! [iptables:5603]
2020-06-05 00:08:32 -07:00
Peter Scheibel
24775697f5
Mirrors: add option to exclude packages from "mirror create" (#14154)
* add an --exclude-file option to 'spack mirror create' which allows a user to specify a file of specs to exclude when creating a mirror. this is anticipated to be useful especially when using the '--all' option

* allow specifying number of versions when mirroring all packages

* when mirroring all specs within an environment, include dependencies of root specs

* add '--exclude-specs' option to allow user to specify that specs should be excluded on the command line

* add test for excluding specs
2020-06-03 17:43:51 -07:00
Greg Becker
3347ef2de4
Feature: add option to create view by copying/relocating files (#16480)
* add subcommand `spack view copy/relocate`

* update bash completions

* add copy/relocate commands to view tests

* allow copied views to be removed
2020-06-03 09:45:13 -07:00
Scott Wittenburg
e0572a7d96 Pipelines: Support DAG scheduling and dynamic child pipelines
This change also adds a code path through the spack ci pipelines
infrastructure which supports PR testing on the Spack repository.
Gitlab pipelines run as a result of a PR (either creation or pushing
to a PR branch) will only verify that the packages in the environment
build without error.  When the PR branch is merged to develop,
another pipeline will run which results in the generated binaries
getting pushed to the binary mirror.
2020-05-14 21:11:07 -07:00
Ben Bergen
37e307e8cd
Added alias and bash completion for spacktivate (#16472) 2020-05-13 12:02:38 -06:00
Massimiliano Culpo
43c9ad3421
Remove 'spack bootstrap' and associated docs (#15179)
fixes #15145

This commit removes the outdated `spack bootstrap`
command and any reference to it in the documentation
and unit tests.
2020-05-11 10:55:18 -07:00
iarspider
08f449ae9a
"spack checksum" QoL (#14311)
* Non-interactive mode for spack checksum; allow passing 'package@version' to spack checksum

* Flake8 fixes

* Update checksum.py

Fix typo

* Update spack-completion script

* Automatically set non-interactive mode if more than one version passed

* Update lib/spack/spack/cmd/checksum.py

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>

* Add documentation and update spack-completion

* Flake8

* Rename option

* Update spack-completion

* Update lib/spack/spack/cmd/checksum.py

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>

* Update checksum.py

* Update stage.py

* Update create.py

Use batch mode when adding a new package

Co-authored-by: Ivan Razumov <ivan.razumov@cern.ch>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-05-07 18:34:36 -05:00
Peter Scheibel
b030a81a5f
Automatically find externals (#15158)
Add a `spack external find` command that tries to populate
`packages.yaml` with external packages from the user's `$PATH`. This
focuses on finding build dependencies. Currently, support has only been
added for `cmake`.

For a package to be discoverable with `spack external find`, it must define:
  * an `executables` class attribute containing a list of
    regular expressions that match executable names.
  * a `determine_spec_details(prefix, specs_in_prefix)` method

Spack will call `determine_spec_details()` once for each prefix where
executables are found, passing in the path to the prefix and the path to
all found executables. The package is responsible for invoking the
executables and figuring out what type of installation(s) are in the
prefix, and returning one or more specs (each with version, variants or
whatever else the user decides to include in the spec).

The found specs and prefixes will be added to the user's `packages.yaml`
file. Providing the `--not-buildable` option will mark all generated
entries in `packages.yaml` as `buildable: False`
2020-05-05 17:37:34 -07:00
Axel Huebl
d0dfa1ea4d
dev-build: --drop-in <shell> (#14887)
* dev-build: --drop-in <shell>

Add a `--drop-in <shell>` option to `spack dev-build`.
This option will automatically run a
`spack build-env <spec> -- <shell>` at the end of a `dev-build`, e.g.
to quickly drop-and-devel into a build phase of a package.

Example usage:
```
spack dev-build --before cmake --drop-in bash openpmd-api@develop
```

* build_env: drop in unit test

Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-05-01 09:37:21 -07:00
Axel Huebl
00d83cd79d
dev-build: stop before phase (#14699)
Add `-b,--before` option to dev-build command to stop before the phase in question.
2020-04-28 09:55:57 -07:00
G-Ragghianti
bae4f91bfe
Add option "--first" for "spack load" (#15622)
* Implemented --first option for "spack load"

* added test for "spack load --first"

Co-authored-by: gragghia <gragghia@localhost.localdomain>
2020-04-03 13:33:20 -07:00
Adam J. Stewart
75a2f8046a
Add commands to facilitate Spack/Python/OS reporting (#15834)
* Add --version arg to spack python command
* Add `spack debug report` command
2020-04-02 23:12:03 -07:00
Peter Scheibel
697719c181
Only use stable versions for public mirror (#15100)
* add --skip-unstable-versions option to 'spack mirror create' which skips sources/resource for packages if their version is not stable (i.e. if they are the head of a git branch rather than a fixed commit)

* '--skip-unstable-versions' should skip all VCS sources/resources, not just those which are not cachable
2020-03-07 13:38:08 +01:00
Patrick Gartung
ae87828520
buildcache cmd: add explicit message with default output dir for buildcaches. (#15090)
* Make -d directory a required option. Print messages about where buildcaches will be written.

* Add mutually exclusive required options

* spack commands --update-completion

* Apply @opadron's patch

* Update share/spack/spack-completion.bash

* Incorporate @opadron's suggestions
2020-02-25 17:49:25 -06:00
Omar Padron
00090f8f97
add --only option to buildcache create cmd (#14921)
* add --only option to buildcache create cmd

replaces the --no-deps option
2020-02-25 17:32:20 -05:00
Patrick Gartung
676eb56ab2
Buildcache cmd: add install -o/--otherarch option for installing macOS buildcaches on linux (#15192)
* Buildcache command: add install option -o/--otherarch
This will allow matching specs from other archs, for example
installing macOS buildcaches on linux hosts.

* spack commands --update-completion
2020-02-25 11:01:59 -06:00
Adam J. Stewart
342200774b
spack extensions prints list of extendable packages (#14473)
* spack extensions prints list of extendable packages

* Update tab completion scripts
2020-02-17 17:41:47 -06:00
Todd Gamblin
a7b43f1015 spack python: add -m option to run modules as scripts
It's often useful to run a module with `python -m`, e.g.:

    python -m pyinstrument script.py

Running a python script this way was hard, though, as `spack python` did
not have a similar `-m` option.  This PR adds a `-m` option to `spack
python` so that we can do things like this:

    spack python -m pyinstrument ./test.py

This makes it easy to write a script that uses a small part of Spack and
then profile it.  Previously thee easiest way to do this was to write a
custom Spack command, which is often overkill.
2020-02-12 16:45:41 -08:00
Oliver Breitwieser
22c9f5cbd8
Allow installing unsigned binary packages (#11107)
This commit introduces a `--no-check-signature` option for
`spack install` so that unsigned packages can be installed. It is
off by default (signatures required).
2020-02-06 18:59:16 -08:00
Patrick Gartung
5ad44477b2
buildcache list: restore original behavior of allowing constraints like @version. (#14732) 2020-02-03 13:40:14 -06:00
Massimiliano Culpo
9635ff3d20
spack containerize generates containers from envs (#14202)
This PR adds a new command to Spack:
```console
$ spack containerize -h
usage: spack containerize [-h] [--config CONFIG]

creates recipes to build images for different container runtimes

optional arguments:
  -h, --help       show this help message and exit
  --config CONFIG  configuration for the container recipe that will be generated
```
which takes an environment with an additional `container` section:
```yaml
spack:
  specs:
  - gromacs build_type=Release 
  - mpich
  - fftw precision=float
  packages:
    all:
      target: [broadwell]

  container:
    # Select the format of the recipe e.g. docker,
    # singularity or anything else that is currently supported
    format: docker
    
    # Select from a valid list of images
    base:
      image: "ubuntu:18.04"
      spack: prerelease

    # Additional system packages that are needed at runtime
    os_packages:
    - libgomp1
```
and turns it into a `Dockerfile` or a Singularity definition file, for instance:
```Dockerfile
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-bionic:prerelease as builder

# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&&  (echo "spack:" \
&&   echo "  specs:" \
&&   echo "  - gromacs build_type=Release" \
&&   echo "  - mpich" \
&&   echo "  - fftw precision=float" \
&&   echo "  packages:" \
&&   echo "    all:" \
&&   echo "      target:" \
&&   echo "      - broadwell" \
&&   echo "  config:" \
&&   echo "    install_tree: /opt/software" \
&&   echo "  concretization: together" \
&&   echo "  view: /opt/view") > /opt/spack-environment/spack.yaml

# Install the software, remove unecessary deps and strip executables
RUN cd /opt/spack-environment && spack install && spack autoremove -y
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
    xargs file -i | \
    grep 'charset=binary' | \
    grep 'x-executable\|x-archive\|x-sharedlib' | \
    awk -F: '{print $1}' | xargs strip -s


# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
    spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh

# Bare OS image to run the installed executables
FROM ubuntu:18.04

COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh

RUN apt-get -yqq update && apt-get -yqq upgrade                                   \
 && apt-get -yqq install libgomp1 \
 && rm -rf /var/lib/apt/lists/*

ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
```
2020-01-30 17:19:55 -08:00
Patrick Gartung
23a7feb917
Limit the number of spec files downloaded to find matches for buildcaches (#14659)
* Limit the number of spec flies downloaded to find matches
2020-01-30 10:56:10 -06:00
Adam J. Stewart
dcd8d7a620 Add spack config list command for tab completion (#14474)
* Add spack config list command for tab completion
* Update tab completion scripts
2020-01-24 17:28:20 -08:00
Todd Gamblin
04a6a55cf8
commands: add simple spack commands --update-completion argument (#14607)
Instead of another script, this adds a simple argument to `spack
commands` that updates the completion script.  Developers can now just
run:

    spack commands --update-completion

This should make it simpler for developers to remember to run this
*before* the tests fail.  Also, this version tab-completes.
2020-01-23 14:48:06 -08:00
Greg Becker
c9e01ff9d7 shell support: spack load no longer needs modules (#14062)
Previously the `spack load` command was a wrapper around `module load`. This required some bootstrapping of modules to make `spack load` work properly.

With this PR, the `spack` shell function handles the environment modifications necessary to add packages to your user environment. This removes the dependence on environment modules or lmod and removes the requirement to bootstrap spack (beyond using the setup-env scripts).

Included in this PR is support for MacOS when using Apple's System Integrity Protection (SIP), which is enabled by default in modern MacOS versions. SIP clears the `LD_LIBRARY_PATH` and `DYLD_LIBRARY_PATH` variables on process startup for executables that live in `/usr` (but not '/usr/local', `/System`, `/bin`, and `/sbin` among other system locations. Spack cannot know the `LD_LIBRARY_PATH` of the calling process when executed using `/bin/sh` and `/usr/bin/python`. The `spack` shell function now manually forwards these two variables, if they are present, as `SPACK_<VAR>` and recovers those values on startup.

- [x] spack load/unload no longer delegate to modules
- [x] refactor user_environment modification calculations
- [x] update documentation for spack load/unload

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-22 22:36:02 -08:00
Adam J. Stewart
11f2b61261 Use spack commands --format=bash to generate shell completion (#14393)
This PR adds a `--format=bash` option to `spack commands` to
auto-generate the Bash programmable tab completion script. It can be
extended to work for other shells.

Progress:

- [x] Fix bug in superclass initialization in `ArgparseWriter`
- [x] Refactor `ArgparseWriter` (see below)
- [x] Ensure that output of old `--format` options remains the same
- [x] Add `ArgparseCompletionWriter` and `BashCompletionWriter`
- [x] Add `--aliases` option to add command aliases
- [x] Standardize positional argument names
- [x] Tests for `spack commands --format=bash` coverage
- [x] Tests to make sure `spack-completion.bash` stays up-to-date
- [x] Tests for `spack-completion.bash` coverage
- [x] Speed up `spack-completion.bash` by caching subroutine calls

This PR also necessitates a significant refactoring of
`ArgparseWriter`. Previously, `ArgparseWriter` was mostly a single
`_write` method which handled everything from extracting the information
we care about from the parser to formatting the output. Now, `_write`
only handles recursion, while the information extraction is split into a
separate `parse` method, and the formatting is handled by `format`. This
allows subclasses to completely redefine how the format will appear
without overriding all of `_write`.

Co-Authored-by: Todd Gamblin <tgamblin@llnl.gov>
2020-01-22 21:31:12 -08:00
Adam J. Stewart
257e71d87a
Reformat Bash tab completion script (#14456) 2020-01-10 11:32:50 -06:00
Adam J. Stewart
eddb42ed43
Fix outdated bash tab completion (#14392) 2020-01-06 23:18:14 -06:00
Todd Gamblin
4beb9fc5d3 tests: improved spack test command line options
Previously, `spack test` automatically passed all of its arguments to
`pytest -k` if no options were provided, and to `pytest` if they were.
`spack test -l` also provided a list of test filenames, but they didn't
really let you completely narrow down which tests you wanted to run.

Instead of trying to do our own weird thing, this passes `spack test`
args directly to `pytest`, and omits the implicit `-k`.  This means we
can now run, e.g.:

```console
$ spack test spec_syntax.py::TestSpecSyntax::test_ambiguous
```

This wasn't possible before, because we'd pass the fully qualified name
to `pytest -k` and get an error.

Because `pytest` doesn't have the greatest ability to list tests, I've
tweaked the `-l`/`--list`, `-L`/`--list-long`, and `-N`/`--list-names`
options to `spack test` so that they help you understand the names
better.  you can combine these options with `-k` or other arguments to do
pretty powerful searches.

This one makes it easy to get a list of names so you can run tests in
different orders (something I find useful for debugging `pytest` issues):

```console
$ spack test --list-names -k "spec and concretize"
cmd/env.py::test_concretize_user_specs_together
concretize.py::TestConcretize::test_conflicts_in_spec
concretize.py::TestConcretize::test_find_spec_children
concretize.py::TestConcretize::test_find_spec_none
concretize.py::TestConcretize::test_find_spec_parents
concretize.py::TestConcretize::test_find_spec_self
concretize.py::TestConcretize::test_find_spec_sibling
concretize.py::TestConcretize::test_no_matching_compiler_specs
concretize.py::TestConcretize::test_simultaneous_concretization_of_specs
spec_dag.py::TestSpecDag::test_concretize_deptypes
spec_dag.py::TestSpecDag::test_copy_concretized
```

You can combine any list option with keywords:

```console
$ spack test --list -k microarchitecture
llnl/util/cpu.py  modules/lmod.py
```

```console
$ spack test --list-long -k microarchitecture
llnl/util/cpu.py::
    test_generic_microarchitecture

modules/lmod.py::TestLmod::
    test_only_generic_microarchitectures_in_root
```

Or just list specific files:

```console
$ spack test --list-long cmd/test.py
cmd/test.py::
    test_list                       test_list_names_with_pytest_arg
    test_list_long                  test_list_with_keywords
    test_list_long_with_pytest_arg  test_list_with_pytest_arg
    test_list_names
```

Hopefully this stuff will help with debugging test issues.

- [x] make `spack test` send args directly to `pytest` instead of trying
  to do fancy things.
- [x] rework `--list`, `--list-long`, and add `--list-names` to make
  searching for tests easier.
- [x] make it possible to mix Spack's list args with `pytest` args
  (they're just fancy parsing around `pytest --collect-only`)
- [x] add docs
- [x] add tests
- [x] update spack completion
2020-01-01 21:37:02 -08:00
Todd Gamblin
4af6303086
copyright: update copyright dates for 2020 (#14328) 2019-12-30 22:36:56 -08:00
Zack Galbreath
0f5724e908 Split out CDash options to a separate help document (#13704)
Prevent `spack help install` from getting too cluttered with CDash-specific documentation.
2019-12-13 10:15:22 -08:00
Greg Becker
6c55a7c85f
cmd/install: remove unused install_status option (#13751)
* cmd/install: remove unused install_status option

* update bash completions for spack install
2019-11-22 11:17:37 -07:00
Todd Gamblin
e0b94dba14
completion: add bash completion for spack spec --json (#13433) 2019-10-25 11:02:52 -07:00
Massimiliano Culpo
76b9c56110 Remove support for generating dotkit files (#11986)
Dotkit is being used only at a few sites and has been deprecated on new
machines. This commit removes all the code that provide support for the
generation of dotkit module files.

A new validator named "deprecatedProperties" has been added to the
jsonschema validators. It permits to prompt a warning message or exit
with an error if a property that has been marked as deprecated is
encountered.

* Removed references to dotkit in the docs
* Removed references to dotkit in setup-env-test.sh
* Added a unit test for the 'deprecatedProperties' schema validator
2019-10-02 22:15:01 -07:00
Glenn Johnson
5397606b34 Add --known-targets to bash completion for arch command (#12887)
This PR adds the new --known-targets flag to the `spack arch` command.
2019-09-20 20:09:44 +02:00
Todd Gamblin
1b877e8e0f tests and completions for spack find --json and spack find --format 2019-09-02 19:24:48 -07:00
Todd Gamblin
b0abbfecb8 new command: spack maintainers queries package maintainers
- We don't currently make enough use of the maintainers field on
  packages, though we could use it to assign reviews.

- add a command that allows maintainers to be queried
  - can ask who is maintaining a package or packages
  - can ask what packages users are maintaining
  - can list all maintained or unmaintained packages

- add tests for the command
2019-07-24 14:10:08 -07:00
Greg Becker
5cf8878185 feature: Allow developers to use Spack for partial builds (#12006)
Added new diy option.
2019-07-17 11:46:56 -07:00
Oliver Breitwieser
d1af7ae5ab Copy --{use,no}-cache from install to bootstrap (#11786) 2019-07-12 16:30:12 -07:00
Tamara Dahlgren
d615d0a1ec Add template creation test, --skip-editor option (#11970)
* Add template creation test
* Added --skip-editor option to "spack create": normally
  "spack create" opens an editor for the user after generating a
  package file; when the --skip-editor option is used, "spack create"
  only generates the package file and does not open an editor
* Added --skip-editor option to bash completion
2019-07-12 11:22:09 -07:00
Todd Gamblin
2a51e07fde
commands: completion for new --update and --header options (#11566) 2019-05-27 14:13:58 -07:00
Adam J. Stewart
72a41a4918 spack versions: only list safe versions (#10004)
* spack versions: only list safe versions
* Add unit tests for spack versions -s
2019-01-10 00:32:47 -08:00
Todd Gamblin
6f50cd52ed copyright: update license headers for 2013-2019 copyright. 2019-01-01 00:44:28 -08:00
Todd Gamblin
d763e92141 commands: add spack resource command to inspect downloadable files
- currently just looks at patches
  - allows you to find out which package applied a patch to a spec

- intended to work with tarballs and resources in the future.

- add tab completion for `spack resource` and subcommands
2018-12-30 00:19:08 -08:00
Adam J. Stewart
29f26da04d
Update Spack's Bash completion script (#10114)
* Update Spack's Bash completion script

* spack mirror create options changed
2018-12-17 13:02:09 -06:00
Todd Gamblin
eea786f4e8 relicense: replace LGPL headers with Apache-2.0/MIT SPDX headers
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
  - core and most packages are (Apache-2.0 OR MIT)
  - a very small number of remaining packages are LGPL-2.1-only
2018-10-17 14:42:06 -07:00
Adam J. Stewart
4de301ebf2 Update bash completion with new spack arch flags 2018-08-03 11:32:08 -05:00
Massimiliano Culpo
5b8b7a5873 The auto-completion file is now consistent with spack module 2018-07-24 11:27:17 -07:00
Adam J. Stewart
ef37cdb380
Autocomplete spack providers (#7783) 2018-04-17 09:33:04 -05:00
Adam J. Stewart
4332e75986
Various updates to Spack's Bash completion (#7590)
* Various updates to Spack's Bash completion

* Equals sign messes up tab completion
2018-03-25 08:03:29 -05:00
Todd Gamblin
54f97d1dec
Update copyright on LLNL files for 2018. (#7592) 2018-03-24 12:13:52 -07:00
Adam J. Stewart
5f6df40f3b Fix bash tab completion for spack install (#6868) 2018-01-10 13:51:04 +01:00
Todd Gamblin
05fa302655
Replace github.com/llnl/spack with github.com/spack/spack (#6142)
We moved to a new GitHub org! Now make the code and docs reflect that.
2017-11-04 17:08:04 -07:00
Michael Kuhn
84ae7872d3 Update copyright notices for 2017 (#5295) 2017-09-06 17:44:16 -10:00
Matthew Scott Krafczyk
c94933343a Add --show-full-compiler option to 'spack find'
When 'spack find' is invoked with the '--show-full-compiler' option,
the compiler flags and version are shown for each spec that is found.
2017-08-28 10:35:46 -07:00
Christoph Junghans
fa1d0a8a4d Add --source option to spack install (#4102)
- -- source will copy source into prefix along with the package.
- added a test for --source, as well
2017-08-23 14:08:52 -07:00
Adam J. Stewart
af02774b3e Add tab completion & update docs for buildcache
This adds tab completion and fixes some formatting issues in the
documentation for the "spack buildcache" command.
2017-08-16 13:58:09 -07:00
Massimiliano Culpo
faeb1b77b2 Merged 'purge' command with 'clean' and deleted 'purge' (#4970)
* Merged 'purge' command with 'clean'. Deleted 'purge'. fixes #2942

'spack purge' has been merged with 'spack clean'. Documentation has been
updated accordingly. The 'clean' and 'purge' behavior are not mutually
exclusive, and they log brief information to tty while they go.

* Fixed a wrong reference to spack clean in the docs

* Added tests for 'spack clean'. Updated bash completion.
2017-08-09 10:02:38 -07:00
paulhopkins
1c7e5724d9 Add --color=[always|never|auto] argument; fix color when piping (#3013)
* Disable spec colorization when redirecting stdout and add command line flag to re-enable
* Add command line `--color` flag to control output colorization
* Add options to `llnl.util.tty.color` to allow color to be auto/always/never
* Add `Spec.cformat()` function to be used when `format()` should have auto-coloring
2017-07-31 12:57:47 -07:00
Todd Gamblin
cac4362f64 Make LICENSE recognizable by GitHub. (#4598) 2017-06-24 22:22:55 -07:00
Adam J. Stewart
0bbafb1673 Fix tab completion of Spack subcommands (#4442) 2017-06-07 11:52:07 -05:00
Massimiliano Culpo
060351e121 bash completion: fixed invalid identifier (#4079)
* bash completion: fixed `_spack_create-db-tarball': not a valid identifier

* bash completion: dashes are translated to underscores

This also fixes the name of the subfunction to be called, as apparently
it was not updated after moving the command `create-db-tarball`.
2017-05-03 08:12:33 -05:00
Todd Gamblin
beeca6bb54 Revert "Override partial installs by default" (#3918)
* Revert "Override partial installs by default (#3530)"

This reverts commit a65c37f15d.
2017-04-20 03:53:41 -07:00
scheibelp
a65c37f15d Override partial installs by default (#3530)
* Package install remove prior unfinished installs

Depending on how spack is terminated in the middle of building a
package it may leave a partially installed package in the install
prefix. Originally Spack treated the package as being installed if
the prefix was present, in which case the user would have to
manually remove the installation prefix before restarting an
install. This commit adds a more thorough check to ensure that a
package is actually installed. If the installation prefix is present
but Spack determines that the install did not complete, it removes
the installation prefix and starts a new install; if the user has
enabled --keep-prefix, then Spack reverts to its old behavior.

* Added test for partial install handling

* Added test for restoring DB

* Style fixes

* Restoring 2.6 compatibility

* Relocated repair logic to separate function

* If --keep-prefix is set, package installs will continue an install from an existing prefix if one is present

* check metadata consistency when continuing partial install

* Added --force option to make spack reinstall a package (and all dependencies) from scratch

* Updated bash completion; removed '-f' shorthand for '--force' for install command

* dont use multiple write modes for completion file
2017-04-19 21:59:18 -07:00
Adam J. Stewart
50df071ad9 Overhaul Spack's URL parsing (#2972)
* Remove fake URLs from Spack

* Ignore long lines for URLs that start with ftp:

* Preliminary changes to version regexes

* New redesign of version regexes

* Allow letters in version-only

* Fix detection of versions that end in Final

* Rearrange a few regexes and add examples

* Add tests for common download repositories

* Add test cases for common tarball naming schemes

* Finalize version regexes

* spack url test -> spack url summary

* Clean up comments

* Rearrange suffix checks

* Use query strings for name detection

* Remove no longer necessary url_for_version functions

* Strip off extraneous information after package name

* Add one more test

* Dot in square brackets does not need to be escaped

* Move renaming outside of parse_name_offset

* Fix versions for a couple more packages

* Fix flake8 and doc tests

* Correctly parse Python, Lua, and Bio++ package names

* Use effective URLs for mfem

* Add checksummed version to mitos

* Remove url_for_version from STAR-CCM+ package

* Revert changes to version numbers with underscores and dashes

* Fix name detection for tbb

* Correctly parse Ruby gems

* Reverted mfem back to shortened URLs.

* Updated instructions for better security

* Remove preferred=True from newest version

* Add tests for new `spack url list` flags

* Add tests for strip_name_suffixes

* Add unit tests for version separators

* Fix bugs related to parseable name but in parseable version

* Remove dead code, update docstring

* Ignore 'binary' at end of version string

* Remove platform from version

* Flip libedit version numbers

* Re-support weird NCO alpha/beta versions

* Rebase and remove one new fake URL

* Add / to beginning of regex to avoid picking up similarly named packages

* Ignore weird tar versions

* Fix bug in url parse --spider when no versions found

* Less strict version matching for spack versions

* Don't rename Python packages

* Be a little more selective, version must begin with a digit

* Re-add fake URLs

* Fix up several other packages

* Ignore more file endings

* Add parsing support for Miniconda

* Update tab completion

* XFAILS are now PASSES for 2 web tests
2017-04-03 15:34:16 -07:00
Adam J. Stewart
c0cfaacbc8 Set default providers for all virtual dependencies (#3634)
* Set default providers for everything

* Add default OpenFOAM provider
2017-04-02 14:48:27 -05:00
Mark Olesen
9e1abb13dc support OpenFOAM package(s) (#3528)
* ENH: add package for building OpenFOAM (1612) from www.openfoam.com
- provide 'openfoam' as virtual package.
- package as openfoam-com to reflect the distribution point.

This initial spack packaging for OpenFOAM supports a number of possible
variants and should handle 64-bit labels properly now that the scotch
package has been updated accordingly.

* ENH: update package for foam-extend (extend-project.de)

- provide 'openfoam' as virtual package.

- much of the build is now aligned with how the openfoam-com package
  looks, with the aim of future refactoring.

- avoid installing intermediate targets.

- contains its own environment sourcing script for the build, for more
  flexibility and robustness (doesn't touch the python build environ)

* ENH: added package for building from openfoam.org

- provide 'openfoam' as a virtual package.

- this is largely a direct copy of the openfoam-com package.
  It has been supplied as a courtesy for users and to ensure maximum
  consistency in quality and naming between the foam-extend,
  openfoam-com and openfoam-org packages.

* CONFIG: add openfoam into bash completion providers list

* ENH: have openfoam-com use spack as USERMPI

- also simplify the generation of mplib/compiler rules

* ENH: have openfoam-org use spack as SYSTEMMPI

- this setup requires more environment settings than USERMPI
  (openfoam-com), but is currently the only means of integration
  for openfoam-org

- simplify generation of mplib/compiler rules

* ENH: simplify generation of mplib/compiler rules (foam-extend)

- rename mpi rules from SPACK,SPACKMPI to USER,USERMPI for consistency
  with openfoam-com and to generalize for any build system.

* STYLE: record spack tree as a log file (openfoam)

- can be useful for future diagnostics and general record keeping
2017-03-30 16:35:57 -07:00
Adam J. Stewart
f0e99456ed Fix tab completion of directories and filenames (#3184) 2017-02-20 15:44:58 -08:00
Adam J. Stewart
c2d210568c Fix tab completion after change to uninstall flag (#3175) 2017-02-18 13:08:59 -06:00
Adam J. Stewart
941dfcbe43 Bash Programmable Completion for Spack (#3026) 2017-02-06 12:34:35 -08:00