Commit Graph

147 Commits

Author SHA1 Message Date
Adam J. Stewart
603569e321
Style: black 23, skip magic trailing comma (#35351)
* Style: black 23, skip magic trailing commas

* isort should use same line length as black

* Fix unused import

* Update version of black used in CI

* Update new packages

* Update new packages
2023-02-16 23:06:12 -06:00
Harmen Stoppels
fce95e2efb
license year bump (#34921)
* license bump year
* fix black issues of modified files
* mypy
* fix 2021 -> 2023
2023-01-18 14:30:17 -08:00
Todd Gamblin
d991ec90e3
new command: spack pkg grep to search package files (#34388)
It's very common for us to tell users to grep through the existing Spack packages to
find examples of what they want, and it's also very common for package developers to do
it. Now, searching packages is even easier.

`spack pkg grep` runs grep on all `package.py` files in repos known to Spack. It has no
special options other than the search string; all options passed to it are forwarded
along to `grep`.

```console
> spack pkg grep --help
usage: spack pkg grep [--help] ...

positional arguments:
  grep_args  arguments for grep

options:
  --help     show this help message and exit
```

```console
> spack pkg grep CMakePackage | head -3
/Users/gamblin2/src/spack/var/spack/repos/builtin/packages/3dtk/package.py:class _3dtk(CMakePackage):
/Users/gamblin2/src/spack/var/spack/repos/builtin/packages/abseil-cpp/package.py:class AbseilCpp(CMakePackage):
/Users/gamblin2/src/spack/var/spack/repos/builtin/packages/accfft/package.py:class Accfft(CMakePackage, CudaPackage):
```

```console
> spack pkg grep -Eho '(\S*)\(PythonPackage\)' | head -3
AwsParallelcluster(PythonPackage)
Awscli(PythonPackage)
Bueno(PythonPackage)
```

## Return Value

This retains the return value semantics of `grep`:
* 0  for found,
* 1  for not found
* >1 for error

## Choosing a `grep`

You can set the ``SPACK_GREP`` environment variable to choose the ``grep``
executable this command should use.
2022-12-09 10:07:54 -08:00
John W. Parent
0e69710f41
Windows: reenable unit tests (#33385)
Unit tests on Windows are supposed to pass for any PR to pass CI.
However, the return code for the unit test command was not being
checked, which meant this check was always passing (effectively
disabled). This PR

* Properly checks the result of the unit tests and fails if the
  unit tests fail
* Fixes (or disables on Windows) a number of tests which have
  "drifted" out of support on Windows since this check was
  effectively disabled
2022-12-09 13:27:46 +00:00
Todd Gamblin
606eef43bd
bugfix: spack load shell test can fail on macos (#34419)
At some point the `a` mock package became an `AutotoolsPackage`, and that means it
depends on `gnuconfig` on macOS. This was causing one of our shell tests to fail on
macOS because it was testing for `{a.prefix.bin}:{b.prefix.bin}` in `PATH`, but
`gnuconfig` shows up between them.

- [x] simplify the test to check `spack load --sh a` and `spack load --sh b` separately
2022-12-09 10:36:54 +00:00
Greg Becker
451e3ff50b
warn about removal of deprecated format strings (#34101)
* warn about removal of deprecated format strings

Co-authored-by: becker33 <becker33@users.noreply.github.com>
2022-11-28 10:03:49 -08:00
Massimiliano Culpo
bd198312c9
Revert "Warn about removal of deprecated format strings (#33829)" (#34056)
This reverts commit 7f9af8d4a0.
2022-11-22 12:35:36 +01:00
Greg Becker
7f9af8d4a0
Warn about removal of deprecated format strings (#33829)
Co-authored-by: becker33 <becker33@users.noreply.github.com>
2022-11-22 10:56:57 +01:00
Massimiliano Culpo
23aef6bb94
Let pytest-cov create the xml directly (#33619)
`coverage` sometimes failed to combine, even if there were multiple reports.
2022-11-01 19:04:45 +01:00
Massimiliano Culpo
4b237349a3
Remove recursive symbolic link in lib/spack/docs from git repository (#33483)
Delete code removing the symlink during CI
2022-10-25 12:27:13 +02:00
Massimiliano Culpo
1427ddaa59
ci: restore coverage computation (#32585)
* ci: restore coverage computation

* Mark "test_foreground_background" as xfail

* Mark "test_foreground_background_output" as xfail

* Make number of processes explicit, remove verbosity on linux

* Run coverage on just 3 Python jobs for linux

* Run coverage on just 3 Python jobs for linux

* Run coverage on just 2 Python jobs for linux

* Add back verbose, since before we didn't encounter the xdist internal error

* Reduce the workers to 2

* Try to use command line
2022-09-10 07:25:44 -06:00
Massimiliano Culpo
67534516c7
ci: avoid running coverage on package only PRs (#32573)
* ci: remove !docs from "core" filters

Written like it is now it causes package only PRs
to run with coverage.

* Try to skip job under condition, see if the workflow proceed

* Try to cancel a running CI job

* Simplify linux unit-tests, skip windows unit-tests on package PRs

* Reduce the inputs to unit-tests workflow

* Move control logic to main workflow, remove inputs

* Revert "Move control logic to main workflow, remove inputs"

This reverts commit 0c46fece4c.

* Do not compute "with_coverage" since it's always == to "core"

* Remove workflow dispatch from unit tests

* Revert "Revert "Move control logic to main workflow, remove inputs""

This reverts commit dd4e4a4e61.

* Try to skip all from the main workflow

* Add back bootstrap to needed checks for "all"

* Restore the correct logic for conditionals
2022-09-08 10:58:53 -07:00
Tom Scogland
762ba27036
Make GHA tests parallel by using xdist (#32361)
* Add two no-op jobs named "all-prechecks" and "all"

These are a suggestion from @tgamblin, they are stable named markers we
can use from gitlab and possibly for required checks to make CI more
resilient to refactors changing the names of specific checks.

* Enable parallel testing using xdist for unit testing in CI

* Normalize tmp paths to deal with macos

* add -u flag compatibility to spack python

As of now, it is accepted and ignored.  The usage with xdist, where it
is invoked specifically by `python -u spack python` which is then passed
`-u` by xdist is the entire reason for doing this.  It should never be
used without explicitly passing -u to the executing python interpreter.

* use spack python in xdist to support python 2

When running on python2, spack has many import cycles unless started
through main.  To allow that, this uses `spack python` as the
interpreter, leveraging the `-u` support so xdist doesn't error out when
it unconditionally requests unbuffered binary IO.

* Use shutil.move to account for tmpdir being in a separate filesystem sometimes
2022-09-07 20:12:57 +02:00
Massimiliano Culpo
01298287f6
Fix CI for package only PRs (#32473) 2022-09-01 14:41:28 +02:00
Massimiliano Culpo
6979d47cb6
CI: reduce the amount of tests run in the original concretizer (#32179)
* CI: reduce the amount of tests run in the original concretizer
* Don't test Python 3.6 on the original concretizer
2022-08-17 09:16:19 +02:00
Harmen Stoppels
ceda5fb46c
Don't set LD_LIBRARY_PATH by default on Linux (#28354)
`LD_LIBRARY_PATH` can break system executables (e.g., when an enviornment is loaded) and isn't necessary thanks to `RPATH`s.  Packages that require `LD_LIBRARY_PATH` can set this in `setup_run_environment`.

- [x] Prefix inspections no longer set `LD_LIBRARY_PATH` by default
- [x] Document changes and workarounds for people who want `LD_LIBRARY_PATH`
2022-08-11 09:33:08 -05:00
Todd Gamblin
549ba1ed32 black: fix style check package and flake8 formatting for black
Black will automatically fix a lot of the exceptions we previously allowed for
directives, so we don't need them in our custom `flake8_formatter` anymore.

- [x] remove `E501` (long line) exceptions for directives from `flake8_formatter`,
      as they won't help us now.
- [x] Refine exceptions for long URLs in the `flake8_formatter`.
- [x] Adjust the mock `flake8-package` to exhibit the exceptions we still allow.
- [x] Update style tests for new `flake8-package`.
- [x] Blacken style test.
2022-07-31 13:29:20 -07:00
Tom Scogland
18c2f1a57a
refactor: packages import spack.package explicitly (#30404)
Explicitly import package utilities in all packages, and corresponding fallout.

This includes:

* rename `spack.package` to `spack.package_base`
* rename `spack.pkgkit` to `spack.package`
* update all packages in builtin, builtin_mock and tutorials to include `from spack.package import *`
* update spack style
  * ensure packages include the import
  * automatically add the new import and remove any/all imports of `spack` and `spack.pkgkit`
    from packages when using `--fix`
  * add support for type-checking packages with mypy when SPACK_MYPY_CHECK_PACKAGES
    is set in the environment
* fix all type checking errors in packages in spack upstream
* update spack create to include the new imports
* update spack repo to inject the new import, injection persists to allow for a deprecation period

Original message below:
 
As requested @adamjstewart, update all packages to use pkgkit.  I ended up using isort to do this,
so repro is easy:

```console
$ isort -a 'from spack.pkgkit import *' --rm 'spack' ./var/spack/repos/builtin/packages/*/package.py
$ spack style --fix
```

There were several line spacing fixups caused either by space manipulation in isort or by packages
that haven't been touched since we added requirements, but there are no functional changes in here.

* [x] add config to isort to make sure this is maintained going forward
2022-05-28 12:55:44 -04:00
Massimiliano Culpo
ff04d1bfc1
Use the non-deprecated MetaPathFinder interface (#29745)
* Extract the MetaPathFinder and Loaders for packages in their own classes

https://peps.python.org/pep-0451/

Currently, RepoPath and Repo implement the (deprecated) interface of
MetaPathFinder (find_module) and of Loader (load_module). This commit
extracts both of them and places the code in their own classes.

The MetaPathFinder interface is updated to contain both the deprecated
"find_module" (for Python 2.7 support) and the recommended "find_spec".
Update of the Loader interface is deferred at a subsequent commit.

* Move the lines to be prepended inside "RepoLoader"

Also adjust the naming of a few variables too

* Remove spack.util.imp, since code is only used in spack.repo

* Remove support from loading Python modules Python > 3 but < 3.5

* Remove `Repo._create_namespace`

This function was interacting badly with the MetaPathFinder
and causing issues with "normal" imports. Removing the
function allows to do things like:
```python
import spack.pkg.builtin.mpich
cls = spack.pkg.builtin.mpich.Mpich
```

* Remove code needed to trigger the Singleton evaluation

The finder is coded in a way to trigger the Singleton,
so we don't need external code now that we register it
at module level into `sys.meta_path`.

* Add unit tests
2022-04-07 15:58:20 -07:00
John Parent
4aee27816e Windows Support: Testing Suite integration
Broaden support for execution of the test suite
on Windows.
General bug and review fixups
2022-03-17 09:01:01 -07:00
John Parent
90c773488c Add Github Actions for Windows (#24504)
Setup Installer CI (#25184), (#25191)

Co-authored-by: Zack Galbreath <zack.galbreath@kitware.com>
Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
Massimiliano Culpo
9b298fd7e4
Add a new test to catch exit code failure (#29244)
* Add a new test to catch exit code failure

fixes #29226

This introduces a new unit test that checks the return
code of `spack unit-test` when it is supposed to fail.

This is to prevent bugs like the one introduced in #25601
in which CI didn't catch a missing return statement.

In retrospective it seems that the shell test we have right
now all go through `tty.die` or similar code paths which
call `sys.exit(a)` explicitly. This new test instead checks
`spack unit-test` which relies on the return code from
command invocation in case of errors.
2022-02-28 12:55:24 -08:00
Todd Gamblin
93377942d1 Update copyright year to 2022 2022-01-14 22:50:21 -08:00
Harmen Stoppels
f0b70b7c8e
Fix spack install --v[tab] spec (#28278) 2022-01-06 14:47:03 +01:00
Massimiliano Culpo
60a5b2a85e Merge tag 'v0.17.1' into develop 2021-12-23 19:48:36 +01:00
Massimiliano Culpo
4381cb5957
New subcommand: spack bootstrap status (#28004)
This command pokes the environment, Python interpreter
and bootstrap store to check if dependencies needed by
Spack are available.

If any are missing, it shows a comprehensible message.
2021-12-23 10:34:04 -08:00
Massimiliano Culpo
e974b44e86 Fix execution of style tests 2021-12-23 16:34:31 +01:00
Harmen Stoppels
654f6839eb ci: run style unit tests only if we target develop (#27472)
Some tests assume the base branch is develop, but this branch may not
have been checked out.
2021-12-23 16:02:09 +01:00
Harmen Stoppels
8f7640dbef
ci: run style unit tests only if we target develop (#27472)
Some tests assume the base branch is develop, but this branch may not
have been checked out.
2021-11-18 13:00:39 +01:00
Todd Gamblin
e13e697067
commands: spack load --list alias for spack find --loaded (#27184)
See #25249 and https://github.com/spack/spack/pull/27159#issuecomment-958163679.
This adds `spack load --list` as an alias for `spack find --loaded`.  The new command is
not as powerful as `spack find --loaded`, as you can't combine it with all the queries or
formats that `spack find` provides.  However, it is more intuitively located in the command
structure in that it appears in the output of `spack load --help`.

The idea here is that people can use `spack load --list`  for simple stuff but fall back to
`spack find --loaded` if they need more.

- add help to `spack load --list` that references `spack find`
- factor some parts of `spack find` out to be called from `spack load`
- add shell tests
- update docs

Co-authored-by: Peter Josef Scheibel <scheibel1@llnl.gov>
Co-authored-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>
2021-11-05 00:58:29 -07:00
Harmen Stoppels
49034abd76
Fix exit codes posix shell wrapper (#27012)
* Correct exit code in sh wrapper

* Fix tests

* SC2069
2021-10-29 08:10:22 +00:00
Harmen Stoppels
574395af93
Fix exit codes in fish (#27028) 2021-10-29 01:10:31 +00:00
Todd Gamblin
233dabbd4f bugfix: config edit should work with a malformed spack.yaml
If you don't format `spack.yaml` correctly, `spack config edit` still fails and
you have to edit your `spack.yaml` manually.

- [x] Add some code to `_main()` to defer `ConfigFormatError` when loading the
  environment, until we know what command is being run.

- [x] Make `spack config edit` use `SPACK_ENV` instead of the config scope
  object to find `spack.yaml`, so it can work even if the environment is bad.

Co-authored-by: scheibelp <scheibel1@llnl.gov>
2021-10-28 15:37:44 -07:00
Harmen Stoppels
6d030ba137
Deactivate previous env before activating new one (#25409)
* Deactivate previous env before activating new one

Currently on develop you can run `spack env activate` multiple times to switch
between environments, but they leave traces, even though Spack only supports
one active environment at a time.

Currently:

```console
$ spack env create a
$ spack env create b
$ spack env activate -p a
[a] $ spack env activate -p b
[b] [a] $ spack env activate -p b
[a] [b] [a] $ spack env activate -p a
[a] [b] [c] $ echo $MANPATH | tr ":" "\n"
/path/to/environments/a/.spack-env/view/share/man
/path/to/environments/a/.spack-env/view/man
/path/to/environments/b/.spack-env/view/share/man
/path/to/environments/b/.spack-env/view/man
```

This PR fixes that:

```console
$ spack env activate -p a
[a] $ spack env activate -p b
[b] $ spack env activate -p a
[a] $ echo $MANPATH | tr ":" "\n"
/path/to/environments/a/.spack-env/view/share/man
/path/to/environments/a/.spack-env/view/man
```
2021-10-28 11:39:25 -07:00
Massimiliano Culpo
3d5444fdd8
Remove documentation tests from GitHub Actions (#26981)
We moved documentation tests to readthedocs since a while,
so remove the one on GitHub.
2021-10-27 19:02:52 +02:00
Harmen Stoppels
c0c9ab113e
Add spack env activate --temp (#25388)
Creates an environment in a temporary directory and activates it, which
is useful for a quick ephemeral environment:

```
$ spack env activate -p --temp
[spack-1a203lyg] $ spack add zlib
==> Adding zlib to environment /tmp/spack-1a203lyg
==> Updating view at /tmp/spack-1a203lyg/.spack-env/view
```
2021-10-11 06:56:03 -04:00
Harmen Stoppels
12e87ebf14
Fix fish test "framework" (#25242)
Remove broken test, see #21699
2021-08-27 10:52:00 +02:00
Massimiliano Culpo
10695f1ed3
Use kcov from official Ubuntu 20.04 repository (#25385)
* Ubuntu 20.04 provides kcov, so don't build from source

* Use two undocumented options for kcov v3.8
2021-08-19 14:03:10 -07:00
Todd Gamblin
6104c31556 bugfix: be careful about GITHUB_BASE_REF in spack style
`spack style` previously used a Travis CI variable to figure out
what the base branch of a PR was, and this was apparently also set
on `develop`.  We switched to `GITHUB_BASE_REF` to support GitHub
Actions, but it looks like this is set to `""` in pushes to develop,
so `spack style` breaks there.

This PR does two things:

- [x] Remove `GITHUB_BASE_REF` knowledge from `spack style` entirely

- [x] Handle `GITHUB_BASE_REF` in style scripts instead, and explicitly
      pass the base ref if it is present, but don't otherwise.

This makes `spack style` *not* dependent on the environment and fixes
handling of the base branch in the right place.
2021-07-27 17:57:17 -07:00
vsoch
4208cf66be spack style: automatically bootstrap dependencies
This uses our bootstrapping logic to automatically install dependencies for
`spack style`. Users should no longer have to pre-install all of the tools
(`isort`, `mypy`, `black`, `flake8`). The command will do it for them.

- [x] add logic to bootstrap specs with specific version requirements in `spack style`
- [x] remove style tools from CI requirements (to ensure we test bootstrapping)
- [x] rework dependencies for `mypy` and `py-typed-ast`
      - `py-typed-ast` needs to be a link dependency
      - it needs to be at 1.4.1 or higher to work with python 3.9

Signed-off-by: vsoch <vsoch@users.noreply.github.com>
2021-07-24 07:07:35 -07:00
Massimiliano Culpo
a30e6c6158
Bump codecov/action to v2.0.2 (#24983)
* build(deps): bump codecov/codecov-action from 1 to 2.0.1

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 1 to 2.0.1.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/master/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v1...v2.0.1)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update arguments to codecov action

* Try to delete the symbolic link to root folder

Hopefully this should get rid of the ELOOP error

* Delete also for shell tests and MacOS

* Bump to v2.0.2

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-07-24 07:23:56 -04:00
Todd Gamblin
24c01d57cf
imports: sort imports everywhere in Spack (#24695)
* fix remaining flake8 errors

* imports: sort imports everywhere in Spack

We enabled import order checking in #23947, but fixing things manually drives
people crazy. This used `spack style --fix --all` from #24071 to automatically
sort everything in Spack so PR submitters won't have to deal with it.

This should go in after #24071, as it assumes we're using `isort`, not
`flake8-import-order` to order things. `isort` seems to be more flexible and
allows `llnl` mports to be in their own group before `spack` ones, so this
seems like a good switch.
2021-07-08 22:12:30 +00:00
Todd Gamblin
24a4d81097 style: clean up and restructure spack style command
This consolidates code across tools in `spack style` so that each
`run_<tool>` function can be called indirecty through a dictionary
of handlers, and os that checks like finding the executable for the
tool can be shared across commands.

- [x] rework `spack style` to use decorators to register tools
- [x] define tool order in one place in `spack style`
- [x] fix python 2/3 issues to Get `isort` checks working
- [x] make isort error regex more robust across versions
- [x] remove unused output option
- [x] change vestigial `TRAVIS_BRANCH` to `GITHUB_BASE_REF`
- [x] update completion
2021-07-07 17:27:31 -07:00
Greg Becker
7490d63c38
Separable module configuration -- without the bugs this time (#23703)
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment).

This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention.

As part of this change, the module roots configuration moved from the config section to inside each module configuration.

Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec.
2021-05-28 14:12:05 -07:00
Tim Haines
cad06a15e0
Dyninst: add elfutils versioning (#19648) 2021-04-27 07:15:37 -05:00
Tim Haines
c23ffd89ff
Dyninst: Add dependencies for v11.0.0 (#23121)
Also update the mpileaks unit test to avoid a conflict on CentOS 6
where Dyninst >=11.0.0 no longer builds due to a compiler version
conflict.
2021-04-26 13:53:53 -07:00
Massimiliano Culpo
629f94b4e1
CI: drastically reduce the number of tests for package only PRs (#22410)
PRs that change only package recipes will only run tests under "package_sanity.py" and without coverage. This should result in a huge drop the cpu-time spent in CI for most PRs.
2021-03-19 11:04:53 -07:00
Massimiliano Culpo
b304b4bdb0
Speed-up CI by reorganizing tests (#22247)
* unit tests: mark slow tests as "maybeslow"

This commit also removes the "network" marker and
marks every "network" test as "maybeslow". Tests
marked as db are maintained, but they're not slow
anymore.

* GA: require style tests to pass before running unit-tests

* GA: make MacOS unit tests fail fast

* GA: move all unit tests into the same workflow, run style tests as a prerequisite

All the unit tests have been moved into the same workflow so that a single
run of the dorny/paths-filter action can be used to ask for coverage based
on the files that have been changed in a PR. The basic idea is that for PRs
that introduce only changes to packages coverage is not necessary, this
resulting in a faster execution of the tests.

Also, for package only PRs slow unit tests are skipped.

Finally, MacOS and linux unit tests are now conditional on style tests passing
meaning that e.g. we won't waste a MacOS worker if we know that the PR has
flake8 issues.

* Addressed review comments

* Skipping slow tests on MacOS for package only recipes

* QA: make tests on changes correct before merging
2021-03-16 08:16:31 -07:00
Harmen Stoppels
0664b90751
Drop compiler variables from spack load (#21699)
Drops:

* C_INCLUDE_PATH
* CPLUS_INCLUDE_PATH
* LIBRARY_PATH
* INCLUDE

We already decided to use C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, INCLUDE over CPATH here:

https://github.com/spack/spack/pull/14749

However, none of these flags apply to Fortran on Linux. So for consistency it seems better to make the user use -I and -L flags by hand or through pkgconfig.
2021-02-23 13:35:19 +01:00
Todd Gamblin
a8ccb8e116 copyrights: update all files with license headers for 2021
- [x] add `concretize.lp`, `spack.yaml`, etc. to licensed files
- [x] update all licensed files to say 2013-2021 using
      `spack license update-copyright-year`
- [x] appease mypy with some additions to package.py that needed
      for oneapi.py
2021-01-02 12:12:00 -08:00