Commit Graph

256 Commits

Author SHA1 Message Date
Harmen Stoppels
8c29e90fa9
Build cache: make signed/unsigned a mirror property (#41507)
* Add `signed` property to mirror config

* make unsigned a tri-state: true/false overrides mirror config, none takes mirror config

* test commands

* Document this

* add a test
2023-12-11 15:14:59 -06:00
Harmen Stoppels
3cefd73fcc
spack buildcache check: use same interface as push (#41378) 2023-12-05 12:44:50 +01:00
Greg Becker
13abfb7013
spack deconcretize command (#38803)
We have two ways to concretize now:
* `spack concretize` concretizes only the root specs that are not concrete in the environment.
* `spack concretize -f` eliminates all cached concretization data and reconcretizes the *entire* environment.

This PR adds `spack deconcretize`, which eliminates cached concretization data for a spec.  This allows
users greater control over what is preserved from their `spack.lock` file and what is reused when not
using `spack concretize -f`.  If you want to update a spec installed in your environment, you can call
`spack deconcretize` on it, and that spec and any relevant dependents will be removed from the lock file.

`spack concretize` has two options:
* `--root`: limits deconcretized specs to *specific* roots in the environment. You can use this to
  deconcretize exactly one root in a `unify: false` environment.  i.e., if `foo` root is a dependent
  of `bar`, both roots, `spack deconcretize bar` will *not* deconcretize `foo`.
* `--all`: deconcretize *all* specs that match the input spec. By default `spack deconcretize`
  will complain about multiple matches, like `spack uninstall`.
2023-11-10 14:55:35 -08:00
Todd Gamblin
f0ced1af42
info: rework spack info command to display variants better (#40998)
This changes variant display to use a much more legible format, and to use screen space
much better (particularly on narrow terminals). It also adds color the variant display
to match other parts of `spack info`.

Descriptions and variant value lists that were frequently squished into a tiny column
before now have closer to the full terminal width.

This change also preserves any whitespace formatting present in `package.py`, so package
maintainers can make easer-to-read descriptions of variant values if they want. For
example, `gasnet` has had a nice description of the `conduits` variant for a while, but
it was wrapped and made illegible by `spack info`. That is now fixed and the original
newlines are kept.

Conditional variants are grouped by their when clauses by default, but if you do not
like the grouping, you can display all the variants in order with `--variants-by-name`.
I'm not sure when people will prefer this, but it makes it easier to tell that a
particular variant is/isn't there. I do think grouping by `when` is the better default.
2023-11-10 12:31:28 +01:00
Harmen Stoppels
4ce80b95f3
spack compiler find --[no]-mixed-toolchain (#40902)
Currently there's some hacky logic in the AppleClang compiler that makes
it also accept `gfortran` as a fortran compiler if `flang` is not found.

This is guarded by `if sys.platform` checks s.t. it only applies to
Darwin.

But on Linux the feature of detecting mixed toolchains is highly
requested too, cause it's rather annoying to run into a failed build of
`openblas` after dozens of minutes of compiling its dependencies, just
because clang doesn't have a fortran compiler.

In particular in CI where the system compilers may change during system
updates, it's typically impossible to fix compilers in a hand-written
compilers.yaml config file: the config will almost certainly be outdated
sooner or later, and maintaining one config file per target machine and
writing logic to select the correct config is rather undesirable too.

---

This PR introduces a flag `spack compiler find --mixed-toolchain` that
fills out missing `fc` and `f77` entries in `clang` / `apple-clang` by
picking the best matching `gcc`.

It is enabled by default on macOS, but not on Linux, matching current
behavior of `spack compiler find`.

The "best matching gcc" logic and compiler path updates are identical to
how compiler path dictionaries are currently flattened "horizontally"
(per compiler id). This just adds logic to do the same "vertically"
(across different compiler ids).

So, with this change on Ubuntu 22.04:

```
$ spack compiler find --mixed-toolchain
==> Added 6 new compilers to /home/harmen/.spack/linux/compilers.yaml
    gcc@13.1.0  gcc@12.3.0  gcc@11.4.0  gcc@10.5.0  clang@16.0.0  clang@15.0.7
==> Compilers are defined in the following files:
    /home/harmen/.spack/linux/compilers.yaml

```

you finally get:

```
compilers:
- compiler:
    spec: clang@=15.0.7
    paths:
      cc: /usr/bin/clang
      cxx: /usr/bin/clang++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    flags: {}
    operating_system: ubuntu23.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: clang@=16.0.0
    paths:
      cc: /usr/bin/clang-16
      cxx: /usr/bin/clang++-16
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    flags: {}
    operating_system: ubuntu23.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
```

The "best gcc" is automatically default system gcc, since it has no
suffixes / prefixes.
2023-11-06 15:17:31 -08:00
Todd Gamblin
f6b23b4653
bugfix: compress aliases for first command in completion (#40890)
This completes to `spack concretize`:

```
spack conc<tab>
```

but this still gets hung up on the difference between `concretize` and `concretise`:

```
spack -e . conc<tab>
```

We were checking `"$COMP_CWORD" = 1`, which tracks the word on the command line
including any flags and their args, but we should track `"$COMP_CWORD_NO_FLAGS" = 1` to
figure out if the arg we're completing is the first real command.
2023-11-05 10:15:37 +00:00
Harmen Stoppels
195f965076
OCI buildcache (#38358)
Credits to @ChristianKniep for advocating the idea of OCI image layers
being identical to spack buildcache tarballs.

With this you can configure an OCI registry as a buildcache:

```console 
$ spack mirror add my_registry oci://user/image # Dockerhub

$ spack mirror add my_registry oci://ghcr.io/haampie/spack-test # GHCR

$ spack mirror set --push --oci-username ... --oci-password ... my_registry  # set login credentials
```

which should result in this config:

```yaml
mirrors:
  my_registry:
    url: oci://ghcr.io/haampie/spack-test
    push:
      access_pair: [<username>, <password>]
```

It can be used like any other registry

```
spack buildcache push my_registry [specs...]
```

It will upload the Spack tarballs in parallel, as well as manifest + config
files s.t. the binaries are compatible with `docker pull` or `skopeo copy`.

In fact, a base image can be added to get a _runnable_ image:

```console
$ spack buildcache push --base-image ubuntu:23.04 my_registry python
Pushed ... as [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack

$ docker run --rm -it [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack
```

which should really be a game changer for sharing binaries.

Further, all content-addressable blobs that are downloaded and verified
will be cached in Spack's download cache. This should make repeated
`push` commands faster, as well as `push` followed by a separate
`update-index` command.

An end to end example of how to use this in Github Actions is here:

**https://github.com/haampie/spack-oci-buildcache-example**


TODO:

- [x] Generate environment modifications in config so PATH is set up
- [x] Enrich config with Spack's `spec` json (this is allowed in the OCI specification)
- [x] When ^ is done, add logic to create an index in say `<image>:index` by fetching all config files (using OCI distribution discovery API)
- [x] Add logic to use object storage in an OCI registry in `spack install`.
- [x] Make the user pick the base image for generated OCI images.
- [x] Update buildcache install logic to deal with absolute paths in tarballs
- [x] Merge with `spack buildcache` command
- [x] Merge #37441 (included here)
- [x] Merge #39077 (included here)
- [x] #39187 + #39285
- [x] #39341
- [x] Not a blocker: #35737 fixes correctness run env for the generated container images

NOTE:

1. `oci://` is unfortunately taken, so it's being abused in this PR to mean "oci type mirror". `skopeo` uses `docker://` which I'd like to avoid, given that classical docker v1 registries are not supported.
2. this is currently `https`-only, given that basic auth is used to login. I _could_ be convinced to allow http, but I'd prefer not to, given that for a `spack buildcache push` command multiple domains can be involved (auth server, source of base image, destination registry). Right now, no urllib http handler is added, so redirects to https and auth servers with http urls will simply result in a hard failure.

CAVEATS:

1. Signing is not implemented in this PR. `gpg --clearsign` is not the nicest solution, since (a) the spec.json is merged into the image config, which must be valid json, and (b) it would be better to sign the manifest (referencing both config/spec file and tarball) using more conventional image signing tools
2. `spack.binary_distribution.push` is not yet implemented for the OCI buildcache, only `spack buildcache push` is. This is because I'd like to always push images + deps to the registry, so that it's `docker pull`-able, whereas in `spack ci` we really wanna push an individual package without its deps to say `pr-xyz`, while its deps reside in some `develop` buildcache.
3. The `push -j ...` flag only works for OCI buildcache, not for others
2023-10-27 15:30:04 +02:00
Harmen Stoppels
1c8073c21f
spack checksum: show long flags in usage output (#40407) 2023-10-26 14:48:35 +02:00
Harmen Stoppels
bd165ebc4d
Support spack env activate --with-view <name> <env> (#40549)
Currently `spack env activate --with-view` exists, but is a no-op.

So, it is not too much of a breaking change to make this redundant flag
accept a value `spack env activate --with-view <name>` which activates
a particular view by name.

The view name is stored in `SPACK_ENV_VIEW`.

This also fixes an issue where deactivating a view that was activated
with `--without-view` possibly removes entries from PATH, since now we
keep track of whether the default view was "enabled" or not.
2023-10-17 15:40:48 +02:00
Massimiliano Culpo
210d221357
Test package detection in a systematic way (#18175)
This PR adds a new audit sub-command to check that detection of relevant packages
is performed correctly in a few scenarios mocking real use-cases. The data for each 
package being tested is in a YAML file called detection_test.yaml alongside the 
corresponding package.py file.

This is to allow encoding detection tests for compilers and other widely used tools, 
in preparation for compilers as dependencies.
2023-09-29 10:24:42 +02:00
Massimiliano Culpo
3b4ca0374e
Use process pool executors for web-crawling and retrieving archives (#39888)
Fix a race condition when searching urls, and updating a shared
set '_visited'.
2023-09-19 15:32:59 +02:00
Todd Gamblin
45d149c7d3
bugfix: only complete aliases for potential aliases (#39887)
Smart alias completion introduced in #39499 wasn't as smart as it needed to be, and
would complete any invalid command prefix and some env names with alias names.

- [x] don't complete aliases if there are no potential completions
      e.g., don't convert `spack isnotacommand` -> `spack concretize`

- [x] don't complete with an aliases if we're not looking at a top-level subcommand.
2023-09-08 12:51:26 +02:00
Massimiliano Culpo
39b9f214a8
Speed-up spack external find execution (#39843)
* Perform external spec detection with multiple workers

The logic to perform external spec detection has been refactored
into classes. These classes use the GoF "template" pattern to account
for the small differences between searching for "executables" and
for "libraries", while unifying the larger part of the algorithm.

A ProcessPoolExecutor is used to parallelize the work.

* Speed-up external find by tagging detectable packages automatically

Querying packages by tag is much faster than inspecting the repository,
since tags are cached. This commit adds a "detectable" tag to every
package that implements the detection protocol, and external detection
uses it to search for packages.

* Pass package names instead of package classes to workers

The slowest part of the search is importing the Python modules
associated with candidate packages. The import is done serially
before we distribute the work to the pool of executors.

This commit pushes the import of the Python module to the job
performed by the workers, and passes just the name of the packages
to the executors.

In this way imports can be done in parallel.

* Rework unit-tests for Windows

Some unit tests were doing a full e2e run of a command
just to check a input handling. Make the test more
focused by just stressing a specific function.

Mark as xfailed 2 tests on Windows, that will be fixed
by a PR in the queue. The tests are failing because we
monkeypatch internals in the parent process, but the
monkeypatching is not done in the "spawned" child
process.
2023-09-08 09:25:50 +02:00
John W. Parent
ca872f9c34
Windows: fix pwsh env activate/deactivate; load/unload (#39118)
These commands are currently broken on powershell (Windows) due to
improper use of the InvokeCommand commandlet and a lack of direct
support for the `--pwsh` argument in `spack load`, `spack unload`,
and `spack env deactivate`.
2023-09-01 11:36:27 -07:00
Todd Gamblin
6693dc5eb8 completion: make bash completion work in zsh
`compgen -W` does not behave the same way in zsh as it does in bash; it seems not to
actually generate the completions we want.

- [x] add a zsh equivalent and `_compgen_w` to abstract it away
- [x] use `_compgen_w` instead of `compgen -W`
2023-08-30 12:42:31 -07:00
Todd Gamblin
396f219011 completion: add alias handling
Bash completion is now smarter about handling aliases. In particular, if all completions
for some input command are aliased to the same thing, we'll just complete with that thing.

If you've already *typed* the full alias for a command, we'll complete the alias.

So, for example, here there's more than one real command involved, so all aliases are
shown:

```console
$ spack con
concretise    concretize    config        containerise  containerize
```

Here, there are two possibilities: `concretise` and `concretize`, but both map to
`concretize` so we just complete that:

```console
$ spack conc
concretize
```

And here, the user has already typed `concretis`, so we just go with it as there is only
one option:

```console
 spack concretis
concretise
```
2023-08-30 12:42:31 -07:00
Todd Gamblin
a3ecd7efed Add concretise and containerise aliases for our UK users
From a user:

> Aargh.
> ```
> ==> Error: concretise is not a recognized Spack command or extension command; check with `spack commands`.
> ```

To make things easier for our friends in the UK, this adds `concretise` and
`containerise` aliases for the `spack concretize` and `spack containerize` commands.

- [x] add aliases
- [x] update completions
2023-08-30 12:42:31 -07:00
Dom Heinzeller
31d5f56913
Add --fail-fast option for generating build caches (#38496)
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-08-23 11:20:33 +02:00
Adam J. Stewart
39d4c402d5
Fix suffix of tab completion scripts (#39154) 2023-08-09 08:28:55 -05:00
kwryankrattiger
0b4631a774
CI: Refactor ci reproducer (#37088)
* CI: Refactor ci reproducer

* Autostart container
* Reproducer paths match CI paths
* Generate start scripts for docker and reproducer

* CI: Add interactive and gpg options to reproduce-build

* Interactive will determine if the docker container persists
  after running reproduction.
* GPG path/url allow downloading GPG keys needed for binary
  cache download validation. This is important for running
  reproducer for protected CI jobs.

* Add exit_on_failure option to CI scripts

* CI: Add runtime option for reproducer
2023-08-02 09:51:12 -07:00
Wouter Deconinck
a14f4b5a02
feat: move -N/--namespace(s) to common args, allow in buildcache list (#36719)
`spack buildcache list` did not have a way to display the namespace of
packages in the buildcache. This PR adds that functionality.

For consistency's sake, it moves the `-N/--namespace` arg definition to
the `common/arguments.py` and modifies `find`, `solve`, `spec` to use
the common definition.

Previously, `find` was using `--namespace` (singular) to control whether
to display the namespace (it doesn't restrict the search to that
namespace). The other commands were using `--namespaces` (plural). For
backwards compatibility and for consistency with `--deps`, `--tags`,
etc, the plural `--namespaces` was chosen. The argument parser ensures
that `find --namespace` will continue to behave as before.

Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-08-02 11:16:14 +00:00
Alec Scott
d4f41b51f4
Add spack checksum --verify, fix --add (#38458)
* Add rewrite of spack checksum to include --verify and better add versions to package.py files
* Fix formatting and remove unused import
* Update checksum unit-tests to correctly test multiple versions and add to package
* Remove references to latest in stage.py
* Update bash-completion scripts to fix unit tests failures
* Fix docs generation
* Remove unused url_dict argument from methods
* Reduce chance of redundant remote_versions work
* Add print() before tty.die() to increase error readablity
* Update version regular expression to allow for multi-line versions
* Add a few unit tests to improve test coverage
* Update command completion
* Add type hints to added functions and fix a few py-lint suggestions
* Add @no_type_check to prevent mypy from failing on pkg.versions
* Add type hints to format.py and fix unit test
* Black format lib/spack/spack/package_base.py
* Attempt ignoring type errors
* Add optional dict type hint and declare versions in PackageBase
* Refactor util/format.py to allow for url_dict as an optional parameter
* Directly reference PackageBase class instead of using TypeVar
* Fix comment typo

---------

Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
2023-07-31 21:49:43 +00:00
Harmen Stoppels
033eb77aa9
spack buildcache push: improve argparse (#38876) 2023-07-13 16:01:09 +02:00
Harmen Stoppels
522d9e260b
mirrors: distinguish between source/binary mirror; simplify schema (#34523)
Allow the following formats:

```yaml
mirrors:
  name: <url>
```

```yaml
mirrors:
  name:
    url: s3://xyz
    access_pair: [x, y]
```

```yaml
mirrors:
  name:
    fetch: http://xyz
    push:
      url: s3://xyz
      access_pair: [x, y]
```

And reserve two new properties to indicate the mirror type (e.g.
mirror.spack.io is a source mirror, not a binary cache)

```yaml
mirrors:
  spack-public:
    source: true
    binary: false
    url: https://mirror.spack.io
```
2023-07-13 11:29:17 +00:00
Todd Gamblin
757f8ae59c
find: add --hashes shortcut for piping to other commands (#38663)
People frequently ask us how to pipe `spack find` output to other commands, and we tell
them to do things like this:

```console
$ spack find --format "/{hash}" | spack uninstall -ay
```

Sometimes users don't know about hash references and come up with potentially ambiguous
formulations like this:

```console
spack find --format {name}@{version}%{compiler} | spack uninstall -ay
```

Since this is a common enough thing to want to do, and to make it more obvious how, this
PR adds a `-H` / `--hashes` as a shortcut, so you can now just do:

```console
spack find -H | spack uninstall -ay
```
2023-07-10 09:43:37 +02:00
John W. Parent
78f33bc002
Windows: Add PowerShell env support (#37951)
PowerShell requires explicit shell and env support in Spack.
This is due to the distinct differences in shell interactions between
cmd and pwsh. Add a doskey in pwsh piping 'spack' commands to a
powershell script similar to the sh function 'spack'. Add
support for PowerShell-specific shell interactions from Spack
(set/unset shell variables).
2023-06-27 18:26:51 -07:00
Harmen Stoppels
da45073ef9
buildcache: remove deprecated api (#37246)
The API was deprecated in v0.20 and is slated for removal in v0.21
2023-06-12 14:33:26 +02:00
Greg Becker
3c672905d0
install status arg: update to default true and add inverse (#38252)
* install status arg: update to default true and add inverse

* update bash completion

* fix tests broken by output change
2023-06-09 10:38:05 -07:00
Tiziano Müller
0f84782fcc
Bugfix: cray manifest parsing regression (#37909)
fa7719a changed syntax for specifying exact versions, which are
required for some compiler specs (including those read as part
of parsing a Cray manifest). This fixes that and also makes a
couple other improvements to manifest parsing.

* Instantiate compiler specs with exact versions (fixes #37893)
* fix slingshot network detection (CPE 22.10+ has libcxi.so
  in /usr/lib64)
* "spack external find": add arg to ignore default dir for cray
  manifests
2023-05-30 18:03:44 -07:00
Massimiliano Culpo
5c7dda7e14
Allow using -j to control the parallelism of concretization (#37608)
fixes #29464

This PR allows to use
```
$ spack concretize -j X
```
to set a cap on the parallelism of concretization from the command line
2023-05-11 13:29:17 -07:00
G-Ragghianti
ebb40ee0d1
New option "--first" for "spack location" (#36283) 2023-05-10 12:26:29 +02:00
Harmen Stoppels
9ef062fcca
Add spack buildcache push (alias to buildcache create) (#34861)
`spack buildcache create` is a misnomer cause it's the only way to push to
an existing buildcache (and it in fact calls binary_distribution.push).

Also we have `spack buildcache update-index` but for create the flag is
`--rebuild-index`, which is confusing (and also... why "rebuild"
something if the command is "create" in the first place, that implies it
wasn't there to begin with).

So, after this PR, you can use either

```
spack buildcache create --rebuild-index
```

or

```
spack buildcache push --update-index
```

Also, alias `spack buildcache rebuild-index` to `spack buildcache
update-index`.
2023-05-05 19:54:26 +02:00
Greg Becker
c3593e5b48
Allow choosing the name of the packages subdirectory in repositories (#36643)
Co-authored-by: becker33 <becker33@users.noreply.github.com>
2023-05-04 23:36:21 +02:00
Michael Kuhn
5bae742826
concretizer: add mode to reuse dependencies only (#30990)
This adds a new mode for `concretizer:reuse` called `dependencies`,
which only reuses dependencies. Currently, `spack install foo` will
reuse older versions of `foo`, which might be surprising to users.
2023-03-14 09:22:20 +01:00
Alex Richert
ce8727cece
Add --exclude option to 'spack external find' (#35013)
* Add --exclude option to 'spack external find' to ignore user-specified external packages
* Update bash completion arg order for external find

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2023-01-21 15:43:20 -08:00
Harmen Stoppels
a357a39963
depfile: --make-target-prefix -> --make-prefix (#35009)
Since SPACK_PACKAGE_IDS is now also "namespaced" with <prefix>, it makes
more sense to call the flag `--make-prefix` and alias the old flag
`--make-target-prefix` to it.
2023-01-19 14:58:34 +01:00
Harmen Stoppels
fce95e2efb
license year bump (#34921)
* license bump year
* fix black issues of modified files
* mypy
* fix 2021 -> 2023
2023-01-18 14:30:17 -08:00
Harmen Stoppels
3489cc0a9b
Refer to mirrors by name, path, or url (#34891)
With this change we get the invariant that `mirror.fetch_url` and
`mirror.push_url` return valid URLs, even when the backing config
file is actually using (relative) paths with potentially `$spack` and
`$env` like variables.

Secondly it avoids expanding mirror path / URLs too early,
so if I say `spack mirror add name ./path`, it stays `./path` in my
config. When it's retrieved through MirrorCollection() we
exand it to say `file://<env dir>/path` if `./path` was set in an
environment scope.

Thirdly, the interface is simplified for the relevant buildcache
commands, so it's more like `git push`:

```
spack buildcache create [mirror] [specs...]
```

`mirror` is either a mirror name, a path, or a URL.

Resolving the relevant mirror goes as follows:
    
- If it contains either / or \ it is used as an anonymous mirror with
   path or url.
- Otherwise, it's interpreted as a named mirror, which must exist.

This helps to guard against typos, e.g. typing `my-mirror` when there
is no such named mirror now errors with:

```
$ spack -e . buildcache create my-mirror
==> Error: no mirror named "my-mirror". Did you mean ./my-mirror?
```

instead of creating a directory in the current working directory. I
think this is reasonable, as the alternative (requiring that a local dir
exists) feels a bit pendantic in the general case -- spack is happy to
create the build cache dir when needed, saving a `mkdir`.

The old (now deprecated) format will still be available in Spack 0.20,
but is scheduled to be removed in 0.21:

```
spack buildcache create (--directory | --mirror-url | --mirror-name) [specs...]
```

This PR also touches `tmp_scope` in tests, because it didn't really
work for me, since spack fixes the possible --scope values once and
for all across tests, so tests failed when run out of order.
2023-01-16 10:14:41 -08:00
Todd Gamblin
d4e714bb2e
spack list: add --count option (#34950)
Sometimes I just want to know how many packages of a certain type there are.

- [x] add `--count` option to `spack list` that output the number of packages that
      *would* be listed.

```console
> spack list --count
6864
> spack list --count py-
2040
> spack list --count r-
1162
```
2023-01-14 16:08:40 -08:00
Massimiliano Culpo
3d961b9a1f
spack graph: rework to use Jinja templates and builders (#34637)
`spack graph` has been reworked to use:

- Jinja templates
- builder objects to construct the template context when DOT graphs are requested. 

This allowed to add a new colored output for DOT graphs that highlights both
the dependency types and the nodes that are needed at runtime for a given spec.
2022-12-27 15:25:53 +01:00
Harmen Stoppels
e055dc0e64
Use file paths/urls correctly (#34452)
The main issue that's fixed is that Spack passes paths (as strings) to
functions that require urls. That wasn't an issue on unix, since there
you can simply concatenate `file://` and `path` and all is good, but on
Windows that gives invalid file urls. Also on Unix, Spack would not deal with uri encoding like x%20y for file paths. 

It also removes Spack's custom url.parse function, which had its own incorrect interpretation of file urls, taking file://x/y to mean the relative path x/y instead of hostname=x and path=/y. Also it automatically interpolated variables, which is surprising for a function that parses URLs.

Instead of all sorts of ad-hoc `if windows: fix_broken_file_url` this PR
adds two helper functions around Python's own path2url and reverse.

Also fixes a bug where some `spack buildcache` commands
used `-d` as a flag to mean `--mirror-url` requiring a URL, and others
`--directory`, requiring a path. It is now the latter consistently.
2022-12-13 23:44:13 +01:00
Todd Gamblin
d991ec90e3
new command: spack pkg grep to search package files (#34388)
It's very common for us to tell users to grep through the existing Spack packages to
find examples of what they want, and it's also very common for package developers to do
it. Now, searching packages is even easier.

`spack pkg grep` runs grep on all `package.py` files in repos known to Spack. It has no
special options other than the search string; all options passed to it are forwarded
along to `grep`.

```console
> spack pkg grep --help
usage: spack pkg grep [--help] ...

positional arguments:
  grep_args  arguments for grep

options:
  --help     show this help message and exit
```

```console
> spack pkg grep CMakePackage | head -3
/Users/gamblin2/src/spack/var/spack/repos/builtin/packages/3dtk/package.py:class _3dtk(CMakePackage):
/Users/gamblin2/src/spack/var/spack/repos/builtin/packages/abseil-cpp/package.py:class AbseilCpp(CMakePackage):
/Users/gamblin2/src/spack/var/spack/repos/builtin/packages/accfft/package.py:class Accfft(CMakePackage, CudaPackage):
```

```console
> spack pkg grep -Eho '(\S*)\(PythonPackage\)' | head -3
AwsParallelcluster(PythonPackage)
Awscli(PythonPackage)
Bueno(PythonPackage)
```

## Return Value

This retains the return value semantics of `grep`:
* 0  for found,
* 1  for not found
* >1 for error

## Choosing a `grep`

You can set the ``SPACK_GREP`` environment variable to choose the ``grep``
executable this command should use.
2022-12-09 10:07:54 -08:00
Massimiliano Culpo
4c05fe569c
Bootstrap most of Spack dependencies using environments (#34029)
This commit reworks the bootstrapping procedure to use Spack environments 
as much as possible.

The `spack.bootstrap` module has also been reorganized into a Python package. 
A distinction is made among "core" Spack dependencies (clingo, GnuPG, patchelf)
and other dependencies. For a number of reasons, explained in the `spack.bootstrap.core`
module docstring, "core" dependencies are bootstrapped with the current ad-hoc
method. 

All the other dependencies are instead bootstrapped using a Spack environment
that lives in a directory specific to the interpreter and the architecture being used.
2022-12-06 11:54:02 +01:00
Greg Becker
99044bedd7
patch command: add concretizer args (#34282)
* patch command: add concretizer args
* tab completion
2022-12-02 14:02:20 -08:00
Michael Kuhn
0fd3c9f451
cmd/checksum: allow adding new versions to package (#24532)
This adds super-lazy maintainer mode to `spack checksum`: Instead of
only printing the new checksums to the terminal, `-a` and
`--add-to-package` will add the new checksums to the `package.py` file
and open it in the editor afterwards for final checks.
2022-11-22 16:30:49 -08:00
Massimiliano Culpo
45b4cedb7e
spack find: remove deprecated "--bootstrap" option (#34015) 2022-11-19 16:09:34 +01:00
Massimiliano Culpo
da0a6280ac
Remove deprecated subcommands from "spack bootstrap" (#33964)
These commands are slated for removal in v0.20
2022-11-17 12:42:57 +01:00
Harmen Stoppels
0f54a63dfd
remove activate/deactivate support in favor of environments (#29317)
Environments and environment views have taken over the role of `spack activate/deactivate`, and we should deprecate these commands for several reasons:

- Global activation is a really poor idea:
   - Install prefixes should be immutable; since they can have multiple, unrelated dependents; see below
   - Added complexity elsewhere: verification of installations, tarballs for build caches, creation of environment views of packages with unrelated extensions "globally activated"... by removing the feature, it gets easier for people to contribute, and we'd end up with fewer bugs due to edge cases.
- Environment accomplish the same thing for non-global "activation" i.e. `spack view`, but better.

Also we write in the docs:

```
However, Spack global activations have two potential drawbacks:

#. Activated packages that involve compiled C extensions may still
   need their dependencies to be loaded manually.  For example,
   ``spack load openblas`` might be required to make ``py-numpy``
   work.

#. Global activations "break" a core feature of Spack, which is that
   multiple versions of a package can co-exist side-by-side.  For example,
   suppose you wish to run a Python package in two different
   environments but the same basic Python --- one with
   ``py-numpy@1.7`` and one with ``py-numpy@1.8``.  Spack extensions
   will not support this potential debugging use case.
```

Now that environments are established and views can take over the role of activation
non-destructively, we can remove global activation/deactivation.
2022-11-11 00:50:07 -08:00
Peter Scheibel
1a3415619e
"spack uninstall": don't modify env (#33711)
"spack install foo" no longer adds package "foo" to the environment
(i.e. to the list of root specs) by default: you must specify "--add".
Likewise "spack uninstall foo" no longer removes package "foo" from
the environment: you must specify --remove. Generally this means
that install/uninstall commands will no longer modify the users list
of root specs (which many users found problematic: they had to
deactivate an environment if they wanted to uninstall a spec without
changing their spack.yaml description).

In more detail: if you have environments e1 and e2, and specs [P, Q, R]
such that P depends on R, Q depends on R, [P, R] are in e1, and [Q, R]
are in e2:

* `spack uninstall --dependents --remove r` in e1: removes R from e1
  (but does not uninstall it) and uninstalls (and removes) P
* `spack uninstall -f --dependents r` in e1: will uninstall P, Q, and
   R (i.e. e2 will have dependent specs uninstalled as a side effect)
* `spack uninstall -f --dependents --remove r` in e1: this uninstalls
   P, Q, and R, and removes [P, R] from e1
* `spack uninstall -f --remove r` in e1: uninstalls R (so it is
  "missing" in both environments) and removes R from e1 (note that e1
  would still install R as a dependency of P, but it would no longer
  be listed as a root spec)
* `spack uninstall --dependents r` in e1: will fail because e2 needs R

Individual unit tests were created for each of these scenarios.
2022-11-08 03:24:51 +00:00
Massimiliano Culpo
7e645f54c5
Deprecate spack bootstrap trust/untrust (#33600)
* Deprecate spack bootstrap trust/untrust
* Update CI
* Update tests
2022-10-29 12:24:26 -07:00