Commit Graph

1072 Commits

Author SHA1 Message Date
Jordan Galby
6b0011c8f1
For "when:" and install_environment.json: Support fully qualified hostname (#45522) 2024-08-28 18:38:54 +02:00
psakievich
1b82779087
Add options for sparse checkout in GitFetcher (#45473)
* Add options for sparse checkout in GitFetcher

Newer versions of git have a beta feature called sparse checkout
that allow users to check out a portion of a large repo.

This feature will be ideal for monolithic repo projects that want to
model their infrastructure via spack.  This PR implements an addition
to the GitFetcher that allows users to add a `git_sparse_paths`
attribute to package classes or versions which will then use sparse
checkout on those directories/files for the package.

* Style

* Split git clone into multiple functions

* Add sparse-checkout impl

* Internalize src clone functions

* Docs

* Adding sparse clone test

* Add test for partial clone

* [@spackbot] updating style on behalf of psakievich

* Small fixes

* Restore default branch status

* Fix attributes for package

* Update lib/spack/docs/packaging_guide.rst

Co-authored-by: Matthew Mosby <44072882+mdmosby@users.noreply.github.com>

* Extend unit test to multiple git versions

* style

---------

Co-authored-by: psakievich <psakievich@users.noreply.github.com>
Co-authored-by: Matthew Mosby <44072882+mdmosby@users.noreply.github.com>
2024-08-15 05:28:34 +00:00
Harmen Stoppels
a66586d749
spack buildcache push: best effort (#45631)
"spack buildcache push" for partially installed environments pushes all it 
can by default, and only dumps errors towards the end.

If --fail-fast is provided, error out before pushing anything if any
of the packages is uninstalled

oci build caches using parallel push now use futures to ensure pushing
goes in best-effort style.
2024-08-13 08:12:48 +02:00
dependabot[bot]
4a8d09dcc1
build(deps): bump flake8 from 7.1.0 to 7.1.1 in /lib/spack/docs (#45588)
Bumps [flake8](https://github.com/pycqa/flake8) from 7.1.0 to 7.1.1.
- [Commits](https://github.com/pycqa/flake8/compare/7.1.0...7.1.1)

---
updated-dependencies:
- dependency-name: flake8
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-06 07:08:57 +02:00
dependabot[bot]
f3bd820374
build(deps): bump pytest from 8.3.1 to 8.3.2 in /lib/spack/docs (#45461)
Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.3.1 to 8.3.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/8.3.1...8.3.2)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-04 05:44:52 +00:00
dependabot[bot]
1090895e72
build(deps): bump black from 24.4.2 to 24.8.0 in /lib/spack/docs (#45563)
Bumps [black](https://github.com/psf/black) from 24.4.2 to 24.8.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/24.4.2...24.8.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-03 21:48:50 -05:00
dependabot[bot]
e983f4a858
build(deps): bump sphinx-design from 0.6.0 to 0.6.1 in /lib/spack/docs (#45562)
Bumps [sphinx-design](https://github.com/executablebooks/sphinx-design) from 0.6.0 to 0.6.1.
- [Release notes](https://github.com/executablebooks/sphinx-design/releases)
- [Changelog](https://github.com/executablebooks/sphinx-design/blob/main/CHANGELOG.md)
- [Commits](https://github.com/executablebooks/sphinx-design/compare/v0.6.0...v0.6.1)

---
updated-dependencies:
- dependency-name: sphinx-design
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-03 21:46:19 -05:00
dependabot[bot]
3ee6507dd6
build(deps): bump mypy from 1.11.0 to 1.11.1 in /lib/spack/docs (#45530)
Bumps [mypy](https://github.com/python/mypy) from 1.11.0 to 1.11.1.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v1.11...v1.11.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-02 17:01:25 -06:00
dependabot[bot]
6ea944bf17
build(deps): bump pytest from 8.2.2 to 8.3.1 in /lib/spack/docs (#45377)
Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.2.2 to 8.3.1.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/8.2.2...8.3.1)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-23 11:53:55 -04:00
dependabot[bot]
8c6177c47f
build(deps): bump sphinx from 7.4.6 to 7.4.7 in /lib/spack/docs (#45376)
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 7.4.6 to 7.4.7.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES.rst)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.4.6...v7.4.7)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-23 11:53:43 -04:00
dependabot[bot]
0d04223ccd
build(deps): bump mypy from 1.10.1 to 1.11.0 in /lib/spack/docs (#45337)
Bumps [mypy](https://github.com/python/mypy) from 1.10.1 to 1.11.0.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v1.10.1...v1.11)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-20 11:25:31 -05:00
Wouter Deconinck
24b73da9e6
docs: util/environment.py: use re.Pattern[str] instead of re (#45329)
* docs: util/environment.py: use `re.Pattern[str]` instead of `re`

* docs: sphinx==7.4.6
2024-07-19 20:03:07 +02:00
Tamara Dahlgren
737b70cbbf
Buildcache: remove deprecated --allow-root and preview subcommand (#45204) 2024-07-11 18:19:04 -07:00
Harmen Stoppels
3212cf86f4
environments.rst: go from simple to advanced (#45004)
* environments.rst: go from simple to advanced
* improvements
* notes about activation
2024-07-08 15:36:18 -07:00
Wouter Deconinck
04044a9744
containers: rm centos7 since EOL (#45049) 2024-07-04 22:22:23 +02:00
Massimiliano Culpo
6b85f6b405
ci: deprecate the --dependencies and --optimize option (#45005) 2024-07-02 22:06:52 +02:00
Tamara Dahlgren
61d6fc70e8
Docs: include cmake spec property for the command (#44956) 2024-07-01 10:28:26 -07:00
dependabot[bot]
d6419f32b8
build(deps): bump mypy from 1.10.0 to 1.10.1 in /lib/spack/docs (#44885)
Bumps [mypy](https://github.com/python/mypy) from 1.10.0 to 1.10.1.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v1.10.0...v1.10.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-27 12:40:14 +02:00
Miguel
60ed682577
Fix typo in docs (#44891)
Fixes on the rst formatting of the make_jobs
2024-06-27 12:32:16 +02:00
Wouter Deconinck
f916b50491
containers: centos:stream -> centos:stream9 (#44876) 2024-06-27 10:38:56 +02:00
Tamara Dahlgren
baf82c0245
Docs: Update stand-alone test information (#44755)
Update and slightly reorganize stand-alone test information to include new and improved examples and more links that can be used in PR feedback.
2024-06-25 13:33:41 -07:00
Miguel
536856874c
add documentation for make_jobs variable (#44838)
* add documentation for make_jobs variable

* apply suggested changes

* Update packaging_guide.rst

add suggestions to the documentation

* Update packaging_guide.rst

fix missing quotes in the documentation

* suggestions to packaging_guide.rst
2024-06-25 11:24:49 -06:00
dependabot[bot]
a82fb33b31
build(deps): bump flake8 from 7.0.0 to 7.1.0 in /lib/spack/docs (#44752) 2024-06-18 08:27:15 +02:00
dependabot[bot]
5f35a90529
build(deps): bump urllib3 from 2.2.1 to 2.2.2 in /lib/spack/docs (#44751) 2024-06-18 08:27:04 +02:00
dependabot[bot]
f80287166e
build(deps): bump sphinx-design from 0.5.0 to 0.6.0 in /lib/spack/docs (#44360)
Bumps [sphinx-design](https://github.com/executablebooks/sphinx-design) from 0.5.0 to 0.6.0.
- [Release notes](https://github.com/executablebooks/sphinx-design/releases)
- [Changelog](https://github.com/executablebooks/sphinx-design/blob/main/CHANGELOG.md)
- [Commits](https://github.com/executablebooks/sphinx-design/compare/v0.5.0...v0.6.0)

---
updated-dependencies:
- dependency-name: sphinx-design
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-06 14:34:59 +02:00
dependabot[bot]
a20d34b8aa
build(deps): bump pytest from 8.2.1 to 8.2.2 in /lib/spack/docs (#44553)
Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.2.1 to 8.2.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/8.2.1...8.2.2)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-06 08:38:54 +02:00
Massimiliano Culpo
f242e0fd0c
remove platform=cray (#43796)
Remove support for `cray` as a separate platform.

Any platform previously detected as `cray` is now detected as `linux`.

Users who still need platform=cray have to stick to Spack 0.22
2024-05-30 14:21:32 +02:00
mSamiolo
b9816a97fc
docs: update chain.rst to improve discussion of upstreams (#43918)
* Update chain.rst

* Update lib/spack/docs/chain.rst

Co-authored-by: Wouter Deconinck <wdconinc@gmail.com>

* Update lib/spack/docs/chain.rst

Co-authored-by: Wouter Deconinck <wdconinc@gmail.com>

* docs: rm leading spaces to avoid indent

---------

Co-authored-by: Wouter Deconinck <wdconinc@gmail.com>
2024-05-22 22:04:53 +00:00
dependabot[bot]
86fb547f7c
bump pytest from 8.2.0 to 8.2.1 --- (#44282)
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-21 13:42:22 +02:00
Todd Gamblin
77c83af17d
docs: remove warning about repositories and package extension (#44247)
Local package repositories are very well supported and we test them extensively, so this
warning from 8 years ago can be removed from the docs.
2024-05-17 22:03:57 +00:00
Chris Marsh
24edc72252
docs: show phase signature for builders (#44067) 2024-05-17 18:16:31 +02:00
Andrew-Dunning-NNL
a24ca50fed
Fix broken link in docs (#44217) 2024-05-17 12:58:11 +00:00
dependabot[bot]
8b4cbbe7b3
build(deps): bump pygments from 2.17.2 to 2.18.0 in /lib/spack/docs (#44044)
Bumps [pygments](https://github.com/pygments/pygments) from 2.17.2 to 2.18.0.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.17.2...2.18.0)

---
updated-dependencies:
- dependency-name: pygments
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-07 18:55:04 +02:00
Richarda Butler
be71f9fdc4
Include concrete environments with include_concrete (#33768)
Add the ability to include any number of (potentially nested) concrete environments, e.g.:

```yaml
   spack:
     specs: []
     concretizer:
         unify: true
     include_concrete:
     - /path/to/environment1
     - /path/to/environment2
```

or, from the CLI:

```console
   $ spack env create myenv
   $ spack -e myenv add python
   $ spack -e myenv concretize
   $ spack env create --include-concrete myenv included_env
```

The contents of included concrete environments' spack.lock files are
included in the environment's lock file at creation time. Any changes
to included concrete environments are only reflected after the environment
is re-concretized from the re-concretized included environments.

- [x] Concretize included envs
- [x] Save concrete specs in memory by hash
- [x] Add included envs to combined env's lock file
- [x] Add test
- [x] Update documentation

    Co-authored-by: Kayla Butler <<butler59@llnl.gov>
    Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.co
m>
    Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
    Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
2024-05-07 09:32:40 -07:00
psakievich
d22bdc1c4e
certs: fix interpolation and disallow relative paths (#44030) 2024-05-07 11:16:32 +02:00
Harmen Stoppels
125206d44d
python: always use a venv (#40773)
This commit adds a layer of indirection to improve build isolation with 
and without external Python, as well as usability of environment views.

It adds `python-venv` as a dependency to all packages that `extends("python")`, 
which has the following advantages:

1. Build isolation: only `PYTHONPATH` is considered in builds, not 
   user / system packages
2. Stable install layout: fixes the problem on Debian, RHEL and Fedora where 
   external / system python produces `bin/local` subdirs in Spack install prefixes. 
3. Environment views are Python virtual environments (and if you add 
   `py-pip` things like `pip list` work)

Views work whether they're symlink, hardlink or copy type.

This commit additionally makes `spec["python"].command` return 
`spec["python-venv"].command`. The rationale is that packages in repos we do 
not own do not pass the underlying python to the build system, which could still 
result in incorrectly computed install layouts.

Other attributes like `libs`, `headers` should be on `python` anyways and need no change.
2024-05-06 16:17:35 +02:00
Massimiliano Culpo
d654d6b1f4
Remove Fedora 37 and 38, Ubuntu 18 from CI (#44006) 2024-05-06 15:51:45 +02:00
Greg Becker
1f31c3374c
External package detection for compilers (#43464)
This creates shared infrastructure for compiler packages to implement the 
detailed search capabilities from the `spack compiler find` command for the 
`spack external find` command.

After this commit, `spack compiler find` can be replaced with 
`spack external find --tag compiler`, with the exception of mixed toolchains.
2024-05-06 10:33:33 +02:00
Harmen Stoppels
715214c1a1
spack env create <env>: dir if dir-like (#44024)
A named env cannot contain `.` and `/`.

So when a user runs `spack env create ./here` do not error but treat it
as `spack env create -d ./here`.

Also fix help string of `spack env create`, which seems to have been
copied from `activate` incorrectly.
2024-05-06 02:00:23 -06:00
dependabot[bot]
b471d62dbd
build(deps): bump black from 24.4.0 to 24.4.2 in /lib/spack/docs (#43878)
Bumps [black](https://github.com/psf/black) from 24.4.0 to 24.4.2.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/24.4.0...24.4.2)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-06 09:55:43 +02:00
dependabot[bot]
74fe498cb8
build(deps): bump mypy from 1.9.0 to 1.10.0 in /lib/spack/docs (#43834)
Bumps [mypy](https://github.com/python/mypy) from 1.9.0 to 1.10.0.
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/1.9.0...v1.10.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-04 15:48:49 -07:00
Todd Gamblin
cc85dc05b7
docs: re-enable google analytics (#43974)
We recently switched to using the new ReadTheDocs with "addons". That includes its own
analytics, which is nice, but we also want to continue using our GA4 analytics.

Adding GA4 is no longer supported by RTD, so we have to add it manually.

- [x] re-add the gtag to all pages, manually

Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
2024-05-02 21:56:19 -04:00
John W. Parent
7e8415a3a6
Windows: auto-add WGL/SDK as externals (#43752)
Adds a pre-concretization check for the Windows SDK and WGL (Windows
GL) packages as non-buildable externals.

This is a redo of https://github.com/spack/spack/pull/43459, but makes
sure to modify the configuration scope outside of the bootstrap scope:
whichever is highest-precedence in the user's environment at the time
the concretization runs, which should either be an env scope or the
~ scope.

Adds pytest fixture mocking the check for WGL and WSDK as if they were
present.
2024-05-02 01:05:03 -06:00
Massimiliano Culpo
4e876b4014
Allow more control over which specs are reused (#42782)
This PR gives users finer control over which specs are reused during concretization.

The value of the `concretizer:reuse` config option now can take an object with the following properties:
- `roots`: true if reusing roots, false if reusing just dependencies
- `exclude`: list of constraints used to select reusable specs 
- `include`: list of constraints used to select reusable specs 
- `from`: allows to select the sources of reused specs

### Examples

#### Reuse only specs compiled with GCC
```yaml
concretizer:
  reuse:
    roots: true
    include:
    - "%gcc"
```

#### `openmpi` must be used from externals, and it must be the only external used
```yaml
concretizer:
  reuse:
    roots: true
    from:
    - type: local
      exclude:
      - "openmpi"
    - type: buildcache
      exclude:
      - "openmpi"
    - type: external
      include:
      - "openmpi"
```
2024-05-01 23:05:26 -04:00
Wouter Deconinck
067155cff5
containers: add ubuntu 24.04 (#43881)
* containers: add ubuntu 24.04

* containers: use python3-boto3 pkg instead of pip install
2024-05-01 13:37:13 +02:00
dependabot[bot]
be48f762a9
build(deps): bump pytest from 8.1.1 to 8.2.0 in /lib/spack/docs (#43908)
Bumps [pytest](https://github.com/pytest-dev/pytest) from 8.1.1 to 8.2.0.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/8.1.1...8.2.0)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-01 13:29:22 +02:00
Jonathon Anderson
ae9f2d4d40
containers: Add Fedora 40, 39 (#43847) 2024-04-26 20:02:04 -05:00
Massimiliano Culpo
ac9012da0c
spack audit externals: allow selecting platforms and checking extra attributes (#43782) 2024-04-26 12:47:17 +02:00
John W. Parent
641ab95a31
Revert "Windows: add win-sdk/wgl externals during bootstrapping (#43459)" (#43819)
This reverts commit 9e2558bd56.
2024-04-24 18:24:28 +02:00
Harmen Stoppels
01f61a2eba
Remove import distro from packages and docs (#43772) 2024-04-23 12:47:33 +02:00