Commit Graph

773 Commits

Author SHA1 Message Date
Tiziano Müller
0f84782fcc
Bugfix: cray manifest parsing regression (#37909)
fa7719a changed syntax for specifying exact versions, which are
required for some compiler specs (including those read as part
of parsing a Cray manifest). This fixes that and also makes a
couple other improvements to manifest parsing.

* Instantiate compiler specs with exact versions (fixes #37893)
* fix slingshot network detection (CPE 22.10+ has libcxi.so
  in /usr/lib64)
* "spack external find": add arg to ignore default dir for cray
  manifests
2023-05-30 18:03:44 -07:00
eugeneswalker
1c7af83d32
update ci ml darwin keypath (#37927) 2023-05-25 16:27:56 -07:00
Adam J. Stewart
2f8cea2792
Add macOS ML CI stacks (#36586)
* Add macOS ML CI stacks

* torchmeta is no longer maintained and requires ancient PyTorch

* Add MXNet

* update darwin aarch64 stacks

* add darwin-aarch64 scoped config.yaml

* remove unnecessary cleanup job

* fix specifications

* fix labels

* fix labels

* fix indent on tags specification

* no tags for trigger jobs

* try overriding tags in stack spack.yaml

* do not use CI_STACK_CONFIG_SCOPES

* incorporate config:install_tree:root: overrides and compiler defs

* copy relevant ci-scoped config settings directly into stack spack.yaml

* remove build-job-remove

* spack ci generate: add debug flag

* include cdash config directly in stack spack.yaml

* customize build-job script section to avoid absolute paths

* add any-job specification

* tags: use aarch64-macos instead of aarch64

* generate tags: use aarch64-macos instead of aarch64

* do not add morepadding

* use shared mirror; comment out known failures

* remove any-job

* nproc || true

* comment out specs failing due to bazel from cache codesign issue

---------

Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
2023-05-25 01:12:54 -04:00
Stephen Sachs
2d77e44f6f
Pcluster local buildcache (#37852)
* [pcluster pipeline] Use local buildcache instead of upstream spack

Spack currently does not relocate compiler references from upstream spack
installations. When using a buildcache we don't need an upstream spack.

* gcc needs to be installed via postinstall to get correct deps

* quantum-espresso@gcc@12.3.0 returns ICE on neoverse_{n,v}1

* Force gitlab to pull the new container

* Revert "Force gitlab to pull the new container"

This reverts commit 3af5f4cd88.

Seems the gitlab version does not yet support "pull_policy" in .gitlab-ci.yml

* Gitlab keeps picking up wrong container. Renaming

* Update containers once more after failed build
2023-05-24 06:55:00 -07:00
eugeneswalker
c62080d498
e4s ci: add dealii (#32484) 2023-05-23 21:34:31 +02:00
Stephen Sachs
125c20bc06
Add aws-plcuster[-aarch64] stacks (#37627)
Add aws-plcuster[-aarch64] stacks.  These stacks build packages defined in
https://github.com/spack/spack-configs/tree/main/AWS/parallelcluster

They use a custom container from https://github.com/spack/gitlab-runners which
includes necessary ParallelCluster software to link and build as well as an
upstream spack installation with current GCC and dependencies.

Intel and ARM software is installed and used during the build stage but removed
from the buildcache before the signing stage.

Files `configs/linux/{arch}/ci.yaml` select the necessary providers in order to
build for specific architectures (icelake, skylake, neoverse_{n,v}1).
2023-05-17 16:21:10 -06:00
kwryankrattiger
4627438373
CI: Expand E4S ROCm stack to include missing DaV packages (#36843)
* CI: Expand E4S ROCm stack to include missing DaV packages

Ascent: Fixup for VTK-m with Kokkos backend

* DaV SDK: Removed duplicated openmp variant for ascent

* Drop visit and add conflict for Kokkos

* E4S: Drop ascent from CUDA builds
2023-05-16 09:34:52 -05:00
Peter Scheibel
7bc5b26c52
Requirements and preferences should not define (non-git) versions (#37687)
Ensure that requirements `packages:*:require:@x` and preferences `packages:*:version:[x]`
fail concretization when no version defined in the package satisfies `x`. This always holds
except for git versions -- they are defined on the fly.
2023-05-16 15:45:11 +02:00
Scott Wittenburg
a184bfc1a6
gitlab ci: reduce job name length of build_systems pipeline (#37686) 2023-05-16 00:26:37 +02:00
Harmen Stoppels
6a47339bf8
oneapi: before script load modules (#37678) 2023-05-15 18:39:58 +02:00
eugeneswalker
91c5b4aeb0
e4s ci stacks: add: hdf5-vol-{log,cache} (#37651) 2023-05-13 04:54:44 +00:00
Scott Wittenburg
c08be95d5e
gitlab ci: release fixes and improvements (#37601)
* gitlab ci: release fixes and improvements

  - use rules to reduce boilerplate in .gitlab-ci.yml
  - support copy-only pipeline jobs
  - make pipelines for release branches rebuild everything
  - make pipelines for protected tags copy-only

* gitlab ci: remove url changes used in testing

* gitlab ci: tag mirrors need public key

Make sure that mirrors associated with release branches and tags
contain the public key needed to verify the signed binaries.  This
also ensures that when stack-specific mirror contents are copied
to the root, the root mirror has the public key as well.

* review: be more specific about tags, curl flags

* Make the check in ci.yaml consistent with the .gitlab-ci.yml

---------

Co-authored-by: Ryan Krattiger <ryan.krattiger@kitware.com>
2023-05-12 15:22:42 -05:00
eugeneswalker
47ec6a6ae5
e4s ci: trilinos +rocm: enable belos to fix build failure (#37617) 2023-05-11 14:02:20 -07:00
Massimiliano Culpo
5c7dda7e14
Allow using -j to control the parallelism of concretization (#37608)
fixes #29464

This PR allows to use
```
$ spack concretize -j X
```
to set a cap on the parallelism of concretization from the command line
2023-05-11 13:29:17 -07:00
Harmen Stoppels
2c17c4e632
ci: remove --mirror-url flag (#37457)
The flags --mirror-name / --mirror-url / --directory were deprecated in 
favor of just passing a positional name, url or directory, and letting spack
figure it out.

---------

Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
2023-05-10 16:34:29 -06:00
G-Ragghianti
ebb40ee0d1
New option "--first" for "spack location" (#36283) 2023-05-10 12:26:29 +02:00
eugeneswalker
3446feff70
use latest trilinos for +cuda variants (#37164) 2023-05-08 12:29:54 -07:00
Harmen Stoppels
e17d6d5eee
gitlab ci: bump tutorial image (#37544) 2023-05-07 16:24:33 +02:00
Harmen Stoppels
9ef062fcca
Add spack buildcache push (alias to buildcache create) (#34861)
`spack buildcache create` is a misnomer cause it's the only way to push to
an existing buildcache (and it in fact calls binary_distribution.push).

Also we have `spack buildcache update-index` but for create the flag is
`--rebuild-index`, which is confusing (and also... why "rebuild"
something if the command is "create" in the first place, that implies it
wasn't there to begin with).

So, after this PR, you can use either

```
spack buildcache create --rebuild-index
```

or

```
spack buildcache push --update-index
```

Also, alias `spack buildcache rebuild-index` to `spack buildcache
update-index`.
2023-05-05 19:54:26 +02:00
Harmen Stoppels
ddea33bdc0
Update tutorial pipeline to Ubuntu 22.04 (#35451) 2023-05-05 17:52:07 +02:00
Harmen Stoppels
803425780e
ci: stop downloading recent gmake (#37458) 2023-05-05 17:09:56 +02:00
Greg Becker
c3593e5b48
Allow choosing the name of the packages subdirectory in repositories (#36643)
Co-authored-by: becker33 <becker33@users.noreply.github.com>
2023-05-04 23:36:21 +02:00
Jonathon Anderson
cba8d1253d
Add container images supporting RHEL alternatives (#36713)
Add container support for AlmaLinux, Fedora 37 and 38 and Rocky Linux
2023-04-28 12:28:33 +02:00
eugeneswalker
75bf6d665c
e4s ci: add xyce (#36841)
* e4s ci: add xyce

* relax trilinos contraints for xyce

* also relax trilinos constraint for e4s-power stack

* allow trilinos~shylu for xyce
2023-04-24 16:32:01 -07:00
eugeneswalker
c76023d7ac
e4s ci: add mgard (#36584) 2023-04-24 23:35:40 +02:00
Adam J. Stewart
b5d66e0144
MXNet: add v1.9 (#36590) 2023-04-24 11:13:09 -05:00
John W. Parent
5c742d4f2b
Hraden Spack's powershell interface (#37079)
Paths with spaces are an issue on Windows and our current powershell
scripts are not sufficiently hardended against their use.
This PR removes promlematic commandlets that do not work well with paths
with spaces and adds escape quotes in other areas where this could be an
issue.
2023-04-21 08:58:37 -05:00
Adam J. Stewart
05215e016b
ML CI: fix mirror name (#37007) 2023-04-20 16:45:15 -07:00
kwryankrattiger
9745865250
DaV SDK: Enable ParaView raytracing with in SDK (#36844)
* DaV SDK: Enable ParaView raytracing with in SDK

* CI: Drop swr testing from Data Vis SDK

* ISPC: extend LLVM requirement to main

* DaV SDK: Disallow concretizing develop unifyfs

No longer needed after mochi-margo patch
2023-04-18 13:39:47 -05:00
Adam J. Stewart
036695ac94
CI: update Linux images in ML pipelines (#36766)
Add missing openssl/curl/pkgconfig deps to py-tokenizers

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2023-04-17 17:42:03 +02:00
Scott Wittenburg
bfa94c5781
gitlab ci: Better tagging of "service" jobs (#36846)
- Tag non-rebuild jobs to target a cheaper (and more highly available)
subset of runners.

- Add missing resource requests to these jobs as well.
2023-04-14 09:03:12 -06:00
eugeneswalker
a9db5620f5
remove x86_64_v3 tags (#36828) 2023-04-13 15:29:15 -05:00
Massimiliano Culpo
d918ae0bde
containerize: strip binaries in a less aggressive way (#36683) 2023-04-13 17:09:34 +02:00
kwryankrattiger
b940468890
CI: Update Data Vis SDK image (#36761)
Supersedes #34224
2023-04-13 09:54:42 -05:00
eugeneswalker
a88fdb216f
ci: gpu-tests stack: swap x86_64-{cuda,rocm} for x86_64 (#36759) 2023-04-10 17:01:18 -07:00
kwryankrattiger
b2310f9e64
Ci backwards compat (#36045)
* CI: Fixup docs for bootstrap.

* CI: Add compatibility shim

* Add an update method for CI

Update requires manually renaming section to `ci`. After
this patch, updating and using the deprecated `gitlab-ci` section
should be possible.

* Fix typos in generate warnings

* Fixup CI schema validation

* Add unit tests for legacy CI

* Add deprecated CI stack for continuous testing

* Allow updating gitlab-ci section directly with env update

* Make warning give good advice for updating gitlab-ci

* Fix typo in CI name

* Remove white space

* Remove unneeded component of deprected-ci
2023-04-10 16:46:45 -05:00
kwryankrattiger
4ace1e660a
Ecp hdf5 vol (#35195)
* ECP-SDK: enable hdf5 VOL adapters
- When +hdf5, enable VOL adapters suitable for the SDK.
- Each VOL package must prepend to the HDF5_PLUGIN_PATH.
- hdf5: 1.13.3 will break existing VOL packages, constrain
  VOLs related to SDK and add note to keep 1.13.2 available.
- hdf5-vol-async:
    - Do not set HDF5_VOL_CONNECTOR, consumers must opt-in.
    - Enforce DAG constraints on MPI to require threaded version.
    - Depend on an explicit version of argbots to relax
      concretization issues in other spack environments.
- paraview: fix compiler flag usage for the 110 ABI (followup to #33617).
* ECP Data and ViS: Add constraits for HDF5 VOLS
* CI: HDF5 1.14 builds without VisIt
* hdf5-vol-async: Update docs string

---------

Co-authored-by: Stephen McDowell <stephen.mcdowell@kitware.com>
2023-04-06 15:21:11 -07:00
Massimiliano Culpo
f91968cf6f
Improve Dockerfile recipe generation (#35187)
- Update default image to Ubuntu 22.04 (previously was still Ubuntu 18.04)
- Optionally use depfiles to install the environment within the container
- Allow extending Dockerfile Jinja2 template
- Allow extending Singularity definition file Jinja2 template
- Deprecate previous options to add extra instructions
2023-04-03 21:05:19 +02:00
Scott Wittenburg
08426ec492
gitlab ci: request more memory for publish job (#36560) 2023-03-31 00:19:59 +02:00
Harmen Stoppels
dba57ff113
ci: require x86_64_v3 everywhere (#36158) 2023-03-29 15:58:48 +02:00
eugeneswalker
c88b30b426
e4s power ci: ecp-data-vis-sdk: disable visit due to build issues (#36475) 2023-03-28 06:02:47 -04:00
Xavier Delaruelle
ea60220a84
modules: enhance help message (#36410)
Update tcl and lmod modulefile template to provide more information on
help message (name, version and target) like done on whatis for lmod
modulefiles.
2023-03-27 10:48:25 +02:00
Xavier Delaruelle
c4923fe3b3
modules: add support for append_flags/remove_flags (#36402)
Adapt tcl and lmod modulefile templates to generate append-path or
remove-path commands in modulefile when respectively append_flags or
remove_flags commands are defined in package for run environment.

Fixes #10299.
2023-03-24 15:38:24 -04:00
Xavier Delaruelle
906151075d
modules tcl: simplify env modification block in template (#36334)
Simplify environment modification block in modulefile Tcl template by
always setting a path delimiter to the prepend-path, append-path and
remove-path commands.

Remove --delim option to the setenv command as this command does not
allow such option.

Update test_prepend_path_separator test to explicitly check the 6
path-like commands that should be present in generated modulefile.
2023-03-24 10:28:10 +01:00
Harmen Stoppels
88d78025a6
spack install: simplify behavior when inside environments (#35206)
Example one:

```
spack install --add x y z
```

is equivalent to

```
spack add x y z
spack concretize
spack install --only-concrete
```

where `--only-concrete` installs without modifying spack.yaml/spack.lock

Example two:

```
spack install
```

concretizes current spack.yaml if outdated and installs all specs.

Example three:

```
spack install x y z
```

concretizes current spack.yaml if outdated and installs *only* concrete
specs in the environment that match abstract specs `x`, `y`, or `z`.
2023-03-20 13:51:30 +01:00
Xavier Delaruelle
41d7fe0a50
modules tcl: fix autoload mechanism in template (#36237)
Adapt tcl modulefile template to call "module load" on autoload
dependency without testing if this dependency is already loaded or not.

The is-loaded test is not necessary, as module commands know how to cope
with an already loaded module. With environment-modules 4.2+ (released
in 2018) it is also important to have this "module load" command even if
dependency is already loaded in order to record that the modulefile
declares such dependency. This is important if you want to keep a
consistent environment when a dependent module is unloaded.

The "Autoloading" verbose message is also removed as recent module
commands will report such information to the user (depending on the
verbosity configured for the module command).

Such change has been test successfully with Modules 3.2 (EL7), 4.5 (EL8)
and 5.2 (latest) and also with Lmod 7 and 8 (as it is mentionned in
Spack docs that Lmod can be used along with tcl modules). Dependencies
are correctly loaded or unloaded, whether they are loaded/unloaded or
not.

This change fixes Tcl quoting issue introduced in #32853.

Fixes #19155.
2023-03-20 09:23:40 +01:00
Xavier Delaruelle
df97827a7b
Fix case spelling for Lmod and Tcl (#36215) 2023-03-19 01:42:50 +00:00
Adam J. Stewart
a84fb716a0
Update the PyTorch ecosystem (#36132)
* py-pytorch-lightning: add v2.0.0

* py-lightning-utilities: add v0.8.0

* Update all PyTorch packages

* Open-CE does not yet have patches for PyTorch 2 on ppc64le
2023-03-17 10:13:44 +01:00
kwryankrattiger
bb73dfc02e
Hotfix: CI: Add CI target for gpu-test stack (#36136) 2023-03-15 21:13:02 -04:00
Harmen Stoppels
1691b7caac
Fix typo affecting Gitlab CI (#36103)
Introduced in #35944
2023-03-14 14:18:05 +01:00