Commit Graph

343 Commits

Author SHA1 Message Date
eugeneswalker
3446feff70
use latest trilinos for +cuda variants (#37164) 2023-05-08 12:29:54 -07:00
Harmen Stoppels
e17d6d5eee
gitlab ci: bump tutorial image (#37544) 2023-05-07 16:24:33 +02:00
Harmen Stoppels
ddea33bdc0
Update tutorial pipeline to Ubuntu 22.04 (#35451) 2023-05-05 17:52:07 +02:00
Harmen Stoppels
803425780e
ci: stop downloading recent gmake (#37458) 2023-05-05 17:09:56 +02:00
eugeneswalker
75bf6d665c
e4s ci: add xyce (#36841)
* e4s ci: add xyce

* relax trilinos contraints for xyce

* also relax trilinos constraint for e4s-power stack

* allow trilinos~shylu for xyce
2023-04-24 16:32:01 -07:00
eugeneswalker
c76023d7ac
e4s ci: add mgard (#36584) 2023-04-24 23:35:40 +02:00
Adam J. Stewart
b5d66e0144
MXNet: add v1.9 (#36590) 2023-04-24 11:13:09 -05:00
Adam J. Stewart
05215e016b
ML CI: fix mirror name (#37007) 2023-04-20 16:45:15 -07:00
kwryankrattiger
9745865250
DaV SDK: Enable ParaView raytracing with in SDK (#36844)
* DaV SDK: Enable ParaView raytracing with in SDK

* CI: Drop swr testing from Data Vis SDK

* ISPC: extend LLVM requirement to main

* DaV SDK: Disallow concretizing develop unifyfs

No longer needed after mochi-margo patch
2023-04-18 13:39:47 -05:00
Adam J. Stewart
036695ac94
CI: update Linux images in ML pipelines (#36766)
Add missing openssl/curl/pkgconfig deps to py-tokenizers

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2023-04-17 17:42:03 +02:00
Scott Wittenburg
bfa94c5781
gitlab ci: Better tagging of "service" jobs (#36846)
- Tag non-rebuild jobs to target a cheaper (and more highly available)
subset of runners.

- Add missing resource requests to these jobs as well.
2023-04-14 09:03:12 -06:00
eugeneswalker
a9db5620f5
remove x86_64_v3 tags (#36828) 2023-04-13 15:29:15 -05:00
kwryankrattiger
b940468890
CI: Update Data Vis SDK image (#36761)
Supersedes #34224
2023-04-13 09:54:42 -05:00
eugeneswalker
a88fdb216f
ci: gpu-tests stack: swap x86_64-{cuda,rocm} for x86_64 (#36759) 2023-04-10 17:01:18 -07:00
kwryankrattiger
b2310f9e64
Ci backwards compat (#36045)
* CI: Fixup docs for bootstrap.

* CI: Add compatibility shim

* Add an update method for CI

Update requires manually renaming section to `ci`. After
this patch, updating and using the deprecated `gitlab-ci` section
should be possible.

* Fix typos in generate warnings

* Fixup CI schema validation

* Add unit tests for legacy CI

* Add deprecated CI stack for continuous testing

* Allow updating gitlab-ci section directly with env update

* Make warning give good advice for updating gitlab-ci

* Fix typo in CI name

* Remove white space

* Remove unneeded component of deprected-ci
2023-04-10 16:46:45 -05:00
kwryankrattiger
4ace1e660a
Ecp hdf5 vol (#35195)
* ECP-SDK: enable hdf5 VOL adapters
- When +hdf5, enable VOL adapters suitable for the SDK.
- Each VOL package must prepend to the HDF5_PLUGIN_PATH.
- hdf5: 1.13.3 will break existing VOL packages, constrain
  VOLs related to SDK and add note to keep 1.13.2 available.
- hdf5-vol-async:
    - Do not set HDF5_VOL_CONNECTOR, consumers must opt-in.
    - Enforce DAG constraints on MPI to require threaded version.
    - Depend on an explicit version of argbots to relax
      concretization issues in other spack environments.
- paraview: fix compiler flag usage for the 110 ABI (followup to #33617).
* ECP Data and ViS: Add constraits for HDF5 VOLS
* CI: HDF5 1.14 builds without VisIt
* hdf5-vol-async: Update docs string

---------

Co-authored-by: Stephen McDowell <stephen.mcdowell@kitware.com>
2023-04-06 15:21:11 -07:00
Scott Wittenburg
08426ec492
gitlab ci: request more memory for publish job (#36560) 2023-03-31 00:19:59 +02:00
Harmen Stoppels
dba57ff113
ci: require x86_64_v3 everywhere (#36158) 2023-03-29 15:58:48 +02:00
eugeneswalker
c88b30b426
e4s power ci: ecp-data-vis-sdk: disable visit due to build issues (#36475) 2023-03-28 06:02:47 -04:00
Adam J. Stewart
a84fb716a0
Update the PyTorch ecosystem (#36132)
* py-pytorch-lightning: add v2.0.0

* py-lightning-utilities: add v0.8.0

* Update all PyTorch packages

* Open-CE does not yet have patches for PyTorch 2 on ppc64le
2023-03-17 10:13:44 +01:00
kwryankrattiger
bb73dfc02e
Hotfix: CI: Add CI target for gpu-test stack (#36136) 2023-03-15 21:13:02 -04:00
Harmen Stoppels
1691b7caac
Fix typo affecting Gitlab CI (#36103)
Introduced in #35944
2023-03-14 14:18:05 +01:00
Harmen Stoppels
2107b6bf00
Set build_jobs dynamically in CI to avoid oversubscription (#35996)
Co-authored-by: Zack Galbreath <zack.galbreath@kitware.com>
Co-authored-by: Ryan Krattiger <ryan.krattiger@kitware.com>
2023-03-13 08:29:58 +01:00
Harmen Stoppels
844701b974
get --dev and drop set -x (#36010) 2023-03-10 22:59:57 -08:00
Harmen Stoppels
08d7f47278
curl flag is not universally supported (#36009) 2023-03-10 16:20:32 -08:00
Harmen Stoppels
26fd1ac5b0
hotfix: fix double double quotes (#36005) 2023-03-10 13:41:01 -08:00
Zack Galbreath
e1301df60c
ci: version bump for ghcr.io/spack/e4s-amazonlinux-2 (#35976)
* ci: version bump for ghcr.io/spack/e4s-amazonlinux-2

This new image comes with GnuPG v2.4.0

* py-cython: upperbounds for Python versions

* fix py-gevent nonsense

---------

Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-03-10 13:32:11 -08:00
kwryankrattiger
f3595da600
CI boilerplate reduction (#34272)
* CI configuration boilerplate reduction and refactor

Configuration:
- New notation for list concatenation (prepend/append)
- New notation for string concatenation (prepend/append)
- Break out configuration files for: ci.yaml, cdash.yaml, view.yaml
- Spack CI section refactored to improve self-consistency and
composability
  - Scripts are now lists of lists and/or lists of strings
  - Job attributes are now listed under precedence ordered list that are
  composed/merged using Spack config merge rules.
  - "service-jobs" are identified explicitly rather than as a batch

CI:
- Consolidate common, platform, and architecture configurations for all CI stacks into composable configuration files
- Make padding consistent across all stacks (256)
- Merge all package -> runner mappings to be consistent across all
stacks

Unit Test:
- Refactor CI module unit-tests for refactor configuration

Docs:
- Add docs for new notations in configuration.rst
- Rewrite docs on CI pipelines to be consistent with refactored CI
workflow

* Script verbose environ, dev bootstrap

* Port #35409
2023-03-10 12:25:35 -07:00
Scott Wittenburg
16c67ff9b4
ci: Increase the amount of pruning possible for PR pipelines (#35944)
By setting the traversal depth to 1, only specs matching the changed
package and direct dependents of those (and of course all dependencies
of that set) are removed from pruning candidacy.
2023-03-10 11:19:52 -08:00
kwryankrattiger
895886959f
CI: Fix timeout for VisIt and ParaView (#35787)
ref. #35400
2023-03-03 09:13:14 +01:00
Massimiliano Culpo
07e251c887
Remove handling of deprecated target names (graviton) (#35537)
* Update target names for Gitlab pipelines

* Remove handling of deprecated names for graviton
2023-03-01 11:03:12 +01:00
eugeneswalker
34441c9eaa
ci: gpu test stack: turn off cuda@11.7 builds (#35467) 2023-02-13 21:54:01 -08:00
eugeneswalker
f7de22eb14
gpu test stack: add test w/ ^cuda@12 builds on A100 w/ newer driver (#35375)
* gpu test stack: test cuda@12 builds on A100 w/ newer driver

* get gpu info via nvidia-smi;

* kokkos+cuda^cuda@12 has genuine failure
2023-02-09 12:18:03 -07:00
eugeneswalker
f6fa64f979
ci: add minimal gpu testing stack (#35251)
* ci: add minimal gpu testing stack

* kokkos +cuda requires +wrapper...

* require pass

* add raja+cuda
2023-02-03 18:33:34 -08:00
eugeneswalker
99f3716346
e4s: add and prefer boost +python +filesystem +iostreams +system (#35180) 2023-01-30 07:37:35 -08:00
eugeneswalker
48a63719b2
e4s oneapi: use oneapi@2023.0.0 (#34634) 2023-01-29 13:22:14 -08:00
eugeneswalker
9f5d9266e6
e4s: add power stack (#32710)
* e4s: restore builds builds

* gitlab ci: allow UO to build protected binaries for signing

* use newer image; comment out failing builds

* gitlab-ci: Some tweaks for e4s power builds

- fix tags (no longer require generate jobs to run on aws)
- fix resource requests for generation jobs resource requests
- remove SPACK_SIGNING_KEY from protected power build jobs
- update UO signing key path
- change the CDash build group to reflect stack name
- retry pipeline generation jobs *always*

* correct double packages: section

* gitlab-ci:script: modernize

* remove new gnu make, not for ppc64le

---------

Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
2023-01-29 10:53:02 -06:00
eugeneswalker
a82fe2010f
e4s: add cusz +cuda (#35004) 2023-01-23 09:15:39 -08:00
Zack Galbreath
91fd6d5a84
Allocate more memory for generate jobs in all stacks (#35040)
* Allocate more memory for generate jobs in all stacks
* Add a 60 minute timeout on generate jobs
2023-01-20 11:06:49 -07:00
eugeneswalker
6713f7feda
e4s: ecp-data-vis-sdk +rocm: remove redundant amdgpu_target specification (#35022) 2023-01-20 00:18:41 +00:00
eugeneswalker
fed4eb2526
e4s ci: clean up unnecessary constraints (#35006) 2023-01-18 21:23:15 -08:00
eugeneswalker
1bf1c7de76
e4s: add sz3 (#35007) 2023-01-18 22:21:30 -06:00
eugeneswalker
ae5408de9a
e4s ci: add veloc (#35003) 2023-01-18 16:48:23 -08:00
eugeneswalker
8eb803ab4c
e4s: add py-h5py ~mpi variant (#34998) 2023-01-18 19:28:47 +00:00
eugeneswalker
886ea7c490
e4s: add py-h5py (#34914) 2023-01-16 15:52:24 +00:00
eugeneswalker
b9a16a599a
e4s ci: py-libensemble: activate variants (#34915) 2023-01-16 10:57:28 +01:00
kwryankrattiger
229ee7cad9
Paraview rocm (#34790)
* paraview: add `rocm` variant

This conflicts with CUDA and requires at least ParaView 5.11.0. More
dependencies are also needed.

* E4S: Add ParaView for ROCm and CUDA stacks

* DAV SDK: Update ParaView version and GPU variants

* Verify using hipcc vs amdclang++ for newer hip

Co-authored-by: Ben Boeckel <ben.boeckel@kitware.com>
2023-01-13 12:41:57 -06:00
Mike VanDenburgh
ea6608edde
Bump memory requests for e4s/ml generate jobs (#34648) 2023-01-11 10:41:42 -07:00
eugeneswalker
9d70f6e6c7
protected runner jobs at uo: trust e4s-uo signing key (#34828) 2023-01-10 18:18:38 -07:00
Scott Wittenburg
8f6f77ac9f
gitlab ci: fix tags for pipeline generation jobs (#34789)
Gitlab does not merge lists when a job extends two other definitions
that include the same list (e.g. tags).  Also, it merges dictionaries
as long as the keys are distinct, but just takes the last mentioned
value when there are key collisions.

This change makes sure that when different tags are needed by a
pipeline, the ones we want are actually provided.  It also changes
the example stack to better follow this pattern so we do not lead
developers astray in the future.
2023-01-10 10:06:26 -07:00
Wileam Y. Phan
0f7f600d1f
Add CUDA 12.0 (#34664) 2023-01-10 11:51:23 +01:00
eugeneswalker
9283a94ee4
e4s ci: use 2023-01-01 runner image (#34827) 2023-01-10 10:47:28 +00:00
Harmen Stoppels
b9f48da560
e4s: move default values for rocm/cuda arch into packags:all:variants (#34772) 2023-01-04 16:09:14 +01:00
Adam J. Stewart
eb67497020
ML CI: Linux x86_64 (#34299)
* ML CI: Linux x86_64

* Update comments

* Rename again

* Rename comments

* Update to match other arches

* No compiler

* Compiler was wrong anyway

* Faster TF
2022-12-22 11:31:40 -06:00
eugeneswalker
7e836b925d
e4s: disable mac stack due to binary relocation issue#32571 (#34560) 2022-12-17 10:53:15 +00:00
Zack Galbreath
aa8e1ba606
gitlab ci: more resources for slow builds (#34505) 2022-12-15 14:35:54 -07:00
Scott Wittenburg
675afd884d
gitlab ci: more resources for paraview and py-torch (#34412) 2022-12-09 11:58:37 -07:00
Andrew W Elble
588d2e295f
py-alphafold: update to 2.2.4, update dependencies (#33876)
* py-alphafold: update to 2.2.4, update dependencies

* style
2022-12-02 22:12:06 +00:00
eugeneswalker
f452741e3d
e4s ci: use 2022-12-01 runner images (#34212) 2022-11-30 09:52:30 -08:00
eugeneswalker
99b68e646d
e4s ci: hpx: set max_cpu_count=512 (#33977) 2022-11-30 09:16:57 -08:00
kwryankrattiger
33422acef0
CI: Update Data and Vis SDK Stack (#34009)
* CI: Update Data and Vis SDK Stack

* Update image to match target deployments (E4S)
* Enable all packages
* Test supported variants of ParaView and VisIt

* Sensei: Update Python hint for newer cmake

* Sensei: add Python3 hint
2022-11-29 14:49:55 -07:00
eugeneswalker
587488882a
e4s ci: add hdf5-vol-async; remove expired comments (#34110) 2022-11-28 19:35:42 +00:00
kwryankrattiger
5d2c9636ff
E4S: Conservatively add ecp-data-vis-sdk (#33621)
* E4S: Conservatively add ecp-data-vis-sdk

* Remove ascent from CUDA SDK stack to stop hanging on Dray

* Adios2: Newer FindPython uses Python_EXECUTABLE
2022-11-23 11:01:30 -08:00
Harmen Stoppels
90fb16033e
gitlab: report load in generate job (#33888) 2022-11-15 13:21:21 +01:00
Harmen Stoppels
f332ac6d21
More jobs in Gitlab CI (#33688)
Use at most 32 jobs when available.
2022-11-08 12:53:11 +01:00
Adam J. Stewart
0f32f7d0e9
py-transformers: add v4.24.0 (#33716)
* py-transformers: add v4.24.0

* Internet access still required
2022-11-05 12:38:49 -05:00
Massimiliano Culpo
ace8c74e20
Revert "gitlab: when_possible -> false (#33443)" (#33552)
This reverts commit b1559cc831.
2022-10-31 13:31:57 -07:00
eugeneswalker
999c460b1e
e4s ci: add mfem +rocm (#31604) 2022-10-29 23:32:46 +00:00
Harmen Stoppels
fa0432c521
gitlab ci: patched make 4.3.0 (#33583) 2022-10-28 14:38:57 +02:00
Harmen Stoppels
df1d233573
Don't fail over cpuinfo (#33546) 2022-10-27 11:14:09 +02:00
Harmen Stoppels
34f9394732
gitlab ci: show build machine info (#33523) 2022-10-26 20:31:16 +02:00
Massimiliano Culpo
30c9ff50dd
Allow for packages with multiple build-systems (#30738)
This commit extends the DSL that can be used in packages
to allow declaring that a package uses different build-systems
under different conditions.

It requires each spec to have a `build_system` single valued
variant. The variant can be used in many context to query, manipulate
or select the build system associated with a concrete spec.

The knowledge to build a package has been moved out of the
PackageBase hierarchy, into a new Builder hierarchy. Customization
of the default behavior for a given builder can be obtained by
coding a new derived builder in package.py.

The "run_after" and "run_before" decorators are now applied to
methods on the builder. They can also incorporate a "when="
argument to specify that a method is run only when certain
conditions apply.

For packages that do not define their own builder, forwarding logic
is added between the builder and package (methods not found in one
will be retrieved from the other); this PR is expected to be fully
backwards compatible with unmodified packages that use a single
build system.
2022-10-26 20:17:32 +02:00
Harmen Stoppels
a2520e80c0
gitlab ci: install binary deps faster (#33248)
* Fast Gitlab CI job setup, and better legibility

* Use a non-broken, recent GNU Make
2022-10-26 09:19:24 +02:00
Harmen Stoppels
d361378553
Improve legibility of Gitlab CI (#33482)
Use --backtrace in ci instead of --debug to reduce verbosity
and don't show log on error, since log is already printed
2022-10-25 12:21:34 +02:00
eugeneswalker
baa21d664b
e4s ci: use an appropriate name for cdash build group name (#33494) 2022-10-24 16:15:01 -07:00
eugeneswalker
a51bd80a5e
e4s ci: add chai +rocm (#32506) 2022-10-24 18:16:56 +02:00
Zack Galbreath
6d3869a7d3
Remove x86_64_v4 target from AHUG and ISC stacks (#33464) 2022-10-22 08:53:07 +00:00
Harmen Stoppels
669bbe1e66
stop building binaries for the 1% (#33463) 2022-10-22 02:01:50 -06:00
Scott Wittenburg
27921c38ce
gitlab: Retry protected publish jobs in certain cases (#32496)
When we lose a running pod (possibly loss of spot instance) or encounter
some other infrastructure-related failure of this job, we need to retry
it.  This retries the job the maximum number of times in those cases.
2022-10-21 10:35:20 -06:00
Harmen Stoppels
b1559cc831
gitlab: when_possible -> false (#33443)
`reuse` and `when_possible` concretization broke the invariant that
`spec[pkg_name]` has unique keys. This invariant is relied on in tons of
places, such as when setting up the build environment.

When using `when_possible` concretization, one may end up with two or
more `perl`s or `python`s among the transitive deps of a spec, because
concretization does not consider build-only deps of reusable specs.

Until the code base is fixed not to rely on this broken property of
`__getitem__`, we should disable reuse in CI.
2022-10-21 14:42:06 +02:00
eugeneswalker
1e4732d5fe
e4s ci: add raja +rocm (#32505) 2022-10-20 14:26:18 -07:00
eugeneswalker
8aac0d09d4
e4s ci: add umpire +rocm (#32504) 2022-10-19 17:05:45 -06:00
eugeneswalker
dd003f66a8
e4s ci stack: add trilinos +rocm (#31601) 2022-10-17 21:04:31 +00:00
Jonathon Anderson
10491e98a8
CI: allow multiple matches to combine tags (#32290)
Currently "spack ci generate" chooses the first matching entry in
gitlab-ci:mappings to fill attributes for a generated build-job,
requiring that the entire configuration matrix is listed out
explicitly. This unfortunately causes significant problems in
environments with large configuration spaces, for example the
environment in #31598 (spack.yaml) supports 5 operating systems,
3 architectures and 130 packages with explicit size requirements,
resulting in 1300 lines of configuration YAML.

This patch adds a configuraiton option to the gitlab-ci schema called
"match_behavior"; when it is set to "merge", all matching entries
are applied in order to the final build-job, allowing a few entries
to cover an entire matrix of configurations.

The default for "match_behavior" is "first", which behaves as before
this commit (only the runner attributes of the first match are used).

In addition, match entries may now include a "remove-attributes"
configuration, which allows matches to remove tags that have been
aggregated by prior matches. This only makes sense to use with
"match_behavior:merge". You can combine "runner-attributes" with
"remove-attributes" to effectively override prior tags.
2022-10-15 17:29:53 +00:00
eugeneswalker
a587bff119
e4s ci: add cabana +rocm (#33177) 2022-10-11 00:37:55 -06:00
eugeneswalker
dc141f5ad6
e4s: add hypre +rocm (#32148) 2022-10-10 16:09:52 -06:00
Adam J. Stewart
831d7979ca
ML CPU pipeline: test py-torch-nvidia-apex (#33158) 2022-10-10 12:18:05 -06:00
Adam J. Stewart
01ede3c595
Add CI stack for ML packages (#31592)
Basic stack of ML packages we would like to test and generate binaries for in CI. 

Spack now has a large CI framework in GitLab for PR testing and public binary generation.
We should take advantage of this to test and distribute optimized binaries for popular ML
frameworks.

This is a pretty extensive initial set, including CPU, ROCm, and CUDA versions of a core
`x96_64_v4` stack.

### Core ML frameworks

These are all popular core ML frameworks already available in Spack.

- [x] PyTorch
- [x] TensorFlow
- [x] Scikit-learn
- [x] MXNet
- [x] CNTK
- [x] Caffe
- [x] Chainer
- [x] XGBoost
- [x] Theano

### ML extensions

These are domain libraries and wrappers that build on top of core ML libraries

- [x] Keras
- [x] TensorBoard
- [x] torchvision
- [x] torchtext
- [x] torchaudio
- [x] TorchGeo
- [x] PyTorch Lightning
- [x] torchmetrics
- [x] GPyTorch
- [x] Horovod

### ML-adjacent libraries

These are libraries that aren't specific to ML but are still core libraries used in ML pipelines

- [x] numpy
- [x] scipy
- [x] pandas
- [x] ONNX
- [x] bazel

Co-authored-by: Jonathon Anderson <17242663+blue42u@users.noreply.github.com>
2022-10-09 15:39:47 -07:00
eugeneswalker
dfbeaff8ae
remove outdated comments (#33123) 2022-10-07 19:47:57 -07:00
eugeneswalker
01e7b89b53
e4s ci: add variorum (#33113) 2022-10-07 14:52:30 -07:00
eugeneswalker
041c1486f8
e4s ci: add h5bench (#33114) 2022-10-07 14:52:08 -07:00
eugeneswalker
35bc158387
e4s ci: add quantum-espresso (#33075) 2022-10-07 08:38:10 -07:00
eugeneswalker
a56cd8ffb6
e4s ci: add charliecloud (#32990) 2022-10-06 12:22:23 +02:00
eugeneswalker
1fe8387d23
e4s ci: add gotcha cpu (#32989) 2022-10-05 11:43:13 -07:00
eugeneswalker
cefc1dc808
e4s ci: add caliper +cuda (#32992) 2022-10-05 10:25:18 -07:00
eugeneswalker
af02dcbd2e
e4s ci: enable caliper cpu (#32988) 2022-10-05 10:25:04 -07:00
eugeneswalker
82cfa68e69
e4s: add omega-h +cuda (#32156) 2022-10-05 09:36:09 -07:00
eugeneswalker
27074d5bec
e4s ci: add unifyfs (#32991) 2022-10-05 09:28:19 -07:00
eugeneswalker
b57d24a5e1
e4s: add vtk-m +rocm (#32151) 2022-10-04 12:28:24 -07:00
eugeneswalker
87b014ed13
e4s: use ubuntu 20.04 image and %gcc@9.4.0 (#32795) 2022-09-27 16:25:56 -06:00
Massimiliano Culpo
ecce657509 Remove "tut" from the "build-systems" pipeline
PR #32615 deprecated Python versions up to 3.6.X. Since
the "build-systems" pipeline requires Python 3.6.15 to
build "tut", it will fail on the first rebuild that
involves Python.

The "tut" package is meant to perform an end-to-end
test of the "Waf" build-system, which is scarcely
used. The fix therefore is just to remove it from
the pipeline.
2022-09-26 14:47:13 +02:00
Harmen Stoppels
2fa9aff206
gitlab: cuda not compatible amazon linux 2 (#32678)
amazon linux 2 ships a glibc that is too old to work with cuda toolkit
for aarch64.

For example:

`libcurand.so.10.2.10.50` requires the symbol `logf@@GLIBC_2.27`, but
glibc is at 2.26.

So, these specs are removed.
2022-09-23 11:01:59 +02:00
Zack Galbreath
4ebdc5643e
Revert "e4s ci: restore power builds (#32296)" (#32633)
This reverts commit 0d18c32bca.
2022-09-14 03:26:26 +00:00
eugeneswalker
bf84cdfcbe
e4s: add py-torch +cuda (#32601) 2022-09-12 09:40:49 +00:00
eugeneswalker
0d18c32bca
e4s ci: restore power builds (#32296) 2022-09-08 21:00:58 -07:00
Scott Wittenburg
6239198d65
Fix cause of checksum failures in public binary mirror (#32407)
Move the copying of the buildcache to a root job that runs after all the child
pipelines have finished, so that the operation can be coordinated across all
child pipelines to remove the possibility of race conditions during potentially
simlutandous copies. This lets us ensure the .spec.json.sig and .spack files
for any spec in the root mirror always come from the same child pipeline
mirror (though which pipeline is arbitrary).  It also allows us to avoid copying
of duplicates, which we now do.
2022-09-01 15:29:44 -06:00
Robert Cohn
c56abe4247
oneapi e4s: use require: to force gcc build for some packages (#32309) 2022-08-24 16:22:02 -04:00
Tamara Dahlgren
3c3b18d858
spack ci: add support for running stand-alone tests (#27877)
This support requires adding the '--tests' option to 'spack ci rebuild'.
Packages whose stand-alone tests are broken (in the CI environment) can
be configured in gitlab-ci to be skipped by adding them to
broken-tests-packages.

Highlights include:
- Restructured 'spack ci' help to provide better subcommand summaries;
- Ensured only one InstallError (i.e., installer's) rather than allowing
  build_environment to have its own; and
- Refactored CI and CDash reporting to keep CDash-related properties and
  behavior in a separate class.

This allows stand-alone tests from `spack ci` to run when the `--tests`
option is used.  With `--tests`, stand-alone tests are run **after** a
**successful** (re)build of the package.  Test results are collected
and report(able) using CDash.

This PR adds the following features:
- Adds `-t` and `--tests` to `spack ci rebuild` to run stand-alone tests;
- Adds `--fail-fast` to stop stand-alone tests after the first failure;
- Ensures a *single* `InstallError` across packages
    (i.e., removes second class from build environment);
- Captures skipping tests for externals and uninstalled packages
    (for CDash reporting);
- Copies test logs and outputs to the CI artifacts directory to facilitate
    debugging;
- Parses stand-alone test results to report outputs from each `run_test` as
    separate test parts (CDash reporting);
- Logs a test completion message to allow capture of timing of the last
    `run_test` part;
- Adds the runner description to the CDash site to better distinguish entries
    in CDash tables;
- Adds `gitlab-ci` `broken-tests-packages` to CI configuration to skip
    stand-alone testing for packages with known issues;
- Changes `spack ci --help` so description of each subcommand is a single line;
- Changes `spack ci <subcommand> --help` to provide the full description of
    each command (versus no description); and
- Ensures `junit` test log file ends in an `.xml` extension (versus default where
    it does not).

Tasks:

- [x] Include the equivalent of the architecture information, or at least the host target, in the CDash output
- [x] Upload stand-alone test results files as  `test` artifacts
- [x] Confirm tests are run in GitLab
- [x] Ensure CDash results are uploaded as artifacts
- [x] Resolve issues with CDash build-and test results appearing on same row of the table 
- [x] Add unit tests  as needed
- [x] Investigate why some (dependency) packages don't have test results (e.g., related from other pipelines)
- [x] Ensure proper parsing and reporting of skipped tests (as `not run`) .. post- #28701 merge
- [x] Restore the proper CDash URLand or mirror ONCE out-of-band testing completed
2022-08-23 00:52:48 -07:00
Robert Cohn
00b244853e
enable packages that use numpy for oneapi e4s stack (#32293)
* enable geopm oneapi

* enable packages that depend on py-numpy

* disable nan warnings

* undo geopm enable

* undo geopm, it has multiple compile issues
2022-08-22 07:02:22 -07:00
eugeneswalker
e1bce8c577
e4s %oneapi: add aml +ze (level zero) (#32161) 2022-08-19 08:25:15 -07:00
eugeneswalker
a42a874faa
e4s oneapi: compiler environment: prepend lib dir to LD_LIBRARY_PATH (#32227) 2022-08-19 06:30:18 -04:00
eugeneswalker
022d59f4a5
e4s: add dealii +cuda (#32159) 2022-08-18 22:41:57 -06:00
eugeneswalker
b2d1d07d2e
e4s %oneapi: add amrex +sycl (#32162) 2022-08-18 18:16:55 -07:00
Robert Cohn
1cdc50e3f6
ascent: fix oneapi build (#32239) 2022-08-18 17:15:39 +00:00
Massimiliano Culpo
7f81a0a25e
CI: move nightly macOS builds from GA to pipelines (#32231)
Modifications:
- [x] Delete the `macOS builds nightly` workflow
- [x] Removed shield in README.md
- [x] Moved corresponding specs to `e4s-macos`
2022-08-18 09:13:37 +00:00
eugeneswalker
d8689b5038
e4s mac ci: try lambda, the new mac studio runner (#32169) 2022-08-17 21:24:47 +00:00
eugeneswalker
5fe8ccca3a e4s oneapi stack: remove notes for now-fixed builds (#31839) 2022-08-17 13:57:13 +02:00
eugeneswalker
0d7a6ec410
e4s: add flecsi +cuda (#32158) 2022-08-16 13:00:03 +00:00
eugeneswalker
fe77cc9b27
e4s: add ascent +cuda (#32153) 2022-08-16 10:47:17 +00:00
eugeneswalker
9bc537bb89
e4s: add amrex +cuda (#32152) 2022-08-16 09:08:26 +00:00
eugeneswalker
2c42ab45a0
e4s: add papi +rocm (#32150) 2022-08-16 08:26:06 +00:00
eugeneswalker
934db62541
e4s: add caliper +rocm (#32149) 2022-08-16 07:52:09 +00:00
eugeneswalker
43c0dac6f6
unifyfs %oneapi: add CFLAGS=-Wno-unused-function to build environment (#32107) 2022-08-15 10:43:39 -07:00
eugeneswalker
5bffd371e7
e4s ci: enable trilinos@13.4.0 +cuda build, issues resolved (#32124) 2022-08-15 09:06:03 -07:00
eugeneswalker
ad51e7d9f0
e4s ci: enable hpctoolkit +rocm (#32126) 2022-08-15 09:05:34 -07:00
eugeneswalker
4ec31003aa
e4s oneapi ci: enable hpx due to merged PR #32097 (#32117) 2022-08-12 23:41:53 -06:00
eugeneswalker
ffb63ca933
e4s oneapi: build tau %oneapi ^binutils%gcc (#32106) 2022-08-12 20:01:04 -07:00
Scott Wittenburg
cb19ca2471
gitlab ci: Make sure PR pipelines can push to buildcache (#32014)
On PR pipelines we need to override the buildcache destination to
point to the "spack-binaries-prs" bucket, otherwise, those pipelines
try to push to the default mirror in a bucket for which they don't
have write permission.
2022-08-12 16:53:59 +00:00
renjithravindrankannath
08e75f7a3e
rocm: add v5.2 for the entire stack (#31591) 2022-08-12 09:20:47 +02:00
eugeneswalker
9215d59599
axom@0.6.1 %oneapi: patch examples (#31991)
* axom@0.6.1 %oneapi: patch examples

* uncomment axom%oneapi ci build now that it is fixed
2022-08-08 19:39:32 +00:00
eugeneswalker
e5eabe0cf6
e4s oneapi ci: unify when possible (#31965) 2022-08-08 09:22:32 +02:00
eugeneswalker
2b04c4f840
sundials@6.1.0:6.2.0 +rocm: patch nvector to use pic (#31910)
* sundials@6.1.0:6.2.0 +rocm: patch nvector to use pic

* e4s ci: add sundials +rocm
2022-08-03 16:01:59 -06:00
eugeneswalker
0462818c2e
e4s oneapi ci: build vtk-m ~openmp due to issue #31830 (#31840) 2022-08-02 14:45:58 -07:00
eugeneswalker
b3f221185e
butterflypack %oneapi: patch CMakeLists to solve issue #31818 (#31848)
* butterflypack %oneapi: patch CMakeLists to solve issue #31818

* uncomment builds affected by failing butterflypack

Co-authored-by: e <e>
2022-08-02 14:02:00 -06:00
eugeneswalker
0fc045392b
e4s oneapi stack: remove notes for now-fixed builds (#31839) 2022-08-01 12:03:39 -07:00
eugeneswalker
d360551c16
e4s oneapi ci: uncomment pdt (#31803)
* e4s oneapi ci: uncomment pdt

* load oneapi compiler module before executing `spack ci rebuild`
2022-07-29 20:57:25 +00:00
eugeneswalker
9e683126f0
e4s ci: add tasmanian +rocm (#31606) 2022-07-29 11:43:34 -07:00
eugeneswalker
70c849f76b
e4s oneapi ci: uncomment parallel-netcdf (#31804) 2022-07-29 17:39:51 +00:00
eugeneswalker
ec8c8e8aa8
e4s ci: add oneapi stack (#31781)
* e4s ci: add oneapi stack

* shorten padded_length to 256

* comment out pdt and add failure note
2022-07-29 03:32:41 +00:00
eugeneswalker
5f845f3f31
e4s ci: add slate +rocm (#31602) 2022-07-27 15:36:59 -07:00
eugeneswalker
b38a814dc9
e4s ci stack: add spec: hdf5-vol-async (#31747)
* e4s ci stack: add spec: hdf5-vol-async

* hdf5-vol-async: add e4s tag
2022-07-26 16:21:56 -06:00
eugeneswalker
b065d69136
e4s ci: add ginkgo +rocm (#31603) 2022-07-18 14:19:31 +02:00
eugeneswalker
3cb6fd140c
update e4s to reflect june status (#31032) 2022-07-14 22:05:57 +00:00
Scott Wittenburg
861edcf996
gitlab: Fix mirror url to match stack name (#31399)
We adopted the convention of putting binaries for each stack into
a dedicated mirror named after the directory in which the stack
(spack.yaml file) resides.  This fixes the mirror url of the
radiuss-aws-aarch64 stack to follow that convention.
2022-06-30 15:38:58 -07:00
David Beckingsale
170c605d6a
AWS RADIUSS builds (#31114)
* Add AWS RADIUSS builds

* Correct variable naming

* Add two more MFEM specs

* Updates to MFEM spec suggested by @v-dobrev

* Simplify MFEM specs
2022-06-28 12:15:53 -07:00
Zack Galbreath
47ac710796
CPU & memory requests for jobs that generate GitLab CI pipelines (#30940)
gitlab ci: make sure pipeline generation isn't resource starved
2022-06-01 09:43:23 -06:00
Evan Bollig
a94438b1f5
Added AWS-AHUG alinux2 pipeline (#24601)
Add spack stacks targeted at Spack + AWS + ARM HPC User Group hackathon.  Includes
a list of miniapps and full-apps that are ready to run on both x86_64 and aarch64.

Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
2022-05-30 10:26:39 -06:00
Evan Bollig
98860c6a5f
Alinux isc buildcache (#30462)
Add two new stacks targeted at x86_64 and arm, representing an initial list of packages 
used by current and planned AWS Workshops, and built in conjunction with the ISC22
announcement of the spack public binary cache.

Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
2022-05-28 11:32:53 -06:00
Scott Wittenburg
85e13260cf
ci: Support secure binary signing on protected pipelines (#30753)
This PR supports the creation of securely signed binaries built from spack
develop as well as release branches and tags. Specifically:

- remove internal pr mirror url generation logic in favor of buildcache destination
on command line
    - with a single mirror url specified in the spack.yaml, this makes it clearer where 
    binaries from various pipelines are pushed
- designate some tags as reserved: ['public', 'protected', 'notary']
    - these tags are stripped from all jobs by default and provisioned internally
    based on pipeline type
- update gitlab ci yaml to include pipelines on more protected branches than just
develop (so include releases and tags)
    - binaries from all protected pipelines are pushed into mirrors including the
    branch name so releases, tags, and develop binaries are kept separate
- update rebuild jobs running on protected pipelines to run on special runners
provisioned with an intermediate signing key
    - protected rebuild jobs no longer use "SPACK_SIGNING_KEY" env var to
    obtain signing key (in fact, final signing key is nowhere available to rebuild jobs)
    - these intermediate signatures are verified at the end of each pipeline by a new
    signing job to ensure binaries were produced by a protected pipeline
- optionallly schedule a signing/notary job at the end of the pipeline to sign all
packges in the mirror
    - add signing-job-attributes to gitlab-ci section of spack environment to allow
    configuration
    - signing job runs on special runner (separate from protected rebuild runners)
    provisioned with public intermediate key and secret signing key
2022-05-26 08:31:22 -06:00
Massimiliano Culpo
f2a81af70e
Best effort co-concretization (iterative algorithm) (#28941)
Currently, environments can either be concretized fully together or fully separately. This works well for users who create environments for interoperable software and can use `concretizer:unify:true`. It does not allow environments with conflicting software to be concretized for maximal interoperability.

The primary use-case for this is facilities providing system software. Facilities provide multiple MPI implementations, but all software built against a given MPI ought to be interoperable.

This PR adds a concretization option `concretizer:unify:when_possible`. When this option is used, Spack will concretize specs in the environment separately, but will optimize for minimal differences in overlapping packages.

* Add a level of indirection to root specs

This commit introduce the "literal" atom, which comes with
a few different "arities". The unary "literal" contains an
integer that id the ID of a spec literal. Other "literals"
contain information on the requests made by literal ID. For
instance zlib@1.2.11 generates the following facts:

literal(0,"root","zlib").
literal(0,"node","zlib").
literal(0,"node_version_satisfies","zlib","1.2.11").

This should help with solving large environments "together
where possible" since later literals can be now solved
together in batches.

* Add a mechanism to relax the number of literals being solved

* Modify spack solve to display the new criteria

Since the new criteria is above all the build criteria,
we need to modify the way we display the output.

Originally done by Greg in #27964 and cherry-picked
to this branch by the co-author of the commit.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>

* Inject reusable specs into the solve

Instead of coupling the PyclingoDriver() object with
spack.config, inject the concrete specs that can be
reused.

A method level function takes care of reading from
the store and the buildcache.

* spack solve: show output of multi-rounds

* add tests for best-effort coconcretization

* Enforce having at least a literal being solved

Co-authored-by: Greg Becker <becker33@llnl.gov>
2022-05-24 12:13:28 -07:00