Commit Graph

357 Commits

Author SHA1 Message Date
Harmen Stoppels
08d7f47278 curl flag is not universally supported (#36009) 2023-03-10 16:20:32 -08:00
Harmen Stoppels
26fd1ac5b0 hotfix: fix double double quotes (#36005) 2023-03-10 13:41:01 -08:00
Zack Galbreath
e1301df60c ci: version bump for ghcr.io/spack/e4s-amazonlinux-2 (#35976)
* ci: version bump for ghcr.io/spack/e4s-amazonlinux-2

This new image comes with GnuPG v2.4.0

* py-cython: upperbounds for Python versions

* fix py-gevent nonsense

---------

Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-03-10 13:32:11 -08:00
kwryankrattiger
f3595da600 CI boilerplate reduction (#34272)
* CI configuration boilerplate reduction and refactor

Configuration:
- New notation for list concatenation (prepend/append)
- New notation for string concatenation (prepend/append)
- Break out configuration files for: ci.yaml, cdash.yaml, view.yaml
- Spack CI section refactored to improve self-consistency and
composability
  - Scripts are now lists of lists and/or lists of strings
  - Job attributes are now listed under precedence ordered list that are
  composed/merged using Spack config merge rules.
  - "service-jobs" are identified explicitly rather than as a batch

CI:
- Consolidate common, platform, and architecture configurations for all CI stacks into composable configuration files
- Make padding consistent across all stacks (256)
- Merge all package -> runner mappings to be consistent across all
stacks

Unit Test:
- Refactor CI module unit-tests for refactor configuration

Docs:
- Add docs for new notations in configuration.rst
- Rewrite docs on CI pipelines to be consistent with refactored CI
workflow

* Script verbose environ, dev bootstrap

* Port #35409
2023-03-10 12:25:35 -07:00
Scott Wittenburg
16c67ff9b4 ci: Increase the amount of pruning possible for PR pipelines (#35944)
By setting the traversal depth to 1, only specs matching the changed
package and direct dependents of those (and of course all dependencies
of that set) are removed from pruning candidacy.
2023-03-10 11:19:52 -08:00
kwryankrattiger
895886959f CI: Fix timeout for VisIt and ParaView (#35787)
ref. #35400
2023-03-03 09:13:14 +01:00
Massimiliano Culpo
07e251c887 Remove handling of deprecated target names (graviton) (#35537)
* Update target names for Gitlab pipelines

* Remove handling of deprecated names for graviton
2023-03-01 11:03:12 +01:00
eugeneswalker
34441c9eaa ci: gpu test stack: turn off cuda@11.7 builds (#35467) 2023-02-13 21:54:01 -08:00
eugeneswalker
f7de22eb14 gpu test stack: add test w/ ^cuda@12 builds on A100 w/ newer driver (#35375)
* gpu test stack: test cuda@12 builds on A100 w/ newer driver

* get gpu info via nvidia-smi;

* kokkos+cuda^cuda@12 has genuine failure
2023-02-09 12:18:03 -07:00
eugeneswalker
f6fa64f979 ci: add minimal gpu testing stack (#35251)
* ci: add minimal gpu testing stack

* kokkos +cuda requires +wrapper...

* require pass

* add raja+cuda
2023-02-03 18:33:34 -08:00
eugeneswalker
99f3716346 e4s: add and prefer boost +python +filesystem +iostreams +system (#35180) 2023-01-30 07:37:35 -08:00
eugeneswalker
48a63719b2 e4s oneapi: use oneapi@2023.0.0 (#34634) 2023-01-29 13:22:14 -08:00
eugeneswalker
9f5d9266e6 e4s: add power stack (#32710)
* e4s: restore builds builds

* gitlab ci: allow UO to build protected binaries for signing

* use newer image; comment out failing builds

* gitlab-ci: Some tweaks for e4s power builds

- fix tags (no longer require generate jobs to run on aws)
- fix resource requests for generation jobs resource requests
- remove SPACK_SIGNING_KEY from protected power build jobs
- update UO signing key path
- change the CDash build group to reflect stack name
- retry pipeline generation jobs *always*

* correct double packages: section

* gitlab-ci:script: modernize

* remove new gnu make, not for ppc64le

---------

Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
2023-01-29 10:53:02 -06:00
eugeneswalker
a82fe2010f e4s: add cusz +cuda (#35004) 2023-01-23 09:15:39 -08:00
Zack Galbreath
91fd6d5a84 Allocate more memory for generate jobs in all stacks (#35040)
* Allocate more memory for generate jobs in all stacks
* Add a 60 minute timeout on generate jobs
2023-01-20 11:06:49 -07:00
eugeneswalker
6713f7feda e4s: ecp-data-vis-sdk +rocm: remove redundant amdgpu_target specification (#35022) 2023-01-20 00:18:41 +00:00
eugeneswalker
fed4eb2526 e4s ci: clean up unnecessary constraints (#35006) 2023-01-18 21:23:15 -08:00
eugeneswalker
1bf1c7de76 e4s: add sz3 (#35007) 2023-01-18 22:21:30 -06:00
eugeneswalker
ae5408de9a e4s ci: add veloc (#35003) 2023-01-18 16:48:23 -08:00
eugeneswalker
8eb803ab4c e4s: add py-h5py ~mpi variant (#34998) 2023-01-18 19:28:47 +00:00
eugeneswalker
886ea7c490 e4s: add py-h5py (#34914) 2023-01-16 15:52:24 +00:00
eugeneswalker
b9a16a599a e4s ci: py-libensemble: activate variants (#34915) 2023-01-16 10:57:28 +01:00
kwryankrattiger
229ee7cad9 Paraview rocm (#34790)
* paraview: add `rocm` variant

This conflicts with CUDA and requires at least ParaView 5.11.0. More
dependencies are also needed.

* E4S: Add ParaView for ROCm and CUDA stacks

* DAV SDK: Update ParaView version and GPU variants

* Verify using hipcc vs amdclang++ for newer hip

Co-authored-by: Ben Boeckel <ben.boeckel@kitware.com>
2023-01-13 12:41:57 -06:00
Mike VanDenburgh
ea6608edde Bump memory requests for e4s/ml generate jobs (#34648) 2023-01-11 10:41:42 -07:00
eugeneswalker
9d70f6e6c7 protected runner jobs at uo: trust e4s-uo signing key (#34828) 2023-01-10 18:18:38 -07:00
Scott Wittenburg
8f6f77ac9f gitlab ci: fix tags for pipeline generation jobs (#34789)
Gitlab does not merge lists when a job extends two other definitions
that include the same list (e.g. tags).  Also, it merges dictionaries
as long as the keys are distinct, but just takes the last mentioned
value when there are key collisions.

This change makes sure that when different tags are needed by a
pipeline, the ones we want are actually provided.  It also changes
the example stack to better follow this pattern so we do not lead
developers astray in the future.
2023-01-10 10:06:26 -07:00
Wileam Y. Phan
0f7f600d1f Add CUDA 12.0 (#34664) 2023-01-10 11:51:23 +01:00
eugeneswalker
9283a94ee4 e4s ci: use 2023-01-01 runner image (#34827) 2023-01-10 10:47:28 +00:00
Harmen Stoppels
b9f48da560 e4s: move default values for rocm/cuda arch into packags:all:variants (#34772) 2023-01-04 16:09:14 +01:00
Adam J. Stewart
eb67497020 ML CI: Linux x86_64 (#34299)
* ML CI: Linux x86_64

* Update comments

* Rename again

* Rename comments

* Update to match other arches

* No compiler

* Compiler was wrong anyway

* Faster TF
2022-12-22 11:31:40 -06:00
eugeneswalker
7e836b925d e4s: disable mac stack due to binary relocation issue#32571 (#34560) 2022-12-17 10:53:15 +00:00
Zack Galbreath
aa8e1ba606 gitlab ci: more resources for slow builds (#34505) 2022-12-15 14:35:54 -07:00
Scott Wittenburg
675afd884d gitlab ci: more resources for paraview and py-torch (#34412) 2022-12-09 11:58:37 -07:00
Andrew W Elble
588d2e295f py-alphafold: update to 2.2.4, update dependencies (#33876)
* py-alphafold: update to 2.2.4, update dependencies

* style
2022-12-02 22:12:06 +00:00
eugeneswalker
f452741e3d e4s ci: use 2022-12-01 runner images (#34212) 2022-11-30 09:52:30 -08:00
eugeneswalker
99b68e646d e4s ci: hpx: set max_cpu_count=512 (#33977) 2022-11-30 09:16:57 -08:00
kwryankrattiger
33422acef0 CI: Update Data and Vis SDK Stack (#34009)
* CI: Update Data and Vis SDK Stack

* Update image to match target deployments (E4S)
* Enable all packages
* Test supported variants of ParaView and VisIt

* Sensei: Update Python hint for newer cmake

* Sensei: add Python3 hint
2022-11-29 14:49:55 -07:00
eugeneswalker
587488882a e4s ci: add hdf5-vol-async; remove expired comments (#34110) 2022-11-28 19:35:42 +00:00
kwryankrattiger
5d2c9636ff E4S: Conservatively add ecp-data-vis-sdk (#33621)
* E4S: Conservatively add ecp-data-vis-sdk

* Remove ascent from CUDA SDK stack to stop hanging on Dray

* Adios2: Newer FindPython uses Python_EXECUTABLE
2022-11-23 11:01:30 -08:00
Harmen Stoppels
90fb16033e gitlab: report load in generate job (#33888) 2022-11-15 13:21:21 +01:00
Harmen Stoppels
f332ac6d21 More jobs in Gitlab CI (#33688)
Use at most 32 jobs when available.
2022-11-08 12:53:11 +01:00
Adam J. Stewart
0f32f7d0e9 py-transformers: add v4.24.0 (#33716)
* py-transformers: add v4.24.0

* Internet access still required
2022-11-05 12:38:49 -05:00
Massimiliano Culpo
ace8c74e20 Revert "gitlab: when_possible -> false (#33443)" (#33552)
This reverts commit b1559cc831.
2022-10-31 13:31:57 -07:00
eugeneswalker
999c460b1e e4s ci: add mfem +rocm (#31604) 2022-10-29 23:32:46 +00:00
Harmen Stoppels
fa0432c521 gitlab ci: patched make 4.3.0 (#33583) 2022-10-28 14:38:57 +02:00
Harmen Stoppels
df1d233573 Don't fail over cpuinfo (#33546) 2022-10-27 11:14:09 +02:00
Harmen Stoppels
34f9394732 gitlab ci: show build machine info (#33523) 2022-10-26 20:31:16 +02:00
Massimiliano Culpo
30c9ff50dd Allow for packages with multiple build-systems (#30738)
This commit extends the DSL that can be used in packages
to allow declaring that a package uses different build-systems
under different conditions.

It requires each spec to have a `build_system` single valued
variant. The variant can be used in many context to query, manipulate
or select the build system associated with a concrete spec.

The knowledge to build a package has been moved out of the
PackageBase hierarchy, into a new Builder hierarchy. Customization
of the default behavior for a given builder can be obtained by
coding a new derived builder in package.py.

The "run_after" and "run_before" decorators are now applied to
methods on the builder. They can also incorporate a "when="
argument to specify that a method is run only when certain
conditions apply.

For packages that do not define their own builder, forwarding logic
is added between the builder and package (methods not found in one
will be retrieved from the other); this PR is expected to be fully
backwards compatible with unmodified packages that use a single
build system.
2022-10-26 20:17:32 +02:00
Harmen Stoppels
a2520e80c0 gitlab ci: install binary deps faster (#33248)
* Fast Gitlab CI job setup, and better legibility

* Use a non-broken, recent GNU Make
2022-10-26 09:19:24 +02:00
Harmen Stoppels
d361378553 Improve legibility of Gitlab CI (#33482)
Use --backtrace in ci instead of --debug to reduce verbosity
and don't show log on error, since log is already printed
2022-10-25 12:21:34 +02:00