Commit Graph

499 Commits

Author SHA1 Message Date
Massimiliano Culpo
41bb68a623
Update pipeline configurations
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2025-01-11 13:57:48 +01:00
Massimiliano Culpo
4be7b98fd2
Perl build system: add a dependency on gmake (#48437) 2025-01-08 09:10:22 +01:00
eugeneswalker
25ce6cce47
e4s cray rhel ci stack: re-enable and update for new cpe (#47697)
* e4s cray rhel ci stack: re-enable and update for new cpe, should fix cray libsci issue

* only run e4s-cray-rhel stack

* Mkae Autotools build_system point at correct build_directory

* remove selective enable of cray-rhel stacks

* restore SPACK_CI_DISABLE_STACKS

* use dot prefix to hide cray-sles jobs instead of comment-out

---------

Co-authored-by: Ryan Krattiger <ryan.krattiger@kitware.com>
2025-01-07 08:56:10 -08:00
Wouter Deconinck
aee2f5cf41
hep stack: add celeritas (#48404) 2025-01-07 09:47:17 -05:00
Harmen Stoppels
6622f7a89c
Revert "CI: Move the build stage to the project root instead of tmp (#47996)" (#48397)
This reverts commit ef615bcc7e.
2025-01-04 11:59:22 +01:00
Wouter Deconinck
1b9d001f6a
feat: add HEP cloud_pipeline stack (#40932)
* dd4hep: depends_on root +root7 in some cases

* dd4hep: avoid self-referential whens -> conflicts

* feat: add HEP cloud_pipeline stack

* [hep] add hep jobs to .gitlab-ci.yml

* [hep] rm hep-generate image override

* [hep] fix root variants; add some more packages

* [hep] disable hep-build for now

* [hep] hepmc3 +interfaces

* [hep] hep-generate on gcc-11.4 runner image

* [hep] sherpa -blackhat

* [hep] align package requirements to e4s for more cache hits

* [hep] lcio -examples due to conflicts

* [hep] marks geant4-data as external

* [hep] enable build

* [hep] elfutils now always has bzip2 xz dependency

* [hep] avoid gaudi ^gdb@8.1 with explicit version gdb@14.1

* [hep] geant4 -vtk for now

* [hep] root ~oracle

* [hep] external geant4-data@11.2.0

* [hep] sherap ~python

* [hep] gaudi ^gdb +python to avoid failing gdb@8.1

* [hep] pythia8 ~madgraph5amc

* [hep] remove garfieldpp which fails to download

* [hep] sherpa ~recola because recola needs pic variant

* [hep] sherpa ~rivet since conflicts("cxxstd=11", when="~rivet")

* [hep] unify: when_possible

* [hep] thepeg ~rivet

* [hep] root ~shadow

* [hep] lcio ~jar

* [hep] root ~postgres

* [hep] herwig3 ~evtgen

* [hep] herwig3 has no evtgen variant

* [hep] no image override in hep-generate

* [hep] re-add ubuntu-22.04 image for newer gcc than 7

* [hep] remove herwig3

* [hep] remove package configs

* [hep] avoid non-existent variants in latest versions

* [hep] allow for sherpa@3:

* [hep] acts cxxstd=20

* [hep] gaudi ^gdb +python

* [hep] gaudi ^gdb +debuginfod

* [hep] add some packages to large and huge kubernetes groups

* [hep] root +rpath

* [hep] add qt-declarative to large 8core 8GB CI group

* [hep] add sherpa to 8 core, 3 GB CI medium group

* [hep] add root to 12 core, 19 GB huge CI group

* [hep] gaudi: touch pytest.ini

* [hep] ~ instead of -

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>

---------

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
2025-01-03 12:22:53 -07:00
eugeneswalker
460bc82632
e4s ci: cleanup; see if failures have been fixed (#48137)
* e4s ci: cleanup; see if failures have been fixed

* comment out dealii+cuda cuda_arch=90 due to cuda@12 conflict

* paraview unpin @5.11

* disable paraview cuda90

* ecp-dav ~paraview cuda90

* disable lbann gfx90a

* disable upcxx +cuda

* comment out nek5000 and lammps +cuda

* comment out failed specs

* comment out slepc+rocm

* comment out failures

* comment out failures

* comment out failing paraview buids;

* comment out petsc+rocm

* sort specs
2025-01-02 13:27:40 -08:00
Mike VanDenburgh
d6f71a2afd
Fix import error in CI log aggregation script (#48245)
Fix CI log aggregation script failing due to an import error.
2024-12-20 16:14:17 -07:00
Zack Galbreath
e0035bd658
ci: request 35G of memory for building composable-kernel (#48227)
This number was determined from the max memory usage recently recorded
for this package in our analytics database.
2024-12-20 06:53:15 -07:00
eugeneswalker
75162be4b6
ci: add developer-tools-aarch64-linux-gnu stack (#48217) 2024-12-20 09:24:10 +01:00
kwryankrattiger
93c09ed3b4
Move allocation override to the generate job (#48199) 2024-12-19 21:59:15 +01:00
eugeneswalker
60be9ea068
ci: update darwin tags (#47993)
* ci: update darwin tags

* tag with apple-clang version

* move darwin aarch64 tagging into configs/darwin/aarch/ci.yaml
2024-12-19 04:55:02 -08:00
Scott Wittenburg
ae6d1538d5
ci: Disable broken specs list (#48194) 2024-12-18 21:26:42 -07:00
Zack Galbreath
9719220e8a
Stop building for neoverse_n1 in our GitLab CI pipelines (#48186) 2024-12-18 17:12:05 +00:00
eugeneswalker
4af8fbeddf
ci: remove unmaintained, inactive gpu-tests stack (#48166) 2024-12-18 12:40:20 +01:00
kwryankrattiger
f211e2f9c4
CI: reduce output from helper scripts (#48145) 2024-12-17 12:37:57 -07:00
kwryankrattiger
f04ea573fa
ci: don't error in CI for missing libs (#48169)
There are still more fix ups required for the missing libs to work as
expected in CI. Dropping the error requirement in favor of moving to a
log scraping method until we can verify all package issues have been
addressed correctly.
2024-12-17 19:43:36 +01:00
Zack Galbreath
364f70c16d
Remove E4S Neoverse V1 pipeline (#48160)
Per discussion with the Spack CI team, our graviton2 runners have been
performing poorly and this stack seems no longer necessary.
2024-12-17 19:37:08 +01:00
kwryankrattiger
dfb529fc6e
Ci set concretiztion pool size (#48077)
* Set the "build_jobs" on concretization/generate for CI

build_jobs also controls the concretization pool size. Set this
in the config section for CI generate.

This config is overwritten by build_job CI using the SPACK_BUILD_JOBS
environment variable. This implicitly will drop the default build
CPU request on all "default" grouped build jobs from (max) 16 to 8.

* Add default allocations for build jobs

* Add common jobs and concretize args to ci generate and rebuild

* CI: Specify parallel concretize and build jobs via argument

* Increase power and cray concretization limits

Lowering limits for these stacks creates timeout

* Increase default pool size to 8

intermittent timeouts with 4 CPU

* Add reduced requests for windows for now
2024-12-17 12:05:15 -06:00
Harmen Stoppels
60b4882d4e
ci: pcluster missing_library_policy: ignore (#48138) 2024-12-17 12:01:56 +01:00
Harmen Stoppels
2f6e30fd24
ci: new image for developer-tools (#48065) 2024-12-16 13:08:05 +01:00
Harmen Stoppels
06eae96ef9
config:shared_linking:missing_library_policy to error/warn about accidental use of system libraries on linux/freebsd (#47365)
This commit adds a config option `config:shared_linking:missing_library_policy:error/warn/ignore` which will cause installation errors or warnings when ELF executables or libraries need shared libraries which cannot be resolved from RPATH search paths. The default is to ignore.

This is a safeguard against accidentally linking to system libraries instead of Spack libraries. It makes it more likely that build cache installs work on different machines. It works only at the level of libraries, not at the level of symbols. Some system dependencies are allowed (e.g. kernel and libc).

Packages can (but are discouraged to) set `unresolved_libraries` to a list of patterns of sonames/library names that are know to be unresolvable in RPATHs.  In the future this could be made more fine-grained in a non-breaking way by allowing a dictionary of patterns `lib => [deps]`.
2024-12-16 12:32:36 +01:00
kwryankrattiger
ef615bcc7e
CI: Move the build stage to the project root instead of tmp (#47996)
On MacOS the tmpdir fills up over time and isn't cleared properly.
Move the build stage to a location it is cleared after each build.
2024-12-13 06:55:01 +00:00
Harmen Stoppels
526af1cbe7
Sprinkle open(..., encoding=utf-8) (#48006)
Add missing encoding=utf-8 to various open calls. This makes
files like spec.json, spack.yaml, spack.lock, config.yaml etc locale
independent w.r.t. text encoding. In practice this is not often an
issue since Python 3.7, where the C locale is promoted to
C.UTF-8. But it's better to enforce UTF-8 explicitly, since there is
no guarantee text files are written in the right encoding.

Also avoid opening in text mode if it can be avoided.
2024-12-12 21:46:08 +01:00
Adam J. Stewart
396a701860
Python: deprecate 3.8 (#46913)
* Python: deprecate 3.8

* Remove preference for EOL Python versions

* Explicitly deprecate things requiring EOL Python

* More deprecations

* deprecate old versions of slepc, py-petsc4py, py-slepc4py in sync with old versions of petsc

---------

Co-authored-by: Satish Balay <balay@mcs.anl.gov>
2024-12-12 15:22:06 +01:00
Scott Wittenburg
a3985e7538
Revert "Set the "build_jobs" on concretization/generate for CI (#47660)" (#48028)
This reverts commit 316dcc1609.
2024-12-11 07:56:36 +01:00
kwryankrattiger
316dcc1609
Set the "build_jobs" on concretization/generate for CI (#47660)
* Set the "build_jobs" on concretization/generate for CI

build_jobs also controls the concretization pool size. Set this
in the config section for CI generate.

This config is overwritten by build_job CI using the SPACK_BUILD_JOBS
environment variable. This implicitly will drop the default build
CPU request on all "default" grouped build jobs from (max) 16 to 8.

* Add default allocations for build jobs

* Add common jobs and concretize args to ci generate and rebuild

* CI: Specify parallel concretize and build jobs via argument

* Increase power and cray concretization limits

Lowering limits for these stacks creates timeout
2024-12-10 14:13:23 -07:00
Massimiliano Culpo
b50dbb8604
pipelines: simplify and lint aws-pcluster-* (#47989) 2024-12-10 12:16:51 +01:00
Harmen Stoppels
30c00353d4
make level_zero variant consistent, add missing instances (#47985) 2024-12-10 09:25:30 +01:00
Stephen Sachs
8a9e16dc3b
aws-pcluster stacks: static spack.yaml (#47918) 2024-12-08 20:26:08 +01:00
eugeneswalker
0b7fc360fa
e4s ci: add lammps +rocm (#47929)
* e4s ci: add lammps +rocm

* e4s rocm external stack: add def for rocm-openmp-extras external

* lammps +rocm external rocm has errors, comment out
2024-12-08 10:21:28 -08:00
Harmen Stoppels
22d104d7a9
ci: add bootstrap stack for python@3.6:3.13 (#47719)
Resurrect latest Python 3.6
Add clingo-bootstrap to Gitlab CI.
2024-12-05 10:07:24 +01:00
Mark Abraham
c037188b59
gromacs: announce deprecation policy and start to implement (#47804)
* gromacs: announce deprecation policy and start to implement

* Style it up

* [@spackbot] updating style on behalf of mabraham

* Bump versions used in CI

---------

Co-authored-by: mabraham <mabraham@users.noreply.github.com>
2024-11-26 05:54:07 -07:00
kwryankrattiger
4f9aa6004b
visit: add v3.4.0, v3.4.1 (#47161)
* Visit: Add new versions 3.4.0 and 3.4.1

* Adios2: Restrict python, 3.11 doesn't not work for older Adios2

* VisIt: Set the VTK_VERSION for @3.4:

Older versions of VTK used the VTK_{MAJOR, MINOR}_VERSION variables for
VTK detection. VisIt >= 3.4 uses the full string VTK_VERSION.

* CI: Don't build llvm-amdgpu for non-HIP stack

* VisIt: v3.4.1 handles newer Adios2 correctly

* Visit: Add missing links in HDF5, set correct VTK version configuration parameter

* VisIt: Add py-pip requirement and patch visit with configuration changes

* HDF5 symlinks move when inside of callback

* VisIt ninja install fails with python module. Using make does not

* VisIt 3.4 has a high minimum cmake requirement

* HDF5: Early return when not mpi for mpi symlinks

* HDF5: Use platform agnostic method for creating legacy compatible MPI symlinks

* Fix VISIT_VTK_VERSION handling for 8.2.1a hack
2024-11-20 18:37:56 +01:00
Massimiliano Culpo
067fefc46a
Set "generic" uarch granularity for a few pipelines (#47640) 2024-11-17 09:07:42 +01:00
Luca Heltai
badb3cedcd
dealii: add v9.6.0 (#45554)
Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
Co-authored-by: Satish Balay <balay@mcs.anl.gov>
2024-11-13 15:34:27 +01:00
Cameron Smith
8f4a0718bf
omega-h: new version and cuda conflicts for prior versions (#47473)
* omegah: add version 10.8.6

* omegah: cuda without kokkos conflict

* omegah: test with latest version in ci
2024-11-06 22:36:59 +00:00
Harmen Stoppels
2b9c6790f2
omega-h: fix versioning and cuda compat (#47433) 2024-11-05 15:50:48 +01:00
eugeneswalker
632c009569
e4s ci stacks: reduce package prefs (#47381) 2024-11-03 00:18:13 +01:00
Adam J. Stewart
32ce278a51
ML CI: Linux aarch64 (#39666)
* ML CI: Linux aarch64

* Add config files

* No aarch64 tag

* Don't specify image

* Use amazonlinux image

Co-authored-by: kwryankrattiger <80296582+kwryankrattiger@users.noreply.github.com>

* Update and require

* GCC is too old

* Fix some builds

* xgboost doesn't support old GCC + cuda

* Run on newer Ubuntu

* Remove mxnet

* Try aarch64 range

* Use main branch

* Conflict applies to all targets

* cuda only required when +cuda

* Use tagged version

* Comment out tf-estimator

* Add ROCm, use newer Ubuntu

* Remove ROCm

---------

Co-authored-by: kwryankrattiger <80296582+kwryankrattiger@users.noreply.github.com>
2024-10-28 10:30:07 +01:00
Scott Wittenburg
1472dcace4
ci: Remove deprecated logic from the ci module (#47062)
ci: Remove deprecated logic from the ci module

Remove the following from the ci module, schema, and tests:

- deprecated ci stack and handling of old ci config
- deprecated mirror handling logic
- support for artifacts buildcache
- support for temporary storage url
2024-10-23 12:50:55 -06:00
kwryankrattiger
fd1c95a432
ParaView: Various fixes to better support no mpi and fides builds (#47114)
* ParaView: Explicitly set the ENABLE_MPI on/off
* Disallow MPI in the DAG when ~mpi
* @5.13 uses 'remove_children', use pugixml@1.11:, See #47098
* cloud_pipelines/stacks/data-vis-sdk: paraview +raytracing: add +adios2 +fides

Co-authored-by: Bernhard Kaindl <bernhardkaindl7@gmail.com>
2024-10-23 11:46:09 -06:00
Tom Scogland
a07d42d35b
Devtools darwin (#46910)
* stacks: add a stack for devtools on darwin

After getting this whole mess building on darwin, let's keep it that
way, and maybe make it so we have some non-ML darwin binaries in spack
as well.

* reuse: false for devtools

* dtc: fix darwin dylib name and id

On mac the convention is `lib<name>.<num>.dylib`, while the makefile
creates a num suffixed one by default. The id in the file is also a
local name rather than rewritten to the full path, this fixes both
problems.

* node-js: make whereis more deterministic

* relocation(darwin): catch Mach-O load failure

The MachO library can throw an exception rather than return no headers,
this happened in an elf file in the test data of go-bootstrap.  Trying
catching the exception and moving on for now.  May also need to look
into why we're trying to rewrite an elf file.

* qemu: add darwin flags to clear out warnings

There's a build failure for qemu in CI, but it's invisible because of
the immense mass of warning output.  Explicitly specify the target macos
version and remove the extraneous unknown-warning-option flag.

* dtc: libyaml is also a link dependency

libyaml is required at runtime to run the dtc binary, lack of it caused
the ci for qemu to fail when the library wasn't found.
2024-10-21 17:32:14 +00:00
Stephen Sachs
9854c9e5f2
Build wrf%oneapi in aws-pcluster-x86_64_v4 stack (#47075) 2024-10-21 02:56:36 -06:00
eugeneswalker
df6427d259
e4s ci stacks: add nwchem (#47055) 2024-10-18 14:30:46 -07:00
eugeneswalker
7ecdc175ff
e4s ci stacks: add fftx cpu, cuda, and rocm builds (#47004)
* e4s ci stacks: add fftx cpu, cuda, and rocm builds

* disable fftx+rocm due to spack github issue #47034

* e4s oneapi: fftx has spack build error https://github.com/spack/spack/issues/47048
2024-10-17 10:11:32 -07:00
eugeneswalker
e4c233710c
e4s external rocm ci: upgrade to v6.2.1 (#46871)
* e4s external rocm ci: upgrade to v6.2.1

* use ghcr.io/spack/spack/ubuntu22.04-runner-amd64-gcc-11.4-rocm6.2.1:2024.10.08

* magma +rocm: add entry for v6.2.1
2024-10-16 19:42:19 -06:00
eugeneswalker
fb53d31d09
ci: use ghcr.io images instead of dockerhub hosted (#46830) 2024-10-16 12:11:26 -04:00
Caetano Melone
1259992159
Reduce noop job resource requests (#46920)
`no-spec-to-rebuild` jobs use far less resources than they request. For example, [this](https://gitlab.spack.io/spack/spack/-/jobs/12944487) job [used](https://prometheus.spack.io/api/v1/query_range?query=container_memory_working_set_bytes{pod=%22runner-dcsgp53u-project-2-concurrent-3-0ubclrr1%22}&start=1728655743&end=1728656543&step=1s) around 3MB.

While this won't lead to any crazy cost savings, k8s requests effectively block other jobs from using the resources, so reducing this to a reasonable number is important.
2024-10-15 12:49:45 -05:00
kwryankrattiger
c8f61c8662
Don't require OIDC initialization for noop (#46921)
ref. https://github.com/spack/spack-infrastructure/pull/957
2024-10-14 23:39:55 -05:00