* e4s ci: add exago +cuda, +rocm builds
* exago: rename 5-18-2022-snapshot to snapshot.5-18-2022
* disable exago +rocm for non-external rocm ci install
* note that hiop +rocm fails to find hip libraries when they are spack-installed
* ci: don't register detectable compilers
Cause they go out of sync...
* remove intel compiler, it can be detected too
* Do not run spack compiler find since compilers are registered in concretize job already
* trilinos: work around +stokhos +cuda +superlu-dist bug due to EMPTY macro
* gromacs +cp2k: build in CI
* libxsmm: x86 only
* attempt to fix dbcsr + new mpich
* use c11 standard
* gromacs: does not depend on dbcsr
* cp2k: build with cmake in CI, s.t. dbcsr is a separate package
* cp2k: cmake patches for config files and C/C++ std
* cp2k: remove unnecessary constraints due to patch
Improve how mirrors are used in gitlab ci, where we have until now thought
of them as only a string.
By configuring ci mirrors ahead of time using the proposed mirror templates,
and by taking advantage of the expressiveness that spack now has for mirrors,
this PR will allow us to easily switch the protocol/url we use for fetching
binary dependencies.
This change also deprecates some gitlab functionality and marks it for
removal in Spack 0.23:
- arguments to "spack ci generate":
* --buildcache-destination
* --copy-to
- gitlab configuration options:
* enable-artifacts-buildcache
* temporary-storage-url-prefix
* Allow branching out of the "generic build" unification set
For cases like the one in https://github.com/spack/spack/pull/39661
we need to relax rules on unification sets.
The issue is that, right now, nodes in the "generic build" unification
set are unified together with their build dependencies. This was done
out of caution to avoid the risk of circular dependencies, which would
ultimately cause a very slow solve.
For build-tools like Cython, however, the build dependencies is masked
by a long chain of "build, run" dependencies that belong in the
"generic build" unification space.
To allow splitting on cases like this, we relax the rule disallowing
branching out of the "generic build" unification set.
* Fix issue with pure build virtual dependencies
Pure build virtual dependencies were not accounted properly in the
list of possible virtuals. This caused some facts connecting virtuals
to the corresponding providers to not be emitted, and in the end
lead to unsat problems.
* Fixed a few issues in packages
py-gevent: restore dependency on py-cython@3
jsoncpp: fix typo in build dependency
ecp-data-vis-sdk: update spack.yaml and cmake recipe
py-statsmodels: add v0.13.5
* Make dependency on "blt" of type "build"
* e4s amd64 gcc ci stack: sync with e4s-23.08
* e4s amd64 oneapi ci stack: sync with e4s-23.08
* e4s ppc64 gcc ci stack: sync with e4s-23.08
* add new ci stack: e4s amd64 gcc w/ external rocm
* add new ci stack: e4s arm gcc ci
* updates
* py-scipy: -fvisibility issue is resolved in 2023.1.0: #39464
* paraview oneapi fails
* comment out pkgs that fail to build on power
* fix arm stack name
* fix cabana +cuda specification
* comment out failing spces
* visit fails build on arm
* comment out slepc arm builds due to make issue
* comment out failing dealii arm builds
* e4s cray rhel stack: expand to full spec list
* comment out gasnet; require %gcc for unzip
* require openblas@0.3.20 to get around %cce error; follow up with issue report
* comment out failing specs;
* comment out axom and xyce due to errors
* improve clarity of failing specs
* Add OIDC tokens to gitlab-ci jobs
This should allow us to start issuing just-in-time generated
credentials for CI jobs that need to modify binary mirrors. The "aud"
claim of the token describes what the token is allowed to do. The
claim is verified against a set of rules on the IAM role using signed
information from GitLab. See spack-infrastructure for the claim
verification logic.
---------
Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
Currently the configuration of the pipeline is such that
there are multiple "optimal" solutions. This is due to
the pipeline making ~lld the default for LLVM, but leaving
+libomptarget from package.py
Since LLVM recipe has a:
conflicts("~lld", "+libomptarget")
clingo is forced to change one of the two defaults, and
the penalty for doing so is the same for each. Hence, it
is random whether we'll get +libomptarget+lld
or ~libomptarget~lld.
This fixes it by changing also the default for libomptarget.
Instead of pointing to the image on DockerHub, which rate limits us and
causes pipeline failures durying busy times, use the version at ghcr.
And we might as well use the ghcr version everywhere else too.
* Write timing information for installs from cache
* CI: aggregate and upload install_times.json to artifacts
* CI: Don't change root directory for artifact generation
* Flat event based timer variation
Event based timer allows for easily starting and stopping timers without
wiping sub-timer data. It also requires less branching logic when
tracking time.
The json output is non-hierarchical in this version and hierarchy is
less rigidly enforced between starting and stopping.
* Add and write timers for top level install
* Update completion
* remove unused subtimer api
* Fix unit tests
* Suppress timing summary option
* Save timers summaries to user_data artifacts
* Remove completion from fish
* Move spack python to script section
* Write timer correctly for non-cache installs
* Re-add hash to timer file
* Fish completion updates
* Fix null timer yield value
* fix type hints
* Remove timer-summary-file option
* Add "." in front of non-package timer name
---------
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
Setting the undocumented variable SPACK_CONCRETIZER_REQUIRE_CHECKSUM
now causes the solver to avoid accounting for versions that are not checksummed.
This feature is used in CI to avoid spurious concretization against e.g. develop branches.
* VTK: Add patch for python 3.8 support
* CI: Re-enable VisIt in CI
* Configure spec matrix for stack with VisIt
* Add pugixml dep for 8.2.0
* Make VTK and ParaView consistent on proj dep
* OpenMPI 3: provides MP support by default
* Add details on proj dep in ParaView
* Add python 3.8 to test mock repo
* Patches to get VisIt VTK interface
* CI: Disable VisIt with GUI in DAV
* AMReX: 23.06+ Multi-Dim Support
This updated the Spack package to allow to install AMReX, modules of
AMReX in E4S deployments and dependent packages with support for
multiple dimensions. Due to an upstream change in AMReX, we do not
longer need to ship three, binary incompatible package variants.
* [E4S] oneAPI AMReX < 23.06 Variant
Work-around the auto-concretization to the multi-dim of `dimensions`,
which only in 23.06+ became a multi-variant.
* e4s cray rhel ci: temporarily disable amrex build until spurious ci failure can be resolved
---------
Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
Refactor gitlab ci configs so that mac and cray jobs can reuse as much higher level
configuration as possible.
* CI: remove redundant sections
* CI: Include base linux CI configs in cray stacks
Relocation and runner mapping is consistent between cray and linux runners.
* Export user cache path in before script
* CI: add GPG root for mac runners
* Disable user configs
Metal runners share a ~ directory
* Disable user config and add configs in activate env
* ci: run spack list in power ci
Let's see if Spack itself is the bottleneck in CI...
* rebuild curl in CI
* more of the same please!
* drop the profiler
* undo rebuildme test in ci variant
* add comment for posterity
* enable profiling
* trigger CI
* See how it goes now that perf regressions are fixed on develop
* try shorter poll intervals
* Revert "try shorter poll intervals"
This reverts commit d60c34ad3eceead0c13a5277cf8e783fd42b7458.
* Remove spec.format call in Database._get_matching_spec_key
* once more in ci please
* undo irrelevant changes
* run spack list in before script
* test in ci
* -:
* Undo CI testing
* openssl: prefer 3.x
This PR is not intended to be merged immediately, but it would be good
to see what packages fail to build in CI so that we can get proper
version constraints on openssl (before all packages update and support
both openssl 1 and 3)
* Disable assembly for 3.x %oneapi
* cmake: depend on spack curl, to deal with curl - openssl compat
* also make zlib external
* remove overly strict & unsafe requirement on py-cryptographty patch version number
* update openssl compat bounds in py-cryptography
* smaller diff
* Make libssh2 an autotools/cmake package
* fix weird upperbound in libssh2 as there is not openssl v2
* libssh2: pc file lists plain -lssl -lcrypto w/o leading -L flag, confusing libgit2 parsing of pkg-config output
* Actually fix the issue in libssh2: its pc file looks broken
* WarpX 23.06
Update WarpX and related Python packages to the lastest releases.
WarpX 23.06 introduces multi-dimension support in a single package,
which will ease deployment in E4S et al. that can ship now a single,
full-feature module/package that is NOT incompatible with itself
anymore.
* e4s ci stacks: multiple specs for each dim variant no longer required
* [@spackbot] updating style on behalf of ax3l
* WarpX: Update CMake CLI and Test/Check
* Add Missing `build-directory`
* [@spackbot] updating style on behalf of ax3l
* Remove `build_directory` again
---------
Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
Co-authored-by: ax3l <ax3l@users.noreply.github.com>
The pcluster image has am internal buildcache without an index.
Also, we need to force reuse to avoid rebuilding GCC, since the default is
to only reuse dependencies - and that is subject to changes in the GCC
recipe.
* e4s oneapi ci: use official intel oneapi-derived runner image
* update oneapi image
* tau builds ok, but only with libdrm - comment out for now, follow up with pr later
* e4s cray ci stack
* e4s ci: add cray
* add zen4 tag
* WIP: new defintions just for cray
* updates
* remove ci signing job overrride, not necessary
* echo $PATH and show modules loaded
* add mirror
* add external def for cray-libsci
* comment out quantum-espresso
* use /etc/protected-runner as key path
* cray ci stack: do not remove tags: [spack, public]
* make cray stack composable
* generate job should run on public tagged runner, override default config:install_tree:root
* CI: Use relative path in default script
* CI: Use relative includes paths for shell runners
* Use concrete_env_dir for relpath
* ml-darwin-aarch64-mps: jax has bazel codesign issue
---------
Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
Co-authored-by: Ryan Krattiger <ryan.krattiger@kitware.com>
* llvm: replace +omp_as_runtime with omp=runtime
* llvm: fetch 'libomp-libflags-as-list.patch' from upstream repo
* llvm: replace 'llvm14-hwloc-ompd.patch' with the official fix from upstream repo
* llvm: fix-up for the black reformatting
* llvm: fetch 'constexpr_longdouble.patch' from upstream repo
* llvm: optionally build libcxx as a runtime
* llvm: fetch 'llvm5-sanitizer-ustat.patch' from upstream repo
* llvm: update 'sanitizer-ipc_perm_mode.patch'
* llvm: refactor compiler conflicts when libcxx=project
* llvm: fetch 'llvm_python_path.patch' from upstream repo
* llvm: update comments and condition for 'xray_buffer_queue-cstddef.patch'
* llvm: optionally build compiler-rt as a runtime
* llvm: fetch 'lldb_external_ncurses-10.patch' from upstream repo
* llvm: fetch 'llvm_py37.patch' from upstream repo
* llvm: rename variant 'internal_unwind' to 'libunwind'
* llvm: optionally build libunwind as a runtime
* llvm: extend the list of maintainers
* llvm: allow for explicit '~clang~flang~libomptarget~lldb~omp_debug~z3'
* llvm: fetch 'llvm5-lld-ELF-Symbols.patch' from FreeBSD port repo
* llvm: fetch most of 'missing-includes.patch' from upstream repo and reuse 'llvm-gcc11.patch'
* llvm: regroup patches for missing include directives and drop compiler constraints for them
* llvm: fetch 'llvm-gcc11.patch' from upstream repo
* llvm: fetch 'no_cyclades.patch' from upstream repo
* llvm: update comments and condition for 'no_cyclades9.patch'
* llvm: rename variant 'omp' to 'openmp'
* llvm: constrain and rename variant 'omp_tsan' to 'libomp_tsan'
* llvm: rename variant 'omp_debug' to 'libomptarget_debug'
* llvm: do not apply same patch twice
* llvm: constrain and document the '*-thread.patch' patches
* llvm: document the '~lld+libomptarget' conflict
* llvm: update comments for the 'D133513.diff' patch
* Add macOS ML CI stacks
* torchmeta is no longer maintained and requires ancient PyTorch
* Add MXNet
* update darwin aarch64 stacks
* add darwin-aarch64 scoped config.yaml
* remove unnecessary cleanup job
* fix specifications
* fix labels
* fix labels
* fix indent on tags specification
* no tags for trigger jobs
* try overriding tags in stack spack.yaml
* do not use CI_STACK_CONFIG_SCOPES
* incorporate config:install_tree:root: overrides and compiler defs
* copy relevant ci-scoped config settings directly into stack spack.yaml
* remove build-job-remove
* spack ci generate: add debug flag
* include cdash config directly in stack spack.yaml
* customize build-job script section to avoid absolute paths
* add any-job specification
* tags: use aarch64-macos instead of aarch64
* generate tags: use aarch64-macos instead of aarch64
* do not add morepadding
* use shared mirror; comment out known failures
* remove any-job
* nproc || true
* comment out specs failing due to bazel from cache codesign issue
---------
Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
* [pcluster pipeline] Use local buildcache instead of upstream spack
Spack currently does not relocate compiler references from upstream spack
installations. When using a buildcache we don't need an upstream spack.
* gcc needs to be installed via postinstall to get correct deps
* quantum-espresso@gcc@12.3.0 returns ICE on neoverse_{n,v}1
* Force gitlab to pull the new container
* Revert "Force gitlab to pull the new container"
This reverts commit 3af5f4cd88.
Seems the gitlab version does not yet support "pull_policy" in .gitlab-ci.yml
* Gitlab keeps picking up wrong container. Renaming
* Update containers once more after failed build
Add aws-plcuster[-aarch64] stacks. These stacks build packages defined in
https://github.com/spack/spack-configs/tree/main/AWS/parallelcluster
They use a custom container from https://github.com/spack/gitlab-runners which
includes necessary ParallelCluster software to link and build as well as an
upstream spack installation with current GCC and dependencies.
Intel and ARM software is installed and used during the build stage but removed
from the buildcache before the signing stage.
Files `configs/linux/{arch}/ci.yaml` select the necessary providers in order to
build for specific architectures (icelake, skylake, neoverse_{n,v}1).
* CI: Expand E4S ROCm stack to include missing DaV packages
Ascent: Fixup for VTK-m with Kokkos backend
* DaV SDK: Removed duplicated openmp variant for ascent
* Drop visit and add conflict for Kokkos
* E4S: Drop ascent from CUDA builds
Ensure that requirements `packages:*:require:@x` and preferences `packages:*:version:[x]`
fail concretization when no version defined in the package satisfies `x`. This always holds
except for git versions -- they are defined on the fly.
* gitlab ci: release fixes and improvements
- use rules to reduce boilerplate in .gitlab-ci.yml
- support copy-only pipeline jobs
- make pipelines for release branches rebuild everything
- make pipelines for protected tags copy-only
* gitlab ci: remove url changes used in testing
* gitlab ci: tag mirrors need public key
Make sure that mirrors associated with release branches and tags
contain the public key needed to verify the signed binaries. This
also ensures that when stack-specific mirror contents are copied
to the root, the root mirror has the public key as well.
* review: be more specific about tags, curl flags
* Make the check in ci.yaml consistent with the .gitlab-ci.yml
---------
Co-authored-by: Ryan Krattiger <ryan.krattiger@kitware.com>
The flags --mirror-name / --mirror-url / --directory were deprecated in
favor of just passing a positional name, url or directory, and letting spack
figure it out.
---------
Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
* DaV SDK: Enable ParaView raytracing with in SDK
* CI: Drop swr testing from Data Vis SDK
* ISPC: extend LLVM requirement to main
* DaV SDK: Disallow concretizing develop unifyfs
No longer needed after mochi-margo patch
