if dump file existed it was not truncating the file, resulting in
a file with unaltered filesize, with the new content at the beginning,
"padded" with the tail of the old content, since the new content was
not enough to overwrite it.
`colify` is an old module in Spack that still uses `**kwargs` liberally.
We should be more explicit. Doing this eliminates the need for many
checks (can't pass the wrong arg if it isn't allowed) and makes the
function documentation more clear.
Fixes a bug introduced in 44ed0de8c0
where the push method of binary_distribution now takes named args
include_root and include_depedencies, to avoid the **kwarg hole.
But the call site wasn't update and we passed a dict of keys/values instead
of arguments, which resulted in a call like this:
```
push(include_root={"include_root": True, "include_dependencies": False})
```
This commit fixes that, and adds a test to see if we push the correct packages.
This error shows up a lot, typically it's harmless because an error
happened before the source build even started, in which case we don't
have build logs to copy. So, warn instead of error, cause it distracts
from the actual CI error.
Currently we attempt to setup the build environment even when
dependencies are not installed, which typically results in error while
searching for libraries or executables in a dependency's prefix.
With this change, we get a more user friendly error:
```
$ spack build-env perl
==> Error: Not all dependencies of perl are installed, cannot setup build environment:
- qpj6dw5 perl@5.36.0%apple-clang@14.0.0+cpanm+open+shared+threads build_system=generic arch=darwin-ventura-m1
- jq2plbe ^berkeley-db@18.1.40%apple-clang@14.0.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=darwin-ventura-m1
...
$ echo $?
1
```
* Allow users to specify root env dir
Environments managed by spack have some advantages over anonymous Environments
but they are tucked away inside spack's directory tree. This PR gives
users the ability to specify where the environments should live.
See #32823
This is also taken as an opportunity to ensure that all references are to "managed environments",
rather than "named environments". Prior to this PR some references to the latter persisted.
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Gregory Becker <becker33@llnl.gov>
* Update exago w/ 1.5.1 and small updates to hiop.
* Fix styling.
* Add RAJA back to ExaGO package.
* Update RAJA requirement for ExaGO and HiOp.
* Update last RAJA requirement in HiOp.
* Add new sphinx rtd theme release 1.2.0
The new release helps with supporting more recent version of docutils
* set docutils officially supported version
* add jquery dependency for sphinx-rtd-theme
* add conflict with jquery version
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* correct dependency
* fix version dependency
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* set sphinx version
* fix sha256
* add version for flit-core
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The call:
```
x.satisfies(y[, strict=False])
```
is commutative, and tests non-empty intersection, whereas:
```
x.satsifies(y, strict=True)
```
is not commutative, and tests set-inclusion.
There are 2 fast paths. When strict=False both self and other need to
be concrete, when strict=True we can optimize when other is concrete.
a) It's used by site administrators, so it's niche
b) If it's used by site administrators, they likely need to modify the config anyhow, so the default config only serves as an example to get started
c) it's too arbitrary to enable tcl, but disable lmod
Spack generally ignores file-file projection clashes in environment
views, but would eventually error when linking the `.spack` directory
for two specs of the same package.
This leads to obscure errors where users have no clue what the issue is
and how to fix it. On top of that, the error comes very late, since it
happens when the .spack dir contents are linked (which happens after
everything else)
This PR improves that by doing a quick check ahead of time if clashes
are going to be anticipated (by simply checking for clashes in the
projection of each spec's .spack metadir). If there are clashes, a
human-readable error is thrown which shows two of the conflicting specs,
and tells users to user unify:true, view:false, or set up custom
projections.
* add pytng
* black
* add setuptools
* fix
* Update var/spack/repos/builtin/packages/py-pytng/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pytng/package.py
* Update var/spack/repos/builtin/packages/py-pytng/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Kokkos when compiled by spack without +wrapper could potentially capture the spack compiler wrappers, resulting in cmake configs and kokkos_launch_compiler trying to run the spack compiler wrapper after installation.
The checksum exception was not detailed enough and not reraised when using cache only, resulting in useless error messages.
Now it dumps the file path, expected
hash, computed hash, and the downloaded file summary.
Batch scripts in general will not function without carriage return line
endings on Windows. We rely on these scripts to support cmd, so we
should not allow these scripts to be converted to lf.
Note: Windows 11 supports lf line endings due to the use of Windows
Terminal. Once support for Windows 10 is dropped, this change can be
reverted.
When running many concurrent spack install processes that need to write
to the db, Spack regularly times out. This is because writing to the DB
after another process has written to it requires deserialization of the
db, mutating it in memory, and serializing it again, which takes some
time. On top of that, I believe there's a 1 second retry when a write
lock cannot be obtained, so I think this means only 3 processes can
really write to the DB at the same time before timing out.
* Style: black 23, skip magic trailing commas
* isort should use same line length as black
* Fix unused import
* Update version of black used in CI
* Update new packages
* Update new packages
* Update package.py
Initial new stuff
* Update package.py
* Update package.py
* Update package.py
* fix targets
* non-llvm backends
* ooops
* fix style
* Somehow that was not caught?
Somehow that was not caught?
* style
* Last fix
make capitalization consistent with Halide not LLVM...
* py-cmake-format: new version, new variants
* Update var/spack/repos/builtin/packages/py-cmake-format/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-cufflinks: new package version with 0.17.3
* Update var/spack/repos/builtin/packages/py-cufflinks/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Specs that did not contribute any files to an env view caused a problem
where zip(specs, files grouped by prefix) got "out of sync", causing the
wrong merge map to be passed to a package's `add_files_to_view`, which
specifically caused an issue where *sometimes* bin/python ended up as a
symlink instead of a copy.
One such example is kokkos + kokkos-nvcc-wrapper, as the latter package
only provides the file bin/nvcc_wrapper, which is also added to view by
kokkos, causing kokkos-nvcc-wrapper to contribute 0 files.
The test feels a bit contrived, but it captures the problem... pkg a is
added first and has 0 files to contribute, pkg b adds a single file, and
we check if pkg b receives a merge map (and a does not).
* pfunit: add v4.6.3
* pfunit: use CMakePackage methods to define arguments
* pfunit: deprecate v3.X, make a variant conditional
* pfunit: simplify setting up environment variables
Reading the docs it seems only v3
needs F90_VENDOR to be set
* pfunit: fix option names
The names set before were unused
* pfunit: shared libraries seem not to be supported
See https://github.com/Goddard-Fortran-Ecosystem/pFUnit/issues/308#issuecomment-874725759
* Add py-mlflow and its dependencies
* mlflow: fix syntax error in package.py
* py-mlflow: cleanup
Process review remarks, add missing dependencies, add skinny variant
* Apply suggestions from code review
* Fix flake8 issues
* More formatting fixes
* Fix py-waitress dependency version
* py-mlflow: platform-specific dependency
* Update var/spack/repos/builtin/packages/py-mlflow/package.py
* Update var/spack/repos/builtin/packages/py-mlflow/package.py
* Process review remarks
* Fix typo in dependency version
* py-shap: fix dependencies
* py-arrow: fix dependencies
* py-slicer: remove py-setuptools explicit version
* py-pyarrow: dataset variant and pass options through environment
It appears there are some issues when using `pip install` instead of
`python setup.py` - this setup_build_environment should fix that.
* py-pyarrow: review remark
* Decouple setup_build_environment from install_options
* py-pyarrow: style
* Bump licenses to 2023
---------
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Matthias Wolf <matthias.wolf@epfl.ch>
`spack gc` removes build deps of explicitly installed specs, but somehow
if you take one of the specs that `spack gc` would remove, and feed it
to `spack uninstall /<hash>` by hash, it complains about all the
dependents that still rely on it.
This resolves the inconsistency by only following run/link type deps in
spack uninstall.
That way you can finally do `spack uninstall cmake` without having to
remove all packages built with cmake.
Default package requirements might contain
variants that are not defined in each package,
so we shouldn't verify them when emitting facts
for the ASP solver.
Account for group when enforcing requirements
packages:all : don't emit facts for requirement conditions
that can't apply to current spec
* Update package.py
Several libraries are need to be present at run time so that the code can be run in parallel.
I have added them as dependencies and to LD_LIBRARY_PATH. Orca comes as a binary so the libraries cannot be added as RPATH at compilation time.
Also, orca 5.0.3 was compiled against 4.1.1, not 4.1.2.
* fortls
* Update var/spack/repos/builtin/packages/py-fortls/package.py
* review
* Update var/spack/repos/builtin/packages/py-fortls/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixes
* review
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* new py-amplpy package
* [@spackbot] updating style on behalf of sm2939
* Update package.py
* Rename var/spack/repos/builtin/py-amplpy/package.py to var/spack/repos/builtin/packages/py-amplpy/package.py
* Edited file to change copyright year/dependencies and changed the directory of the file
---------
Co-authored-by: Sangu Mbekelu <s.mbekelu9@gmail.com>
#35098 added the correct extraction of toolset version for the MSVC
compiler. This updates the associated method in MSBuilder to retrieve
the (now correct) property.
Meme 4.5.0 has the first occurrence of the string
```
use XML::Simple
```
I found this by doing a binary search manually extracting tarballs until `grep` came up empty.
* new ampltools package
* [@spackbot] updating style on behalf of sm2939
* Update and rename var/spack/repos/builtin/py-ampltools/package.py to var/spack/repos/builtin/packages/py-ampltools/package.py
Edited file to change copyright year/dependencies and edited directory
---------
Co-authored-by: Sangu Mbekelu <s.mbekelu9@gmail.com>
* Add Score-P 8.0 and Cube 4.7/4.8 packages.
* Score-P 8.0 requires 4.8, not 4.7, Cube packages
* Add maintainer
* Add CUDA and HIP variants. Add version checks for CUDA (Score-P 8 requires CUDA 7), ROCm (variant only valid as of Score-P 8), and MPI (Score-P 7 requires at least version 2.2 of the MPI standard).
* Deprecate everything pre-7.0.
* Fix HIP dependencies and enable CUDA and HIP variants for configure.
* Deprecate OTF2 pre-2.3 and Cube pre-4.6
* Add "fake" mpi compiler wrappers to msmpi: msmpi doesn't actually
provide wrappers, so this just assigns the wrappers to be whatever
compiler that a dependent is using. Packages referencing the
wrappers would otherwise break. This is assumed to be workable
because build scripts will need to assemble appropriate information
to pass to the compiler anyway
* Fix msmpi detection stanza ('executable' is not the correct name of
the property)
* Fix compiler pkg dereference
* add initial package.
* Update package.py
* Update var/spack/repos/builtin/packages/tiramisu/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/tiramisu/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/tiramisu/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Hopefully this will be fine.
* Update package.py
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/tiramisu/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* cvise: new package
* cvise: colordiff as optional dependency
* cvise: remove old versions and correctly name master version
* cvise: update license date
* cvise: use maintainers directive
* Remove @olupton as maintainer
After live discussion: it's been too long since he did anything with this package.
* add halide package.
* some style changes.
* small fix
* Update var/spack/repos/builtin/packages/halide/package.py
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* Update var/spack/repos/builtin/packages/halide/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/halide/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
add comment to requirements.txt
* Update package.py
Fix version order.
* Update package.py
style
* Update package.py
Removed unneeded vars.
* Update var/spack/repos/builtin/packages/halide/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/halide/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Fix some deps
* Update package.py
* Fix finding llvm cmake info
* Update package.py
---------
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This PR enables the successful execution of the spack binary cache
tutorial on Windows. It assumes gnupg and file are available (they
can be installed with choco).
* Fix handling of args with quotes in spack.bat
* `file` utility can be installed on Windows (e.g. with choco): update
error message accordingly
I don't know if this is new in version 7.0, but to build `info`, which is a required executable at the end of the recipe, it is necessary to have a terminal library, otherwise you get
```
[...]
checking for tgetent in -ltinfo... no
checking for tgetent in -lncurses... no
checking for tgetent in -lcurses... no
checking for tgetent in -ltermlib... no
checking for tgetent in -ltermcap... no
checking for tgetent in -lterminfo... no
configure: WARNING: info needs a terminal library, one of: tinfo ncurses curses termlib termcap terminfo
[...]
configure: WARNING: Could not find a terminal library among tinfo ncurses curses termlib termcap terminfo
configure: WARNING: The programs from `info' directory will not be built.
```
then compilation runs, `info` is not built and installation fails according to Spack because the required executable is missing.
* Support packages for using scitokens on OSG
The Open Science Grid (OSG) encourages scitokens to provide
certain services (e.g. writing to xrootd). Spack already
supports this through scitokens-cpp and xrootd +scitokens-cpp.
This adds py-htgettoken, a python utility to get a scitoken
from a vault through web authentication. To support htgettoken,
this also adds py-gssapi.
This also adds the OSG CA cert collection which is typically
at /etc/grid-security but pointed to in user installations by
the X509_CERTS_DIR variable.
This allows userspace through spack for functionality that
otherwise depends on installing the RPMs provided by OSG.
* fine, I'll fix style myself then
* fix maintainers
* py-gssapi: version before depends_on
* remove list_url
* add documentation on reason for git describe version numbers
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* better BEARER_TOKEN definition
* import os
* remove older version that don't build with setuptools
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
At least with ZSH, prefix inspections containing `./bin` result in a
`$PREFIX/./bin` and result in strange `$PATH` handling.
I.e., `module load git` will prepend `/path/to/git/./bin`, `which git`
will find the right executable, but `git --version` will print the
system one. Normalize the relative path to avoid this behavior.
See also spack/spack#31867.
* changes to enable LLVM_ENABLE_RUNTIMES for libcxx and libcxxabi
* remove version update for 5.3.0 as it is done thru PR #33320 to enable
ci and reviews
* initial commit for rocm-5.4.0 release
* update the versions for more packages for 5.4.0 release
* update the gallium patch for mesa for libllvm-15 for ROCm-5.4.0 release
* update rocm-openmp-extras and rocwmma recipes for 5.4.0 release
* fix build error for rocfft for 5.4.0
* address review comments for rocfft for 5.4.0 change
* undo the removal of the older patch file
* bump up the version for hipfft for 5.4.0
* fix the failure after the merge with develop
* add recipes updates for 5.4.0 for migraphx.miopen-hip,miopen-opencl
* address the review comments on the mesa patch.update the rdc package for
5.4.0 release
* fix style errors
* acts: new versions 21.1.1, 22.0.1, 23.0.0
New versions:
- [major 23.0.0](https://github.com/acts-project/acts/compare/v22.0.0...v23.0.0):
- new option `ACTS_BUILD_PLUGIN_GEANT4` -> enabled with existing variant `geant4`
- new option `ACTS_BUILD_EXAMPLES_BINARIES`:
- it is my understanding that the binaries for examples are deprecated (in favor of python examples); warnings to this effect have been printed for a few versions, and now the building of binaries is disabled by default,
- rather than introducing a variant to enable deprecated behavior for only one or two versions, I propose that we just follow the default and keep this disabled.
- [bugfix 22.0.1](https://github.com/acts-project/acts/compare/v22.0.0...v22.0.1) (no build system changes)
- [bugfix 21.1.1](https://github.com/acts-project/acts/compare/v21.1.0...v21.1.1) (no build system changes)
* acts: correct 23.0.0 sha
Co-authored-by: Hadrien G. <knights_of_ni@gmx.com>
As of 2.4.113, the flag for man-pages is now a feature,
so true/false is now enabled/disabled. Other similarly
changed options are not used in the spack recipe (i.e.
experimental kms drivers).
* qt: new versions 6.4.0, 6.4.1
- New libpsl vendored dependency in qt-base.
- New embree and tinyexr dependency in qt-quick3d.
We need to figure out a better way to deal with these vendored
dependencies in src/3rdparty. Removing them was a way to make sure
they are not used unintentionally. Many of these dependencies cannot
be overridden with a QT_FEATURE_system_* flag and are included directly
in cpp files. Many change versions from release to release, so even if
they use system (ie spack managed) versions we need to support this in
the depends_on lines.
What we can rely on?
- src/3rdparty is where vendored stuff is stored
- not much else...
Possible ways to deal with this:
- Change vendor_deps_to_keep to dict with versions, eg
```
vendor_deps_to_keep = {
"xatlas": "@6:",
"embree": "@6.4:",
"tinyexr": "@6.4:",
}
```
- Similarly introduce system_deps_to_use:
```
system_deps_to_use = {
"assimp@5.2:": "@6:",
}
```
and derive depends_on and QT_FEATURE_system_* from this dict.
* qt-*: new version 6.4.2, invert vendored pkgs logic
* qt-base: fix vendor_deps_to_avoid typo
* qt-*: move lots into QtPackage base layer
* py-minkowskiengine: new package (sparse tensor autodiff by Nvidia)
This python package (with cuda support) provides torch support for sparse
tensors. The `pybind11` headers are not found without the patch to `setup.py`.
* [@spackbot] updating style on behalf of wdconinc
* py-minkowskiengine: depends_on numpy, pybind11 type=link; no patch
* [@spackbot] updating style on behalf of wdconinc
---------
Co-authored-by: wdconinc <wdconinc@users.noreply.github.com>
The fastqc script was using the system perl. This PR sets the script to
use the spack built/provided perl. This PR also removes the code that
adds the java path. That should be handled by module loading as far as I
know.
* Add HDF5 version 1.13.3.
* Remove maintainers no longer with The HDFGroup.
* Add version hdf5-vol-async@1.4
* Add HDF5 version 1.14.0, develop-1.14, develop-1.15.
Add missing conflicts for api version and develop versions.
* Add conflicts statement to hdf5/package.py to avoid building hdf5 with
MPICH 4.0.x versions with bug that causes testphdf5 test to fail.
* Add patch to call find_package(MPI) for dependent packages not finding
it, not having called it themselves.
* Remove language components from find_package(MPI) in
hdf5_1_14_0_config_find_mpi.patch.
* Add HDF5 version 1.14.0, develop-1.14, develop-1.15.
Add missing conflicts for api version and develop versions.
* Add conflicts statement to hdf5/package.py to avoid building hdf5 with
MPICH 4.0.x versions with bug that causes testphdf5 test to fail.
* Add patch to call find_package(MPI) for dependent packages not finding
it, not having called it themselves.
* Remove language components from find_package(MPI) in
hdf5_1_14_0_config_find_mpi.patch.
* Don't guard ParaView patch on HDF5 variant
ParaView always needsd HDF5 and ignores the variant.
* py-h5py: Newer versions of HDF5 introduce breaking API changes
---------
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Ryan Krattiger <ryan.krattiger@kitware.com>
* Add trilinos-solvers variant to nalu-wind package.
This allows nalu-wind to be built against a trilinos installation
which doesn't have amesos2, belos, ifpack2, or muelu enabled, if
the nalu-wind user provides the spec 'nalu-wind@master~trilinos-solvers'
Support for these solver-packages remains on by default.
* Fixed a style issue reported by CI.
* Incorporate change in wording suggested from review comments.
... to clarify that at least one, or both, of hypre and/or
trilinos-solvers must be enabled. The error condition is if
both are disabled.
* That style checker is picky...
* It really did want a trailing comma...
* py-jinja2-cli: new package
* Update var/spack/repos/builtin/packages/py-jinja2-cli/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
---------
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add py-docker@5:
* [@spackbot] updating style on behalf of spoutn1k
* Ignore `tls` variant
* Update var/spack/repos/builtin/packages/py-docker/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* `py-docker`: `py-paramiko` version fix
---------
Co-authored-by: spoutn1k <spoutn1k@users.noreply.github.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The gfx906:xnack- and gfx908:xnack- targets were introduced in ROCm 4.1
and replaced gfx906 and gfx908 as default build targets, but the library
can still be built for gfx906 and gfx908 if requested.
* e4s: restore builds builds
* gitlab ci: allow UO to build protected binaries for signing
* use newer image; comment out failing builds
* gitlab-ci: Some tweaks for e4s power builds
- fix tags (no longer require generate jobs to run on aws)
- fix resource requests for generation jobs resource requests
- remove SPACK_SIGNING_KEY from protected power build jobs
- update UO signing key path
- change the CDash build group to reflect stack name
- retry pipeline generation jobs *always*
* correct double packages: section
* gitlab-ci:script: modernize
* remove new gnu make, not for ppc64le
---------
Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
* Added e4s-cl package
* Version order change
* Added e4s-cl dependencies
* Added python-sotools dependency
* [@spackbot] updating style on behalf of spoutn1k
* Add missing versions to py- packages
* Fix style
* [@spackbot] updating style on behalf of spoutn1k
* Update var/spack/repos/builtin/packages/e4s-cl/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/e4s-cl/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-python-sotools/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add docker removing patch for e4s-cl
Co-authored-by: spoutn1k <spoutn1k@users.noreply.github.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-nexusforge: add with dependencies
* py-pyshacl, py-sseclient: more style
* py-hjson, py-nexus-sdk, py-nexusforge, py-puremagic: more style
* py-pyshacl: license update
* py-nexusforge, py-prettytable, py-pyshacl: review remarks
* py-nexusforge: make the variant mean something
Too hasty to commit...
* py-ipyparallel: add 8.4.1, which builds with py-hatchling
* py-ipyparallel: copyright and redundant py-setuptools dependency
* py-ipyparallel: py-packaging was dropped after 8.0.0
fixes#34879
This commit adds a new maintainer directive,
which by default extend the list of maintainers
for a given package.
The directive is backward compatible with the current
practice of having a "maintainers" list declared at
the class level.
Move the relocation of binary text in its own class
Drop threaded text replacement, since the current bottleneck
is decompression. It would be better to parallellize over packages,
instead of over files per package.
A small improvement with separate classes for text replacement is that we
now compile the regex in the constructor; previously it was compiled per
binary to be relocated.
The regex doesn't actually work because dollar signs and parentheses have to be
escaped. Also, compiling with OpenMPI requires defining the macro
`MPI2SUPPORT`.
This commit makes explicit the format version of the spec file
we are reading from.
Before there were different functions capable of reading some
part of the spec file at multiple format versions. The decision
was implicit, since checks were based on the structure of the
JSON without ever checking a format version number.
The refactor makes also explicit which spec file format is used
by which database and lockfile format, since the information is
stored in global mappings.
To ensure we don't change the hash of old specs, JSON representations
of specs have been added as data. A unit tests checks that we read
the correct hash in, and that the hash stays the same when we
re-serialize the spec using the most recent format version.
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
```
File ".../spack/var/spack/environments/scale-mpi/.spack-env/._view/4yiorsdd4pefrnwgrwlwt3yzo5i235il/lib/python3.10/site-packages/h5py/_hl/base.py", line 19, in <module>
from collections import (Mapping, MutableMapping, KeysView,
ImportError: cannot import name 'Mapping' from 'collections' (.../spack/var/spack/environments/scale-mpi/.spack-env/._view/4yiorsdd4pefrnwgrwlwt3yzo5i235il/lib/python3.10/collections/__init__.py)
```
Fixed in https://github.com/h5py/h5py/pull/1069 which was first merged
in v2.9.
* py-flatten-dict: require poetry to build.
The sources seem to contain a bundled, auto-generated `setup.py`.
Building with `pip` insist on using Poetry as mentioned in
`pyproject.toml`, so require it as a build dependency.
* Update var/spack/repos/builtin/packages/py-flatten-dict/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-flatten-dict/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Currently we print "sha256 checksum failed for [file]. Expected X but
got Y".
This PR extends that message with file size and contents info:
"... but got Y. File size = 123456 bytes. Contents = b'abc...def'"
That way we can immediately see if the file was downloaded only
partially, or if we downloaded a text page instead of a binary, etc.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
py-scipy 1.6 and older come with pre cython-ized files that
use the _PyGen_Send symbol that was removed from python 3.10.0.161,
so do not build these old versions with python 3.10.1 and later
Parts of libgcrypt should not be optimized with -O1/2/3, so it's best to
let the build system do that; the build system cannot know the compiler
wrapper would inject optimization flags
When running unit-test the test/ci.py module is leaving
garbage (help.sh, test.sh files) in the current working
directory.
This commit changes the current working directory to a
temporary path before those files are created.
* freeimage: fails to compile with c++17, use c++14
Only `opencascade` when a (non-default) variant depends on `freeimage`, which seems to have gone unmaintained. There are c++17 standard violations [[1]]( https://en.cppreference.com/w/cpp/language/except_spec) in the code, so we can at most expect c++14. Since some compilers default to c++17 (gcc-12) we need to be explicit.
* freeimage: install directly in prefix
* freeimage: fix inverted patch
* environments: don't rewrite relative view path, expand path on cli ahead of time
Currently if you have a spack.yaml that specifies a view by relative
path, Spack expands it to an absolute path on `spack -e . install` and
persists that to disk.
This is rather annoying when you have a `spack.yaml` file inside a git
repo, cause you want to use relative paths to make it relocatable, but
you constantly have to undo the changes made to spack.yaml by Spack.
So, as an alternative:
1. Always stick to paths as they are provided in spack.yaml, never
replace them with a canonicalized version
2. Turn relative paths on the command line into absolute paths before
storing to spack.yaml. This way you can do `spack env create --dir
./env --with-view ./view` and both `./env` and `./view` are resolved
to the current working dir, as expected (not `./env/view`). This
corresponds to the old behavior of `spack env create`.
* create --with-view always takes a value
All packages with explicit Windows support can be found with
`spack list --tags=windows`.
This also removes the documentation which explicitly lists
supported packages on Windows (which is currently out of date and
is now unnecessary with the added tags).
Note that if a package does not appear in this list, it *may*
still build on Windows, but it likely means that no explicit
attempt has been made to support it.
* nextflow recipe: added latest stable version
* tower-cli recipe: added latest release
* recipes tower-agent and tower-cli renamed to nf-tower-agent and nf-tower-cli
* recipes nf-tower-agent and nf-tower-cli: small fix
* nf-core-tools recipe: added most py- dependencies
* nf-core-tools: recipe without galaxy-tool-util (for testing)
* fixed typos in py-yacman recipe
* fixed typos in py-pytest-workflow recipe
* fixed typo in nf-core-tools recipe
* fixed typos in py-yacman recipe
* fixes in recipes for py-questionary and py-url-normalize
* fixes to py-yacman recipe
* style fixes to py- packages that are dependencies to nf-core-tools
* fix in py-requests-cache recipe
* added missing dep in py-requests-cache recipe
* nf-core-tools deps: removed redundant python dep for py packages oyaml and piper
* nf-core-tools recipe: final, incl dep on py-galaxy-tool-util
* nf-core-tools: new version with extra dependency
* added py-galaxy-util, draft: added some required dep versions, still have to add 40+ deps
* nextflow and nf-core-tools packages: added my self as maintainer
* style fixes
* style fix for nf-core-tools recipe
* added license to py-logmuse recipe
* audit fixes
* style fix after audit fix
* py-galaxy-tool-util: added deps 1st bunch
* audit/style fixes, including adding missing dep package
* more audit/style fixes
* more more audit/style fixes
* moooore audit fixes
* py-galaxy-tool-util: dependencies 2nd chunk
* silly audit fix
* py-galaxy-util deps: 3rd bunch - first 20 done
* fixes
* style fix
* py-galaxy-tool-util: 4th bunch of deps
* stashing dep recipe backbones for py-galaxy-tool-util
* nf-core-tools: using pre-built wheel for dependency py-galaxy-tool-util
* nf-core-tools: adding also py-galaxy-util, as wheel
* fix
* nextflow: added latest bugfix version
* Update var/spack/repos/builtin/packages/nf-core-tools/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/nf-core-tools/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* nf-core-tools pr: 1st bunch of review edits
* nf-core-tools: 2nd bunch of review edits
* adding back tower-agent and tower-cli as deprecated
* nf-core-tools: 3rd bunch of review edits
* small style fix
* prepping py-galaxy-tool-util for further work
* nf-core-tools: last bunch of deps, except for galaxy-tool-util and pulsar
* audit fixes
* updates to py-galaxy-tool-util and its deps, still 2 to work on
* one style fix
* updated recipe for py-galaxy-util
* updated recipe for py-pulsar-galaxy-lib
* typo fix
* shasum fixes
* updated py-sqlalchemy from develop
* added newest versions (today) for nf-tower-agent and nf-tower-cli
* Update var/spack/repos/builtin/packages/py-requests-cache/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-requests-cache/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* adding 2nd bunch of nf-core deps from update/nextflow-tools
* adding 3rd bunch of nf-core deps from update/nextflow-tools
* 4th chunk of nf-core deps from update/nextflow-tools
* Update var/spack/repos/builtin/packages/py-dnspython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-dnspython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-dnspython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi-utils/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pastedeploy/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pebble/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-gunicorn/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-starlette/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-starlette/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-starlette/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-parsley/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-paste/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-paste/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-gxformat2: added comment
* py-lagom: now using github tarballs
* fix for py-lagom
* adding missing deps to py-fastapi-utils
* another fix to py-lagom
* Update var/spack/repos/builtin/packages/py-dnspython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi-utils/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi-utils/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi-utils/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-lagom/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-supervisor/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-social-auth-core/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixes from PR review
* adding missing deps, from PR review
* py-galaxy2cwl from github tarball, as per PR review
* fix to py-tuswsgi, as per PR review
* nf-tools: edits from PR review
* adding 3x more galaxy deps
* fix
* fixing circular dep of py-poetry-plugin-export with py-poetry
* added newest nf-core-tools version
* Update var/spack/repos/builtin/packages/py-galaxy-util/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fix in py-poetry-plugin-export
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Currently, the `python` package tries to set `CPATH` in `setup_run_environment()`.
We no longer set `CPATH` and other destructive environment variables (like
`LD_LIBRARY_PATH`) in modules, so we shouldn't do something special for Python.
Also, the way `python` sets `CPATH` causes issues. Because it does a header search to
find directories containing headers, if you bootstrap `mypy` or other style tools on a
fresh Ubuntu image with *no* python devel headers installed, you'll get an error like
this when trying to load the thing you just installed:
```console
[root@980de539843d /]# spack -b load py-mypy
==> Error: Unable to locate python headers in any of these locations:
/usr/include/python3.6m
/usr/include/3.6
/usr/Headers
```
The headers and includes aren't needed to get `mypy` in the path or for `mypy` to work,
so we're failing unnecessarily here.
- [x] remove `setup_run_environment()` from `python/package.py`
Since SPACK_PACKAGE_IDS is now also "namespaced" with <prefix>, it makes
more sense to call the flag `--make-prefix` and alias the old flag
`--make-target-prefix` to it.
1. add variant cray-static, older crays build hpcprof-mpi static,
newer ones build dynamic.
2. move URL patches from github to gitlab.
3. add workaround for a bug where a file is mistakenly overwritten.
4. add conflict for hpcprof-mpi at 2022.10.01.
* [@spackbot] updating style on behalf of mwkrentel
Co-authored-by: mwkrentel <mwkrentel@users.noreply.github.com>
Normally when using external packages in concretization, Spack ignores
all dependencies of the external. #33777 updated this logic to attach
a Python Spec to external Python extensions (most py-* packages), but
as implemented there were a couple issues:
* this did not account for concretization groups and could generate
multiple different python specs for a single DAG
* in some cases this created a fake Python spec with insufficient
details to be usable (concretization/installation of the
extension would fail)
This PR addresses both of these issues:
* For environment specs that are concretized together, external python
extensions in those specs will all be assigned the same Python spec
* If Spack needs to "invent" a Python spec, then it will have all the
needed details (e.g. compiler/architecture)
* npm: Add latest version, update build
The `npm` package had gotten a bit long in the tooth and only suported the last version
for which running `configure` / `make` / `make install` actually worked.
- [x] Update the package to support npm@9, in which `npm install .` works properly and
installation is easier.
- [x] Update the package so that `npm@6:8` also install successfullly. The incantation
that is *supposed* to work on these versions is `node bin/npm-cli.js install $(node
bin/npm-cli.js pack . | tail -1)`, but depending on the version one of `npm install`
or `npm pack` will fail when run straight from the install directory. So now we just
manually copies things over.
This seems to make the `npm` install much more reliable for all of `npm@6:9` (at least
for me).
udpates the `npm` build to support versions 6-9 and fixes the install for all of
them on macos.
* update for review
With the new variable [prefix/]SPACK_PACKAGE_IDS you can conveniently execute
things after each successful install.
For example push just-built packages to a buildcache
```
SPACK ?= spack
export SPACK_COLOR = always
MAKEFLAGS += -Orecurse
MY_BUILDCACHE := $(CURDIR)/cache
.PHONY: all clean
all: push
ifeq (,$(filter clean,$(MAKECMDGOALS)))
include env.mk
endif
# the relevant part: push has *all* example/push/<pkg identifier> as prereqs
push: $(addprefix example/push/,$(example/SPACK_PACKAGE_IDS))
$(SPACK) -e . buildcache update-index --directory $(MY_BUILDCACHE)
$(info Pushed everything, yay!)
# and each example/push/<pkg identifier> has the install target as prereq,
# and the body can use target local $(HASH) and $(SPEC) variables to do
# things, such as pushing to a build cache
example/push/%: example/install/%
@mkdir -p $(dir $@)
$(SPACK) -e . buildcache create --allow-root --only=package --unsigned --directory $(MY_BUILDCACHE) /$(HASH) # push $(SPEC)
@touch $@
spack.lock: spack.yaml
$(SPACK) -e . concretize -f
env.mk: spack.lock
$(SPACK) -e . env depfile -o $@ --make-target-prefix example
clean:
rm -rf spack.lock env.mk example/
``
* [armpl-gcc] Make pkg-config files available
ARMpl pkgconfig files are located in a non-default location and do not have the
.pc extension. Changing those both helps pkgconfig pick them up correctly, e.g.
in the meson build of `py-scipy`.
* Address @annop-w comments
* symlink instead of cp.
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
In the past we checked remote binary mirrors for existence of a spec
before attempting to download it. That changed to only checking local
copies of index.jsons (if available) to prioritize certain mirrors where
we expect to find a tarball. That was faster for CI since fetching
index.json and loading it just to order mirrors takes more time than
just attempting to fetch tarballs -- and also if we have a direct hit
there's no point to look at other mirrors.
Long story short: the info message only makes sense in the old version
of Spack, so it's better to remove it.
* py-sphinx-immaterial: new package
This is a new-ish theme for Sphinx that's based on MkDocs's `immaterial` theme. More on
the theme here: https://jbms.github.io/sphinx-immaterial/, but it seems to be very clear
and readable. We *might* consider switching to it for Spack's docs.
* Update var/spack/repos/builtin/packages/py-sphinx-immaterial/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-sphinx-immaterial/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-sphinx-immaterial/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-sphinx-immaterial/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* add note about node.js requirements
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Forward lookup of "test_log_file" and "test_failures"
refers #34531closes#34487fixes#34440
* Add unit test
* py-libensemble: fix tests
* Support stand-alone tests with cached files as install-time tests
Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
Ensure `spack mirror add <name> <url/path>` without further arguments translates to `<name>: <url>` key value pairs in mirrors.yaml. If --s3-* flags are provided, only store the provided ones.
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
This is only a work-around for the actual problem that Python is used to install libraries instead of CMake, so we end up with BUILD_RPATH not INSTALL_RPATHs.
Signed-off-by: Loïc Pottier <pottier1@llnl.gov>
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
* Add new file for MVAPICH 3.0a release
Creating this as a new package since it requires some new configuration
options and because we are moving to the name "MVAPICH" and droping the
2 (following a similar move by MPICH).
Co-authored-by: Nat Shineman <shineman.5@osu.edu>
Co-authored-by: Matthew Lieber <lieber.31@osu.edu>
* OpenCV: checksum for 4.5.5, make contrib optional
* [@spackbot] updating style on behalf of iarspider
* Add conflicts for contrib modules
* Fix typo
* Implement changes from review
* Update var/spack/repos/builtin/packages/opencv/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: iarspider <iarspider@users.noreply.github.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Revert "Revert "4th chunk of nf-core deps from update/nextflow-tools (#34564)" (#34960)"
This reverts commit 891a63cae6.
* fix to py-python-multipart, as per PR review
* py-macs2: add version 2.2.7.1 and support python@3.10:
The tarball from PyPi includes the Cythonized C files. The tarball from
github does not. Remove the Cythonized C files from the source so that
they are rebuilt with the Spack Python/Cython combination. This is
necessary for python-3.10 but make sense for other combinations as well.
* Edits based on review
- set python version constraint on version 2.2.4
- removed all python-2 versions and related constraints.
* Add package file for xtb and py-xtb
* Retain maintainers from PythonPackage
* Update package files
- use extends("python") instead of tampering with PYTHONPATH
- use PyPI for downloading sdist of py-xtb
- add simple-dftd3 0.7.0
- add dftd4 3.5.0
- remove --wrap-mode=nodownload from toml-f
* Remove logic for download URL
With this change we get the invariant that `mirror.fetch_url` and
`mirror.push_url` return valid URLs, even when the backing config
file is actually using (relative) paths with potentially `$spack` and
`$env` like variables.
Secondly it avoids expanding mirror path / URLs too early,
so if I say `spack mirror add name ./path`, it stays `./path` in my
config. When it's retrieved through MirrorCollection() we
exand it to say `file://<env dir>/path` if `./path` was set in an
environment scope.
Thirdly, the interface is simplified for the relevant buildcache
commands, so it's more like `git push`:
```
spack buildcache create [mirror] [specs...]
```
`mirror` is either a mirror name, a path, or a URL.
Resolving the relevant mirror goes as follows:
- If it contains either / or \ it is used as an anonymous mirror with
path or url.
- Otherwise, it's interpreted as a named mirror, which must exist.
This helps to guard against typos, e.g. typing `my-mirror` when there
is no such named mirror now errors with:
```
$ spack -e . buildcache create my-mirror
==> Error: no mirror named "my-mirror". Did you mean ./my-mirror?
```
instead of creating a directory in the current working directory. I
think this is reasonable, as the alternative (requiring that a local dir
exists) feels a bit pendantic in the general case -- spack is happy to
create the build cache dir when needed, saving a `mkdir`.
The old (now deprecated) format will still be available in Spack 0.20,
but is scheduled to be removed in 0.21:
```
spack buildcache create (--directory | --mirror-url | --mirror-name) [specs...]
```
This PR also touches `tmp_scope` in tests, because it didn't really
work for me, since spack fixes the possible --scope values once and
for all across tests, so tests failed when run out of order.
Packages that use docbook-xml may specify a specific entity version.
When this is specified as a version constraint in the package recipe it
will cause problems when using `unify = True` in a Spack environment, as
there could be multiple versions of docbook-xml in the spec. In
practice, any entity version should work with any other version and
everything should work with the latest version. This PR maps all Spack
docbook-xml entity versions to the docbook-xml version in the spec.
Ideally, the version in the spec would be the latest version. With this
PR, even if a package specifies an older entity version, it will map
to the entity version (latest) in the spec. This means that there can be one
docbook-xml version in a Spack environment spec and packages requesting
older entity versions will still work.
To help facilitate this, docbook-xml version constraints for packages
that have them have been removed. Those packages are dbus and gtk-doc.
What's in AOCL 4.0:
1. amdblis
LPGEMM variants with post-ops support
AMD "Zen4" support for BLIS
2. amdlibflame
Upgrade to LAPACK 3.10.1 specification
Improvements in a few more variants of SVD and Eigen Value routines
Multithread support enabled for selected APIs
3. amdfftw
AVX-512 enablement of DFT kernels
AVX-512 optimization of copy and transpose routines
5. amdlibm
Black & Scholes support (logf, expf, erff, both scalar and vector)
AVX-512 variants of vector functions
6. aocl-sparse
New Iterative Solver APIs
AVX-512 support for SPMV API
7. amdscalapack
Upgrade to Netlib ScaLAPACK 2.2.0
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Sometimes I just want to know how many packages of a certain type there are.
- [x] add `--count` option to `spack list` that output the number of packages that
*would* be listed.
```console
> spack list --count
6864
> spack list --count py-
2040
> spack list --count r-
1162
```
* Add packages
* Style
* Respond to comments
* Change conflct dep type
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* paraview: add `rocm` variant
This conflicts with CUDA and requires at least ParaView 5.11.0. More
dependencies are also needed.
* E4S: Add ParaView for ROCm and CUDA stacks
* DAV SDK: Update ParaView version and GPU variants
* Verify using hipcc vs amdclang++ for newer hip
Co-authored-by: Ben Boeckel <ben.boeckel@kitware.com>
* adding 1st version of py-strawberryfields
* py-strawberryfields: minor edit
* added backbone for 4x new SF dependencies
* edits to SF and its 4x new deps
* added all deps for SF
* added one version to py-lark-parser
* py-quantum-xir with tarball from github
* Update var/spack/repos/builtin/packages/py-strawberryfields/package.py
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* updated version for py-quantum-xir : pypy fixed
* py-pennylane: added pythonpackage.maintainers, too
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Currently, all of the replacements in `spack.util.path.replacements()` get evaluated for
each replacement. This makes it easy to get bootstrap issues, because config is used
very early on in Spack.
Right now, if I run `test_autotools_gnuconfig_replacement_no_gnuconfig` on my M1 mac, I
get the circular reference error below. This fixes the issue by making all of the path
replacements lazy lambdas.
As a bonus, this cleans up the way we do substitution for `$env` -- it's consistent with
other substitutions now.
- [x] make all path `replacements()` lazy
- [x] clean up handling of `$env`
```console
> spack unit-test -k test_autotools_gnuconfig_replacement_no_gnuconfig
...
==> [2022-12-31-15:44:21.771459] Error: AttributeError:
The 'autotools-config-replacement' package cannot find an attribute while trying to build from sources. This might be due to a change in Spack's package format to support multiple build-systems for a single package. You can fix this by updating the build recipe, and you can also report the issue as a bug. More information at https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure
/Users/gamblin2/src/spack/lib/spack/spack/package_base.py:1332, in prefix:
1330 @property
1331 def prefix(self):
>> 1332 """Get the prefix into which this package should be installed."""
1333 return self.spec.prefix
Traceback (most recent call last):
File "/Users/gamblin2/src/spack/lib/spack/spack/build_environment.py", line 1030, in _setup_pkg_and_run
kwargs["env_modifications"] = setup_package(
^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/build_environment.py", line 757, in setup_package
set_module_variables_for_package(pkg)
File "/Users/gamblin2/src/spack/lib/spack/spack/build_environment.py", line 596, in set_module_variables_for_package
m.std_cmake_args = spack.build_systems.cmake.CMakeBuilder.std_args(pkg)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/build_systems/cmake.py", line 241, in std_args
define("CMAKE_INSTALL_PREFIX", pkg.prefix),
^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/package_base.py", line 1333, in prefix
return self.spec.prefix
^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/spec.py", line 1710, in prefix
self.prefix = spack.store.layout.path_for_spec(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/directory_layout.py", line 336, in path_for_spec
path = self.relative_path_for_spec(spec)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/directory_layout.py", line 106, in relative_path_for_spec
projection = spack.projections.get_projection(self.projections, spec)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/projections.py", line 13, in get_projection
if spec.satisfies(spec_like, strict=True):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/spec.py", line 3642, in satisfies
if not self.virtual and other.virtual:
^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/spec.py", line 1622, in virtual
return spack.repo.path.is_virtual(self.name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 890, in is_virtual
return have_name and pkg_name in self.provider_index
^^^^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 770, in provider_index
self._provider_index.merge(repo.provider_index)
^^^^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 1096, in provider_index
return self.index["providers"]
~~~~~~~~~~^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 592, in __getitem__
self._build_all_indexes()
File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 607, in _build_all_indexes
self.indexes[name] = self._build_index(name, indexer)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/spack/repo.py", line 616, in _build_index
index_mtime = self.cache.mtime(cache_filename)
^^^^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/llnl/util/lang.py", line 826, in __getattr__
return getattr(self.instance, name)
^^^^^^^^^^^^^
File "/Users/gamblin2/src/spack/lib/spack/llnl/util/lang.py", line 825, in __getattr__
raise AttributeError()
AttributeError
```
* Added support for libfabric to find an external installation and
identify variants supported.
* Change the fabrics definition to only include CXI when on a cray
system with a libfabric-based slingshot network.
* Added a conflict when trying to build the CXI fabric value since it is
only available as closed source at this time.
#33128 Introduces a dependency on re2c into the Ninja build recipe.
This is problematic on Windows as we use CMake to build re2c, and
Ninja to drive the CMake build. This PR resolves this issue by
adding a variant to toggle the use of re2c with ninja.
* Added a more robust check for an external version of the library.
Included a guard to identify when the library gives no discernible
version information and then to substitute with "unknown_ver"
identifier.
* gaudi: new versions 36.8, 36.9
As of 36.8, the tests use catch2 ([commit](https://gitlab.cern.ch/gaudi/Gaudi/- /commit/f2cafb5c9d04c9d497d49182258aa3a0440622c0)).
* gaudi: still depends_on fmt@:8
* unifyfs: new release v1.0.1
Add 1.0.1 release
Add new variant for new configure time option
Co-authored-by: CamStan <CamStan@users.noreply.github.com>
Now that the `tix` variant is conditional, it should also be detected
condititionally, otherwise the spec is invalid and cannot be used during
concretization.
ShellCheck is installed with a downloaded binary instead of being
compiled from source, and there should be comments to point out this
unorthodox approach.
autoconf 2.70 uses use warnings instead of -w so that PERL=/usr/bin/env perl can be passed, but we want to fix absolute paths anyhow through sbang upon install. So, we stick to patching the one perl script that's used during the build.
Gitlab does not merge lists when a job extends two other definitions
that include the same list (e.g. tags). Also, it merges dictionaries
as long as the keys are distinct, but just takes the last mentioned
value when there are key collisions.
This change makes sure that when different tags are needed by a
pipeline, the ones we want are actually provided. It also changes
the example stack to better follow this pattern so we do not lead
developers astray in the future.
Since we dropped support for Python 2.7, we can embrace using keyword only arguments
for many functions in Spack that use **kwargs in the function signature. Here this is done
for the llnl.util.filesystem module.
There were a couple of bugs lurking in the code related to typo-like errors when retrieving
from kwargs. Those have been fixed as well.
* Update xsbench to version 20
XSBench version 20 has implementations for new
architectures and accelerators.
* Added CUDA support for XSBench
* Fixed style issues
The code in FileCache for write_transaction attempts to delete the temporary file when an exception occurs under the context by calling shutil.rmtree. However, rmtree only operates on directories while the rest of FileCache uses normal files. This causes an empty file to be written to the cache key when unneeded.
Use os.remove instead which operates on normal files.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
* cernlib: depends_on freetype, libnsl, libxcrypt, openssl; and patch
In addition to #34448, cernlib depends on these additional packages.
This also applies a patch to the current release in which crypto is
specified wbere libcrypt (in libxcrypt) is actually needed. Because
the upstream git repository is behind a CERN login, we cannot patch
by gitlab URL link.
* [@spackbot] updating style on behalf of wdconinc
Co-authored-by: wdconinc <wdconinc@users.noreply.github.com>
* update the version for 5.3.0 release
* update the rocwmma for 5.3.0 release
* fix the +hip variant
* update the version for rocm-openmp-extras package for 5.3.0 release
* update the hipsolver and hipfft as per review comments
* address review comments
* revert changes to mivisionx with regard to change added for clangrt
* fix for the petsc failure
Spack was running an external detection of Python during each invocation
of the setup script for Windows CMD/PWSH, which has dramatic performance
implications each time the script is invoked, and is completely
unneccesary. Remove this operation.
Windows CMD prompt does not automatically support ASCI color control
characters on the console from Python. Enable this behavior by
accessing the current console and allowing the interpreation of ASCI
control characters from Python via the win32 API.
* adding 2nd bunch of nf-core deps from update/nextflow-tools
* adding 3rd bunch of nf-core deps from update/nextflow-tools
* Update var/spack/repos/builtin/packages/py-dnspython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-dnspython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-dnspython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi-utils/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pastedeploy/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pebble/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-gunicorn/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-starlette/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-starlette/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-starlette/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-parsley/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-paste/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-paste/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-gxformat2: added comment
* py-lagom: now using github tarballs
* fix for py-lagom
* adding missing deps to py-fastapi-utils
* another fix to py-lagom
* Update var/spack/repos/builtin/packages/py-dnspython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi-utils/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi-utils/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-fastapi-utils/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-lagom/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-supervisor/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This commit allows (remote) spec.json files to be clearsigned and gzipped.
The idea is to reduce the number of requests and number of bytes transferred
* added recipes for py-qutip and py-qutip-qip
* small fix
* updated qutip 2x versions
* py-qutip-qip: tarball url from github
* style fix in py-qutip-qip
* Update var/spack/repos/builtin/packages/py-qutip-qip/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-qutip-qip/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-qutip-qip/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-qutip/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-qutip/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* draft for py-pennylane recipe
* first draft for py-strawberryfields recipe
* minimal fix
* small fixes
* accounting for circular dep in py-pennylane and py-pennylane-lightning
* removing py-strawberryfields from this branch
* updated versions for py-pennylane 2x
* needs cmake
* py-pennylane-lightning using github tarball
* Update var/spack/repos/builtin/packages/py-autoray/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pennylane-lightning/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The Intel compiler isn't able to deal with noinline member functions of template classses defined in headers. As such it outputs
```
warning #2196: routine is both "inline" and "noinline"
```
cmake bootstrap will fail due to the word 'warning'.
See spack/var/spack/repos/builtin/packages/protobuf/intel-v2.patch for reference.
The issue does not appear with intel@2021.7.0 or later:
```
$~: compiler=/shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-12.2.0/intel-oneapi-compilers-2022.2.0-uqvb2553zy5toeapvoopacndd27x6p5m/compiler/2022.2.0/linux/bin/intel64/icpc
$~: $compiler unique.c
icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
```
This is a clean version of https://github.com/spack/spack/pull/34167
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
* package/libproxy: fix py3 install
* improve readability
* fix bug
* also add extend
* make flake happy
* [@spackbot] updating style on behalf of Sinan81
* Update var/spack/repos/builtin/packages/libproxy/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* python dependency implied by extends const.
* disable python variant by default
* add run_env, add py conflict
* Update var/spack/repos/builtin/packages/libproxy/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* set env for macos as well
* generalize lib dir detection
Co-authored-by: sbulut <sbulut@3vgeomatics.com>
Co-authored-by: Sinan81 <Sinan81@users.noreply.github.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* add_new_package: py-file-magic
* re-order depends...
* Update var/spack/repos/builtin/packages/py-file-magic/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [@spackbot] updating style on behalf of Sinan81
Co-authored-by: sbulut <sbulut@3vgeomatics.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Sinan81 <Sinan81@users.noreply.github.com>
* Add py-svgpath and dependency
* Update copyright expiration
* [@spackbot] updating style on behalf of heerener
* Process review remarks
* Update var/spack/repos/builtin/packages/py-trimesh/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fix style issue
* py-trimesh: cleanup and optional dependencies
* Fix formatting issue
* py-trimesh: complete dependency list for easy variant
Two new packages: py-mapbox-earcut and py-pycollada
* Some more missing dependencies
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* new package: py-kb-python + dependencies
- py-loompy
- py-ngs-tools
- py-numpy-groupies
* Update var/spack/repos/builtin/packages/py-kb-python/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-shortuuid: add version 1.0.11
* Update var/spack/repos/builtin/packages/py-shortuuid/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update packages config to indicate that MSVC is the preferred compiler
* Update packages config to indicate that msmpi is the preferred MPI provider
* Fix msmpi external detection
Spack imports `pytest`, which *can* import `numpy`. Recent versions of `numpy` require
Python 3.8 or higher, and they use 3.8 type annotations in their type stubs (`.pyi`
files). At the same time, we tell `mypy` to target Python 3.7, as we still support older
versions of Python.
What all this means is that if you run `mypy` on `spack`, `mypy` will follow all the
static import statements, and it ends up giving you this error when it finds numpy stuff
that is newer than the target Python version:
```
==> Running mypy checks
src/spack/var/spack/environments/default/.spack-env/._view/4g7jd4ibkg4gopv4rosq3kn2vsxrxm2f/lib/python3.11/site-packages/numpy/__init__.pyi:638: error: Positional-only parameters are only supported in Python 3.8 and greater [syntax]
Found 1 error in 1 file (errors prevented further checking)
mypy found errors
```
We can fix this by telling `mypy` to skip all imports of `numpy` in `pyproject.toml`:
```toml
[[tool.mypy.overrides]]
module = 'numpy'
follow_imports = 'skip'
follow_imports_for_stubs = true
```
- [x] don't follow imports from `numpy` in `mypy`
- [x] get rid of old rule not to follow `jinja2` imports, as we now require Python 3
The code in Spack to generate install and test reports currently suffers from unneeded complexity. For
instance, we have classes in Spack core packages, like `spack.reporters.CDash`, that need an
`argparse.Namespace` to be initialized and have "hard-coded" string literals on which they branch to
change their behavior:
```python
if do_fn.__name__ == "do_test" and skip_externals:
package["result"] = "skipped"
else:
package["result"] = "success"
package["stdout"] = fetch_log(pkg, do_fn, self.dir)
package["installed_from_binary_cache"] = pkg.installed_from_binary_cache
if do_fn.__name__ == "_install_task" and installed_already:
return
```
This PR attempt to polish the major issues encountered in both `spack.report` and `spack.reporters`.
Details:
- [x] `spack.reporters` is now a package that contains both the base class `Reporter` and all
the derived classes (`JUnit` and `CDash`)
- [x] Classes derived from `spack.reporters.Reporter` don't take an `argparse.Namespace` anymore
as argument to `__init__`. The rationale is that code for commands should be built upon Spack
core classes, not vice-versa.
- [x] An `argparse.Action` has been coded to create the correct `Reporter` object based on command
line arguments
- [x] The context managers to generate reports from either `spack install` or from `spack test` have
been greatly simplified, and have been made less "dynamic" in nature. In particular, the `collect_info`
class has been deleted in favor of two more specific context managers. This allows for a simpler
structure of the code, and less knowledge required to client code (in particular on which method to patch)
- [x] The `InfoCollector` class has been turned into a simple hierarchy, so to avoid conditional statements
within methods that assume a knowledge of the context in which the method is called.
On systems with remote groups, the primary user group may be remote and may not exist on
the local system (i.e., it might just be a number). On the CLI, it looks like this:
```console
> touch foo
> l foo
-rw-r--r-- 1 gamblin2 57095 0 Dec 29 22:24 foo
> chmod 2000 foo
chmod: changing permissions of 'foo': Operation not permitted
```
Here, the local machine doesn't know about per-user groups, so they appear as gids in
`ls` output. `57095` is also `gamblin2`'s uid, but the local machine doesn't know that
`gamblin2` is in the `57095` group.
Unfortunately, it seems that Python's `os.chmod()` just fails silently, setting
permissions to `0o0000` instead of `0o2000`. We can avoid this by ensuring that the file
has a group the user is known to be a member of.
- [x] Add `ensure_known_group()` in the permissions tests.
- [x] Call `ensure_known_group()` on tempfile in `test_chmod_real_entries_ignores_suid_sgid`.
There are a number of places in our docstrings where we write "list of X" as the type, even though napoleon doesn't actually support this. It ends up causing warnings when generating docs.
Now that we require Python 3, we don't have to rely on type hints in docs -- we can just use Python type hints and omit the types of args and return values from docstrings.
We should probably do this for all types in docstrings eventually, but this PR focuses on the ones that generate warnings during doc builds.
Some `mypy` annoyances we should consider in the future:
1. Adding some of these type annotations gets you:
```
note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs [annotation-unchecked]
```
because they are in unannotated functions (like constructors where we don't really need any annotations).
You can silence these with `disable_error_code = "annotation-unchecked"` in `pyproject.toml`
2. Right now we support running `mypy` in Python `3.6`. That means we have to support `mypy` `.971`, which does not support `disable_error_code = "annotation-unchecked"`, so I just filter `[annotation-unchecked]` lines out in `spack style`.
3. I would rather just turn on `check_untyped_defs` and get more `mypy` coverage everywhere, but that will require about 1,000 fixes. We should probably do that eventually.
4. We could also consider only running `mypy` on newer python versions. This is not easy to do while supporting `3.6`, because you have to use `if TYPE_CHECKING` for a lot of things to ensure that 3.6 still parses correctly. If we only supported `3.7` and above we could use [`from __future__ import annotations`](https://mypy.readthedocs.io/en/stable/runtime_troubles.html#future-annotations-import-pep-563), but we have to support 3.6 for now. Sigh.
- [x] Convert a number of docstring types to Python type hints
- [x] Get rid of "list of" wherever it appears
Based on the following lines in the top level `CMakeLists.txt` (I can't deep link since gitlab.cern.ch not public), `cernlib` needs an explicit dependency on `libxaw` and `libxt`:
```cmake
find_package(X11 REQUIRED)
message(STATUS "CERNLIB: X11_Xt_LIB=${X11_Xt_LIB} X11_Xaw_LIB=${X11_Xaw_LIB} X11_LIBRARIES=${X11_LIBRARIES}")
```
Per https://github.com/spack/spack/issues/34192, apptainer does not support `--without-conmon`, so we introduce a base class `config_options` property that can be overridden in the `apptainer` package.
`texinfo` depends on `gettext`, and it builds a perl module that uses gettext via XS
module FFI. Unfortunately, the XS modules build asks perl to tell it what compiler to
use instead of respecting the one passed to configure.
Without this change, the build fails with this error:
```
parsetexi/api.c:33:10: fatal error: 'libintl.h' file not found
^~~~~~~~~~~
```
We need the gettext dependency and the spack wrappers to ensure XS builds properly.
- [x] Add needed `gettext` dependency to `texinfo`
- [x] Override XS compiler with `PERL_EXT_CC`
Co-authored-by: Paul Kuberry <pakuber@sandia.gov>
Local `git` tests will fail with `fatal: transport 'file' not allowed` when using git 2.38.1 or higher, due to a fix for `CVE-2022-39253`.
This was fixed in CI in #33429, but that doesn't help the issue for anyone's local environment. Instead of fixing this with git config in CI, we should ensure that the tests run anywhere.
- [x] Introduce `spack.util.git`.
- [x] Use `spack.util.git.get_git()` to get a git executable, instead of `which("git")` everywhere.
- [x] Make all `git` tests use a `git` fixture that goes through `spack.util.git.get_git()`.
- [x] Add `-c protocol.file.allow=always` to all `git` invocations under `pytest`.
- [x] Revert changes from #33429, which are no longer needed.
`spack graph` has been reworked to use:
- Jinja templates
- builder objects to construct the template context when DOT graphs are requested.
This allowed to add a new colored output for DOT graphs that highlights both
the dependency types and the nodes that are needed at runtime for a given spec.
`spack solve` is supposed to show you times you can compare. setup, ground, solve, etc.
all in a list. You're also supposed to be able to compare easily across runs. With
`pretty_seconds()` (introduced in #33900), it's easy to miss the units, e.g., spot the
bottleneck here:
```console
> spack solve --timers tcl
setup 22.125ms
load 16.083ms
ground 8.298ms
solve 848.055us
total 58.615ms
```
It's easier to see what matters if these are all in the same units, e.g.:
```
> spack solve --timers tcl
setup 0.0147s
load 0.0130s
ground 0.0078s
solve 0.0008s
total 0.0463s
```
And the units won't fluctuate from run to run as you make changes.
-[x] make `spack solve` timings consistent like before
* py-pytest-datadir: Init at 1.4.1
* py-pytest-data-dir: Fix missing dep
Co-authored-by: "Adam J. Stewart" <ajstewart426@gmail.com>
Co-authored-by: "Adam J. Stewart" <ajstewart426@gmail.com>
* ML CI: Linux x86_64
* Update comments
* Rename again
* Rename comments
* Update to match other arches
* No compiler
* Compiler was wrong anyway
* Faster TF
Avoid text decoding and encoding when combining log files, instead
combine in binary mode.
Also do a buffered copy which is sometimes faster for large log files.
Currently, the Spack docs show documentation for submodules *before* documentation for
submodules on package doc pages. This means that if you put docs in `__init__.py` in
some package, the docs in there will be shown *after* the docs for all submodules of the
package instead of at the top as an intro to the package. See, e.g.,
[the lockfile docs](https://spack.readthedocs.io/en/latest/spack.environment.html#module-spack.environment),
which should be at the
[top of that page](https://spack.readthedocs.io/en/latest/spack.environment.html).
- [x] add the `--module-first` option to sphinx so that it generates module docs at top of page.
* librsvg: add 2.40.21, which does not require rust and has some security backports
https://download.gnome.org/sources/librsvg/2.40/librsvg-2.40.21.news
* librsvg: prevent finding broken gtkdoc binaries when ~doc is selected.
On my CentOS7 hosts, ./configure finds e.g. /bin/gtkdoc-rebase even when
~doc is selected. These tools use Python2, and fail with an error:
"ImportError: No module named site"
So prevent ./configure from finding these broken tools when not building
the +doc variant.
without this patch, build of paraview has a meltdown when reaching 3rd party catalyst and other packages
with these types of errors:
335 /tmp/foo/spack-stage/spack-stage-paraview-5.10.1-gscoqxhhakjyyfirdefuhmi2bzw4scho/spack-src/VTK/ThirdParty/fmt/vtkfmt/vtkfmt/format.h:1732:11: error: cannot capture a bi
t-field by reference
336 if (sign) *it++ = static_cast<Char>(data::signs[sign]);
337 ^
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
The sticky property will prevent clingo from changing the amdgpu_target
to work around conflicts. This is the same behaviour as was adopted for
cuda_arch in 055c9d125d.
Replace the filter_file for older configure with rocm 5.3 with an
upstream patch. Further, the patch is no longer needed for develop or
later releases.
Implement an alternative strategy to do index.json invalidation.
The current approach of pairs of index.json / index.json.hash is
problematic because it leads to races.
The standard solution for cache invalidation is etags, which are
supported by both http and s3 protocols, which allows one to do
conditional fetches.
This PR implements that for the http/https schemes. It should also work
for s3 schemes, but that requires other prs to be merged.
Also it improves unit tests for index.json fetches.
* py-scikit-image: add 0.19.3
* Update var/spack/repos/builtin/packages/py-scikit-image/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* LLVM: replace libelf dependency with elf
I didn't test this extensively, but in CMS LLVM builds just fine with elfutils.
* [@spackbot] updating style on behalf of iarspider
Co-authored-by: iarspider <iarspider@users.noreply.github.com>
* exiv2: add new versions
* babl: new package required to build GIMP
* gegl: new package required to build GIMP
* gexiv2: new package required to build GIMP
* libmypaint: new package required to build GIMP
* mypaint-brushes: new package required to build GIMP
* vala: new package required to build GIMP
* GIMP: new package definition for building GIMP-2.10 from source
* libjxl: update for 0.7.0
* libwmf: a library for reading vector images in Windows Metafile Format (WMF)
* libde265: an open source implementation of the h.265 video codec
* libwebp: add new versions
* GIMP: additional variants for building GIMP-2.10 from source
* libde265: remove boilerplate
* fixes for style precheck
* updates based on feedback
* fixes for style precheck
Update the depends_on("perl") to depends_on("perl+threads").
This and #34074 is needed to properly handle e.g. the perl-Thread-Queue
rpm package:
It may not be installed on RedHat-based hosts, which can lead to automake
build failures when `spack external find perl` or `spack external find --all`
was used to use the system-provided perl install.
* nextflow recipe: added latest stable version
* tower-cli recipe: added latest release
* recipes tower-agent and tower-cli renamed to nf-tower-agent and nf-tower-cli
* recipes nf-tower-agent and nf-tower-cli: small fix
* nf-core-tools recipe: added most py- dependencies
* nf-core-tools: recipe without galaxy-tool-util (for testing)
* fixed typos in py-yacman recipe
* fixed typos in py-pytest-workflow recipe
* fixed typo in nf-core-tools recipe
* fixed typos in py-yacman recipe
* fixes in recipes for py-questionary and py-url-normalize
* fixes to py-yacman recipe
* style fixes to py- packages that are dependencies to nf-core-tools
* fix in py-requests-cache recipe
* added missing dep in py-requests-cache recipe
* nf-core-tools deps: removed redundant python dep for py packages oyaml and piper
* nf-core-tools recipe: final, incl dep on py-galaxy-tool-util
* nf-core-tools: new version with extra dependency
* commit to merge packages on focus from update/nextflow-tools
* nf-core: commenting galaxy dep for this pr
* Update var/spack/repos/builtin/packages/py-requests-cache/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-requests-cache/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* removed nf-core-tools from this branch, will be back at the end
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Interim fix for #34559
Spack's MSVC compiler definition uses ifx as the Fortran compiler.
Prior to #33385, the Spack MSVC compiler definition required the
executable to be called "ifx.exe"; #33385 replaced this with just
"ifx", which inadvertently led to ifx falsely indicating the
presence of MSVC on non-Windows systems (which leads to future
errors when attempting to query/use those compiler objects).
This commit applies a short-term fix by updating MSVC Fortran
version detection to always indicate a failure on non-Windows.
fixes#34518
Fix an issue due to the MRO chain of the package wrapper
during build. Before this PR we were always returning
False when the builder object was created before the
run_tests method was monkey patched.
openPMD, a metadata standard on top of backends like ADIOS2 and HDF5,
is implemented in ParaView 5.9+ via a Python3 module.
Simplify Conflicts & Variant
Add to ECP Data Vis SDK
This reverts commit 8035eeb36d.
And also removes logic around an additional HEAD request to prevent
a more expensive GET request on wrong content-type. Since large files
are typically an attachment and only downloaded when reading the
stream, it's not an optimization that helps much, and in fact the logic
was broken since the GET request was done unconditionally.
* Add py-bmtk and py-neurotools
* py-bmtk: version bump
* [@spackbot] updating style on behalf of heerener
* Maybe the copyright needs to be extended to 2022 for the check to pass
* Process review remarks
* Update var/spack/repos/builtin/packages/py-neurotools/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The main issue that's fixed is that Spack passes paths (as strings) to
functions that require urls. That wasn't an issue on unix, since there
you can simply concatenate `file://` and `path` and all is good, but on
Windows that gives invalid file urls. Also on Unix, Spack would not deal with uri encoding like x%20y for file paths.
It also removes Spack's custom url.parse function, which had its own incorrect interpretation of file urls, taking file://x/y to mean the relative path x/y instead of hostname=x and path=/y. Also it automatically interpolated variables, which is surprising for a function that parses URLs.
Instead of all sorts of ad-hoc `if windows: fix_broken_file_url` this PR
adds two helper functions around Python's own path2url and reverse.
Also fixes a bug where some `spack buildcache` commands
used `-d` as a flag to mean `--mirror-url` requiring a URL, and others
`--directory`, requiring a path. It is now the latter consistently.
* Add GFE packages, Update pFUnit
* Remove citibeth as maintainer per her request
* Version 3.3.0 is an odd duck. Needs a v
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
When installing binary tarballs, Spack has to download from its
binary mirrors.
Sometimes Spack has cache available for these mirrors.
That cache helps to order mirrors to increase the likelihood of
getting a direct hit.
However, currently, when Spack can't find a spec in any local cache
of mirrors, it's very dumb:
- A while ago it used to query each mirror to see if it had a spec,
and use that information to order the mirror again, only to go
about and do exactly a part of what it just did: fetch the spec
from that mirror confused
- Recently, it was changed to download a full index.json, which
can be multiple dozens of MBs of data and may take a minute to
process thanks to the blazing fast performance you get with
Python.
In a typical use case of concretizing with reuse, the full index.json
is already available, and it likely that the local cache gives a perfect
mirror ordering on install. (There's typically no need to update any
caches).
However, in the use case of Gitlab CI, the build jobs don't have cache,
and it would be smart to just do direct fetches instead of all the
redundant work of (1) and/or (2).
Also, direct fetches from mirrors will soon be fast enough to
prefer these direct fetches over the excruciating slowness of
index.json files.
* include Drishti
* fix syntax
* Update var/spack/repos/builtin/packages/drishti/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update var/spack/repos/builtin/packages/drishti/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-sphinxcontrib-devhelp: add 1.0.2
* Update var/spack/repos/builtin/packages/py-sphinxcontrib-devhelp/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-sphinxcontrib-applehelp: add 1.0.2
* Update var/spack/repos/builtin/packages/py-sphinxcontrib-applehelp/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* ROCm 5.3.0 updates
* New patches for 5.3.0 on hip and hsakmt
* Adding additional build arguments in hip and llvm
* RVS updates for 5.3.0 release
* New patches and rocm-tensile, rocprofiler-dev, roctracer-dev recipe updates for 5.3.0
* Reverting OPENMP fix from rocm-tensile
* Removing the patch to compile without git and adding witout it
* Install library in to lib directory instead of lib64 across all platform
* Setting lib install directory to lib
* Disable gallivm coroutine for libllvm15
* Update llvm-amdgpu prefix path in hip-config.cmake.in
Removing libllvm15 from Mesa dependency removing
* hip-config.cmake.in update required from 5.2
* hip-config.cmake.in update required from 5.2 and above
* hip-config.cmake.in update required for all 5.2 release above
* Style check correction in hip update
* ginkgo: add missing include
* Patching hsa include path for rocm 5.3
* Restricting patch for llvm-15
* Style check error correction
* PIC flag required for the new test applications
* Passing -DCMAKE_POSITION_INDEPENDENT_CODE=ON in the cmake_args instead of setting -fPIC in CFLAGS
Co-authored-by: Cordell Bloor <Cordell.Bloor@amd.com>
gnulib/lib/malloca.c uses single value `static_assert()` only available in c-11
syntax. `gcc` seems to be fine, but `icc` needs extra flag.
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
Since it's a header-only library there's nothing to build. However, the
default targets include tests and examples and there's no option to turn
them off during configuration time.
Per https://geant4.web.cern.ch/node/1837 the correct dependency for 10.6 is on `g4emlow@7.9.1`, not on both `g4emlow@7.9` and `g4emlow@7.9.1`.
This is a minor cosmetic fix. The concretization for 10.6 works just fine here. But this removes the duplicate entry.
This PR patches the f_check script to detect the ifort compiler and
ensure that F_COMPILER is iset to INTEL. This problem was introduced with
openblas-0.3.21. Without this patch, the value of F_COMPILER falls back
to G77 and icc rather than ifort is used for the linking stage. That
results in the openblas library missing libifcore, which in turn means
many Fotran programs can not be compiled with ifort.
Writing a long dependency like:
```python
depends_on(
"llvm"
"targets=amdgpu,bpf,nvptx,webassembly"
"version_suffix=jl +link_llvm_dylib ~internal_unwind"
)
```
when it should be formatted like this:
```python
depends_on(
"llvm"
" targets=amdgpu,bpf,nvptx,webassembly"
" version_suffix=jl +link_llvm_dylib ~internal_unwind"
)
```
can cause really subtle errors. Specifically, you'll get something like this in
the package sanity tests:
```
AttributeError: 'NoneType' object has no attribute 'rpartition'
```
because Spack happily constructs a class that has a dependency with name `None`.
We can catch this earlier by banning anonymous dependency specs directly in
`depends_on()`. This causes the package itself to fail to parse, and emits
a much better error message:
```
==> Error: Invalid dependency specification in package 'julia':
llvmtargets=amdgpu,bpf,nvptx,webassemblyversion_suffix=jl +link_llvm_dylib ~internal_unwind
```
* Only restrict CMake version in umpire when examples and rocm are enabled
* Add CMAKE_HIP_ARCHITECTURES to Umpire and lift cmake version restriction
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
This patch:
https://gcc.gnu.org/legacy-ml/gcc-patches/2018-01/msg01962.html
is actually in Amazon Linux GCC 7.3.1, which we use in CI.
So we should not hold openblas back because of it.
Old versions of OpenBLAS fail to detect the host arch of some of the
AVX512 cpus of build nodes, causing build failures.
Of course we should try to set ARCH properly in OpenBLAS to avoid that
it looks up the build arch, but that's quite some work.
* Debian like distros use multiarch implementation spec
https://wiki.ubuntu.com/MultiarchSpec
Instead of being limited to /usr/lib64, architecture based
lib directories are used. For instance, under ubuntu a library package
on x86_64 installs binaries under /usr/lib/x86_64-linux-gnu.
Building pmix with external dependencies like hwloc or libevent
fail as with prefix set to /usr, that prefix works for
headers and binaries but does not work for libraries. The default
location for library /usr/lib64 does not hold installed binaries.
Pmix build options --with-libevent and --with-libhwloc allow us to
specify dependent library locations. This commit is an effort to
highlight and resolve such an issue when a users want to use Debian like
distro library packages and use spack to build pmix.
There maybe other packages that might be impacted in a similar way.
* Adding libs property to hwloc and libevent and some cleanups to pmix patch
* Fixing style and adding comment on Pmix' 32-bit hwloc version detection issue
* `url_exists` improvements (take 2)
Make `url_exists` do HEAD request for http/https/s3 protocols
Rework the opener: construct it once and only once, dynamically dispatch
to the right one based on config.
* geant4: version bumps for Geant4 11.1.0
- Version bumps for new data libraries
- g4ndl 4.7
- g4emlow 8.2
- Add geant4-data@11.1.0
- Checksum new Geant4 11.1.0 release
- Limit +python variant to maximum of :11.0 due to removal of
Geant4Py in 11.1
- Update CLHEP dependency to at least 2.4.6.0 for this release
- Update VecGeom dependency to at least 1.2.0 for this release,
closing version ranges for older releases to prevent multiple
versions satisfying requirement
* geant4: correct max version for python support
It's very common for us to tell users to grep through the existing Spack packages to
find examples of what they want, and it's also very common for package developers to do
it. Now, searching packages is even easier.
`spack pkg grep` runs grep on all `package.py` files in repos known to Spack. It has no
special options other than the search string; all options passed to it are forwarded
along to `grep`.
```console
> spack pkg grep --help
usage: spack pkg grep [--help] ...
positional arguments:
grep_args arguments for grep
options:
--help show this help message and exit
```
```console
> spack pkg grep CMakePackage | head -3
/Users/gamblin2/src/spack/var/spack/repos/builtin/packages/3dtk/package.py:class _3dtk(CMakePackage):
/Users/gamblin2/src/spack/var/spack/repos/builtin/packages/abseil-cpp/package.py:class AbseilCpp(CMakePackage):
/Users/gamblin2/src/spack/var/spack/repos/builtin/packages/accfft/package.py:class Accfft(CMakePackage, CudaPackage):
```
```console
> spack pkg grep -Eho '(\S*)\(PythonPackage\)' | head -3
AwsParallelcluster(PythonPackage)
Awscli(PythonPackage)
Bueno(PythonPackage)
```
## Return Value
This retains the return value semantics of `grep`:
* 0 for found,
* 1 for not found
* >1 for error
## Choosing a `grep`
You can set the ``SPACK_GREP`` environment variable to choose the ``grep``
executable this command should use.
Unit tests on Windows are supposed to pass for any PR to pass CI.
However, the return code for the unit test command was not being
checked, which meant this check was always passing (effectively
disabled). This PR
* Properly checks the result of the unit tests and fails if the
unit tests fail
* Fixes (or disables on Windows) a number of tests which have
"drifted" out of support on Windows since this check was
effectively disabled
At some point the `a` mock package became an `AutotoolsPackage`, and that means it
depends on `gnuconfig` on macOS. This was causing one of our shell tests to fail on
macOS because it was testing for `{a.prefix.bin}:{b.prefix.bin}` in `PATH`, but
`gnuconfig` shows up between them.
- [x] simplify the test to check `spack load --sh a` and `spack load --sh b` separately
* Add 20 as a valid option for cxxstd to fmt
* Add pika 0.11.0
* Fix version constraint for p2300 variant in pika package
* Add pika-algorithms package
* py-reportlab: add 3.6.12
* Update var/spack/repos/builtin/packages/py-reportlab/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* adding recipe for singularity-hpc - 1st go
* typo in singularity-hpc recipe
* singularity-hpc, spython recipes: added platform variant
* singularity-hpc, spython recipes: platform variant renamed to runtime
* style fix
* another style fix
* yet another style fix (why are they not reported altogether)
* singularity-hpc recipe: added Vanessa as maintainer
* singularity-hpc recipe: add podman variant
* singularity-hpc recipe: added variant for module system
* shpc recipe: add version for py-semver dependency
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-spython recipe: no need to specify generic python dep for a python pkg
* py-spython: py-requests not needed
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
## Motivation
Our parser grew to be quite complex, with a 2-state lexer and logic in the parser
that has up to 5 levels of nested conditionals. In the future, to turn compilers into
proper dependencies, we'll have to increase the complexity further as we foresee
the need to add:
1. Edge attributes
2. Spec nesting
to the spec syntax (see https://github.com/spack/seps/pull/5 for an initial discussion of
those changes). The main attempt here is thus to _simplify the existing code_ before
we start extending it later. We try to do that by adopting a different token granularity,
and by using more complex regexes for tokenization. This allow us to a have a "flatter"
encoding for the parser. i.e., it has fewer nested conditionals and a near-trivial lexer.
There are places, namely in `VERSION`, where we have to use negative lookahead judiciously
to avoid ambiguity. Specifically, this parse is ambiguous without `(?!\s*=)` in `VERSION_RANGE`
and an extra final `\b` in `VERSION`:
```
@ 1.2.3 : develop # This is a version range 1.2.3:develop
@ 1.2.3 : develop=foo # This is a version range 1.2.3: followed by a key-value pair
```
## Differences with the previous parser
~There are currently 2 known differences with the previous parser, which have been added on purpose:~
- ~No spaces allowed after a sigil (e.g. `foo @ 1.2.3` is invalid while `foo @1.2.3` is valid)~
- ~`/<hash> @1.2.3` can be parsed as a concrete spec followed by an anonymous spec (before was invalid)~
~We can recover the previous behavior on both ones but, especially for the second one, it seems the current behavior in the PR is more consistent.~
The parser is currently 100% backward compatible.
## Error handling
Being based on more complex regexes, we can possibly improve error
handling by adding regexes for common issues and hint users on that.
I'll leave that for a following PR, but there's a stub for this approach in the PR.
## Performance
To be sure we don't add any performance penalty with this new encoding, I measured:
```console
$ spack python -m timeit -s "import spack.spec" -c "spack.spec.Spec(<spec>)"
```
for different specs on my machine:
* **Spack:** 0.20.0.dev0 (c9db4e50ba045f5697816187accaf2451cb1aae7)
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-icelake
* **Concretizer:** clingo
results are:
| Spec | develop | this PR |
| ------------- | ------------- | ------- |
| `trilinos` | 28.9 usec | 13.1 usec |
| `trilinos @1.2.10:1.4.20,2.0.1` | 131 usec | 120 usec |
| `trilinos %gcc` | 44.9 usec | 20.9 usec |
| `trilinos +foo` | 44.1 usec | 21.3 usec |
| `trilinos foo=bar` | 59.5 usec | 25.6 usec |
| `trilinos foo=bar ^ mpich foo=baz` | 120 usec | 82.1 usec |
so this new parser seems to be consistently faster than the previous one.
## Modifications
In this PR we just substituted the Spec parser, which means:
- [x] Deleted in `spec.py` the `SpecParser` and `SpecLexer` classes. deleted `spack/parse.py`
- [x] Added a new parser in `spack/parser.py`
- [x] Hooked the new parser in all the places the previous one was used
- [x] Adapted unit tests in `test/spec_syntax.py`
## Possible future improvements
Random thoughts while working on the PR:
- Currently we transform hashes and files into specs during parsing. I think
we might want to introduce an additional step and parse special objects like
a `FileSpec` etc. in-between parsing and concretization.
* py-pywavelets: add 1.4.1
* Update var/spack/repos/builtin/packages/py-pywavelets/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pywavelets/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [quantum-espresso] Parallel make fails for 6.{6,7}
I run into a race condition in `make` with Intel compiler on icelake when building QE 6.6 and 6.7.
* Fix comment
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
* fix location of input for darshan-util tests
Darshan log file used for test input was removed from the Darshan
repo after the 3.4.0 release. This commit adds logic to use a
different log file as test input for later Darshan versions.
* libctl: add new version
Change-Id: I16f91cfab198c66b60407ab5bb2cb3ebeac6bc19
* New package: libgdsii
Change-Id: I34b52260ab68ecc857ddf8cc63b124adc2689a51
* New package: mpb
Change-Id: I6fdf5321c33d6bdbcaa1569026139a8483a3bcf8
* meep: add new version and variants
Change-Id: I0b60a9a4d9a329f7bde9027514467e17376e6a39
* meep: use with_or_without
Change-Id: I05584cb13df8ee153ed385e77d367cb34e39777e
* LibCatalyst: Fix version of pre-release develop version
* ParaView: Requires libcatalyst@2:
* ParaView: Apply adios2 module no kit patch to 5.11
This patch is still pending in VTK and didn't make it into 5.11 as anticipated.
* bcache: support external gettext when `libintl` is in glibc
Many glibc-based Linux systems don't have gettext's libintl because
libintl is included in the standard system's glibc (libc) itself.
When using `spack external find gettext` on those, packages like
`bcache` which unconditionally to link using `-lintl` fail to link
with -lintl.
Description of the fix:
The libs property of spack's gettext recipe returns the list of libs,
so when gettext provides libintl, use it. When not, there is no
separate liblint library and the libintl API is provided by glibc.
Tested with `spack external find gettext` on glibc-based Linux and
in musl-based Alpine Linux to make sure that when -lintl is really
needed, it is really used and nothing breaks.
* Fix mercurial print_str failure.
* Perform same fix on py-pybind11 for print_string missing method.
Co-authored-by: Nicholas Cameron Sly <sly1@llnl.gov>
* Add new root versions and associated dependency constraints
* Please style guide
* Avoid conflicts where possible
* Untested prototype of macOS version detection
* Fixes for macOS version prototype
* More logical ordering
* More correctness and style fixes
* Try to use spack's macos_version
* Add some forgotten @s
* Actually, Spack can't build Python 3.6 anymore, and thus no older PyROOT
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* Fix compile errors with latest HDF5 1.13.3
* format
* Update var/spack/repos/builtin/packages/hdf5-vol-async/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This commit reworks the bootstrapping procedure to use Spack environments
as much as possible.
The `spack.bootstrap` module has also been reorganized into a Python package.
A distinction is made among "core" Spack dependencies (clingo, GnuPG, patchelf)
and other dependencies. For a number of reasons, explained in the `spack.bootstrap.core`
module docstring, "core" dependencies are bootstrapped with the current ad-hoc
method.
All the other dependencies are instead bootstrapped using a Spack environment
that lives in a directory specific to the interpreter and the architecture being used.
The cray manifest shows dependency information for cray-mpich, which we never previously cared about
because it can only be used as an external. This updates Spack's dependency information to make cray-mpich
specs read in from the cray external manifest usable.
* initial changes for rocm recipes
* drop support for older releases
* drop support for older rocm releases - add more recipes
* drop support for older releases
* address style issues
* address style error
* fix errors
* address review comments
* Added plumed version 2.8.1 including gromacs compatibility
* Corrected ~mpi and +mpi variants in new depends
* Fixed regression logic plumed+gromacs@2020.6 support
All the vermin annotations we were using were for optional features introduced in early
Python 3 versions. We no longer need any of them, as we only support Python 3.6+. If we
start optionally using features from newer Pythons than 3.6, we will need more vermin
annotations.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
We no longer support Python <3.6, so we don't need to check whether Python supports SSL
verification in `spack.util.web`.
- [x] Remove a bunch of logic we needed to appease Python 2
We've stopped supporting Python 2, and contributors are noticing that our CI no longer
allows Python 2.7 comment type hints. They end up having to adapt them, but this adds
extra unrelated work to PRs.
- [x] Move to 3.6 type hints across the entire code base
* doxygen: add build-tool tag
This allows it to be included automatically as an external. No one links against doxygen so this should be ok.
* doxygen: add self as maintainer
* vecgeom: add new 1.2.1 version
* vecgeom: introduce conflict between gcc/cuda
Recent tests of vecgeom in Spack environments have shown that the build
with +cuda fails with GCC >= 11.3 and CUDA < 11.7 with error
...lib/gcc/x86_64-pc-linux-gnu/11.3.0/include/serializeintrin.h(41):
error: identifier "__builtin_ia32_serialize" is undefined
1 error detected in the compilation of
".../VecGeom/source/BVHManager.cu".
Other GCC/CUDA combinations appear o.k.
Avoid this error in spack, and document it for users, with a conflict
directive to express the restriction.
All Spec attributes are now represented as `attr(attribute_name, ... args ...)`, e.g.
`attr(node, "hdf5")` instead of `node("hdf5")`, as we *have* to maintain the `attr()`
form anyway, and it simplifies the encoding to just maintain one form of the Spec
information.
Background
----------
In #20644, we unified the way conditionals are done in the concretizer, but this
introduced a nasty aspect to the encoding: we have to maintain everything we want in
general conditions in two forms: `predicate(...)` and `attr("predicate", ...)`. For
example, here's the start of the table of spec attributes we had to maintain:
```prolog
node(Package) :- attr("node", Package).
virtual_node(Virtual) :- attr("virtual_node", Virtual).
hash(Package, Hash) :- attr("hash", Package, Hash).
version(Package, Version) :- attr("version", Package, Version).
...
```
```prolog
attr("node", Package) :- node(Package).
attr("virtual_node", Virtual) :- virtual_node(Virtual).
attr("hash", Package, Hash) :- hash(Package, Hash).
attr("version", Package, Version) :- version(Package, Version).
...
```
This adds cognitive load to understanding how the concretizer works, as you have to
understand the equivalence between the two forms of spec attributes. It also makes the
general condition logic in #20644 hard to explain, and it's easy to forget to add a new
equivalence to this list when adding new spec attributes (at least two people have been
bitten by this).
Solution
--------
- [x] remove the equivalence list from `concretize.lp`
- [x] simplify `spec_clauses()`, `condition()`, and other functions in `asp.py` that need
to deal with `Spec` attributes.
- [x] Convert all old-form spec attributes in `concretize.lp` to the `attr()` form
- [x] Simplify `display.lp`, where we also had to maintain a list of spec attributes. Now
we only need to show `attr/2`, `attr/3`, and `attr/4`.
- [x] Simplify model extraction logic in `asp.py`.
Performance
-----------
This seems to result in a smaller grounded problem (as there are no longer duplicated
`attr("foo", ...)` / `foo(...)` predicates in the program), but it also adds a slight
performance overhead vs. develop. Ultimately, simplifying the encoding will be a win,
particularly for improving error messages.
Notes
-----
This will simplify future node refactors in `concretize.lp` (e.g., not identifying nodes
by package name, which we need for separate build dependencies).
I'm still not entirely used to reading `attr()` notation, but I thnk it's ultimately
clearer than what we did before. We need more uniform naming, and it's now clear what is
part of a solution. We should probably continue making the encoding of `concretize.lp`
simpler and more self-explanatory. It may make sense to rename `attr` to something like
`node_attr` and to simplify the names of node attributes. It also might make sense to do
something similar for other types of predicates in `concretize.lp`.
* Add py-python-lsp-server and dependencies
* Update var/spack/repos/builtin/packages/py-python-lsp-server/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Relax version range constraints on py-python-lsp-jsonrpc and add missing dep
* Add runtime dependency flag to setuptools dependencies
* Remove unused python@3.6: dependency and move setuptools-scm to build dep only
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fix recipe
* Update package.py
* Update recipe following review
* Update var/spack/repos/builtin/packages/py-onnx-runtime/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* remove unused imports
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* CI: Update Data and Vis SDK Stack
* Update image to match target deployments (E4S)
* Enable all packages
* Test supported variants of ParaView and VisIt
* Sensei: Update Python hint for newer cmake
* Sensei: add Python3 hint
* Add new version of snakemake
* Add myself as a maintainer
* py-retry -> py-reretry
* Added snakemake variants for storage systems
* Updated comments
* Responded to Adam's comments
* Fixed spack style
* Add build/run dependency types
* New package py-statmorph w/ dependecies. Add py-astropy@5.1
* [@spackbot] updating style on behalf of meyersbs
* [py-statmorph,py-astropy,py-pyerfa] minor fixes
* Add checksum for py-rsa 4.9
* Update package.py
* Update var/spack/repos/builtin/packages/py-rsa/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-fastpath: new package
* Update var/spack/repos/builtin/packages/py-fastpath/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Historically, development of the Octopus code was done on the "develop" branch
on https://gitlab.com/octopus-code/octopus but now development takes place on
"main" (since Q3 2022).
The suggestion in this PR to solve the issue is to keep the spack label
`octopus@develop` as this better indicates this is the development branch on git
than `octopus@main`, but of course to use the `main` branch (there is no choice
here - the `develop` branch is not touched anymore). Sticking to
`octopus@develop` as the version label also keeps backwards compatibility.
This reverts commit d06fd26c9a.
The problem is that Bitbucket's API forwards download requests to an S3 bucket using a temporary URL. This URL includes a signature for the request, which embeds the HTTP verb. That means only GET requests are allowed, and HEAD requests would fail verification, leading to 403 erros. The same is observed when using `curl -LI ...`
v0.5 does not build due to a change in setting `arch` introduced in v0.5.2, compatibility with which was not kept in `arbor/package.py`. Since v0.5.2 is compatible with `arbor/package.py`, and is API compatible with v0.5, any users relying on v0.5 can rely on v0.5.2.
Using `-Werror` is good practice for development and testing, but causes us a great
deal of heartburn supporting multiple compiler versions, especially as newer compiler
versions add warnings for released packages. This PR adds support for suppressing
`-Werror` through spack's compiler wrappers. There are currently three modes for
the `flags:keep_werror` setting:
* `none`: (default) cancel all `-Werror`, `-Werror=*` and `-Werror-*` flags by
converting them to `-Wno-error[=]*` flags
* `specific`: preserve explicitly selected warnings as errors, such as
`-Werror=format-truncation`, but reverse the blanket `-Werror`
* `all`: keeps all `-Werror` flags
These can be set globally in config.yaml, through the config command-line flags, or
overridden by a particular package (some packages use Werror as a proxy for determining
support for other compiler features). We chose to use this approach because:
1. removing `-Werror` flags entirely broke *many* build systems, especially autoconf
based ones, because of things like checking `-Werror=feature` and making the
assumption that if that did not error other flags related to that feature would also work
2. Attempting to preserve `-Werror` in some phases but not others caused similar issues
3. The per-package setting came about because some packages, even with all these
protections, still use `-Werror` unsafely. Currently there are roughly 3 such packages
known.
* Working updates to py-antlr4-python3-runtime and py-omegaconf
* [py-antlr4-python3-runtime] added version 4.9.3
* [@spackbot] updating style on behalf of qwertos
Co-authored-by: Benjamin Meyers <bsmits@rit.edu>
Co-authored-by: qwertos <qwertos@users.noreply.github.com>
For reasons beyond me Python thinks it's a great idea to upgrade HEAD
requests to GET requests when following redirects. So, this PR adds a
better `HTTPRedirectHandler`, and also moves some ad-hoc logic around
for dealing with disabling SSL certs verification.
Also, I'm stumped by the fact that Spack's `url_exists` does not use
HEAD requests at all, so in certain cases Spack awkwardly downloads
something first to see if it can download it, and then downloads it
again because it knows it can download it. So, this PR ensures that both
urllib and botocore use HEAD requests.
Finally, it also removes some things that were there to support currently
unsupported Python versions.
Notice that the HTTP spec [section 10.3.2](https://datatracker.ietf.org/doc/html/rfc2616.html#section-10.3.2) just talks about how to deal
with POST request on redirect (whether to follow or not):
> If the 301 status code is received in response to a request other
> than GET or HEAD, the user agent MUST NOT automatically redirect the
> request unless it can be confirmed by the user, since this might
> change the conditions under which the request was issued.
> Note: When automatically redirecting a POST request after
> receiving a 301 status code, some existing HTTP/1.0 user agents
> will erroneously change it into a GET request.
Python has a comment about this, they choose to go with the "erroneous change".
But they then mess up the HEAD request while following the redirect, probably
because they were too busy discussing how to deal with POST.
See https://github.com/python/cpython/pull/99731
The package.py assumed "+mpi" in many places, without checking for the variant.
This problem went undetected, as a hard dependency on scalapack pulled an mpi
implementation into the dependency chain (this is also fixed).
Also, the +mpi variant is used select between serial and parallel mode:
It has to enable MPI and ScaLAPACK: They are inter-dependent. Compile
fails because of checks for the other if the other is not enabled.
Co-authored-by: Bernhard Kaindl <contact@bernhard.kaindl.dev>
This adds super-lazy maintainer mode to `spack checksum`: Instead of
only printing the new checksums to the terminal, `-a` and
`--add-to-package` will add the new checksums to the `package.py` file
and open it in the editor afterwards for final checks.
This PR removes [end of life](https://endoflife.date/python) versions of Python from Spack. Specifically, this includes all versions of Python older than 3.7.
See https://github.com/spack/spack/discussions/31824 for rationale. Deprecated in #32615. And #28003.
For anyone using software that relies on Python 2, you have a few options:
* Upgrade the software to support Python 3. The `3to2` tool may get you most of the way there, although more complex libraries may need manual tweaking.
* Add Python 2 as an [external package](https://spack.readthedocs.io/en/latest/build_settings.html#external-packages). Many Python libraries do not support Python 2, but you may be able to add older versions that did once upon a time.
* Use Spack 0.19. Spack 0.19 is the last release to officially support Python 3.6 and older
* Create and maintain your own [custom repository](https://spack.readthedocs.io/en/latest/repositories.html). Basically, you would need a package for Python 2 and any other Python 2-specific libraries you need.
* Add a WindowsRegistryView class, which can query for existing
package installations on Windows. This is particularly important
because some Windows packages (including those added here)
do not allow two simultaneous installs, and this can be
queried in order to provide a clear error message.
* Consolidate external path detection logic for Windows into
WindowsKitExternalPaths and WindowsCompilerExternalPaths objects.
* Add external-only packages win-sdk and wgl
* Add win-wdk (including external detection) which depends on
win-sdk
* Replace prior msmpi implementation with a source-based install
(depends on win-wdk). This install can control the install
destination (unlike the binary installation).
* Update MSVC compiler to choose vcvars based on win-sdk dependency
* Provide "msbuild" module-level variable to packages during build
* When creating symlinks on Windows, need to explicitly specify when
a symlink target is a directory
* executables_in_path no-longer defaults to using PATH (this is
now expected to be taken care of by the caller)
Spec traversals can now specify a topological ordering. A topologically-
ordered traversal with input specs X1, X2... will
* include all of X1, X2... and their children
* be ordered such that a given node is guaranteed to appear before any
of its children in the traversal
Other notes:
* Input specs can be children of other input specs (this is useful if
a user specifies a set of specs to uninstall: some of those specs
might be children of others)
* `direction="parents"` will produce a reversed topological order
(children always come before parents).
* `cover="edges"` will generate a list of edges L such that (a) input
edges will always appear before output edges and (b) if you create
a list with the destination of each edge in L the result is
topologically ordered
* test_suite.py: speed up slow test by using mock packages
* Don't resolve the sha during unit-tests
* Skip long-running test that fails, instead of executing it
* uninstall: fix accidental cubic complexity
Currently spack uninstall runs in worst case cubic time complexity
thanks to traversal during traversal during traversal while collecting
the specs to be uninstalled.
Also brings down the number of error messages printed to something
linear in the amount of matching specs instead of quadratic.
* qt6: initial commit of several basic qt6 packages
* Qt6: fix style issues
* [qt6] fix style issues, trailing spaces
* [qt6] rename to qt-* ecosystem; remove imports
* [qt6] rename dependencies; change version strings
* [qt6] list_urls
* [qt6] homepage links
* [qt6] missing closing quotes failed style check
* qt-declarative: use private _versions
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* qt-quick3d, qt-quicktimeline, qt-shadertools: use private _versions
* qt-base: rework feature defines and use run_tests
* qt: new version 6.2.4
* flake8 whitespace before comma
* qt-base: variant opengl when +gui
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* qt6: rebase and apply new black style
* qt6: apply style isort fixes
* qt6: new version 6.3.0 and 6.3.1
* qt6: add 6.3.0 and 6.3.1 to versions list
* qt6: multi-argument join_path
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* qt-base: fix isort
* qt-shadertools: no cmake_args needed
* qt-declarative: imports up front
* qt-quick3d: fix import
* qt-declarative: remove useless cmake_args
* qt-shadertools: imports and join_path fixes
* qt-quick3d: join_path fixes
* qt-declarative: join_path fixes
* Update features based on gui usage
* Update dependencies, cmake args, mac support
* Update features based on linux
* More updates
* qt-base: fix style
* qt-base: archive_files join_path
* qt-base: new version 6.3.2
* qt-{declarative,quick3d,quicktimeline,shadertools}@6.3.2
* qt-base: require libxcb@1.13: and use system xcb_xinput when on linux
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update dask and related packages
* Update package dependency specs
* Run spack style
* Add new version of locket
* Respond to comments
* Added constraints
* Add version constraints for py-dask+distributed
* Run spack style
* Update var/spack/repos/builtin/packages/py-dask/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Deprecated dask versions
* Deprecated more dask and distirbuted
* spack style --fix
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* graphviz: remove cyclic dep to svg when pangocairo and poppler+glib failure
* graphviz: remove cyclic dep to svg when pangocairo and poppler+glib failure
* Add a regression test for 33928
* PackageBase should not set `(build|install)_time_test_callbacks`
* Fix audits by preserving the current semantic
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
gcc@10: Newer binutils than RHEL7/8's are required to for guaranteed operaton. Therefore, on RHEL7/8, reject ~binutils. You need to add +binutils to be sure to have binutils which are recent enough.
See this discussion with the OpenBLAS devs for reference:
https://github.com/xianyi/OpenBLAS/issues/3805#issuecomment-1319878852
Co-authored-by: Bernhard Kaindl <contact@bernhard.kaindl.dev>
Add a dependency on python versions less than 3.10 in order to work
around a bug in libxml2's configure script that fails to parse python
version strings with more than one character for the minor version.
The bug is present in v2.10.1, but has been fixed in 2.10.2.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Bernhard Kaindl <43588962+bernhardkaindl@users.noreply.github.com>
* add version 1.46.0 to bioconductor package r-a4
* add version 1.46.0 to bioconductor package r-a4base
* add version 1.46.0 to bioconductor package r-a4classif
* add version 1.46.0 to bioconductor package r-a4core
* add version 1.46.0 to bioconductor package r-a4preproc
* add version 1.46.0 to bioconductor package r-a4reporting
* add version 1.52.0 to bioconductor package r-absseq
* add version 1.28.0 to bioconductor package r-acde
* add version 1.76.0 to bioconductor package r-acgh
* add version 2.54.0 to bioconductor package r-acme
* add version 1.68.0 to bioconductor package r-adsplit
* add version 1.70.0 to bioconductor package r-affxparser
* add version 1.76.0 to bioconductor package r-affy
* add version 1.74.0 to bioconductor package r-affycomp
* add version 1.58.0 to bioconductor package r-affycompatible
* add version 1.56.0 to bioconductor package r-affycontam
* add version 1.70.0 to bioconductor package r-affycoretools
* add version 1.46.0 to bioconductor package r-affydata
* add version 1.50.0 to bioconductor package r-affyilm
* add version 1.68.0 to bioconductor package r-affyio
* add version 1.74.0 to bioconductor package r-affyplm
* add version 1.44.0 to bioconductor package r-affyrnadegradation
* add version 1.46.0 to bioconductor package r-agdex
* add version 3.30.0 to bioconductor package r-agilp
* add version 2.48.0 to bioconductor package r-agimicrorna
* add version 1.30.0 to bioconductor package r-aims
* add version 1.30.0 to bioconductor package r-aldex2
* add version 1.36.0 to bioconductor package r-allelicimbalance
* add version 1.24.0 to bioconductor package r-alpine
* add version 2.60.0 to bioconductor package r-altcdfenvs
* add version 2.22.0 to bioconductor package r-anaquin
* add version 1.26.0 to bioconductor package r-aneufinder
* add version 1.26.0 to bioconductor package r-aneufinderdata
* add version 1.70.0 to bioconductor package r-annaffy
* add version 1.76.0 to bioconductor package r-annotate
* add version 1.60.0 to bioconductor package r-annotationdbi
* add version 1.22.0 to bioconductor package r-annotationfilter
* add version 1.40.0 to bioconductor package r-annotationforge
* add version 3.6.0 to bioconductor package r-annotationhub
* add version 3.28.0 to bioconductor package r-aroma-light
* add version 1.30.0 to bioconductor package r-bamsignals
* add version 2.14.0 to bioconductor package r-beachmat
* add version 2.58.0 to bioconductor package r-biobase
* add version 2.6.0 to bioconductor package r-biocfilecache
* add version 0.44.0 to bioconductor package r-biocgenerics
* add version 1.8.0 to bioconductor package r-biocio
* add version 1.16.0 to bioconductor package r-biocneighbors
* add version 1.32.1 to bioconductor package r-biocparallel
* add version 1.14.0 to bioconductor package r-biocsingular
* add version 2.26.0 to bioconductor package r-biocstyle
* add version 3.16.0 to bioconductor package r-biocversion
* add version 2.54.0 to bioconductor package r-biomart
* add version 1.26.0 to bioconductor package r-biomformat
* add version 2.66.0 to bioconductor package r-biostrings
* add version 1.46.0 to bioconductor package r-biovizbase
* add version 1.8.0 to bioconductor package r-bluster
* add version 1.66.1 to bioconductor package r-bsgenome
* add version 1.34.0 to bioconductor package r-bsseq
* add version 1.40.0 to bioconductor package r-bumphunter
* add version 2.64.0 to bioconductor package r-category
* add version 2.28.0 to bioconductor package r-champ
* add version 2.30.0 to bioconductor package r-champdata
* add version 1.48.0 to bioconductor package r-chipseq
* add version 4.6.0 to bioconductor package r-clusterprofiler
* add version 1.34.0 to bioconductor package r-cner
* add version 1.30.0 to bioconductor package r-codex
* add version 2.14.0 to bioconductor package r-complexheatmap
* add version 1.72.0 to bioconductor package r-ctc
* add version 2.26.0 to bioconductor package r-decipher
* add version 0.24.0 to bioconductor package r-delayedarray
* add version 1.20.0 to bioconductor package r-delayedmatrixstats
* add version 1.38.0 to bioconductor package r-deseq2
* add version 1.44.0 to bioconductor package r-dexseq
* add version 1.40.0 to bioconductor package r-dirichletmultinomial
* add version 2.12.0 to bioconductor package r-dmrcate
* add version 1.72.0 to bioconductor package r-dnacopy
* add version 3.24.1 to bioconductor package r-dose
* add version 2.46.0 to bioconductor package r-dss
* add version 3.40.0 to bioconductor package r-edger
* add version 1.18.0 to bioconductor package r-enrichplot
* add version 2.22.0 to bioconductor package r-ensembldb
* add version 1.44.0 to bioconductor package r-exomecopy
* add version 2.6.0 to bioconductor package r-experimenthub
* add version 1.24.0 to bioconductor package r-fgsea
* add version 2.70.0 to bioconductor package r-gcrma
* add version 1.34.0 to bioconductor package r-gdsfmt
* add version 1.80.0 to bioconductor package r-genefilter
* add version 1.34.0 to bioconductor package r-genelendatabase
* add version 1.70.0 to bioconductor package r-genemeta
* add version 1.76.0 to bioconductor package r-geneplotter
* add version 1.20.0 to bioconductor package r-genie3
* add version 1.34.3 to bioconductor package r-genomeinfodb
* update r-genomeinfodbdata
* add version 1.34.0 to bioconductor package r-genomicalignments
* add version 1.50.2 to bioconductor package r-genomicfeatures
* add version 1.50.1 to bioconductor package r-genomicranges
* add version 2.66.0 to bioconductor package r-geoquery
* add version 1.46.0 to bioconductor package r-ggbio
* add version 3.6.2 to bioconductor package r-ggtree
* add version 2.8.0 to bioconductor package r-glimma
* add version 1.10.0 to bioconductor package r-glmgampoi
* add version 5.52.0 to bioconductor package r-globaltest
* update r-go-db
* add version 1.18.0 to bioconductor package r-gofuncr
* add version 2.24.0 to bioconductor package r-gosemsim
* add version 1.50.0 to bioconductor package r-goseq
* add version 2.64.0 to bioconductor package r-gostats
* add version 1.76.0 to bioconductor package r-graph
* add version 1.60.0 to bioconductor package r-gseabase
* add version 1.30.0 to bioconductor package r-gtrellis
* add version 1.42.0 to bioconductor package r-gviz
* add version 1.26.0 to bioconductor package r-hdf5array
* add version 1.70.0 to bioconductor package r-hypergraph
* add version 1.34.0 to bioconductor package r-illumina450probevariants-db
* add version 0.40.0 to bioconductor package r-illuminaio
* add version 1.72.0 to bioconductor package r-impute
* add version 1.36.0 to bioconductor package r-interactivedisplaybase
* add version 2.32.0 to bioconductor package r-iranges
* add version 1.58.0 to bioconductor package r-kegggraph
* add version 1.38.0 to bioconductor package r-keggrest
* add version 3.54.0 to bioconductor package r-limma
* add version 2.50.0 to bioconductor package r-lumi
* add version 1.74.0 to bioconductor package r-makecdfenv
* add version 1.76.0 to bioconductor package r-marray
* add version 1.10.0 to bioconductor package r-matrixgenerics
* add version 1.6.0 to bioconductor package r-metapod
* add version 2.44.0 to bioconductor package r-methylumi
* add version 1.44.0 to bioconductor package r-minfi
* add version 1.32.0 to bioconductor package r-missmethyl
* add version 1.78.0 to bioconductor package r-mlinterfaces
* add version 1.10.0 to bioconductor package r-mscoreutils
* add version 2.24.0 to bioconductor package r-msnbase
* add version 2.54.0 to bioconductor package r-multtest
* add version 1.36.0 to bioconductor package r-mzid
* add version 2.32.0 to bioconductor package r-mzr
* add version 1.60.0 to bioconductor package r-oligoclasses
* update r-org-hs-eg-db
* add version 1.40.0 to bioconductor package r-organismdbi
* add version 1.38.0 to bioconductor package r-pathview
* add version 1.90.0 to bioconductor package r-pcamethods
* update r-pfam-db
* add version 1.42.0 to bioconductor package r-phyloseq
* add version 1.60.0 to bioconductor package r-preprocesscore
* add version 1.30.0 to bioconductor package r-protgenerics
* add version 1.32.0 to bioconductor package r-quantro
* add version 2.30.0 to bioconductor package r-qvalue
* add version 1.74.0 to bioconductor package r-rbgl
* add version 2.38.0 to bioconductor package r-reportingtools
* add version 2.42.0 to bioconductor package r-rgraphviz
* add version 2.42.0 to bioconductor package r-rhdf5
* add version 1.10.0 to bioconductor package r-rhdf5filters
* add version 1.20.0 to bioconductor package r-rhdf5lib
* add version 2.0.0 to bioconductor package r-rhtslib
* add version 1.74.0 to bioconductor package r-roc
* add version 1.26.0 to bioconductor package r-rots
* add version 2.14.0 to bioconductor package r-rsamtools
* add version 1.58.0 to bioconductor package r-rtracklayer
* add version 0.36.0 to bioconductor package r-s4vectors
* add version 1.6.0 to bioconductor package r-scaledmatrix
* add version 1.26.0 to bioconductor package r-scater
* add version 1.12.0 to bioconductor package r-scdblfinder
* add version 1.26.0 to bioconductor package r-scran
* add version 1.8.0 to bioconductor package r-scuttle
* add version 1.64.0 to bioconductor package r-seqlogo
* add version 1.56.0 to bioconductor package r-shortread
* add version 1.72.0 to bioconductor package r-siggenes
* add version 1.20.0 to bioconductor package r-singlecellexperiment
* add version 1.32.0 to bioconductor package r-snprelate
* add version 1.48.0 to bioconductor package r-snpstats
* add version 2.34.0 to bioconductor package r-somaticsignatures
* add version 1.10.0 to bioconductor package r-sparsematrixstats
* add version 1.38.0 to bioconductor package r-spem
* add version 1.36.0 to bioconductor package r-sseq
* add version 1.28.0 to bioconductor package r-summarizedexperiment
* add version 3.46.0 to bioconductor package r-sva
* add version 1.36.0 to bioconductor package r-tfbstools
* add version 1.20.0 to bioconductor package r-tmixclust
* add version 2.50.0 to bioconductor package r-topgo
* add version 1.22.0 to bioconductor package r-treeio
* add version 1.26.0 to bioconductor package r-tximport
* add version 1.26.0 to bioconductor package r-tximportdata
* add version 1.44.0 to bioconductor package r-variantannotation
* add version 3.66.0 to bioconductor package r-vsn
* add version 2.4.0 to bioconductor package r-watermelon
* add version 2.44.0 to bioconductor package r-xde
* add version 1.56.0 to bioconductor package r-xmapbridge
* add version 0.38.0 to bioconductor package r-xvector
* add version 1.24.0 to bioconductor package r-yapsa
* add version 1.24.0 to bioconductor package r-yarn
* add version 1.44.0 to bioconductor package r-zlibbioc
* make version resource consistent for r-bsgenome-hsapiens-ucsc-hg19
* make version resource consistent for r-go-db
* make version resource consistent for r-kegg-db
* make version resource consistent for r-org-hs-eg-db
* make version resource consistent for r-pfam-db
* new package: r-ggrastr
* Patches not needed for new version
* new package: r-hdo-db
* new package: r-ggnewscale
* new package: r-gson
* Actually depends on ggplot2@3.4.0:
* Fix formatting of r-hdo-db
* Fix dependency version specifiers
* Clean up duplicate dependency references
* Enable hdf5 build (including +mpi) on Windows
* This includes updates to hdf5 dependencies openssl (minor edit) and
bzip2 (more-extensive edits)
* Add binary-based installation of msmpi (this is currently the only
supported MPI implementation in Spack for Windows). Note that this
does not install to the Spack-specified prefix. This implementation
will be replaced with a source-based implementation
Co-authored-by: John Parent <john.parent@kitware.com>
Setting PYTHONHOME is rarely needed (since each interpreter has
various ways of setting it automatically) and very often it is
difficult to get right manually.
For instance, the change done to set PYTHONHOME to
sysconfig["base_prefix"] broke bootstrapping dev dependencies
of Spack for me, when working inside a virtual environment in Linux.
* replace mpi as a variant instead of dependency
* separate serial and MPI dependencies
* configure args depending on serial or mpi variant
* reformat with black
In #3113, `https` was removed to ensure that `curl` can be bootstrapped
without SSL being present. This was lost in #25672 which aimed to use
`https` where possible.
Co-authored-by: Bernhard Kaindl <43588962+bernhardkaindl@users.noreply.github.com>
#32942 fixed bootstrapping on Windows by having the core Spack
code explicitly add the Clingo package bin/ directory as a
DLL path.
Since then, #33400 has been merged, which ensures that the Python
module installed by the Spack `clingo` package can find the DLLs
in bin/.
Note that this only works for Spack instances which have been
bootstrapped after #33400: for installations bootstrapped before
then, you will need to run `spack clean -b` (this would only
be needed for Spack instances running on Windows).
Buildin neovim@0.8.0 complains (for me) about Lua's lpeg and mpack
packages not being available at build time. Removing the link-only
setting in the dependencies for these two packages fixes the build for
me.
Revamp the timer so we always have a designated begin and end.
Fix a bug where the phase timer was stopped before the phase started,
resulting in incorrect timing reports in timers.json.
Add spack.ld_so_conf.host_dynamic_linker_search_paths
Retrieve the current host runtime search paths for shared libraries;
for GNU and musl Linux we try to retrieve the dynamic linker from the
current Python interpreter and then find the corresponding config file
(e.g. ld.so.conf or ld-musl-<arch>.path). Similar can be done for
BSD and others, but this is not implemented yet. The default paths
are always returned. We don't check if the listed directories exist.
Use this in spack external find for libraries.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
While binaries built for PRs that get merged must still be rebuilt
in develop pipelines, they can be used by other PRs that find they
would otherwise need to rebuild them. Now that spackbot is
managing copying PR binaries from merged PRs into a shared location,
keeping it pruned to a reasonable size, and making sure the indices
are up to date, spack can use these mirrors as a potential source
of binaries.
I'm finding I often want the date in my paths and it would be nice if spack had a config variable for this.
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* Remove CI jobs related to Python 2.7
* Remove Python 2.7 specific code from Spack core
* Remove externals for Python 2 only
* Remove llnl.util.compat
We added a hotfix to releases/v0.19 with a feature flag, but the flag
is incompatible with the config schema on `develop`.
- [x] Ensure schema is compatible on develop even though config option is unused.
* Speed-up bootstrap mirror unit test
The unit test doesn't need to concretize, since it checks
only metadata for the mirror.
* architecture.py: use "default_mock_concretization" for slow test
Environments and environment views have taken over the role of `spack activate/deactivate`, and we should deprecate these commands for several reasons:
- Global activation is a really poor idea:
- Install prefixes should be immutable; since they can have multiple, unrelated dependents; see below
- Added complexity elsewhere: verification of installations, tarballs for build caches, creation of environment views of packages with unrelated extensions "globally activated"... by removing the feature, it gets easier for people to contribute, and we'd end up with fewer bugs due to edge cases.
- Environment accomplish the same thing for non-global "activation" i.e. `spack view`, but better.
Also we write in the docs:
```
However, Spack global activations have two potential drawbacks:
#. Activated packages that involve compiled C extensions may still
need their dependencies to be loaded manually. For example,
``spack load openblas`` might be required to make ``py-numpy``
work.
#. Global activations "break" a core feature of Spack, which is that
multiple versions of a package can co-exist side-by-side. For example,
suppose you wish to run a Python package in two different
environments but the same basic Python --- one with
``py-numpy@1.7`` and one with ``py-numpy@1.8``. Spack extensions
will not support this potential debugging use case.
```
Now that environments are established and views can take over the role of activation
non-destructively, we can remove global activation/deactivation.
* add version 1.7-20 to r-ade4
* add version 1.1-13 to r-adephylo
* add version 0.3-20 to r-adespatial
* add version 1.2-0 to r-afex
* add version 0.8-19 to r-amap
* add version 0.1.8 to r-aplot
* add version 2.0.8 to r-biasedurn
* add version 2.4-4 to r-bio3d
* add version 1.30.19 to r-biocmanager
* add version 1.2-9 to r-brobdingnag
* add version 0.4.1 to r-bslib
* add version 3.7.3 to r-callr
* add version 3.1-1 to r-car
* add version 0.3-62 to r-clue
* add version 1.2.0 to r-colourpicker
* add version 1.8.1 to r-commonmark
* add version 0.4.3 to r-cpp11
* add version 1.14.4 to r-data-table
* add version 1.34 to r-desolve
* add version 2.4.5 to r-devtools
* add version 0.6.30 to r-digest
* add version 0.26 to r-dt
* add version 1.7-12 to r-e1071
* add version 1.8.2 to r-emmeans
* add version 1.1.16 to r-exomedepth
* add version 0.1-8 to r-expint
* add version 0.4.0 to r-fontawesome
* add version 1.5-2 to r-fracdiff
* add version 1.10.0 to r-future-apply
* add version 3.0.1 to r-ggmap
* add version 3.4.0 to r-ggplot2
* add version 2.1.0 to r-ggraph
* add version 0.6.4 to r-ggsignif
* add version 0.6-7 to r-gmp
* add version 2022.10-2 to r-gparotation
* add version 0.8.3 to r-graphlayouts
* add version 1.8.8 to r-grbase
* add version 2.1-0 to r-gstat
* add version 0.18.6 to r-insight
* add version 1.3.1 to r-irkernel
* add version 1.8.3 to r-jsonlite
* add version 1.7.0 to r-lava
* add version 1.1-31 to r-lme4
* add version 5.6.17 to r-lpsolve
* add version 5.5.2.0-17.9 to r-lpsolveapi
* add version 1.2.9 to r-mapproj
* add version 3.4.1 to r-maps
* add version 1.1-5 to r-maptools
* add version 1.3 to r-markdown
* add version 6.0.0 to r-mclust
* add version 4.2-2 to r-memuse
* add version 1.8-41 to r-mgcv
* add version 1.2.5 to r-minqa
* add version 0.3.7 to r-nanotime
* add version 2.4.1.1 to r-nfactors
* add version 3.1-160 to r-nlme
* add version 2.7-1 to r-nmof
* add version 0.60-16 to r-np
* add version 2.0.4 to r-openssl
* add version 4.2.5.1 to r-openxlsx
* add version 0.3-8 to r-pbdzmq
* add version 2.0-3 to r-pcapp
* add version 0.7.0 to r-philentropy
* add version 2.0.3 to r-pkgcache
* add version 1.3.1 to r-pkgload
* add version 1.10-4 to r-polyclip
* add version 3.8.0 to r-processx
* add version 1.7.2 to r-ps
* add version 2.12.1 to r-r-utils
* add version 1.2.4 to r-ragg
* add version 3.7 to r-rainbow
* add version 0.0.20 to r-rcppannoy
* add version 0.3.3.9.3 to r-rcppeigen
* add version 0.3.12 to r-rcppgsl
* add version 1.0.2 to r-recipes
* add version 1.1.10 to r-rmutil
* add version 0.10.24 to r-rmysql
* add version 2.4.8 to r-rnexml
* add version 4.1.19 to r-rpart
* add version 1.7-2 to r-rrcov
* add version 0.8.28 to r-rsconnect
* add version 0.25 to r-servr
* add version 1.7.3 to r-shiny
* add version 3.0-0 to r-spatstat-data
* add version 3.0-3 to r-spatstat-geom
* add version 3.0-1 to r-spatstat-random
* add version 3.0-0 to r-spatstat-sparse
* add version 3.0-1 to r-spatstat-utils
* add version 1.8.0 to r-styler
* add version 3.4.1 to r-sys
* add version 1.5-2 to r-tclust
* add version 0.0.9 to r-tfmpvalue
* add version 1.2.0 to r-tidyselect
* add version 0.10-52 to r-tseries
* add version 4.2.2 to r-v8
* add version 0.5.0 to r-vctrs
* add version 2.6-4 to r-vegan
* add version 0.7.0 to r-wk
* add version 0.34 to r-xfun
* add version 1.0.6 to r-xlconnect
* add version 3.99-0.12 to r-xml
* add version 0.12.2 to r-xts
* add version 1.0-33 to r-yaimpute
* add version 2.3.6 to r-yaml
* add version 2.2.2 to r-zip
* add version 2.2-7 to r-deoptim
* add version 4.3.1 to r-ergm
* add version 0.18 to r-evaluate
* add version 1.29.0 to r-future
* add version 0.0.8 to r-ggfun
* add version 0.9.2 to r-ggrepel
* add version 1.9.0 to r-lubridate
* add version 4.10.1 to r-plotly
* add version 0.2.12 to r-rcppcctz
* add version 1.2 to r-rook
* add version 1.6-1 to r-segmented
* add version 4.2.1 to r-seurat
* add version 4.1.3 to r-seuratobject
* add version 1.0-9 to r-sf
* add version 1.5-1 to r-sp
* add version 1.8.1 to r-styler
* new package: r-timechange
* new package: r-stars
* new package: r-sftime
* new package: r-spatstat-explore
Co-authored-by: glennpj <glennpj@users.noreply.github.com>
* updates and fixes for libpressio
* differentiate between standalone and build tests
* add e4s tags
Co-authored-by: Robert Underwood <runderwood@anl.gov>
* include py-darshan
* include requested changes
* fix required versions
* fix style
* fix style
* Update package.py
* Update var/spack/repos/builtin/packages/py-darshan/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Boost 1.64.0 has build errors when building the python and MPI modules. This was previously just a comment in the package.py which allowed broken specs to concretize. The comments are now expressed in conflicts to prevent this.
Currently, external `PythonPackage`s cause install failures because the logic in `PythonPackage` assumes that it can ask for `spec["python"]`. Because we chop off externals' dependencies, an external Python extension may not have a `python` dependency.
This PR resolves the issue by guaranteeing that a `python` node is present in one of two ways:
1. If there is already a `python` node in the DAG, we wire the external up to it.
2. If there is no existing `python` node, we wire up a synthetic external `python` node, and we assume that it has the same prefix as the external.
The assumption in (2) isn't always valid, but it's better than leaving the user with a non-working `PythonPackage`.
The logic here is specific to `python`, but other types of extensions could take advantage of it. Packages need only define `update_external_dependencies(self)`, and this method will be called on externals after concretization. This likely needs to be fleshed out in the future so that any added nodes are included in concretization, but for now we only bolt on dependencies post-concretization.
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
* [mfem] updates related to building with cuda
* [hypre] tweak to support building with external ROCm/HIP
* [mfem] more tweaks related to building with +rocm
* [mfem] temporary (?) workaround for issue #33684
* [mfem] fix style
* [mfem] fix +shared+miniapps install
* Add checksum for py-protobuf 4.21.7, protobuf 21.7
* Update var/spack/repos/builtin/packages/protobuf/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update var/spack/repos/builtin/packages/protobuf/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/protobuf/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update package.py
* Delete protoc2.5.0_aarch64.patch
* Update package.py
* Restore but deprecate py-protobuf 3.0.0a/b; deprecate py-tensorflow 0.x
* Fix audit
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Spack currently creates a temporary sbang that is moved "atomically" in place,
but this temporary causes races when multiple processes start installing sbang.
Let's just stick to an idempotent approach. Notice that we only re-install sbang
if Spack updates it (since we do file compare), and sbang was only touched
18 times in the past 6 years, whereas we hit the sbang tempfile issue
frequently with parallel install on a fresh spack instance in CI.
Also fixes a bug where permissions weren't updated if config changed but
the latest version of the sbang file was already installed.
The `intel` compiler at versions > 20 is provided by the `intel-oneapi-compilers-classic`
package (a thin wrapper around the `intel-oneapi-compilers` package), and the `oneapi`
compiler is provided by the `intel-oneapi-compilers` package.
Prior to this work, neither of these compilers could be bootstrapped by Spack as part of
an install with `install_missing_compilers: True`.
Changes made to make these two packages bootstrappable:
1. The `intel-oneapi-compilers-classic` package includes a bin directory and symlinks
to the compiler executables, not just logical pointers in Spack.
2. Spack can look for bootstrapped compilers in directories other than `$prefix/bin`,
defined on a per-package basis
3. `intel-oneapi-compilers` specifies a non-default search directory for the
compiler executables.
4. The `spack.compilers` module now can make more advanced associations between
packages and compilers, not just simple name translations
5. Spack support for lmod hierarchies accounts for differences between package
names and the associated compiler names for `intel-oneapi-compilers/oneapi`,
`intel-oneapi-compilers-classic/intel@20:`, `llvm+clang/clang`, and
`llvm-amdgpu/rocmcc`.
- [x] full end-to-end testing
- [x] add unit tests
* Add zstd support for elfutils
Not defining `+zstd` implies `--without-zstd` flag to configure.
This avoids automatic library detection and thus make the build only
depends on Spack installed dependencies.
* Use autotools helper "with_or_without"
* Revert use of with_or_without
Using `with_or_without()` with `variant` keyword does not seem to work.
"spack install foo" no longer adds package "foo" to the environment
(i.e. to the list of root specs) by default: you must specify "--add".
Likewise "spack uninstall foo" no longer removes package "foo" from
the environment: you must specify --remove. Generally this means
that install/uninstall commands will no longer modify the users list
of root specs (which many users found problematic: they had to
deactivate an environment if they wanted to uninstall a spec without
changing their spack.yaml description).
In more detail: if you have environments e1 and e2, and specs [P, Q, R]
such that P depends on R, Q depends on R, [P, R] are in e1, and [Q, R]
are in e2:
* `spack uninstall --dependents --remove r` in e1: removes R from e1
(but does not uninstall it) and uninstalls (and removes) P
* `spack uninstall -f --dependents r` in e1: will uninstall P, Q, and
R (i.e. e2 will have dependent specs uninstalled as a side effect)
* `spack uninstall -f --dependents --remove r` in e1: this uninstalls
P, Q, and R, and removes [P, R] from e1
* `spack uninstall -f --remove r` in e1: uninstalls R (so it is
"missing" in both environments) and removes R from e1 (note that e1
would still install R as a dependency of P, but it would no longer
be listed as a root spec)
* `spack uninstall --dependents r` in e1: will fail because e2 needs R
Individual unit tests were created for each of these scenarios.
Somehow a network error when cloning the repo for ci gets
categorized by gitlab as a script failure. To make sure we retry
jobs that failed for that reason or a similar one, include
"script_failure" as one of the reasons for retrying service jobs
(which include "no specs to rebuild" jobs, update buildcache
index jobs, and temp storage cleanup jobs.
Add a `project` block to the toml config along with development and CI
dependencies and a minimal `build-system` block, doing basically
nothing, so that spack can be bootstrapped to a full development
environment with:
```shell
$ hatch -e dev shell
```
or for a minimal environment without hatch:
```shell
$ python3 -m venv venv
$ source venv/bin/activate
$ python3 -m pip install --upgrade pip
$ python3 -m pip install -e '.[dev]'
```
This means we can re-use the requirements list throughout the workflow
yaml files and otherwise maintain this list in *one place* rather than
several disparate ones. We may be stuck with a couple more temporarily
to continue supporting python2.7, but aside from that it's less places
to get out of sync and a couple new bootstrap options.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This change uses the aws cli, if available, to retrieve spec files
from the mirror to a local temp directory, then parallelizes the
reading of those files from disk using multiprocessing.ThreadPool.
If the aws cli is not available, then a ThreadPool is used to fetch
and read the spec files from the mirror.
Using aws cli results in ~16 times speed up to recreate the binary
mirror index, while just parallelizing the fetching and reading
results in ~3 speed up.
The compiler bootstrapping logic currently does not add a task when the compiler package is already in the install task queue. This causes failures when the compiler package is added without the additional metadata telling the task to update the compilers list.
Solution: requeue compilers for bootstrapping when needed, to update `task.compiler` metadata.
Currently, develop specs that are not roots and are not explicitly listed dependencies
of the roots are not applied.
- [x] ensure dev specs are applied.
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
`spack env create` enables a view by default (in a weird hidden
directory, but well...). This is asking for trouble with the other
default of `concretizer:unify:false`, since having different flavors of
the same spec in an environment, leads to collision errors when
generating the view.
A change of defaults would improve user experience:
However, `unify:true` makes most sense, since any time the issue is
brought up in Slack, the user changes the concretization config, since
it wasn't the intention to have different flavors of the same spec, and
install times are decreased.
Further we improve the docs and drop the duplicate root spec limitation
Dependencies specified by hash are unique in Spack in that the abstract
specs are created with internal structure. In this case, the constraint
generation for spec matrices fails due to flattening the structure.
It turns out that the dep_difference method for Spec.constrain does not
need to operate on transitive deps to ensure correctness. Removing transitive
deps from this method resolves the bug.
- [x] Includes regression test
Without this, Meson will use its Wraps to automatically download and
install dependencies. We want to manage dependencies explicitly,
therefore disable this functionality.
Currently, Spack can fail for a valid spec if the spec is constructed from overlapping, but not conflicting, concrete specs via the hash.
For example, if abcdef and ghijkl are the hashes of specs that both depend on zlib/mnopqr, then foo ^/abcdef ^/ghijkl will fail to construct a spec, with the error message "Cannot depend on zlib... twice".
This PR changes this behavior to check whether the specs are compatible before failing.
With this PR, foo ^/abcdef ^/ghijkl will concretize.
As a side-effect, so will foo ^zlib ^zlib and other specs that are redundant on their dependencies.
* ADD version 0.19.0 in py-gym recipe
* Fix py-gym download url and dependencies for v0.19.0
* Fix stupid error in previous commit: no change in py-cloudpickle dep
* Yes, I should've paid more attention! O:)
I think now it is right, thanks!
Argparse started raising ArgumentError exceptions
when the same parser is added twice. Therefore, we
perform the addition only if the parser is not there
already
Port match syntax to our unparser
Compilers and linker optimize string constants for space by aliasing
them when one is a suffix of another. For gcc / binutils this happens
already at -O1, due to -fmerge-constants. This means that we have
to take care during relocation to always preserve a certain length
of the suffix of those prefixes that are C-strings.
In this commit we pick length 7 as a safe suffix length, assuming the
suffix is typically the 7 characters from the hash (i.e. random), so
it's unlikely to alias with any string constant used in the sources.
In general we now pad shortened strings from the left with leading
dir seperators, but in the case of C-strings that are much shorter
and don't share a common suffix (due to projections), we do allow
shrinking the C-string, appending a null, and retaining the old part
of the prefix.
Also when rewiring, we ensure that the new hash preserves the last
7 bytes of the old hash.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
A user may want to set some attributes on a package without actually modifying the package (e.g. if they want to git pull updates to the package without conflicts). This PR adds a per-package configuration section called "set", which is a dictionary of attribute names to desired values. For example:
packages:
openblas:
package_attributes:
submodules: true
git: "https://github.com/myfork/openblas"
in this case, the package will always retrieve git submodules, and will use an alternate location for the git repo.
While git, url, and submodules are the attributes for which we envision the most usage, this allows any attribute to be overridden, and the acceptable values are any value parseable from yaml.
Newer versions of the CrayPE for EX systems have standalone compiler executables for CCE and compiler wrappers for Cray MPICH. With those, we can treat the cray systems as part of the linux platform rather than having a separate cray platform.
This PR:
- [x] Changes cray platform detection to ignore EX systems with Craype version 21.10 or later
- [x] Changes the cce compiler to be detectable via paths
- [x] Changes the spack compiler wrapper to understand the executable names for the standalone cce compiler (`craycc`, `crayCC`, `crayftn`).
For some instances of externally-provided Python (e.g. Homebrew),
the LDLIBRARY/LIBRARY config variables don't actually refer to
libraries and should therefore be excluded from ".libs".
Only enable the hdf5-vfd-gds package if it can compile.
- hdf5-vfd-gds needs cuda@11.7.1+ to be able to `find_library` for cuFile.
- Only enable hdf5-vfd-gds in the sdk if cuda@11.7.1+ is available.
If an earlier version of cuda is being used, do not depend on the
hdf5-vfd-gds package at all.
* take two
* Add missing import statement
* Group dependencies together
* Extract libtiff arguments
* Extract libpng arguments
* Push preamble variable into png_args and tiff_args
* Extract setting args associated with the screenshot variant
* Inlined a few variables
* Modify only build targets and install targets
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Whenever the rpath string actually _grows_, it falls back to patchelf,
when it stays the same length or gets shorter, we update it in-place,
padded with null bytes.
This PR only deals with absolute -> absolute rpath replacement. We don't
use `_build_tarball(relative=True)` in our CI. If `relative` then it falls
back to the old replacement code.
With this PR, relocation time goes down significantly, likely because patchelf
does some odd things with mmap, causing lots of overhead. Example:
- `binutils`: 700MB installed, goes from `1.91s` to `0.57s`, or `3.4x` faster.
Relocation time: 27% -> 10% of total install time
- `llvm`: 6.8GB installed, goes from `28.56s` to `5.38`, or `5.3x` faster.
Relocation time: 44% -> 13% of total install time
The bottleneck is now decompression.
Note: I'm somewhat confused about the "relative rpath" code paths. Right
now this PR only deals with absolute -> absolute replacement. As far as
I understand, if you embrace relative rpaths when uploading to the
buildcache, the whole point is you _don't_ want to patch rpaths on
install? So it seems fine to not expand `$ORIGIN` again imho.
When a package asks for non-parallel make, we need to force `make -j1` because just doing `make` will run in parallel under jobserver (e.g. `spack env depfile`).
We now always add `-j1` when asked for a non-parallel execution (even if there is no jobserver).
And each `MakeExecutable` can now ask for jobserver support or not. For example: the default `ninja` does not support jobserver so spack applies the default `-j`, but `ninja@kitware` or `ninja-fortran` does, so spack doesn't add `-j`.
Tips: you can run `SPACK_INSTALL_FLAGS=-j1 make -f spack-env-depfile.make -j8` to avoid massive job-spawning because of build tools that don't support jobserver (ninja).
* testing ssh key
* test
* LR : Creating the packge to install the gegelati app
* LR : Gegelati, a TPG C++ library added and fully tested
* LR : adjusting for fork
* LR: taking out the boilerplates
* LR: taking out the rest
We try to avoid non-default variant values in the concretizer, but this doesn't make
sense for variants forced to take some non-default value by variant propagation.
Counting this as a penalty effectively biases the concretizer for small specs dependency
graphs -- something we try very hard to avoid elsewhere because it can lead to very
strange decisions.
Example: with the penalty, `spack spec hdf5` will choose the default `openmpi` as its
`mpi` provider, but `spack spec hdf5 ~~shared` will choose `mpich` because it has to set
fewer non-default variant values because `mpich`'s DAG is smaller. That's not a good
reason to prefer a non-default virtual provider.
To fix this, if the user explicitly requests a non-default value to be propagated, there
shouldn't be a penalty. Variant values set on the CLI already don't count as default; we
just need to extend that to propagated values.
Adds another post install hook that loops over the install prefix, looking for shared libraries type of ELF files, and sets the soname to their own absolute paths.
The idea being, whenever somebody links against those libraries, the linker copies the soname (which is the absolute path to the library) as a "needed" library, so that at runtime the dynamic loader realizes the needed library is a path which should be loaded directly without searching.
As a result:
1. rpaths are not used for the fixed/static list of needed libraries in the dynamic section (only for _actually_ dynamically loaded libraries through `dlopen`), which largely solves the issue that Spack's rpaths are a heuristic (`<prefix>/lib` and `<prefix>/lib64` might not be where libraries really are...)
2. improved startup times (no library search required)
Untouched spec pruning was added to reduce the number of specs
developers see getting rebuilt in their PR pipelines that they
don't understand. Because the state of the develop mirror lags
quite far behind the tip of the develop branch, PRs often find
they need to rebuild things untouched by their PR.
Untouched spec pruning was previously implemented by finding all
specs in the environment with names of packages touched by the PR,
traversing in both directions the DAGS of those specs, and adding
all dependencies as well as dependents to a list of concrete specs
that should not be considered for pruning.
We found that this heuristic results in too many pruned specs, and
that dependents of touched specs must have all their dependencies
added to the list of specs that should not be considered for pruning.
* SEACAS: Update package.py to handle new SEACAS project name
The base project name for the SEACAS project has changed from
"SEACASProj" to "SEACAS" as of @2022-10-14, so the package
needed to be updated to use the new project name when needed.
The refactor also changes several:
"-DSome_CMAKE_Option:BOOL=ON"
to
define("Some_CMAKE_Option", True)
* SEACAS: Additional refactorings
* Replaced all cmake "-Dsomething=other" lines with either `define`
or `define_from_variant` functions.
Consolidated the application (fortran, legacy, all) enabling lines
into loops over the code names. Easier to see categorization of
applications and also to add/move/remove an application
Reordered some lines; general cleanup and restructuring.
* Address flake8 issues
* Remove trailing whitespace
* Reformat using black
* add new package: py-pylatex
* fix bugs
* add extras indicated in setup.py
* Update var/spack/repos/builtin/packages/py-pylatex/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pylatex/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* improvements
* remove git merge related lines
* tidy
* Update var/spack/repos/builtin/packages/py-pylatex/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* remove variant
* [@spackbot] updating style on behalf of Sinan81
Co-authored-by: sbulut <sbulut@3vgeomatics.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Sinan81 <Sinan81@users.noreply.github.com>
This issue was introduced in #29761:
```
==> Installing ncurses-6.3-22hz6q6cvo3ep2uhrs3erpp2kogxncbn
==> No binary for ncurses-6.3-22hz6q6cvo3ep2uhrs3erpp2kogxncbn found: installing from source
==> Using cached archive: /spack/var/spack/cache/_source-cache/archive/97/97fc51ac2b085d4cde31ef4d2c3122c21abc217e9090a43a30fc5ec21684e059.tar.gz
==> No patches needed for ncurses
==> ncurses: Executing phase: 'autoreconf'
==> ncurses: Executing phase: 'configure'
==> ncurses: Executing phase: 'build'
==> ncurses: Executing phase: 'install'
==> Error: AttributeError: 'str' object has no attribute 'propagate'
The 'ncurses' package cannot find an attribute while trying to build from sources. This might be due to a change in Spack's package format to support multiple build-systems for a single package. You can fix this by updating the build recipe, and you can also report the issue as a bug. More information at https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure
/spack/lib/spack/spack/build_environment.py:1075, in _setup_pkg_and_run:
1072 tb_string = traceback.format_exc()
1073
1074 # build up some context from the offending package so we can
>> 1075 # show that, too.
1076 package_context = get_package_context(tb)
1077
1078 logfile = None
```
It turns out this was caused by a bug that had been around much longer, in which the flags were passed by reference to the flag_handler, and the flag_handler was modifying the spec object, not just the flags given to the build system. The scope of this bug was limited by the forking model in Spack, which is how it went under the radar for so long.
PR includes regression test.
* remove deptype_query remnants
* deptypes -> deptype
These arguments haven't existed since 2017, but `traverse` now fails on unknown **kwargs, so they have finally popped up.
The base project name for the SEACAS project has changed from
"SEACASProj" to "SEACAS" as of @2022-10-14, so the package
needed to be updated to use the new project name when needed.
The refactor also changes several:
"-DSome_CMAKE_Option:BOOL=ON"
to
define("Some_CMAKE_Option", True)
This updates the propagation logic used in `concretize.lp` to avoid rules with `path()`
in the body and instead base propagation around `depends_on()`.
Currently, compiler flags and variants are inconsistent: compiler flags set for a
package are inherited by its dependencies, while variants are not. We should have these
be consistent by allowing for inheritance to be enabled or disabled for both variants
and compiler flags.
- [x] Make new (spec language) operators
- [x] Apply operators to variants and compiler flags
- [x] Conflicts currently result in an unsatisfiable spec
(i.e., you can't propagate two conflicting values)
What I propose is using two of the currently used sigils to symbolized that the variant
or compiler flag will be inherited:
Example syntax:
- `package ++variant`
enabled variant that will be propagated to dependencies
- `package +variant`
enabled variant that will NOT be propagated to dependencies
- `package ~~variant`
disabled variant that will be propagated to dependencies
- `package ~variant`
disabled variant that will NOT be propagated to dependencies
- `package cflags==True`
`cflags` will be propagated to dependencies
- `package cflags=True`
`cflags` will NOT be propagated to dependencies
Syntax for string-valued variants is similar to compiler flags.
Fixes an issue on the RHEL8 UBI container where this test would fail because `gr_mem`
was empty for every entry in the `grp` DB.
You have to check *both* the `pwd` database (which has primary groups) and `grp` (which
has other gorups) to do this correctly.
- [x] update `llnl.util.filesystem.group_ids()` to do this
- [x] use it in the `sbang` test
This PR introduces breadth-first traversal, and moves depth-first traversal
logic out of Spec's member functions, into `traverse.py`.
It introduces a high-level API with three main methods:
```python
spack.traverse.traverse_edges(specs, kwargs...)
spack.traverse.traverse_nodes(specs, kwags...)
spack.traverse.traverse_tree(specs, kwargs...)
```
with the usual `root`, `order`, `cover`, `direction`, `deptype`, `depth`, `key`,
`visited` kwargs for the first two.
What's new is that `order="breadth"` is added for breadth-first traversal.
The lower level API is not exported, but is certainly useful for advanced use
cases. The lower level API includes visitor classes for direction reversal and
edge pruning, which can be used to create more advanced traversal methods,
especially useful when the `deptype` is not constant but depends on the node
or depth.
---
There's a couple nice use-cases for breadth-first traversal:
- Sometimes roots have to be handled differently (e.g. follow build edges of
roots but not of deps). BFS ensures that root nodes are always discovered at
depth 0, instead of at any depth > 1 as a dep of another root.
- When printing a tree, it would be nice to reduce indent levels so it fits in the
terminal, and ensure that e.g. `zlib` is not printed at indent level 10 as a
dependency of a build dep of a build dep -- rather if it's a direct dep of my
package, I wanna see it at depth 1. This basically requires one breadth-first
traversal to construct a tree, which can then be printed with depth-first traversal.
- In environments in general, it's sometimes inconvenient to have a double
loop: first over the roots then over each root's deps, and maintain your own
`visited` set outside. With BFS, you can simply init the queue with the
environment root specs and it Just Works. [Example here](3ec7304699/lib/spack/spack/environment/environment.py (L1815-L1816))
Currently, many tests hardcode to older versions of gcc for comparisons of
concretization among compiler versions. Those versions are too old to concretize for
`aarch64`-family targets, which leads to failing tests on `aarch64`.
This PR fixes those tests by updating the compiler versions used for testing.
Currently, many tests hardcode the expected architecture result in concretization to the
`x86_64` family of architectures.
This PR generalizes the tests that can be generalized, to cover multiple architecture
families. For those that test specific relationships among `x86_64`-family targets, it
ensures that concretization uses the `x86_64`-family targets in those cases.
Currently, many tests rely on the fact that `AutotoolsPackage` imposes no dependencies
on the inheriting package. That is not true on `aarch64`-family architectures.
This PR ensures that the fact `AutotoolsPackage` on `aarch64` pulls in a dependency on
`gnuconfig` is ignored when testing for the appropriate relationships among dependencies
Additionally, 5 tests currently prompt the user for input when `gpg` is available in the
user's path. This PR fixes that issue. And 7 tests fail currently when the user has a
yubikey available. This PR fixes the incorrect gpg argument causing those issues.
The `spack info <package>` command does not show the `Virtual Packages:` output unless the `--virtuals` command option is passed. Before this changes, the information that the command is supposed to be illustrating is not shown in the example and is confusing.
* julia: don't look for the openlibm libraries when unneeded
Cause spack to *not* check for the existence of the openlibm libraries (by adding it to the pkgs list) when ~openlibm is specified.
* [@spackbot] updating style on behalf of downloadico
Co-authored-by: downloadico <downloadico@users.noreply.github.com>
- the updated OpenFOAM wmake rules now allow multiple locations for
compiler flags:
* wmake/General/common/c++Opt [central]
* wmake/linux64Gcc/c++Opt [traditional]
- match both '=' and ':=' make rule lines
Co-authored-by: Mark Olesen <Mark.Olesen@esi-group.com>
Changes to improve locating shared libraries on Windows, which in
turn enables the use of Clingo. This PR attempts to establish a
proper distinction between linking on Windows vs. Linux/Mac: on
Windows, linking is always done with .lib files (never .dll files).
This somewhat complicates the model since the Spec.lib method could
return libraries that were used for both linking and loading, but
since these are not always the same on Windows, it was decided to
treat Spec.libs as being for link-time libraries. Additional functions
are added to help dependents locate run-time libraries.
* Clingo is now the default concretizer on Windows
* Clingo is now the concretizer used for unit tests on Windows
* Fix a permissions issue that can occur while moving Git files during
fetching/staging
* Packages can now implement "win_add_library_dependent" to register
files/directories that include libraries that would need to link
to dependency dlls
* Packages can now implement "win_add_rpath" to register the locations
of dlls that dependents would want to load
* "Spec.libs" on Windows is updated to return link-time libraries
(i.e. .lib files, rather than .dll files)
* PackageBase.rpath on Windows is now updated to return the most-likely
locations where .dlls will be found (which is generally in the bin/
directory)
* Nalu-Wind: Allow for standard versions of trilinos
This will allow us to utilize custom numeric versions for trilinos in `spack-manager` while we continue to develop `nalu-wind`.
Pinging @eugeneswalker @jrood-nrel @tasmith4
* Update var/spack/repos/builtin/packages/nalu-wind/package.py
Currently there's a slow sequential step in binary relocation where all
strings of a binary are collected, with rpaths removed, and then
filtered for the old install root.
This is completely unnecessary, and also incorrect, since we replace
more than just the old install root in the prefix to prefix mapping. And
in fact the prefix to prefix mapping is parallel, and a single pass. So
even as an optimization, this filter makes no sense anymore.
Therefor we remove it
- single pass over the binary data matching all prefixes
- collect offsets and replacement strings
- do in-place updates with `fseek` / `fwrite`, since typically our
replacement touch O(few bytes) while the file is O(many megabytes)
- be nice: leave the file untouched if some string can't be
replaced
* Added py-medaka and dependencies
* fixed py-parasail build error
* medaka still doesn't have correct linked libdeflate
* fixed pyspoa deps
* added htslib.patch, confirmed builds and runs
* fixed style
* Update var/spack/repos/builtin/packages/py-auditwheel/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* made requested changes
* added targets for pyspoa dep
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
/home/xsdk/spack.x/lib/spack/env/oneapi/icx -DAdd_ -Dscalapack_EXPORTS -I/opt/intel/oneapi/mpi/2021.7.0/include -O3 -DNDEBUG -fPIC -MD -MT CMakeFiles/scalapack.dir/BLACS/SRC/dgamx2d_.c.o -MF CMakeFiles/scalapack.dir/BLACS/SRC/dgamx2d_.c.o.d -o CMakeFiles/scalapack.dir/BLACS/SRC/dgamx2d_.c.o -c /home/xsdk/spack.x/spack-stage/spack-stage-netlib-scalapack-2.2.0-uj3jepiowz5is4hmdmjrzjltetgdr3lx/spack-src/BLACS/SRC/dgamx2d_.c
/home/xsdk/spack.x/spack-stage/spack-stage-netlib-scalapack-2.2.0-uj3jepiowz5is4hmdmjrzjltetgdr3lx/spack-src/BLACS/SRC/igsum2d_.c:154:7: error: call to undeclared function 'BI_imvcopy'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
BI_imvcopy(Mpval(m), Mpval(n), A, tlda, bp->Buff);
^
Currently the vasp package always enables the use of shmem to reduce algorithm memory usage (see
https://www.vasp.at/wiki/index.php/Precompiler_options). This is great,but on some systems gives compile errors with the interoperability of C and Fortran. This PR makes that shmem flag optional, but retains the
existing default on behavior.
* Added support for building the DiHydrogen package and LBANN extensions
to DiHydrogen with ROCm libraries.
Fixed a bug on Cray systems where CMake didn't try hard enough to find
an MPI-compatible compiler wrapper. Make it look more.
Added support for the roctracer package when using ROCm libraries.
* Fixed how ROCm support is defined for pre-v0.3 versions.
- hdf5-vfd-gds:
- Add new version 1.0.2 compatible with hdf5@1.13.
- CMake is a build dependency.
- Set `HDF5_PLUGIN_PATH` in the runtime environment, this plugin
is loaded dynamically.
- SDK:
- The VFD GDS driver only has utility when CUDA is enabled.
- Require hdf5-vfd-gds@1.0.2+ (1.0.1 and earlier do not compile).
* Add patches for building clingo with MSVC
* Help python find clingo DLL
* If an executable is located in "C:\Program Files", Executable was
running into issues with the extra space. This quotes the exe
to ensure that it is treated as a single value.
Signed-off-by: Kiruya Momochi <65301509+KiruyaMomochi@users.noreply.github.com>
This commit extends the DSL that can be used in packages
to allow declaring that a package uses different build-systems
under different conditions.
It requires each spec to have a `build_system` single valued
variant. The variant can be used in many context to query, manipulate
or select the build system associated with a concrete spec.
The knowledge to build a package has been moved out of the
PackageBase hierarchy, into a new Builder hierarchy. Customization
of the default behavior for a given builder can be obtained by
coding a new derived builder in package.py.
The "run_after" and "run_before" decorators are now applied to
methods on the builder. They can also incorporate a "when="
argument to specify that a method is run only when certain
conditions apply.
For packages that do not define their own builder, forwarding logic
is added between the builder and package (methods not found in one
will be retrieved from the other); this PR is expected to be fully
backwards compatible with unmodified packages that use a single
build system.
* Updating package file for osu-micro-benchmarks for the 6.2 release
* updating sha hash for 6.2 tarball
Co-authored-by: natshineman <shineman.5@osu.edu>
* Add netcdf-c 4.9.0 and netcdf-fortran 4.6.0
With v4.9.0 netcdf-c introduces zstandard compression option which is added as a variant.
* Fix when= in dependency
* Turn on variant zstd by default
Co-authored-by: kgerheiser <kgerheiser@icloud.com>
* alquimia, pflotran, plasma, py-mpi4py, strumpack - add in new versions
* Fix hip CI failure
Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
Instead of looping over multiple regexes and the entire text file
contents, create a giant regex with all literal prefixes and do a single
pass over files to detect prefixes. Not only is a single pass faster,
it's also likely that the regex is compiled better, given that most
prefixes share a common ... prefix.
In the dfs code, flip edges so that `parent` means `from` and
`spec` means `to` in the direction of traversal. This makes it slightly
easier to write generic/composable code. For example when using visitors
where one visitor reverses direction, and another only cares about
accepting particular edges or not depending on whether the target node
is seen before, it would be good if the second visitor didn't have to
know whether the order was changed or not.
Use the same compression level as `gzip` (6) instead of what Python uses
(9).
The LLVM tarball takes 4m instead of 12m to create, and is <1% larger.
That's not worth the wait...
* udunits: Update download URL
* udunits: Deprecate older versions
Unidata now only provides the latest version of each X.Y branch. Older 2.2 versions have been deprecated accordingly but are still available in the build cache.
Co-authored-by: RemiLacroix-IDRIS <RemiLacroix-IDRIS@users.noreply.github.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* Update wgrib2 from JCSDA/NOAA-EMC fork
* var/spack/repos/builtin/packages/wgrib2/package.py: fix typo in comment, add conflict for variants netcdf3, netcdf4
* wget hdf5/netcdf4 internal dependencies for wgrib2
* Black-format var/spack/repos/builtin/packages/wgrib2/package.py
* More format changes in var/spack/repos/builtin/packages/wgrib2/package.py
#32137 added an option to update() a BinaryCacheIndex with a
cooldown: repeated attempts within this cooldown would not
actually retry. However, the cooldown was not properly
tracked for failures (which is common when the mirror
does not store any binaries and therefore has no index.json).
This commit ensures that update(..., with_cooldown=True) will
also skip the update even if a failure has occurred within the
cooldown period.
Due to reuse concretization, we may generate DAGs with two occurrences
of the same package corresponding to distinct specs. This happens when
build dependencies are reused, since their dependencies are ignored in
concretization.
This caused a regression, for example: `spec['openssl']` would take the
'openssl' of the build dep `cmake`, instead of the direct `openssl`
dependency, simply because the edge to `cmake` was traversed first and
we do depth first traversal.
One solution that was discussed is to limit `spec[name]` to just direct
deps, or direct deps + transitive link deps, but this is too breaking.
Instead, this PR simply prioritizes transitive link and direct
build/run/test deps, and then falls back to a full DAG traversal. So,
it's just about order of iteration.
* Update py-sphinxcontrib-mermaid
* Add py-myst-parser
* Fix py-mdit-py-plugins and py-myst-parser dependencies
* Add py-exhale package
* Update var/spack/repos/builtin/packages/py-mdit-py-plugins/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-myst-parser/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-myst-parser/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-myst-parser/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update py-exhale and py-myst-parser dependencies
* Add @svenevs as py-exhale maintainer
* Update var/spack/repos/builtin/packages/py-mdit-py-plugins/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Scan the text files for relocatable prefixes *before* creating a tarball,
to reduce the amount of work to be done during install from binary
cache.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
* py-drep: new package
* fixed file extension
* added darwin conflict
* py-checkm-genome and py-pysam: bumped version and updated deps (#10)
added checkm and pysam deps
* added dep documentation and fixed style
* changed checkm and pysam back to dev version for upstreaming
* added url and perl run dep
* fixed style
Instead of showing
```
==> Error: Timed out waiting for a write lock.
```
show
```
==> Error: Timed out waiting for a write lock after 1.200ms and 4 attempts on file: /some/file
```
s.t. we actually get to see where acquiring a lock failed even when not
running in debug mode.
And use pretty time units everywhere, so we don't get 1.45e-9 seconds
but 1.450ns etc.
* backtraces without --debug
Currently `--debug` is too verbose and not-`--debug` gives to little
context about where exceptions are coming from.
So, instead, it'd be nice to have `spack --backtrace` and
`SPACK_BACKTRACE=1` as methods to get something inbetween: no verbose
debug messages, but always a full backtrace.
This is useful for CI, where we don't want to drown in debug messages
when installing deps, but we do want to get details where something goes
wrong if it goes wrong.
* completion
* acts: new versions
In the 20.x release line, these are the changes, https://github.com/acts-project/acts/compare/v20.0.0...v20.2.0
- `option(ACTS_SETUP_ACTSVG "Build ActSVG display plugin" OFF)` introduced in v20.1.0
- `option(ACTS_USE_SYSTEM_ACTSVG "Use the ActSVG system library" OFF)` introduced in v20.1.0
- `option(ACTS_BUILD_PLUGIN_ACTSVG "Build SVG display plugin" OFF)` introduced in v20.1.0
- `option(ACTS_USE_EXAMPLES_TBB "Use Threading Building Blocks library in examples" ON)` introduced in v20.1.0
- `option(ACTS_EXATRKX_ENABLE_ONNX "Build the Onnx backend for the exatrkx plugin" OFF)` introduced in v20.2.0
- `option(ACTS_EXATRKX_ENABLE_TORCH "Build the torchscript backend for the exatrkx plugin" ON)` introduced in v20.2.0
In the 19.x release line, these are the changes: https://github.com/acts-project/acts/compare/v19.7.0...v19.9.0
- `option(ACTS_USE_EXAMPLES_TBB "Use Threading Building Blocks library in examples" ON)` introduced in v19.8.0
The new build options have not been implemented in this commit but will be implemented next.
* acts: new variant svg
* actsvg: new package
* actsvg: style fixes
* acts: new versions 20.3.0 and 19.10.0
* astsvg: depends_on boost googletest
* actsvg: new version 0.4.26 (and style fix)
Includes fix to build issue when +examples, https://github.com/acts-project/actsvg/pull/23
* acts: new variant tbb when +examples @19.8:19 @20.1:
* acts: set ACTS_USE_EXAMPLES_TBB
* acts: no need for ACTS_SETUP_ACTSVG
* acts: move tbb variant to examples block
* acts: ACTS_USE_SYSTEM_ACTSDD4HEP removed in 20.3
* acts: use new ACTS_USE_SYSTEM_LIBS
* acts-dd4hep: new version 1.0.1, maintainer handle fixed
* acts: simplify variant tbb condition
* py-checkm-genome and py-pysam: bumped version and updated deps
* updated setuptools dep type
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
When we lose a running pod (possibly loss of spot instance) or encounter
some other infrastructure-related failure of this job, we need to retry
it. This retries the job the maximum number of times in those cases.
Currently `relocate_text` and `relocate_text_bin` are unsafe in the
sense that they run in parallel, and lead to races when modifying
different items pointing to the same inode.
This leads to the issue observed in #33453.
This PR:
1. Renames those functions to `unsafe_*` so people are aware
2. Adds logic to deal with hardlinks in current binary packages
3. Adds logic to deal with hardlinks when creating new binary tarballs,
so the install side doesn't have to de-dupe hardlinks.
4. Adds a test for 3
The assumption is that all our relocation logic preserves inodes. That
is, we should never copy a file, modify it, and then move it back. I
quickly verified, and its seems like this is true for (binary) text
relocation, as well as rpath patching in patchelf (even when the file
grows) and mach-o fixes.
* axom@0.7.0: require cmake@3.21:
* Update var/spack/repos/builtin/packages/axom/package.py
Co-authored-by: Chris White <white238@llnl.gov>
Co-authored-by: Chris White <white238@llnl.gov>
`reuse` and `when_possible` concretization broke the invariant that
`spec[pkg_name]` has unique keys. This invariant is relied on in tons of
places, such as when setting up the build environment.
When using `when_possible` concretization, one may end up with two or
more `perl`s or `python`s among the transitive deps of a spec, because
concretization does not consider build-only deps of reusable specs.
Until the code base is fixed not to rely on this broken property of
`__getitem__`, we should disable reuse in CI.
* fixed version numbers to python 2 and old biopython
* changed shortbred pacakge to pypi, removed python 2 version
* added package description
* re-added shortbred package with depreciated flag
* fixed style and removed unnecessary python dep (it can't build with python 2 anyway)
* removed whitespace and readded the python2.7.9+ dep
* fixed style
* gitlab: Do not use root_spec['pkg_name'] anymore
For a long time it was fine to index a concrete root spec with the name
of a dependency in order to access the concrete dependency spec. Since
pipelines started using `--use-buildcache dependencies:only,package:never`
though, it has exposed a scheduling issue in how pipelines are
generated. If a concrete root spec depends on two different hashes of
`openssl` for example, indexing that root with just the package name
is ambiguous, so we should no longer depend on that approach when
scheduling jobs.
* env: make sure exactly one spec in env matches hash
When installing some/all specs from a buildcache, build edges are pruned
from those specs. This can result in a much smaller effective DAG. Until
now, `spack env depfile` would always generate a full DAG.
Ths PR adds the `spack env depfile --use-buildcache` flag that was
introduced for `spack install` before. This way, not only can we drop
build edges, but also we can automatically set the right buildcache
related flags on the specific specs that are gonna get installed.
This way we get parallel installs of binary deps without redundancy,
which is useful for Gitlab CI.
When downloading from binary cache not only replace RPATHs to dependencies, but
also text references to dependencies.
Example:
`autoconf@2.69` contains a text reference to the executable of its dependency
`perl`:
```
$ grep perl-5 /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/autoconf-2.69-q3lo/bin/autoreconf
eval 'case $# in 0) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0";; *) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0" "$@";; esac'
```
These references need to be replace or any package using `autoreconf` will fail
as it cannot find the installed `perl`.
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
"spack install" will not update the binary index if given a concrete
spec, which causes it to fall back to direct fetches when a simple
index update would have helped. For S3 buckets in particular, this
significantly and needlessly slows down the install process.
This commit alters the logic so that the binary index is updated
whenever a by-hash lookup fails. The lookup is attempted again with
the updated index before falling back to direct fetches. To avoid
updating too frequently (potentially once for each spec being
installed), BinaryCacheIndex.update now includes a "cooldown"
option, and when this option is enabled it will not update more
than once in a cooldown window (set in config.yaml).
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Without this patch one hits this error trying to compiler papi with Intel OneAPI:
icx: error: Note that use of '-g' without any optimization-level option will turn off most compiler optimizations similar to use of '-O0' [-Werror,-Wdebug-disables-optimization]
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
* Fixed two bugs in the HIP package recipe. The first is that the
HIP_PATH was being set to the actual spec, and not the spec prefix.
The second bug is that HIP is expected to be in /opt/rocm-x.y.z/hip
but it's libraries can exist at both /opt/rocm-x.y.z/hip/lib and
/opt/rocm-x.y.z/lib. This means that the external detection logic may
find it in either and it turns out that some modules only expose one
of those two locations. Logic is added to ensure that the internal
HIP_PATH and associated ROCM_PATH are correctly set in both cases.
* Added support for Aluminum to use the libfabric plugin with either
RCCL or NCCL.
* Add libpressio and dependencies; some of these packages are
maintained as forks of the original repositories and in those
cases the docstring mentions this.
* Add optional dependency in adios2 on libpressio
* cub package: set CUB_DIR environment variable for dependent
installations
* Clear R_HOME/R_ENVIRON before Spack installation (avoid sources
outside of Spack from affecting the installation in Spack)
* Rename dlib to dorian3d-dlib and update dependents; add new dlib
implementation. Pending an official policy on how to handle
packages with short names, reviewer unilaterally decided that
the rename was acceptable given that the new Spack dlib package
is referenced more widely (by orders of magnitude) than the
original
Co-authored-by: Samuel Li <shaomeng@users.noreply.github.com>
* Classic Intel compilers do not support gcc-toolchain
This fix removes `--gcc-toolchain=` from the ~.fcg` files for the classic Intel
compilers. AFAIK this option is only supported for Clang based compilers.
This lead to an issue when installing cmake. Reproducer:
```
spack install cmake@3.24.2%intel@2021.7.0~doc+ncurses+ownlibs~qt
build_type=Release arch=linux-amzn2-skylake_avx512
```
Tagging maintainer @rscohn2
* Add `-gcc-name` for icc
.. and `-gxx-name` for icpc.
AFAIK this is used for modern C++ support, so we can ignore `ifort`.
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
When installing an individual spec `spack --only=package --cache-only /xyz`
from a buildcache, Spack currently issues tons of warnings about
missing build deps (and their deps) in the database.
This PR disables these warnings, since it's fine to have a spec without
its build deps in the db (they are just "missing").
Currently `traverse_dependencies` fixes deptypes to traverse once and
for all in the recursion, but this is incorrect, since deptypes depend
on the node (e.g. if it's a dependency and cache-only, don't follow
build type edges, even if the parent is build from sources and needs
build deps.)
Support spackbot rebuilding all specs from source when asked (with "rebuild everything")
- Allow overriding --prune-dag cli opt with env var
- Use job variable to optionally prevent rebuild jobs early exit behavior
- ci rebuild: Use new install argument to insist deps are always installed from binary, but
package is only installed from source.
- gitlab: fix bug w/ untouched pruning
- ci rebuild: install from hash rather than json file
- When doing a "rebuild everything" pipeline, make sure that each install job only consumes
binary dependencies from the mirror being populated by the current pipeline. This avoids
using, e.g. binaries from develop, when rebuilding everything on a PR.
- When running a pipeline to rebuild everything, do not die because we generated a hash on
the broken specs list. Instead only warn in that case.
- bugfix: Replace broken no-args tty.die() with sys.exit(1)
Print a message of the form
```
Fetch mm:ss. Build: mm:ss. Total: mm:ss
```
when installing from buildcache.
Previously this only happened for source builds.
Currently "spack ci generate" chooses the first matching entry in
gitlab-ci:mappings to fill attributes for a generated build-job,
requiring that the entire configuration matrix is listed out
explicitly. This unfortunately causes significant problems in
environments with large configuration spaces, for example the
environment in #31598 (spack.yaml) supports 5 operating systems,
3 architectures and 130 packages with explicit size requirements,
resulting in 1300 lines of configuration YAML.
This patch adds a configuraiton option to the gitlab-ci schema called
"match_behavior"; when it is set to "merge", all matching entries
are applied in order to the final build-job, allowing a few entries
to cover an entire matrix of configurations.
The default for "match_behavior" is "first", which behaves as before
this commit (only the runner attributes of the first match are used).
In addition, match entries may now include a "remove-attributes"
configuration, which allows matches to remove tags that have been
aggregated by prior matches. This only makes sense to use with
"match_behavior:merge". You can combine "runner-attributes" with
"remove-attributes" to effectively override prior tags.
* petsc,py-petsc4py,slepc,py-slepc4py: add version 3.18.0
* workaround for dealii build failure [with petsc version check]
* pism: add compatibility fix to for petsc@3.18
* add in hipsolver dependency
* Add checksum for py-gitpython 3.1.27
* Update package.py
* Update var/spack/repos/builtin/packages/py-gitpython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
When a pipeline generation job is automatically failed because it
generated jobs for specs known to be broken on develop, print better
information about the broken specs that were encountered. Include
at a minimum the hash and the url of the job whose failure caused it
to be put on the broken specs list in the first place.
* new package + deps
* Update var/spack/repos/builtin/packages/py-about-time/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-alive-progress/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* removed unnecessary python version dep
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* add new lua releases
* split install phase and move it into a build phase, remove hardcoded standard flag
* revert back to the original hardcoded std flag, guard patch against versions above 5.4
* py-pyopenssl: add version 22.1.0
* Update var/spack/repos/builtin/packages/py-pyopenssl/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add checksum for py-regex 2022.8.17
* Update var/spack/repos/builtin/packages/py-regex/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add checksum for py-async-lru 1.0.3
* [@spackbot] updating style on behalf of iarspider
* Update var/spack/repos/builtin/packages/py-async-lru/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: iarspider <iarspider@users.noreply.github.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* env depfile: allow deps only install
- Refactor `spack env depfile` to use a Jinja template, making it a bit
easier to follow as a human being.
- Add a layer of indirection in the generated Makefile through an
`<prefix>/.install-deps/<hash>` target, which allows one to specify
different options when installing dependencies. For example, only
verbose/debug mode on when installing some particular spec:
```
$ spack -e my_env env depfile -o Makefile --make-target-prefix example
$ make example/.install-deps/<hash> -j16
$ make example/.install/<hash> SPACK="spack -d" SPACK_INSTALL_FLAGS=--verbose -j16
```
This could be used to speed up `spack ci rebuild`:
- Parallel install of dependencies from buildcache
- Better readability of logs, e.g. reducing verbosity when installing
dependencies, and splitting logs into deps.log and current_spec.log
* Silence please!
* Add bioc attribute to r-do-db
* add version 1.38.1 to bioconductor package r-annotationforge
* add version 1.30.4 to bioconductor package r-biocparallel
* add version 2.64.1 to bioconductor package r-biostrings
* add version 4.4.4 to bioconductor package r-clusterprofiler
* add version 2.12.1 to bioconductor package r-complexheatmap
* add version 1.18.1 to bioconductor package r-delayedmatrixstats
* add version 3.22.1 to bioconductor package r-dose
* add version 3.38.4 to bioconductor package r-edger
* add version 1.16.2 to bioconductor package r-enrichplot
* add version 2.20.2 to bioconductor package r-ensembldb
* add version 1.32.4 to bioconductor package r-genomeinfodb
* add version 1.32.1 to bioconductor package r-genomicalignments
* add version 1.48.4 to bioconductor package r-genomicfeatures
* add version 1.44.1 to bioconductor package r-ggbio
* add version 3.4.4 to bioconductor package r-ggtree
* add version 1.24.2 to bioconductor package r-hdf5array
* add version 2.30.1 to bioconductor package r-iranges
* add version 1.36.3 to bioconductor package r-keggrest
* add version 3.52.4 to bioconductor package r-limma
* add version 1.8.1 to bioconductor package r-matrixgenerics
* update r-org-hs-eg-db
* add version 1.38.1 to bioconductor package r-organismdbi
* add version 1.36.1 to bioconductor package r-pathview
* add version 1.56.1 to bioconductor package r-rtracklayer
* add version 1.4.1 to bioconductor package r-scaledmatrix
* add version 1.24.1 to bioconductor package r-scran
* add version 1.6.3 to bioconductor package r-scuttle
* add version 1.18.1 to bioconductor package r-singlecellexperiment
* add version 1.20.2 to bioconductor package r-treeio
* Revert "Add bioc attribute to r-do-db"
This reverts commit 36be5c6072.
* Fix quotes on versions
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
For older versions of intel-oneapi-compilers, running the compiler in
preprocessor / compilation mode would trigger warnings that
`-Wl,-rpath,...` flags were unused.
This in turn caused certain configure scripts to fail as they did not
expect output from the compiler (it's treated as an error). Notably
cmake's bootstrap phase failed to detect c++ features of the compiler.
As a workaround, add this flag to silence the warning, since I don't
think we can scope the flags to compile+link mode.
* bump version for libvterm, required by neovim
* bump version for neovim and add related dep constraints
see release note:
d367ed9b23
in particular:
'deps: Bump required libvterm to v0.3'
https://github.com/neovim/neovim/pull/20222
* spack.compiler.Compiler: introduce prefix property
We currently don't really have something that gives the GCC install
path, which is used by many LLVM-based compilers (llvm, llvm-amdgpu,
nvhpc, ...) to fix the GCC toolchain once and for all.
This `prefix` property is dynamic in the sense that it queries the
compiler itself. This is necessary because it's not easy to deduce the
install path from the `cc` property (might be a symlink, might be a
filename like `gcc` which works by having the compiler load a module
that sets the PATH variable, might be a generic compiler wrapper based
on environment variables like on cray...).
With this property introduced, we can clean up some recipes that have
the logic repeated for GCC.
* intel-oneapi-compilers: set --gcc-sysroot to %gcc prefix
Caches used by repositories don't reference the global spack.repo.path instance
anymore, but get the repository they refer to during initialization.
Spec.virtual now use the index, and computation done to compute the index
use Repository.is_virtual_safe.
Code to construct mock packages and mock repository has been factored into
a unique MockRepositoryBuilder that is used throughout the codebase.
Add debug print for pushing and popping config scopes.
Changed spack.repo.use_repositories so that it can override or not previous repos
spack.repo.use_repositories updates spack.config.config according to the modifications done
Removed a peculiar behavior from spack.config.Configuration where push would always
bubble-up a scope named command_line if it existed
Resolves#31782
With this change, if a spec is concrete after parsing (e.g. spec.yaml
or /hash-based), then it is not disambiguated (a process which requires
(a) that the spec be installed and (b) that it be part of the
currently-active environment).
This commit allows you to:
* Diff specs from an environment regardless of whether they have
been installed (more useful for projection/matrix-based envs)
* Diff specs read from .yaml files which may or may not be entirely
different installations of Spack
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* hpctoolkit: add version 2022.10.01
1. add version 2022.10.01
2. remove version for master branch, develop is now the main branch
3. add CPATH and LD_LIBRARY_PATH to module run environment,
this is for apps that want to use the start/stop interface
4. cleanup style in variants, depends and conflicts
5. remove all-static variant, nothing uses it
6. deprecate more old versions
* [@spackbot] updating style on behalf of mwkrentel
* Add when(+level_zero) to the gtpin variant.
* Test commit to see if this passes E4S.
* Another test commit to see if E4S succeeds.
* Add temporary hack to ignore +mpi for version 2022.10.01 and issue a
warning instead.
Co-authored-by: mwkrentel <mwkrentel@users.noreply.github.com>
* Add checksum for py-msgpack 1.0.4
* Update var/spack/repos/builtin/packages/py-msgpack/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add checksum for py-virtualenv 20.16.4
* Add checksum for py-werkzeug 2.2.2
* Restore py-virtualenv/package.py
* Update var/spack/repos/builtin/packages/py-werkzeug/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Docs: Getting Started Dependencies
Finally document what one needs to install to use Spack on
Linux and Mac :-)
With <3 for minimal container users and my colleagues with
their fancy Macs.
* Debian Update Packages: GCC, Python
- build-essential: includes gcc, g++ (thx Cory)
- Python: add python3-venv, python3-distutils (thx Pradyun)
* Add RHEL8 Dependencies
* Add checksum for py-skl2onnx 1.12
* Update var/spack/repos/builtin/packages/py-skl2onnx/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add checksum for py-tables 3.7.0
* Update package.py
* Update var/spack/repos/builtin/packages/py-tables/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
ROOT has a webgui which is available with the `+root7` variant. This is a fairly large part of a ROOT install (275MB out of 732MB on my system) which is not necessarily useful in all use cases (e.g. inside containers on network-restricted HPC/HTC compute nodes). This new variant adds the option to retain the ROOT7 functionality but not necessarily include the `webgui` aspects.
`__unused__` defined in `general.h` conflict with the one defined by libc headers,
so change it to `__attribute__unused__` according to s.zharkoff:
https://bugs.gentoo.org/828550#c11
cmd:
`grep -rl "__unused__" . | xargs -n1 sed -i -e 's/\b__unused__\b/__attribute__unused__/g' -e 's/(unused)/(__unused__)/g'`
Basic stack of ML packages we would like to test and generate binaries for in CI.
Spack now has a large CI framework in GitLab for PR testing and public binary generation.
We should take advantage of this to test and distribute optimized binaries for popular ML
frameworks.
This is a pretty extensive initial set, including CPU, ROCm, and CUDA versions of a core
`x96_64_v4` stack.
### Core ML frameworks
These are all popular core ML frameworks already available in Spack.
- [x] PyTorch
- [x] TensorFlow
- [x] Scikit-learn
- [x] MXNet
- [x] CNTK
- [x] Caffe
- [x] Chainer
- [x] XGBoost
- [x] Theano
### ML extensions
These are domain libraries and wrappers that build on top of core ML libraries
- [x] Keras
- [x] TensorBoard
- [x] torchvision
- [x] torchtext
- [x] torchaudio
- [x] TorchGeo
- [x] PyTorch Lightning
- [x] torchmetrics
- [x] GPyTorch
- [x] Horovod
### ML-adjacent libraries
These are libraries that aren't specific to ML but are still core libraries used in ML pipelines
- [x] numpy
- [x] scipy
- [x] pandas
- [x] ONNX
- [x] bazel
Co-authored-by: Jonathon Anderson <17242663+blue42u@users.noreply.github.com>
* Add checksum for py-scikit-build 0.15.0 and use sources from pypi
* Update var/spack/repos/builtin/packages/py-scikit-build/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add checksum for py-terminado 0.15.0
* Update var/spack/repos/builtin/packages/py-terminado/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* added metaphlan v4, cleaned up phylophlan
* added iqtree2
* fixed phylophlan, builds now
* changed config.yaml to default
* fixed style
* py-jsonschema: add 4.16.0 and new package py-hatch-fancy-pypi-readme (#32929)
* acfl: add v22.1 (#32915)
Co-authored-by: Annop Wongwathanarat <annop.wongwathanarat@arm.com>
* Fixup errors introduced by Clingo Pr: (#32905)
* re2c depends on cmake on Windows
* Winbison properly added to bootstrap package search list
* Set CMAKE_HIP_ARCHITECTURES with the value of amdgpu_target (#32901)
* libtiff: default to +zlib+jpeg (#32945)
* octave: add version 7.2.0 (#32943)
* simgrid new releases (#32920)
* [rocksdb] Added rtti variant (#32918)
* rvs binary path updated for 5.2 rocm release (#32892)
* Add checksum for py-pytest-runner 6.0.0 (#32957)
* py-einops: add v0.5.0 (#32959)
* Replace repo with the NVIDIA one (#32951)
* Add checksum for py-tomli 2.0.1 (#32949)
* QMCPACK: add @3.15.0 (#32931)
* Tidied up configure arguments to use special spack autotools features. (#32930)
* casper: old domain fell off, adding github repo (#32928)
* unifyfs: pin mercury version; add boost variant (#32911)
Mercury has a new version (v2.2) releasing soon that UnifyFS does not build with and hasn't been tested with. This pins UnifyFS to the last version of Mercury used/tested.
Add a variant to avoid building/using boost
Append -std=gnu99 to cflags if building with gcc@4. Needed for mochi-margo to compile
* trilinos: constrain superlu-dist version (#32889)
* trilinos: constrain superlu-dist version for 13.x
* syntax
* FEniCSx: Updates for 0.5.1 (#32665)
* Updates for DOLFINx 0.5.1 and associated packages
* xtensor needed on anything less than main
* Switch back to Python 3.7 minimum.
* Might be good to point out in our README how to fix Python version?
* Fix basix, xtensor dep
* Add numba feature
* Fix checksum
* Make slepc optional
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* simgrid: add variant and remove flag (#32797)
* simgrid: remove std c++11 flag
* simgrid: add msg variant
* Axom: bring in changes from axom repo (#32643)
* bring in changes from axom repo
Co-authored-by: white238 <white238@users.noreply.github.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* Add checksum for py-pyparsing 3.0.9 (#32952)
* rdma-core: fix syntax for external discoverability (#32962)
* Add checksum for py-flatbuffers 2.0.7 (#32955)
* amrex: add v22.10 (#32966)
* Remove CMakePackage.define alias from most packages (#32950)
* Bug fix for `ca-certificates-mozilla/package.py` to enable `spack install --source` (#32953)
* made suggested changes to iqtree2, py-dendropy, py-metaphlan, and py-pkgconfig. Poetry install still broken
* reverted py-pkgconfig deps to poetry-core
* made iqtree2 less dedundant, changes to py-dendropy and py-pkgconfig deps
Co-authored-by: Manuela Kuhn <36827019+manuelakuhn@users.noreply.github.com>
Co-authored-by: Annop Wongwathanarat <annop.wongwathanarat@gmail.com>
Co-authored-by: Annop Wongwathanarat <annop.wongwathanarat@arm.com>
Co-authored-by: John W. Parent <45471568+johnwparent@users.noreply.github.com>
Co-authored-by: Auriane R <48684432+aurianer@users.noreply.github.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Kai Torben Ohlhus <k.ohlhus@gmail.com>
Co-authored-by: Vinícius <viniciusvgp@gmail.com>
Co-authored-by: Matthieu Dorier <mdorier@anl.gov>
Co-authored-by: renjithravindrankannath <94420380+renjithravindrankannath@users.noreply.github.com>
Co-authored-by: iarspider <iarspider@gmail.com>
Co-authored-by: Paul R. C. Kent <kentpr@ornl.gov>
Co-authored-by: Brian Van Essen <vanessen1@llnl.gov>
Co-authored-by: snehring <7978778+snehring@users.noreply.github.com>
Co-authored-by: Cameron Stanavige <stanavige1@llnl.gov>
Co-authored-by: Cody Balos <balos1@llnl.gov>
Co-authored-by: Jack S. Hale <mail@jackhale.co.uk>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Lucas Nesi <lucas31nesi@hotmail.com>
Co-authored-by: Chris White <white238@llnl.gov>
Co-authored-by: white238 <white238@users.noreply.github.com>
Co-authored-by: Martin Pokorny <mpokorny@caltech.edu>
Co-authored-by: Weiqun Zhang <WeiqunZhang@lbl.gov>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Co-authored-by: Dom Heinzeller <dom.heinzeller@icloud.com>
* filter_file: introduce argument 'start_at'
* autotools: extend patching of the libtool script
* autotools: refactor _patch_usr_bin_file
* autotools: improve readability of the filtering
* autotools: keep the modification time of the configure scripts
* autotools: do not try to patch directories
* autotools: explain libtool patching for posterity
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
* Add checksum for py-onnxmltools 1.11.0
* Add checksum for py-onnxmltools 1.11.0
* Fix patch name
* Update var/spack/repos/builtin/packages/py-onnx-runtime/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add checksum for py-pkginfo 1.8.3
* Update var/spack/repos/builtin/packages/py-pkginfo/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* trilinos and xyce: fix fortran library handling
xyce:
- add pymi_static_blas variant and logic
handles blas and hdf5 conflicts for Xyce-PyMi
- make +isorropia and +zoltan conditional on mpi
* xyce: clean up CMake options
* xyce: change pymi_static_blas to pymi_static_tpls
* xyce: made pymi_static_tpls only when +pymi
Remove `module-info mode load` condition that prevents auto-unloading when autoloading is enabled. It looks like this condition was added to work around an issue in environment-modules that is no longer necessary.
Add quotes to make is-loaded happy
* Add checksum for py-urllib3 1.26.12
* Update var/spack/repos/builtin/packages/py-urllib3/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fixes from review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add checksum for py-soupsieve 2.3.2.post1
* Update package.py
* Update var/spack/repos/builtin/packages/py-soupsieve/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update to PyFR v1.15.0
* allow unsupported compilers for cuda
* update requirement for py-gimmik version
* update requirement for cuda version
* update versions, style fix
* lib directory changed, style fixes
* PYFR_METIS_LIBRARY_PATH set
There is a new OpenBLAS release out that can be compiled w/o
a Fortran compiler.
macOS XCode developers, rejoice. Maybe at some point Spack
becomes a package manager that can be used without using
another package manager (to get gfortran) 🎉
phist: add conflict on reference netlib-lapack due to API change in lapack.h
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
* Added a package for the aws-ofi-nccl plug-in from to enable NCCL to
use libfabric communication library as a network provider.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
When concurrent misc_cache provider index rebuilds happen, try to
rebuild it only once, so we don't exceed misc_cache lock timeout.
For example, when using `spack env depfile`, with no previous
misc_cache, running `make -f depfile -j8` could run at most 8 concurrent
`spack install` locking on misc_cache to rebuild the provider index. If
one rebuild takes 30s, before this fix, the "worst" lock could wait up
to 30s * 7, easily exceeding misc_cache lock timeout. Now, the "worst"
lock would take 30s * 1 + ~1s * 6.
* Add checksums for Rivet 3.1.7 and YODA 1.9.7
* Add checksum for fastjet 3.4.0
* Add v3.1.7b...
... in which version requirement for autoconf was lowered to 2.68
Currently, module changes from `setup_dependent_package` are applied only to the module of the package class, but not to any parent classes' modules between the package class module and `spack.package_base`.
In this PR, we create a custom class to accumulate module changes, and apply those changes to each class that requires it. This design allows us to code for a single module, while applying the changes to multiple modules as needed under the hood, without requiring the user to reason about package inheritance.
* find/list: display package counts last
We have over 6,600 packages now, and `spack list` still displays the number of packages
before it lists them all. This is useless for large sets of results (e.g., with no args)
as the number has scrolled way off the screen before you can see it. The same is true
for `spack find` with large installations.
This PR changes `spack find` and `spack list` so that they display the package count
last.
* add some quick testing
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
* Updates for DOLFINx 0.5.1 and associated packages
* xtensor needed on anything less than main
* Switch back to Python 3.7 minimum.
* Might be good to point out in our README how to fix Python version?
* Fix basix, xtensor dep
* Add numba feature
* Fix checksum
* Make slepc optional
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Mercury has a new version (v2.2) releasing soon that UnifyFS does not build with and hasn't been tested with. This pins UnifyFS to the last version of Mercury used/tested.
Add a variant to avoid building/using boost
Append -std=gnu99 to cflags if building with gcc@4. Needed for mochi-margo to compile
Allow environment variables and spack-specific path substitution variables (e.g. `$spack`) to be
used in the paths associated with develop specs, while maintaining the ability to keep those
paths relative to the environment rather than the working directory.
* Added new versions
* New slate version
* Adding GPU support for lapackpp package
* Modified dependency on lapackpp
* Added rocblas and rocsolver to deps
* Testing with custom lapackpp repo
* Added chaining depends_on for +rocm
* Removing testing repo
* py-ipython: add 8.5.0 and py-win-unicode-console: new package
* Fix style
* Fix dependency version
* Deprecate version 2.3.1 and 3.1.0
* Always skip IPython.kernel
* Move skip_module definition to top
Install: Add use-buildcache option to install
* Allow differentiating between top level packages and dependencies when
determining whether to install from the cache or not.
* Add unit test for --use-buildcache
* Use metavar to display use-buildcache options.
* Update spack-completion
* Added a package for the aws-ofi-rccl plug-in from the ROCm software
stack. It allows RCCL to use the libfabric communication library.
Added support for using libfabric in Aluminum.
* Updated the run environment so that the plugin would get loaded.
* Added support for setting up the the LD_LIBRARY_PATH for dependent packages.
* Added package for RCCL tests to assess the impact of OFI libfabric RCCL plug-in.
* Add tag for xgboost 1.6.2
* Update py-xgboost as well
* Update package.py
* Update var/spack/repos/builtin/packages/py-xgboost/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* intel-oneapi-compilers-classic: refactor setup_run_environment
* intel-oneapi-compilers-classic: extend setup_run_environment with the one from intel-oneapi-compilers
* py-interface-meta: add 1.3.0 (incl new dependency packages)
* Update var/spack/repos/builtin/packages/py-poetry-dynamic-versioning/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
superlu:oneapi-deal with non ISO C99 complianc in the package.
The Intel OneAPI compilers are based on LLVM 14. A recent enhancement to LLVM -
https://reviews.llvm.org/D122983
results in superlu-dist not compiling because of some non ISO C99 compliant stuff.
A workaround is to use an llvm compile line option noted in the above URL.
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Make it possible to install the Clingo package on Windows; this
also provides a means to use Clingo with Spack on Windows.
This includes
* A new "winbison" package: Windows has a port of bison and flex where
the two packages are grouped together. Clingo dependencies have been
updated to use winbison on Windows and bison elsewhere (this avoids
complicating the existin bison/flex packages until we can add support
for implied virtuals).
* The CMake build system was incorrectly converting CMAKE_INSTALL_PREFIX
to POSIX format.
* The re2c package has been modified to use CMake on Windows; for now
this is done by overloading the configure/build/install methods to
perform CMake-appropriate operations; the package should be refactored
once support for multiple build systems in one Package is available.
This commit fixes#27027.
The root cause of the issue is that the `SPACK_OLD_PROMPT` variable
was evaluated in string interpolation regardless of whether the
guard condition above evaluates to true or false. This commit uses
the `eval` keyword to defer evaluation until the command is executed.
Co-authored-by: Alexander Hornburg <alexande@xilinx.com>
The file rbg.txt is needed for many PGPLOT application, such as the MESA
stellar evolution code. This change installs the file to the PGPLOT_DIR,
where the library expects it.
PGPLOT_DIR was previously set to the prefix itself, which is an odd
place to install rgb.txt. This commit changes it to lib/pgplot5,
following the convention used by Debian.
Co-authored-by: Philipp Edelmann <edelmann@fs.tum.de>
* py-datalad-metalad: add 0.4.5
* Update var/spack/repos/builtin/packages/py-datalad-metalad/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
In addition to the new release, made intel-oneapi-compilers-classic version number match the compiler version number, instead of the version of the package that contains it.
Co-authored-by: Robert Cohn <robert.s.cohn@intel.com>
* root: make X11 really optional on macOS
* Update var/spack/repos/builtin/packages/root/package.py
* remove when clauses in provides
Co-authored-by: Hadrien G. <knights_of_ni@gmx.com>
This modifications breaks `develop` since it doesn't
pass audits with Python 2.7 It is to be investigated
why audits pass in CI for the PR and the issue is
revealed only when the package is pushed to develop.
PR #32615 deprecated Python versions up to 3.6.X. Since
the "build-systems" pipeline requires Python 3.6.15 to
build "tut", it will fail on the first rebuild that
involves Python.
The "tut" package is meant to perform an end-to-end
test of the "Waf" build-system, which is scarcely
used. The fix therefore is just to remove it from
the pipeline.
Spack currently depends on parsing filenames of downloaded files to
determine what type of archive they are and how to decompress them.
This commit adds a preliminary check based on magic numbers to
determine archive type (but falls back on name parsing if the
extension type cannot be determined).
As part of this work, this commit also enables decompression of
.tar.xz-compressed archives on Windows.
* Update py-poetry to 1.2.1
* Update py-xattr
* Apply style from review
* Apply suggestions from code review (part 1)
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Changes from review - 2
* Fix typo
* Fix style
* Add missing py-dulwich version
* Update var/spack/repos/builtin/packages/py-poetry-core/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-poetry/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-poetry/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Apply changes from review
* Add py-backports-cached-property and fix style
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-eccodes: ensure the minimal recommended shared version of libeccodes
* py-eccodes: set less general environment variables to enable location of libeccodes
* py-eccodes: add version 1.5.0
* py-eccodes: make flake8 happy
amazon linux 2 ships a glibc that is too old to work with cuda toolkit
for aarch64.
For example:
`libcurand.so.10.2.10.50` requires the symbol `logf@@GLIBC_2.27`, but
glibc is at 2.26.
So, these specs are removed.
* New packages: py-conan, py-node-semver, py-patch-ng
* Update var/spack/repos/builtin/packages/py-conan/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-conan/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-babel: add 2.10.3
* Update var/spack/repos/builtin/packages/py-babel/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New package: py-jaraco-classes
* Update var/spack/repos/builtin/packages/py-jaraco-classes/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [py-tinydb] added package py-tinydb
* [py-tinydb] corrected version in dependency for py-tinydb
* [py-tinydb] update python dependency
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [py-tinydb] update dependency
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Include exception info related to url retrieval in debug messages
which otherwise would be swallowed. This is intended to be useful
for detecting if CA configuration interferes with downloads from
HTTPS links.
* add the 2 variants OPENCL and HIP and their dependencies correctly
for OPENCL - rocm-opencl, miopengemm and miopen-opencl
for HIP - miopen-hip
Earlier this was adding both the dependencies -miopen-hip and miopen-opencl
for both the backends which did not seem correct.
Also corrected the miopen-hip or miopen-opencl config.h in patch() depending on the
backend
Also added libjpeg-turbo as it is required for building ROCAl .
the AMDRpp is still required for ROCAL inclusion but it currently does not build
AMDRpp will be added as a new spack recipe and the mivisionx will refer to that as a
dependency in future.
* fix style errors
* bump up the version for 5.2.3 release. tested +opencl, +hip and ~hip~opencl(cpu backend)
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* Initial opencatalyst proxy app spackage
Initial spackage for opencatalyst proxy app. Includes exposing of
versions in dependency spackages
* Verified Functionality and Spack Style
Verified build of mlperf-opencatalyst and fixed lingering spack style
issues
* Making requested changes to py-ocp
Making requested changed to spackage
* Further Requested Changes
Since fmt@9.0.0 and 9.1.0 were [added](6c4acfbf83) to spack a few days ago, gaudi fails to compile with default concretization. Since gaudi developers are usually paying attention to new versions of dependencies, I'm going to assume (perhaps optimistically) that the next bugfix version of gaudi will fix this (even though the issue has not been reported yet to Gaudi; I posted on the [key4hep public mirror](https://github.com/key4hep/Gaudi/issues/1)).
* Add fixed version 0.7.0 to cpu-features
* [@spackbot] updating style on behalf of iarspider
Co-authored-by: iarspider <iarspider@users.noreply.github.com>
* started updating multiqc package
* working now
* added py-rich-click
* fixed style
* changed py-matlibplot versions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* changed py-networkx versions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* changed py-coloredlogs versions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* changed python versions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* changed py-markdown versions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* changed py-pyyaml requirement
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* changed py-requests requirements
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* changed py-spectra requirements
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Bump up the version for rocm-5.2.1-initial commit
* Bump up the version for rocm-5.2.1 release
* Bump up the version for rocm-5.2.1 release
* correct the PROF_API_HEADER_PATH to include
* Bump up the version of rocm-openmp-extras for rocm-5.2.1 release
* bump up the version of rocwmma for 5.2.1
* imagemagick: new variant ghostscript
Ghostscript adds about 75 dependencies to an installation of
imagemagick (97 without, 172 with Ghostscript). This adds a
variant (defaulting to true for backward compatibility) that
allows users to turn off Ghostscript support.
* imagemagick: be explicit when `--with-gslib`
* imagemagick: fix suggestion fail
* imagemagick: use spec.prefix.share.font
* imagemagick: default ghostscript false
* imagemagick: no need for join_path anymore
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
This PR deprecates using Spack to install [EOL Python versions](https://endoflife.date/python), including Python 2.7, 3.1–3.6. It also deprecates running Spack with Python 2.7. Going forward, we expect Spack to have the following Python support timeline.
### Running Spack
* Spack 0.18 (spring 2022): Python 2.7, 3.5–3.10
* Spack 0.19 (fall 2022): Python 2.7 (deprecated), 3.6–3.11
* Spack 0.20 (spring 2023): Python 3.6–3.11
### Building with Spack
* Spack 0.18 (spring 2022): Python 2.7, 3.1–3.10
* Spack 0.19 (fall 2022): Python 2.7, 3.1–3.6 (deprecated), 3.7–3.11
* Spack 0.20 (spring 2023): Python 3.7–3.11
This is a reboot of #28003. See #31824 for a detailed discussion of the motivation for this PR.
If you have concerns about this change, please comment on #31824.
* py-datalad: add 0.17.5
* Fix description of py-types-urllib3
* Update var/spack/repos/builtin/packages/py-datalad/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-datalad/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add colorama dependency for windows
* Fix importlib-metadata dependency
* Update var/spack/repos/builtin/packages/py-pytest/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pytest/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pytest/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Use conflict to avoid dependency duplication
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Allow the nccl flag to be specified even for ROCm builds so that NCCL kernels are included in the build.
In this case the NCCL kernels will use RCCL as the backend implementation.
* Adding intel-oneapi-itac package
* Make black happy
* add rscohn2 as maintainer
* black prefers double quotes
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
Co-authored-by: Robert Cohn <rscohn2@gmail.com>
* Don't run bootstrap on package only PRs
* Run bootstrap tests when ci.yaml is modified
* Test a package only PR
* Revert "Test a package only PR"
This reverts commit af96b1af60b0c31efcc9a2875ffb1a809ef97241.
Now that `podio` can support `cxxstd` variants 17 and 20, we can allow `edm4hep` to use `cxxstd=20` as well, but must ensure that `edm4hep` uses the same `cxxstd` variant as `podio`. Solution as in `celeritas`.
This fixes a bug where two installations that differ only by package hash will not show
up in `spack find`.
The bug arose because `_cmp_node` on `Spec` didn't include the package hash in its
yielded fields. So, any two `Spec` objects that were only different by package hash
would appear to be equal and would overwrite each other when inserted into the same
`dict`. Note that we could still *install* specs with different package hashes, and they
would appear in the database, but we code that needed to put them into data structures
that use `__hash__` would have issues.
This PR makes `Spec.__hash__` and `Spec.__eq__` include the `process_hash()`, and it
makes `Spec._cmp_node` include the package hash. All of these *should* include all
information in a spec so that we don't end up in a situation where we are blind to
particular field differences.
Eventually, we should unify the `_cmp_*` methods with `to_node_dict` so there aren't two
sources of truth, but this needs some thought, since the `_cmp_*` methods exist for
speed. We should benchmark whether it's really worth having two types of hashing now
that we use `json` instead of `yaml` for spec hashing.
- [x] Add `package_hash` to `Spec._cmp_node`
- [x] Add `package_hash` to `spack.solve.asp.spec_clauses` so that the `package_hash`
will show up in `spack diff`.
- [x] Add `package_hash` to the `process_hash` (which doesn't affect abstract specs
but will make concrete specs correct)
- [x] Make `_cmp_iter` report the dag_hash so that no two specs with different
process hashes will be considered equal.
* initial commit of 0.5.0 changes
* updated dependencies
* updated ffcx sha
* comment style
* llvm compilers
* introduce pugixml dependency for 0.5.0:
* update compilers to support C++20 features
* style fixes
* xtensor and xtl not needed for basix 0.5.0 and above
* Skip to Basix 0.5.1
The 0.5.1 release removes the C++ build dependency on Python that sneaked into the 0.5.0 build system.
* Improve depends on version ranges
* More dependency version improvements
Co-authored-by: Chris Richardson <chris@bpi.cam.ac.uk>
Co-authored-by: Garth N. Wells <gnw20@cam.ac.uk>
* added missing dependency for py-msgpack that breaks neovim
* Update var/spack/repos/builtin/packages/py-pynvim/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Robert Underwood <runderwood@anl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The issue is that we are not not able to install (Fetch URL error) any
version of catalyst other than the specified in the spack package.py.
This very version is accessible only because it is cached by Spack. The
real URL does not exist anymore, I believe the reason is that there used
to be a tag in catalyst that does not exist anymore.
53a7b49 created a reference error which broke `.libs` (and
`find_libraries`) for many packages. This fixes the reference
error and improves the testing for `find_libraries` by actually
checking the extension types of libraries that are retrieved by
the function.
* catch json schema errors and reraise as property of SpackError
* no need to catch subclass of given error
* Builtin json library for Python 2 uses more generic type
* Correct instantiation of SpackError (requires a string rather than an exception)
* Use exception chaining (where possible)
* Add libraqm package
* py-pillow: Add optional raqm dependency/variant
* Use sha256
* Use " instead of '
* Use more explicit import
* Only add raqm from @8.4.0:
* Make the docstring shorter to satisfy flake
* Add conflict, silence warning, adjust version
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Add a post-install step which runs (only) on Windows to modify an
install prefix, adding symlinks to all dependency libraries.
Windows does not have the same concept of RPATHs as Linux, but when
resolving symbols will check the local directory for dependency
libraries; by placing a symlink to each dependency library in the
directory with the library that needs it, the package can then
use all Spack-built dependencies.
Note:
* This collects dependency libraries based on Package.rpath, which
includes only direct link dependencies
* There is no examination of libraries to check what dependencies
they require, so all libraries of dependencies are symlinked
into any directory of the package which contains libraries
UnifyFS:
- Add 1.0 release
- Deprecate older, unsupported versions
- Set fortran variant to true by default
- Update gotcha and mochi-margo dependency versions for unifyfs@1.0
and unifyfs@develop
- Add conflict of unifyfs with libfabric 1.13.* versions
- Update configure_args to use helper functions
GOTCHA: Hasn't been under active development for a couple years but
the develop branch has some fixes UnifyFS uses. To avoid having
UnifyFS v1.0 depend on a develop branch of a dependency, this creates
a new release in the Gotcha Spack package based on the most recent
commit of the develop branch.
* Adds hyperqueue package
* Adds maintainers
* Adds missing dockstring
* Bump year for Copyright
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* Fixes checksum, selects preferred version
Don't use a release candidate as the default version
* Switch maintainer to one of the developers for HQ
* Update package.py
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* Add new intel-mpi-benchmarks version
* Add new versions of intel mpi benchmarks
* Fix style bugs
* Fix style bugs
* Switch to using url_for_version formatting and improve patch ranges
* p2p benchmark is not included on older versions
* Set patch to proper version
* Add url field, improve patch versioning, improve version detection
* Add url field, improve patch versioning, improve version detection
* Bug fix
Syntax fix
* Remove 2019 from valid version on reorder_benchmark_macros patch
* OpenMPI isn't supported on older versions of the benchmark. Prevents OpenMPI from being selected on those versions
* Add new requirement of gmake for older versions
* Require intel-mpi for older versions of benchmark
* Minor changes to build directory for older versions
* Remove repeated conflict
* Minor style changes
* Minor change
* Correct fix for intel-mpi-benchmarks
* Bug fix
* Bug fix
* Attempted fix for install bug
* Attempted fix for install bug
* Remove duplicate build_directory setting
* new packages: py-arm-pyart and dependencies
- py-arm-pyart
- py-cylp
- rsl
* Update var/spack/repos/builtin/packages/py-cylp/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fix dependencies
- xarray is not optional
- pandas is needed
- pylab is needed
- new package, py-pylab-sdk
- setuptools is needed at run time
* Patch for import of StringIO
* Update var/spack/repos/builtin/packages/py-arm-pyart/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fix call to `StringIO` in patch
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* ci: restore coverage computation
* Mark "test_foreground_background" as xfail
* Mark "test_foreground_background_output" as xfail
* Make number of processes explicit, remove verbosity on linux
* Run coverage on just 3 Python jobs for linux
* Run coverage on just 3 Python jobs for linux
* Run coverage on just 2 Python jobs for linux
* Add back verbose, since before we didn't encounter the xdist internal error
* Reduce the workers to 2
* Try to use command line
* Fix a version cmp bug in asp.py
* Fix submodule bug for git refs
* Add branch in logic for submodules
* Fix git version comparisons
main does not satisfy git.foo=main
git.foo=main does satisfy main
* [py-python-bioformats] New package
* [py-python-bioformats] Added version 4.0.0
* [py-python-bioformats] Added types
* [py-python-bioformats] setuptools is build only
* [py-python-bioformats] fixup import
* [@spackbot] updating style on behalf of qwertos
Co-authored-by: James A Zilberman <jazrc@rit.edu>
Co-authored-by: qwertos <qwertos@users.noreply.github.com>
* Fixed the py-protobuf recipe so that when cpp support is require so
that it uses the same major and minor version range for the protobuf
package.
* Fixed the range bound for the 3.x py-protobuf packages.
Added mappings for 4.x py-protobuf packages to 3.x protobuf packages.
Removed a hash for v21.1 protobuf and replaced with v3.21.1 to keep a
standard versioning convention. Note that while Google has started
releasing both 3.x.y and a tag that dropped the leading 3. so it is
just x.y. This provides the appearance of a new major version, but
really is just a new minor version. These packages still report
versions as 3.x.y, so switching to versions and hashes with that
convention.
* Simplified constraints based on reviewer comments.
* Fixed flake8 errors
* Update var/spack/repos/builtin/packages/py-protobuf/package.py
* Fixed constraints on v2. versions and addressed Flake8 comments.
* Fixed flake8
* Fixed range dependencies for version 2.x
* Update var/spack/repos/builtin/packages/py-protobuf/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fixed version ranges to skip unknown versions.
* Fixed the dependencies on protobuf to solve weird build issues.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* ci: remove !docs from "core" filters
Written like it is now it causes package only PRs
to run with coverage.
* Try to skip job under condition, see if the workflow proceed
* Try to cancel a running CI job
* Simplify linux unit-tests, skip windows unit-tests on package PRs
* Reduce the inputs to unit-tests workflow
* Move control logic to main workflow, remove inputs
* Revert "Move control logic to main workflow, remove inputs"
This reverts commit 0c46fece4c.
* Do not compute "with_coverage" since it's always == to "core"
* Remove workflow dispatch from unit tests
* Revert "Revert "Move control logic to main workflow, remove inputs""
This reverts commit dd4e4a4e61.
* Try to skip all from the main workflow
* Add back bootstrap to needed checks for "all"
* Restore the correct logic for conditionals
* Versions added for each dep, but I think I'll need to remove them
* py-tesseract now builds and will import in python
* Fixed flake style error as raised by pipeline
* changed to proper python dependency
* added pil as a dependency
* Fixed flake style errors
* [py-pytesseract] py-pillow and py-wheel are redundant
* [py-pytesseract]
- fixed spelling
- removed unneeded dependency
* [py-pytesseract] update import
Co-authored-by: Viv Eric Hafener <vehrc@sporcbuild.rc.rit.edu>
* Add two no-op jobs named "all-prechecks" and "all"
These are a suggestion from @tgamblin, they are stable named markers we
can use from gitlab and possibly for required checks to make CI more
resilient to refactors changing the names of specific checks.
* Enable parallel testing using xdist for unit testing in CI
* Normalize tmp paths to deal with macos
* add -u flag compatibility to spack python
As of now, it is accepted and ignored. The usage with xdist, where it
is invoked specifically by `python -u spack python` which is then passed
`-u` by xdist is the entire reason for doing this. It should never be
used without explicitly passing -u to the executing python interpreter.
* use spack python in xdist to support python 2
When running on python2, spack has many import cycles unless started
through main. To allow that, this uses `spack python` as the
interpreter, leveraging the `-u` support so xdist doesn't error out when
it unconditionally requests unbuffered binary IO.
* Use shutil.move to account for tmpdir being in a separate filesystem sometimes
update libflame for work with crayCC, craycc, crayftn compiler wrappers. These lightweight compiler drivers do not add the `-L<lib_path>` like the CC/cc/ftn compiler drivers do. I've made a slight change to add the lib directories.
This change adds support for building the rocthrust tests and adds the `amdgpu_target`
variant to the `rocthrust` package.
- [x] rocthrust: add amdgpu_target and spack build test
- [x] Drop numactl as it is not a direct dependency
* add workaround for broken behavior in HIP
Hip has a longstanding cmake issue where they calculate include paths
incorrectly, this works around it for raja and adds an explicit rocprim
dependency.
* propagate openmp requirement and workaround to camp
* refactor and include umpire
* propagate openmp option to camp in umpire and use main camp for main and develop raja and umpire
* bump camp to new patch release
This patchset refactors our GitHub actions into a single top-level ci workflow that
invokes a series of reusable actions. The main goal of this is to be able to easily
control which tests run and in what order based on the success or failure of top-level
prechecks. Our previous workflows ran in three sets:
* nix tests: style and verification first, then linux and macos tests if successful
* windows tests: style and verification first, then linux and macos tests if successful
* bootstrap tests
As a result, the bootstrap tests ran even if the style failed, and style and verification
had to run on two different platforms despite running identical checks. I'm relatively
sure that's because of the limitation on dependencies between steps in the jobs.
Reusable workflows allow us to run the style, verification and now audit checks once,
then depending on the results, and the files changed, run the appropriate nix, windows
and bootstrap tests. While it saves only a few minutes by itself, this makes it easier to
refactor checks to subset tests without having to replicate tests or other workflow
components in the future.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Fix GCC compiler warnings due to not using C99 mode
CC should be overriden with Spack's value, and the other flags needed
to be copied from the Makefile.
Move the copying of the buildcache to a root job that runs after all the child
pipelines have finished, so that the operation can be coordinated across all
child pipelines to remove the possibility of race conditions during potentially
simlutandous copies. This lets us ensure the .spec.json.sig and .spack files
for any spec in the root mirror always come from the same child pipeline
mirror (though which pipeline is arbitrary). It also allows us to avoid copying
of duplicates, which we now do.
If you have an environment like
```
$ cat spack.yaml
spack:
specs: [openmpi@4.1.0+cuda]
```
this PR provides a new command `spack change` that you can use to adjust environment specs from the command line:
```
$ spack change openmpi~cuda
$ cat spack.yaml
spack:
specs: [openmpi@4.1.0~cuda]
```
in other words, this allows you to tweak the details of environment specs from the command line.
Notes:
* This is only allowed for environments that do not define matrices
* This is possible but not anticipated to be needed immediately
* If this were done, it should probably only be done for "named"/not-anonymous specs (i.e. we can change `openmpi+cuda` but not spec like `+cuda` or `@4.0.1~cuda`)
The building of tests is optional [as of 2.42.9](801eef111d). This applies this option in the build.
The reason the option was added was to deal with test build failures in sandboxed environments and with certain glibc versions (caused by glib gresources). For example, with the latest version glibc and in the latest version of docker these tests [cannot be built](https://github.com/moby/moby/issues/43595).
* llvm: fix 15.0.0rc builds on MacOS with command-line-tools
Ref: https://github.com/llvm/llvm-project/issues/57037
i.e use -DBUILTINS_CMAKE_ARGS=-DCOMPILER_RT_ENABLE_IOS=OFF. But this needs switching "compiler-rt" from "projects" to "runtimes".
Also fixing the warnings below fixes compile errors
CMake Warning at CMakeLists.txt:101 (message):
Using LLVM_ENABLE_PROJECTS=libcxx is deprecated now, please use
-DLLVM_ENABLE_RUNTIMES=libcxx or see the instructions at
https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.
CMake Warning at CMakeLists.txt:101 (message):
Using LLVM_ENABLE_PROJECTS=libcxxabi is deprecated now, please use
-DLLVM_ENABLE_RUNTIMES=libcxxabi or see the instructions at
https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.
CMake Warning at CMakeLists.txt:101 (message):
Using LLVM_ENABLE_PROJECTS=libunwind is deprecated now, please use
-DLLVM_ENABLE_RUNTIMES=libunwind or see the instructions at
https://libcxx.llvm.org/BuildingLibcxx.html for building the runtimes.
/private/var/folders/nt/_m1t_x7j76q6sl3xt91tqgs00000gn/T/balay/spack-stage/spack-stage-llvm-15.0.0-rc2-h2t5bohzyy7exz2ub3m42pfycjcmbndk/spack-build-h2t5boh/include/c++/v1/cstdlib:135:9: error: no member named 'at_quick_exit' in the global namespace
using ::at_quick_exit _LIBCPP_USING_IF_EXISTS;
~~^
* Update var/spack/repos/builtin/packages/llvm/package.py
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
fixes#31484
Before this change if anything was matching an external
condition, it was considered "external" and thus something
to be "built".
This was happening in particular to external packages
that were re-read from the DB, which then couldn't be
reused, causing the problems shown in #31484.
This PR fixes the issue by excluding specs with a
"hash" from being considered "external"
* Test that users have a way to select a virtual
This ought to be solved by extending the "require"
attribute to virtual packages, so that one can:
```yaml
mpi:
require: 'multi-provider-mpi'
```
* Prevent conflicts to be enforced on specs that can be reused.
* Rename the "external_only" fact to "buildable_false", to better reflect its origin
* py-breathe: add new version and improve version constraints
* py-breathe: everyone loves versions
```
py-breathe, py-breathe in the air
don't be afraid to care
````
* Update var/spack/repos/builtin/packages/py-breathe/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* add comment
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Preliminary support for include URLs in spack.yaml (environment) files
This commit adds support in environments for external configuration files obtained from a URL with a preference for grabbing raw text from GitHub and gitlab for efficient downloads of the relevant files. The URL can also be a link to a directory that contains multiple configuration files.
Remote configuration files are retrieved and cached for the environment. Configuration files with the same name will not be overwritten once cached.
Extend the semantics of package requirements to
allow using them also under a virtual package
attribute in packages.yaml
These requirements are enforced whenever that
virtual spec is present in the DAG.
Per #32214, the existing patch `92b2c...` for `gcc@11:` only applies
cleanly for `libzmq@4.3.3:4.3.4`. This PR adds a conflict for earlier
versions, as they cannot be patched due to different context.
For `gcc@11`, this leaves the most recent two versions available (a
satisfactory compromise).
For `gcc@12`, however, there is another existing conflict that makes
these most recent two versions unavailable. This PR adds an upstream
patch for the single most recent version that allows compilation with
`gcc@12` for that most recent version.
Starting point:
- `gcc@11` concretizes on all versions, attempts to apply patch on
`@4.2.3:4.3.4`, and only succeeds to apply patch on `@4.3.3:4.3.4`,
- `gcc@12` concretizes on `@:4.3.1` (and `@master`), attempts to apply
patch on `@4.2.3:4.3.1`, fails to apply patch on all.
Ending point:
- `gcc@11` concretizes on `@4.3.3:4.3.4` (and `@master`), attempts and
succeeds to apply patch on `@4.3.3:4.3.4`,
- `gcc@12` concretizes on `@4.3.4` (and `@master`), attempts and
succeeds to apply patch on `@4.3.4`.
Verified with environment build:
```yaml
spack:
specs:
- libzmq@4.3.4%gcc@12.1.0
- libzmq@4.3.4%gcc@11.3.0
- libzmq@4.3.3%gcc@11.3.0
view: false
```
which returns the following:
```console
16:14:47 wdconinc@menelaos ~/git/spack (libzmq-patch-gcc12 *+$%=) $
spack install --fresh
==> Installing environment libzmq
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-12.1.0/libmd-1.0.4-egpgd6eoaqtsl5fja2iwsl6gyc4o32p5
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-12.1.0/libsodium-1.0.18-af3rsfnvck6anxf7eeog3f2bph44tjia
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-12.1.0/pkgconf-1.8.0-z5of2hj2c6ygd3kxr4cwv7u7t42sxair
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-11.3.0/libmd-1.0.4-tec234gco2sd7n52dkwbrad43sdhaw4o
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-11.3.0/libsodium-1.0.18-uljf675u3yrn5c7fdjdpa5c7qnnkynke
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-11.3.0/pkgconf-1.8.0-l4hzc2g4pnn7dwyttphmxivt3xghvpoq
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-12.1.0/libbsd-0.11.5-fi3ri64moy45ksr4sf5pcwd6k23dsa4o
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-11.3.0/libbsd-0.11.5-2matmm7im7oygrr77k7wznttv4rbupfz
==> Installing libzmq-4.3.4-t7ad54q3atrnte4rzq7g7cfjcw5227pr
==> No binary for libzmq-4.3.4-t7ad54q3atrnte4rzq7g7cfjcw5227pr found:
installing from source
==> Fetching
c593001a89.tar.gz
==> Fetching
310b8aa57a
==> Fetching
https://github.com/zeromq/libzmq/pull/4334.patch?full_index=1
==> Applied patch
92b2c38a2c.patch?full_index=1
==> Applied patch
https://github.com/zeromq/libzmq/pull/4334.patch?full_index=1
==> libzmq: Executing phase: 'autoreconf'
==> libzmq: Executing phase: 'configure'
==> libzmq: Executing phase: 'build'
==> libzmq: Executing phase: 'install'
==> libzmq: Successfully installed
libzmq-4.3.4-t7ad54q3atrnte4rzq7g7cfjcw5227pr
Fetch: 0.61s. Build: 1m 31.57s. Total: 1m 32.18s.
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-12.1.0/libzmq-4.3.4-t7ad54q3atrnte4rzq7g7cfjcw5227pr
==> Installing libzmq-4.3.3-pxrd6piprucu65bkro2ixms6d3x2eudz
==> No binary for libzmq-4.3.3-pxrd6piprucu65bkro2ixms6d3x2eudz found:
installing from source
==> Fetching
9d9285db37.tar.gz
==> Using cached archive:
/home/wdconinc/.spack/cache/_source-cache/archive/31/310b8aa57a8ea77b7ac74debb3bf928cbafdef5e7ca35beaac5d9c61c7edd239
==> Applied patch
92b2c38a2c.patch?full_index=1
==> libzmq: Executing phase: 'autoreconf'
==> libzmq: Executing phase: 'configure'
==> libzmq: Executing phase: 'build'
==> libzmq: Executing phase: 'install'
==> libzmq: Successfully installed
libzmq-4.3.3-pxrd6piprucu65bkro2ixms6d3x2eudz
Fetch: 0.93s. Build: 11.55s. Total: 12.48s.
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-11.3.0/libzmq-4.3.3-pxrd6piprucu65bkro2ixms6d3x2eudz
==> Installing libzmq-4.3.4-hiil6dzy2reb4nk555zztwh44rpbyv4h
==> No binary for libzmq-4.3.4-hiil6dzy2reb4nk555zztwh44rpbyv4h found:
installing from source
==> Using cached archive:
/home/wdconinc/.spack/cache/_source-cache/archive/c5/c593001a89f5a85dd2ddf564805deb860e02471171b3f204944857336295c3e5.tar.gz
==> Using cached archive:
/home/wdconinc/.spack/cache/_source-cache/archive/31/310b8aa57a8ea77b7ac74debb3bf928cbafdef5e7ca35beaac5d9c61c7edd239
==> Applied patch
92b2c38a2c.patch?full_index=1
==> libzmq: Executing phase: 'autoreconf'
==> libzmq: Executing phase: 'configure'
==> libzmq: Executing phase: 'build'
==> libzmq: Executing phase: 'install'
==> libzmq: Successfully installed
libzmq-4.3.4-hiil6dzy2reb4nk555zztwh44rpbyv4h
Fetch: 0.01s. Build: 10.77s. Total: 10.78s.
[+]
/opt/software/linux-ubuntu22.04-skylake/gcc-11.3.0/libzmq-4.3.4-hiil6dzy2reb4nk555zztwh44rpbyv4h
```
* ddt: Initial commit
* ddt: Minor change to license date to appease the CI gods
* Get rid of unattractive extra line
* Switch to sha256 instead of md5
* Initial Draft of E3SM-Kernels Spackage
Initial draft of E3SM-Kernels. Currently no support for nested_loops due
to build system limitations
* Style Check and Fixed gfortran Check
Fixed style issues and changed gnu toolchain check to a gfortran check
due to hybrid compilers (e.g. clang+gfortran)
* Fixed Style Issues
* tixi: new variants (fortran,shared)
Since some tixi 3 versions, additional CMake flags are needed to build
tixi with shared libraries, respectively with Fortran support.
* tixi: fix style
* Changes for ROCm-5.2.0 changes and new recipe rocwmma
* modify the maintainers for hipify-clang
* address review comments
* update the rocwmma new recipe as per latest syntax
* fix style errors
* modify the patch file to provide the details about the patch
* fix style errors
* whizard: Fix passing of build options, update versions
The dependency of whizard on libtirpc is now correctly passed down as
an autotools option.
Update known versions of package with 3.0.2 and 3.0.3.
* Express path to headers via spec object methods
Allow users to express default requirements in packages.yaml.
These requirements are overridden if more specific requirements
are present for a given package.
* cpu-features: fix fetch failure
`master` branch was renamed to `main`
* Update var/spack/repos/builtin/packages/cpu-features/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [py-loguru] New package
* [py-loguru] Removed commented out line
* [py-loguru] Added types removed extra dependencies
* [py-loguru] missing windows dependency. listing windows as a conflict for now
* [py-loguru] depends on py-colorama when platform=windows
* [py-loguru] flake8
* [py-loguru] Import update
* [py-loguru]
- python is a runtime dependency
- setuptools is a build dependency
* [@spackbot] updating style on behalf of qwertos
Co-authored-by: James A Zilberman <jazrc@rit.edu>
Co-authored-by: qwertos <qwertos@users.noreply.github.com>
* libfabric: add new versions
1.15.0, 1.15.1, main (previously named master)
* Add OPX fabric option, with conflict for versions before v1.15.0
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* geant4: use define function
* geant4: Change new feature from conflicts to when
* geant4: add support/conflicts for nvhpc
* fixup! geant4: add support/conflicts for nvhpc
This support requires adding the '--tests' option to 'spack ci rebuild'.
Packages whose stand-alone tests are broken (in the CI environment) can
be configured in gitlab-ci to be skipped by adding them to
broken-tests-packages.
Highlights include:
- Restructured 'spack ci' help to provide better subcommand summaries;
- Ensured only one InstallError (i.e., installer's) rather than allowing
build_environment to have its own; and
- Refactored CI and CDash reporting to keep CDash-related properties and
behavior in a separate class.
This allows stand-alone tests from `spack ci` to run when the `--tests`
option is used. With `--tests`, stand-alone tests are run **after** a
**successful** (re)build of the package. Test results are collected
and report(able) using CDash.
This PR adds the following features:
- Adds `-t` and `--tests` to `spack ci rebuild` to run stand-alone tests;
- Adds `--fail-fast` to stop stand-alone tests after the first failure;
- Ensures a *single* `InstallError` across packages
(i.e., removes second class from build environment);
- Captures skipping tests for externals and uninstalled packages
(for CDash reporting);
- Copies test logs and outputs to the CI artifacts directory to facilitate
debugging;
- Parses stand-alone test results to report outputs from each `run_test` as
separate test parts (CDash reporting);
- Logs a test completion message to allow capture of timing of the last
`run_test` part;
- Adds the runner description to the CDash site to better distinguish entries
in CDash tables;
- Adds `gitlab-ci` `broken-tests-packages` to CI configuration to skip
stand-alone testing for packages with known issues;
- Changes `spack ci --help` so description of each subcommand is a single line;
- Changes `spack ci <subcommand> --help` to provide the full description of
each command (versus no description); and
- Ensures `junit` test log file ends in an `.xml` extension (versus default where
it does not).
Tasks:
- [x] Include the equivalent of the architecture information, or at least the host target, in the CDash output
- [x] Upload stand-alone test results files as `test` artifacts
- [x] Confirm tests are run in GitLab
- [x] Ensure CDash results are uploaded as artifacts
- [x] Resolve issues with CDash build-and test results appearing on same row of the table
- [x] Add unit tests as needed
- [x] Investigate why some (dependency) packages don't have test results (e.g., related from other pipelines)
- [x] Ensure proper parsing and reporting of skipped tests (as `not run`) .. post- #28701 merge
- [x] Restore the proper CDash URLand or mirror ONCE out-of-band testing completed
* gaudi: consistent test dependencies when +examples
Gaudi requires testing to be enabled for examples to be built, so all
test dependencies are also there for `+examples`. This PR fixes a
missing pytest dependency when `+examples` is used but no testing is
enabled. The construct with the loop is to ensure the identical
dependencies are always used, even as version ranges my start to differ.
Testing with gaudi_add_tests was added in v35r0. Some nosetests and
QMtests were in the tree before, but not accessible it seems. The
effective dependency since 35.0 is also applied for pytest, extending
the range that was there before disentangling `optional`, at
9d67d1e034/var/spack/repos/builtin/packages/gaudi/package.py
* gaudi: version 36.7 in other PR...
The "release" tarball provided by github lacks several files in
the SFconv/expat/xmlparse directory, including xmlparse. Using
tarballs based off of version tags solves the problem.
o Changes version() to use commits associated with version tags.
o Adds several additional versions.
o Adds myself as maintainer.
o Adds hook to execute autogen.sh.
o Adds autotools &ct dependences.
o Removes expat dependence.
* openmc: add v0.13.1
* Add @paulromano as maintainer of openmc and py-openmc
* Address review comments from @adamjstewart
* Add back MPI variant in openmc package
In a fast-moving project with as many forks as LLVM, it's difficult to
accurately determine if a function exists just by checking the version
number. The existing version check fails, for example, with llvm-amdgpu
from ROCm 4.5. It is more robust to directly check if the function
exists.
Added the SHA256 for version 6.3.2 and added logic to eal with the change of naming pattern for the makefile.include files that now appears to leave out the "linux_" prefix. (Changes should be. backwards compatible.)
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Co-authored-by: alansill <alansill@users.noreply.github.com>
* modified list.py and added functionality for --tag
* Removed long and very long, shifted rest of code above return statement
* removed results variable
* added import statement at top
* added the line accidentally deleted
* added line accidentally deleted
* changed p.name to p, added line inside if statement
* line order switched
* [@spackbot] updating style on behalf of sparkyniner
* ran update completion command
* add tests
* Update lib/spack/spack/test/cmd/list.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [@spackbot] updating style on behalf of sparkyniner
* changed argument to mock_packages and moved code under filter by tag
* removed bad rebase code and added additional test
* [@spackbot] updating style on behalf of sparkyniner
* added line removed earlier
* added line removed earlier
* replaced function
* added more recommended changes
Co-authored-by: sairaj <sairaj@sairajs-MacBook-Pro.local>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* VTK-h: don't assume I have Fortran
Don't assume I have a working Fortran compiler in my toolchain :)
* Conduit: Do not Assume Fortran
* Ascent: Do not Assume Fortran
* fix style
* Build Tensorflow using the fork for rocm. Initial commit
* re-order the versions
* fix style errors
* address review comments
* add conflicts for rocm version
* address review comments
* remove rocm variant as its added by ROCmPackage
* add mothra tests
* py-globus-sdk: add new versions; unpin py-cryptography version constraint
* Update var/spack/repos/builtin/packages/py-globus-sdk/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-pyjwt: fix py-cryptography version constraints
* Update var/spack/repos/builtin/packages/py-pyjwt/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* ForTrilinos: new versions 2.0.1, 2.1.0
I also had to update the checksum for the released 2.0.0: see #32000 for
the explanation and solution to keep this from happening again.
* Soooo stylish
Assertions without messages if/when hit create a blank error message for users.
This PR adds error messages to all assertions in asp.py even
if it seems unlikely they will ever be needed.
* Sensei: Refactor package to work with v4.0.0
* Add missing MPI dependency
* Patch bug in libsim adapter
* Simplify conflicts with when-clauses
* Conflict variants that are incompatible (catalyst/libsim/ascent)
* Fix paraview version constraints to be more clear
* Add version constraints for VTK
* Drop unneeded visit restrictions
* Specify +vtkm dependency on ParaView's VTKm
* +hl is not needed for VTK, and is already specified in the VTK recipe
when it is needed
* Pass paths for adios2 and ascent packages
* ECP-SDK: Enable sensei
* CI: Add sensei to the data-vis-sdk pipeline
* Sensei: Change VISIT_DIR to work on linux
* Fixup: style check
* Sensei: Add patch for version detection
* CI: revert SDK pipeline in favor of new matrices
* Sensei: Formatting fixes
The argument is very likely a typo, and was meant to
be given to the fixture decorator. Since the value
being passed is the default, let's just remove it.
* metaeuk: new package
* sepp: new package
* busco: adding version 5.4.3
* busco: adding maintainers
* metauk: more accurate cmake dep
* sepp: adding missing dep and accurate python dep
* sepp: remove install_tree
* sepp: extend python
Ensure that build tools with module-level commands in spack use
the version built as part of their build graph if one exists.
This is now also required for mesa, scons, cmake and ctest, out
of graph versions of these tools in path will not be found unless
added as an external.
This bug appeared because a new version of rocprim needs cmake
3.16, while I have 3.14 in my path I had added an external for
cmake 3.20 to the dag, but 3.14 was still used to configure
rocprim causing it to fail. As far as I can tell, all the build
tools added in build_environment.py had this problem, despite the
fact that they should have been resolving these tools by name
with a path search and find the one in the dag that way. I'm
still investigating why the path searching and Executable logic
didn't do it, but this makes three of the build systems much more
explicit, and leaves only gmake and ninja as dependencies from
out in the system while ensuring the version in the dag is used
if there is one.
The additional sqlite version is to perturb the hash of python to
work around a relocation bug which will be fixed in a subsequent
PR.
The Python that comes with XCode command line tools is a bit weird.
This fixes two issues that were preventing it from bootstrapping:
- [x] CommandLineTools python does not come with a `python-config` executable. If you
don't have one of these, CMake's FindPython will, for some reason, assume that you
want Python 2, so you need to set `CLINGO_PYTHON_VERSION` to ensure that clingo
tells `find_package(Python <VERSION>)` the right thing.
- [x] We weren't setting `PYTHONHOME` correctly in Python's `package.py`. We were
setting it to the `prefix` from Python `sysconfig`, but `prefix` tells you where
the executable lives. CommandLineTools python reports its prefix as
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8
but that doesn't exist. It looks like Apple builds the full python and just copies
pieces of it into command line tools. PYTHONHOME is supposed to tell you where the
default python library location is. So you have to look at the `base`, which
`sysconfig` reports as
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8
On most systems this is probably the same as `prefix`, but not with
CommandLineTools, which has its system library in a different place.
The second change here was cherry-picked to 0d981a012d before merging and doesn't show
up here.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
- [x] Rework `headers` to search a sequence of directories and to display all searched
locations on error, as opposed to handling each directory with a variable
- [x] Make `headers` and `libs` do the same type of search and raise the same sort of
errors.
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
* [py-bitshuffle] New package
* [py-bitshuffle] compiling against spack hdf5
* [py-bitshuffle] flake8
* [py-bitshuffle] update import ine
* [@spackbot] updating style on behalf of qwertos
Co-authored-by: James A Zilberman <jazrc@rit.edu>
Co-authored-by: qwertos <qwertos@users.noreply.github.com>
We're seeing errors pop up with older versions of Kokkos and newer versions of
`oneapi`, specifically:
error: no member named 'ONEAPI' in namespace 'sycl'
This hapens because `sycl::ONEAPI` is `sycl::ext::oneapi` since oneapi `2022.0.0`.
`kokkos@3.6:` uses the new namespace. A conflict was present for this, but it was too
specific -- both to `dpcpp` and to the `2022.0.0` version.
- [x] Expand version ranges in `kokkos` conflict
- [x] Add conflict for `oneapi` in addition to `dpcpp`
* filesystem: use lstat in recursive mtime
When a `develop` path contains a dead symlink, the `os.stat` in the recursive `mtime` determination trips up over it.
Closes#32165.
`requirement_policy/3` is generated and may not be in Spack's inputs to Clingo.
Currently this is causing warnings like:
```
$ spack spec zlib
/global/u2/t/tgamblin/src/spack/lib/spack/spack/solver/concretize.lp:510:3-43: info: atom does not occur in any rule head:
requirement_policy(Package,X,"one_of")
/global/u2/t/tgamblin/src/spack/lib/spack/spack/solver/concretize.lp:517:3-43: info: atom does not occur in any rule head:
requirement_policy(Package,X,"one_of")
/global/u2/t/tgamblin/src/spack/lib/spack/spack/solver/concretize.lp:523:3-43: info: atom does not occur in any rule head:
requirement_policy(Package,X,"any_of")
/global/u2/t/tgamblin/src/spack/lib/spack/spack/solver/concretize.lp:534:3-43: info: atom does not occur in any rule head:
requirement_policy(Package,X,"any_of")
Input spec
--------------------------------
zlib
Concretized
--------------------------------
zlib@1.2.11%gcc@7.5.0+optimize+pic+shared arch=cray-sles15-haswell
```
- [x] Silence warning with `#defined requirement_policy/3`
Spack doesn't have an easy way to say something like "If I build
package X, then I *need* version Y":
* If you specify something on the command line, then you ensure
that the constraints are applied, but the package is always built
* Likewise if you `spack add X...`` to your environment, the
constraints are guaranteed to hold, but the environment always
builds the package
* You can add preferences to packages.yaml, but these are not
guaranteed to hold (Spack can choose other settings)
This commit adds a 'require' subsection to packages.yaml: the
specs added there are guaranteed to hold. The commit includes
documentation for the feature.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* celeritas: update checksum for git describe difference
Using `$Format:%h$` and the export-subst gitattribute property mean that
the release checksum can change post release...
```
$ diff -ru spack-stage-celeritas-0.1.0-hv7ewpjouekqws2y5iaql2cnp6tn76iz/ spack-stage-celeritas-0.1.0.wtf-ynyck3df2a7kkgkqrpwapgz3l2i62omz/
diff -ru spack-stage-celeritas-0.1.0-hv7ewpjouekqws2y5iaql2cnp6tn76iz/spack-src/cmake/CgvFindVersion.cmake spack-stage-celeritas-0.1.0.wtf-ynyck3df2a7kkgkqrpwapgz3l2i62omz/spack-src/cmake/CgvFindVersion.cmake
--- spack-stage-celeritas-0.1.0-hv7ewpjouekqws2y5iaql2cnp6tn76iz/spack-src/cmake/CgvFindVersion.cmake 2022-07-31 19:45:03.000000000 -0400
+++ spack-stage-celeritas-0.1.0.wtf-ynyck3df2a7kkgkqrpwapgz3l2i62omz/spack-src/cmake/CgvFindVersion.cmake 2022-07-31 19:45:03.000000000 -0400
@@ -75,7 +75,7 @@
# Get a possible Git version generated using git-archive (see the
# .gitattributes file)
- set(_ARCHIVE_TAG "HEAD -> master, tag: v0.1.0")
+ set(_ARCHIVE_TAG "tag: v0.1.0")
set(_ARCHIVE_HASH "04fe945d9")
set(_TAG_REGEX "v([0-9.]+)(-dev[0-9.]+)?")
Binary files spack-stage-celeritas-0.1.0-hv7ewpjouekqws2y5iaql2cnp6tn76iz/v0.1.0.tar.gz and spack-stage-celeritas-0.1.0.wtf-ynyck3df2a7kkgkqrpwapgz3l2i62omz/v0.1.0.tar.gz differ
```
* celeritas: Use exported and re-uploaded tar.gz for reproducibility
All PRs are failing the docs build on account of an error with
pygments. These errors coincide with a new release of pygments
(2.13.0) and restricting to < 2.13 allows the doc tests to pass,
so this commit enforces that constraint for the docs build.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
A few attribute in packages are meant to be reserved for
Spack internal use. This audit checks packages to ensure none
of these attributes are overridden.
- [x] add additional audit check
In file included from /data/xsdkci/VS1mJQ1K/0/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-oce-0.18.3-rim5zhuacb7z4plxag52fjj3gbc4znv3/spack-src/src/OSD/OSD_Parallel.cxx:17:
/data/xsdkci/VS1mJQ1K/0/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-oce-0.18.3-rim5zhuacb7z4plxag52fjj3gbc4znv3/spack-src/src/OSD/OSD_Parallel.hxx: In static member function 'static void OSD_Parallel::ForEach(InputIterator, InputIterator, const Functor&, Standard_Boolean)':
/data/xsdkci/VS1mJQ1K/0/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-oce-0.18.3-rim5zhuacb7z4plxag52fjj3gbc4znv3/spack-src/src/OSD/OSD_Parallel.hxx:223:18: error: 'captured_exception' in namespace 'tbb' does not name a type
223 | catch ( tbb::captured_exception& anException )
| ^~~~~~~~~~~~~~~~~~
/data/xsdkci/VS1mJQ1K/0/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-oce-0.18.3-rim5zhuacb7z4plxag52fjj3gbc4znv3/spack-src/src/OSD/OSD_Parallel.hxx:225:38: error: 'anException' was not declared in this scope
225 | Standard_NotImplemented::Raise(anException.what());
| ^~~~~~~~~~~
/data/xsdkci/VS1mJQ1K/0/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-oce-0.18.3-rim5zhuacb7z4plxag52fjj3gbc4znv3/spack-src/src/OSD/OSD_Parallel.hxx: In static member function 'static void OSD_Parallel::For(Standard_Integer, Standard_Integer, const Functor&, Standard_Boolean)':
/data/xsdkci/VS1mJQ1K/0/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-oce-0.18.3-rim5zhuacb7z4plxag52fjj3gbc4znv3/spack-src/src/OSD/OSD_Parallel.hxx:272:18: error: 'captured_exception' in namespace 'tbb' does not name a type
272 | catch ( tbb::captured_exception& anException )
| ^~~~~~~~~~~~~~~~~~
/data/xsdkci/VS1mJQ1K/0/xsdk-project/spack-xsdk/spack-stage/balay/spack-stage-oce-0.18.3-rim5zhuacb7z4plxag52fjj3gbc4znv3/spack-src/src/OSD/OSD_Parallel.hxx:274:38: error: 'anException' was not declared in this scope
274 | Standard_NotImplemented::Raise(anException.what());
| ^~~~~~~~~~~
`setup_dependent_build_environment(self, env, dependent_spec)` does not have a variable `spec`.
This causes several issues right now:
```console
==> Installing gaudi-36.6-cjjrpjwpcqrtojyrdqml3jpzkbn55hpb
==> No binary for gaudi-36.6-cjjrpjwpcqrtojyrdqml3jpzkbn55hpb found: installing from source
==> Error: NameError: name 'spec' is not defined
/home/wdconinc/git/spack/var/spack/repos/builtin/packages/root/package.py:614, in setup_dependent_build_environment:
611 env.prepend_path("ROOT_INCLUDE_PATH", dependent_spec.prefix.include)
612 if "+rpath" not in self.spec:
613 env.prepend_path("LD_LIBRARY_PATH", self.prefix.lib.root)
>> 614 if "platform=darwin" in spec:
615 # Newer deployment targets cause fatal errors in rootcling
616 env.unset("MACOSX_DEPLOYMENT_TARGET")
```
On PR pipelines we need to override the buildcache destination to
point to the "spack-binaries-prs" bucket, otherwise, those pipelines
try to push to the default mirror in a bucket for which they don't
have write permission.
This PR fixes the performance regression reported in #31985 and a few
other issues found while refactoring the spack mirror create command.
Modifications:
* (Primary) Do not require concretization for
`spack mirror create --all`
* Forbid using --versions-per-spec together with --all
* Fixed a few issues when reading specs from input file (specs were
not concretized, command would fail when trying to mirror
dependencies)
* Fix issue with default directory for spack mirror create not being
canonicalized
* Add more unit tests to poke spack mirror create
* Skip externals also when mirroring environments
* Changed slightly the wording for reporting (it was mentioning
"Successfully created" even in presence of errors)
* Fix issue with colify (was not called properly during error
reporting)
* [py-pynndescent] New package
* [py-pynndescent] Removed white space
* [py-pynndescent] fixed dependencies
* [py-pynndescent] flake8
* [py-pynndescent] Import update
Co-authored-by: James A Zilberman <jazrc@rit.edu>
- [x] Add release v0.2 of the AML package, deprecate v0.1, and add support for
OpenCL, HIP, and CUDA variants of the library. Also update repo and
release URL, as the previous one is not accessible anymore.
- [x] aml: add oneapi-level-zero support
- [x] Change openmp flags to force compatibility when compiling with
intel-oneapi compilers.
`LD_LIBRARY_PATH` can break system executables (e.g., when an enviornment is loaded) and isn't necessary thanks to `RPATH`s. Packages that require `LD_LIBRARY_PATH` can set this in `setup_run_environment`.
- [x] Prefix inspections no longer set `LD_LIBRARY_PATH` by default
- [x] Document changes and workarounds for people who want `LD_LIBRARY_PATH`
These changes make many packages build on nixos where nearly nothing
comes from /bin or /usr/bin (the only things in "system locations" are
/bin/sh and /usr/bin/env, all the rest is found through PATH).
Many configuration scripts hardcode /usr/bin/file instead of using the
one from PATH. This patches them to use file from PATH.
* ADIOS2: ZFP<1.0
The tagged ADIOS2 releases in Spack (and develop) do not yet
work with ZFP 1.0. Express valid depedency range before someone
updates the ZFP package and triggers the incompatibility.
* Use semver
* flux packages used in a spack view will already be unshallow
and this particular command will error and break the entire build. Instead
we want to catch the ProcessError and allow for a regular fetch.
* final tweaks to add missing sqlite and fallback to fetch
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
fixes#31736
Catch errors when concretizing specs and report them as
debug messages. The corresponding spec is skipped.
Co-authored-by: Greg Becker <becker33@llnl.gov>
* Extracted two functions in cmd/install.py
* Extracted a function to perform installation from the active environment
* Rename a few functions, remove args from their arguments
* Rework conditional in install_from_active_environment to reduce nesting in the function
* Extract functions to parsespecs from cli and files
* Extract functions to getuser confirmation for overwrite
* Extract functions to install specs inside and outside environments
* Rename a couple of functions
* Fix outdated comment
* Add missing imports
* Split conditional to dedent one level
* Invert check and exit early to dedent one level when requiring user confirmation
* tassel: add version 3.0.174
fix naming of 5.2.39 from "2017-07-22"
* changed single quotes to double quotes to appease the gods ;)
* fixed style issues
The current use of git ref's as a version requires a search algorithm to pick the right matching version based on the tags in the git history of the package.
This is less than ideal for the use case where users already know the specific version they want the git ref to be associated with. This PR makes a new version syntax [package]@[ref]=[version] to allow the users to specify the exact hash they wish to use.
* Add checksum for py-arror 1.2.1 and 1.2.2
* Update package.py
* Update var/spack/repos/builtin/packages/py-arrow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update package.py
* Update package.py
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [py-glymur] New Package: py-glymur
* [py-glymur] fixed copyright line
* [py-glumur] update import ine
* [@spackbot] updating style on behalf of qwertos
Co-authored-by: James A Zilberman <jazrc@rit.edu>
Co-authored-by: qwertos <qwertos@users.noreply.github.com>
There is a problem with the git repo for rhtslib that apparently led to
a bad version entry during the previous round of package updates. A git
checkout on the commit also fails so use branch for most recent version.
* new package: alphafold
and related dependencies, depends on #27138
* [@spackbot] updating style on behalf of aweits
* fix
Co-authored-by: aweits <aweits@users.noreply.github.com>
On some systems the shell in login mode wipes important parts of the
environment, such as PATH. This causes the build to fail since it can't
find `spack`.
For better robustness, don't use a login shell.
In a full CI job the final spack install is run in an environment formed by scripts running in this order:
export AWS_SECRET=... # 1. Load environment from GitLab project variables
source spack/share/spack/setup-env.sh # 2. Load Spack into the environment (PATH)
spack env activate -V concrete_env # 3. Activate the concrete environment
source /etc/profile # 4. Bash login shell (from -l)
spack install ...
Whereas when a user launches their own container with (docker|podman) run -it, they end up running spack install in an environment formed in this order:
source /etc/bash.bashrc # (not 4). Bash interactive shell (default with TTY)
export AWS_SECRET=... #~1. Manually load environment from GitLab project variables
source spack/share/spack/setup-env.sh # 2. Load Spack into the environment (PATH)
spack env activate -V concrete_env # 3. Activate the concrete environment
spack install ...
The big problem being that (4) has a completely different position and content (on Leap 15 and possibly other containers).
So in context, this PR removes (4) from the CI job case, leaving us with the simpler:
export AWS_SECRET=... # 1. Load environment from GitLab project variables
source spack/share/spack/setup-env.sh # 2. Load Spack into the environment (PATH)
spack env activate -V concrete_env # 3. Activate the concrete environment
spack install ...
* database: don't sort on return from query_local
* ASP-based solver: don't build the hash-lookup dictionary twice
Building this dictionary twice and traversing all the specs
might be time-consuming for large buildcaches.
This test relied on an old version of the `flake8_package` fixture that modified
the spack repository, but it doesn't do that anymore. There are other tests for
`changed_files()` that do a better job of mocking up a git repository with
changes, so we can just delete this one.
A GitHub rebase merge seems to rewrite commits even if it would be a
fast-forward, which means that the commit merged from #24718 is wrong.
- [x] update `.git-blame-ignore-revs` with real commit from `develop`
`spack style` tests were annoyingly brittle because we could not easily be
specific about which tools to run (we had to use `--no-black`, `--no-isort`,
`--no-flake8`, and `--no-mypy`). We should be able to specify what to run OR
what to skip.
Now you can run, e.g.:
spack style --tool black,flake8
or:
spack style --skip black,isort
- [x] Remove `--no-black`, `--no-isort`, `--no-flake8`, and `--no-mypy` args.
- [x] Add `--tool TOOL` argument.
- [x] Add `--skip TOOL` argument.
- [x] Allow either `--tool black --tool flake8` or `--tool black,flake8` syntax.
- [x] remove alignment spaces from tempaltes
- [x] replace single with double quotes
- [x] Makefile template now generates parsable code
(function body is `pass` instead of just a comment)
- [x] template checks now run black to check output
Previously we'd accept any version for bootstrapping black, but we need <= 21.
- [x] modify bootstrapping code to check black version before accepting an
executable from `PATH`.
- [x] add `.git-blame-ignore-revs` to ignore black reformatting
- [x] make `spack blame` respect `.git-blame-ignore-revs`
(even if the user hasn't configured git to do so)
Some of our tests rely on single vs. double quotes, and others rely on specific
line numbers in the source. These needed fixing after the switch to Black.
Black will automatically fix a lot of the exceptions we previously allowed for
directives, so we don't need them in our custom `flake8_formatter` anymore.
- [x] remove `E501` (long line) exceptions for directives from `flake8_formatter`,
as they won't help us now.
- [x] Refine exceptions for long URLs in the `flake8_formatter`.
- [x] Adjust the mock `flake8-package` to exhibit the exceptions we still allow.
- [x] Update style tests for new `flake8-package`.
- [x] Blacken style test.
Many noqa's in the code are no longer necessary now that the column limit is 99
characters. Others can easily be eliminated, and still more can just be made more
specific if they do not have to do with line length.
The only E501's still in the code are in the tests for `spack.util.path` and the tests
for `spack style`.
This adds necessary configuration for flake8 and black to work together.
This also sets the line length to 99, per the data here:
* https://github.com/spack/spack/pull/24718#issuecomment-876933636
Given the data and the spirit of black's 88-character limit, we set the limit to 99
characters for all of Spack, because:
* 99 is one less than 100, a nice round number, and all lines will fit in a
100-character wide terminal (even when the text editor puts a \ at EOL).
* 99 is just past the knee the file size curve for packages, and it means that packages
remain readable and not significantly longer than they are now.
* It doesn't seem to hurt core -- files in core might change length by a few percent but
seem like they'll be mostly the same as before -- just a bit more roomy.
- [x] set line length to 99
- [x] remove most exceptions from `.flake8` and add the ones black cares about
- [x] add `[tool.black]` to `pyproject.toml`
- [x] make `black` run if available in `spack style --fix`
Co-Authored-By: Tom Scogland <tscogland@llnl.gov>
* Add checksum for numba 0.55.2 and 0.56
* Add checksum for py-llvmlite 0.39.0
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* OpenGL: Restructures the OpenGL packages
This provides concrete glx and osmesa packages which delegate to
virtual libglx and libosmesa packages provided by mesa. This was
necessary because GLX and OSMesa both provide gl implementations but
with mesa providing the girtual gl package there was no way to properly
distinguish which of the two opengl implementations was beiing requested
when querying the spec['gl'] dependency. This additional level of
indirection allows for that.
* OpenGL: Adjust downstream dependents of OpenGL for the restructure
This implements the necessary fixes in the packages that depend on
OpenGL to work with the restructuring. This also attempts to create a
consistent variant for specifying glx or osmesa.
the patch urls dynamically generate a diff, which includes metadata
about the git version used, meaning they are not content-addressable.
instead ship the patches with spack.
In #31618 the idea was to determine the file extension heuristically by dropping query params etc from a url and then consider it as a file path. That broke for URLs that only have query params like http://example.com/?patch=x as it would result in empty string as basename. This PR reverts to the old behavior of saving files as ?patch=x in that case.
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
* herwig3: change lhapdfsets dependency type to build
These data sets are needed for a check during build, but due to the difficulty of versioning the datasets it is preferred not to keep the them in the run environment.
* herwig3: explicitly state needed boost libs
* thepeg: explicitly state needed boost libs
* style
* stylestyle
* Tau must get GCC path from environment on Cray
self.compiler doesn't provide the path to the gcc compiler when using cray cc and the spack internal compiler overrides the location in PATH. If possible get the location from the GCC_PATH variable instead.
* Fix flake8 issues
* Update package.py
* llvm: Use variant when clauses for many of the expressed conflicts
* llvm: Remove the shared variant as it wasn't really used
* llvm: Remove unnecessary deps and make explicit the ones that are
* llvm: Cleanup patch conditions
* pocl: Update for llvm cleanup
* unit-test: update unparse package hash with the updated llvm package
* llvm: Fix ppc long double patching and add clarifying comments
`self.archive_file` is (among others) a symlink to a tarball. `extension()` on a
symlink will result in no extension. This patch fixes the behavior introduced in
https://github.com/spack/spack/pull/31618.
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
When
1. Spack installs libtool,
2. system libtool is installed too, and
3. system automake is used
Spack passes system automake's `-I <prefix>` flag to itself, even though
it's a default search path. This takes precedence over spack's libtool
prefix dir. This causes the wrong `libtool.m4` file to be used (since
system libtool is in the same prefix as system automake).
And that leads to error messages about incompatible libtool, something
something LT_INIT.
fixes#31627
spack.mirror.get_all_versions now uses the package class
instead of the package object in its implementation.
Ensure spec is concrete before staging for mirrors
* Update open-ce patches for py-torch to us immutable URLs. Update magma dependency specs to be more explicit.
* Address comments for PR regarding URLs and conflicting variants.
Co-authored-by: Nicholas Cameron Sly <sly1@llnl.gov>
* rocblas: make tensile dependencies conditional
* Remove rocm-smi from the rocblas dependency list
rocm-smi was added to the rocblas dependency list because Tensile was a
dependency of rocBLAS and rocm-smi was a dependency of Tensile. However,
that reasoning was not correct.
Tensile is composed of three components:
1. A command-line tool for generating kernels, benchmarking them, and
saving the parameters used for generating the best kernels
(a.k.a. "Solutions") in YAML files.
2. A build system component that reads YAML solution files, generates
kernel source files, and invokes the compiler to compile then into
code object files (*.co, *.hsco). An index of the kernels and their
associated parameters is also generated and stored in either YAML
or MessagePack format (TensileLibrary.yaml or TensileLibrary.dat).
3. A runtime library that will load the TensileLibrary and code object
files when asked to execute a GEMM and choose the ideal kernel for
your specific input parameters.
rocBLAS developers use (1) during rocBLAS development. This is when
Tensile depends on rocm-smi. The GPU clock speed and temperature must be
controlled to ensure consistency when doing the solution benchmarking.
That control is provided by rocm-smi. When building rocBLAS, Tensile is
used for (2). However, there is no need for control of the GPU at that
point and rocm-smi is not a dependency. At runtime, the rocBLAS library
uses Tensile for (3), but there is again no dependency on rocm-smi.
tl;dr: rocm-smi is a dependency of the tensile benchmarking tool,
which is not a build dependency or runtime dependency of rocblas.
This PR contains several fixes for the kallisto package.
- create hdf5 variant as hdf5 is optional beginning with 0.46.2
- provide patch for 0.43 to link against libz
- provide patch for older versions to build again gcc-11 and up
- patch and code to use autoconf-2.70 and up
Newer versions of botocore (>=1.23.47) support the full IOBase
interface, so the hacks added to supplement the missing attributes are
no longer needed. Conditionally disable the hacks if they appear to be
unnecessary based on the class hierarchy found at runtime.
* py-panaroo: new package
* moving panaroo to branch
* updated mizani, plotnine, and pystan versions and requirements
* made suggested fixes
* adding more requested fixes
* added new versions of statsmodels and httpstan
* py-torch: add version 0.23.0 and fix to built on aarch64
* Add newer versions, fix build issues
* Fix tests
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add connection information to buildcache update command
Ensure that the s3 connection made when attempting to update the content
of a buildcache attempts to use the extra connection information
from the mirror creation.
* Add unique help for endpoint URL argument
Fix copy/paste error for endpoint URL help which was the same as
the access token
* Re-work URL checking for S3 mirrors
Due to the fact that nested bucket URLs would never match the string used
for checking that the mirror is the same, switch the check used.
Sort all mirror URLs by length to have the most specific cases first
and see if the desired URL "starts with" the mirror URL.
* Long line style fixes
Add execptions for long lines and fix other style errors
* Use format() function to rebuild URL
Use the format command to rebuild the url instead of crafing a
formatted string out of known values
* Add early exit for URL checking
When a valid mirror is found, break from the loop
For a long time the module configuration has had a few settings that use
`blacklist`/`whitelist` terminology. We've been asked by some of our users to replace
this with more inclusive language. In addition to being non-inclusive, `blacklist` and
`whitelist` are inconsistent with the rest of Spack, which uses `include` and `exclude`
for the same concepts.
- [x] Deprecate `blacklist`, `whitelist`, `blacklist_implicits` and `environment_blacklist`
in favor of `exclude`, `include`, `exclude_implicits` and `exclude_env_vars` in module
configuration, to be removed in Spack v0.20.
- [x] Print deprecation warnings if any of the deprecated names are in module config.
- [x] Update tests to test old and new names.
- [x] Update docs.
- [x] Update `spack config update` to fix this automatically, and include a note in the error
that you can use this command.
Python's built-in tarfile support doesn't address some general
cases of malformed tarfiles that are already handled by the system
'tar' utility; until these can be addressed, use that exclusively.
Add support for CMake builds while preserving autotools support
for older versions of GDAL
* Add GDAL 3.5.0
* Remove GDAL 1
* Add support for new CMake build system
* Change defaults to build all recommended dependencies
* Simplify Autotools flag handling
* Determine version range for drivers
The goal of this PR is to make clearer where we need a package object in Spack as opposed to a package class.
We currently instantiate a lot of package objects when we could make do with a class. We should use the class
when we only need metadata, and we should only instantiate and us an instance of `PackageBase` at build time.
Modifications:
- [x] Remove the `spack.repo.get` convenience function (which was used in many places, and not really needed)
- [x] Use `spack.repo.path.get_pkg_class` wherever possible
- [x] Try to route most of the need for `spack.repo.path.get` through `Spec.package`
- [x] Introduce a non-data descriptor, that can be used as a decorator, to have "class level properties"
- [x] Refactor unit tests that had to be modified to reduce code duplication
- [x] `Spec.package` and `Repo.get` now require a concrete spec as input
- [x] Remove `RepoPath.all_packages` and `Repo.all_packages`
* fixed the cgal recipe and added the latest release.
* Update var/spack/repos/builtin/packages/cgal/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* updated cgal recipe to new URL for tarballs
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
There's a race condition in `remove()` as the lockfile is removed after
releasing the lock, which is a problem when another process acquires a
write lock during deletion.
Also simplify life a bit in multiprocessing when a file is possibly
removed multiple times, which currently is an error on the second
deletion, so the proposed fix is to make remove(...) idempotent and not
error when deleting non-existing cache entries.
Don't tests for existence of lockfile, cause windows/linux behavior is different
Oversight in #31433, the non-phony `env` target was missing a file being
created for it, which can cause make to infinitely loop when including
multiple generated makefiles.
* Metall package: add dependency to GCC for build test
* Package Metall: add v.017
* Package Metall: update the package file
* Update var/spack/repos/builtin/packages/metall/package.py
Co-authored-by: Bernhard Kaindl <43588962+bernhardkaindl@users.noreply.github.com>
* Metall package: add v0.18 and v0.19
* Metall Package: add v0.20
* Metall package: add v0.21
Co-authored-by: Bernhard Kaindl <43588962+bernhardkaindl@users.noreply.github.com>
When no default editor is installed and no environment variable is set,
which_string would return None and this would be passed to os.execv
resulting in a TypeError. The message presented to the user would be:
Error: execv: path should be string, bytes or os.PathLike,
not NoneType
This change checks that which_string has returned successfully before
attempting to execute the result, resulting in a new error message:
Error: No text editor found! Please set the VISUAL and/or EDITOR
environment variable(s) to your preferred text editor.
It's not strictly necessary, but I've also changed try_exec to catch
all errors rather than just OSErrors. This would have provided slightly
more context for the original error message.
Use 8 byte reals only when `precision=double` is set
The pre-defined compilation targets do not follow the usual behavior of
Makefiles. Compiler flags are set as strings (not Makefile variables) and as
such are not able to be overridden by environment variables. This patch changes
the behavior to the expected behavior of a Makefile such that `fflags` etc have
the desired effect.
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
* [py-ipyparallel] setuptools.config.StaticModule moved
... in py-setuptools@61
* [py-ipyparallel] setuptools fix only added to release 8.2.1
https://github.com/ipython/ipyparallel/pull/680
* [py-ipyparallel] Fix typo
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
There were two choices: 1) remove '-p' from '-a' or 2) allow monkeypatching
the cleaning of the python cache since clean's test_function_calls isn't
supposed to be actually cleaning anything.
This commit supports the latter and adds a test case for `-p`.
* add recipe for improved-rdock
* style: fix format
* style: fix format again
* Fix location of the directory
* Fix copyright
* Fix according to the reviewer's comments.
* Revert "Fix according to the reviewer's comments."
This reverts commit 4877877daf.
* Fix according to the reviewer's comments.
* style: fix format
* Update var/spack/repos/builtin/packages/improved-rdock/package.py
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Yuichi Otsuka <otsukay@riken.jp>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* guppy: new package
* adding different hashs/urls
* working now
* guppy: new package
* ont-guppy: new package
* ont-guppy: new package
* ont-guppy: fixed style
- rocprim is a header-only library; its amdgpu_target variant is only
meaningful for its client executables, not the library itself
- comgr and llvm-amdgpu are merely indirect dependencies (via hip)
* Initial implementation of Omnitrace package
* Fix flake8 errors
* Fix run environment when +python
* String normalization and fix for build env when +tau
* Tweak to style
Release branches and tags run protected pipelines, and we noticed
that those pipelines were generating all jobs in the stack, even
when the mirror contained all the built specs and an up to date
index. The problem was caused because the override mirror was
not present in spacks mirror configuration at the time when the
binary_distribution.update() method was called. This fixes that
by always adding the mirror override, if present, to the list of
configured mirrors.
* py-altair: new package
* Update var/spack/repos/builtin/packages/py-altair/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-altair/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-altair/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-altair: update
* Update var/spack/repos/builtin/packages/py-altair/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* boost: do not access the context-impl variant for versions below 1.65.0
* boost: check if spec.variants contains context-impl
* boost: improve error message when the context-impl variant causes a conflict
Executing spack solve boost@1.63.0 +context context-impl=fcontext
triggers the following error message:
==> Error: No version for 'boost' satisfies '@1.63.0' and '@1.79.0'
With this change, the error message becomes the following:
==> Error: Cannot set variant 'context-impl' for package 'boost' because the variant condition cannot be satisfied for the given spec
We adopted the convention of putting binaries for each stack into
a dedicated mirror named after the directory in which the stack
(spack.yaml file) resides. This fixes the mirror url of the
radiuss-aws-aarch64 stack to follow that convention.
`make_target` can be used to instruct Spack to build one of the pre-defined make
targets in the MPAS makefile. Spack will guess the targret based on the value of
`spack_fc`, but the user can overwrite the target as variant. E.g.
```
spack install mpas-model make_target=pgi-nersc
```
`precision` is used to set the `PRECISION` flag in the Makefile to {single,
double}. Default is double.
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
* remove unhelpful comment
* Filter compiler duplicates while reading manifest
* more-specific version matching edited to use module-specific version (to avoid an issue where a user might add a compiler with the same version to the initial test configuration
* Set R_LIBS_USER='' in dependent build environment
Despite R packages being installed with the --vanilla flag, which
ignores Rprofile and Renviron files, R will still set R_LIBS_USER if the
default directory exists. This could lead to pulling in packages from
that directory during the build which could cause the build to fail. To
avoid that, explicitly set R_LIB_USER='' to ensure that packages from
the HOME/R directory are not in the library path.
* Update var/spack/repos/builtin/packages/r/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* PythonPackage: add default libs/headers attributes
* Style fix
* libs and headers should be properties
* Check both platlib and include
* Fix variable reference
Building on #24639, this allows versions to be prefixed by `git.`. If a version begins `git.`, it is treated as a git ref, and handled as git commits are starting in the referenced PR.
An exception is made for versions that are `git.develop`, `git.main`, `git.master`, `git.head`, or `git.trunk`. Those are assumed to be greater than all other versions, as those prefixed strings are in other contexts.
This commit adds some changes which improve use of Spack-installed
oneAPI packages with Spack-generated modules, but not within Spack
(e.g. if you install some of these packages with Spack, then load
their associated modules to build other packages outside of Spack).
The majority of the PR diff is refactoring. The functional changes
are:
* intel-oneapi-mkl:
* setup_run_environment: update Intel compiler flags to RPATH the
mkl libs
* intel-oneapi-compilers: update the compiler configuration to RPATH
libraries installed by this package (note that Spack already handled
this when installing dependent packages with Spack, but this is
specifically to use the intel-oneapi-compilers package outside
of Spack). Specifically:
* inject_rpaths: this modifies the binaries installed as part of
the intel-oneapi-compilers package to RPATH libraries provided
by this package
* extend_config_flags: this instructs the compiler executables
provided by this package to RPATH those same libraries
Refactoring includes:
* intel-oneapi-compilers: in addition to the functional changes,
a large portion of this diff is reorganizing the creation of
resources/archives downloaded for the install
* The base oneAPI package renamed component_path, so several packages
changed as a result:
* intel-oneapi-dpl
* intel-oneapi-dnn
* extrae
* intel-oneapi-mpi:
* Perform file filtering in one pass rather than two
Allow `spack external find` (with no extra args) to proceed if the manifest file exists but
without sufficient permissions; in that case, print a warning. Also add a test for that behavior.
TODOs:
- [x] continue past any exception raised during manifest parsing as part of `spack external find`,
except for CTRL-C (and other errors that indicate immediate program termination)
- [x] Semi-unrelated but came up when discussing this with the user who reported this issue to
me: the manifest parser now accepts older schemas
See: https://github.com/spack/spack/issues/31191
nvc doesn't know this flag: `-Wno-unknown-pragmas`, it doesn't hurt to
remove the specific warning flags, from:
```
-O3 -Wall -Wextra -Wno-unknown-pragmas -Wcast-qual
```
to:
```
-O3 -Wall
```
fixes#30997
Instead of giving a penalty of 30 to all nodes when preferences
are not package specific, give a penalty of 100 to all targets
of a node where we have package specific preferences, if the target
is not explicitly preferred.
* Add the rocm tag to packages from the Radeon Open Compute Platform.
* Added more packages from the ROCm software distribution based on
reviewer feedback.
* Updated tags based on reviewer feedback
* add first version of py-pyrr package
* added license
* update license HEADER
* Remove preferred as there is only one version available
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Better specify dependency type
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* remove useless variable license
Co-authored-by: Jérôme Dubois <jerome.dubois@cea.fr>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fixed a bug in the 'external find --all' command where the call failed
to find packages by both executable and library. The bug was that the
call `path.all_packages()` incorrectly turned the variable
`packages_to_check` into a generator rather than keeping it a list.
Thus the second call to `detection.by_library` had no work to do.
* Fixed the help message for the find external and compiler commands as
well as others that used the `scopes_metavar` field to define where
the results should be stored in configuration space. Specifically,
the fact that configuration could be added to the environment was not
mentioned in the help message.
os.path.dirname was being used to compare compilers. If two compilers
are in the same directory then it will pick up the first one it encounters.
Compare the full compiler path instead.
When installing setuptools from sources in Spack, we might
get into weird failures due to the way we use pip.
In particular, for Spack it's necessary to install in a
non-isolated pip environment to allow using PYTHONPATH as a
selection method for all the build requirements of a
Python package.
This can fail when installing setuptools since there might
be a setuptools version already installed for the Python
interpreter being used, with different entry points than
the one we want to install.
Installing from wheels both pip and setuptools should
harden our installation procedure in the context of:
- Bootstrapping Python dependencies of Spack
- Using external Python packages
On Cray systems that use Cray Data Virtualization Service (DVS),
symlinks across filesystems are not allowed, either due to a bug, or
because they're simply not POSIX compliant [1].
Spack's OpenSSL package defaults to `certs=system` which comes down to
symlinking `/etc/ssl` in the Spack install prefix, triggering this
problem, resulting in mysterious installation failures.
Instead of relying on system certs, we can just use
`ca-certificates-mozilla`, and if users really need system certs, then
they're probably better off marking OpenSSL entirely as external.
[1] https://github.com/glennklockwood/cray-dvs
* Created package and added description
* Add py-markdown-include
* Created package
* Finished creating package
* Added py-md-environ
* Added build dependencies
* Added other deps
* Add python-markdown-math (#4)
* Created package and started to add info
* Removed unneeded global/install options
* Figured out version spec for markdown-math
* Removed type=build from unnecessary dependencies
* Removed unneeded install/global options, added version spec to dependency
* Added wscullin as interim maintainer for packages
* Fixed style issues
* Took care of trailing whitespace
* Removed comment line before imports
* Changed file charset to match other packages
* Update var/spack/repos/builtin/packages/py-ford/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-ford/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Removed test dependency after review feedback
* Added new 6.1.12 version to py-ford
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add LevelZero variant to hwloc package
This permits the hwloc package to build with with support for the
Intel Level Zero low-level layer, analogous to CUDA, ROCm, and OpenCL.
* Fix typo
Add variant +gssapi to enable authentication via Kerberos through GSSAPI
When openssh is installed at sites using Kerberos, openssh needs to auth
via Kerberos through GSSAPI in order to work in such environments.
* Update h5bench maintainers and versions
* Include version 1.1 for h5bench
* Correct release hash and set default version
* Update .tar.gz version
* Update HDF5 VOL async version and environment variable syntax
* added package gptune with all its dependencies: adding py-autotune, pygmo, py-pyaml, py-autotune, py-gpy, py-lhsmdu, py-hpbandster, pagmo2, py-opentuner; modifying superlu-dist, py-scikit-optimize
* adding gptune package
* minor fix for macos spack test
* update patch for py-scikit-optimize; update test files for gptune
* fixing gptune package style error
* fixing unit tests
* a few changes reviewed in the PR
* improved gptune package.py with a few newly added/improved dependencies
* fixed a few style errors
* minor fix on package name py-pyro4
* fixing more style errors
* Update var/spack/repos/builtin/packages/py-scikit-optimize/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* resolved a few issues in the PR
* fixing file permissions
* a few minor changes
* style correction
* minor correction to jq package file
* Update var/spack/repos/builtin/packages/py-pyro4/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixing a few issues in the PR
* adding py-selectors34 required by py-pyro4
* improved the superlu-dist package
* improved the superlu-dist package
* moree changes to gptune and py-selectors34 based on the PR
* Update var/spack/repos/builtin/packages/py-selectors34/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* improved gptune package: 1. addressing comments of tldahlgren in PR 26936; 2. adding variant openmpi
* fixing style issue of gptune
* changing file mode
* improved gptune package: add variant mpispawn which depends on openmpi; add variant superlu and hypre for installing the drivers; modified hypre package file to add a gptune variant
* fixing style error
* corrected pddrive_spawn path in gptune test; enforcing gcc>7
* fixing style error
* setting environment variables when loading gptune
* removing debug print in hypre/package.py
* adding superlu-dist v7.2.0; fixing an issue with CMAKE_INSTALL_LIBDIR
* changing site_packages_dir to python_platlib
* not using python3.9 for py-gpy, which causes due to dropped support of tp_print
* more replacement of site_packages_dir
* fixing a few dependencies in gptune; added a gptune version
* adding url for gptune
* minor correction of gptune
* updating versions in butterflypack
* added a version for openturns
* added a url for openturns
* minor fix for openturns
* Update var/spack/repos/builtin/packages/openturns/package.py
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* added version2.1.1 for butterflypack
* fixing a tag in butterflypack
* minor fix for superlu-dist
* removed file filter for superlu-dist
* Update var/spack/repos/builtin/packages/superlu-dist/package.py
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* superlu-dist: add version detection for the testing; removing unused lines
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* add test to verify fix works
* fix spec cflags/variants parsing test (breaking change)
* fix `spack spec` arg quoting issue
* add error report for deprecated cflags coalescing
* use .group(n) vs subscript regex group extraction for 3.5 compat
* add random test for untested functionality to pass codecov
* fix new test failure since rebase
Fix a bug introduced in #21720. `spack_json.dump()` calls `_strify()` on dictionaries to
convert `unicode` to `str`, but it constructs `dict` objects instead of
`collections.OrderedDict` objects, so in Python 2 (or earlier versions of 3) it can
scramble dictionary order.
This can cause hashes to differ between Python 2 and Python 3, or between Python 3.7
and earlier Python 3's.
- [x] use `OrderedDict` in `_strify`
- [x] add a regression test
* nfft, pnfft: fix detection of fftw variants (precision)
* nfft, pnfft: use fftw's selected_precisions; avoid repetitive calls to spec
Co-authored-by: Martin Lang <martin.lang@mpsd.mpg.de>
The "submodules" argument of the "version" directive can now accept
a callable that returns a list of submodules, in addition to the usual
Boolean values
* Cancel running workflows automatically on PR update
* Add the last update later to check cancellation is working
* Use github.run_number instead of github.sha
* [py-sentry-sdk] audited dependencies and relisted extras as variants
* [py-sentry-sdk] added version 1.5.5
* [py-sentry-sdk] flake8
* [py-sentry-sdk]
- added "descriptions" to variants
- removed conflicts
- replaced with when statements in varients
* bootstrap: account for disabled sources
Fix a bug introduced in #30192, which effectively skips
any prescription on disabled bootstrapping sources.
* Add unit test to avoid regression
Fixes#31021
With #25185, we no longer default to using tar when we can't
determine the extension type, opting to fail instead.
This broke fetching for the pcre package, where we couldn't determine
the extension. To determine the extension, we were attempting to
extract it from the destination filename; however, this file name
may omit details of the origin URL that are required to determine the
extension, so instead we examine the URL directly.
This also updates the decompressor_for method not to set ext=None
by default: it must now always be set by the caller.
* Update prmon recipe, v3.0.2
Update recipe to v3.0.2, using external dependency
option for the build (as these are satisfied so easily with Spack)
* Remove "broken" 3.0.1 version
This one could not build properly with Spack, due to
missing submodule sources
Installation process of libint runs a Python script:
<69cc7b9bc6/export/cmake/CMakeLists.txt.export (L410)>.
If Python isn't explicitly listed as build dependency, system Python will be
picked up, which can cause troubles.
* Fix py-ray dependencies and build system
* Update var/spack/repos/builtin/packages/py-ray/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Added documentation for unusual dependencies.
* Fixed npm preinstall script per @adamjstewart suggestion.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Most package installations include compressed source files. This
adds support for common archive types on Windows:
* Add support for using system 7zip functionality to decompress .Z
files when available (and on Windows, use 7zip for .xz archives)
* Default to using built-in Python support for tar/bz2 decompression
(note that Python tar documentation mentions preservation of file
permissions)
* Add tests for decompression support
* Extract logic for handling exploding archives (i.e. compressed
archives that expand to more than one base file) into an
exploding_archive_catch context manager in the filesystem module
* [py-httpx] python dependencies are type=('build', 'run')
* [py-httpx] py-wheel is now implied by PythonPackage
* [py-httpx] fixed older version dependencies
* [py-httpx] added version 0.22.0
Spack's staging logic constructs a file path based on a URL. The URL
may contain characters which are not allowed in valid file paths on
the system (e.g. Windows prohibits ':' and '?' among others). This
commit adds a function to remove such offending characters (but
otherwise preserves the URL string when constructing a file path).
* add pacakge: abacus
* rename
* fix some style bugs
* update package:abacus
* fix some style bugs
* format code
* Delete a line of useless comments
* updatee abacus
* Update package.py
add new version info and fix a bug on mkl
* update version info and fix a bug on mkl
* Update package.py
* fix style bugs
* trailing whitespace
Anticipate openPMD-api changes in the next major release that are
already in `dev` (aka Spack `develop`):
- C++17 requirement
- drop: `mpark-variant` public dependency
- add: `toml11` private dependency
Also add @franzpoeschel as co-maintainer for the Spack package.
Updates to improve Spack-generated modules for Intel oneAPI compilers:
* intel-oneapi-compilers set CC etc.
* Add a new package intel-oneapi-compilers-classic which can be used to
generate a module which sets CC etc. to older compilers (e.g. icc)
* lmod module logic now updated to treat the intel-oneapi-compilers*
packages as compilers
* acts-dd4hep: new package, separated from new acts@19.1.0
* acts-dd4hep: improved versioning
* acts-dd4hep: don't use curl | sha256sum
* acts: new variant `odd` for Open Data Detector
* acts-dd4hep: style changes
Add spack stacks targeted at Spack + AWS + ARM HPC User Group hackathon. Includes
a list of miniapps and full-apps that are ready to run on both x86_64 and aarch64.
Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
Add two new stacks targeted at x86_64 and arm, representing an initial list of packages
used by current and planned AWS Workshops, and built in conjunction with the ISC22
announcement of the spack public binary cache.
Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
Explicitly import package utilities in all packages, and corresponding fallout.
This includes:
* rename `spack.package` to `spack.package_base`
* rename `spack.pkgkit` to `spack.package`
* update all packages in builtin, builtin_mock and tutorials to include `from spack.package import *`
* update spack style
* ensure packages include the import
* automatically add the new import and remove any/all imports of `spack` and `spack.pkgkit`
from packages when using `--fix`
* add support for type-checking packages with mypy when SPACK_MYPY_CHECK_PACKAGES
is set in the environment
* fix all type checking errors in packages in spack upstream
* update spack create to include the new imports
* update spack repo to inject the new import, injection persists to allow for a deprecation period
Original message below:
As requested @adamjstewart, update all packages to use pkgkit. I ended up using isort to do this,
so repro is easy:
```console
$ isort -a 'from spack.pkgkit import *' --rm 'spack' ./var/spack/repos/builtin/packages/*/package.py
$ spack style --fix
```
There were several line spacing fixups caused either by space manipulation in isort or by packages
that haven't been touched since we added requirements, but there are no functional changes in here.
* [x] add config to isort to make sure this is maintained going forward
referred targets are currently the only minimization criteria for Spack for which we allow
negative values. That means Spack may be incentivized to add nodes to the DAG if they
match the preferred target.
This PR re-norms the minimization criteria so that preferred targets are weighted from 0,
and default target weights are offset by the number of preferred targets per-package to
calculate node_target_weight.
Also fixes a bug in the test for preferred targets that was making the test easier to pass
than it should be.
* Call Numpy package's set_blas_lapack() and setup_build_environment() in Scipy package
* Remove broken link from comment
* Use .package attribute of spec to avoid import
This PR fixes several issues I noticed while trying to get Spack working on Apple M1.
- [x] `build_environment.py` attempts to add `spec['foo'].libs` and `spec['foo'].headers` to our compiler wrappers for all dependencies using a try-except that ignores `NoLibrariesError` and `NoHeadersError` respectively. However, The `libs` and `headers` attributes of the Python package were erroneously using `RuntimeError` instead.
- [x] `spack external find python` (used during bootstrapping) currently has no way to determine whether or not an installation is `+shared`, so previously we would only search for static Python libs. However, most distributions including XCode/Conda/Intel ship shared Python libs. I updated `libs` to search for both shared and static (order based on variant) as a fallback.
- [x] The `headers` attribute was recursively searching in `prefix.include` for `pyconfig.h`, but this could lead to non-deterministic behavior if multiple versions of Python are installed and `pyconfig.h` files exist in multiple `<prefix>/include/pythonX.Y` locations. It's safer to search in `sysconfig.get_path('include')` instead.
- [x] The Python installation that comes with XCode is broken, and `sysconfig.get_paths` is hard-coded to return specific directories. This meant that our logic for `platlib`/`purelib`/`include` where we replace `platbase`/`base`/`installed_base` with `prefix` wasn't working and the `mkdirp` in `setup_dependent_package` was trying to create a directory in root, giving permissions issues. Even if you commented out those `mkdirp` calls, Spack would add the wrong directories to `PYTHONPATH`. Added a fallback hard-coded to `lib/pythonX.Y/site-packages` if sysconfig is broken (this is what distutils always did).
This PR supports the creation of securely signed binaries built from spack
develop as well as release branches and tags. Specifically:
- remove internal pr mirror url generation logic in favor of buildcache destination
on command line
- with a single mirror url specified in the spack.yaml, this makes it clearer where
binaries from various pipelines are pushed
- designate some tags as reserved: ['public', 'protected', 'notary']
- these tags are stripped from all jobs by default and provisioned internally
based on pipeline type
- update gitlab ci yaml to include pipelines on more protected branches than just
develop (so include releases and tags)
- binaries from all protected pipelines are pushed into mirrors including the
branch name so releases, tags, and develop binaries are kept separate
- update rebuild jobs running on protected pipelines to run on special runners
provisioned with an intermediate signing key
- protected rebuild jobs no longer use "SPACK_SIGNING_KEY" env var to
obtain signing key (in fact, final signing key is nowhere available to rebuild jobs)
- these intermediate signatures are verified at the end of each pipeline by a new
signing job to ensure binaries were produced by a protected pipeline
- optionallly schedule a signing/notary job at the end of the pipeline to sign all
packges in the mirror
- add signing-job-attributes to gitlab-ci section of spack environment to allow
configuration
- signing job runs on special runner (separate from protected rebuild runners)
provisioned with public intermediate key and secret signing key
Old concrete specs were slipping through in `_assign_hash`, and `package_hash` was
attempting to recompute a package hash when we could not know the package a time
of concretization.
Part of this was that the logic for `_assign_hash` was hard to understand -- it was
called twice from `_finalize_concretization` and had special cases for both args it
was called with. It's much easier to understand the logic here if we just inline it.
- [x] Get rid of `_assign_hash` and just integrate it with `_finalize_concretization`
- [x] Don't call `_package_hash` at all for already-concrete specs.
- [x] Add regression test.
Use `spack build` as build dir to avoid recursive link error.
```
config.status: linking /var/folders/fy/x2xtwh1n7fn0_0q2kk29xkv9vvmbqb/T/s3j/spack-stage/spack-stage-sed-4.8-wraqsot6ofzvr3vrgusx4mj4mya5xfux/spack-src/GNUmakefile to GNUmakefile
config.status: executing depfiles commands
config.status: executing po-directories commands
config.status: creating po/POTFILES
config.status: creating po/Makefile
==> sed: Executing phase: 'build'
==> [2022-05-25-14:15:51.310333] 'make' '-j8' 'V=1'
make: GNUmakefile: Too many levels of symbolic links
make: stat: GNUmakefile: Too many levels of symbolic links
make: *** No rule to make target `GNUmakefile'. Stop.
```
This PR introduces a new build cache layout and package format, with improvements for
both efficiency and security.
## Old Format
Currently a binary package consists of a `spec.json` file at the root and a `.spack` file,
which is a `tar` archive containing a copy of the `spec.json` format, possibly a detached
signature (`.asc`) file, and a tar-gzip compressed archive containing the install tree.
```
build_cache/
# metadata (for indexing)
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json
<arch>/
<compiler>/
<name>-<ver>/
# tar archive
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spack
# tar archive contents:
# metadata (contains sha256 of internal .tar.gz)
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json
# signature
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json.asc
# tar.gz-compressed prefix
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.tar.gz
```
After this change, the nesting has been removed so that the `.spack` file is the
compressed archive of the install tree. Now signed binary packages, will take the
form of a clearsigned `spec.json` file (a `spec.json.sig`) at the root, while unsigned
binary packages will contain a `spec.json` at the root.
## New Format
```
build_cache/
# metadata (for indexing, contains sha256 of .spack file)
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json
# clearsigned spec.json metadata
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json.sig
<arch>/
<compiler>/
<name>-<ver>/
# tar.gz-compressed prefix (may support more compression formats later)
<arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spack
```
## Benefits
The major benefit of this change is that the signatures on binary packages can be
verified without:
1. Having to download the tarball, or
2. having to extract an unknown tarball.
(1) is an improvement in efficiency; (2) is a security fix: we now ensure that we trust the
binary before we try to run it through `tar`, which avoids potential attacks.
## Backward compatibility
Also after this change, spack should still be able to handle the previous buildcache
structure and binary mirrors with mixed layouts.
This PR builds on #28392 by adding a convenience command to create a local mirror that can be used to bootstrap Spack. This is to overcome the inconvenience in setting up this mirror manually, which has been reported when trying to setup Spack on air-gapped systems.
Using this PR the user can create a bootstrapping mirror, on a machine with internet access, by:
% spack bootstrap mirror --binary-packages /opt/bootstrap
==> Adding "clingo-bootstrap@spack+python %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding "gnupg@2.3: %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding "patchelf@0.13.1:0.13.99 %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding binary packages from "https://github.com/alalazo/spack-bootstrap-mirrors/releases/download/v0.1-rc.2/bootstrap-buildcache.tar.gz" to the mirror at /opt/bootstrap/local-mirror
To register the mirror on the platform where it's supposed to be used run the following command(s):
% spack bootstrap add --trust local-sources /opt/bootstrap/metadata/sources
% spack bootstrap add --trust local-binaries /opt/bootstrap/metadata/binaries
The mirror has to be moved over to the air-gapped system, and registered using the commands shown at prompt. The command has options to:
1. Add pre-built binaries downloaded from Github (default is not to add them)
2. Add development dependencies for Spack (currently the Python packages needed to use spack style)
* bootstrap: refactor bootstrap.yaml to move sources metadata out
* bootstrap: allow adding/removing custom bootstrapping sources
This operation can be performed from the command line since
new subcommands have been added to `spack bootstrap`
* Add --trust argument to spack bootstrap add
* Add a command to generate a local mirror for bootstrapping
* Add a unit test for mirror creation
* Allow Kokkos with OpenMPTarget backend
* Restrict SYCL and OpenMPTarget to C++17 or higher
* Improve C++ standard check for SYCL and OpenMPTarget
* Fix indentation
Currently, environments can either be concretized fully together or fully separately. This works well for users who create environments for interoperable software and can use `concretizer:unify:true`. It does not allow environments with conflicting software to be concretized for maximal interoperability.
The primary use-case for this is facilities providing system software. Facilities provide multiple MPI implementations, but all software built against a given MPI ought to be interoperable.
This PR adds a concretization option `concretizer:unify:when_possible`. When this option is used, Spack will concretize specs in the environment separately, but will optimize for minimal differences in overlapping packages.
* Add a level of indirection to root specs
This commit introduce the "literal" atom, which comes with
a few different "arities". The unary "literal" contains an
integer that id the ID of a spec literal. Other "literals"
contain information on the requests made by literal ID. For
instance zlib@1.2.11 generates the following facts:
literal(0,"root","zlib").
literal(0,"node","zlib").
literal(0,"node_version_satisfies","zlib","1.2.11").
This should help with solving large environments "together
where possible" since later literals can be now solved
together in batches.
* Add a mechanism to relax the number of literals being solved
* Modify spack solve to display the new criteria
Since the new criteria is above all the build criteria,
we need to modify the way we display the output.
Originally done by Greg in #27964 and cherry-picked
to this branch by the co-author of the commit.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* Inject reusable specs into the solve
Instead of coupling the PyclingoDriver() object with
spack.config, inject the concrete specs that can be
reused.
A method level function takes care of reading from
the store and the buildcache.
* spack solve: show output of multi-rounds
* add tests for best-effort coconcretization
* Enforce having at least a literal being solved
Co-authored-by: Greg Becker <becker33@llnl.gov>
* Py-x21 now works, needs dependencies
Conflicts:
var/spack/repos/rit-rc/packages/py-x21/package.py
* Added dependencies to py-x21
* Making flake style check happy
* [py-x21] flake8
* [py-x21]
- added homepage
- added placeholder description
- added comment about checksums
* [py-x21] added darwin support and fixed issue with python 3.7 wheel name
* [py-x21] adding checksum hash
* [py-x21] removed duplicate py-pynacl
* [py-x21]
- updated description
- updated version listing to have a different version for each version
of python. Also, versions dependent on sys.platform
- updated url_for_version to not require post concretized information so
that spack checksum works
* [py-x21] isort
Co-authored-by: vehrc <vehrc@rit.edu>
Previously the regex was only checking for presence of quotes as a beginning
or end character and not a matching set. This erroneously identified the
following *single* argument as being quoted:
source bashenvfile &> /dev/null && python3 -c "import os, json; print(json.dumps(dict(os.environ)))"
rocm-5.1.0 removed librocrand.so from ROCM_DIR/rocrand/lib location (but includes are still at this location)
/opt/rocm-5.0.2/lib/librocrand.so
/opt/rocm-5.0.2/rocrand/lib/librocrand.so
/opt/rocm-5.1.0/lib/librocrand.so
drwxr-xr-x 2 root root 617 Mar 8 08:20 /opt/rocm-5.0.2/rocrand/include
drwxr-xr-x 2 root root 617 Mar 31 09:48 /opt/rocm-5.1.0/rocrand/include
Add a config option to strip `-Werror*` or `-Werror=*` from compile lines everywhere.
```yaml
config:
keep_werror: false
```
By default, we strip all `-Werror` arguments out of compile lines, to avoid unwanted
failures when upgrading compilers. You can re-enable `-Werror` in your builds if
you really want to, with either:
```yaml
config:
keep_werror: all
```
or to keep *just* specific `-Werror=XXX` args:
```yaml
config:
keep_werror: specific
```
This should make swapping in newer versions of compilers much smoother when
maintainers have decided to enable `-Werror` by default.
Parse error information is kept for specs, but it doesn't seem like we propagate it
to the user when we encounter an error. This fixes that.
e.g., for this error in a package:
```python
depends_on("python@:3.8", when="0.900:")
```
Before, with no context and no clue that it's even from a particular spec:
```
==> Error: Unexpected token: ':'
```
With this PR:
```
==> Error: Unexpected token: ':'
Encountered when parsing spec:
0.900:
^
```
* Added autotools configure flags to ensure that hwloc finds the correct
version of CUDA that it was concretized against, rather than the first
one that package config finds.
* Added support for finding the correct version of ROCm libraries. Fixed Flake8.
* Fixed guard on finding ROCm library
* [py-h2] py-wheel is implied by PythonPackage
* [py-h2] python dependencies should be type=('build', 'run')
* [py-h2] fixed dependencies for py-h2@4.0.0
* [py-h2] added version 3.2.0
* [py-h2] added version 4.1.0
* [py-h2] Older version requires py-enum34 for older versions of python
Add two new cloud pipelines for E4S on Amazon Linux, include arm and x86 (v3 + v4) stacks.
Notes:
- Updated mpark-variant to remove conflict that no longer exists in Amazon Linux
- Which command on Amazon Linux prefixes on all results when padded_length is too high. In this case, padded_length<=503 works as expected. Chose conservative length of 384.
* Introduce concretizer:unify option to replace spack:concretization
* Deprecate concretization
* Make spack:concretization overrule concretize:unify for now
* Add environment update logic to move from spack:concretization to spack:concretizer:reuse
* Migrate spack:concretization to spack:concretize:unify in all locations
* For new environments make concretizer:unify explicit, so that defaults can be changed in 0.19
* Update h5bench maintainers and versions
* Include version 1.1 for h5bench
* Correct release hash and set default version
* Update .tar.gz version
* Include new version and update runtime
* Update year
* Update package.py
* Update package.py
fixes#30700
To avoid clingo adding penalties for not using the
default value for a variant, it's better to model
the variant as conditional where possible.
* This commit removes the Boost.with_default_variants to variants that packages are precisely dependant upon. This is the third batch of 16 packages with modified boost dependencies.
* style fix
* Update var/spack/repos/builtin/packages/sympol/package.py
Co-authored-by: Tim Haines <thaines.astro@gmail.com>
* fix style
* Apply suggestions from code review
Co-authored-by: Tim Haines <thaines.astro@gmail.com>
* Fix Trilinos boost deps
* Fix style
Co-authored-by: Tim Haines <thaines.astro@gmail.com>
Co-authored-by: Tom Scogland <tom.scogland@gmail.com>
Add a `build_type` variant, which allows building optimized compilers,
as well as target libraries (libstdc++ and friends).
The default is `build_type=RelWithDebInfo`, which corresponds to GCC's
default of -O2 -g.
When building with `+bootstrap %gcc`, also add Spack's arch specific
flags using the common denominator between host and new GCC.
It is done by creating a config/spack.mk file in def patch, that looks
as follows:
```
BOOT_CFLAGS := $(filter-out -O% -g%, $(BOOT_CFLAGS)) -O2 -g -march=znver2 -mtune=znver2
CFLAGS_FOR_TARGET := $(filter-out -O% -g%, $(CFLAGS_FOR_TARGET)) -O2 -g -march=znver2 -mtune=znver2
CXXFLAGS_FOR_TARGET := $(filter-out -O% -g%, $(CXXFLAGS_FOR_TARGET)) -O2 -g -march=znver2 -mtune=znver2
```
The oneapi and dpcpp compilers are essentially the same except for which
binary is used foc CXX. Spack will detect them as "mixed toolchain" and
not inject compiler optimization flags. This will be needed once
archspec has entries for the oneapi and dpcpp compilers. This PR detects
when dpcpp and oneapi are in the toolchains list and explicitly sets
`is_mixed_toolchain` to `False`.
* [py-openslide-python] added verion 1.1.2 and set max py-setuptools version for 1.1.1
* [py-openslide-python]
- setuptools required for all possible newer versions
- python is type build run
* [py-openslide-python] use pil provider
* Add version 3.0 and 3.1 and prelim OpenMP support
* Fix flag handler missing spec variable
* Use self.compiler.openmp_flag instead of -fopenmp
* Fix whitespace
Fixes qt configure errors with external openssl on older systems (rhel7)
See
efc02f9cc3/dist/changes-5.15.0 (L346)
This means for now on, `qt ^openssl@1.0` gets you `qt@5.15.4 ~ssl`:
clingo chooses latest qt version **but disables ssl support**.
Error messages for the clingo concretizer have proven challenging. The current messages are incredibly vague and often don't help users at all. Unsat cores in clingo are not guaranteed to be minimal, and lead to cores that are either not useful or need to be post-processed for hours to reach a minimal core.
Following up on an idea from a slack conversation with kwryankrattiger on slack, this PR takes a new approach. We eliminate most integrity constraints and minima/maxima on choice rules in clingo, and instead force invalid states to imply an error predicate. The error predicate can include context on the cause of the error (Package, Version, etc). These error predicates are then heavily optimized against, to ensure that we do not include error facts in the solution when a solution with no error facts could be generated. When post-processing the clingo solution to construct specs, any error facts cause the program to raise an exception. This leads to much more legible error messages. Each error predicate includes a priority and an error message. The error message is formatted by the remaining arguments to produce the error message. The priority is used to ensure that when clingo has a choice of which rules to violate, it chooses the one which will be most informative to the user.
Performance:
"fresh" concretizations appear to suffer a ~20% performance penalty under this branch, while "reuse" concretizations see a speedup of around 33%.
Possible optimizations if users still see unhelpful messages:
There are currently 3 levels of priority of the error messages. Additional priorities are possible, and can allow us finer granularity to ensure more informative error messages are provided in lieu of less informative ones.
Future work:
Improve tests to ensure that every possible rule implying an error message is exercised
A non-existent upstream should not be fatal: it could only mean it is
not deployed yet. In the meantime, it should not block the user to
rebuild anything it needs.
A warning is still emitted, to let the user decide if this is ok or not.
* Fix for xtensor-xsimd
* Add sha256 for all new releases
* renamed ufcx package
* Update sha for ffcx
* fixed hashes and modified fenics-dolfinx to depend on ufcx
* cleaned and fixed dependency types
* use spec.satisfies in cmake_args
* bumped to ufcx@0.4.1
* address PR comments
* fix hashes
* update parmetis in cmake_args to reflect default setting
* update versions
* renamed ufcx package
* fixed hashes and modified fenics-dolfinx to depend on ufcx
* cleaned and fixed dependency types
* use spec.satisfies in cmake_args
* bumped to ufcx@0.4.1
* address PR comments
* fix hashes
* update parmetis in cmake_args to reflect default setting
* update versions
* Add dependency fix
* bump basix to 0.4.2 and address PR comments
* Versioning fixes
* Use xtensor-0.24: and loosen pybind11
* Add conflicts for partitioners
* Updates on partitioners
* use define_from_variant
* Tidy up some dependencies
* Work on multi-variants for graph partitioners
* Fix KaHIP issue.
KaHIP changed the name of its library from 'interface' to 'kahip'. Pin earlier versions of DOLFINx to earlier verisons of KaHIP for proper detection.
Co-authored-by: Chris Richardson <chris@bpi.cam.ac.uk>
Co-authored-by: Garth N. Wells <gnw20@cam.ac.uk>
Fixes missing chgrp on symlinks in package installations, and errors on
symlinks referencing non-existent or non-writable locations.
Note: `os.chown(.., follow_symlinks=False)` is python3 only, but
`os.lchown` exists in both versions.
* Change license dir from hard-coded to a configurable item
* Change config item to be a string not an array
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Trying to compute `dag_hash()` or `package_hash()` on a concrete spec that doesn't have
a `_package_hash` attribute would attempt to recompute the package hash.
This most commonly manifests as a failed lookup of a namespace if you attempt to uninstall
or compute the hashes of packages in exsternal repositories that aren't registered, e.g.:
```console
> spack spec --json c/htno
==> Error: Unknown namespace: myrepo
```
While it wouldn't change the already-assigned `dag_hash` value, this behavior is
incorrect, since the package file for a previously concrete spec:
1. might have changed since concretization,
2. might not exist anymore, or
3. might just not be findable by Spack.
This PR ensures that the package hash can't be computed on older concrete specs. Instead
of calling `package_hash()` from within `to_node_dict()`, we now check for the `_package_hash`
attribute and only add the package_hash to the spec record if it's there.
This PR also handles the tricky semantics of computing `package_hash()` at concretization
time. We have to compute it *before* marking the spec concrete so that `to_node_dict` can
use it. But this means that the logic for `package_hash()` can't rely on `spec.concrete`,
as it is called *during* concretization. Instead of checking for concreteness, `package_hash()`
now checks `_patches_assigned()` to determine whether it should add them to the package
hash.
- [x] Add an assert to `package_hash()` so it can't be called on specs for which it
would be wrong.
- [x] Add an `_assign_hash()` method to handle tricky semantics of `package_hash`
and `dag_hash`.
- [x] Rework concretization to call `_assign_hash()` before and after marking specs
concrete.
- [x] Rework content hash part of package hash to check for `_patches_assigned()`
instead of `spec.concrete`.
- [x] regression test
* [py-tensorflow-hub] applied patch for newer version of zlib
* [py-tensorflow-hub] patch also applies to 0.11.0
* [py-tensorflow-hub] Audit fix
1. patch URL in package py-tensorflow-hub must end with ?full_index=1
Newer versions of gobject-introspection require Meson to build. Convert
the package into a hybrid one that still supports older versions using
Autotools.
* arm-forge: Download via HTTPS
Update download URL to use HTTPS (rather than HTTP)
* arm-forge: Allow +probe to depend on python3
Allow python dependency required for arm-forge+probe to be python3 as
well as 2.7.x
* arm-forge: Add versions up to 22.0.1
By default, libfuse install helper programs like `fusermount3`, which
are mostly useless if not installed with setuid (that is, `+useroot`).
However, their presence makes it complicated to use globally installed
versions, which can be combined with a Spack-installed FUSE library.
In particular, on systems that have a setuid fusermount3 binary, but no
libfuse-dev installed, it is nice to be able to build libfuse with Spack, and
have it call the system setuid executable.
* Correcting include and library paths using patch file for RVS to build
following library files in spack.
libperf.so.0.0
libpebb.so.0.0
libiet.so.0.0
libgst.so.0.0
libpqt.so.0.0
libmem.so.0.0
libbabel.so.0.0
* Correcting include and library paths using patch file for RVS to build
following library files in spack.
libperf.so.0.0
libpebb.so.0.0
libiet.so.0.0
libgst.so.0.0
libpqt.so.0.0
libmem.so.0.0
libbabel.so.0.0
* Replacing ROCM_PATH with RPATH in the deviceid.sh before installing in Spack build.
* Reducing multiple enviroment variable for HIP and HSA path
- Removed gl dependency.
- Specify clang as cmake compiler as gcc was being
improperly picked up. As a result, ffi include
path was needed in C/CXX flags.
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Previously we sorted by hash values for `spack graph`, but changing hashes can make the
test brittle and the node order seem nondeterministic to users.
- [x] Sort nodes in `spack graph` by the default edge order, which takes into account
parent and child names as well as dependency types.
- [x] Update ASCII test output for new order.
The dependency check currently checks whether there are only build
dependencies left for a particular package. However, the database also
contains uninstalled packages, which can cause the check to fail.
For instance, with `bison` and `flex` having already been uninstalled,
`m4` will have the following dependents:
```
bison ('build', 'run')--> m4
flex ('build',)--> m4
libnl ('build',)--> m4
```
`bison` and `flex` should be ignored in this case because they are not
installed anymore.
Fixes#30673
* ceed50: add ceed 5.0.0 and pumi 2.2.7
* libceed-0.10
* ceed50: add omegah
* omega-h: mpi and cuda builds work
* omega-h: fix style
* New package: libfms
* New version: gslib@1.0.7
CEED: add some TODO items for the 5.0 release
* ceed: variant name consistent with package name
* LAGHOS: allow newer versions of MFEM to be used with v3.1
* LIBCEED: add missing 'install' target in 'install_targets'
* CEED: address some TODO items + some tweaks
* MFEM: add new variant for FMS (libfms)
* CEED: v5.0.0 depends on 'libfms' and 'mfem+fms'
* RATEL: add missing 'install' target in 'install_targets'
* CEED: add dependency for v5.0.0 on Ratel v0.1.2
* CEED: add Nek-related dependencies for ceed@5.0.0
* CEED: v5.0.0 depends on MAGMA v2.6.2
* libCEED: set the `CUDA_ARCH` makefile parameter
* libCEED: set the `HIP_ARCH` makefile parameter
Co-authored-by: Jed Brown <jed@jedbrown.org>
Co-authored-by: Veselin Dobrev <dobrev@llnl.gov>
Co-authored-by: Veselin Dobrev <v-dobrev@users.noreply.github.com>
#24556 merged in support for Python's .zip file support via ZipFile.
However as per #30200 ZipFile does not preserve file permissions of
the extracted contents. This PR returns to using the `unzip`
executable on non-Windows systems (as was the case before #24556)
and now uses `tar` on Windows to extract .zip files.
We previously had checks in `directory_layout` to check for build-dependency
conflicts when we weren't storing build dependencies. We don't need
those anymore; we can just rely on the DAG hash now that it includes everything
we know about each spec.
- [x] Remove vestigial code for checking installed spec against concrete spec
in `ensure_installed()`
- [x] Remove `SpecHashCollisionError` -- if specs have the same hash now, they're
the same as far as `DirectoryLayout` should be concerned.
- [x] Convert spec comparison to `dag_hash()` comparison when adding extensions.
The database now stores full hashes, so we need to adjust the criteria we use to
determine if something can be uninstalled. Specifically, it's ok to uninstall thing that
have remaining build-only dependents.
With the original DAG hash, we did not store build dependencies in the database, but
with the full DAG hash, we do. Previously, we'd never tell the concretizer about build
dependencies of things used by hash, because we never had them. Now, we have to avoid
telling the concretizer about them, or they'll unnecessarily constrain build
dependencies for new concretizations.
- [x] Make database track all dependencies included in the `dag_hash`
- [x] Modify spec_clauses so that build dependency information is optional
and off by default.
- [x] `spack diff` asks `spec_clauses` for build dependencies for completeness
- [x] Modify `concretize.lp` so that reuse optimization doesn't affect fresh
installations.
- [x] Modify concretizer setup so that it does *not* prioritize installed versions
over package versions. We don't need this with reuse, so they're low priority.
- [x] Fix `test_installed_deps` for full hash and new concretizer (does not work
for old concretizer with full hash -- leave this for later if we need it)
- [x] Move `test_installed_deps` mock packages to `builtin.mock` for easier debugging
with `spack -m`.
- [x] Fix `test_reuse_installed_packages_when_package_def_changes` for full hash
- [x] update test to use `build_hash` instead of `dag_hash`, as we're testing for
graph structure, and specifically NOT testing for package changes.
- [x] make hash descriptors callable on specs to simplify syntax for invoking them
- [x] make `Spec.spec_hash()` public
This removes all but one usage of runtime hash. The runtime hash was being used to write
historical lockfiles for tests, but we don't need it for that; we can just save those
lockfiles.
- [x] add legacy lockfiles for v1, v2, v3
- [x] fix bugs with v1 lockfile tests (the dummy lockfile we were writing was not actually
a v1 lockfile because it used the new spec file format).
- [x] remove all but one runtime_hash usage -- that one needs a small rework of the
concretizer to really fix, as it's about separate concretization of build
dependencies.
- [x] Document the history of the lockfile format in `environment/__init__.py`
Some test cases had to be modified in a kludgy way so that abstract specs made
concrete would have versions on them. We shouldn't *need* to do this, as the
only reason we care is because the content hash has to be able to get an archive
for a version.
This modifies the content hash so that it can be called on abstract specs,
including only relevant content.
This does NOT add a partial content hash to the DAG hash, as we do not really
want that -- we don't need in-memory spec hashes to need to load package files.
It just makes `Package.content_hash()` less prickly and tests easier to
understand.
`spack monitor` expects a field called `spec_full_hash`, so we shouldn't change that.
Instead, we can pass a `dag_hash` (which is now the full hash) but not change the field
name.
`hashes_final` was used to indicate when a spec was concrete but possibly lacked
`full_hash` or `build_hash` fields. This was only necessary because older Spacks
didn't generate them, and we want to avoid recomputing them, as we likely do not
have the same package files as existed at concretization time.
Now, we don't need to do that -- there is only the DAG hash and specs are either
concrete and have a `dag_hash`, or not concrete and have no `dag_hash`. There's
no middle ground.
Without some enforcement of spec ordering, python 2 produced
different results in the affected test than did python 3. This
change makes the arbitrary but reproducible decision to sort
the specs by their lockfile key alphabetically.
The full hash appears twice in the spec dict now, replacing just
the value replaces it under "hash" and "full_hash". Only replace
the one that appears after "full_hash".
I'm actually not sure what purpose this test served, so maybe it
could be removed, as it may be testing some distinction between
full and dag hash which no longer exists.
For a long time, Spack has used a coarser hash to identify packages
than it likely should. Packages are identified by `dag_hash()`, which
includes only link and run dependencies. Build dependencies are
stripped before hashing, and we have notincluded hashes of build
artifacts or the `package.py` files used to build. This means the
DAG hash actually doesn't represent all the things Spack can build,
and it reduces reproducibility.
We did this because, in the early days, users were (rightly) annoyed
when a new version of CMake, autotools, or some other build dependency
would necessitate a rebuild of their entire stack. Coarsening the hash
avoided this issue and enabled a modicum of stability when only reusing
packages by hash match.
Now that we have `--reuse`, we don't need to be so careful. Users can
avoid unnecessary rebuilds much more easily, and we can add more
provenance to the spec without worrying that frequent hash changes
will cause too many rebuilds.
This commit starts the refactor with the following major change:
- [x] Make `Spec.dag_hash()` include build, run, and link
dependencides and the package hash (it is now equivalent to
`full_hash()`).
It also adds a couple of bugfixes for problems discovered during
the switch:
- [x] Don't add a `package_hash()` in `to_node_dict()` unless
the spec is concrete (fixes breaks on abstract specs)
- [x] Don't add source ids to the package hash for packages without
a known fetch strategy (may mock packages are like this)
- [x] Change how `Spec.patches` is memoized. Using
`llnl.util.lang.memoized` on `Spec` objects causes specs to
be stored in a `dict`, which means they need a hash. But,
`dag_hash()` now includes patch `sha256`'s via the package
hash, which can lead to infinite recursion
For tutorial builds, we should continue to allow deprecated builds to be installed. We
can update them as needed when we update the tutorial, but we don't need to correct them
immediately on deprecation in CI.
- [x] add `deprecated:true` to tutorial `spack.yaml` config.
* updating googletest version to 1.11 to avoid GTEST_DISALLOW_ASSIGN_ error
* limiting the version scope
* modified the version limit
Co-authored-by: mohan babu <mohbabul@amd.com>
Upstream neovim builds with luajit-openresty or luajit in almost all
cases. To support the current usage, a user can specify that they want
lua, but this will allow the use of the normal (faster, better tested
and better maintained) setup.
* Add checksum for py-pylint@2.13.5
* Update dependencies
* Add checksum for py-astroid@2.11.4
* Correct py-toml addition and add py-tomli dependency
* Remove py-pytoml dependency for versions @2.13:
* Modify py-astroid version range
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Discontinue py-astroid dependency @2.8.0:2.8 for new versions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Discontinue py-mccabe dependency @0.6.0:0.6 for new versions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Remove mccabe and setuptools-scm dependencies
* Update astroid dependencies
* Extend py-typed-ast version range to future releases
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-dill only required for version 2.13.5 and above
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add maccabe dependency and correct setuptools run dependency
* Setuptools fix
* Add setuptools as run dependency
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-pyarrow: Add version 7.0.0
* Add version constraints on dependencies
* Add version 8.0.0
* arrow: Add version 8.0.0
* py-pyarrow: Allow version 8.0.0 of arrow
* Bump up rocm release version to rocm-5.1.0
* update rocm-opencl for rocm-5.1.0 release
* update the migraphx,miopen(hip,opencl),mivisionx,rocm-tensile
* update the mlirmiopen checksum version
- [x] Add `mkdir -p` and `chmod` to ensure `/home/spack-test` exists and
has correct permissions.
- [x] Remove version comments from dependabot-managed action commits
- [x] Don't duplicate comment describing required fixes for distros with
patched git
`spack pkg list` tests were broken by #29593 for cases when your `builtin.mock` repo
still has stale backup files (or, really, stale directories) sitting around. This
happens if you switch branches a lot. In this case, things like this were causing
erroneous packages in the mock listing:
```
var/spack/repos/builtin.mock/packages/
foo/
package.py~
```
- [x] make `list_packages` consider only directories with one-deep `package.py` files.
Reworking lua to allow easier substitution of the base lua implementation.
Also adding in a maintained version of luajit and re-factoring the entire stack
to use a custom build-system to centralize functionality like environment
variable management and luarocks installation.
The `lua-lang` virtual is now versioned so that a package that requires
Lua 5.1 semantics can get any lua, but one that requires 5.2 will only
get upstream lua.
The luaposix package requires lua-bit32, but only when built with a
lua conforming to version 5.1. This adds the package, and the
dependencies, but exposed a problem with luarocks dependency
detection. Since we're installing each package in its own "tree" and
there's no environment variable to list extra trees, spack now
generates a luarocks config file that lists all the trees of all the
dependencies, and references it by setting `LUAROCKS_CONFIG`
in the build environment of every LuaPackage. This allows luarocks
to find the spack installed dependencies correctly rather than
trying (and failing) to download them.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Tom Scogland <tscogland@llnl.gov>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Some of our `git` tests still fail when `init.defaultBranch` is set to something other
than `master`.
- [x] get rid of all hard-coded `master` refs
- [x] Use `'default'` to key tests that use the default branch
When running on Windows, Spack may generate files in the stage/install
prefixes that do not have write permissions, which prevents the
removal of those directories (e.g. when cleaning stages or uninstalling).
There should be a refactoring to avoid this in the first place, but that
is assumed to be longer term, so the temporary fix is to make such files
writable if they are not. This PR:
* Automatically handles these permissions errors when uninstalling
packages from the Spack root (makes then writable)
* Updates similar already-existing logic when removing Spack-managed
stage directories (the error-handling was assuming all errors were
permissions errors and was therefore handling other errors
inappropriately)
Note: these permissions issues only appear on Windows so this logic is
only applied there (permissions are not modified for this purpose on
Linux etc.).
This also adds special handling for a case where calling `isdir`
on an `os.DirEntry` object would fail for improperly-created symlinks
(e.g. on Windows, using `os.symlink` without `target_is_directory=True`).
Note this specific issue only came up when enabling link_tree tests
(specifically `source_merge_visitor_cant_be_cyclical`).
* create function for translating compiler names on specs/compiler entries in manifest
* add tests for translating compiler names on spec/compiler entries
* use higher-level function in test and add comment to prefer testing via higher-level function
* opensuse clingo check should not fail on account of this pr, but I cannot get it to pass by restarting via CI UI
* Addition of 1.1.9dev version.
* Small style fix -- extra blank line.
* Update var/spack/repos/builtin/packages/py-maestrowf/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-maestrowf/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-maestrowf/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-maestrowf/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Additional dependencies and version constraints.
* Revert to py-poetry.
* Remove run from cryptography (build only).
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Force GCC to always provide a C++14 flag
Updated gnu logic so that the c++14 flag for g++ is always propagated.
This fixes issues with build systems that error out if passed an empty
string for a flag.
Engaging in the best kind of software engineering by updating the unit
test to pass with the value it is now passed. This should better match
the expected flag for g++ compiling with the C++14 standard
* Add py-docutils@0.16
* Add sphinx-tabs package
* Update var/spack/repos/builtin/packages/py-sphinx-tabs/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-sphinx-tabs/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-sphinx-tabs/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This ensures that multiple spack instances called from `make` will respect the maximum number of jobs in the POSIX jobserver across packages.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Problem: GCC 9.4.0 catches a bad integer comparison in
resource/hlapi/bindings/c++/reapi_cli_impl.hpp in flux-sched@0.22.0
and current master.
Add a patch to work around the problem until an upstream fix is
available.
* use the init.defaultBranch name, not master
* make tcl and modules/common independent
Both used to use not just the same directory, but the same *file* for
their outputs. In parallel this can cause problems, but it can also
accidentally allow expected failures to pass if the file is left around
by mistake.
* use a non-global misc_cache in tests
* make pkg tests resilient to gitignore
* make source cache and module directories non-global
`make` solves a lot of headaches that would otherwise have to be implemented in Spack:
1. Parallelism over packages through multiple `spack install` processes
2. Orderly output of parallel package installs thanks to `make --sync-output=recurse` or `make -Orecurse` (works well in GNU Make 4.3; macOS is unfortunately on a 16 years old 3.x version, but it's one `spack install gmake` away...)
3. Shared jobserver across packages, which means a single `-j` to rule them all, instead of manually finding a balance between `#spack install processes` & `#jobs per package` (See #30302).
This pr adds the `spack env depfile` command that generates a Makefile with dag hashes as
targets, and dag hashes of dependencies as prerequisites, and a command
along the lines of `spack install --only=packages /hash` to just install
a single package.
It exposes two convenient phony targets: `all`, `fetch-all`. The former installs the environment, the latter just fetches all sources. So one can either use `make all -j16` directly or run `make fetch-all -j16` on a login node and `make all -j16` on a compute node.
Example:
```yaml
spack:
specs: [perl]
view: false
```
running
```
$ spack -e . env depfile --make-target-prefix env | tee Makefile
```
generates
```Makefile
SPACK ?= spack
.PHONY: env/all env/fetch-all env/clean
env/all: env/env
env/fetch-all: env/fetch
env/env: env/.install/cdqldivylyxocqymwnfzmzc5sx2zwvww
@touch $@
env/fetch: env/.fetch/cdqldivylyxocqymwnfzmzc5sx2zwvww env/.fetch/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.fetch/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.fetch/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk env/.fetch/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws env/.fetch/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.fetch/73t7ndb5w72hrat5hsax4caox2sgumzu env/.fetch/trvdyncxzfozxofpm3cwgq4vecpxixzs env/.fetch/sbzszb7v557ohyd6c2ekirx2t3ctxfxp env/.fetch/c4go4gxlcznh5p5nklpjm644epuh3pzc
@touch $@
env/dirs:
@mkdir -p env/.fetch env/.install
env/.fetch/%: | env/dirs
$(info Fetching $(SPEC))
$(SPACK) -e '/tmp/tmp.7PHPSIRACv' fetch $(SPACK_FETCH_FLAGS) /$(notdir $@) && touch $@
env/.install/%: env/.fetch/%
$(info Installing $(SPEC))
+$(SPACK) -e '/tmp/tmp.7PHPSIRACv' install $(SPACK_INSTALL_FLAGS) --only-concrete --only=package --no-add /$(notdir $@) && touch $@
# Set the human-readable spec for each target
env/%/cdqldivylyxocqymwnfzmzc5sx2zwvww: SPEC = perl@5.34.1%gcc@10.3.0+cpanm+shared+threads arch=linux-ubuntu20.04-zen2
env/%/gv5kin2xnn33uxyfte6k4a3bynhmtxze: SPEC = berkeley-db@18.1.40%gcc@10.3.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu20.04-zen2
env/%/cuymc7e5gupwyu7vza5d4vrbuslk277p: SPEC = bzip2@1.0.8%gcc@10.3.0~debug~pic+shared arch=linux-ubuntu20.04-zen2
env/%/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk: SPEC = diffutils@3.8%gcc@10.3.0 arch=linux-ubuntu20.04-zen2
env/%/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws: SPEC = libiconv@1.16%gcc@10.3.0 libs=shared,static arch=linux-ubuntu20.04-zen2
env/%/yfz2agazed7ohevqvnrmm7jfkmsgwjao: SPEC = gdbm@1.19%gcc@10.3.0 arch=linux-ubuntu20.04-zen2
env/%/73t7ndb5w72hrat5hsax4caox2sgumzu: SPEC = readline@8.1%gcc@10.3.0 arch=linux-ubuntu20.04-zen2
env/%/trvdyncxzfozxofpm3cwgq4vecpxixzs: SPEC = ncurses@6.2%gcc@10.3.0~symlinks+termlib abi=none arch=linux-ubuntu20.04-zen2
env/%/sbzszb7v557ohyd6c2ekirx2t3ctxfxp: SPEC = pkgconf@1.8.0%gcc@10.3.0 arch=linux-ubuntu20.04-zen2
env/%/c4go4gxlcznh5p5nklpjm644epuh3pzc: SPEC = zlib@1.2.12%gcc@10.3.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu20.04-zen2
# Install dependencies
env/.install/cdqldivylyxocqymwnfzmzc5sx2zwvww: env/.install/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.install/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.install/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.install/c4go4gxlcznh5p5nklpjm644epuh3pzc
env/.install/cuymc7e5gupwyu7vza5d4vrbuslk277p: env/.install/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk
env/.install/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk: env/.install/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws
env/.install/yfz2agazed7ohevqvnrmm7jfkmsgwjao: env/.install/73t7ndb5w72hrat5hsax4caox2sgumzu
env/.install/73t7ndb5w72hrat5hsax4caox2sgumzu: env/.install/trvdyncxzfozxofpm3cwgq4vecpxixzs
env/.install/trvdyncxzfozxofpm3cwgq4vecpxixzs: env/.install/sbzszb7v557ohyd6c2ekirx2t3ctxfxp
env/clean:
rm -f -- env/env env/fetch env/.fetch/cdqldivylyxocqymwnfzmzc5sx2zwvww env/.fetch/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.fetch/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.fetch/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk env/.fetch/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws env/.fetch/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.fetch/73t7ndb5w72hrat5hsax4caox2sgumzu env/.fetch/trvdyncxzfozxofpm3cwgq4vecpxixzs env/.fetch/sbzszb7v557ohyd6c2ekirx2t3ctxfxp env/.fetch/c4go4gxlcznh5p5nklpjm644epuh3pzc env/.install/cdqldivylyxocqymwnfzmzc5sx2zwvww env/.install/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.install/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.install/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk env/.install/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws env/.install/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.install/73t7ndb5w72hrat5hsax4caox2sgumzu env/.install/trvdyncxzfozxofpm3cwgq4vecpxixzs env/.install/sbzszb7v557ohyd6c2ekirx2t3ctxfxp env/.install/c4go4gxlcznh5p5nklpjm644epuh3pzc
```
Then with `make -O` you get very nice orderly output when packages are built in parallel:
```console
$ make -Orecurse -j16
spack -e . install --only-concrete --only=package /c4go4gxlcznh5p5nklpjm644epuh3pzc && touch c4go4gxlcznh5p5nklpjm644epuh3pzc
==> Installing zlib-1.2.12-c4go4gxlcznh5p5nklpjm644epuh3pzc
...
Fetch: 0.00s. Build: 0.88s. Total: 0.88s.
[+] /tmp/tmp.b1eTyAOe85/store/linux-ubuntu20.04-zen2/gcc-10.3.0/zlib-1.2.12-c4go4gxlcznh5p5nklpjm644epuh3pzc
spack -e . install --only-concrete --only=package /sbzszb7v557ohyd6c2ekirx2t3ctxfxp && touch sbzszb7v557ohyd6c2ekirx2t3ctxfxp
==> Installing pkgconf-1.8.0-sbzszb7v557ohyd6c2ekirx2t3ctxfxp
...
Fetch: 0.00s. Build: 3.96s. Total: 3.96s.
[+] /tmp/tmp.b1eTyAOe85/store/linux-ubuntu20.04-zen2/gcc-10.3.0/pkgconf-1.8.0-sbzszb7v557ohyd6c2ekirx2t3ctxfxp
```
For Perl, at least for me, using `make -j16` versus `spack -e . install -j16` speeds up the builds from 3m32.623s to 2m22.775s, as some configure scripts run in parallel.
Another nice feature is you can do Makefile "metaprogramming" and depend on packages built by Spack. This example fetches all sources (in parallel) first, print a message, and only then build packages (in parallel).
```Makefile
SPACK ?= spack
.PHONY: env
all: env
spack.lock: spack.yaml
$(SPACK) -e . concretize -f
env.mk: spack.lock
$(SPACK) -e . env depfile -o $@ --make-target-prefix spack
fetch: spack/fetch
@echo Fetched all packages && touch $@
env: fetch spack/env
@echo This executes after the environment has been installed
clean:
rm -rf spack/ env.mk spack.lock
ifeq (,$(filter clean,$(MAKECMDGOALS)))
include env.mk
endif
```
* Use patches from IBM's Open CE project to enable PyTorch to build on
Power systems.
Cherry-pick a patch to allow earlier versions of PyTorch to build with
CUDA 11.4.
* Update var/spack/repos/builtin/packages/py-torch/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* octopus: adding versions up to 11.4
* octopus: add smoke tests
* octopus: add necessary flags for gcc@10
* octopus: update to compilation and dependencies
* octopus: adding new variants
* octopus: remove 'poke' (as this poke is not in spack [yet])
* octopus: allow compilation from git repo develop branch
* octopus: adapt to spack style requirements
* octopus: add maintainer
* octopus: make tests after install optional
Thank you @tldahlgren
* octopus: follow recommended practice for test input data
Move the two configuration files we use for smoke tests into `test`
subdirectory. Thanks @tldahlgren.
* Adding maintainer
with their agreement by email
* octopus: reduce duplication of flags
- part of code review
* octopus: https is preferred over http
* octopus: remove .99 from versioning information
Thanks to https://github.com/spack/spack/pull/26402, we can drop the
"2:3.99" notation when we mean all versions 2.x and 3.x
Examples: b9e72557e8 (diff-b8373d30b3a141c495c2281273ee6184fc513413142afaf2adac1f406cd6b0d7)
(from review)
* octopus: args.extend([x]) -> args.append(x)
(hint from review)
libassimp has been a dependency for all of 5.x but expressing that has
varied significantly throughout the 5.x lifecycle:
v5.0: qt3d uses internal-only libassimp
v5.5: external-only libassimp
v5.6: either internal or external libassimp via autodetection
v5.9: user-selectable internal-vs-external via -assimp
v5.14: additional qtquick3d module uses -assimp
v5.15: qtquick3d switches to the -quick3d-assimp option
* current bug where the incorrect target is setup
* Add mimalloc package
* Add mimalloc as allocator option to pika
* Add mimalloc as allocator option to hpx
* Set git property globally instead of per-version in pika, hpx, and mimalloc packages
Co-authored-by: Mikael Simberg <mikael.simberg@iki.if>
Strictly, `sed` is a `build` and `run` dependency in all gpi-2
versions, whereas `gawk` is a `run` (`build` and `run`) dependency for
gpi-2 versions greater or equal (less) than 1.4.0
Gitlab pipelines run for spack already have other S3 storage locations
configured for storage of binaries, so this PR removes the redundant
per-pipeline mirror. As a result, the "cleanup" jobs will no longer be
generated at the end of each pipeline, removing one possible point of
pipeline failure.
The go-bootstrap package doesn't work on aarch64 platforms, so the only way
to build Go is to use gccgo.
Also, some versions of gccgo have a bug that prevents them from compiling
go (see golang/go#47771), so this patch limits gcc to versions newer than
10.4.0 or 11.3.0.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* new package: pytaridx
* fixed copyright year
* Update git link
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* added type in python depends
* added pypi link
* Update package.py
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Metall package: add dependency to GCC for build test
* Package Metall: add v.017
* Package Metall: update the package file
* Update var/spack/repos/builtin/packages/metall/package.py
Co-authored-by: Bernhard Kaindl <43588962+bernhardkaindl@users.noreply.github.com>
* Metall package: add v0.18 and v0.19
* Metall Package: add v0.20
Co-authored-by: Bernhard Kaindl <43588962+bernhardkaindl@users.noreply.github.com>
Starting with MPICH 3.4, we offer different datatype engine options
(dataloop or yaksa). The default is 'auto', which will choose based on
the device configuration. Starting with MPICH 4.0, building against an
external yaksa library is supported.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Added support for finding the OpenCV package via the find external
command. Included support for identifying variants based on available
shared libraries.
Added support to finding the OpenBLAS package via the find external
command.
Enabled packages to show that they can be discovered via the find
external command in the info message.
Updated the OpenCV and OpenBLAS packages to use the extensible search
mechanism for library extensions on multiple OS platforms.
Corrected how find externals works on Darwin for OpenCV and OpenBLAS
to accommodate that the version numbers are placed before the file
extension instead of after it, as on Linux.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* librsb: added v1.2.0.10 (#26043)
* librsb: add v1.2.0.11/v1.3.0.0 (#28636)
* librsb: add v1.3.0.1 (#30424)
* unconflict clang
* address apparent style issues
given
https://github.com/spack/spack/runs/6248126997?check_suite_focus=true
and its excerpt
var/spack/repos/builtin/packages/librsb/package.py:27: [E265] block comment should start with '# '
var/spack/repos/builtin/packages/librsb/package.py:52: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:53: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:53: [E501] line too long (89 > 88 characters)
var/spack/repos/builtin/packages/librsb/package.py:54: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:55: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:56: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:57: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:59: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:60: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:62: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:63: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:64: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:66: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:68: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:70: [E211] whitespace before '('
var/spack/repos/builtin/packages/librsb/package.py:71: [E211] whitespace before '('
let these changes flow in.
* +asan+native: mark as conflict; thanks @tldahlgren
* +asan conflict grouped with other conflicts
As suggested as good Spack style by @tldahlgren .
- Keep long lists in alphabetical order for easier reading
- Add a placeholder for Exa.TrkX plugin since we're missing a dep on the
Spack side
- Add support for the ONNX plugin since Spack now has an ONNX runtime
package
- Use spack's pybind11 package now that we're given the option to do so
This adds the newest stable version (and removes old development
versions), a few missing dependencies and workarounds for build
failures. Without the environment variables, sysstat will try creating
directories in `/var/log`, and without `--disable-file-attr`, sysstat
will try to change file ownership.
* gdal: changing behavior of configure for +xml2 with 3.0+
* Update var/spack/repos/builtin/packages/gdal/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add checksum for py-more-itertools@8.12.0 and fix python dependency
* Add checksum for py-prettytable@3.2.0
* Package version 8.11.0 is the only version that requires python 3.6+
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add reference to python@3.6 support when 8.11
* Revert "Add reference to python@3.6 support when 8.11"
This reverts commit 0ba0002193.
* Add python 3.7: requirement
* Revert range for python 3.6
* Revert py-more-itertools modifications
Co-authored-by: aandvalenzuela <andrea.valenzuela.ramirez@cern.ch>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This is an amended version of https://github.com/spack/spack/pull/24894 (reverted in https://github.com/spack/spack/pull/29603). https://github.com/spack/spack/pull/24894
broke all instances of `spack external find` (namely when it is invoked without arguments/options)
because it was mandating the presence of a file which most systems would not have.
This allows `spack external find` to proceed if that file is not present and adds tests for this.
- [x] Add a test which confirms that `spack external find` successfully reads a manifest file
if present in the default manifest path
--- Original commit message ---
Adds `spack external read-cray-manifest`, which reads a json file that describes a
set of package DAGs. The parsed results are stored directly in the database. A user
can see these installed specs with `spack find` (like any installed spec). The easiest
way to use them right now as dependencies is to run
`spack spec ... ^/hash-of-external-package`.
Changes include:
* `spack external read-cray-manifest --file <path/to/file>` will add all specs described
in the file to Spack's installation DB and will also install described compilers to the
compilers configuration (the expected format of the file is described in this PR as well including examples of the file)
* Database records now may include an "origin" (the command added in this PR
registers the origin as "external-db"). In the future, it is assumed users may want
to be able to treat installs registered with this command differently (e.g. they may
want to uninstall all specs added with this command)
* Hash properties are now always preserved when copying specs if the source spec
is concrete
* I don't think the hashes of installed-and-concrete specs should change and this
was the easiest way to handle that
* also specs that are concrete preserve their `.normal` property when copied
(external specs may mention compilers that are not registered, and without this
change they would fail in `normalize` when calling `validate_or_raise`)
* it might be this should only be the case if the spec was installed
- [x] Improve testing
- [x] Specifically mark DB records added with this command (so that users can do
something like "uninstall all packages added with `spack read-external-db`)
* This is now possible with `spack uninstall --all --origin=external-db` (this will
remove all specs added from manifest files)
- [x] Strip variants that are listed in json entries but don't actually exist for the package
* ASP-based solver: discard unknown packages from reuse
This is an add-on to #28259 that cover for the case of
a single package.py being removed from a repository,
rather than an entire custom repository being removed.
* Add unit test
CTest determines whether to enable tests using the BUILD_TESTING variable.
This should be used by projects to conditionally enable the compilation of tests.
Spack knowns which packages have to run tests and can thus automatically define this variable.
I tried to use --overwrite on nvhpc, but nvhpc's install size is 16GB. Seems
better to do os.rename in the same directory than moving the directory to
`/tmp`.
- [x] install --overwrite: use rename instead of tmpdir
- [x] use tempfile
By default `openmpi` needs `rsh` from `openssh`, which is a somewhat
redundant dependency for clusters using slurm. This PR adds a toggle to
allow users to disable the ssh/rsh plm altogether.
This package was not setting FFTW when +mklfft was used with +cuda.
Since both were set to 'True', the default build was not linked to
any FFTW, leading to a run time error. It seems MKL support was
conflated with alternative CPU acceleration support. This PR does the
following:
- adds the altcpu variant to specify non-GPU/CPU acceleration
- sets a conflict between +altcpu and +cuda
- sets an FFTW implementation
- sets fltk+xft when +gui to get a decent looking GUI interface
- sets tbb dependency only when +altcpu
- adds dependency on ctffind
- adds variant and dependency on motioncor2
- sets defaults for
- qsub template location
- ctffind location
- motioncor2 location
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
fixes#28259
This commit discard specs from unknown namespaces from the
ones that can be "reused" during concretization. Previously
Spack would just error out when encountering them.
1. Add version 2022.04.17 (new numbering scheme) and update mbuild resource.
2. Branch 'master' is now 'main'.
3. Old rev 10.2019.03 needs a patch for python vs python3.
The parent thread in the process stdout redirection logic on Windows
was closing a file that was being read in child thread, which lead to
error-based termination of the reader thread. This updates the
interaction to avoid the error.
* Add checksum for py-ipywidgets@7.7.0
* Correct py-widgetsnbextension and py-ipython dependencies
* Update widgetsnbextension dependency to 3.6
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Allow requirement to next versions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Revert ipyhton dependencies
* Add widgetsnbextension@3.6.0 checksum
Co-authored-by: aandvalenzuela <andrea.valenzuela.ramirez@cern.ch>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* ASP-based solver: allow configuring target selection
This commit adds a new "concretizer:targets" configuration
section, and two options under it.
- "concretizer:targets:granularity" allows switching from
considering only generic targets to consider all possible
microarchitectures.
- "concretizer:targets:host_compatible" instead controls
whether we can concretize for microarchitectures that
are incompatible with the current host.
* Add documentation
* Add unit-tests
* MAINT: Add a debug flag
* MAINT: Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* openmpi: always require pmix for 4:
`~pmix` is not applicable to version 4+, since it always builds a vendored
copy of pmix (currently 3.2.3).
* pmix: relax version requirements
When the version range was specified, newer versions didn't exist.
Also use normalized spack versions rather than artificial .9.9 /.0.0.
* openmpi: restrict pmix versions
pmix option isn't available for OpenMPI@1, and according to
https://github.com/open-mpi/ompi/issues/7988 , OpenMPI 4.0.1 will not
build with pmix@3.1.5.
* pmix: add newer versions
* OpenMPI: re-express conflicts/configure logic as conditional variants
This relies partly on `self.enable_or_disable` and its ilk to emit an
empty list when the variant isn't applicable.
* ASP-based solver: always consider version of installed packages
fixes#29201
Explicitly add facts for versions of installed software when
using the --reuse option, so that we could consider versions
that are not declared in package.py
The parser is already committing a crime of querying the database for
specs when it encounters a `/hash`. It's helpful, but unfortunately not
helpful when trying to install a specific spec in an environment by
hash. Therefore, consider the environment first, then the database.
This allows the following:
```console
$ spack -e . concretize
==> Starting concretization
==> Environment concretized in 0.27 seconds.
==> Concretized diffutils
- 7vangk4 diffutils@3.8%gcc@10.3.0 arch=linux-ubuntu20.04-zen2
- hyb7ehx ^libiconv@1.16%gcc@10.3.0 libs=shared,static arch=linux-ubuntu20.04-zen2
$ spack -e . install /hyb7ehx
==> Installing libiconv-1.16-hyb7ehxxyqqp2hiw56bzm5ampkw6cxws
...
==> libiconv: Successfully installed libiconv-1.16-hyb7ehxxyqqp2hiw56bzm5ampkw6cxws
Fetch: 0.01s. Build: 17.54s. Total: 17.55s.
[+] /tmp/tmp.VpvYApofVm/store/linux-ubuntu20.04-zen2/gcc-10.3.0/libiconv-1.16-hyb7ehxxyqqp2hiw56bzm5ampkw6cxws
```
1. update for rocm 4.5 and drop support for earlier rocm.
2. no longer use mbedtls or gotcha, they are only for old revs.
3. update version requirements for dyninst and libmonitor
4. begin to deprecate old versions
Fix bug introduced in #30191. `Spec.installed` and `Spec.installed_upstream` should just return
`False` for abstract specs, as they can be called in that context.
- [x] `Spec.installed` returns `False` now instead of asserting that the `Spec`
is concrete.
- [x] `Spec.installed_upstream` returns `False` now instead of asserting that the `Spec`
is concrete.
- [x] `Spec.installed_upstream` no longer caches its result, as install status seems
like a bad thing to cache -- it can easily be invalidated. Calling code should
use transactions if there are peformance issues, as with other places in Spack.
- [x] add tests for `Spec.installed` and `Spec.installed_upstream`
This PR moves the `installed` and `installed_upstream` properties from `PackageBase` to `Spec` and is a step towards being able to reuse specs for which we don't have a `package.py` available. It _should_ be sufficient to complete the concretization step and see the spec in the concretized DAG.
To fully reuse a spec without a package.py though we need a way to serialize enough data to reconstruct the results of calls to:
- `Spec.libs`, `Spec.headers` and `Spec.ommand`
- `Package.setup_dependent_*_environment` and `Package.setup_run_environment`
- [x] Add stub methods to packages with warnings
- [x] Add a missing "root=False" in cmd/fetch.py
- [x] Assert that a spec is concrete before checking installation status
* Add checksum for jupyter-console@6.4.3
* Update py-jupyter-console dependency
* Extend jupyter-client@7.0.0 dependency to newer versions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: aandvalenzuela <andrea.valenzuela.ramirez@cern.ch>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-pystan: Add new package
* Fix dependencies
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add run dependency to py-setuptools
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add py-httpstan@4.7.2 and py-pysimdjson@3.2.0
* Dependency
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This PR updates the list of images we build nightly, deprecating
Ubuntu 16.04 and CentOS 8 and adding Ubuntu 20.04, Ubuntu 22.04
and CentOS Stream. It also removes a lot of duplication by generating
the Dockerfiles during the CI workflow and uploading them as artifacts
for later inspection or reuse.
* ipopt: add goxberry as maintainer
This commit adds 'goxberry' (me, Geoff Oxberry) as a maintainer of the
Ipopt Spack package.
* ipopt: use github url instead of coin-or.org url
This commit changes the package URL for Ipopt from one containing
`coin-or.org` to one containing `github.com`. The rationale for
using `github.com` is as follows:
- The COIN-OR webpage now directs users interested in Ipopt source to
GitHub.
- Ipopt used to have a COIN-OR project homepage actually hosted on
coin-or.org using an SVN-Trac web page. A link to this project
homepage no longer appears within the "Projects" section of
COIN-OR's website.
- COIN-OR issued a 2021-12-15 post on the News section of its web site
(see https://www.coin-or.org/news/) that discusses the impact that
lack of financial support has on COIN-OR software maintenance. It
seems reasonable to suspect that the GitHub project is likely to
outlast the COIN-OR web site.
The sha256 hashes for ipopt@:3.12 downloaded from GitHub differ from
the corresponding COIN-OR versions, so these hashes are also updated.
* ipopt 3.14.5: add new version
This commit adds the latest version of Ipopt, 3.14.5, to the Ipopt
Spack package.
* git: add 2.35.2, explicit version(...)
git 2.35.2 fixes CVE-2022-24765 which seems to only affect Windows. But
nonetheless we should maybe set deprecated=True on older versions... The
restructure allows for that.
* deprecate over CVE-2022-24765
In WarpX 22.04, we introduced the openPMD `thetaMode` for fields in
RZ geometry. That means we need to name the fields differently than
the reconstructured Cartesian slice that we default to in plotfiles.
* ncurses: add wide, nowide headers, libs query parameter options
* readline: only link with libncursesw
Needed for python to detect proper ncurses library #27369
Alter the `install_components/install` script to pass the `-gcc $SPACK_CC`,
`-gpp $SPACK_CXX`, and `-g77 $SPACK_F77` flags to `makelocalrc`. This
ensures that nvhpc is configured to use the spack gcc spec, rather than
whatever gcc is found on the path.
Co-authored-by: Mikael Simberg <simberg@cscs.ch>
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Fix test_ci_generate_prune_untouched(), which would fail if run when
the latest commit changed the .gitlab-ci.yml. This change mocks the
get_stack_changed() method in that test to disregard the state of
the current spack repo in favor of a mock repo under test control.
* The configure script on Windows requires that CC/CXX be enclosed
in quotes if the paths to those compiler executables contain
spaces (so unlike most instances of Executable, the arguments
need to contain the quotes)
* OpenSSL requires the nasm package on Windows
* Restore parallel build from 075e942 (accidentally reverted in
#27021)
* py-ipympl: Add new package
* Update var/spack/repos/builtin/packages/py-ipympl/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-ipympl/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-ipympl/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-ipympl/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Remove trailing whitespaces
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-webargs: Add new package
* Fix python requirement
* Add run dependency to py-packaging
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
gitlab ci: Set resource requests explicitly
This PR sets resource requests for the Kubernetes executor, which should aid in
better workload scheduling in the cluster. The specific values were derived from
profile data taken from several full "from scratch" rebuilds in a separate worker pool.
Co-authored-by: Zack Galbreath <zack.galbreath@kitware.com>
* serialbox: setup the run and dependent build environments
* Update var/spack/repos/builtin/packages/serialbox/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* rocmlibs: relax rocm-cmake version requirements
The rocm-cmake modules tend to be backwards-compatible, to the extent
that most ROCm math libraries were built using rocm-cmake@master
for a long while without anybody noticing. (That was fixed in
97f0c3ccd9f0a40896998a7580150a514ec3bc37.)
Some packages, like comgr, barely use rocm-cmake for anything, and
we can easily set a very minimal version requirement. For most
packages, however, it would be a lot of effort to determine the
minimum rocm-cmake version required for each release. For those
packages, I just turned the exact version requirement into a
minimum version requirement.
Since I was looking through the CMakeLists.txt for a large number of
libraries, I also took note of the cmake_minimum_required and adjusted
the cmake minimum requirements to match.
* Add rocblas build dependency to hipblas
The rocblas library is required both for both building and linking
hipblas.
* Remove rocm-cmake from vtk-m dependency list
The rocm-cmake package provides CMake scripts that facilitate common
build configuration tasks in the ROCm libraries. It is never needed at
link-time. Also, there are no calls to find_package(ROCM) or
include(ROCM.*)in vtk-m, so this dependency will never be used.
- older versions are no longer available for download so mark them
deprecated
- set manual_download
- set url_for_version
- only install the binary that matches the cuda version
In #26630, I assumed "glu" was needed by glew because it included glu.h, but
actually, glew can be used without glu when GLEW_NO_GLU is defined and this
is documented in the announcement of glew-1.6.0:
> https://www.geeks3d.com/20110430/opengl-glew-1-6-0-available/
> * Define GLEW_NO_GLU for no glu dependency
It is therefore the duty of users of glew to decide if they use glu,
and then they need to have a depends_on("glu").
Thus, move the depends_on("glu") which I changed from "gl" in #26630
to vapor, which itself uses glu as well.
For about a decade GCC has an option `-f[no]-canonical-system-headers`
which basically runs `realpath` on all "system headers", to possibly
reduce the length of paths in diagnostics. [1]
Spack usually installs the "system headers" of GCC in very deeply nested
directories. Calling `realpath` there results in stat calls on every
level, for every header file. On some slow filesystem I have,
`-fno-canonical-system-headers` gives about 5x speedup to compile hello
world in C, meaning that ./configure scripts would be much faster when
using this flag by default.
[1] https://codereview.appspot.com/6495088
Add option to allow using OpenSSL (by default this uses the SSL
implementation that comes with Windows, since that is more likely
to have needed certificates).
* py-awkward: Add new versions
* py-awkward: Update dependencies
* Make setuptools a runtime dependency as well
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Don't rely on NASM's nmake to export install target. Spack
now handles NASM installation; the install tree structure
mimics NASM Windows installer behavior.
* Add dependency on perl
we switched to an optional sphinx based way of
generating docs, so remove pandoc, which can cause
issues with latex conflicts.
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
Bug fixes for package netcdf-cxx4 so that it builds on macOS semi
case-sensitive filesystems; this includes additional changes to build
netcdf-cxx4 consistently with netcdf-fortran.
* netcdf-fortran: remove unused config_flags
* netcdf-fortran: avoid building without the optimization flags
* netcdf-cxx4: do not enforce autoreconf. This was a rudiment from the
times when the package was fetched with git, which broke timestamp
order of the automatically generated Autoconf files.
* netcdf-cxx4: inject PIC flags for C++ when '+pic'
* netcdf-cxx4: inject C/CXXFLAGS via the wrapper
* netcdf-cxx4: fix the underlinking problem for platforms other than darwin
(add netcdf-c libs netcdf-cxx4 ldlibs flags)
* netcdf-cxx4: remove redundant extension of CPPFLAGS
* netcdf-cxx4: only need to use MPI compiler wrapper when building C
(vs both C and C++)
* netcdf-cxx4: remove variant 'static'
This makes it consistent with other packages from the NetCDF
constellation: always build the static libraries and additionally
build the shared ones when '+shared'.
* netcdf-cxx4: do not configure --with/--without-pic.
This makes it consistent with other packages from the NetCDF
constellation: build the shared libraries with the PIC flag and
the static ones without it (the default for Autotools) when
'~pic', and build the static libraries with PIC when '+pic' (to
make them injectable into other shared libraries).
* netcdf-cxx4: run the tests serially
* netcdf-cxx4: build the plugins only when the tests are run
Co-authored-by: Sergey Kosukhin <sergey.kosukhin@mpimet.mpg.de>
gitlab ci: Remove code for relating CDash builds
Relating CDash builds to their dependencies was a seldom used feature. Removing
it will make it easier for us to reorganize our CDash projects & build groups in the
future by eliminating the needs to keep track of CDash build ids in our binary mirrors.
* Allow packages to add a 'submodules' property that determines when ad-hoc Git-commit-based versions should initialize submodules
* add support for ad-hoc git-commit-based versions to instantiate submodules if the associated package has a 'submodules' property and it indicates this should happen for the associated spec
* allow Package-level submodule request to influence all explicitly-defined version() in the Package
* skip test on windows which fails because of long paths
* Set CUDA architectures in ArrayFire based on cuda_arch
The cuda_arch flag was not recognized by the ArrayFire package and
therefore any setting was not respected. This commit adds the appropriate
cmake flags if cuda_arch is specified. If no cuda_arch is specified,
then the flag is set to "Auto" which checks the installed compute
architectures on the build system.
* ArrayFire only requires boost headers to build. Update version to 1.75
ArrayFire only requires boost headers at build time. This commit also
updates the version to 1.75 to avoid some errors in Boost Compute
* Disable tests in ArrayFire by default
* Add support for ArrayFire v3.8.1
* Add maintainer for ArrayFire package
* Remove test variant from ArrayFire. Use comprehensions
* Reduce boost requirement in ArrayFire
* Address cuda_arch suggestions
* Add commit hashes to Release versions of ArrayFire
* Fix style issues in ArrayFire package
Ubuntu patched git v2.25.1 with a security fix that also
introduced a breaking change, so v2.25.1 behaves like
v2.35.2 with respect to the use cases in CVE-2022-24765
* llvm7_intel.patch required for intel@19.1.3 too
* apply llvm7_intel.patch forall intel@19.0 and intel@19.1
Co-authored-by: Daryl W. Grunau <dwg@lanl.gov>
Spack added support in #24639 for ad-hoc Git-commit-hash-based
versions: A user can install a package x@hash, where X is a package
that stores its source code in a Git repository, and the hash refers
to a commit in that repository which is not recorded as an explicit
version in the package.py file for X.
A couple issues were found relating to this:
* If an environment defines an alternative package repo (i.e. with
repos.yaml), and spack.yaml contains user Specs with ad-hoc
Git-commit-hash-based versions for packages in that repo,
then as part of retrieving the data needed for version comparisons
it will attempt to retrieve the package before the environment's
configuration is instantiated.
* The bookkeeping information added to compare ad-hoc git versions was
being stripped from Specs during concretization (such that user
Specs which succeeded before concretizing would then fail after)
This addresses the issues:
* The first issue is resolved by deferring access to the associated
Package until the versions are actually compared to one another.
* The second issue is resolved by ensuring that the Git bookkeeping
information is explicitly applied to Specs after they are concretized.
This also:
* Resolves an ambiguity in the mock_git_version_info fixture used to
create a tree of Git commits and provide a list where each index
maps to a known commit.
* Isolates the cache used for Git repositories in tests using the
mock_git_version_info fixture
* Adds a TODO which points out that if the remote Git repository
overwrites tags, that Spack will then fail when using
ad-hoc Git-commit-hash-based versions
This commit updates the `gpg publish` command to work with the mirror
arguments, when trying to push keys to a mirror.
- [x] update `gpg publish command
- [x] add test for publishing GPG keys and rebuilding the key index within a mirror
* zstd: bring back libs=shared,static and compression=zlib,lz4,lzma variants
Should make building `gcc+binutils ^zstd libs=static` a bit easier (this
is the case where we don't control the compiler wrappers of gcc because
of bootstrapping, nor of ld because of how gcc invokes the linker).
In a typical call to spack, the OperatingSystem gets instantiated
multiple times. For macOS, each one requires a call to `sw_vers`, which
is done through the Executable helper class. Memoizing
reduces the call count from "spac spec" from three to one.
Currently environments are indexed by build hashes. When looking into this bug I noticed there is a disconnect between environments that are concretized in memory for the first time and environments that are read from a `spack.lock`. The issue is that specs read from a `spack.lock` don't have a full hash, since they are indexed by a build hash which is strictly coarser. They are also marked "final" as they are read from a file, so we can't compute additional hashes.
This bugfix PR makes "first concretization" equivalent to re-reading the specs from a corresponding `spack.lock`, and doing so unveiled a few tests were we were making wrong assumptions and relying on the fact that a `spack.lock` file was not there already.
* Add unit test
* Modify mpich to trigger jobs in pipelines
* Fix two failing unit tests
* Fix another full_hash vs. build_hash mismatch in tests
* Ignore top-level module config; add auto-update
In Spack 0.17 we got module sets (modules:[name]:[prop]), and for
backwards compat modules:[prop] was short for modules:default:[prop].
But this makes it awkward to define default config for the "default"
module set.
Since 0.17 is branched off, we can now deprecate top-level module config
(that is, just ignore it with a warning).
This PR does that, and it implements `spack config update modules` to
make upgrading easy (we should have added that to 0.17 already...)
It also removes references to `dotkit` stuff which was already
deprecated in 0.13 and could have been removed in 0.14.
Prefix inspections are the only exception, since the top-level prefix inspections
used for `spack load` and `spack env activate`.
Spack currently allows dependencies to be concretized for an
architecture incompatible with the root. This commit adds rules
to make this situation impossible by design.
* Extract the MetaPathFinder and Loaders for packages in their own classes
https://peps.python.org/pep-0451/
Currently, RepoPath and Repo implement the (deprecated) interface of
MetaPathFinder (find_module) and of Loader (load_module). This commit
extracts both of them and places the code in their own classes.
The MetaPathFinder interface is updated to contain both the deprecated
"find_module" (for Python 2.7 support) and the recommended "find_spec".
Update of the Loader interface is deferred at a subsequent commit.
* Move the lines to be prepended inside "RepoLoader"
Also adjust the naming of a few variables too
* Remove spack.util.imp, since code is only used in spack.repo
* Remove support from loading Python modules Python > 3 but < 3.5
* Remove `Repo._create_namespace`
This function was interacting badly with the MetaPathFinder
and causing issues with "normal" imports. Removing the
function allows to do things like:
```python
import spack.pkg.builtin.mpich
cls = spack.pkg.builtin.mpich.Mpich
```
* Remove code needed to trigger the Singleton evaluation
The finder is coded in a way to trigger the Singleton,
so we don't need external code now that we register it
at module level into `sys.meta_path`.
* Add unit tests
OpenMPI includes cuda_runtime.h, which errors with `#error --
unsupported GNU version! gcc versions later than 9 are not supported!`
By inheriting CudaPackage, the proper conflicts between `cuda` and
`gcc`/`clang` are added.
* mesa, mesa18: Implement the swr variant consistently between mesa and mesa18
* mesa: Bump to 21.3.7
* mesa: Build release by default tie swr to release builds
* mesa, mesa18: re-enable the llvm variant by default
This reverts the change made in #29360
Some servers require `User-Agent` to be set, and otherwise error with
access denied. One such example is mpich.
To fix this, set `User-Agent: Spackbot/[version]` as a header.
Apparently by convention, it should include the word `bot`.
#27021 broke fetching for CVS-based packages because:
- The mirror logic was using URL parsing to extract a path from the
CVS repository location
- #27021 added sanity checks to enforce that strings passed to the
URL parser were actually URLs
This replaces the call to "url_util.parse" with logic that is
customized for CVS. This implies that VCSFetchStrategy should
rename the "url_attr" attribute to something more generic, but
that should be handled separately.
* mpich: add 3.4.3, 4.0, 4.0.1
* mpich: add url_for_version function
For versions 4.0 and up, get tarballs from GitHub. This will help with
CI builds, since the MPICH website denies the urllib user-agent from
downloading release tarballs.
* mpich: disable cuda support
MPICH is failing to build in CI due to a configuration script bug in
detecting CUDA support. Disable CUDA support by default until we add a
proper variant.
Allow declaring possible values for variants with an associated condition. If the variant takes one of those values, the condition is imposed as a further constraint.
The idea of this PR is to implement part of the mechanisms needed for modeling [packages with multiple build-systems]( https://github.com/spack/seps/pull/3). After this PR the build-system directive can be implemented as:
```python
variant(
'build-system',
default='cmake',
values=(
'autotools',
conditional('cmake', when='@X.Y:')
),
description='...',
)
```
Modifications:
- [x] Allow conditional possible values in variants
- [x] Add a unit-test for the feature
- [x] Add documentation
* tests for rewiring pure specs to spliced specs
* relocate text, binaries, and links
* using llnl.util.symlink for windows compat.
Note: This does not include CLI hooks for relocation.
Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
From the tempfile module docs:
The default directory is chosen from a platform-dependent list, but the
user of the application can control the directory location by setting
the TMPDIR, TEMP or TMP environment variables
missing dependencies
- boost
- lzo
Also, turn off libuv. This does not build properly with libuv so it is
not a dependency. However, configure will look for libuv on the system
and try to use it if found, thus breaking the build.
- Add variants for various common build flags, including support for both versions of the Racket VM environment.
- Prevent `-j` flags to `make`, which has been known to cause problems with Racket builds.
- Prefer the minimal release to improve install times. Bells and whistles carry their own runtime dependencies and should be installed via `raco`. An enterprising user may even create a `RacketPackage` class to make spack aware of `raco` installed packages.
- Match the official version numbering scheme.
- Update to version 1.2.12.
- Mark older versions as deprecated because they have security bugs.
- mfem: Update list of system library directories
- zlib patch: cc patch
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Update "spack external find --all" to also find library-only packages.
A Package can add a ".libraries" attribute, which is a list of regular
expressions to use to find libraries associated with the Package.
"spack external find --all" will search LD_LIBRARY_PATH for potential
libraries.
This PR adds examples for NCCL, RCCL, and hipblas packages. These
examples specify the suffix ".so" for the regular expressions used
to find libraries, so generally are only useful for detecting library
packages on Linux.
Do not prompt user with checksum warning when using git commit hashes
as versions. Spack was incorrectly reporting this as a potential
problem: it would display a prompt asking the user whether they
want to proceed if Spack was running in a terminal, or it would
terminate the running instance of Spack if running as part of a
script.
* rocm-cmake: remove ldconfig variant
The packages built for `rocm-cmake~ldconfig` and `rocm-cmake+ldconfig`
are identical, so the variant is unnecessary.
The `ROCM_DISABLE_LDCONFIG` option changes how `rocm_create_package`
generates DEB and RPM packages with CPack. rocm-cmake itself uses
`rocm_create_package`, however, this option is has no effect because
Spack does not build the CPack packages. It is also unnecessary on
rocm-cmake, because rocm-cmake does not contain any shared libraries
for ldconfig to configure. The rocm-cmake package is purely composed
of CMake scripts.
* Tighten CMake version dependency
* Improve package description
* Add pl2bat to PATH: Windows on Perl requires the script pl2bat.bat
and Perl to be available to the installer via the PATH. The build
and dependent environments of Perl on Windows have the install
prefix bin added to the PATH.
* symlink with win32file module instead of using Executable to
call mklink (mklink is a shell function and so is not accessible
in this manner).
* py-marshmallow: Add new package
* Modify py-packaging dependency type
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add run dependency to py-packaging
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
We've previously generated CI pipelines for PRs, and they rebuild any packages that don't have
a binary in an existing build cache. The assumption we were making was that ALL prior merged
builds would be in cache, but due to the way we do security in the pipeline, they aren't. `develop`
pipelines can take a while to catch up with the latest PRs, and while it does that, there may be a
bunch of redundant builds on PRs that duplicate things being rebuilt on `develop`. Until we can
do better caching of PR builds, we'll have this problem.
We can do better in PRs, though, by *only* rebuilding things in the CI environment that are actually
touched by the PR. This change computes exactly what packages are changed by a PR branch and
*only* includes those packages' dependents and dependencies in the generated pipeline. Other
as-yet unbuilt packages are pruned from CI for the PR.
For `develop` pipelines, we still want to build everything to ensure that the stack works, and to ensure
that `develop` catches up with PRs. This is especially true since we do not do rebuilds for *every* commit
on `develop` -- just the most recent one after each `develop` pipeline finishes. Since we skip around,
we may end up missing builds unless we ensure that we rebuild everything.
We differentiate between `develop` and PR pipelines in `.gitlab-ci.yml` by setting
`SPACK_PRUNE_UNTOUCHED` for PRs. `develop` will still have the old behavior.
- [x] Add `SPACK_PRUNE_UNTOUCHED` variable to `spack ci`
- [x] Refactor `spack pkg` command by moving historical package checking logic to `spack.repo`
- [x] Implement pruning logic in `spack ci` to remove untouched packages
- [x] add tests
* py-pysimdjson: Add new package
* Cleanup
* Fix python requirement
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* libtiff: add missing dependencies
- gl
- glu
- freeglut
* Make X/GL only for Darwin/Mac
* Catch the force_autoreconf property
* add platform=darwin to the autotools deps as well
* Update var/spack/repos/builtin/packages/libtiff/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Fixes the following error on %clang@13.0.1
>> 2413 bison: error while loading shared libraries: libtextstyle.so.0: cannot open shared object file: No such file or directory
>> 2414 make[2]: *** [<builtin>: getdate.c] Error 127
VecCore's new home is on github (hashes have changed even though commit
IDs and presumably contents are the same), and it does not need any configuration
options. See discussion at https://gitlab.cern.ch/VecGeom/VecCore/-/merge_requests/1 .
Updated flecsi spackage to better support changes in control variables
in post 2.1.0 releases while also making legacy versions clearer as to
what is a tagged release and what is a rolling-ish development branch
* py-reportlab: add missing dependency on freetype
* Add missing dependencies
* Update var/spack/repos/builtin/packages/py-reportlab/package.py
Use pil virtual.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* ExaGO: Handling of cuda architectures and amdgpu targets changed
to effectively handle multiple targets. See #28441.
* Add ROCm support to ExaGO and update ROCm support in HiOp
* ExaGO+rocm requires HiOp+rocm
* Newer versions of CMake may set HIP_CLANG_INCLUDE_PATH incorrectly:
add comments to the ExaGO/HiOp packages explaining how to address
this problem if it occurs.
* cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH
Spack has a heuristic to add rpaths for packages it knows are required,
but it's really a heuristic, and it does not work when the dependencies
put their libraries in a different folder than `<prefix>/lib{64,}`.
CMake patches binaries after install with the "install rpaths", which by
default are provided by Spack and its heuristic through
`CMAKE_INSTALL_RPATH`.
CMake however knows better what libraries are effectively being linked
to, and has an option to include those in the install rpath too, through
`CMAKE_INSTALL_RPATH_USE_LINK_PATH`.
These two CMake options are complementary, repeated rpaths seem to be
filtered, and the "use link path" paths are appended to Spack's
heuristic "install rpath".
So, it seems like a good idea to enable "use link path" by default, so
that:
- `dlopen` by library name uses Spack's heuristic search paths
- linked libraries in non-standard locations within a prefix get an
rpath thanks to CMake.
* docs
- Use define/define_from_variant
- Remove unused "fortran_flags"
- Fix CUDA architectures when using multiple (needs semicolon not comma
separators)
- Add `when=` variant restrictions to simplify logic
Add output of build- and install-time tests to info command
Enable dependencies, variants, and versions by default (i.e., provide --no*
options; add gcc to test_info_fields to increase coverage for c_names->v_names
* New package: spiner
* Update dependencies for spiner package
* Update var/spack/repos/builtin/packages/spiner/package.py
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* Update var/spack/repos/builtin/packages/spiner/package.py
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* Remove versions that can't be installed and use ports-of-call@1.1.0
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* py-torch: fix build with fujitsu-ssl2
* fix to use fujitsu-ssl2 in py-torch v1.5.0 to v1.11.0
* fix to use fujitsu-ssl2 in py-torch v1.2.0 to v1.11.0
* Delete fj-ssl2.patch
* renamed the patches
* Rename fj-ssl2.1.5.patch to fj-ssl2_1.5.patch
* Delete fj-ssl2_1.5.patch
We shouldn't be using "remove_linked_tree" to remove the lock file,
since that function expects to receive a directory path as an
argument.
Also, as a further measure to avoid regression, this commit restores
the "ignore_errors=True" argument on linux and adds a unit test
checking that "remove_linked_tree" doesn't change file permissions
as a side effect of a failure to remove.
* Fix py-onnx-runtime recipe
* Add missing dependencies
* Update var/spack/repos/builtin/packages/py-cerberus/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Better fix for py-onnx-runtime
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* omegah: v10.1.0
this version is from the SCOREC fork of Omega_h
* prefix version with scorec
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
Reduces the number of stat calls to a bare minimum:
- Single pass over src prefixes
- Handle projection clashes in memory
Symlinked directories in the src prefixes are now conditionally
transformed into directories with symlinks in the dst dir. Notably
`intel-mkl`, `cuda` and `qt` has top-level symlinked directories that
previously resulted in empty directories in the view. We now avoid
cycles and possible exponential blowup by only expanding symlinks that:
- point to dirs deeper in the folder structure;
- are a fixed depth of 2.
* py-cffi: add compiler flags to fix build with clang
For %clang@13.0.1, this avoids the
```
clang-13: warning: optimization flag '-ffat-lto-objects' is not supported [-Wignored-optimization-argument]
```
warning being turned into an error, and fixes this link error:
```
build/temp.linux-x86_64-3.10/c/_cffi_backend.o: file not recognized: file format not recognized
```
* style
Currently `old_root` is computed by reading the symlink at `self.root`.
We should be more defensive in removing it by checking that it is in the
same directory as the new root. Otherwise, in the worst case, when
someone runs `spack env create --with-view=./view -d .` and `view`
already exists and is a symlink to `/`, Spack effectively runs `rm -rf /`.
`file` was used to detect Python scripts with shebangs, so that the interpreter could be changed from <python prefix> to <view path>. With this change, we detect shebangs using Python instead, so that `file` is no longer required.
The number of commit characters in patch files fetched from GitHub can change,
so we should use `full_index=1` to enforce full commit hashes (and a stable
patch `sha256`).
Similarly, URLs for branches like `master` don't give us stable patch files,
because branches are moving targets. Use specific tags or commits for those.
- [x] update all github patch URLs to use `full_index=1`
- [x] don't use `master` or other branches for patches
- [x] add an audit check and a test for `?full_index=1`
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Known issues reports only 2 issues, among the bugs reported on GitHub.
One of the two is also outdated, since the issue has been solved
with the new concretizer. Thus, this commit removes the section.
* This commit removes the Boost.with_default_variants to variants
that packages are precisely dependant upon. This is the first batch
of 20 packages with modified boost dependencies.
* Style fixes
* Tested bridger: works for gcc-4.9.3 and gcc-8.3.1
Commit 26ff443 made the Gitlab pipeline failing on develop
(while it was not failing in the original PR) due to errors in the
fetcher. This change preserves the new versions, but will give
some time for use to sync our tarball mirror for better reliability
* vecgeom: fix cuda arch
* vecgeom: change 'options' to 'args'
* vecgeom: add spec to locals
* vecgeom: suppress architecture specializations when cuda
- constrain samtools to version 1.13
- replace lzma dependency with xz
- add missing dependencies for libdeflate and openssl
- explicitly set LD_FLAGS for dependencies in makefile
From the release announcement: "This is a special bugfix release ahead of
schedule to address a memory leak that was happening on certain function calls
when using Cython. The memory leak consisted of a small constant amount of bytes
in certain function calls from Cython code. Although in most cases this was not
very noticeable, it was very impactful for long-running applications and certain
usage patterns. Check bpo-46347 for more information."
When you install Spack from a tarball, it will always show an exact
version for Spack itself, even when you don't download a tagged commit:
```
$ wget -q https://github.com/spack/spack/archive/refs/heads/develop.tar.gz
$ tar -xf develop.tar.gz
$ ./spack-develop/bin/spack --version
0.16.2
```
This PR sets the Spack version to `0.18.0.dev0` on develop, following [PEP440](https://github.com/spack/spack/pull/25267#issuecomment-896340234) as
suggested by Adam Stewart.
```
spack (fix/set-dev-version)$ spack --version
0.18.0.dev0 (git 0.17.1-1526-e270464ae0)
spack (fix/set-dev-version)$ mv .git .git_
spack $ spack --version
0.18.0.dev0
```
- [x] Update the release guide
- [x] Add __version__ to spack's __init__.py
- [x] Use PEP 440 canonical version strings
- [x] Make spack --version output [actual version] (git version)
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
* rivet: fix dependency build types
If it isn't a python package, there is no good reason to change the default build type to remove link
* rivet: turn swig into build dependency
* Add tests to ensure google cloud storage urls work as mirrors
This commit adds two tests to track that GCS buckets can work as
mirrors, and can be parsed as valid URLs.
Currently, gs:// format URLs are not correctly parsed.
* Fix URL parsing for GCS buckets
This commit adds GCS bucket URLs as valid URLs.
* lower priority of package-provided urls
This change favors urls found in a scraped page over those provided by
the package from `url_for_version`. In most cases this doesn't matter,
but R specifically returns known bad URLs in some cases, and the
fallback path for a failed fetch uses `fetch_remote_versions` to find a
substitute. This fixes that problem.
fixes#29204
* consider what links actually exist in all cases
Checksum was only actually scraping when called with no versions. It
now always scrapes and then selects URLs from the set of URLs known to
exist whenever possible.
fixes#25831
* bow to the wrath of flake8
* test-fetch urls from package, prefer if successful
* Update lib/spack/spack/package.py
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* reword as suggested
* re-enable mypy specific ignore and ignore pyflakes
* remove flake8 ignore from .flake8
* address review comments
* address comments
* add sneaky missing substitute
I missed this one because we call substitute on a URL that doesn't
contain a version component. I'm not sure how that's supposed to work,
but apparently it's required by at least one mock package, so back in it
goes.
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
Adds `spack external read-cray-manifest`, which reads a json file that describes a set of package DAGs. The parsed results are stored directly in the database. A user can see these installed specs with `spack find` (like any installed spec). The easiest way to use them right now as dependencies is to run `spack spec ... ^/hash-of-external-package`.
Changes include:
* `spack external read-cray-manifest --file <path/to/file>` will add all specs described in the file to Spack's installation DB and will also install described compilers to the compilers configuration (the expected format of the file is described in this PR as well including examples of the file)
* Database records now may include an "origin" (the command added in this PR registers the origin as "external-db"). In the future, it is assumed users may want to be able to treat installs registered with this command differently (e.g. they may want to uninstall all specs added with this command)
* Hash properties are now always preserved when copying specs if the source spec is concrete
* I don't think the hashes of installed-and-concrete specs should change and this was the easiest way to handle that
* also specs that are concrete preserve their `.normal` property when copied (external specs may mention compilers that are not registered, and without this change they would fail in `normalize` when calling `validate_or_raise`)
* it might be this should only be the case if the spec was installed
- [x] Improve testing
- [x] Specifically mark DB records added with this command (so that users can do something like "uninstall all packages added with `spack read-external-db`)
* This is now possible with `spack uninstall --all --origin=external-db` (this will remove all specs added from manifest files)
- [x] Strip variants that are listed in json entries but don't actually exist for the package
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
* Use same cxx value as root
* Remove pointer syntax from non-pointer type in source
* Run patch function before build
* Use raw string in filter_file and merge edit function with patch
* Escape parentheses
* Use gDirectory from ROOT instead of CurrentDirectory function
This PR removes a few outdated sections from the "Basics" part of the
documentation. It also makes a few topic under the environment section
more prominent by removing an unneeded spack.yaml subsection and
promoting everything under it.
* Make boost composable
Currently Boost enables a few components through variants by default,
which means that if you want to use only what you need and no more, you
have to explicitly disable these variants, leading to concretization
errors whenever a second package explicitly needs those components.
For instance if package A only needs `+component_a` it might depend on
`boost +component_a ~component_b`. And if packge B only needs
`+component_b` it might depend on `boost ~component_a +component_b`. If
package C now depends on both A and B, this leads to unsatisfiable
variants and hence a concretization error.
However, if we default to disabling all components, package A can simply
depend on `boost +component_a` and package B on `boost +component_b` and
package C will concretize to depending on `boost +component_a
+component_b`, and whatever you install, you get the bare minimum.
* Fix style
* Added composable boost dependencies for folly
* fixing akantu merge issue
* hpctoolkit boost dependencies already defined
* Fix Styles
* Fixup style once more
* Adding isort fix
* isort one more time
* Fix for package audit issue
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Ryan O'Malley <rd.omalley@comcast.net>
Consolidate Spack's internal filepath logic to a select
few places and refactor to consistent internal useage of
os.path utilities. Creates a prefix, and a series of utilities
in the path utility module that facilitate handling paths
in a platform agnostic manner.
Convert Windows paths to posix paths internally
Prefer posixpath.join instead of os.path.join
Updated util/ directory to account for Windows integration
Co-authored-by: Stephen Crowell <stephen.crowell@khq.kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
Module template format for windows (#23041)
* Incorporate new search location
* Add external user option
* proper doc string
* Explicit commands in getting started
* raise during chgrp on Win
recover installer changes
Notate admin privleges
Windows phase install hooks
Find external python and install ninja (#23496)
Allow external find python to find windows python and spack install ninja
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
Fixup common tests
* Remove requirement for Python 2.6
* Skip new failing test
Windows: Update url util to handle Windows paths (#27959)
* update url util to handle windows paths
* Update tests to handle fixed url handling
* canonicalize path only when the path type matches the host platform
* Skip some url tests on Windows
Co-authored-by: Omar Padron <omar.padron@kitware.com>
Use threading.TIMEOUT_MAX when available (#24246)
This value was introduced in Python 3.2. Specifying a timeout greater than
this value will raise an OverflowError.
Co-authored-by: Lou Lawrence <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
Add compiler hint to the root spec for Windows
Reporters on Windows (#26038)
Reporters use Jinja2 as the templating engine, and Jinja2 indexes
templates by Unix separators, even on Windows, so search using Unix paths
on all systems.
Support patching on win via git (#25871)
Handle GRP on windows
CMake - Windows Bootstrap (#25825)
Remove hardcoded cmake compiler (#26410)
Revert breaking cmake changes
Ensure no autotools on Windows
Perl on Windows (#26612)
Python source build windows (#26313)
Reconfigure sysconf for Windows
Python2.6 compatibility
Fxixup new sbang tests for windows
Ruby support (#28287)
Add NASM support (#28319)
Add mock Ninja package for testing
* Style fixes
* Use Python's zipfile, if available
The compression libs are optional in Python. Rely on python as a
first attempt then fall back to `unzip`
MSVC's internal CMake and Ninja now detected by spack external find and added to packages.yaml
Saving progress on packaging zlib for Windows
Fixing the shared CMake flag
* Loading Intel's ifx Fortran compiler into MSVC; if there are multiple
versions of MSVC installed and detected, ifx will only be placed into
the first block written in compilers.yaml. The version number of ifx can
be detected using MSVC's version flag (instead of /QV) by using
ignore_version_errors. This commit also provides support for detection
of Intel compilers in their own compiler block by adding ifx.exe to the
fc/f77_name blocks inside intel.py
* Giving CMake a Fortran compiler argument
* Adding patch file for removing duplicated mangling header for versions 3.9.1 and older; static and shared now successfully building on Windows
* Have netlib-lapack depend on ninja@1.10
Co-authored-by: John R. Cary <cary@txcorp.com>
Co-authored-by: Jared Popelar <jpopelar@txcorp.com>
Making a default config.yaml for Windows
Small path length for build_stage
Provide more prerequisite details, mention default config.yaml
Killing an unnecessary setvars call
Replacing some lost changes, proofreading, updating windows-supported package list
Co-authored-by: John Parent <john.parent@kitware.com>
* Add 'make-installer' command for Windows
* Add '--bat' arg to env activate, env deactivate and unload commands
* An equivalent script to setup-env on linux: spack_cmd.bat. This script
has a wrapper to evaluate cd, load/unload, env activate/deactivate.(#21734)
* Add spacktivate and config editor (#22049)
* spack_cmd: will find python and spack on its own. It preferentially
tries to use python on your PATH (#22414)
* Ignore Windows python installer if found (#23134)
* Bundle git in windows installer (#23597)
* Add Windows section to Getting Started document
(#23131), (#23295), (#24240)
Co-authored-by: Stephen Crowell <stephen.crowell@kitware.com>
Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
Co-authored-by: Jared Popelar <jpopelar@txcorp.com>
Co-authored-by: Ben Cowan <benc@txcorp.com>
Update Installer CI
Co-authored-by: John Parent <john.parent@kitware.com>
Made the vcvars batch script location a member variable of the msvc compiler subclass, initialized from the compiler executable path. Added a setup_custom_environment() method to the msvc subclass that sources the vcvars script, dumps the environment, and copies the relevant environment variables to the Spack environment. Added class variables to the Windows OS and MSVC compiler subclasses to enable finding the compiler executables and determining their versions.
* Fixed path and uid issues.
* Added needed import statement; kluged .exe extension.
* Got package to build. Some manual intervention necessary, including sourcing the MSVC setup script and having certain configuration parameters.
* Removed CMake executable suffix hack.
To provide Windows-compatible functionality, spack code should use
llnl.util.symlink instead of os.symlink. On non-Windows platforms
and on Windows where supported, os.symlink will still be used.
Use junctions when symlinks aren't supported on Windows (#22583)
Support islink for junctions (#24182)
Windows: Update llnl/util/filesystem
* Use '/' as path separator on Windows.
* Recognizing that Windows paths start with '<Letter>:/' instead of '/'
Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
os.rename() fails on Windows if file already exists.
Create getuid utility function (#21736)
On Windows, replace os.getuid with ctypes.windll.shell32.IsUserAnAdmin().
Tests: Use getuid util function
Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
1. Forwarding sys.stdin, e.g. use input_multiprocess_fd,
gives an error on Windows. Skipping for now
3. subprocess_context needs to serialize for Windows, like it does
for Mac.
Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
* Snapshot of some MSVC infrastructure added during experiments a while ago. Rebasing from spack/develop.
* Added platform and OS definitions for Windows.
* Updated Windows platform file to conform to new archspec use.
* Added Windows as a platform; introduced some debugging code.
* Added type annotations.
* Fixed copyright.
* Removed print statements.
* Ensure `spack arch` returns correctly on Windows (#21428)
* Correctly identify windows as 'windows-Windows10-AMD64'
* python: allow versions with garbage suffix
Ubuntu 22.04 preview python prints version as 3.10.2+, the + causes
version parsing to fail and breaks detection.
* Add version comment
* match VALID_VERSION regex
* libiconv: compile with pic even when static build
* lmod: require shared lua
It seems to be unable to detect lua-posix when using a static lua:
```
Error: The follow lua module(s) are missing: posix
```
Re-work the checks and comparisons around commit versions, when no
commit version is involved the overhead is now in the noise, where one
is the overhead is now constant rather than linear.
develop in the version string. The versions from the HDF5 code were not
matching because 'develop-' is not part of the HDF5 version. Also, the
develop-x.x versions in spack omit the release version (third) number
because the branch spans all of the release versions.
* Update: py-cmake
Add additional dependencies as declared by the `py-cmake` repository.
Note: for either from-source or from-binary builds, this downloads
additional software via the network. We might want to propose upstream
patches to make this work on nodes without internet connection.
* Add Review Comments + Newest Version
* Add: Ninja
Preferred generator according to outputs and upstream repo logic
* Attempt to use resource() for CMake source
* [py-watchdog] switched to pypi and audited dependencies
* [py-watchdog] added version 2.1.6
* [py-watchdog] updated dependencies for old versions
* [py-watchdog] added when for variant
* [py-watchdog] added some newlines to make flake8 happy
* hsa-rocr-dev, llvm-amdgpu: change dependency libelf to elf
Change the libelf dependency to the virtual elf for two rocm packages.
This allows other packages (hpctoolkit) to combine rocm and dyninst
(with elfutils) while still being able to build rocm with libelf when
needed, eg darwin.
* add comment describing include path for libelf vs elfutils
fixes#29446
The new setup_*_environment functions have been falling back
to calling the old functions and warn the user since #11115.
This commit removes the fallback behavior and any use of:
- setup_environment
- setup_dependent_environment
in the codebase
Change the internal representation of `Spec` to allow for multiple dependencies or
dependents stemming from the same package. This change permits to represent cases
which are frequent in cross compiled environments or to bootstrap compilers.
Modifications:
- [x] Substitute `DependencyMap` with `_EdgeMap`. The main differences are that the
latter does not support direct item assignment and can be modified only through its
API. It also provides a `select_by` method to query items.
- [x] Reworked a few public APIs of `Spec` to get list of dependencies or related edges.
- [x] Added unit tests to prevent regression on #11983 and prove the synthetic construction
of specs with multiple deps from the same package.
Since #22845 went in first, this PR reuses that format and thus it should not change hashes.
The same package may be present multiple times in the list of dependencies with different
associated specs (each with its own hash).
* The new version of Wonton requires the new version of Jali
* Wonton: versions after 1.2.10 don't require boost at all
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* environment.py: allow link:run
Some users want minimal views, excluding run-type dependencies, since
those type of dependencies are covered by rpaths and the symlinked
libraries in the view aren't used anyways.
With this change, an environment like this:
```
spack:
specs: ['py-flake8']
view:
default:
root: view
link: run
```
includes python packages and python, but no link type deps of python.
* ECP-SDK/VTK-m: Update ROCm variant
VTK-m set contraint for when rocm/kokkos are available.
SDK Make ROCmPackage and propagate amdgpu_arch and rocm variant to
VTK-m.
Note: SDK has to check vtk-m@ 1.7: and :1.6 explicitly in orderer to have 1.7
be selected by default if +rocm in the SDK.
* ECP-SDK: Enable ROCm + VTK-m constraints
* Adding Panzer as Default
* Set Panzer as non-default
* Updated the conflict for Panzer.
* Updated the conflict for Panzer.
* Resolve the issue with Stratimikos and Thyra
* Fixing stk build issues.
* Fixing stk build issues.
* Adding another conflict for Thrya
* cray-libsci: only be a provider for scalapack with +mpi
If a package explicitly links the scalapack provider we might otherwise end up with different variants of libsci being linked: the explicitly linked one and the one added by the Cray compiler wrappers.
* cp2k: require cray-libsci+openmp with +openmp for consistency
otherwise we might get 2 different libsci linked: one explicitly, the other one via the Cray compiler wrappers, leading at least to segfaults during cleanup
* cp2k: depend on cray-fftw+openmp with +openmp
* hdf5: mark +fortran+shared conflict for older version
This version was only activated unintentionally by silo's conflict
statement, but `@1.8.15+shared+fortran+cxx` errors out in configure:
```
CMake Error at CMakeLists.txt:814 (message):
**** Shared FORTRAN libraries are unsupported ****
```
* silo: refine hdf5 conflicts to avoid building old version
Before this, `silo+hdf5` concretized to 1.10.7 or sometimes 1.8.15. Now
I've verified it works for the following configurations:
```
silo@4.10.2 patches=7b5a1dc,952d3c9
^ hdf5@1.10.7 api=default
silo@4.10.2 patches=7b5a1dc,952d3c9,eb2a3a0
^ hdf5@1.10.8 api=v18
silo@4.10.2 patches=7b5a1dc,952d3c9,eb2a3a0
^ hdf5@1.12.1 api=v110
silo@4.11-bsd patches=eb2a3a0
^ hdf5@1.12.1 api=v110
silo@4.11-bsd patches=eb2a3a0
^ hdf5@1.10.8 api=default
silo@4.11-bsd patches=eb2a3a0
^ hdf5@1.12.1 api=default
```
and verified that the following fail:
```
silo@4.10.2 ^hdf5@1.12.1 api=default
silo@4.11 ^hdf5 api=v18
silo@4.11-bsd ^hdf5@1.13.0 api=v12
silo@4.11-bsd ^hdf5@1.13.0 api=default
```
and have updated the constraints to match. Hdf5 no longer has to be
downgraded to work with Silo.
* silo: fix dependency conflicts
* py-h5py: shorten and add comments to py-h5py hdf5 dependencies
* e4s: remove slightly outdated hdf5 requirement
* e4s: remove excessive hdf5 variant constraints
These I think are holdovers from the old concretizer.
- `hdf5_compat` can be expressed as `+hdf5 ^hdf5@1.8`
- The extra variants on hdf5 shouldn't break conduit
- axom unnecessarily restricts hdf5 version
* conduit: restore hdf5_compat flag
New versions don't try to configure docs targets at all when the
BUILD_DOCS option is turned off. This avoids CMake warnings
when docs dependencies are not found.
Speeds up comparison on `Version` by ~2.5x, e.g.
```python
In [1]: v = spack.version.Version('1.0.0'); w = spack.version.Version('1.0.2')
In [2]: %timeit v < w
1.47 µs ± 5.59 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
535 ns ± 1.75 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```
* Bugfix in var/spack/repos/builtin/packages/esmf/package.py
* Bug fixes in var/spack/repos/builtin/packages/esmf/package.py to build ESMF on macOS with clang+gfortran and on cray
* Add maintainer to var/spack/repos/builtin/packages/esmf/package.py
* Fix style errors
* Fix more style errors
* py-jupytext: add version 0.13.6
From da3fcc305d:
markdown-it-py v2.0 implements some internal changes, but won't affect jupytext
* py-jupytext: keep mdit-py version restricted to 1
* py-jupytext: update dependencies
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add HiOp v0.5.4, update magma constraint
* Add v2.6.2rc1 to magma, make hiop depend on it
* Update var/spack/repos/builtin/packages/hiop/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The 'multicore' backend always uses SMP, so reverse
the logic of the `conflict` clause. This resolves an issue
where the '+smp' default caused the 'backend' to switch
away from 'multicore' unintentionally (#29234).
fixes#29203
This PR fixes a subtle bug we have when importing
Spack packages as Python modules that can lead to
multiple module objects being created for the same
package.
It also fixes all the places in unit-tests where
"relying" on the old bug was crucial to have a new
"clean" state of the package class.
This commit reverts the GCS fetch strategy to before commit:
d759612523
The previous commit added some s3 syntax to handle connections, but
added them into the GCS fetch strategy in a way that prevents GCS from
working anymore.
* rocmcc compiler: initial commit based on aocc and clang
Co-authored-by: luker <luke.roskop@hpe.com>
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
Recipes that are not actually required for LBANN or DiHydrogen to
build. These should be concretized within the same environment or
installed via PIP using the same Python that installed LBANN.
Removing these will help eliminate build time failures that are
actually associated with Python tools, not LBANN.
The status displayed in the terminal title could be wrong when doing
distributed builds. For instance, doing `spack install glib` in two
different terminals could lead to the current package being reported as
`40/29` due to the way Spack handles retrying locks.
Work around this by keeping track of the package IDs that were already
encountered to avoid counting packages twice.
* HIP: Change mesa18 dep to gl
* Mesa: Conflict with llvm-amdgpu when +llvm and swr
* Add def for suffix
* Disable llvm suffix patch.
* LLVM: Remove version suffix patches
* ECP-SDK: ParaView 5.11: required for CUDA
* Add conflict with ParaView@master
Because of the additional constraints for cuda, ParaView@master may be
selected unintentionally. Prefer older versions of ParaView without cuda
to master with cuda.
* hypre: Add releases 2.21.0 and 2.22.0
* Revert "hypre: Add releases 2.21.0 and 2.22.0"
This reverts commit 8921cdb3ac.
* Address external linkage failures in elfutils 0.185:
https://bugs.gentoo.org/794601https://sourceware.org/pipermail/elfutils-devel/2021q2/003862.html
Encountered while building within a Spack environment.
* Revert "Address external linkage failures in elfutils 0.185:"
This reverts commit 76b93e4504.
* paraview: The ninja generator has problems with XL and CCE
See https://gitlab.kitware.com/paraview/paraview/-/issues/21223
* paraview: Add variant to allow choice of cmake generator.
This will be necessary until problems with cmake+ninja on XL and
CCE builds can be resolved.
See https://gitlab.kitware.com/paraview/paraview/-/issues/21223
* paraview: ninja generator problems with XL/CCE
By popular preference, abandon the idea of a special variant
and select the generator based on compiler.
* Greg Becker suggested using the dedicated "generator" method to
pass the choice of makefile generator to cmake.
* paraview: The build errors I saw before with paraview%cce + ninja
have not reappeared in subsequent testing, so I'm dropping it from this
PR. If they re-occur I'll report the issue separately to KitWare.
* py-nbclassic: add 0.3.5
* Update var/spack/repos/builtin/packages/py-nbclassic/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fix style
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add a new test to catch exit code failure
fixes#29226
This introduces a new unit test that checks the return
code of `spack unit-test` when it is supposed to fail.
This is to prevent bugs like the one introduced in #25601
in which CI didn't catch a missing return statement.
In retrospective it seems that the shell test we have right
now all go through `tty.die` or similar code paths which
call `sys.exit(a)` explicitly. This new test instead checks
`spack unit-test` which relies on the return code from
command invocation in case of errors.
* Add 'develop' version for dmtcp
* Update var/spack/repos/builtin/packages/dmtcp/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The `spack external find binutils` command was failing to find my system
binutils because the regex was not matching. The name of the executable
follows the string 'GNU' that I tested with three different
installations so I changed the regex to look for that. On my CentOS-7
system, the version had the RPM details so I set the version to capture
the first three parts of the version.
The system compiler on RHEL7 fails to build the latest linux-uuid.
```
util-linux-uuid@2.37.4%gcc@4.8.5 arch=linux-rhel7-haswell
```
results in:
```
libuuid/src/unparse.c:42:73: error: expected ';', ',' or ')' before 'fmt'
static void uuid_fmt(const uuid_t uuid, char *buf, char const *restrict fmt)
```
It looks like it's assuming C99 by default so there may be a better way
to handle this... but this at least works
See https://github.com/spack/spack/pull/28468/files#r809156986
If we exit before generating the:
error("Dependencies must have compatible OS's with their dependents").
...
facts we'll output a problem that is effectively
different by the one solved by clingo.
* cmd/checksum: prefer url matching url_from_version
This is a minimal change toward getting the right archive from places
like github. The heuristic is:
* if an archive url exists, take its version
* generate a url from the package with pkg.url_from_version
* if they match
* stop considering other URLs for this version
* otherwise, continue replacing the url for the version
I doubt this will always work, but it should address a variety of
versions of this bug. A good test right now is `spack checksum gh`,
which checksums macos binaries without this, and the correct source
packages with it.
fixes#15985
related to #14129
related to #13940
* add heuristics to help create as well
Since create can't rely on an existing package, this commit adds another
pair of heuristics:
1. if the current version is a specifically listed archive, don't
replace it
2. if the current url matches the result of applying
`spack.url.substitute_version(a, ver)` for any a in archive_urls,
prefer it and don't replace it
fixes#13940
* clean up style and a lingering debug import
* ok flake8, you got me
* document reference_package argument
* Update lib/spack/spack/util/web.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* try to appease sphinx
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
We can see what is in the bootstrap store with `spack find -b`, and you can clean it with `spack
clean -b`, but we can't do much else with it, and if there are bootstrap issues they can be hard to
debug.
We already have `spack --mock`, which allows you to swap in the mock packages from the command
line. This PR introduces `spack -b` / `spack --bootstrap`, which runs all of spack with
`ensure_bootstrap_configuration()` set. This means that you can run `spack -b find`, `spack -b
install`, `spack -b spec`, etc. to see what *would* happen with bootstrap configuration, to remove
specific bootstrap packages, etc. This will hopefully make developers' lives easier as they deal
with bootstrap packages.
This PR also uses a `nullcontext` context manager. `nullcontext` has been implemented in several
other places in Spack, and this PR consolidates them to `llnl.util.lang`, with a note that we can
delete the function if we ever reqyire a new enough Python.
- [x] introduce `spack --bootstrap` option
- [x] consolidated all `nullcontext` usages to `llnl.util.lang`
* py-imageio: add 2.16.0
* Update var/spack/repos/builtin/packages/py-imageio/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Some "concrete" versions on the command line, e.g. `qt@5` are really
meant to satisfy some actual concrete version from a package. We should
only assume the user is introducing a new, unknown version on the CLI
if we, well, don't know of any version that satisfies the user's
request. So, if we know about `5.11.1` and `5.11.3` and they ask for
`5.11.2`, we'd ask the solver to consider `5.11.2` as a solution. If
they just ask for `5`, though, `5.11.1` or `5.11.3` are fine solutions,
as they satisfy `@5`, so use them.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
* geant4-data: use build+run-only depends
* geant4: point to dependent datadir
This is "used" in the configure step to set up the Geant4Config.cmake
file's persistent pointers to the data directory, but the dependency
is still listed as "run" -- though I'm not sure this is the right behavior
since the geant4 installation really does change as a function of the
data directory, and the installation is incomplete/erroneous
without using one.
* Style
* trilinos: disable dl on macOS
* py-sphinx-argparse: add explicit poetry dependency
* libzmq: fix libbsd dependency
libbsd is *always* required when +libbsd (introduced in #28503) . #20893
had previously removed the macos dependency because libbsd wasn't always
enabled. Libbsd support is only available after 4.3.2 so change it to a
conflict rather than bumping the dependency.
* hdf5: work around GCC11.2 monterey fortran bug
* go-bootstrap: mark conflict for monterey
* py-tensorflow: add versions 2.5.0 and 2.6.0
- add version 2.5.0
- add version 2.6.0
- add patches for newer protobuf
- set constraints
* Remove import os. left over from testing
* Remove unused patch file
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add py-clang dependency
* Adjust py-clang constraint
* Build tensorflow with tensorboard
- tensorflow
- added 2.6.1 and 2.6.2 versions
- tensorboard
- have bazel use number of jobs set by spack
- add versions and constraints
- new package: py-tensorboard-data-server
- use wheel for py-tensorboard-plugin-wit
This package can not build with newer versions of bazel that are
needed for newer versions of py-tensorboard.
* Update var/spack/repos/builtin/packages/py-clang/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Remove empty line at end of file
* Fix import sorting
* Adjust python dependencies on py-clang
* Add version 2.7.0 of pt-tensorflow and py-tensorboard
* Adjust bazel constraints
* bazel-4 support begins with py-tensorflow-2.7.0
* Adjust dependencies
* Loosen cuda constraint on versions > 2.5
Tensorflow-2.5 and above can use cuda up to version 11.4.
* Add constraints to patch
The 0008-Fix-protobuf-errors-when-using-system-protobuf.patch patch
should only apply to versions 2.5 and above.
* Adjust constraints
- versions 2.4 and below need protobuf-3.12 and below
- versions 2.4 and above can use up to cuda-11.4
- versions 2.2 and below can not use cudnn-8
- the null_linker_bin patch should only be applied to versions 2.5 and
above.
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fix py-grpcio dependency for version 2.7
Also, make sure py-h5py mpi specs are consistent.
* Add llvm as run dependency.
* Fix python spec for py-tensorboard
* Fix py-google-auth spec for py-tensorboard
* Do not override the pip spec for tensorboard-plugin-wit
* Converted py-tensorboard-plugin-wit to wheel only package
* Fix bazel dependency spec in tensorflow
* Adjust pip masks
- allow tensorboard to be specified in pip constraints
- mask tensorflow-estimator
* Remove blank line at end of file
* Adjust pip constraints in setup.py
Also, adjust constraint on a patch that is fixed in 2.7
* Fix flake8 error
Adjust formatting for consistency.
* Get bazel dep right
* Fix old cudnn dependency, caught in audit test
* Adjust the regex to ensure proper line is changed
* Add py-libclang package
- Stripped the py-clang package down to just version 5
- added comments to indicate the purpose of py-clang and that
py-libclang should be preferred
- set dependencies accordingly in py-tensorflow
* Remove cap on py-h5py dependency for v2.7
* Add TODO entries for tensorflow-io-gcs-filesystem
* Edit some comments
* Add phases and select python in PATH for tensorboard-data-server
* py-libclang
- remove py-wheel dependency
- remove raw string notation in filter_file
* py-tensorboard-data-server
- remove py-wheel dep
- remove py-pip dep
- use python from package class
* py-tensorboard-plugin-wit
- switch to PythonPackage
- add version 1.8.1
- remove unneeded code
* Add comment as to why a wheel is need for tensorboard-plugin-wit
* remove which pip from tensorboard-data-server
* Fix dependency specs in tensorboard
* tweak dependencies for tensorflow
* fix python constraint
* Use llvm libs property
* py-tensorboard-data-server
- merge build into install
- use std_pip_args
* remove py-clang dependency
* remove my edits to py-tensorboard-plugin-wit
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
See https://github.com/spack/spack/issues/25353#issuecomment-1041868116
This commit changes the default behavior of
```
$ spack external find
```
from searching all the possible packages Spack knows about to
search only for the ones tagged as being a "build-tool".
It also introduces a `--all` option to restore the old behavior.
Prefer `sw_vers` to `platform.mac_ver`. In anaconda3 installation, for example, the latter reports 10.16 on Monterey -- I think this is affected by how and where the python instance was built.
Use MACOSX_DEPLOYMENT_TARGET if present to override the operating system choice.
It will be useful for metrics gathering and possibly debugging to
have this environment variable available in the runner pods that
do the actual rebuilds.
Since Spack does not install external packages, this commit skips them by
default when running stand-alone tests. The assumption is that such packages
have likely undergone an acceptance test process.
However, the tests can be run against installed externals using
```
% spack test run --externals ...
```
fixes#28260
Since we iterate over different variants from many packages, the variant
values may have types which are not comparable, which causes errors
at runtime. This is not a real issue though, since we don't need the facts
to be ordered. Thus, to avoid needless sorting, the sorted function has
been removed and a comment has been added to tip any developer that
might need to inspect these clauses for debugging to add back sorting
on the first two items only.
It's kind of difficult to add a test for this, since the error depends on
whether Python sorting algorithm ever needs to compare the third
value of a tuple being ordered.
* extensions: allow multiple "extends" directives
This will allow multiple extends directives in a package as long as only one of
them is selected as a dependency in the concrete spec.
* document the option to have multiple extends
Reuse previously was a very invasive change that required parameters to be added to all
the methods that called `concretize()` on a `Spec` object. With the addition of
concretizer configuration, we can use the config system to simplify this argument
passing and keep the code cleaner.
We decided that concretizer config options should be read at `Solver` instantiation
time, and if config changes between instnatiation of a particular solver and
`solve()` invocation, the `Solver` should use the settings from `__init__()`.
- [x] remove `reuse` keyword argument from most concretize functions
- [x] refactor usages to use `spack.config.override("concretizer:reuse", True)`
- [x] rework argument passing in `Solver` so that parameters are set from config
at instantiation time
`--reuse` was previously handled individually by each command that
needed it. We are growing more concretization options, and they'll
need their own section for commands that support them.
Now there are two concretization options:
* `--reuse`: Attempt to reuse packages from installs and buildcaches.
* `--fresh`: Opposite of reuse -- traditional spack install.
To handle thes, this PR adds a `ConfigSetAction` for `argparse`, so
that you can write argparse code like this:
```
subgroup.add_argument(
'--reuse', action=ConfigSetAction, dest="concretizer:reuse",
const=True, default=None,
help='reuse installed dependencies/buildcaches when possible'
)
```
With this, you don't need to add logic to pull the argument out and
handle it; the `ConfigSetAction` just does it for you. This can probably
be used to clean up some other commands later, as well.
Code that was previously passing `reuse=True` around everywhere has
been refactored to use config, and config is set from the CLI using
a new `add_concretizer_args()` function in `spack.cmd.common.arguments`.
- [x] Add `ConfigSetAction` to simplify concretizer config on the CLI
- [x] Refactor code so that it does not pass `reuse=True` to every function.
- [x] Refactor commands to use `add_concretizer_args()` and to pass
concretizer config using the config system.
Config scopes were different for `config` and `mutable_config`,
and `mutable_config` did not have a command line scope.
- [x] Fix by consolidating the creation logic for the two fixtures.
The concretizer is going to grow to have many more configuration,
and we really need some structured config for that.
* We have the `config:concretizer` option that chooses the solver,
but extending that is awkward (we'd need to replace a string with
a `dict`) and the solver choice will be deprecated eventually.
* We have the `concretization` option in environments, but it's
not a top-level config section -- it's just for environments,
and it also only admits a string right now.
To avoid overlapping with either of these and to allow the most
extensibility in the future, this adds a new `concretizer` config
section that can be used in and outside of environments. There
is only one option right now: `reuse`. This can expand to include
other options later.
Likely, we will soon deprecate `config:concretizer` and warn when
the user doesn't use `clingo`, and we will eventually (sometime later)
move the `together` / `separately` options from `concretization` into
the top-level `concretizer` section.
This commit just adds the new section and schema. Fully wiring it
up is TBD.
The solver has a lot of configuration associated with it. Rather
than adding arguments to everything, we should encapsulate that
in a class. This is the start of that work; it replaces `solve()`
and its kwargs with a class and properties.
* Add 'stable' to the list of infinity version names.
Rename libunwind 1.5-head to 1.5-stable.
* Add stable to the infinite version list in packaging_guide.rst.
* py-etelemetry: add 0.3.0
* Update var/spack/repos/builtin/packages/py-etelemetry/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
No version of py-nbconvert@5: can be concretized due to conflicting versions
of flit-core that are required. This issue could be solved by separate
concretization of build deps.
* archspec: remove pyproject.toml to workaround PEP517
If pyproject.toml is in the folder, that is preferred to the
setup.py packaged by poetry itself. Adding a dependency on
poetry for deploying a pure Python package seems wasteful,
since the package to be deployed just needs to be copied in
place, so we don't want to built rust for that.
* archspec: patch pyproject.toml to comply to PEP517
See https://python-poetry.org/docs/pyproject/#poetry-and-pep-517
* Fix style issues
The new HDF5 version 1.12 API causes compiler errors due to modified function prototypes. Note that version 1.11 is the development version of HDF5 1.12.
* py-numba: add 0.55.1
* Remove comment
* Pin down py-llvmlite version for older py-numba releases
* Remove py-llvmlite deps for releases not in spack
* Set upper bounds for python and py-numpy
* Add stricter upper bound to py-numpy for releases <=0.47
Setting Spack's `$prefix` to `$DESTDIR` and not to `$PREFIX` install the
package in `$prefix/usr/local` and not in `$prefix`, thus when it is
loaded the executable `direnv` in not "seen" by the environment.
* Added support to LBANN, Hydrogen, DiHydrogen, and Aluminum to capture
a gcc-toolchain cxxflags argument and pass it to a CMAKE_CUDA_FLAG
argument when set. This helps deal with compiling with clang on
systems with old base gcc installations.
* Added a dependency on py-scipy when enabling tests on LBANN.
* Updated the C++ standard for Hydrogen to C++17.
* Added a new variant +apps to enable (or disable) python packages that
are used by applications in the LBANN repo, but are not strictly
required for building and using LBANN.
* Added a run time dependency for both py-pytest and py-scipy so that
they are activated in any environment.
* Added support for building LBANN, Hydrogen, and DiHydrogen with the
IBM ESSL BLAS library. This requires explicit identification of
additional LAPACK libraries, since ESSL does not implement LAPACK, but
is found by CMake.
* Fixed a bug in the LBANN dependency on OpenCV for Power architectures.
The +powerpc variant is only required for GCC toolchains and causes
Clang to break. Switched to only enabling when using %gcc on power.
- Installation often hangs building the documentation. This happens when
doxygen and latex are found. To avoid the issue, comment-out that part
of the code until an explicit cmake variable to disable documentation
generation is available.
* sundials: fix smoke tests
* sundials: add new version
* use cmake+make instead of make for tests, fix style
* use cmake_bin workaround from https://github.com/spack/spack/pull/28622
Note that the SDK is not the same as the system version: using
apple-clang@13 is a better match than `os=monterey` since this actually
fails on bigsur as well, as long as xcode 13 is being used.
* core: Make platform environment an instance not class method
In preparation for accessing data constructed in __init__.
* macos: set consistent macosx deployment target
This should silence numerous warnings from mixed gcc/macos toolchains.
* perl: prevent too-new deployment target version
```
*** Unexpected MACOSX_DEPLOYMENT_TARGET=11
***
*** Please either set it to a valid macOS version number (e.g., 10.15) or to empty.
```
* Stylin'
* Add deployment target overrides to failing autoconf packages
* Move configure workaround to base autoconf package
This reverts commit 3c119eaf8b4fb37c943d503beacf5ad2aa513d4c.
* Stylin'
* macos: add utility functions for SDK
These aren't yet used but should probably be added to spack debug
report.
* Remove node_target_satisfies/3 in favor of target_satisfies/2
When emitting input facts we don't need to couple target with
packages, but we can emit fewer facts independently and let
the grounder combine them.
* Remove compiler_version_satisfies/4 in favor of compiler_version_satisfies/3
When emitting input facts we don't need to couple compilers with
packages, but we can emit fewer facts independently and let
the grounder combine them.
* Introduce heuristic in the ASP-program
With heuristic we can drive clingo to make better
initial guesses, which lead to fewer choices and
conflicts in the overall solve
This improves the stand-alone tests for slate by providing most
of the dependencies to the test framework and enabling stand-alone
tests on all versions except the oldest.
* AMReX: +tiny_profile
The tiny profiler options in AMReX are by default off but needed
by WarpX. Adds a new variant to control it.
* Add Erik Palmer as Co-Maintainer
... so he receives pings on updates of the package for review.
The version of the ONNX submodule was updated between the PyTorch
1.9 and 1.10 releases, which fixed builds with newer protobuf but
broke builds with older protobuf.
Also this adds minimum version reqs for numpy/typing-extensions
(which were not present before).
* gcc: revise patch range on darwin
* gcc: add conflict to work around bootstrap failure
closes#23296 . See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100340
.
```
Comparing stages 2 and 3
Bootstrap comparison failure!
gcc/tree-ssa-operands.o differs
gcc/tree-ssanames.o differs
gcc/ipa-inline.o differs
gcc/tree-ssa-pre.o differs
gcc/gimple-loop-interchange.o differs
...
```
639 total differences.
* gcc: bump conflict up to correct later version
* Fix reindex with uninstalled deps
When a prefix of a dep is removed, and the db is reindexed, it is added
through the dependent, but until now it incorrectly listed the spec as
'installed'.
There was also some questionable behavior in the db when the same spec
was added multiple times, it would always be marked installed.
* Always reserve path
* Only add installed spec's prefixes to install prefixes set
* Improve warning, and ensure ensure only ensures
* test: reindex with every file system remnant removed except for the old index; it should give a database with nothing installed, including records with installed==False,external==False,ref_count==0,explicit=True, and these should be removable from the database
* stacks: add regression tests for matrix expansion
* Use constrain semantics to construct spec lists for stacks
* Fix semantics for constraining an anonymous spec. Add tests
Since in Spack we pull binaries out of the `warpx` package, we don't
need `py-cmake` to build `py-warpx`.
Generally, `py-cmake` in `pyproject.toml` is just a mean for us to
tell `pip` to make a `cmake` CLI tool available.
* added package gptune with all its dependencies: adding py-autotune, pygmo, py-pyaml, py-autotune, py-gpy, py-lhsmdu, py-hpbandster, pagmo2, py-opentuner; modifying superlu-dist, py-scikit-optimize
* adding gptune package
* minor fix for macos spack test
* update patch for py-scikit-optimize; update test files for gptune
* fixing gptune package style error
* fixing unit tests
* a few changes reviewed in the PR
* improved gptune package.py with a few newly added/improved dependencies
* fixed a few style errors
* minor fix on package name py-pyro4
* fixing more style errors
* Update var/spack/repos/builtin/packages/py-scikit-optimize/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* resolved a few issues in the PR
* fixing file permissions
* a few minor changes
* style correction
* minor correction to jq package file
* Update var/spack/repos/builtin/packages/py-pyro4/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixing a few issues in the PR
* adding py-selectors34 required by py-pyro4
* improved the superlu-dist package
* improved the superlu-dist package
* moree changes to gptune and py-selectors34 based on the PR
* Update var/spack/repos/builtin/packages/py-selectors34/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* improved gptune package: 1. addressing comments of tldahlgren in PR 26936; 2. adding variant openmpi
* fixing style issue of gptune
* changing file mode
* improved gptune package: add variant mpispawn which depends on openmpi; add variant superlu and hypre for installing the drivers; modified hypre package file to add a gptune variant
* fixing style error
* corrected pddrive_spawn path in gptune test; enforcing gcc>7
* fixing style error
* setting environment variables when loading gptune
* removing debug print in hypre/package.py
* adding superlu-dist v7.2.0; fixing an issue with CMAKE_INSTALL_LIBDIR
* changing site_packages_dir to python_platlib
* not using python3.9 for py-gpy, which causes due to dropped support of tp_print
* more replacement of site_packages_dir
* fixing a few dependencies in gptune; added a gptune version
* adding url for gptune
* minor correction of gptune
* updating versions in butterflypack
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [py-xonsh] added py-xonsh package
* [py-xonsh] change dependency to python 3.6
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add sticky variants
* Add unit tests for sticky variants
* Add documentation for sticky variants
* Revert "Revert 19736 because conflicts are avoided by clingo by default (#26721)"
This reverts commit 33ef7d57c1.
* Add stickiness to "allow-unsupported-compiler"
- To retrieve the correct spack version we need to get it from the git
repo.
- Recommend installing the package with root for production
- Add Tomas as maintainer to sarus' spack package
- Add the option to disable unit tests in latest versions
Fixes the following build failure when building with gcc 11:
478 ../../../../CPP/7zip/Archive/Wim/WimHandler.cpp: In member function 'virtual LONG NArchive::NWim::CHandler::GetArchiveProperty(PROPID, PROPVARIANT*)':
>> 479 ../../../../CPP/7zip/Archive/Wim/WimHandler.cpp:308:11: error: use of an operand of type 'bool' in 'operator++' is forbidden in C++17
480 308 | numMethods++;
481 | ^~~~~~~~~~
>> 482 ../../../../CPP/7zip/Archive/Wim/WimHandler.cpp:318:9: error: use of an operand of type 'bool' in 'operator++' is forbidden in C++17
483 318 | numMethods++;
484 | ^~~~~~~~~~
* opencv: add new version, variant, and patch
- added version 4.5.4
- added tesseract variant
- added patch to not add system paths
* Add leptonica depends and contrib conflicts
* Add dependencies for 1394 support
- new package: libraw1394
- add sdl dependency to libdc1394
- add conflict for openjpeg and jasper
* Adjust dependencies and conflicts for opencv modules
* rewrite of opencv
- all prebuilt apps are now variants and can be installed
- core is no longer a variant. It was always built anyway so it was not
really a variant.
- contrib is no longer a variant. All of the contrib modules are now
available as variants.
- components that can not be built with Spack are no longer variants.
They are set to 'off' to prevent pulling from system.
- handle the case where a module and a component have the same name
- use `with when` framework
- adjust dependencies and conflicts
- new package: libraw1394
- have libdc1394 depend on libraw1394
- patch to find clp
- patch to find onnx
- patch for cvv to find Qt
- format with black
* Incorporate recommended changes
- fix variants and dependencies on packages that depend on opencv
- remove opencv-3.2 and patches
- add some new patches to handle different versions
- cntk needs further work
- the openvslam package was markde deprecated as it is no longer an
active project and the repository has no code
* Remove gmake dependency.
* Remove sdl support
SDL is only used in an example case, but the examples are not built.
* remove openvslam
* Remove opencv+flann variant from 3dtk
* Back out cfitsio constraint from py-astropy
* remove opencv+flann variant from dlib
* remove boost constraint from 3dtk
* Remove non-opencv related bohrium changes
* Adjustments for cntk
- protobuf constraint at version 3.10
- need specific variants for opencv
- improve patch
* Deprecate CNTK package
* variant tweaks
- added appropriate conflicts for cublas
- made cuda/cudev relationship explicit
- moved openx to pending components as it needs an openvx package
* fix isort style error
* Use date version from kaldi rather than commit
* Revert changes from a bad rebase
* Add +flann to 3dtk and dlib
* Use compression support with libtiff
* remove `+datasets` from opencv dependency
The py-torchgeo package does not need opencv+datasets.
* fix typo
zip --> zlib
* added package gptune with all its dependencies: adding py-autotune, pygmo, py-pyaml, py-autotune, py-gpy, py-lhsmdu, py-hpbandster, pagmo2, py-opentuner; modifying superlu-dist, py-scikit-optimize
* adding gptune package
* minor fix for macos spack test
* update patch for py-scikit-optimize; update test files for gptune
* fixing gptune package style error
* fixing unit tests
* a few changes reviewed in the PR
* improved gptune package.py with a few newly added/improved dependencies
* fixed a few style errors
* minor fix on package name py-pyro4
* fixing more style errors
* Update var/spack/repos/builtin/packages/py-scikit-optimize/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* resolved a few issues in the PR
* fixing file permissions
* a few minor changes
* style correction
* minor correction to jq package file
* Update var/spack/repos/builtin/packages/py-pyro4/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixing a few issues in the PR
* adding py-selectors34 required by py-pyro4
* improved the superlu-dist package
* improved the superlu-dist package
* moree changes to gptune and py-selectors34 based on the PR
* Update var/spack/repos/builtin/packages/py-selectors34/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* improved gptune package: 1. addressing comments of tldahlgren in PR 26936; 2. adding variant openmpi
* fixing style issue of gptune
* changing file mode
* improved gptune package: add variant mpispawn which depends on openmpi; add variant superlu and hypre for installing the drivers; modified hypre package file to add a gptune variant
* fixing style error
* corrected pddrive_spawn path in gptune test; enforcing gcc>7
* fixing style error
* setting environment variables when loading gptune
* removing debug print in hypre/package.py
* adding superlu-dist v7.2.0; fixing an issue with CMAKE_INSTALL_LIBDIR
* changing site_packages_dir to python_platlib
* not using python3.9 for py-gpy, which causes due to dropped support of tp_print
* more replacement of site_packages_dir
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* espnet first build with depends
* fixed flake8
* updated to lastest version and removed python dependency
* changed to pypi and version 2.17.2
* [py-kaldiio] depends on py-pytest-runner
* [py-kaldiio] updated copyright
Co-authored-by: Sid Pendelberry <sid@rit.edu>
* prmon: make sure integration tests do not run in parallel
Some integration tests fail if not run on an otherwise idle machine.
* prmon: run unittests based on googletest
* prmon: fix checksums
* superlu-dist: use CMakePackage helper functions
* Fix#28609
It's OK to have CUDA in the dependency tree as long as it's not being
used for superlu-cuda.
* Update prmon package to latest versions
Add recent versions of the prmon package
Add the spdlog dependency for versions >=3
Add package developers as additional maintainers
Update name of development branch to 'main'
* Correct checksum for v3.0.1
at-spi2-core is automatically selecting dbus-broker and enabling systemd if it finds dbus-broker-launch which some systems might have even without systemd being part of the actual spack environment. This is not ideal for a spack package.
ucx has the configure option --[enable|disable]-backtrace detail.
This option is not explicitely set by spack, causing problems on my system, because
./configure does not find the bfd.h header file / libbfd.so library.
Added variant + dependencies (binutils). Disabled by default
* trilinos: version 12 requires cxxstd=11
* trilinos: use cmake version 3.21 or old when trilinos version 12
* conflict cxxstd=17 and cmake@3.2.[01]
* trilinos: version 12 requires cxxstd=11.
* Trilinos_CXX11_FLAGS is set to ' ' to avoid inject C++11 flag.
* set Trilinos_CXX11_FLAGS only version 12 or older.
* trilinos: update dependencies
Use the tribits deps to clarify some dependencies, and group some together
using `with` statements, eliminating some transitive conflict duplication.
* trilinos: Restricit cuda incompatibility
* e4s: vastly reduce number of packages in trilinos-cuda build
Not clear who the customers of cuda-enabled trilinos are, or what options
they need, or which sets of options conflict...
* e4s: remove ~wrapper from trilinos+cuda
* VTK-m: Make vtk-m consistent with ROCmPackage
* VTKm: Add kokkos variant
Specifying +kokkos will enable kokkos backend.
Specifying +kokkos with +rocm will require a kokkos with a ROCm backend.
Specifying +cuda enables VTK-m native CUDA backend. VTK-m native cuda backend
is not compatible with the kokkos +cuda backend.
* VTK-m: Add cuda_native variant
Required to allow specifying a vtk-m spec the selects a
cuda_arch and predictably propagate that to the underlying kokkos
dependency.
This also makes explicit selecting kokkos with a cuda backend or using
the VTK-m cuda backend.
* Mesa(18): Use libllvm virtual package
* Mesa patch configuration
Patch Mesa to define LLVM_VERSION_SUFFIX if llvm is pre-release
* Patch llvm-config to define LLVM_VERSION_SUFFIX
* Add a new version to track development
The released versions do not properly install via cmake which leads to
errors when linking against the library. These upstream problems have
been addressed on the glm development branch.
* Move git to class level and remove redundant depends
* vecgeom: require exact version of veccore
Fixes configure error from downstream package:
```
CMake Error at /rnsdhpc/code/spack/opt/spack/apple-clang/cmake/7zgbrwt/share/cmake-3.22/Modules/CMakeFindDependencyMacro.cmake:47 (find_package):
Could not find a configuration file for package "VecCore" that is
compatible with requested version "0.8.0".
The following configuration files were considered but not accepted:
/rnsdhpc/code/spack/var/spack/environments/celeritas/.spack-env/view/lib/cmake/VecCore/VecCoreConfig.cmake, version: 0.6.0
```
* veccore: add new versions
* Add flags to cabana to enable hypre and heffte when they are part of spec. Also add googletest to build dependencies
* Fixed mixed spaced and tabs
* Update package.py
* Update package.py
* Update package.py
* Modified to request specifically heFFTe version 2.0.0 due to
limitations in heFFTe cmakefiles.
* Update var/spack/repos/builtin/packages/cabana/package.py
Co-authored-by: Christoph Junghans <christoph.junghans@gmail.com>
* Integrated more heffte and hypre versions into cabana requests
Co-authored-by: Christoph Junghans <christoph.junghans@gmail.com>
* ParaView/VTK: Constrain version for ADIOS2 patch.
Older available versions of ParaView/VTK predate
ADIODS2 support.
ParaView lower bound is 5.8 and VTK lower bound is 8.2.0
* ParaView: Gate the ADIOS2 by verison
It seems that spack reads the output of `setup_run_environment` to build the actual spack modules and lmod modules. So, any output here will used verbatim on the shell.
This patch fixes https://github.com/spack/spack/issues/26733
1. adding latest release 3.5.0
2. updating cmake requirement to match that of Kokkos
3. adding logic to depend on the right version of Kokkos by default
* Kokkos: updating package list, maintainers and minimum cmake version
* Kokkos: updating maintainers list
Updating maintainers list to have the correct GitHub handle for Jan.
`spack license update-copyright-year` was updating license headers but not the MIT
license file. Make it do that and add a test.
Also simplify the way we bump the latest copyright year so that we only need to
update it in one place.
* [kaldi] Added version 2021-11-16
* [kaldi] Added logic for new version and when cuda 11 is used
* [kaldi] Added patch file when cuda 11 as cub is now built into it
* [kaldi] removed .999 and simplified some logic
Co-authored-by: Doug Heckman <dahdco@rit.edu>
* add py-ats package
* add new 7.0.10 tag
* add myself as a maintainer
* add dependencies for python and setuptools
* style
* added todo for flux
* words
* update versions users should use
* Use pip to bootstrap pip
* Bootstrap wheel from source
* Update PythonPackage to install using pip
* Update several packages
* Add wheel as base class dep
* Build phase no longer exists
* Add py-poetry package, fix py-flit-core bootstrapping
* Fix isort build
* Clean up many more packages
* Remove unused import
* Fix unit tests
* Don't directly run setup.py
* Typo fix
* Remove unused imports
* Fix issues caught by CI
* Remove custom setup.py file handling
* Use PythonPackage for installing wheels
* Remove custom phases in PythonPackages
* Remove <phase>_args methods
* Remove unused import
* Fix various packages
* Try to test Python packages directly in CI
* Actually run the pipeline
* Fix more packages
* Fix mappings, fix packages
* Fix dep version
* Work around bug in concretizer
* Various concretization fixes
* Fix gitlab yaml, packages
* Fix typo in gitlab yaml
* Skip more packages that fail to concretize
* Fix? jupyter ecosystem concretization issues
* Solve Jupyter concretization issues
* Prevent duplicate entries in PYTHONPATH
* Skip fenics-dolfinx
* Build fewer Python packages
* Fix missing npm dep
* Specify image
* More package fixes
* Add backends for every from-source package
* Fix version arg
* Remove GitLab CI stuff, add py-installer package
* Remove test deps, re-add install_options
* Function declaration syntax fix
* More build fixes
* Update spack create template
* Update PythonPackage documentation
* Fix documentation build
* Fix unit tests
* Remove pip flag added only in newer pip
* flux: add explicit dependency on jsonschema
* Update packages that have been added since this was branched off of develop
* Move Python 2 deprecation to a separate PR
* py-neurolab: add build dep on py-setuptools
* Use wheels for pip/wheel
* Allow use of pre-installed pip for external Python
* pip -> python -m pip
* Use python -m pip for all packages
* Fix py-wrapt
* Add both platlib and purelib to PYTHONPATH
* py-pyyaml: setuptools is needed for all versions
* py-pyyaml: link flags aren't needed
* Appease spack audit packages
* Some build backend is required for all versions, distutils -> setuptools
* Correctly handle different setup.py filename
* Use wheels for py-tomli to avoid circular dep on py-flit-core
* Fix busco installation procedure
* Clarify things in spack create template
* Test other Python build backends
* Undo changes to busco
* Various fixes
* Don't test other backends
* Add new package to spack. survey is a lightweight application performance tool that also gathers system information and stores it as metadata.
* Add maintainer and note about source access.
* Update the man path per spack reviewer suggestion.
* Remove redundant settings for PYTHONPATH, PATH, and MANPATH.
* Move to a one mpi collector approach for cce/tce integration.
* Add pyyaml dependency
* Make further spack reviewer changes to python type specs, mpi args, build type variant.
* Add reviewer requested changes.
* Add reviewer docstring requested changes.
* Add more updates from spack reviewer comments.
* Update the versions to use tags, not branches
* Redo dashes to fix issue with spack testing.
Co-authored-by: Jim Galarowicz <jgalarowicz@newmexicoconsortium.org>
When `spack compiler list` is run without being restricted to a
particular scope, and no compilers are found, say that none are
available, and hint that the use should run spack compiler find to
auto detect compilers.
* Improve docs
* Check if stdin is a tty
* add a test
Backport a patch for v1.3.4 that fixes an unsigned typedef problem
on macOS: https://github.com/xiph/ogg/pull/64
Also add v1.3.5 that has this issue fixed.
spack paths can be long and this overflows (at least) these buffers
inside of the bundled T1lib inside of the grace distribution, leading
to crashes on startup.
Charm++ versions below 7.0.0 have build issues on macOS, mainly due to the
pre-7.0.0 `VERSION` file conflicting with other version files on the
system: https://github.com/UIUC-PPL/charm/issues/2844. Specifically, it
conflicts with LLVM's `<version>` header that was added in llvm@7.0.0 to
comply with the C++20 standard:
https://en.cppreference.com/w/cpp/header/version. The conflict only occurs
on case-insensitive file systems, as typically used on macOS machines.
Many packages implement logic at the class level to handle complex dependencies and
conflicts. Others have started using `with when("@1.0"):` blocks since we added that
capability. The loops and other control logic can cause some pure directive logic not to
be removed by our package hashing logic -- and in many cases that's a lot of code that
will cause unnecessary rebuilds.
This commit changes the unparser so that it will descend into these blocks. Specifically:
1. Descend into loops, if statements, and with blocks at the class level.
2. Don't look inside function definitions (in or outside a class).
3. Don't look at nested class definitions (they don't have directives)
4. Add logic to *remove* empty loops/with blocks/if statements if all directives
in them were removed.
This allows our package hash to ignore a lot of pure metadata that it was not ignoring
before, and makes it less sensitive.
In addition, we add `maintainers` and `tags` to the list of metadata attributes that
Spack should remove from packages when constructing canonoical source for a package
hash.
- [x] Make unparser handle if/for/while/with at class level.
- [x] Add tests for control logic removal.
- [x] Add a test to ensure that all packages are not only unparseable, but also
that their canonical source is still compilable. This is a test for
our control logic removal.
- [x] Add another unparse test package that has complex logic.
These are the unit tests from astunparse, converted to pytest, with a few backports from
upstream cpython. These should hopefully keep `unparser.py` well covered as we change it.
We can't tell `print(a, b, c)` and `print((a, b, c))` apart -- both of these expressions
generate different ASTs in Python 2 and Python 3. However, we can decide that we don't
care. This commit treats both of them the same when `py_ver_consistent` is set with
`unparse()`.
This means that the package hash won't notice changes from printing a tuple to printing
multiple values, but we don't care, because this is extremely unlikely to affect the build.
More than likely this is just an error message for the user of the package.
- [x] treat `print(a, b, c)` and `print((a, b, c))` the same in py2 and py3
- [x] add another package parsing test -- legion -- that exercises this feature
To make it easier to see how package hashes change and how they are computed, add two
commands:
* `spack pkg source <spec>`: dumps source code for a package to the terminal
* `spack pkg source --canonical <spec>`: dumps canonicalized source code for a
package to the terminal. It strips comments, directives, and known-unused
multimethods from the package. It is used to generate package hashes.
* `spack pkg hash <spec>`: This gives the package hash for a particular spec.
It is generated from the canonical source code for the spec.
- [x] `add spack pkg source` and `spack pkg hash`
- [x] add tests
- [x] fix bug in multimethod resolution with boolean `@when` values
Co-authored-by: Greg Becker <becker33@llnl.gov>
We are planning to switch to using full hashes for Spack specs, which means that the
package hash will be included in the deployment descriptor. This means we need a more
robust package hash than simply dumping the `repr` of the AST.
The AST repr that we previously used for package content is unreliable because it can
vary between python versions (Python's AST actually changes fairly frequently).
- [x] change `package_hash`, `package_ast`, and `canonical_source` to accept a string for
alternate source instead of a filename.
- [x] consolidate package hash tests in `test/util/package_hash.py`.
- [x] remove old `package_content` method.
- [x] make `package_hash` do what `canonical_source_hash` was doing before.
- [x] modify `content_hash` in `package.py` to use the new `package_hash` function.
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Our package hash is supposed to be consistent from python version to python version.
Test this by adding some known unparse inputs and ensuring that they always have the
same canonical hash. This test relies on the fact that we run Spack's unit tests
across many python versions. We can't compute for several python versions within the
same test run so we precompute the hashes and check them in CI.
Package hashing was not properly handling multimethods. In particular, it was removing
any functions that had decorators from the output, so we'd miss things like
`@run_after("install")`, etc.
There were also problems with handling multiple `@when`'s in a single file, and with
handling `@when` functions that *had* to be evaluated dynamically.
- [x] Rework static `@when` resolution for package hash
- [x] Ensure that functions with decorators are not removed from output
- [x] Add tests for many different @when scenarios (multiple @when's,
combining with other decorators, default/no default, etc.)
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Previously we used `directives.__all__` to get directive names, but it wasn't
quite right -- it included `DirectiveMeta`, etc. It's not wrong, but it's also
not the clearest way to do this.
- [x] Refactor `@directive` to track names in `directive_names` global
- [x] Rename `_directive_names` to `_directive_dict_names` in `DirectiveMeta`
- [x] Add a test for `RemoveDirectives`
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Some packages use top-level unassigned strings instead of comments, either just after a
docstring on in the body somewhere else. Ignore those strings becasue they have no
effect on package behavior.
- [x] adjust RemoveDocstrings to remove all free-standing strings.
- [x] move tests for util/package_hash.py to test/util/package_hash.py
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Python 2 and 3 represent string literals differently in the AST. Python 2 requires '\x'
literals, and Python 3 source is always unicode, and allows unicode to be written
directly. These also unparse differently by default.
- [x] modify unparser to write both out the way `repr` would in Python 2 when
`py_ver_consistent` is provided.
Backport operator precedence algorithm from here:
397b96f6d7
This eliminates unnecessary parentheses from our unparsed output and makes Spack's unparser
consistent with the one in upstream Python 3.9+, with one exception.
Our parser normalizes argument order when `py_ver_consistent` is set, so that star arguments
in function calls come last. We have to do this because Python 2's AST doesn't have information
about their actual order.
If we ever support only Python 3.9 and higher, we can easily switch over to `ast.unparse`, as
the unparsing is consistent except for this detail (modulo future changes to `ast.unparse`)
Previously, there were differences in the unparsed code for Python 2.7 and for 3.5-3.10.
This makes unparsed code the same across these Python versions by:
1. Ensuring there are no spaces between unary operators and
their operands.
2. Ensuring that *args and **kwargs are always the last arguments,
regardless of the python version.
3. Always unparsing print as a function.
4. Not putting an extra comma after Python 2 class definitions.
Without these changes, the same source can generate different code for different
Python versions, depending on subtle AST differences.
One place where single source will generate an inconsistent AST is with
multi-argument print statements, e.g.:
```
print("foo", "bar", "baz")
```
In Python 2, this prints a tuple; in Python 3, it is the print function with
multiple arguments. Use `from __future__ import print_function` to avoid
this inconsistency.
Add `astunparse` as `spack_astunparse`. This library unparses Python ASTs and we're
adding it under our own name so that we can make modifications to it.
Ultimately this will be used to make `package_hash` consistent across Python versions.
Add an abstraction around libllvm to allow libllvm
providers to be specified for all packages.
This is targeting allowing mesa to build against
llvm-amdgpu or intel-llvm or llvm or any other
custom llvm variant that arises for specific GPU
toolchains
* Python: set default config_vars
* Add missing commas
* dso_suffix not present for some reason
* Remove use of default_site_packages_dir
* Use config_vars during bootstrapping too
* Catch more errors
* Fix unit tests
* Catch more errors
* Update docstring
* Update existing 2020.0 version to use tag
* Add versions 2018.2 and master
* Add patches for GCC/Intel
* Use MPI compiler wrappers when +mpi
* Constrain CMake build dependency (need >= 3.1)
* Add variants for optional components (e.g QFIT library)
./configure tries to execute an MPI test, which is not possible on
most HPC platforms (if you don't build on a compute node), so this
check is disabled to allow the build to proceed. Ideally we could
check this by placing constraints on the MPI that Spack builds (e.g.
require building a version that is guaranteed to have threading
support).
* rocm recipes updates for 4.5.0
* update to rocm recipes for 4.5.0 release
* updates to the rocm recipes for rocm-4.5.0 release
* fix style errors
* update to rocm-validation-suite for rocm-4.5.0 release
* bump up rccl recipe for rocm-4.5.0
* bump up version for rdc for rocm-4.5.0
* update miopengemm, miopen-opencl,rocm-opencl recipes for 4.5.0 release
* bump up version for mivisiox for rocm-4.5.0 release
* update the rocm-validation-suite recipe
* no need to change the perl path for 4.5.0
* fix the build failure with the recent change made for hip package
* modify checksum for the llvm-amdgpu for 4.5.0
* fix the build issue aftere recent changes made for enabling test
* fix the build issue with 4.5.0
* add new recipe for hipsolver
* address review comments
* llvm: make targets a multivalued variant
* Fix the targets variant values
1. Make them lowercase and add a mapping to cmake equivalent
2. auto -> all
2. Restore composability by using a multivalued variant, so that
`targets=all` and `targets=x86` is combined to `targets=all,x86`
which is then transformed into LLVM_TARGETS_TO_BUILD=all.
* use targets=x86 in iwyu
* Default to nvptx/amdgpu/host arch targets
* default to none
* Update var/spack/repos/builtin/packages/zig/package.py
This reports the kernel version (vs. the distro version) on Linux and
returns a valid Version (stripping characters like '+' which may be
present for custom-built kernels).
Reading appdirs.py without explicitly requesting UTF-8 decoding results
in the build process to fail for Python 3.6.
See https://github.com/ActiveState/appdirs/pull/152 for the upstream
fix.
* snakemake: New version
The newer versions of snakemake have a lot of new dependencies. The
optional dependencies still have to be added.
* removed comment
* some changes
* added reports variant
* deprecate older version and add me as maintainer
* Added dependency py-ratelimiter
* Fix: py-adios Cython run
Always run Cython before `py-adios` installs.
This makes sure the `.cpp` files from `.pyx` files are freshly
created and work with newer CPython versions than the one checked
in.
* Cleanup: `rm` -> `os.remove`
* Drop preferred hdf5 version
* Fix conduit
* Add compat bound for silo on hdf5
* Update var/spack/repos/builtin/packages/conduit/package.py
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
* hdf5 <= 1.10 for conduit <= 0.7
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
Doxygen's build system uses CMake's deprecated `FindPythonInterp`,
which can get confused by other `python` executables in the PATH.
See issue: https://github.com/spack/spack/issues/28215
Patch to add --embed flag to config-python when interface=python and
using python@3.8:
This is because python@3.8 changed behavior of python-config --ldflags
(and --libs) such that it no longer includes -lpython unless --embed
flag is used.
See e.g. https://github.com/mesonbuild/meson/issues/5629
* [py-aiohttp] added new version
* [py-aiohttp] changed dependency on setuptools
* [py-aiohttp] fixed some of the dependencies
* [py-aiohttp] updated dependency to py-typing-extensions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New package: py-azure-storage-blob
* fixed typo
* Update var/spack/repos/builtin/packages/py-azure-storage-blob/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-azure-storage-blob/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-azure-storage-blob/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* libtiff: fix build on macOS Monterey
* patch configure, not configure.ac
* Revert "patch configure, not configure.ac"
This reverts commit 8bf315cb22.
* Force Spack to run autoreconf using new patch
* [py-aiohttp] added new version
* [py-aiohttp] updated py-setuptools version dependency
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
For versions of aws-parallelcluster >= 2.9, the pyyaml dependency had to be >= 5.3.1 and == 5.1.2
at the same time making impossible to install ParallelCluster >= 2.9 from spack repository.
See issue: https://github.com/spack/spack/issues/28172
Fixed by limiting pyyaml 5.1.2 version to aws-parallelcluster < 2.8, according to this commit:
7255d314b7
Tested with a manual installation of aws-parallelcluster@2.11.4
Add a new check to `spack audit` to scan and verify that version constraints may be satisfied
Modifications:
- [x] Add a new check to `spack audit` to scan and verify that version constraints may be satisfied by some version declared in the built-in repository
- [x] Fix issues found by CI
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-datrie: New package, initial commit
* removed boilerplate
* Update var/spack/repos/builtin/packages/py-datrie/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-datrie/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* trilinos: fix define_tpl to handle depspecs w/out headers
This should address #27758 (i.e. errors due to netlib-scalapack not
having headers)
* trilinos: This fixes a mismatch in variant name and spec name for x11/libx11
* [py-minio] Added py-minio package
* [py-minio] added missing dependencies for py-minio
* [py-minio] added type=('build', 'run') in dependencies
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [py-dh-scikit-optimize] Added new versions
* [py-dh-scikit-optimize] fixing dependencies
* making py-configspace a required dependency of py-dh-scikit-learn
* r-hdf5r: use pkg-config to find hdf5
Since hdf5 was switched from autotools to cmake, the hdf5 compiler
wrappers can not be used to find and configure hdf5. This PR switches to
using pkg-config for configuration.
* Add comment about configure patch
* The Class F problem has been added to seven of the benchmarks
(BT, SP, LU, CG, MG, FT, and EP).
* The Class E problem has been added to the IS benchmark.
* In version 3.4.1, 'the number of processes' option does not apply.
* MPIFC and FC flags were added.
These versions change the install location of CMake files used
by dependents, but most dependents don't seem to look in this
new location.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Chris White <white238@llnl.gov>
This command pokes the environment, Python interpreter
and bootstrap store to check if dependencies needed by
Spack are available.
If any are missing, it shows a comprehensible message.
* locks: allow locks to work under high contention
This is a bug found by Harshitha Menon.
The `lock=None` line shouldn't be a release but should be
```
return (lock_type, None)
```
to inform the caller it couldn't get the lock type requested without
disturbing the existing lock object in the database. There were also a
couple of bugs due to taking write locks at the beginning without any
checking or release, and not releasing read locks before requeueing.
This version no longer gives me read upgrade to write errors, even
running 200 instances on one box.
* Change lock in check_deps_status to read, release if not installed,
not sure why this was ever write, but read definitely is more
appropriate here, and the read lock is only held out of the scope if
the package is installed.
* Release read lock before requeueing to reduce chance of livelock, the
timeout that caused the original issue now happens in roughly 3 of 200
workers instead of 199 on average.
With this commit:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
==> Updating view at /tmp/spack-faiirgmt/.spack-env/view
$ spack install zlib
==> All of the packages are already installed
```
Before this PR:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
$ spack install zlib
==> All of the packages are already installed
```
No view was generated
This commit introduces the command
spack module tcl setdefault <package>
similar to the one already available for lmod
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* upcxx: Update the UPC++ package to 2021.9.0
* Add the new release, and a missing older one.
* Remove the spack package cruft for supporting the obsolete build system that
was present in older versions that are no longer supported.
* General cleanups.
Support for library versions older than 2020.3.0 is officially retired,
for two reasons:
1. Releases prior to 2020.3.0 had a required dependency on Python 2,
which is [officially EOL](https://www.python.org/doc/sunset-python-2/)
as of Jan 1 2020, and is no longer considered secure.
2. (Most importantly) The UPC++ development team is unable/unwilling to
support releases more than two years old. UPC++ provides robust
backwards-compatibility to earlier releases of UPC++ v1.0, with very
rare well-documented/well-motivated exceptions. Users are strongly
encouraged to update to a current version of UPC++.
NOTE: Most of the lines changed in this commit are simply re-indentation,
and thus might be best reviewed in a diff that ignores whitespace.
* upcxx: Detect Cray XC more explicitly
This change is necessary to prevent false matches occuring on new Cray Shasta
systems, which do not use the aries network but were incorrectly being treated
as a Cray XC + aries platform.
UPC++ has not yet deployed official native support for Cray Shasta, but this
change is sufficient to allow building the portable backends there.
When running `spack install --log-format junit|cdash ...`, install
errors were ignored. This made spack continue building dependents of
failed install, ignoring `--fail-fast`, and exit 0 at the end.
* locks: allow locks to work under high contention
This is a bug found by Harshitha Menon.
The `lock=None` line shouldn't be a release but should be
```
return (lock_type, None)
```
to inform the caller it couldn't get the lock type requested without
disturbing the existing lock object in the database. There were also a
couple of bugs due to taking write locks at the beginning without any
checking or release, and not releasing read locks before requeueing.
This version no longer gives me read upgrade to write errors, even
running 200 instances on one box.
* Change lock in check_deps_status to read, release if not installed,
not sure why this was ever write, but read definitely is more
appropriate here, and the read lock is only held out of the scope if
the package is installed.
* Release read lock before requeueing to reduce chance of livelock, the
timeout that caused the original issue now happens in roughly 3 of 200
workers instead of 199 on average.
Fixes#27652
Ensure that mirror's to_dict function returns a syaml_dict object for all code
paths.
Switch to using the .get function for accessing the potential information from
the S3 mirror objects. If the key is not there, it will gracefully return
None instead of failing with a KeyError
Additionally, check that the connection object is a dictionary before trying
to "get" from it.
Add a test for the capturing of the new S3 information.
* perl: fix macOS build
With both 5.34.0 and 5.32.1 the build fails on macos-bigsur-skylake
%clang@12.0.5 and %clang13.0.0 :
```
2 errors found in build log:
579013 /private/var/folders/fy/x2xtwh1n7fn0_0q2kk29xkv9vvmbqb/T/s3j/spack-stage/spack-stage-perl-5.34.0-tpha2u52qfwaraidpzzbf6u4dbqg7dk5/spack-src/cpan/
Math-BigInt-FastCalc/../../miniperl "-I../../lib" -MExtUtils::Command::MM -e 'cp_nonempty' -- FastCalc.bs ../../lib/auto/Math/BigInt/FastCalc/Fas
tCalc.bs 644
579014
579015 Everything is up to date. Type '/Applications/Xcode.app/Contents/Developer/usr/bin/make test' to run test suite.
579016 DYLD_LIBRARY_PATH=/private/var/folders/fy/x2xtwh1n7fn0_0q2kk29xkv9vvmbqb/T/s3j/spack-stage/spack-stage-perl-5.34.0-tpha2u52qfwaraidpzzbf6u4dbqg7d
k5/spack-src ./perl -Ilib -I. installperl --destdir=
579017 WARNING: You've never run 'make test' or some tests failed! (Installing anyway.)
579018 /rnsdhpc/code/spack/opt/spack/apple-clang/perl/tpha2u5/bin/perl5.34.0
>> 579019 install_name_tool: error: sh -c '/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode.app/Contents/Developer/Pl
atforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -find install_name_tool 2> /dev/null' failed with exit code 256: (null) (errno=Invalid argument
)
579020 xcode-select: Failed to locate 'install_name_tool', requesting installation of command line developer tools.
579021 Cannot update /rnsdhpc/code/spack/opt/spack/apple-clang/perl/tpha2u5/bin/perl5.34.0 dependency paths
>> 579022 make: *** [install-all] Error 72
```
This is due to SYSTEM_VERSION_COMPAT being set.
* perl: conditionally set SYSTEM_VERSION_COMPAT based on CLT
The version of command line tools is the only difference between
@alalazo and my builds: his (v11) works only when SYSTEM_VERSION_COMPAT
is set to 1, and mine (v12.5 and v13) only work when it is unset.
With this commit:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
==> Updating view at /tmp/spack-faiirgmt/.spack-env/view
$ spack install zlib
==> All of the packages are already installed
```
Before this PR:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
$ spack install zlib
==> All of the packages are already installed
```
No view was generated
* New package: py-onnxmltools and dependencies
* Small fix
* Changes from review
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update recipe following review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Updates to installer.py did not account for spack monitor, so as currently implemented
there are three cases of failure that spack monitor will not account for. To fix this we add additional
hooks, including an on cancel and also do a custom action on concretization fail.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
The latest version of `jsonschema` fails if we're not specific about which schema draft
specification we're using. Update all of them to use the latest one (draft-07).
Our `jsonschema` external won't support Python 3.10, so we need to upgrade it.
It currently generates this warning:
lib/spack/external/jsonschema/compat.py:6: DeprecationWarning: Using or importing the ABCs
from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and
in 3.10 it will stop working
This upgrades `jsonschema` to 3.2.0, the latest version with support for Python 2.7. The next
version after this (4.0.0) drops support for 2.7 and 3.6, so we'll have to wait to upgrade to it.
Dependencies have been added in prior commits.
* [geant4] new version 11.0.0
* [geant4] prefer 10.7.3 for now
* [vecgeom] new version 1.1.18
* [clhep] new version 2.4.5.1
* [g4emlow] new version 8.0
* [g4particlexs] new version 4.p
* [geant4-data] new version 11.0.0
* [geant4] @11.0.0: cxxstd=17: ^clhep@2.4.5.1: ^vecgeom@1.1.18:
* [geant4] depends_on cmake@3.16:
* [geant4-data] remove g4tendl comment
* [g4tendl] new version 1.4
* [geant4] default cxxstd=11 when @10, 17 when @11; use CMAKE_CXX_STANDARD
* [geant4] variant tbb whe @11:, depends_on tbb, sets GEANT4_USE_TBB
* [geant4] new variant vtk when @11:, depends_on vtk@8.2:
* [geant4] simplify GEANT4_USE_VTK with define_from_variant
* [geant4] remove variant cxxstd conditional again
* [geant4] flake8 space after comma
* What's new in AOCL 3.1
1) AMD BLIS:
1.a) Supports Dynamic Dispatch and AOCL Dynamic feature
1.b) Improvements in DGEMM, ZGEMM, DTRSM, DSYRK, xGEMV, and DOTV
2) AMD libFLAME:
2.a) Supports LAPACK 3.10.0 specification
2.b) Optimized factorization and ZGEEV routines
3) AMD FFTW:
3.a) Features like 'AMD application optimization layer', 'Fast MPI transpose algorithm' and 'Top N planner' are added
4) AMD LibM:
4.a) Optimized exp2, log2 (Single and Double precision) scalar and vector
4.b) Optimized log10f (scalar and vector) and powf vector variants to support WRF4.1.2 benchmark
5) AOCL-Sparse:
5.a) New API for sparse matrix and dense matrix multiplication
6) AMD ScaLAPACK:
6.a) ILP64 support has been enabled
7) AOCL enabled MUMPS library:
7.a) CMake based build system on Windows for AOCL enabled MUMPS sparse solver library will be available shortly on GitHub
7.a.i) Refer https://github.com/amd/mumps-build
7.b) Spack-based recipe on Linux for AOCL enabled MUMPS sparse solver library will be enabled shortly
* Fix invalid version range error
* Incorporated review comments
1) Restore to previous url value
2) Instead of if else statements, used spack's enable_or_disable feature
* Incorporated following review comments:
1. Use of satisfies() for spec checks
2. Seperate conflict statements to check for minimum and maximum GCC versions
3. Used CMakePackage helpers
4. Code rearrangement to have the directives listed before methods
* New package: py-xrootdpyfs
* Add old version of py-fs
* Replace 2to3.patch with running 2to3
* Just restriuct setuptools version
* Update var/spack/repos/builtin/packages/py-fs/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
spack monitor now requires authentication as each build must be associated
with a user, so it does not make sense to allow the --monitor-no-auth flag
and this commit will remove it
* suite-sparse: Fix check for +/- tbb variant
Changed
'tbb' in spec
to
'+tbb' in spec
The former would configure suite-sparse to use tbb if any dependency
package (e.g. intel-oneapi-mkl) depends on tbb, even if
suite-sparse~tbb was specified.
* suite-sparse: conflict when trying to use 2021.x versions of tbb
See https://github.com/DrTimothyAldenDavis/SuiteSparse/issues/72
suite-sparse depends on task_schedule_init to control the number
of threads when e.g. interfacing with MATLAB. However, Intel
dropped task_schedule_init in the 2021.x releases of TBB (it has
been deprecated since TBB 4.3.5).
We just raise a spack conflict when using tbb @2021.x and +tbb
Because tbb is a virtual package and is not versioned, I have
instead checked for either intel-oneapi-tbb@2021: or
intel-tbb@2021:, not the most elegant but should do the job
* suite-sparse: fix style issues
* Added installation of OpenMP as an option
* Added a softlink (dpcpp) to clang++ to
mimic the packaged version of dpcpp
Co-authored-by: ravil <ravil.dorozhinskii@tum.de>
* Fix building container images
Patchelf is bootstrapped from sources, so we cannot
disable that mechanism until a finer selection is
possible in the configuration.
* Build on changes to the Dockerfile
* Don't login to Dockerhub on PRs
This commit introduces the command
spack module tcl setdefault <package>
similar to the one already available for lmod
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* Update hdf5/package.py to add HDF5 1.10.8 release and move
preferred version from 1.10.7 to 1.10.8.
* silo: versions before 4.11 conflict with hdf5 >= 1.10.8.
* Add patch file for silo@4.11 with hdf5 1.10 >=1.10.8 and
hdf5 1.12 >= 1.12.1.
* New package: py-scinum
* Update var/spack/repos/builtin/packages/py-scinum/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fix for OpenFOAM pthread issue for AOCC 3.2
* addressing the review comments
* updating when command for aocc v3.2.0 and above
Co-authored-by: mohan babu <mohbabul@amd.com>
* PETSc is a core dependency, yet we left a variant for PETSc.
This was removed, and ExaGO always depends on PETSc. The CMake
arguments were updated accordingly.
* camp+cuda is only a dependency when we build with RAJA and CUDA.
* py-radical-entk: add version 1.9.0
* py-radical-pilot: add version 1.10.1
* py-radical-utils: add version 1.9.1
* py-radical-utils: requires py-pymongo less than version 4
* New package: py-pysqlite3
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This PR also slightly changes the behavior in ci_rebuild().
We now still attempt to submit `spack install` results to CDash
even if the initial registration failed due to connection issues.
This commit follows in the spirit of #24299. We do not want `spack install`
to exit with a non-zero status when something goes wrong while attempting to
report results to CDash.
* Add a CI job to audit all the packages in the built-in repository
* flecsi: fixed typo for dependency on legion
* py-pythonqwt: fix a typo in variant name
* sollve: removed a conflict with a non-existing variant
* acts: fixed use of wrong variant in dd4hep
Also removed duplicated variant declaration in dd4hep
* aoflagger: update variant of a dependency
Issues introduced indirectly in #22925
* camellia: removed unused variant
Issue introduced indirectly in #26150
* cbtf-*: remove cti variants and dependency on mrnet+cti
Issue introduced in #14178
* flecsale: update variants to match flecsi
Issue introduced in #11679
* grnboost: fixed issue with non-existing variant in a dependency
This package possibly never worked since #8763
* nalu: fixed issue with non-existing variant in a dependency
* open-iscsi: fixed issue with non-existing variant in a dependency
* openspeedshop-*: remove use of non-existing mrnet+cti variant
* percept: fixed issue with non-existing variant in a dependency
* phyluce: fixed issue with non-existing variant in a dependency
Issue introduced in #12952
* phyluce: fixed issue with non-existing variant in a dependency
Issue introduced in #22340
* CPU Architecture Support
This commit removes the `native` variant in favor of Spack's built-in support for specifying a target cpu architecture. It also passes this information to the Legion build system so that it correctly passes the architecture to GASNet when built internally
* fixing whitespace
* Update package.py
based on a conversation with @streichler, this change sets `BUILD_MARCH` to an empty string, which will prevent legion's CMake build system from inserting `-march=native` and allow Spack to provide the correct architecture flags.
* Update package.py
adding a comment on what problem this MR solves.
* Update package.py
formatting
* PICMI: 0.16 & 0.18 & WarpX 1D
Update the `py-picmistandard` and the latest WarpX release.
Preparing 1D support (testable inputs coming for 22.01+).
* Fix style: overlong line
* Update pypi example link
* Fix requirement ranges
* WarpX 21.12: Update Patch
Follow-up from
https://github.com/ECP-WarpX/WarpX/pull/2646
* fix style
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* openjpeg: add missing dependencies and optionally disable them
* openjpeg: remove variant 'ownlibs'
* openjpeg: bugfix
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* openjpeg: do not build CODEC executables by default
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New package: py-hist and it's dependencies
* Update var/spack/repos/builtin/packages/py-hist/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-histoprint/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-mplhep/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update py-hist recipe
* Update package.py
* Fix py-iminuit recipe (requires py-cmake now)
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New package: py-correctionlib
* Update var/spack/repos/builtin/packages/py-correctionlib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-correctionlib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add py-make and update py-correctionlib recipe
* Add py-bind11 dependency
* Update package.py
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This PR is meant to move code with "business logic" from `spack.cmd.buildcache` to appropriate core modules[^1].
Modifications:
- [x] Add `spack.binary_distribution.push` to create a binary package from a spec and push it to a mirror
- [x] Add `spack.binary_distribution.install_root_node` to install only the root node of a concrete spec from a buildcache (may check the sha256 sum if it is passed in as input)
- [x] Add `spack.binary_distribution.install_single_spec` to install a single concrete spec from a buildcache
- [x] Add `spack.binary_distribution.download_single_spec` to download a single concrete spec from a buildcache to a local destination
- [x] Add `Spec.from_specfile` that construct a spec given the path of a JSON or YAML spec file
- [x] Removed logic from `spack.cmd.buildcache`
- [x] Removed calls to `spack.cmd.buildcache` in `spack.bootstrap`
- [x] Deprecate `spack buildcache copy` with a message that says it will be removed in v0.19.0
[^1]: The rationale is that commands should be lightweight wrappers of the core API, since that helps with both testing and scripting (easier mocking and no need to invoke `SpackCommand`s in a script).
* New package: py-cmsml
* Update var/spack/repos/builtin/packages/py-cmsml/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New package: py-cx-oracle
* Add dependencies from pyproject.toml
* Update var/spack/repos/builtin/packages/py-cx-oracle/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixing bugs in spack monitor
updates to installer.py did not account for spack monitor, so as currently implemented
there are three cases of failure that spack monitor will not account for. To fix this we add additional
hooks, including an on cancel and also do a custom action on concretization fail.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
* New package: py-climate
* Revert "fixing bugs in spack monitor"
This reverts commit bf7f6bf0e3.
* Flake-8
* Update package.py
* Update package.py
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
After this PR an error in a single package while detecting
external software won't abort the entire procedure.
The error is reported to screen as a warning.
* New package: py-boost-histogram
* Update var/spack/repos/builtin/packages/py-boost-histogram/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Remove a try/catch for an error with no handling. If the affected
code doesn't execute successfully, then the associated variable
is undefined and another (more-obscure) error occurs shortly after.
Since #27185, the cuda_arch variant values are conditional on +cuda. This means that for -cuda specs, the installation fails with:
```
==> acts: Executing phase: 'cmake'
==> Error: KeyError: 'cuda_arch'
/home/wdconinc/git/spack/var/spack/repos/builtin/packages/acts/package.py:222, in cmake_args:
219 log_failure_threshold = spec.variants['log_failure_threshold'].value
220 args.append("-DACTS_LOG_FAILURE_THRESHOLD={0}".format(log_failure_threshold))
221
>> 222 cuda_arch = spec.variants['cuda_arch'].value
223 if cuda_arch != 'none':
224 args.append('-DCUDA_FLAGS=-arch=sm_{0}'.format(cuda_arch[0]))
225
```
* new package: py-tensorflow-datasets
- includes new dependency package: py-tensorflow-metadata
* Update var/spack/repos/builtin/packages/py-tensorflow-datasets/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow-metadata/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Added gdb Dependency
When using spack to install cgdb, a spack-built gdb is necessary to
avoid dynamic link errors.
- Added maintainer: tuxfan
- Set preferred to 'master' (best version for spack currently)
* Update: The gdb dependency added by this PR is for runtime
Co-authored-by: Bernhard Kaindl <43588962+bernhardkaindl@users.noreply.github.com>
This adds support in spack for both build/install tests (spack install
--run-tests) and post-install smoke tests (spack test run).
Hpctoolkit itself only recently added tests, so for now, this only
applies to branch master.
To use this, you can "spack install intel-oneapi-compilers" and then
"spack compiler add" the new compiler. You would need to install with
"spack install ginkgo+oneapi%dpcpp"
- Use .tar.gz archive
- Update 2.3.3 to use .tar.gz archive (and update checksum)
- autoreconf dependency is no-longer required
- The new version depends on gperf
- Add sensei to the SDK with appropriate propagations
- Rework variants to SENSEI package to avoid providing broken builds
- Turn off miniapps by default, these are examples and not critical to using sensei
This PR updates the vtk package to use new variable names and adds some
dependencies.
- add version 9.1.0
- add version 1.4.2 to gl2ps package. This is needed to use gl2ps as a
dependency.
- new package: utf8cpp, used as a dependency for version 9.
- add dependencies when possible
- pugixml
- libogg
- libtheora
- utf8cpp
- gl2ps
- proj
- turn off configuring MPI if ~mpi
- always use the package-provided FindHDF5.cmake for versions up to 8.
Version @9: already does this so does not need a patch.
- use new CMake variables for version 9
- remove unused CMake variables depending on version
* py-packaging: add 21.3
* Update var/spack/repos/builtin/packages/py-packaging/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
A part of the gpgme testsuite by default even runs during normal
make and make install phases, creating a public keyring in ~/.gnupg.
Prevent this and avoid build failures in containers due to another
problem of the test suite and fix a test case of the new 0.16.0 release.
When the uidmap tools are installed on a system, this allows to run
containers as unprivileged user (rootless and daemonless) slimilar
to singularity, but using a familiar CLI: "alias docker=podman"
This is helpful to run e.g. spack builds in containers to reproduce
build failures from CI without requiring a installation of docker.
The required dependencies of podman are added as well.
Fix to not attempt to patch a nonexisting file for old versions
when building with gcc-11. Skip the build-time tests as all access
the X DISPLAY and open many windows on the screen.
I was browsing package metadata, as one does on a Sunday, and stumbled across a new kind of version attribute - brancch! I suspect this is supposed to be "branch."
Update flux-core and flux-sched package.py to include latest releases.
For flux-sched:
- Add patch to disable false-positive-happy valgrind test
- pin yaml-cpp to 0.6.3 due to issue described at:
https://github.com/flux-framework/flux-sched/issues/886
Starting with meson 0.60, unknown args produce errors and
the -Dx11 arg is only present in @:2.40
https://gitlab.gnome.org/GNOME/gtk-osx/-/issues/44
Add tiff variant: Default disabled since fails the tests in part.
Only libtiff@:3.9 pass, but these old versions have severe security issues.
Deprecate @:2.41 as they are affected by the high-severity CVE-2021-20240:
https://nvd.nist.gov/vuln/detail/CVE-2021-20240
Remove a custom bootstrapping procedure to
use spack.bootstrap instead
Modifications:
* Reference count the bootstrap context manager
* Avoid SpackCommand to make the bootstrapping
procedure more transparent
* Put back requirement on patchelf being in PATH for unit tests
* Add an e2e test to check bootstrapping patchelf
* Fix version constrains in py-ipykernel and py-ipython
Before the fix:
```
$ spack spec py-ipykernel@6.4.1 ^py-jupyter-client@7.0.6
==> Error: py-ipykernel@6.4.1 ^py-jupyter-client@7.0.6 is unsatisfiable, conflicts are:
no version satisfies the given constraints
```
After the fix:
```
```
(thanks god the old concretizer is still there - it provides sane error messages!)
* Fix py-ipython recipe
* Revert "Fix py-ipython recipe"
This reverts commit d65071665f.
* added package gptune with all its dependencies: adding py-autotune, pygmo, py-pyaml, py-autotune, py-gpy, py-lhsmdu, py-hpbandster, pagmo2, py-opentuner; modifying superlu-dist, py-scikit-optimize
* adding gptune package
* minor fix for macos spack test
* update patch for py-scikit-optimize; update test files for gptune
* fixing gptune package style error
* fixing unit tests
* a few changes reviewed in the PR
* improved gptune package.py with a few newly added/improved dependencies
* fixed a few style errors
* minor fix on package name py-pyro4
* fixing more style errors
* Update var/spack/repos/builtin/packages/py-scikit-optimize/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* resolved a few issues in the PR
* fixing file permissions
* a few minor changes
* style correction
* minor correction to jq package file
* Update var/spack/repos/builtin/packages/py-pyro4/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixing a few issues in the PR
* adding py-selectors34 required by py-pyro4
* improved the superlu-dist package
* improved the superlu-dist package
* moree changes to gptune and py-selectors34 based on the PR
* Update var/spack/repos/builtin/packages/py-selectors34/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
I think this test should be removed, but when it stays, it should at
least follow the symlink, cause it fails for me if I let spack build
patchelf and have a symlink in a view.
- Prevent `-j` flags to `make`, which has been known to cause problems
with Racket builds.
- Add variants for various common build flags, including support
for both versions of the Racket VM environment.
In addition:
- Prefer the minimal release to improve install times. Bells and
whistles carry their own runtime dependencies and should be
installed via `raco`. An enterprising user may even create a
`RacoPackage` class to make spack aware of `raco` installed packages.
- Match the official version numbering scheme.
Modifications:
- [x] Removed `centos:6` unit test, adjusted vermin checks
- [x] Removed backport of `collections.OrderedDict`
- [x] Removed backport of `functools.total_ordering`
- [x] Removed Python 2.6 specific skip markers in unit tests
- [x] Fixed a few minor Python 2.6 related TODOs in code
Updating the vendored dependencies will be done in separate PRs
* py-cython: add 3.0.0a9
* Update var/spack/repos/builtin/packages/py-cython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: py-singledispatch 3.7.0
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Make CUDA and ROCm architecture conditional
fixes#14337
The variant to specify which architecture to use
for CUDA and ROCm are now conditional on +cuda and
+rocm respectively.
* cp2k: make all CUDA related variants conditional on +cuda
* New version: py-pyzmq 22.3.0
* Update var/spack/repos/builtin/packages/py-pyzmq/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add connection specification to mirror creation
This allows each mirror to contain information about the credentials
used to access it.
Update command and tests based on comments
Switch to only "long form" flags for the s3 connection information.
Use the "any" function instead of checking for an empty list when looking
for s3 connection information.
Split test to use the access token separately from the access id and key.
Use long flag form in test.
Add endpoint_url to available S3 options.
Extend the special parameters for an S3 mirror to accept the
endpoint_url parameter.
Add a test.
* Add connection information per URL not per mirror
Expand the mirror-based connection information to be per-URL.
This will allow a user to specify different S3 connection information
for both the fetch and the push URLs.
Add a parameter for "profile", another way of storing the id/secret pair.
* Switch from "access_profile" to "profile"
* py-cycler: add 0.11.0
* Update var/spack/repos/builtin/packages/py-cycler/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: py-qtconsole 5.2.0
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: openldap 2.6.0;
fix recipe for groff (requires pkg-config to find uchardet);
fix recipe for openldap (requires groff to build documentation)
* Restrict openldap versions of py-python-ldap and percona-server
* Update var/spack/repos/builtin/packages/groff/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This causes `otfconfig` to also list zlib's library directory.
Otherwise, `-lz` cannot be found. Also remove `--with-zlibsymbols`,
which doesn't seem to be supported anymore.
Include several several patches to vtk 8.1 for building on a
system with no system install X11 libraries or include files.
Specify specific versions of dependent packages that are known to work with 3.2.1.
Tested on spock.olcf.ornl.gov. The GUI came up and rendered images
and an image was successfully saved using off screen rendering from
data from curv2d.silo.
Remove the "get_executable" function from the
spack.bootstrap module. Now "flake8", "isort",
"mypy" and "black" will use the same
bootstrapping method as GnuPG.
Currently Spack vendors `pytest` at a version which is three major
versions behind the latest (3.2.5 vs. 6.2.4). We do that since v3.2.5
is the latest version supporting Python 2.6. Remaining so much
behind the currently supported versions though might introduce
some incompatibilities and is surely a technical debt.
This PR modifies Spack to:
- Use the vendored `pytest@3.2.5` only as a fallback solution,
if the Python interpreter used for Spack doesn't provide a newer one
- Be able to parse `pytest --collect-only` in all the different output
formats from v3.2.5 to v6.2.4 and use it consistently for `spack unit-test --list-*`
- Updating the unit tests in Github Actions to use a more recent `pytest` version
The previous workaround of using CMAKE_INSTALL_RPATH=ON was used to
avoid CMake trying to write an RPATH into the linker script libcxx.so,
which is nonsensical. See commit f86ed1e.
However, CMAKE_INSTALL_RPATH=ON seems to disable the build RPATH, which
breaks LLVM during the build when it has to locate its build-time shared
libraries (e.g. libLLVM.so). That required yet another workaround, where
some shared libraries were installed "by hand", so that they were picked
up from the install libdir. See commit 8a81229.
This was a dirty workaround, and also makes it impossible to use ninja,
since we explicitly invoked make.
This commit removes the two old workaround, and sets
LIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON, so that libc++abi.a is linked into
libc++.so, which makes it enough to link with -lc++ or invoke clang++
with -stdlib=libc++, so that our install succeeds and linking LLVM's c++
standard lib is still easy.
Some packages use a 64_ or _64 symbol suffix for the ilp64 (= 64-bit
integers) intefrace for BLAS. In particular if we want to support shim
libraries like libopenblastrampoline supporting both the 32 and 64 bit
integer version of blas, it must be possible to distinguish between the
two.
mesa inherits MesonPackage (since October 2020) which depends on Py@3.
The conflicts('mesa') enables a regular build of `qt@5.7:5.15+webkit`
without having to specify the exact version by causing the concretizer
to select mesa18 which does not depend on python@3.
Co-authored-by: Bernhard Kaindl <bernhard.kaindl@ait.ac.at>
This type of error is skipped:
make[1]: *** [Makefile:222: /tmp/user/spack-stage/.../spack-src/usr/lib/julia/libopenblas64_.so.so] Error 1
but it's useful to have it, especially when a package sets a variable
incorrectly in makefiles
* New version: py-pylint 2.8.2; new package py-platformdirs
* Update var/spack/repos/builtin/packages/py-platformdirs/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pylint/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Intel mpi comes with an installation of libfabric (which it needs as a
dependency). It can use other implementations of libfabric at runtime
though, so if you install a package that depends on `mpi` and
`libfabric`, you can specify `intel-mpi+external-libfabric` and ensure
that the Spack-built instance is used (both by `intel-mpi` and the
root).
Apply analogous change to intel-oneapi-mpi.
* New version: py-pyrsistent 0.18.0
* Update package.py
* Update var/spack/repos/builtin/packages/py-pyrsistent/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: py-pytest 6.2.5
* Update var/spack/repos/builtin/packages/py-pytest/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Use setup_run_environment to search for libraries and set env variables for module generation.
Libraries are installed with CMAKE_INSTALL_LIBDIR, which can be lib or lib64 depending on the machine, which makes it impossible to hardcode through modules.yaml.
If the perl that perl-forks is built against is non-threaded the build
system will drop into interactive mode to ask about simulating ithreads.
This causes the build to hang. Set FORKS_SIMULATE_USEITHREADS to avoid
going into interactive mode.
If building Qt on a system with a recent glibc but an older kernel, it
is possible that Qt configures features based on glibc that are not
supported in the kernel. This PR tests the kernel version and ensures
certain features are disabled if the kernel does not support them.
* New version: py-prettytable 2.4.0; update homepage
* Update var/spack/repos/builtin/packages/py-prettytable/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: py-prometheus-client 0.12.0; new dependency (py-twisted) version 21.7.0 + it's dependencies
* Apply suggestions from code review (1/?)
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Changes from review (2/?)
* Changes from review (3/?)
* Changes from review (4/?)
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
- disable graphblas by default (very slow to compile)
- fix patch upperbound for cuda 11
- remove find_system_libs; not sure why it was added in the first place,
but it makes spack rather unusable as it introduces an rpath to /lib/...
* New version: py-oauthlib 3.1.1
* Update var/spack/repos/builtin/packages/py-oauthlib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Generalize env var PYTHON to avoid version conflicts
* Use available python executable
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* upcxx: Update the UPC++ package to 2021.9.0
* Add the new release, and a missing older one.
* Remove the spack package cruft for supporting the obsolete build system that
was present in older versions that are no longer supported.
* General cleanups.
Support for library versions older than 2020.3.0 is officially retired,
for two reasons:
1. Releases prior to 2020.3.0 had a required dependency on Python 2,
which is [officially EOL](https://www.python.org/doc/sunset-python-2/)
as of Jan 1 2020, and is no longer considered secure.
2. (Most importantly) The UPC++ development team is unable/unwilling to
support releases more than two years old. UPC++ provides robust
backwards-compatibility to earlier releases of UPC++ v1.0, with very
rare well-documented/well-motivated exceptions. Users are strongly
encouraged to update to a current version of UPC++.
NOTE: Most of the lines changed in this commit are simply re-indentation,
and thus might be best reviewed in a diff that ignores whitespace.
* upcxx: Detect Cray XC more explicitly
This change is necessary to prevent false matches occuring on new Cray Shasta
systems, which do not use the aries network but were incorrectly being treated
as a Cray XC + aries platform.
UPC++ has not yet deployed official native support for Cray Shasta, but this
change is sufficient to allow building the portable backends there.
* py-setupmeta: add new package
* Update var/spack/repos/builtin/packages/py-setupmeta/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
When running `spack install --log-format junit|cdash ...`, install
errors were ignored. This made spack continue building dependents of
failed install, ignoring `--fail-fast`, and exit 0 at the end.
* Add hdf5-vol-async package.
Add HDF5 1.13.0-rc6 version for building vol-async.
* Style test required another blank line.
* Change hdf5 dependency to develop-1.13+mpi+threadsafe.
* Update args for hdf5-vol-async.
Open MPI currently fails to build with scheduler=slurm if +pmix is
not given with a fatal error due to ``config_args +=
self.with_or_without('pmix', ...)`` resulting in --without-pmix.
However, Open MPI's configure points out "Note that Open MPI does
not support --without-pmix."
The PR only adds "--with-pmix=PATH" if +pmix is part of the spec.
Otherwise, nothing is added and Open MPI can fall back to its
internal PMIX sources.
(The other alternative would be to depend on +pmix in for
scheduler=slurm as is done for +pmi.)
- Added new checksums for 4.3.
- Now using llvm-amdgpu ~openmp in order to use the rocm-device-libs
build as external project in llvm-amdgpu package. We still need
to pull device-libs in using resource for the build as some headers
are not installed.
- Updated symlink creation to now remove an existing link if present
to avoid issues on partial reinstalls when debugging.
- Adjusted the flang_warning to be a part of Cmake options instead of
a filter_file for better compatability.
- The dependency on hsa-rocr-dev created some problems as type was changed
to the default build/link. This issue was because ROCr uses libelf and
the build of openmp expects elfutils. When link is specified libelf
was being found in the path first, causing errors. This was
introduced with the llvm-amdgpu external project build of device-libs.
- On a bare bone installation of sles15 it was noted that libquadmath0 was
needed as a dependency. On 18.04 gcc-multilib was also needed.
* Workaround for libelf headers being used instead of elfutils.
Due to Kitware API changes, default ANTs builds were failing, presumably for all versions (https://github.com/ANTsX/ANTs/issues/1236).
This commit defaults BUILD_TESTING to OFF, preventing calls against
these APIs and fixing all versions.
Note that the ANTs test suite was not clean anyway (e.g. ANTs/#842).
* Python tests: allow importing weirdly-named modules
e.g. with dashes in name
* SIP tests: allow importing weirdly-named modules
* Skip modules with invalid names
* Changes from review
* Update from review
* Update from review
* Cleanup
* New version: py-lz4 3.1.3; use external lz4 instead of bundled one
* Update var/spack/repos/builtin/packages/py-lz4/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Changes from review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add spack package py-ytopt-team-ytopt and required dependencies.
* Removed old ytop package.
* Added author as maintainer.
* Fix style.
* Update var/spack/repos/builtin/packages/py-config-space/package.py
Update python dependency to 3.7
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-config-space/package.py
Remove run dependency from py-cython.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-config-space/package.py
Added run dependency type for py-pyparsing.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Updated description of py-dh-scikit-optimize.
* Source py-dh-scikit-optimize from PyPI.
* Added latest py-dh-scikit-optimize version.
* Made plots option False by default for py-dh-scikit-optimize.
* Removed 0.9.4 as it needs additional dependencies.
* Added version dependencies.
* Added missing py-joblib dependencies.
* Added run dependency type.
* Added python 2.7+ as supported for py-pyaml.
* Change py-config-space to py-configspace.
* Added dependency on python 3.6+.
* Fix py-configspace package naming.
* Changed py-autotune to py-ytopt-autotune.
* Update var/spack/repos/builtin/packages/py-pyaml/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Added debug variant with py-ray dependency.
* Added missing py-mpi4py missing dependency.
* Removed erroneous variant.
* Added debug variant to py-ray.
* Fix indentation.
* Removed debug variant of py-ray.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2021-11-06 10:52:30 -05:00
9068 changed files with 372788 additions and 243332 deletions
description:Some package in Spack had stand-alone tests that didn't pass
title:"Testing issue: "
labels:[test-error]
body:
- type:textarea
id:reproduce
attributes:
label:Steps to reproduce the failure(s) or link(s) to test output(s)
description:|
Fill in the test output from the exact spec that is having stand-alone test failures. Links to test outputs (e.g., CDash) can also be provided.
value:|
```console
$ spack spec -I <spec>
...
```
- type:textarea
id:error
attributes:
label:Error message
description:|
Please post the error message from spack inside the `<details>` tag below:
value:|
<details><summary>Error message</summary><pre>
...
</pre></details>
validations:
required:true
- type:textarea
id:information
attributes:
label:Information on your system or the test runner
description:Please include the output of `spack debug report` for your system.
validations:
required:true
- type:markdown
attributes:
value:|
If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well.
- type:textarea
id:additional_information
attributes:
label:Additional information
description:|
Please upload test logs or any additional information about the problem.
- type:markdown
attributes:
value:|
Some packages have maintainers who have volunteered to debug build failures. Run `spack maintainers <name-of-the-package>` and **@mention** them here if they exist.
- type:checkboxes
id:checks
attributes:
label:General information
options:
- label:I have reported the version of Spack/Python/Platform/Runner
required:true
- label:I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
required:true
- label:I have uploaded any available logs
required:true
- label:I have searched the issues of this repo and believe this is not a duplicate
@@ -1090,6 +1097,8 @@ could depend on ``mpich@1.2:`` if it can only build with version
Below are more details about the specifiers that you can add to specs.
.._version-specifier:
^^^^^^^^^^^^^^^^^
Version specifier
^^^^^^^^^^^^^^^^^
@@ -1105,6 +1114,37 @@ set of arbitrary versions, such as ``@1.0,1.5,1.7`` (``1.0``, ``1.5``,
or ``1.7``). When you supply such a specifier to ``spack install``,
it constrains the set of versions that Spack will install.
For packages with a ``git`` attribute, ``git`` references
may be specified instead of a numerical version i.e. branches, tags
and commits. Spack will stage and build based off the ``git``
reference provided. Acceptable syntaxes for this are:
..code-block::sh
# branches and tags
foo@git.develop # use the develop branch
foo@git.0.19 # use the 0.19 tag
# commit hashes
foo@abcdef1234abcdef1234abcdef1234abcdef1234 # 40 character hashes are automatically treated as git commits
foo@git.abcdef1234abcdef1234abcdef1234abcdef1234
Spack versions from git reference either have an associated version supplied by the user,
or infer a relationship to known versions from the structure of the git repository. If an
associated version is supplied by the user, Spack treats the git version as equivalent to that
version for all version comparisons in the package logic (e.g. ``depends_on('foo', when='@1.5')``).
The associated version can be assigned with ``[git ref]=[version]`` syntax, with the caveat that the specified version is known to Spack from either the package definition, or in the configuration preferences (i.e. ``packages.yaml``).
..code-block::sh
foo@git.my_ref=3.2 # use the my_ref tag or branch, but treat it as version 3.2 for version comparisons
foo@git.abcdef1234abcdef1234abcdef1234abcdef1234=develop # use the given commit, but treat it as develop for version comparisons
If an associated version is not supplied then the tags in the git repo are used to determine
the most recent previous version known to Spack. Details about how versions are compared
and how Spack determines if one version is less than another are discussed in the developer guide.
If the version spec is not provided, then Spack will choose one
according to policies set for the particular spack installation. If
the spec is ambiguous, i.e. it could match multiple versions, Spack
@@ -1190,6 +1230,23 @@ variants using the backwards compatibility syntax and uses only ``~``
for disabled boolean variants. The ``-`` and spaces on the command
line are provided for convenience and legibility.
Spack allows variants to propagate their value to the package's
dependency by using ``++``, ``--``, and ``~~`` for boolean variants.
For example, for a ``debug`` variant:
..code-block::sh
mpileaks ++debug # enabled debug will be propagated to dependencies
mpileaks +debug # only mpileaks will have debug enabled
To propagate the value of non-boolean variants Spack uses ``name==value``.
For example, for the ``stackstart`` variant:
..code-block::sh
mpileaks stackstart==4# variant will be propagated to dependencies
mpileaks stackstart=4# only mpileaks will have this variant value
^^^^^^^^^^^^^^
Compiler Flags
^^^^^^^^^^^^^^
@@ -1197,10 +1254,15 @@ Compiler Flags
Compiler flags are specified using the same syntax as non-boolean variants,
but fulfill a different purpose. While the function of a variant is set by
the package, compiler flags are used by the compiler wrappers to inject
flags into the compile line of the build. Additionally, compiler flags are
inherited by dependencies. ``spack install libdwarf cppflags="-g"`` will
install both libdwarf and libelf with the ``-g``flag injected into their
compile line.
flags into the compile line of the build. Additionally, compiler flags can
be inherited by dependencies by using ``==``.
``spack install libdwarf cppflags=="-g"``will install both libdwarf and
libelf with the ``-g`` flag injected into their compile line.
..note::
versions of spack prior to 0.19.0 will propagate compiler flags using
the ``=`` syntax.
Notice that the value of the compiler flags must be quoted if it
contains any spaces. Any of ``cppflags=-O3``, ``cppflags="-O3"``,
@@ -1280,7 +1342,7 @@ Normally users don't have to bother specifying the architecture if they
are installing software for their current host, as in that case the
values will be detected automatically. If you need fine-grained control
over which packages use which targets (or over *all* packages' default
target), see :ref:`concretization-preferences`.
target), see :ref:`package-preferences`.
..admonition:: Cray machines
@@ -1402,7 +1464,7 @@ built.
You can see what virtual packages a particular package provides by
getting info on it:
..command-output:: spack info mpich
..command-output:: spack info --virtuals mpich
Spack is unique in that its virtual packages can be versioned, just
like regular packages. A particular version of a package may provide
@@ -1610,9 +1672,13 @@ own install prefix. However, certain packages are typically installed
`Python <https://www.python.org>`_ packages are typically installed in the
==> Adding "clingo-bootstrap@spack+python %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding "gnupg@2.3: %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding "patchelf@0.13.1:0.13.99 %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding binary packages from "https://github.com/alalazo/spack-bootstrap-mirrors/releases/download/v0.1-rc.2/bootstrap-buildcache.tar.gz" to the mirror at /opt/bootstrap/local-mirror
To register the mirror on the platform where it's supposed to be used run the following command(s):
Notice that this function, if you define it, requires a result object (generated by
``run()``, a monitor (if you want to send), and a boolean ``overwrite`` to be used
to check if a result exists first, and not write to it if the result exists and
overwrite is False. Also notice that since we already saved these files to the analyzer metadata folder, we return early if a monitor isn't defined, because this function serves to send results to the monitor. If you haven't saved anything to the analyzer metadata folder
yet, you might want to do that here. You should also use ``tty.info`` to give
the user a message of "Writing result to $DIRNAME."
.._writing-commands:
@@ -671,6 +415,13 @@ If you need to write a hook that is relevant to a failure within a build
process, you would want to instead use ``on_phase_failure``.
"""""""""""""""""""""""""""
``on_install_cancel(spec)``
"""""""""""""""""""""""""""
The same, but triggered if a spec install is cancelled for any reason.
"""""""""""""""""""""""""""""""""""""""""""""""
``on_phase_success(pkg, phase_name, log_file)``
"""""""""""""""""""""""""""""""""""""""""""""""
@@ -692,23 +443,6 @@ with a hook, and this is the purpose of this particular hook. Akin to
``on_phase_success`` we require the same variables - the package that failed,
the name of the phase, and the log file where we might find errors.
"""""""""""""""""""""""""""""""""
``on_analyzer_save(pkg, result)``
"""""""""""""""""""""""""""""""""
After an analyzer has saved some result for a package, this hook is called,
and it provides the package that we just ran the analysis for, along with
the loaded result. Typically, a result is structured to have the name
of the analyzer as key, and the result object that is defined in detail in
:ref:`analyzer_run_function`.
..code-block::python
defon_analyzer_save(pkg,result):
"""given a package and a result...
"""
print('Do something extra with a package analysis result here')
^^^^^^^^^^^^^^^^^^^^^^
Adding a New Hook Type
@@ -1050,39 +784,39 @@ Release branches
^^^^^^^^^^^^^^^^
There are currently two types of Spack releases: :ref:`major releases
<major-releases>` (``0.13.0``, ``0.14.0``, etc.) and :ref:`point releases
<point-releases>` (``0.13.1``, ``0.13.2``, ``0.13.3``, etc.). Here is a
<major-releases>` (``0.17.0``, ``0.18.0``, etc.) and :ref:`point releases
<point-releases>` (``0.17.1``, ``0.17.2``, ``0.17.3``, etc.). Here is a
diagram of how Spack release branches work::
o branch: develop (latest version)
o branch: develop (latest version, v0.19.0.dev0)
|
o merge v0.14.1 into develop
|\
| o branch: releases/v0.14, tag: v0.14.1
o | merge v0.14.0 into develop
|\|
| o tag: v0.14.0
o
| o branch: releases/v0.18, tag: v0.18.1
o |
| o tag: v0.18.0
o |
| o
|/
o merge v0.13.2 into develop
|\
| o branch: releases/v0.13, tag: v0.13.2
o | merge v0.13.1 into develop
|\|
| o tag: v0.13.1
o | merge v0.13.0 into develop
|\|
| o tag: v0.13.0
o
|
o
| o branch: releases/v0.17, tag: v0.17.2
o |
| o tag: v0.17.1
o |
| o tag: v0.17.0
o |
| o
|/
o
The ``develop`` branch has the latest contributions, and nearly all pull
requests target ``develop``.
requests target ``develop``. The ``develop`` branch will report that its
version is that of the next **major** release with a ``.dev0`` suffix.
Each Spack release series also has a corresponding branch, e.g.
``releases/v0.14`` has ``0.14.x`` versions of Spack, and
``releases/v0.13`` has ``0.13.x`` versions. A major release is the first
``releases/v0.18`` has ``0.18.x`` versions of Spack, and
``releases/v0.17`` has ``0.17.x`` versions. A major release is the first
tagged version on a release branch. Minor releases are back-ported from
develop onto release branches. This is typically done by cherry-picking
bugfix commits off of ``develop``.
@@ -1093,12 +827,20 @@ packages. They should generally only contain fixes to the Spack core.
However, sometimes priorities are such that new functionality needs to
be added to a minor release.
Both major and minor releases are tagged. After each release, we merge
the release branch back into ``develop`` so that the version bump and any
other release-specific changes are visible in the mainline. As a
convenience, we also tag the latest release as ``releases/latest``,
so that users can easily check it out to get the latest
stable version. See :ref:`merging-releases` for more details.
Both major and minor releases are tagged. As a convenience, we also tag
the latest release as ``releases/latest``, so that users can easily check
it out to get the latest stable version. See :ref:`updating-latest-release`
for more details.
..note::
Older spack releases were merged **back** into develop so that we could
do fancy things with tags, but since tarballs and many git checkouts do
not have tags, this proved overly complex and confusing.
We have since converted to using `PEP 440 <https://peps.python.org/pep-0440/>`_
compliant versions. `See here <https://github.com/spack/spack/pull/25267>`_ for
details.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Scheduling work for releases
@@ -1156,10 +898,11 @@ completed, the steps to make the major release are:
``releases/vX.Y``. That is, you should create a ``releases/vX.Y``
branch if you are preparing the ``X.Y.0`` release.
#.Bump the version in ``lib/spack/spack/__init__.py``.
#.Remove the ``dev0`` development release segment from the version tuple in
==> Installing "patchelf@0.16.1%gcc@10.2.1 ldflags="-static-libstdc++ -static-libgcc" build_system=autotools arch=linux-centos7-x86_64" from a buildcache
and ``CMAKE_PREFIX_PATH``, as well as ``DYLD_FALLBACK_LIBRARY_PATH``
on macOS. On Linux however, the corresponding ``LD_LIBRARY_PATH``
variable is *not* set, because it affects the behavior of
system executables too.
..note::
In general, the ``LD_LIBRARY_PATH`` variable is not required
when using packages built with Spack, thanks to the use of RPATH.
Some packages may still need the variable, which is best handled
on a per-package basis instead of globally, as explained in
:ref:`overide-api-calls-in-package-py`.
There is a special case for prefix inspections relative to environment
views. If all of the following conditions hold for a module set
@@ -533,8 +554,7 @@ configuration:
#. The configuration is for an :ref:`environment <environments>` and
will never be applied outside the environment,
#. The environment in question is configured to use a :ref:`view
<filesystem-views>`,
#. The environment in question is configured to use a view,
#. The :ref:`environment view is configured
<configuring_environment_views>` with a projection that ensures
every package is linked to a unique directory,
@@ -586,16 +606,17 @@ Filter out environment modifications
Modifications to certain environment variables in module files are there by
default, for instance because they are generated by prefix inspections.
If you want to prevent modifications to some environment variables, you can
do so by using the environment blacklist:
do so by using the ``exclude_env_vars``:
..code-block::yaml
modules:
tcl:
all:
filter:
# Exclude changes to any of these variables
environment_blacklist:['CPATH','LIBRARY_PATH']
default:
tcl:
all:
filter:
# Exclude changes to any of these variables
exclude_env_vars:['CPATH','LIBRARY_PATH']
The configuration above will generate module files that will not contain
modifications to either ``CPATH`` or ``LIBRARY_PATH``.
@@ -607,42 +628,39 @@ modifications to either ``CPATH`` or ``LIBRARY_PATH``.
Autoload dependencies
"""""""""""""""""""""
In some cases it can be useful to have module files that automatically load
their dependencies. This may be the case for Python extensions, if not
activated using ``spack activate``:
Often it is required for a module to have its (transient) dependencies loaded as well.
One example where this is useful is when one package needs to use executables provided
by its dependency; when the dependency is autoloaded, the executable will be in the
PATH. Similarly for scripting languages such as Python, packages and their dependencies
have to be loaded together.
Autoloading is enabled by default for LMod, as it has great builtin support for through
the ``depends_on`` function. For Environment Modules it is disabled by default.
Autoloading can also be enabled conditionally:
..code-block::yaml
modules:
tcl:
^python:
autoload:'direct'
modules:
default:
tcl:
all:
autoload:none
^python:
autoload:direct
The configuration file above will produce module files that will
load their direct dependencies if the package installed depends on ``python``.
The allowed values for the ``autoload`` statement are either ``none``,
``direct`` or ``all``. The default is ``none``.
..tip::
Building external software
Setting ``autoload`` to ``direct`` for all packages can be useful
when building software outside of a Spack installation that depends on
artifacts in that installation. E.g. (adjust ``lmod`` vs ``tcl``
as appropriate):
..code-block::yaml
modules:
lmod:
all:
autoload:'direct'
``direct`` or ``all``.
..note::
TCL prerequisites
In the ``tcl`` section of the configuration file it is possible to use
the ``prerequisites`` directive that accepts the same values as
``autoload``. It will produce module files that have a ``prereq``
statement instead of automatically loading other modules.
statement, which can be used to autoload dependencies in some versions
of Environment Modules.
------------------------
Maintaining Module Files
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
