os.path.dirname was being used to compare compilers. If two compilers
are in the same directory then it will pick up the first one it encounters.
Compare the full compiler path instead.
When installing setuptools from sources in Spack, we might
get into weird failures due to the way we use pip.
In particular, for Spack it's necessary to install in a
non-isolated pip environment to allow using PYTHONPATH as a
selection method for all the build requirements of a
Python package.
This can fail when installing setuptools since there might
be a setuptools version already installed for the Python
interpreter being used, with different entry points than
the one we want to install.
Installing from wheels both pip and setuptools should
harden our installation procedure in the context of:
- Bootstrapping Python dependencies of Spack
- Using external Python packages
On Cray systems that use Cray Data Virtualization Service (DVS),
symlinks across filesystems are not allowed, either due to a bug, or
because they're simply not POSIX compliant [1].
Spack's OpenSSL package defaults to `certs=system` which comes down to
symlinking `/etc/ssl` in the Spack install prefix, triggering this
problem, resulting in mysterious installation failures.
Instead of relying on system certs, we can just use
`ca-certificates-mozilla`, and if users really need system certs, then
they're probably better off marking OpenSSL entirely as external.
[1] https://github.com/glennklockwood/cray-dvs
* Created package and added description
* Add py-markdown-include
* Created package
* Finished creating package
* Added py-md-environ
* Added build dependencies
* Added other deps
* Add python-markdown-math (#4)
* Created package and started to add info
* Removed unneeded global/install options
* Figured out version spec for markdown-math
* Removed type=build from unnecessary dependencies
* Removed unneeded install/global options, added version spec to dependency
* Added wscullin as interim maintainer for packages
* Fixed style issues
* Took care of trailing whitespace
* Removed comment line before imports
* Changed file charset to match other packages
* Update var/spack/repos/builtin/packages/py-ford/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-ford/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Removed test dependency after review feedback
* Added new 6.1.12 version to py-ford
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add LevelZero variant to hwloc package
This permits the hwloc package to build with with support for the
Intel Level Zero low-level layer, analogous to CUDA, ROCm, and OpenCL.
* Fix typo
Add variant +gssapi to enable authentication via Kerberos through GSSAPI
When openssh is installed at sites using Kerberos, openssh needs to auth
via Kerberos through GSSAPI in order to work in such environments.
* Update h5bench maintainers and versions
* Include version 1.1 for h5bench
* Correct release hash and set default version
* Update .tar.gz version
* Update HDF5 VOL async version and environment variable syntax
* added package gptune with all its dependencies: adding py-autotune, pygmo, py-pyaml, py-autotune, py-gpy, py-lhsmdu, py-hpbandster, pagmo2, py-opentuner; modifying superlu-dist, py-scikit-optimize
* adding gptune package
* minor fix for macos spack test
* update patch for py-scikit-optimize; update test files for gptune
* fixing gptune package style error
* fixing unit tests
* a few changes reviewed in the PR
* improved gptune package.py with a few newly added/improved dependencies
* fixed a few style errors
* minor fix on package name py-pyro4
* fixing more style errors
* Update var/spack/repos/builtin/packages/py-scikit-optimize/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* resolved a few issues in the PR
* fixing file permissions
* a few minor changes
* style correction
* minor correction to jq package file
* Update var/spack/repos/builtin/packages/py-pyro4/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixing a few issues in the PR
* adding py-selectors34 required by py-pyro4
* improved the superlu-dist package
* improved the superlu-dist package
* moree changes to gptune and py-selectors34 based on the PR
* Update var/spack/repos/builtin/packages/py-selectors34/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* improved gptune package: 1. addressing comments of tldahlgren in PR 26936; 2. adding variant openmpi
* fixing style issue of gptune
* changing file mode
* improved gptune package: add variant mpispawn which depends on openmpi; add variant superlu and hypre for installing the drivers; modified hypre package file to add a gptune variant
* fixing style error
* corrected pddrive_spawn path in gptune test; enforcing gcc>7
* fixing style error
* setting environment variables when loading gptune
* removing debug print in hypre/package.py
* adding superlu-dist v7.2.0; fixing an issue with CMAKE_INSTALL_LIBDIR
* changing site_packages_dir to python_platlib
* not using python3.9 for py-gpy, which causes due to dropped support of tp_print
* more replacement of site_packages_dir
* fixing a few dependencies in gptune; added a gptune version
* adding url for gptune
* minor correction of gptune
* updating versions in butterflypack
* added a version for openturns
* added a url for openturns
* minor fix for openturns
* Update var/spack/repos/builtin/packages/openturns/package.py
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* added version2.1.1 for butterflypack
* fixing a tag in butterflypack
* minor fix for superlu-dist
* removed file filter for superlu-dist
* Update var/spack/repos/builtin/packages/superlu-dist/package.py
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* superlu-dist: add version detection for the testing; removing unused lines
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* nfft, pnfft: fix detection of fftw variants (precision)
* nfft, pnfft: use fftw's selected_precisions; avoid repetitive calls to spec
Co-authored-by: Martin Lang <martin.lang@mpsd.mpg.de>
* [py-sentry-sdk] audited dependencies and relisted extras as variants
* [py-sentry-sdk] added version 1.5.5
* [py-sentry-sdk] flake8
* [py-sentry-sdk]
- added "descriptions" to variants
- removed conflicts
- replaced with when statements in varients
* Update prmon recipe, v3.0.2
Update recipe to v3.0.2, using external dependency
option for the build (as these are satisfied so easily with Spack)
* Remove "broken" 3.0.1 version
This one could not build properly with Spack, due to
missing submodule sources
* add some new version of abacus and fix bug on mkl
* Add Package:PSCMC
* update maintainer
* rebase
* update maintainers
* Update var/spack/repos/builtin/packages/pscmc/package.py
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Installation process of libint runs a Python script:
<69cc7b9bc6/export/cmake/CMakeLists.txt.export (L410)>.
If Python isn't explicitly listed as build dependency, system Python will be
picked up, which can cause troubles.
* Fix py-ray dependencies and build system
* Update var/spack/repos/builtin/packages/py-ray/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Added documentation for unusual dependencies.
* Fixed npm preinstall script per @adamjstewart suggestion.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Most package installations include compressed source files. This
adds support for common archive types on Windows:
* Add support for using system 7zip functionality to decompress .Z
files when available (and on Windows, use 7zip for .xz archives)
* Default to using built-in Python support for tar/bz2 decompression
(note that Python tar documentation mentions preservation of file
permissions)
* Add tests for decompression support
* Extract logic for handling exploding archives (i.e. compressed
archives that expand to more than one base file) into an
exploding_archive_catch context manager in the filesystem module
* Update backage bear
Newer versions of rtags aren't available via older url
* Fix sha256 as download link has changed
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* [py-httpx] python dependencies are type=('build', 'run')
* [py-httpx] py-wheel is now implied by PythonPackage
* [py-httpx] fixed older version dependencies
* [py-httpx] added version 0.22.0
* add pacakge: abacus
* rename
* fix some style bugs
* update package:abacus
* fix some style bugs
* format code
* Delete a line of useless comments
* updatee abacus
* Update package.py
add new version info and fix a bug on mkl
* update version info and fix a bug on mkl
* Update package.py
* fix style bugs
* trailing whitespace
* Add flag to enable WRF-Chem module
* Update var/spack/repos/builtin/packages/wrf/package.py
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* Set chem variant only for v4+
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Anticipate openPMD-api changes in the next major release that are
already in `dev` (aka Spack `develop`):
- C++17 requirement
- drop: `mpark-variant` public dependency
- add: `toml11` private dependency
Also add @franzpoeschel as co-maintainer for the Spack package.
Updates to improve Spack-generated modules for Intel oneAPI compilers:
* intel-oneapi-compilers set CC etc.
* Add a new package intel-oneapi-compilers-classic which can be used to
generate a module which sets CC etc. to older compilers (e.g. icc)
* lmod module logic now updated to treat the intel-oneapi-compilers*
packages as compilers
* acts-dd4hep: new package, separated from new acts@19.1.0
* acts-dd4hep: improved versioning
* acts-dd4hep: don't use curl | sha256sum
* acts: new variant `odd` for Open Data Detector
* acts-dd4hep: style changes
Add two new stacks targeted at x86_64 and arm, representing an initial list of packages
used by current and planned AWS Workshops, and built in conjunction with the ISC22
announcement of the spack public binary cache.
Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
Explicitly import package utilities in all packages, and corresponding fallout.
This includes:
* rename `spack.package` to `spack.package_base`
* rename `spack.pkgkit` to `spack.package`
* update all packages in builtin, builtin_mock and tutorials to include `from spack.package import *`
* update spack style
* ensure packages include the import
* automatically add the new import and remove any/all imports of `spack` and `spack.pkgkit`
from packages when using `--fix`
* add support for type-checking packages with mypy when SPACK_MYPY_CHECK_PACKAGES
is set in the environment
* fix all type checking errors in packages in spack upstream
* update spack create to include the new imports
* update spack repo to inject the new import, injection persists to allow for a deprecation period
Original message below:
As requested @adamjstewart, update all packages to use pkgkit. I ended up using isort to do this,
so repro is easy:
```console
$ isort -a 'from spack.pkgkit import *' --rm 'spack' ./var/spack/repos/builtin/packages/*/package.py
$ spack style --fix
```
There were several line spacing fixups caused either by space manipulation in isort or by packages
that haven't been touched since we added requirements, but there are no functional changes in here.
* [x] add config to isort to make sure this is maintained going forward
* Call Numpy package's set_blas_lapack() and setup_build_environment() in Scipy package
* Remove broken link from comment
* Use .package attribute of spec to avoid import
This PR fixes several issues I noticed while trying to get Spack working on Apple M1.
- [x] `build_environment.py` attempts to add `spec['foo'].libs` and `spec['foo'].headers` to our compiler wrappers for all dependencies using a try-except that ignores `NoLibrariesError` and `NoHeadersError` respectively. However, The `libs` and `headers` attributes of the Python package were erroneously using `RuntimeError` instead.
- [x] `spack external find python` (used during bootstrapping) currently has no way to determine whether or not an installation is `+shared`, so previously we would only search for static Python libs. However, most distributions including XCode/Conda/Intel ship shared Python libs. I updated `libs` to search for both shared and static (order based on variant) as a fallback.
- [x] The `headers` attribute was recursively searching in `prefix.include` for `pyconfig.h`, but this could lead to non-deterministic behavior if multiple versions of Python are installed and `pyconfig.h` files exist in multiple `<prefix>/include/pythonX.Y` locations. It's safer to search in `sysconfig.get_path('include')` instead.
- [x] The Python installation that comes with XCode is broken, and `sysconfig.get_paths` is hard-coded to return specific directories. This meant that our logic for `platlib`/`purelib`/`include` where we replace `platbase`/`base`/`installed_base` with `prefix` wasn't working and the `mkdirp` in `setup_dependent_package` was trying to create a directory in root, giving permissions issues. Even if you commented out those `mkdirp` calls, Spack would add the wrong directories to `PYTHONPATH`. Added a fallback hard-coded to `lib/pythonX.Y/site-packages` if sysconfig is broken (this is what distutils always did).
* Adding new package bricks for x86, cuda
* Fixed complaints from "spack style" that CI found
* add license comment at top
Co-authored-by: drhansj <drhansj@berkeley.edu>
Co-authored-by: eugeneswalker <38933153+eugeneswalker@users.noreply.github.com>
Use `spack build` as build dir to avoid recursive link error.
```
config.status: linking /var/folders/fy/x2xtwh1n7fn0_0q2kk29xkv9vvmbqb/T/s3j/spack-stage/spack-stage-sed-4.8-wraqsot6ofzvr3vrgusx4mj4mya5xfux/spack-src/GNUmakefile to GNUmakefile
config.status: executing depfiles commands
config.status: executing po-directories commands
config.status: creating po/POTFILES
config.status: creating po/Makefile
==> sed: Executing phase: 'build'
==> [2022-05-25-14:15:51.310333] 'make' '-j8' 'V=1'
make: GNUmakefile: Too many levels of symbolic links
make: stat: GNUmakefile: Too many levels of symbolic links
make: *** No rule to make target `GNUmakefile'. Stop.
```
* Allow Kokkos with OpenMPTarget backend
* Restrict SYCL and OpenMPTarget to C++17 or higher
* Improve C++ standard check for SYCL and OpenMPTarget
* Fix indentation
Currently, environments can either be concretized fully together or fully separately. This works well for users who create environments for interoperable software and can use `concretizer:unify:true`. It does not allow environments with conflicting software to be concretized for maximal interoperability.
The primary use-case for this is facilities providing system software. Facilities provide multiple MPI implementations, but all software built against a given MPI ought to be interoperable.
This PR adds a concretization option `concretizer:unify:when_possible`. When this option is used, Spack will concretize specs in the environment separately, but will optimize for minimal differences in overlapping packages.
* Add a level of indirection to root specs
This commit introduce the "literal" atom, which comes with
a few different "arities". The unary "literal" contains an
integer that id the ID of a spec literal. Other "literals"
contain information on the requests made by literal ID. For
instance zlib@1.2.11 generates the following facts:
literal(0,"root","zlib").
literal(0,"node","zlib").
literal(0,"node_version_satisfies","zlib","1.2.11").
This should help with solving large environments "together
where possible" since later literals can be now solved
together in batches.
* Add a mechanism to relax the number of literals being solved
* Modify spack solve to display the new criteria
Since the new criteria is above all the build criteria,
we need to modify the way we display the output.
Originally done by Greg in #27964 and cherry-picked
to this branch by the co-author of the commit.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* Inject reusable specs into the solve
Instead of coupling the PyclingoDriver() object with
spack.config, inject the concrete specs that can be
reused.
A method level function takes care of reading from
the store and the buildcache.
* spack solve: show output of multi-rounds
* add tests for best-effort coconcretization
* Enforce having at least a literal being solved
Co-authored-by: Greg Becker <becker33@llnl.gov>
* Py-x21 now works, needs dependencies
Conflicts:
var/spack/repos/rit-rc/packages/py-x21/package.py
* Added dependencies to py-x21
* Making flake style check happy
* [py-x21] flake8
* [py-x21]
- added homepage
- added placeholder description
- added comment about checksums
* [py-x21] added darwin support and fixed issue with python 3.7 wheel name
* [py-x21] adding checksum hash
* [py-x21] removed duplicate py-pynacl
* [py-x21]
- updated description
- updated version listing to have a different version for each version
of python. Also, versions dependent on sys.platform
- updated url_for_version to not require post concretized information so
that spack checksum works
* [py-x21] isort
Co-authored-by: vehrc <vehrc@rit.edu>
rocm-5.1.0 removed librocrand.so from ROCM_DIR/rocrand/lib location (but includes are still at this location)
/opt/rocm-5.0.2/lib/librocrand.so
/opt/rocm-5.0.2/rocrand/lib/librocrand.so
/opt/rocm-5.1.0/lib/librocrand.so
drwxr-xr-x 2 root root 617 Mar 8 08:20 /opt/rocm-5.0.2/rocrand/include
drwxr-xr-x 2 root root 617 Mar 31 09:48 /opt/rocm-5.1.0/rocrand/include
* Added autotools configure flags to ensure that hwloc finds the correct
version of CUDA that it was concretized against, rather than the first
one that package config finds.
* Added support for finding the correct version of ROCm libraries. Fixed Flake8.
* Fixed guard on finding ROCm library
* [py-h2] py-wheel is implied by PythonPackage
* [py-h2] python dependencies should be type=('build', 'run')
* [py-h2] fixed dependencies for py-h2@4.0.0
* [py-h2] added version 3.2.0
* [py-h2] added version 4.1.0
* [py-h2] Older version requires py-enum34 for older versions of python
Add two new cloud pipelines for E4S on Amazon Linux, include arm and x86 (v3 + v4) stacks.
Notes:
- Updated mpark-variant to remove conflict that no longer exists in Amazon Linux
- Which command on Amazon Linux prefixes on all results when padded_length is too high. In this case, padded_length<=503 works as expected. Chose conservative length of 384.
