Recipes that are not actually required for LBANN or DiHydrogen to
build. These should be concretized within the same environment or
installed via PIP using the same Python that installed LBANN.
Removing these will help eliminate build time failures that are
actually associated with Python tools, not LBANN.
The status displayed in the terminal title could be wrong when doing
distributed builds. For instance, doing `spack install glib` in two
different terminals could lead to the current package being reported as
`40/29` due to the way Spack handles retrying locks.
Work around this by keeping track of the package IDs that were already
encountered to avoid counting packages twice.
* HIP: Change mesa18 dep to gl
* Mesa: Conflict with llvm-amdgpu when +llvm and swr
* Add def for suffix
* Disable llvm suffix patch.
* LLVM: Remove version suffix patches
* ECP-SDK: ParaView 5.11: required for CUDA
* Add conflict with ParaView@master
Because of the additional constraints for cuda, ParaView@master may be
selected unintentionally. Prefer older versions of ParaView without cuda
to master with cuda.
* hypre: Add releases 2.21.0 and 2.22.0
* Revert "hypre: Add releases 2.21.0 and 2.22.0"
This reverts commit 8921cdb3ac.
* Address external linkage failures in elfutils 0.185:
https://bugs.gentoo.org/794601https://sourceware.org/pipermail/elfutils-devel/2021q2/003862.html
Encountered while building within a Spack environment.
* Revert "Address external linkage failures in elfutils 0.185:"
This reverts commit 76b93e4504.
* paraview: The ninja generator has problems with XL and CCE
See https://gitlab.kitware.com/paraview/paraview/-/issues/21223
* paraview: Add variant to allow choice of cmake generator.
This will be necessary until problems with cmake+ninja on XL and
CCE builds can be resolved.
See https://gitlab.kitware.com/paraview/paraview/-/issues/21223
* paraview: ninja generator problems with XL/CCE
By popular preference, abandon the idea of a special variant
and select the generator based on compiler.
* Greg Becker suggested using the dedicated "generator" method to
pass the choice of makefile generator to cmake.
* paraview: The build errors I saw before with paraview%cce + ninja
have not reappeared in subsequent testing, so I'm dropping it from this
PR. If they re-occur I'll report the issue separately to KitWare.
* py-nbclassic: add 0.3.5
* Update var/spack/repos/builtin/packages/py-nbclassic/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fix style
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add a new test to catch exit code failure
fixes#29226
This introduces a new unit test that checks the return
code of `spack unit-test` when it is supposed to fail.
This is to prevent bugs like the one introduced in #25601
in which CI didn't catch a missing return statement.
In retrospective it seems that the shell test we have right
now all go through `tty.die` or similar code paths which
call `sys.exit(a)` explicitly. This new test instead checks
`spack unit-test` which relies on the return code from
command invocation in case of errors.
* Add 'develop' version for dmtcp
* Update var/spack/repos/builtin/packages/dmtcp/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The `spack external find binutils` command was failing to find my system
binutils because the regex was not matching. The name of the executable
follows the string 'GNU' that I tested with three different
installations so I changed the regex to look for that. On my CentOS-7
system, the version had the RPM details so I set the version to capture
the first three parts of the version.
The system compiler on RHEL7 fails to build the latest linux-uuid.
```
util-linux-uuid@2.37.4%gcc@4.8.5 arch=linux-rhel7-haswell
```
results in:
```
libuuid/src/unparse.c:42:73: error: expected ';', ',' or ')' before 'fmt'
static void uuid_fmt(const uuid_t uuid, char *buf, char const *restrict fmt)
```
It looks like it's assuming C99 by default so there may be a better way
to handle this... but this at least works
See https://github.com/spack/spack/pull/28468/files#r809156986
If we exit before generating the:
error("Dependencies must have compatible OS's with their dependents").
...
facts we'll output a problem that is effectively
different by the one solved by clingo.
* cmd/checksum: prefer url matching url_from_version
This is a minimal change toward getting the right archive from places
like github. The heuristic is:
* if an archive url exists, take its version
* generate a url from the package with pkg.url_from_version
* if they match
* stop considering other URLs for this version
* otherwise, continue replacing the url for the version
I doubt this will always work, but it should address a variety of
versions of this bug. A good test right now is `spack checksum gh`,
which checksums macos binaries without this, and the correct source
packages with it.
fixes#15985
related to #14129
related to #13940
* add heuristics to help create as well
Since create can't rely on an existing package, this commit adds another
pair of heuristics:
1. if the current version is a specifically listed archive, don't
replace it
2. if the current url matches the result of applying
`spack.url.substitute_version(a, ver)` for any a in archive_urls,
prefer it and don't replace it
fixes#13940
* clean up style and a lingering debug import
* ok flake8, you got me
* document reference_package argument
* Update lib/spack/spack/util/web.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* try to appease sphinx
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
We can see what is in the bootstrap store with `spack find -b`, and you can clean it with `spack
clean -b`, but we can't do much else with it, and if there are bootstrap issues they can be hard to
debug.
We already have `spack --mock`, which allows you to swap in the mock packages from the command
line. This PR introduces `spack -b` / `spack --bootstrap`, which runs all of spack with
`ensure_bootstrap_configuration()` set. This means that you can run `spack -b find`, `spack -b
install`, `spack -b spec`, etc. to see what *would* happen with bootstrap configuration, to remove
specific bootstrap packages, etc. This will hopefully make developers' lives easier as they deal
with bootstrap packages.
This PR also uses a `nullcontext` context manager. `nullcontext` has been implemented in several
other places in Spack, and this PR consolidates them to `llnl.util.lang`, with a note that we can
delete the function if we ever reqyire a new enough Python.
- [x] introduce `spack --bootstrap` option
- [x] consolidated all `nullcontext` usages to `llnl.util.lang`
* py-imageio: add 2.16.0
* Update var/spack/repos/builtin/packages/py-imageio/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Some "concrete" versions on the command line, e.g. `qt@5` are really
meant to satisfy some actual concrete version from a package. We should
only assume the user is introducing a new, unknown version on the CLI
if we, well, don't know of any version that satisfies the user's
request. So, if we know about `5.11.1` and `5.11.3` and they ask for
`5.11.2`, we'd ask the solver to consider `5.11.2` as a solution. If
they just ask for `5`, though, `5.11.1` or `5.11.3` are fine solutions,
as they satisfy `@5`, so use them.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
* geant4-data: use build+run-only depends
* geant4: point to dependent datadir
This is "used" in the configure step to set up the Geant4Config.cmake
file's persistent pointers to the data directory, but the dependency
is still listed as "run" -- though I'm not sure this is the right behavior
since the geant4 installation really does change as a function of the
data directory, and the installation is incomplete/erroneous
without using one.
* Style
* trilinos: disable dl on macOS
* py-sphinx-argparse: add explicit poetry dependency
* libzmq: fix libbsd dependency
libbsd is *always* required when +libbsd (introduced in #28503) . #20893
had previously removed the macos dependency because libbsd wasn't always
enabled. Libbsd support is only available after 4.3.2 so change it to a
conflict rather than bumping the dependency.
* hdf5: work around GCC11.2 monterey fortran bug
* go-bootstrap: mark conflict for monterey
* py-tensorflow: add versions 2.5.0 and 2.6.0
- add version 2.5.0
- add version 2.6.0
- add patches for newer protobuf
- set constraints
* Remove import os. left over from testing
* Remove unused patch file
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add py-clang dependency
* Adjust py-clang constraint
* Build tensorflow with tensorboard
- tensorflow
- added 2.6.1 and 2.6.2 versions
- tensorboard
- have bazel use number of jobs set by spack
- add versions and constraints
- new package: py-tensorboard-data-server
- use wheel for py-tensorboard-plugin-wit
This package can not build with newer versions of bazel that are
needed for newer versions of py-tensorboard.
* Update var/spack/repos/builtin/packages/py-clang/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Remove empty line at end of file
* Fix import sorting
* Adjust python dependencies on py-clang
* Add version 2.7.0 of pt-tensorflow and py-tensorboard
* Adjust bazel constraints
* bazel-4 support begins with py-tensorflow-2.7.0
* Adjust dependencies
* Loosen cuda constraint on versions > 2.5
Tensorflow-2.5 and above can use cuda up to version 11.4.
* Add constraints to patch
The 0008-Fix-protobuf-errors-when-using-system-protobuf.patch patch
should only apply to versions 2.5 and above.
* Adjust constraints
- versions 2.4 and below need protobuf-3.12 and below
- versions 2.4 and above can use up to cuda-11.4
- versions 2.2 and below can not use cudnn-8
- the null_linker_bin patch should only be applied to versions 2.5 and
above.
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fix py-grpcio dependency for version 2.7
Also, make sure py-h5py mpi specs are consistent.
* Add llvm as run dependency.
* Fix python spec for py-tensorboard
* Fix py-google-auth spec for py-tensorboard
* Do not override the pip spec for tensorboard-plugin-wit
* Converted py-tensorboard-plugin-wit to wheel only package
* Fix bazel dependency spec in tensorflow
* Adjust pip masks
- allow tensorboard to be specified in pip constraints
- mask tensorflow-estimator
* Remove blank line at end of file
* Adjust pip constraints in setup.py
Also, adjust constraint on a patch that is fixed in 2.7
* Fix flake8 error
Adjust formatting for consistency.
* Get bazel dep right
* Fix old cudnn dependency, caught in audit test
* Adjust the regex to ensure proper line is changed
* Add py-libclang package
- Stripped the py-clang package down to just version 5
- added comments to indicate the purpose of py-clang and that
py-libclang should be preferred
- set dependencies accordingly in py-tensorflow
* Remove cap on py-h5py dependency for v2.7
* Add TODO entries for tensorflow-io-gcs-filesystem
* Edit some comments
* Add phases and select python in PATH for tensorboard-data-server
* py-libclang
- remove py-wheel dependency
- remove raw string notation in filter_file
* py-tensorboard-data-server
- remove py-wheel dep
- remove py-pip dep
- use python from package class
* py-tensorboard-plugin-wit
- switch to PythonPackage
- add version 1.8.1
- remove unneeded code
* Add comment as to why a wheel is need for tensorboard-plugin-wit
* remove which pip from tensorboard-data-server
* Fix dependency specs in tensorboard
* tweak dependencies for tensorflow
* fix python constraint
* Use llvm libs property
* py-tensorboard-data-server
- merge build into install
- use std_pip_args
* remove py-clang dependency
* remove my edits to py-tensorboard-plugin-wit
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
See https://github.com/spack/spack/issues/25353#issuecomment-1041868116
This commit changes the default behavior of
```
$ spack external find
```
from searching all the possible packages Spack knows about to
search only for the ones tagged as being a "build-tool".
It also introduces a `--all` option to restore the old behavior.
Prefer `sw_vers` to `platform.mac_ver`. In anaconda3 installation, for example, the latter reports 10.16 on Monterey -- I think this is affected by how and where the python instance was built.
Use MACOSX_DEPLOYMENT_TARGET if present to override the operating system choice.
It will be useful for metrics gathering and possibly debugging to
have this environment variable available in the runner pods that
do the actual rebuilds.
Since Spack does not install external packages, this commit skips them by
default when running stand-alone tests. The assumption is that such packages
have likely undergone an acceptance test process.
However, the tests can be run against installed externals using
```
% spack test run --externals ...
```
fixes#28260
Since we iterate over different variants from many packages, the variant
values may have types which are not comparable, which causes errors
at runtime. This is not a real issue though, since we don't need the facts
to be ordered. Thus, to avoid needless sorting, the sorted function has
been removed and a comment has been added to tip any developer that
might need to inspect these clauses for debugging to add back sorting
on the first two items only.
It's kind of difficult to add a test for this, since the error depends on
whether Python sorting algorithm ever needs to compare the third
value of a tuple being ordered.
* extensions: allow multiple "extends" directives
This will allow multiple extends directives in a package as long as only one of
them is selected as a dependency in the concrete spec.
* document the option to have multiple extends
Reuse previously was a very invasive change that required parameters to be added to all
the methods that called `concretize()` on a `Spec` object. With the addition of
concretizer configuration, we can use the config system to simplify this argument
passing and keep the code cleaner.
We decided that concretizer config options should be read at `Solver` instantiation
time, and if config changes between instnatiation of a particular solver and
`solve()` invocation, the `Solver` should use the settings from `__init__()`.
- [x] remove `reuse` keyword argument from most concretize functions
- [x] refactor usages to use `spack.config.override("concretizer:reuse", True)`
- [x] rework argument passing in `Solver` so that parameters are set from config
at instantiation time
`--reuse` was previously handled individually by each command that
needed it. We are growing more concretization options, and they'll
need their own section for commands that support them.
Now there are two concretization options:
* `--reuse`: Attempt to reuse packages from installs and buildcaches.
* `--fresh`: Opposite of reuse -- traditional spack install.
To handle thes, this PR adds a `ConfigSetAction` for `argparse`, so
that you can write argparse code like this:
```
subgroup.add_argument(
'--reuse', action=ConfigSetAction, dest="concretizer:reuse",
const=True, default=None,
help='reuse installed dependencies/buildcaches when possible'
)
```
With this, you don't need to add logic to pull the argument out and
handle it; the `ConfigSetAction` just does it for you. This can probably
be used to clean up some other commands later, as well.
Code that was previously passing `reuse=True` around everywhere has
been refactored to use config, and config is set from the CLI using
a new `add_concretizer_args()` function in `spack.cmd.common.arguments`.
- [x] Add `ConfigSetAction` to simplify concretizer config on the CLI
- [x] Refactor code so that it does not pass `reuse=True` to every function.
- [x] Refactor commands to use `add_concretizer_args()` and to pass
concretizer config using the config system.
Config scopes were different for `config` and `mutable_config`,
and `mutable_config` did not have a command line scope.
- [x] Fix by consolidating the creation logic for the two fixtures.
The concretizer is going to grow to have many more configuration,
and we really need some structured config for that.
* We have the `config:concretizer` option that chooses the solver,
but extending that is awkward (we'd need to replace a string with
a `dict`) and the solver choice will be deprecated eventually.
* We have the `concretization` option in environments, but it's
not a top-level config section -- it's just for environments,
and it also only admits a string right now.
To avoid overlapping with either of these and to allow the most
extensibility in the future, this adds a new `concretizer` config
section that can be used in and outside of environments. There
is only one option right now: `reuse`. This can expand to include
other options later.
Likely, we will soon deprecate `config:concretizer` and warn when
the user doesn't use `clingo`, and we will eventually (sometime later)
move the `together` / `separately` options from `concretization` into
the top-level `concretizer` section.
This commit just adds the new section and schema. Fully wiring it
up is TBD.
The solver has a lot of configuration associated with it. Rather
than adding arguments to everything, we should encapsulate that
in a class. This is the start of that work; it replaces `solve()`
and its kwargs with a class and properties.
* Add 'stable' to the list of infinity version names.
Rename libunwind 1.5-head to 1.5-stable.
* Add stable to the infinite version list in packaging_guide.rst.
* py-etelemetry: add 0.3.0
* Update var/spack/repos/builtin/packages/py-etelemetry/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
No version of py-nbconvert@5: can be concretized due to conflicting versions
of flit-core that are required. This issue could be solved by separate
concretization of build deps.
* archspec: remove pyproject.toml to workaround PEP517
If pyproject.toml is in the folder, that is preferred to the
setup.py packaged by poetry itself. Adding a dependency on
poetry for deploying a pure Python package seems wasteful,
since the package to be deployed just needs to be copied in
place, so we don't want to built rust for that.
* archspec: patch pyproject.toml to comply to PEP517
See https://python-poetry.org/docs/pyproject/#poetry-and-pep-517
* Fix style issues
The new HDF5 version 1.12 API causes compiler errors due to modified function prototypes. Note that version 1.11 is the development version of HDF5 1.12.
* py-numba: add 0.55.1
* Remove comment
* Pin down py-llvmlite version for older py-numba releases
* Remove py-llvmlite deps for releases not in spack
* Set upper bounds for python and py-numpy
* Add stricter upper bound to py-numpy for releases <=0.47
Setting Spack's `$prefix` to `$DESTDIR` and not to `$PREFIX` install the
package in `$prefix/usr/local` and not in `$prefix`, thus when it is
loaded the executable `direnv` in not "seen" by the environment.
* Added support to LBANN, Hydrogen, DiHydrogen, and Aluminum to capture
a gcc-toolchain cxxflags argument and pass it to a CMAKE_CUDA_FLAG
argument when set. This helps deal with compiling with clang on
systems with old base gcc installations.
* Added a dependency on py-scipy when enabling tests on LBANN.
* Updated the C++ standard for Hydrogen to C++17.
* Added a new variant +apps to enable (or disable) python packages that
are used by applications in the LBANN repo, but are not strictly
required for building and using LBANN.
* Added a run time dependency for both py-pytest and py-scipy so that
they are activated in any environment.
* Added support for building LBANN, Hydrogen, and DiHydrogen with the
IBM ESSL BLAS library. This requires explicit identification of
additional LAPACK libraries, since ESSL does not implement LAPACK, but
is found by CMake.
* Fixed a bug in the LBANN dependency on OpenCV for Power architectures.
The +powerpc variant is only required for GCC toolchains and causes
Clang to break. Switched to only enabling when using %gcc on power.
- Installation often hangs building the documentation. This happens when
doxygen and latex are found. To avoid the issue, comment-out that part
of the code until an explicit cmake variable to disable documentation
generation is available.
* sundials: fix smoke tests
* sundials: add new version
* use cmake+make instead of make for tests, fix style
* use cmake_bin workaround from https://github.com/spack/spack/pull/28622
Note that the SDK is not the same as the system version: using
apple-clang@13 is a better match than `os=monterey` since this actually
fails on bigsur as well, as long as xcode 13 is being used.
* core: Make platform environment an instance not class method
In preparation for accessing data constructed in __init__.
* macos: set consistent macosx deployment target
This should silence numerous warnings from mixed gcc/macos toolchains.
* perl: prevent too-new deployment target version
```
*** Unexpected MACOSX_DEPLOYMENT_TARGET=11
***
*** Please either set it to a valid macOS version number (e.g., 10.15) or to empty.
```
* Stylin'
* Add deployment target overrides to failing autoconf packages
* Move configure workaround to base autoconf package
This reverts commit 3c119eaf8b4fb37c943d503beacf5ad2aa513d4c.
* Stylin'
* macos: add utility functions for SDK
These aren't yet used but should probably be added to spack debug
report.
* Remove node_target_satisfies/3 in favor of target_satisfies/2
When emitting input facts we don't need to couple target with
packages, but we can emit fewer facts independently and let
the grounder combine them.
* Remove compiler_version_satisfies/4 in favor of compiler_version_satisfies/3
When emitting input facts we don't need to couple compilers with
packages, but we can emit fewer facts independently and let
the grounder combine them.
* Introduce heuristic in the ASP-program
With heuristic we can drive clingo to make better
initial guesses, which lead to fewer choices and
conflicts in the overall solve
This improves the stand-alone tests for slate by providing most
of the dependencies to the test framework and enabling stand-alone
tests on all versions except the oldest.
* AMReX: +tiny_profile
The tiny profiler options in AMReX are by default off but needed
by WarpX. Adds a new variant to control it.
* Add Erik Palmer as Co-Maintainer
... so he receives pings on updates of the package for review.
The version of the ONNX submodule was updated between the PyTorch
1.9 and 1.10 releases, which fixed builds with newer protobuf but
broke builds with older protobuf.
Also this adds minimum version reqs for numpy/typing-extensions
(which were not present before).
* gcc: revise patch range on darwin
* gcc: add conflict to work around bootstrap failure
closes#23296 . See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100340
.
```
Comparing stages 2 and 3
Bootstrap comparison failure!
gcc/tree-ssa-operands.o differs
gcc/tree-ssanames.o differs
gcc/ipa-inline.o differs
gcc/tree-ssa-pre.o differs
gcc/gimple-loop-interchange.o differs
...
```
639 total differences.
* gcc: bump conflict up to correct later version
* Fix reindex with uninstalled deps
When a prefix of a dep is removed, and the db is reindexed, it is added
through the dependent, but until now it incorrectly listed the spec as
'installed'.
There was also some questionable behavior in the db when the same spec
was added multiple times, it would always be marked installed.
* Always reserve path
* Only add installed spec's prefixes to install prefixes set
* Improve warning, and ensure ensure only ensures
* test: reindex with every file system remnant removed except for the old index; it should give a database with nothing installed, including records with installed==False,external==False,ref_count==0,explicit=True, and these should be removable from the database
* stacks: add regression tests for matrix expansion
* Use constrain semantics to construct spec lists for stacks
* Fix semantics for constraining an anonymous spec. Add tests
Since in Spack we pull binaries out of the `warpx` package, we don't
need `py-cmake` to build `py-warpx`.
Generally, `py-cmake` in `pyproject.toml` is just a mean for us to
tell `pip` to make a `cmake` CLI tool available.
* added package gptune with all its dependencies: adding py-autotune, pygmo, py-pyaml, py-autotune, py-gpy, py-lhsmdu, py-hpbandster, pagmo2, py-opentuner; modifying superlu-dist, py-scikit-optimize
* adding gptune package
* minor fix for macos spack test
* update patch for py-scikit-optimize; update test files for gptune
* fixing gptune package style error
* fixing unit tests
* a few changes reviewed in the PR
* improved gptune package.py with a few newly added/improved dependencies
* fixed a few style errors
* minor fix on package name py-pyro4
* fixing more style errors
* Update var/spack/repos/builtin/packages/py-scikit-optimize/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* resolved a few issues in the PR
* fixing file permissions
* a few minor changes
* style correction
* minor correction to jq package file
* Update var/spack/repos/builtin/packages/py-pyro4/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixing a few issues in the PR
* adding py-selectors34 required by py-pyro4
* improved the superlu-dist package
* improved the superlu-dist package
* moree changes to gptune and py-selectors34 based on the PR
* Update var/spack/repos/builtin/packages/py-selectors34/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* improved gptune package: 1. addressing comments of tldahlgren in PR 26936; 2. adding variant openmpi
* fixing style issue of gptune
* changing file mode
* improved gptune package: add variant mpispawn which depends on openmpi; add variant superlu and hypre for installing the drivers; modified hypre package file to add a gptune variant
* fixing style error
* corrected pddrive_spawn path in gptune test; enforcing gcc>7
* fixing style error
* setting environment variables when loading gptune
* removing debug print in hypre/package.py
* adding superlu-dist v7.2.0; fixing an issue with CMAKE_INSTALL_LIBDIR
* changing site_packages_dir to python_platlib
* not using python3.9 for py-gpy, which causes due to dropped support of tp_print
* more replacement of site_packages_dir
* fixing a few dependencies in gptune; added a gptune version
* adding url for gptune
* minor correction of gptune
* updating versions in butterflypack
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [py-xonsh] added py-xonsh package
* [py-xonsh] change dependency to python 3.6
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add sticky variants
* Add unit tests for sticky variants
* Add documentation for sticky variants
* Revert "Revert 19736 because conflicts are avoided by clingo by default (#26721)"
This reverts commit 33ef7d57c1.
* Add stickiness to "allow-unsupported-compiler"
- To retrieve the correct spack version we need to get it from the git
repo.
- Recommend installing the package with root for production
- Add Tomas as maintainer to sarus' spack package
- Add the option to disable unit tests in latest versions
Fixes the following build failure when building with gcc 11:
478 ../../../../CPP/7zip/Archive/Wim/WimHandler.cpp: In member function 'virtual LONG NArchive::NWim::CHandler::GetArchiveProperty(PROPID, PROPVARIANT*)':
>> 479 ../../../../CPP/7zip/Archive/Wim/WimHandler.cpp:308:11: error: use of an operand of type 'bool' in 'operator++' is forbidden in C++17
480 308 | numMethods++;
481 | ^~~~~~~~~~
>> 482 ../../../../CPP/7zip/Archive/Wim/WimHandler.cpp:318:9: error: use of an operand of type 'bool' in 'operator++' is forbidden in C++17
483 318 | numMethods++;
484 | ^~~~~~~~~~
* opencv: add new version, variant, and patch
- added version 4.5.4
- added tesseract variant
- added patch to not add system paths
* Add leptonica depends and contrib conflicts
* Add dependencies for 1394 support
- new package: libraw1394
- add sdl dependency to libdc1394
- add conflict for openjpeg and jasper
* Adjust dependencies and conflicts for opencv modules
* rewrite of opencv
- all prebuilt apps are now variants and can be installed
- core is no longer a variant. It was always built anyway so it was not
really a variant.
- contrib is no longer a variant. All of the contrib modules are now
available as variants.
- components that can not be built with Spack are no longer variants.
They are set to 'off' to prevent pulling from system.
- handle the case where a module and a component have the same name
- use `with when` framework
- adjust dependencies and conflicts
- new package: libraw1394
- have libdc1394 depend on libraw1394
- patch to find clp
- patch to find onnx
- patch for cvv to find Qt
- format with black
* Incorporate recommended changes
- fix variants and dependencies on packages that depend on opencv
- remove opencv-3.2 and patches
- add some new patches to handle different versions
- cntk needs further work
- the openvslam package was markde deprecated as it is no longer an
active project and the repository has no code
* Remove gmake dependency.
* Remove sdl support
SDL is only used in an example case, but the examples are not built.
* remove openvslam
* Remove opencv+flann variant from 3dtk
* Back out cfitsio constraint from py-astropy
* remove opencv+flann variant from dlib
* remove boost constraint from 3dtk
* Remove non-opencv related bohrium changes
* Adjustments for cntk
- protobuf constraint at version 3.10
- need specific variants for opencv
- improve patch
* Deprecate CNTK package
* variant tweaks
- added appropriate conflicts for cublas
- made cuda/cudev relationship explicit
- moved openx to pending components as it needs an openvx package
* fix isort style error
* Use date version from kaldi rather than commit
* Revert changes from a bad rebase
* Add +flann to 3dtk and dlib
* Use compression support with libtiff
* remove `+datasets` from opencv dependency
The py-torchgeo package does not need opencv+datasets.
* fix typo
zip --> zlib
* added package gptune with all its dependencies: adding py-autotune, pygmo, py-pyaml, py-autotune, py-gpy, py-lhsmdu, py-hpbandster, pagmo2, py-opentuner; modifying superlu-dist, py-scikit-optimize
* adding gptune package
* minor fix for macos spack test
* update patch for py-scikit-optimize; update test files for gptune
* fixing gptune package style error
* fixing unit tests
* a few changes reviewed in the PR
* improved gptune package.py with a few newly added/improved dependencies
* fixed a few style errors
* minor fix on package name py-pyro4
* fixing more style errors
* Update var/spack/repos/builtin/packages/py-scikit-optimize/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* resolved a few issues in the PR
* fixing file permissions
* a few minor changes
* style correction
* minor correction to jq package file
* Update var/spack/repos/builtin/packages/py-pyro4/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixing a few issues in the PR
* adding py-selectors34 required by py-pyro4
* improved the superlu-dist package
* improved the superlu-dist package
* moree changes to gptune and py-selectors34 based on the PR
* Update var/spack/repos/builtin/packages/py-selectors34/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* improved gptune package: 1. addressing comments of tldahlgren in PR 26936; 2. adding variant openmpi
* fixing style issue of gptune
* changing file mode
* improved gptune package: add variant mpispawn which depends on openmpi; add variant superlu and hypre for installing the drivers; modified hypre package file to add a gptune variant
* fixing style error
* corrected pddrive_spawn path in gptune test; enforcing gcc>7
* fixing style error
* setting environment variables when loading gptune
* removing debug print in hypre/package.py
* adding superlu-dist v7.2.0; fixing an issue with CMAKE_INSTALL_LIBDIR
* changing site_packages_dir to python_platlib
* not using python3.9 for py-gpy, which causes due to dropped support of tp_print
* more replacement of site_packages_dir
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* espnet first build with depends
* fixed flake8
* updated to lastest version and removed python dependency
* changed to pypi and version 2.17.2
* [py-kaldiio] depends on py-pytest-runner
* [py-kaldiio] updated copyright
Co-authored-by: Sid Pendelberry <sid@rit.edu>
* prmon: make sure integration tests do not run in parallel
Some integration tests fail if not run on an otherwise idle machine.
* prmon: run unittests based on googletest
* prmon: fix checksums
* superlu-dist: use CMakePackage helper functions
* Fix#28609
It's OK to have CUDA in the dependency tree as long as it's not being
used for superlu-cuda.
* Update prmon package to latest versions
Add recent versions of the prmon package
Add the spdlog dependency for versions >=3
Add package developers as additional maintainers
Update name of development branch to 'main'
* Correct checksum for v3.0.1
at-spi2-core is automatically selecting dbus-broker and enabling systemd if it finds dbus-broker-launch which some systems might have even without systemd being part of the actual spack environment. This is not ideal for a spack package.
ucx has the configure option --[enable|disable]-backtrace detail.
This option is not explicitely set by spack, causing problems on my system, because
./configure does not find the bfd.h header file / libbfd.so library.
Added variant + dependencies (binutils). Disabled by default
* trilinos: version 12 requires cxxstd=11
* trilinos: use cmake version 3.21 or old when trilinos version 12
* conflict cxxstd=17 and cmake@3.2.[01]
* trilinos: version 12 requires cxxstd=11.
* Trilinos_CXX11_FLAGS is set to ' ' to avoid inject C++11 flag.
* set Trilinos_CXX11_FLAGS only version 12 or older.
* trilinos: update dependencies
Use the tribits deps to clarify some dependencies, and group some together
using `with` statements, eliminating some transitive conflict duplication.
* trilinos: Restricit cuda incompatibility
* e4s: vastly reduce number of packages in trilinos-cuda build
Not clear who the customers of cuda-enabled trilinos are, or what options
they need, or which sets of options conflict...
* e4s: remove ~wrapper from trilinos+cuda
* VTK-m: Make vtk-m consistent with ROCmPackage
* VTKm: Add kokkos variant
Specifying +kokkos will enable kokkos backend.
Specifying +kokkos with +rocm will require a kokkos with a ROCm backend.
Specifying +cuda enables VTK-m native CUDA backend. VTK-m native cuda backend
is not compatible with the kokkos +cuda backend.
* VTK-m: Add cuda_native variant
Required to allow specifying a vtk-m spec the selects a
cuda_arch and predictably propagate that to the underlying kokkos
dependency.
This also makes explicit selecting kokkos with a cuda backend or using
the VTK-m cuda backend.
* Mesa(18): Use libllvm virtual package
* Mesa patch configuration
Patch Mesa to define LLVM_VERSION_SUFFIX if llvm is pre-release
* Patch llvm-config to define LLVM_VERSION_SUFFIX
* Add a new version to track development
The released versions do not properly install via cmake which leads to
errors when linking against the library. These upstream problems have
been addressed on the glm development branch.
* Move git to class level and remove redundant depends
* vecgeom: require exact version of veccore
Fixes configure error from downstream package:
```
CMake Error at /rnsdhpc/code/spack/opt/spack/apple-clang/cmake/7zgbrwt/share/cmake-3.22/Modules/CMakeFindDependencyMacro.cmake:47 (find_package):
Could not find a configuration file for package "VecCore" that is
compatible with requested version "0.8.0".
The following configuration files were considered but not accepted:
/rnsdhpc/code/spack/var/spack/environments/celeritas/.spack-env/view/lib/cmake/VecCore/VecCoreConfig.cmake, version: 0.6.0
```
* veccore: add new versions
* Add flags to cabana to enable hypre and heffte when they are part of spec. Also add googletest to build dependencies
* Fixed mixed spaced and tabs
* Update package.py
* Update package.py
* Update package.py
* Modified to request specifically heFFTe version 2.0.0 due to
limitations in heFFTe cmakefiles.
* Update var/spack/repos/builtin/packages/cabana/package.py
Co-authored-by: Christoph Junghans <christoph.junghans@gmail.com>
* Integrated more heffte and hypre versions into cabana requests
Co-authored-by: Christoph Junghans <christoph.junghans@gmail.com>
* ParaView/VTK: Constrain version for ADIOS2 patch.
Older available versions of ParaView/VTK predate
ADIODS2 support.
ParaView lower bound is 5.8 and VTK lower bound is 8.2.0
* ParaView: Gate the ADIOS2 by verison
It seems that spack reads the output of `setup_run_environment` to build the actual spack modules and lmod modules. So, any output here will used verbatim on the shell.
This patch fixes https://github.com/spack/spack/issues/26733
1. adding latest release 3.5.0
2. updating cmake requirement to match that of Kokkos
3. adding logic to depend on the right version of Kokkos by default
* Kokkos: updating package list, maintainers and minimum cmake version
* Kokkos: updating maintainers list
Updating maintainers list to have the correct GitHub handle for Jan.
`spack license update-copyright-year` was updating license headers but not the MIT
license file. Make it do that and add a test.
Also simplify the way we bump the latest copyright year so that we only need to
update it in one place.
* [kaldi] Added version 2021-11-16
* [kaldi] Added logic for new version and when cuda 11 is used
* [kaldi] Added patch file when cuda 11 as cub is now built into it
* [kaldi] removed .999 and simplified some logic
Co-authored-by: Doug Heckman <dahdco@rit.edu>
* add py-ats package
* add new 7.0.10 tag
* add myself as a maintainer
* add dependencies for python and setuptools
* style
* added todo for flux
* words
* update versions users should use
* Use pip to bootstrap pip
* Bootstrap wheel from source
* Update PythonPackage to install using pip
* Update several packages
* Add wheel as base class dep
* Build phase no longer exists
* Add py-poetry package, fix py-flit-core bootstrapping
* Fix isort build
* Clean up many more packages
* Remove unused import
* Fix unit tests
* Don't directly run setup.py
* Typo fix
* Remove unused imports
* Fix issues caught by CI
* Remove custom setup.py file handling
* Use PythonPackage for installing wheels
* Remove custom phases in PythonPackages
* Remove <phase>_args methods
* Remove unused import
* Fix various packages
* Try to test Python packages directly in CI
* Actually run the pipeline
* Fix more packages
* Fix mappings, fix packages
* Fix dep version
* Work around bug in concretizer
* Various concretization fixes
* Fix gitlab yaml, packages
* Fix typo in gitlab yaml
* Skip more packages that fail to concretize
* Fix? jupyter ecosystem concretization issues
* Solve Jupyter concretization issues
* Prevent duplicate entries in PYTHONPATH
* Skip fenics-dolfinx
* Build fewer Python packages
* Fix missing npm dep
* Specify image
* More package fixes
* Add backends for every from-source package
* Fix version arg
* Remove GitLab CI stuff, add py-installer package
* Remove test deps, re-add install_options
* Function declaration syntax fix
* More build fixes
* Update spack create template
* Update PythonPackage documentation
* Fix documentation build
* Fix unit tests
* Remove pip flag added only in newer pip
* flux: add explicit dependency on jsonschema
* Update packages that have been added since this was branched off of develop
* Move Python 2 deprecation to a separate PR
* py-neurolab: add build dep on py-setuptools
* Use wheels for pip/wheel
* Allow use of pre-installed pip for external Python
* pip -> python -m pip
* Use python -m pip for all packages
* Fix py-wrapt
* Add both platlib and purelib to PYTHONPATH
* py-pyyaml: setuptools is needed for all versions
* py-pyyaml: link flags aren't needed
* Appease spack audit packages
* Some build backend is required for all versions, distutils -> setuptools
* Correctly handle different setup.py filename
* Use wheels for py-tomli to avoid circular dep on py-flit-core
* Fix busco installation procedure
* Clarify things in spack create template
* Test other Python build backends
* Undo changes to busco
* Various fixes
* Don't test other backends
* Add new package to spack. survey is a lightweight application performance tool that also gathers system information and stores it as metadata.
* Add maintainer and note about source access.
* Update the man path per spack reviewer suggestion.
* Remove redundant settings for PYTHONPATH, PATH, and MANPATH.
* Move to a one mpi collector approach for cce/tce integration.
* Add pyyaml dependency
* Make further spack reviewer changes to python type specs, mpi args, build type variant.
* Add reviewer requested changes.
* Add reviewer docstring requested changes.
* Add more updates from spack reviewer comments.
* Update the versions to use tags, not branches
* Redo dashes to fix issue with spack testing.
Co-authored-by: Jim Galarowicz <jgalarowicz@newmexicoconsortium.org>
When `spack compiler list` is run without being restricted to a
particular scope, and no compilers are found, say that none are
available, and hint that the use should run spack compiler find to
auto detect compilers.
* Improve docs
* Check if stdin is a tty
* add a test
Backport a patch for v1.3.4 that fixes an unsigned typedef problem
on macOS: https://github.com/xiph/ogg/pull/64
Also add v1.3.5 that has this issue fixed.
spack paths can be long and this overflows (at least) these buffers
inside of the bundled T1lib inside of the grace distribution, leading
to crashes on startup.
Charm++ versions below 7.0.0 have build issues on macOS, mainly due to the
pre-7.0.0 `VERSION` file conflicting with other version files on the
system: https://github.com/UIUC-PPL/charm/issues/2844. Specifically, it
conflicts with LLVM's `<version>` header that was added in llvm@7.0.0 to
comply with the C++20 standard:
https://en.cppreference.com/w/cpp/header/version. The conflict only occurs
on case-insensitive file systems, as typically used on macOS machines.
Many packages implement logic at the class level to handle complex dependencies and
conflicts. Others have started using `with when("@1.0"):` blocks since we added that
capability. The loops and other control logic can cause some pure directive logic not to
be removed by our package hashing logic -- and in many cases that's a lot of code that
will cause unnecessary rebuilds.
This commit changes the unparser so that it will descend into these blocks. Specifically:
1. Descend into loops, if statements, and with blocks at the class level.
2. Don't look inside function definitions (in or outside a class).
3. Don't look at nested class definitions (they don't have directives)
4. Add logic to *remove* empty loops/with blocks/if statements if all directives
in them were removed.
This allows our package hash to ignore a lot of pure metadata that it was not ignoring
before, and makes it less sensitive.
In addition, we add `maintainers` and `tags` to the list of metadata attributes that
Spack should remove from packages when constructing canonoical source for a package
hash.
- [x] Make unparser handle if/for/while/with at class level.
- [x] Add tests for control logic removal.
- [x] Add a test to ensure that all packages are not only unparseable, but also
that their canonical source is still compilable. This is a test for
our control logic removal.
- [x] Add another unparse test package that has complex logic.
These are the unit tests from astunparse, converted to pytest, with a few backports from
upstream cpython. These should hopefully keep `unparser.py` well covered as we change it.
We can't tell `print(a, b, c)` and `print((a, b, c))` apart -- both of these expressions
generate different ASTs in Python 2 and Python 3. However, we can decide that we don't
care. This commit treats both of them the same when `py_ver_consistent` is set with
`unparse()`.
This means that the package hash won't notice changes from printing a tuple to printing
multiple values, but we don't care, because this is extremely unlikely to affect the build.
More than likely this is just an error message for the user of the package.
- [x] treat `print(a, b, c)` and `print((a, b, c))` the same in py2 and py3
- [x] add another package parsing test -- legion -- that exercises this feature
To make it easier to see how package hashes change and how they are computed, add two
commands:
* `spack pkg source <spec>`: dumps source code for a package to the terminal
* `spack pkg source --canonical <spec>`: dumps canonicalized source code for a
package to the terminal. It strips comments, directives, and known-unused
multimethods from the package. It is used to generate package hashes.
* `spack pkg hash <spec>`: This gives the package hash for a particular spec.
It is generated from the canonical source code for the spec.
- [x] `add spack pkg source` and `spack pkg hash`
- [x] add tests
- [x] fix bug in multimethod resolution with boolean `@when` values
Co-authored-by: Greg Becker <becker33@llnl.gov>
We are planning to switch to using full hashes for Spack specs, which means that the
package hash will be included in the deployment descriptor. This means we need a more
robust package hash than simply dumping the `repr` of the AST.
The AST repr that we previously used for package content is unreliable because it can
vary between python versions (Python's AST actually changes fairly frequently).
- [x] change `package_hash`, `package_ast`, and `canonical_source` to accept a string for
alternate source instead of a filename.
- [x] consolidate package hash tests in `test/util/package_hash.py`.
- [x] remove old `package_content` method.
- [x] make `package_hash` do what `canonical_source_hash` was doing before.
- [x] modify `content_hash` in `package.py` to use the new `package_hash` function.
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Our package hash is supposed to be consistent from python version to python version.
Test this by adding some known unparse inputs and ensuring that they always have the
same canonical hash. This test relies on the fact that we run Spack's unit tests
across many python versions. We can't compute for several python versions within the
same test run so we precompute the hashes and check them in CI.
Package hashing was not properly handling multimethods. In particular, it was removing
any functions that had decorators from the output, so we'd miss things like
`@run_after("install")`, etc.
There were also problems with handling multiple `@when`'s in a single file, and with
handling `@when` functions that *had* to be evaluated dynamically.
- [x] Rework static `@when` resolution for package hash
- [x] Ensure that functions with decorators are not removed from output
- [x] Add tests for many different @when scenarios (multiple @when's,
combining with other decorators, default/no default, etc.)
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Previously we used `directives.__all__` to get directive names, but it wasn't
quite right -- it included `DirectiveMeta`, etc. It's not wrong, but it's also
not the clearest way to do this.
- [x] Refactor `@directive` to track names in `directive_names` global
- [x] Rename `_directive_names` to `_directive_dict_names` in `DirectiveMeta`
- [x] Add a test for `RemoveDirectives`
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Some packages use top-level unassigned strings instead of comments, either just after a
docstring on in the body somewhere else. Ignore those strings becasue they have no
effect on package behavior.
- [x] adjust RemoveDocstrings to remove all free-standing strings.
- [x] move tests for util/package_hash.py to test/util/package_hash.py
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Python 2 and 3 represent string literals differently in the AST. Python 2 requires '\x'
literals, and Python 3 source is always unicode, and allows unicode to be written
directly. These also unparse differently by default.
- [x] modify unparser to write both out the way `repr` would in Python 2 when
`py_ver_consistent` is provided.
Backport operator precedence algorithm from here:
397b96f6d7
This eliminates unnecessary parentheses from our unparsed output and makes Spack's unparser
consistent with the one in upstream Python 3.9+, with one exception.
Our parser normalizes argument order when `py_ver_consistent` is set, so that star arguments
in function calls come last. We have to do this because Python 2's AST doesn't have information
about their actual order.
If we ever support only Python 3.9 and higher, we can easily switch over to `ast.unparse`, as
the unparsing is consistent except for this detail (modulo future changes to `ast.unparse`)
Previously, there were differences in the unparsed code for Python 2.7 and for 3.5-3.10.
This makes unparsed code the same across these Python versions by:
1. Ensuring there are no spaces between unary operators and
their operands.
2. Ensuring that *args and **kwargs are always the last arguments,
regardless of the python version.
3. Always unparsing print as a function.
4. Not putting an extra comma after Python 2 class definitions.
Without these changes, the same source can generate different code for different
Python versions, depending on subtle AST differences.
One place where single source will generate an inconsistent AST is with
multi-argument print statements, e.g.:
```
print("foo", "bar", "baz")
```
In Python 2, this prints a tuple; in Python 3, it is the print function with
multiple arguments. Use `from __future__ import print_function` to avoid
this inconsistency.
Add `astunparse` as `spack_astunparse`. This library unparses Python ASTs and we're
adding it under our own name so that we can make modifications to it.
Ultimately this will be used to make `package_hash` consistent across Python versions.
Add an abstraction around libllvm to allow libllvm
providers to be specified for all packages.
This is targeting allowing mesa to build against
llvm-amdgpu or intel-llvm or llvm or any other
custom llvm variant that arises for specific GPU
toolchains
* Python: set default config_vars
* Add missing commas
* dso_suffix not present for some reason
* Remove use of default_site_packages_dir
* Use config_vars during bootstrapping too
* Catch more errors
* Fix unit tests
* Catch more errors
* Update docstring
* Update existing 2020.0 version to use tag
* Add versions 2018.2 and master
* Add patches for GCC/Intel
* Use MPI compiler wrappers when +mpi
* Constrain CMake build dependency (need >= 3.1)
* Add variants for optional components (e.g QFIT library)
./configure tries to execute an MPI test, which is not possible on
most HPC platforms (if you don't build on a compute node), so this
check is disabled to allow the build to proceed. Ideally we could
check this by placing constraints on the MPI that Spack builds (e.g.
require building a version that is guaranteed to have threading
support).
* rocm recipes updates for 4.5.0
* update to rocm recipes for 4.5.0 release
* updates to the rocm recipes for rocm-4.5.0 release
* fix style errors
* update to rocm-validation-suite for rocm-4.5.0 release
* bump up rccl recipe for rocm-4.5.0
* bump up version for rdc for rocm-4.5.0
* update miopengemm, miopen-opencl,rocm-opencl recipes for 4.5.0 release
* bump up version for mivisiox for rocm-4.5.0 release
* update the rocm-validation-suite recipe
* no need to change the perl path for 4.5.0
* fix the build failure with the recent change made for hip package
* modify checksum for the llvm-amdgpu for 4.5.0
* fix the build issue aftere recent changes made for enabling test
* fix the build issue with 4.5.0
* add new recipe for hipsolver
* address review comments
* llvm: make targets a multivalued variant
* Fix the targets variant values
1. Make them lowercase and add a mapping to cmake equivalent
2. auto -> all
2. Restore composability by using a multivalued variant, so that
`targets=all` and `targets=x86` is combined to `targets=all,x86`
which is then transformed into LLVM_TARGETS_TO_BUILD=all.
* use targets=x86 in iwyu
* Default to nvptx/amdgpu/host arch targets
* default to none
* Update var/spack/repos/builtin/packages/zig/package.py
This reports the kernel version (vs. the distro version) on Linux and
returns a valid Version (stripping characters like '+' which may be
present for custom-built kernels).
Reading appdirs.py without explicitly requesting UTF-8 decoding results
in the build process to fail for Python 3.6.
See https://github.com/ActiveState/appdirs/pull/152 for the upstream
fix.
* snakemake: New version
The newer versions of snakemake have a lot of new dependencies. The
optional dependencies still have to be added.
* removed comment
* some changes
* added reports variant
* deprecate older version and add me as maintainer
* Added dependency py-ratelimiter
* Fix: py-adios Cython run
Always run Cython before `py-adios` installs.
This makes sure the `.cpp` files from `.pyx` files are freshly
created and work with newer CPython versions than the one checked
in.
* Cleanup: `rm` -> `os.remove`
* Drop preferred hdf5 version
* Fix conduit
* Add compat bound for silo on hdf5
* Update var/spack/repos/builtin/packages/conduit/package.py
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
* hdf5 <= 1.10 for conduit <= 0.7
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
Doxygen's build system uses CMake's deprecated `FindPythonInterp`,
which can get confused by other `python` executables in the PATH.
See issue: https://github.com/spack/spack/issues/28215
Patch to add --embed flag to config-python when interface=python and
using python@3.8:
This is because python@3.8 changed behavior of python-config --ldflags
(and --libs) such that it no longer includes -lpython unless --embed
flag is used.
See e.g. https://github.com/mesonbuild/meson/issues/5629
* [py-aiohttp] added new version
* [py-aiohttp] changed dependency on setuptools
* [py-aiohttp] fixed some of the dependencies
* [py-aiohttp] updated dependency to py-typing-extensions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New package: py-azure-storage-blob
* fixed typo
* Update var/spack/repos/builtin/packages/py-azure-storage-blob/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-azure-storage-blob/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-azure-storage-blob/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* libtiff: fix build on macOS Monterey
* patch configure, not configure.ac
* Revert "patch configure, not configure.ac"
This reverts commit 8bf315cb22.
* Force Spack to run autoreconf using new patch
* [py-aiohttp] added new version
* [py-aiohttp] updated py-setuptools version dependency
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
For versions of aws-parallelcluster >= 2.9, the pyyaml dependency had to be >= 5.3.1 and == 5.1.2
at the same time making impossible to install ParallelCluster >= 2.9 from spack repository.
See issue: https://github.com/spack/spack/issues/28172
Fixed by limiting pyyaml 5.1.2 version to aws-parallelcluster < 2.8, according to this commit:
7255d314b7
Tested with a manual installation of aws-parallelcluster@2.11.4
Add a new check to `spack audit` to scan and verify that version constraints may be satisfied
Modifications:
- [x] Add a new check to `spack audit` to scan and verify that version constraints may be satisfied by some version declared in the built-in repository
- [x] Fix issues found by CI
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-datrie: New package, initial commit
* removed boilerplate
* Update var/spack/repos/builtin/packages/py-datrie/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-datrie/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* trilinos: fix define_tpl to handle depspecs w/out headers
This should address #27758 (i.e. errors due to netlib-scalapack not
having headers)
* trilinos: This fixes a mismatch in variant name and spec name for x11/libx11
* [py-minio] Added py-minio package
* [py-minio] added missing dependencies for py-minio
* [py-minio] added type=('build', 'run') in dependencies
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [py-dh-scikit-optimize] Added new versions
* [py-dh-scikit-optimize] fixing dependencies
* making py-configspace a required dependency of py-dh-scikit-learn
* r-hdf5r: use pkg-config to find hdf5
Since hdf5 was switched from autotools to cmake, the hdf5 compiler
wrappers can not be used to find and configure hdf5. This PR switches to
using pkg-config for configuration.
* Add comment about configure patch
* The Class F problem has been added to seven of the benchmarks
(BT, SP, LU, CG, MG, FT, and EP).
* The Class E problem has been added to the IS benchmark.
* In version 3.4.1, 'the number of processes' option does not apply.
* MPIFC and FC flags were added.
These versions change the install location of CMake files used
by dependents, but most dependents don't seem to look in this
new location.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Chris White <white238@llnl.gov>
This command pokes the environment, Python interpreter
and bootstrap store to check if dependencies needed by
Spack are available.
If any are missing, it shows a comprehensible message.
* locks: allow locks to work under high contention
This is a bug found by Harshitha Menon.
The `lock=None` line shouldn't be a release but should be
```
return (lock_type, None)
```
to inform the caller it couldn't get the lock type requested without
disturbing the existing lock object in the database. There were also a
couple of bugs due to taking write locks at the beginning without any
checking or release, and not releasing read locks before requeueing.
This version no longer gives me read upgrade to write errors, even
running 200 instances on one box.
* Change lock in check_deps_status to read, release if not installed,
not sure why this was ever write, but read definitely is more
appropriate here, and the read lock is only held out of the scope if
the package is installed.
* Release read lock before requeueing to reduce chance of livelock, the
timeout that caused the original issue now happens in roughly 3 of 200
workers instead of 199 on average.
With this commit:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
==> Updating view at /tmp/spack-faiirgmt/.spack-env/view
$ spack install zlib
==> All of the packages are already installed
```
Before this PR:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
$ spack install zlib
==> All of the packages are already installed
```
No view was generated
This commit introduces the command
spack module tcl setdefault <package>
similar to the one already available for lmod
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* upcxx: Update the UPC++ package to 2021.9.0
* Add the new release, and a missing older one.
* Remove the spack package cruft for supporting the obsolete build system that
was present in older versions that are no longer supported.
* General cleanups.
Support for library versions older than 2020.3.0 is officially retired,
for two reasons:
1. Releases prior to 2020.3.0 had a required dependency on Python 2,
which is [officially EOL](https://www.python.org/doc/sunset-python-2/)
as of Jan 1 2020, and is no longer considered secure.
2. (Most importantly) The UPC++ development team is unable/unwilling to
support releases more than two years old. UPC++ provides robust
backwards-compatibility to earlier releases of UPC++ v1.0, with very
rare well-documented/well-motivated exceptions. Users are strongly
encouraged to update to a current version of UPC++.
NOTE: Most of the lines changed in this commit are simply re-indentation,
and thus might be best reviewed in a diff that ignores whitespace.
* upcxx: Detect Cray XC more explicitly
This change is necessary to prevent false matches occuring on new Cray Shasta
systems, which do not use the aries network but were incorrectly being treated
as a Cray XC + aries platform.
UPC++ has not yet deployed official native support for Cray Shasta, but this
change is sufficient to allow building the portable backends there.
When running `spack install --log-format junit|cdash ...`, install
errors were ignored. This made spack continue building dependents of
failed install, ignoring `--fail-fast`, and exit 0 at the end.
* locks: allow locks to work under high contention
This is a bug found by Harshitha Menon.
The `lock=None` line shouldn't be a release but should be
```
return (lock_type, None)
```
to inform the caller it couldn't get the lock type requested without
disturbing the existing lock object in the database. There were also a
couple of bugs due to taking write locks at the beginning without any
checking or release, and not releasing read locks before requeueing.
This version no longer gives me read upgrade to write errors, even
running 200 instances on one box.
* Change lock in check_deps_status to read, release if not installed,
not sure why this was ever write, but read definitely is more
appropriate here, and the read lock is only held out of the scope if
the package is installed.
* Release read lock before requeueing to reduce chance of livelock, the
timeout that caused the original issue now happens in roughly 3 of 200
workers instead of 199 on average.
Fixes#27652
Ensure that mirror's to_dict function returns a syaml_dict object for all code
paths.
Switch to using the .get function for accessing the potential information from
the S3 mirror objects. If the key is not there, it will gracefully return
None instead of failing with a KeyError
Additionally, check that the connection object is a dictionary before trying
to "get" from it.
Add a test for the capturing of the new S3 information.
* perl: fix macOS build
With both 5.34.0 and 5.32.1 the build fails on macos-bigsur-skylake
%clang@12.0.5 and %clang13.0.0 :
```
2 errors found in build log:
579013 /private/var/folders/fy/x2xtwh1n7fn0_0q2kk29xkv9vvmbqb/T/s3j/spack-stage/spack-stage-perl-5.34.0-tpha2u52qfwaraidpzzbf6u4dbqg7dk5/spack-src/cpan/
Math-BigInt-FastCalc/../../miniperl "-I../../lib" -MExtUtils::Command::MM -e 'cp_nonempty' -- FastCalc.bs ../../lib/auto/Math/BigInt/FastCalc/Fas
tCalc.bs 644
579014
579015 Everything is up to date. Type '/Applications/Xcode.app/Contents/Developer/usr/bin/make test' to run test suite.
579016 DYLD_LIBRARY_PATH=/private/var/folders/fy/x2xtwh1n7fn0_0q2kk29xkv9vvmbqb/T/s3j/spack-stage/spack-stage-perl-5.34.0-tpha2u52qfwaraidpzzbf6u4dbqg7d
k5/spack-src ./perl -Ilib -I. installperl --destdir=
579017 WARNING: You've never run 'make test' or some tests failed! (Installing anyway.)
579018 /rnsdhpc/code/spack/opt/spack/apple-clang/perl/tpha2u5/bin/perl5.34.0
>> 579019 install_name_tool: error: sh -c '/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode.app/Contents/Developer/Pl
atforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -find install_name_tool 2> /dev/null' failed with exit code 256: (null) (errno=Invalid argument
)
579020 xcode-select: Failed to locate 'install_name_tool', requesting installation of command line developer tools.
579021 Cannot update /rnsdhpc/code/spack/opt/spack/apple-clang/perl/tpha2u5/bin/perl5.34.0 dependency paths
>> 579022 make: *** [install-all] Error 72
```
This is due to SYSTEM_VERSION_COMPAT being set.
* perl: conditionally set SYSTEM_VERSION_COMPAT based on CLT
The version of command line tools is the only difference between
@alalazo and my builds: his (v11) works only when SYSTEM_VERSION_COMPAT
is set to 1, and mine (v12.5 and v13) only work when it is unset.
With this commit:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
==> Updating view at /tmp/spack-faiirgmt/.spack-env/view
$ spack install zlib
==> All of the packages are already installed
```
Before this PR:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
$ spack install zlib
==> All of the packages are already installed
```
No view was generated
* New package: py-onnxmltools and dependencies
* Small fix
* Changes from review
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update recipe following review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Updates to installer.py did not account for spack monitor, so as currently implemented
there are three cases of failure that spack monitor will not account for. To fix this we add additional
hooks, including an on cancel and also do a custom action on concretization fail.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
The latest version of `jsonschema` fails if we're not specific about which schema draft
specification we're using. Update all of them to use the latest one (draft-07).
Our `jsonschema` external won't support Python 3.10, so we need to upgrade it.
It currently generates this warning:
lib/spack/external/jsonschema/compat.py:6: DeprecationWarning: Using or importing the ABCs
from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and
in 3.10 it will stop working
This upgrades `jsonschema` to 3.2.0, the latest version with support for Python 2.7. The next
version after this (4.0.0) drops support for 2.7 and 3.6, so we'll have to wait to upgrade to it.
Dependencies have been added in prior commits.
* [geant4] new version 11.0.0
* [geant4] prefer 10.7.3 for now
* [vecgeom] new version 1.1.18
* [clhep] new version 2.4.5.1
* [g4emlow] new version 8.0
* [g4particlexs] new version 4.p
* [geant4-data] new version 11.0.0
* [geant4] @11.0.0: cxxstd=17: ^clhep@2.4.5.1: ^vecgeom@1.1.18:
* [geant4] depends_on cmake@3.16:
* [geant4-data] remove g4tendl comment
* [g4tendl] new version 1.4
* [geant4] default cxxstd=11 when @10, 17 when @11; use CMAKE_CXX_STANDARD
* [geant4] variant tbb whe @11:, depends_on tbb, sets GEANT4_USE_TBB
* [geant4] new variant vtk when @11:, depends_on vtk@8.2:
* [geant4] simplify GEANT4_USE_VTK with define_from_variant
* [geant4] remove variant cxxstd conditional again
* [geant4] flake8 space after comma
* What's new in AOCL 3.1
1) AMD BLIS:
1.a) Supports Dynamic Dispatch and AOCL Dynamic feature
1.b) Improvements in DGEMM, ZGEMM, DTRSM, DSYRK, xGEMV, and DOTV
2) AMD libFLAME:
2.a) Supports LAPACK 3.10.0 specification
2.b) Optimized factorization and ZGEEV routines
3) AMD FFTW:
3.a) Features like 'AMD application optimization layer', 'Fast MPI transpose algorithm' and 'Top N planner' are added
4) AMD LibM:
4.a) Optimized exp2, log2 (Single and Double precision) scalar and vector
4.b) Optimized log10f (scalar and vector) and powf vector variants to support WRF4.1.2 benchmark
5) AOCL-Sparse:
5.a) New API for sparse matrix and dense matrix multiplication
6) AMD ScaLAPACK:
6.a) ILP64 support has been enabled
7) AOCL enabled MUMPS library:
7.a) CMake based build system on Windows for AOCL enabled MUMPS sparse solver library will be available shortly on GitHub
7.a.i) Refer https://github.com/amd/mumps-build
7.b) Spack-based recipe on Linux for AOCL enabled MUMPS sparse solver library will be enabled shortly
* Fix invalid version range error
* Incorporated review comments
1) Restore to previous url value
2) Instead of if else statements, used spack's enable_or_disable feature
* Incorporated following review comments:
1. Use of satisfies() for spec checks
2. Seperate conflict statements to check for minimum and maximum GCC versions
3. Used CMakePackage helpers
4. Code rearrangement to have the directives listed before methods
* New package: py-xrootdpyfs
* Add old version of py-fs
* Replace 2to3.patch with running 2to3
* Just restriuct setuptools version
* Update var/spack/repos/builtin/packages/py-fs/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
spack monitor now requires authentication as each build must be associated
with a user, so it does not make sense to allow the --monitor-no-auth flag
and this commit will remove it
* suite-sparse: Fix check for +/- tbb variant
Changed
'tbb' in spec
to
'+tbb' in spec
The former would configure suite-sparse to use tbb if any dependency
package (e.g. intel-oneapi-mkl) depends on tbb, even if
suite-sparse~tbb was specified.
* suite-sparse: conflict when trying to use 2021.x versions of tbb
See https://github.com/DrTimothyAldenDavis/SuiteSparse/issues/72
suite-sparse depends on task_schedule_init to control the number
of threads when e.g. interfacing with MATLAB. However, Intel
dropped task_schedule_init in the 2021.x releases of TBB (it has
been deprecated since TBB 4.3.5).
We just raise a spack conflict when using tbb @2021.x and +tbb
Because tbb is a virtual package and is not versioned, I have
instead checked for either intel-oneapi-tbb@2021: or
intel-tbb@2021:, not the most elegant but should do the job
* suite-sparse: fix style issues
* Added installation of OpenMP as an option
* Added a softlink (dpcpp) to clang++ to
mimic the packaged version of dpcpp
Co-authored-by: ravil <ravil.dorozhinskii@tum.de>
* Fix building container images
Patchelf is bootstrapped from sources, so we cannot
disable that mechanism until a finer selection is
possible in the configuration.
* Build on changes to the Dockerfile
* Don't login to Dockerhub on PRs
This commit introduces the command
spack module tcl setdefault <package>
similar to the one already available for lmod
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* Update hdf5/package.py to add HDF5 1.10.8 release and move
preferred version from 1.10.7 to 1.10.8.
* silo: versions before 4.11 conflict with hdf5 >= 1.10.8.
* Add patch file for silo@4.11 with hdf5 1.10 >=1.10.8 and
hdf5 1.12 >= 1.12.1.
* New package: py-scinum
* Update var/spack/repos/builtin/packages/py-scinum/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fix for OpenFOAM pthread issue for AOCC 3.2
* addressing the review comments
* updating when command for aocc v3.2.0 and above
Co-authored-by: mohan babu <mohbabul@amd.com>
* PETSc is a core dependency, yet we left a variant for PETSc.
This was removed, and ExaGO always depends on PETSc. The CMake
arguments were updated accordingly.
* camp+cuda is only a dependency when we build with RAJA and CUDA.
* py-radical-entk: add version 1.9.0
* py-radical-pilot: add version 1.10.1
* py-radical-utils: add version 1.9.1
* py-radical-utils: requires py-pymongo less than version 4
* New package: py-pysqlite3
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This PR also slightly changes the behavior in ci_rebuild().
We now still attempt to submit `spack install` results to CDash
even if the initial registration failed due to connection issues.
This commit follows in the spirit of #24299. We do not want `spack install`
to exit with a non-zero status when something goes wrong while attempting to
report results to CDash.
* Add a CI job to audit all the packages in the built-in repository
* flecsi: fixed typo for dependency on legion
* py-pythonqwt: fix a typo in variant name
* sollve: removed a conflict with a non-existing variant
* acts: fixed use of wrong variant in dd4hep
Also removed duplicated variant declaration in dd4hep
* aoflagger: update variant of a dependency
Issues introduced indirectly in #22925
* camellia: removed unused variant
Issue introduced indirectly in #26150
* cbtf-*: remove cti variants and dependency on mrnet+cti
Issue introduced in #14178
* flecsale: update variants to match flecsi
Issue introduced in #11679
* grnboost: fixed issue with non-existing variant in a dependency
This package possibly never worked since #8763
* nalu: fixed issue with non-existing variant in a dependency
* open-iscsi: fixed issue with non-existing variant in a dependency
* openspeedshop-*: remove use of non-existing mrnet+cti variant
* percept: fixed issue with non-existing variant in a dependency
* phyluce: fixed issue with non-existing variant in a dependency
Issue introduced in #12952
* phyluce: fixed issue with non-existing variant in a dependency
Issue introduced in #22340
* CPU Architecture Support
This commit removes the `native` variant in favor of Spack's built-in support for specifying a target cpu architecture. It also passes this information to the Legion build system so that it correctly passes the architecture to GASNet when built internally
* fixing whitespace
* Update package.py
based on a conversation with @streichler, this change sets `BUILD_MARCH` to an empty string, which will prevent legion's CMake build system from inserting `-march=native` and allow Spack to provide the correct architecture flags.
* Update package.py
adding a comment on what problem this MR solves.
* Update package.py
formatting
* PICMI: 0.16 & 0.18 & WarpX 1D
Update the `py-picmistandard` and the latest WarpX release.
Preparing 1D support (testable inputs coming for 22.01+).
* Fix style: overlong line
* Update pypi example link
* Fix requirement ranges
* WarpX 21.12: Update Patch
Follow-up from
https://github.com/ECP-WarpX/WarpX/pull/2646
* fix style
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* openjpeg: add missing dependencies and optionally disable them
* openjpeg: remove variant 'ownlibs'
* openjpeg: bugfix
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* openjpeg: do not build CODEC executables by default
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New package: py-hist and it's dependencies
* Update var/spack/repos/builtin/packages/py-hist/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-histoprint/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-mplhep/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update py-hist recipe
* Update package.py
* Fix py-iminuit recipe (requires py-cmake now)
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New package: py-correctionlib
* Update var/spack/repos/builtin/packages/py-correctionlib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-correctionlib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add py-make and update py-correctionlib recipe
* Add py-bind11 dependency
* Update package.py
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This PR is meant to move code with "business logic" from `spack.cmd.buildcache` to appropriate core modules[^1].
Modifications:
- [x] Add `spack.binary_distribution.push` to create a binary package from a spec and push it to a mirror
- [x] Add `spack.binary_distribution.install_root_node` to install only the root node of a concrete spec from a buildcache (may check the sha256 sum if it is passed in as input)
- [x] Add `spack.binary_distribution.install_single_spec` to install a single concrete spec from a buildcache
- [x] Add `spack.binary_distribution.download_single_spec` to download a single concrete spec from a buildcache to a local destination
- [x] Add `Spec.from_specfile` that construct a spec given the path of a JSON or YAML spec file
- [x] Removed logic from `spack.cmd.buildcache`
- [x] Removed calls to `spack.cmd.buildcache` in `spack.bootstrap`
- [x] Deprecate `spack buildcache copy` with a message that says it will be removed in v0.19.0
[^1]: The rationale is that commands should be lightweight wrappers of the core API, since that helps with both testing and scripting (easier mocking and no need to invoke `SpackCommand`s in a script).
* New package: py-cmsml
* Update var/spack/repos/builtin/packages/py-cmsml/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New package: py-cx-oracle
* Add dependencies from pyproject.toml
* Update var/spack/repos/builtin/packages/py-cx-oracle/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixing bugs in spack monitor
updates to installer.py did not account for spack monitor, so as currently implemented
there are three cases of failure that spack monitor will not account for. To fix this we add additional
hooks, including an on cancel and also do a custom action on concretization fail.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
* New package: py-climate
* Revert "fixing bugs in spack monitor"
This reverts commit bf7f6bf0e3.
* Flake-8
* Update package.py
* Update package.py
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
After this PR an error in a single package while detecting
external software won't abort the entire procedure.
The error is reported to screen as a warning.
* New package: py-boost-histogram
* Update var/spack/repos/builtin/packages/py-boost-histogram/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Remove a try/catch for an error with no handling. If the affected
code doesn't execute successfully, then the associated variable
is undefined and another (more-obscure) error occurs shortly after.
Since #27185, the cuda_arch variant values are conditional on +cuda. This means that for -cuda specs, the installation fails with:
```
==> acts: Executing phase: 'cmake'
==> Error: KeyError: 'cuda_arch'
/home/wdconinc/git/spack/var/spack/repos/builtin/packages/acts/package.py:222, in cmake_args:
219 log_failure_threshold = spec.variants['log_failure_threshold'].value
220 args.append("-DACTS_LOG_FAILURE_THRESHOLD={0}".format(log_failure_threshold))
221
>> 222 cuda_arch = spec.variants['cuda_arch'].value
223 if cuda_arch != 'none':
224 args.append('-DCUDA_FLAGS=-arch=sm_{0}'.format(cuda_arch[0]))
225
```
* new package: py-tensorflow-datasets
- includes new dependency package: py-tensorflow-metadata
* Update var/spack/repos/builtin/packages/py-tensorflow-datasets/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-tensorflow-metadata/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Added gdb Dependency
When using spack to install cgdb, a spack-built gdb is necessary to
avoid dynamic link errors.
- Added maintainer: tuxfan
- Set preferred to 'master' (best version for spack currently)
* Update: The gdb dependency added by this PR is for runtime
Co-authored-by: Bernhard Kaindl <43588962+bernhardkaindl@users.noreply.github.com>
This adds support in spack for both build/install tests (spack install
--run-tests) and post-install smoke tests (spack test run).
Hpctoolkit itself only recently added tests, so for now, this only
applies to branch master.
To use this, you can "spack install intel-oneapi-compilers" and then
"spack compiler add" the new compiler. You would need to install with
"spack install ginkgo+oneapi%dpcpp"
- Use .tar.gz archive
- Update 2.3.3 to use .tar.gz archive (and update checksum)
- autoreconf dependency is no-longer required
- The new version depends on gperf
- Add sensei to the SDK with appropriate propagations
- Rework variants to SENSEI package to avoid providing broken builds
- Turn off miniapps by default, these are examples and not critical to using sensei
This PR updates the vtk package to use new variable names and adds some
dependencies.
- add version 9.1.0
- add version 1.4.2 to gl2ps package. This is needed to use gl2ps as a
dependency.
- new package: utf8cpp, used as a dependency for version 9.
- add dependencies when possible
- pugixml
- libogg
- libtheora
- utf8cpp
- gl2ps
- proj
- turn off configuring MPI if ~mpi
- always use the package-provided FindHDF5.cmake for versions up to 8.
Version @9: already does this so does not need a patch.
- use new CMake variables for version 9
- remove unused CMake variables depending on version
* py-packaging: add 21.3
* Update var/spack/repos/builtin/packages/py-packaging/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
A part of the gpgme testsuite by default even runs during normal
make and make install phases, creating a public keyring in ~/.gnupg.
Prevent this and avoid build failures in containers due to another
problem of the test suite and fix a test case of the new 0.16.0 release.
When the uidmap tools are installed on a system, this allows to run
containers as unprivileged user (rootless and daemonless) slimilar
to singularity, but using a familiar CLI: "alias docker=podman"
This is helpful to run e.g. spack builds in containers to reproduce
build failures from CI without requiring a installation of docker.
The required dependencies of podman are added as well.
Fix to not attempt to patch a nonexisting file for old versions
when building with gcc-11. Skip the build-time tests as all access
the X DISPLAY and open many windows on the screen.
I was browsing package metadata, as one does on a Sunday, and stumbled across a new kind of version attribute - brancch! I suspect this is supposed to be "branch."
Update flux-core and flux-sched package.py to include latest releases.
For flux-sched:
- Add patch to disable false-positive-happy valgrind test
- pin yaml-cpp to 0.6.3 due to issue described at:
https://github.com/flux-framework/flux-sched/issues/886
Starting with meson 0.60, unknown args produce errors and
the -Dx11 arg is only present in @:2.40
https://gitlab.gnome.org/GNOME/gtk-osx/-/issues/44
Add tiff variant: Default disabled since fails the tests in part.
Only libtiff@:3.9 pass, but these old versions have severe security issues.
Deprecate @:2.41 as they are affected by the high-severity CVE-2021-20240:
https://nvd.nist.gov/vuln/detail/CVE-2021-20240
Remove a custom bootstrapping procedure to
use spack.bootstrap instead
Modifications:
* Reference count the bootstrap context manager
* Avoid SpackCommand to make the bootstrapping
procedure more transparent
* Put back requirement on patchelf being in PATH for unit tests
* Add an e2e test to check bootstrapping patchelf
* Fix version constrains in py-ipykernel and py-ipython
Before the fix:
```
$ spack spec py-ipykernel@6.4.1 ^py-jupyter-client@7.0.6
==> Error: py-ipykernel@6.4.1 ^py-jupyter-client@7.0.6 is unsatisfiable, conflicts are:
no version satisfies the given constraints
```
After the fix:
```
```
(thanks god the old concretizer is still there - it provides sane error messages!)
* Fix py-ipython recipe
* Revert "Fix py-ipython recipe"
This reverts commit d65071665f.
* added package gptune with all its dependencies: adding py-autotune, pygmo, py-pyaml, py-autotune, py-gpy, py-lhsmdu, py-hpbandster, pagmo2, py-opentuner; modifying superlu-dist, py-scikit-optimize
* adding gptune package
* minor fix for macos spack test
* update patch for py-scikit-optimize; update test files for gptune
* fixing gptune package style error
* fixing unit tests
* a few changes reviewed in the PR
* improved gptune package.py with a few newly added/improved dependencies
* fixed a few style errors
* minor fix on package name py-pyro4
* fixing more style errors
* Update var/spack/repos/builtin/packages/py-scikit-optimize/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* resolved a few issues in the PR
* fixing file permissions
* a few minor changes
* style correction
* minor correction to jq package file
* Update var/spack/repos/builtin/packages/py-pyro4/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* fixing a few issues in the PR
* adding py-selectors34 required by py-pyro4
* improved the superlu-dist package
* improved the superlu-dist package
* moree changes to gptune and py-selectors34 based on the PR
* Update var/spack/repos/builtin/packages/py-selectors34/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
I think this test should be removed, but when it stays, it should at
least follow the symlink, cause it fails for me if I let spack build
patchelf and have a symlink in a view.
- Prevent `-j` flags to `make`, which has been known to cause problems
with Racket builds.
- Add variants for various common build flags, including support
for both versions of the Racket VM environment.
In addition:
- Prefer the minimal release to improve install times. Bells and
whistles carry their own runtime dependencies and should be
installed via `raco`. An enterprising user may even create a
`RacoPackage` class to make spack aware of `raco` installed packages.
- Match the official version numbering scheme.
Modifications:
- [x] Removed `centos:6` unit test, adjusted vermin checks
- [x] Removed backport of `collections.OrderedDict`
- [x] Removed backport of `functools.total_ordering`
- [x] Removed Python 2.6 specific skip markers in unit tests
- [x] Fixed a few minor Python 2.6 related TODOs in code
Updating the vendored dependencies will be done in separate PRs
* py-cython: add 3.0.0a9
* Update var/spack/repos/builtin/packages/py-cython/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: py-singledispatch 3.7.0
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Make CUDA and ROCm architecture conditional
fixes#14337
The variant to specify which architecture to use
for CUDA and ROCm are now conditional on +cuda and
+rocm respectively.
* cp2k: make all CUDA related variants conditional on +cuda
* New version: py-pyzmq 22.3.0
* Update var/spack/repos/builtin/packages/py-pyzmq/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add connection specification to mirror creation
This allows each mirror to contain information about the credentials
used to access it.
Update command and tests based on comments
Switch to only "long form" flags for the s3 connection information.
Use the "any" function instead of checking for an empty list when looking
for s3 connection information.
Split test to use the access token separately from the access id and key.
Use long flag form in test.
Add endpoint_url to available S3 options.
Extend the special parameters for an S3 mirror to accept the
endpoint_url parameter.
Add a test.
* Add connection information per URL not per mirror
Expand the mirror-based connection information to be per-URL.
This will allow a user to specify different S3 connection information
for both the fetch and the push URLs.
Add a parameter for "profile", another way of storing the id/secret pair.
* Switch from "access_profile" to "profile"
* py-cycler: add 0.11.0
* Update var/spack/repos/builtin/packages/py-cycler/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: py-qtconsole 5.2.0
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: openldap 2.6.0;
fix recipe for groff (requires pkg-config to find uchardet);
fix recipe for openldap (requires groff to build documentation)
* Restrict openldap versions of py-python-ldap and percona-server
* Update var/spack/repos/builtin/packages/groff/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This causes `otfconfig` to also list zlib's library directory.
Otherwise, `-lz` cannot be found. Also remove `--with-zlibsymbols`,
which doesn't seem to be supported anymore.
Include several several patches to vtk 8.1 for building on a
system with no system install X11 libraries or include files.
Specify specific versions of dependent packages that are known to work with 3.2.1.
Tested on spock.olcf.ornl.gov. The GUI came up and rendered images
and an image was successfully saved using off screen rendering from
data from curv2d.silo.
Remove the "get_executable" function from the
spack.bootstrap module. Now "flake8", "isort",
"mypy" and "black" will use the same
bootstrapping method as GnuPG.
Currently Spack vendors `pytest` at a version which is three major
versions behind the latest (3.2.5 vs. 6.2.4). We do that since v3.2.5
is the latest version supporting Python 2.6. Remaining so much
behind the currently supported versions though might introduce
some incompatibilities and is surely a technical debt.
This PR modifies Spack to:
- Use the vendored `pytest@3.2.5` only as a fallback solution,
if the Python interpreter used for Spack doesn't provide a newer one
- Be able to parse `pytest --collect-only` in all the different output
formats from v3.2.5 to v6.2.4 and use it consistently for `spack unit-test --list-*`
- Updating the unit tests in Github Actions to use a more recent `pytest` version
The previous workaround of using CMAKE_INSTALL_RPATH=ON was used to
avoid CMake trying to write an RPATH into the linker script libcxx.so,
which is nonsensical. See commit f86ed1e.
However, CMAKE_INSTALL_RPATH=ON seems to disable the build RPATH, which
breaks LLVM during the build when it has to locate its build-time shared
libraries (e.g. libLLVM.so). That required yet another workaround, where
some shared libraries were installed "by hand", so that they were picked
up from the install libdir. See commit 8a81229.
This was a dirty workaround, and also makes it impossible to use ninja,
since we explicitly invoked make.
This commit removes the two old workaround, and sets
LIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON, so that libc++abi.a is linked into
libc++.so, which makes it enough to link with -lc++ or invoke clang++
with -stdlib=libc++, so that our install succeeds and linking LLVM's c++
standard lib is still easy.
Some packages use a 64_ or _64 symbol suffix for the ilp64 (= 64-bit
integers) intefrace for BLAS. In particular if we want to support shim
libraries like libopenblastrampoline supporting both the 32 and 64 bit
integer version of blas, it must be possible to distinguish between the
two.
mesa inherits MesonPackage (since October 2020) which depends on Py@3.
The conflicts('mesa') enables a regular build of `qt@5.7:5.15+webkit`
without having to specify the exact version by causing the concretizer
to select mesa18 which does not depend on python@3.
Co-authored-by: Bernhard Kaindl <bernhard.kaindl@ait.ac.at>
This type of error is skipped:
make[1]: *** [Makefile:222: /tmp/user/spack-stage/.../spack-src/usr/lib/julia/libopenblas64_.so.so] Error 1
but it's useful to have it, especially when a package sets a variable
incorrectly in makefiles
* New version: py-pylint 2.8.2; new package py-platformdirs
* Update var/spack/repos/builtin/packages/py-platformdirs/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pylint/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Intel mpi comes with an installation of libfabric (which it needs as a
dependency). It can use other implementations of libfabric at runtime
though, so if you install a package that depends on `mpi` and
`libfabric`, you can specify `intel-mpi+external-libfabric` and ensure
that the Spack-built instance is used (both by `intel-mpi` and the
root).
Apply analogous change to intel-oneapi-mpi.
* New version: py-pyrsistent 0.18.0
* Update package.py
* Update var/spack/repos/builtin/packages/py-pyrsistent/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: py-pytest 6.2.5
* Update var/spack/repos/builtin/packages/py-pytest/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Use setup_run_environment to search for libraries and set env variables for module generation.
Libraries are installed with CMAKE_INSTALL_LIBDIR, which can be lib or lib64 depending on the machine, which makes it impossible to hardcode through modules.yaml.
If the perl that perl-forks is built against is non-threaded the build
system will drop into interactive mode to ask about simulating ithreads.
This causes the build to hang. Set FORKS_SIMULATE_USEITHREADS to avoid
going into interactive mode.
If building Qt on a system with a recent glibc but an older kernel, it
is possible that Qt configures features based on glibc that are not
supported in the kernel. This PR tests the kernel version and ensures
certain features are disabled if the kernel does not support them.
* New version: py-prettytable 2.4.0; update homepage
* Update var/spack/repos/builtin/packages/py-prettytable/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: py-prometheus-client 0.12.0; new dependency (py-twisted) version 21.7.0 + it's dependencies
* Apply suggestions from code review (1/?)
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Changes from review (2/?)
* Changes from review (3/?)
* Changes from review (4/?)
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
- disable graphblas by default (very slow to compile)
- fix patch upperbound for cuda 11
- remove find_system_libs; not sure why it was added in the first place,
but it makes spack rather unusable as it introduces an rpath to /lib/...
* New version: py-oauthlib 3.1.1
* Update var/spack/repos/builtin/packages/py-oauthlib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Generalize env var PYTHON to avoid version conflicts
* Use available python executable
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* upcxx: Update the UPC++ package to 2021.9.0
* Add the new release, and a missing older one.
* Remove the spack package cruft for supporting the obsolete build system that
was present in older versions that are no longer supported.
* General cleanups.
Support for library versions older than 2020.3.0 is officially retired,
for two reasons:
1. Releases prior to 2020.3.0 had a required dependency on Python 2,
which is [officially EOL](https://www.python.org/doc/sunset-python-2/)
as of Jan 1 2020, and is no longer considered secure.
2. (Most importantly) The UPC++ development team is unable/unwilling to
support releases more than two years old. UPC++ provides robust
backwards-compatibility to earlier releases of UPC++ v1.0, with very
rare well-documented/well-motivated exceptions. Users are strongly
encouraged to update to a current version of UPC++.
NOTE: Most of the lines changed in this commit are simply re-indentation,
and thus might be best reviewed in a diff that ignores whitespace.
* upcxx: Detect Cray XC more explicitly
This change is necessary to prevent false matches occuring on new Cray Shasta
systems, which do not use the aries network but were incorrectly being treated
as a Cray XC + aries platform.
UPC++ has not yet deployed official native support for Cray Shasta, but this
change is sufficient to allow building the portable backends there.
* py-setupmeta: add new package
* Update var/spack/repos/builtin/packages/py-setupmeta/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
When running `spack install --log-format junit|cdash ...`, install
errors were ignored. This made spack continue building dependents of
failed install, ignoring `--fail-fast`, and exit 0 at the end.
* Add hdf5-vol-async package.
Add HDF5 1.13.0-rc6 version for building vol-async.
* Style test required another blank line.
* Change hdf5 dependency to develop-1.13+mpi+threadsafe.
* Update args for hdf5-vol-async.
Open MPI currently fails to build with scheduler=slurm if +pmix is
not given with a fatal error due to ``config_args +=
self.with_or_without('pmix', ...)`` resulting in --without-pmix.
However, Open MPI's configure points out "Note that Open MPI does
not support --without-pmix."
The PR only adds "--with-pmix=PATH" if +pmix is part of the spec.
Otherwise, nothing is added and Open MPI can fall back to its
internal PMIX sources.
(The other alternative would be to depend on +pmix in for
scheduler=slurm as is done for +pmi.)
- Added new checksums for 4.3.
- Now using llvm-amdgpu ~openmp in order to use the rocm-device-libs
build as external project in llvm-amdgpu package. We still need
to pull device-libs in using resource for the build as some headers
are not installed.
- Updated symlink creation to now remove an existing link if present
to avoid issues on partial reinstalls when debugging.
- Adjusted the flang_warning to be a part of Cmake options instead of
a filter_file for better compatability.
- The dependency on hsa-rocr-dev created some problems as type was changed
to the default build/link. This issue was because ROCr uses libelf and
the build of openmp expects elfutils. When link is specified libelf
was being found in the path first, causing errors. This was
introduced with the llvm-amdgpu external project build of device-libs.
- On a bare bone installation of sles15 it was noted that libquadmath0 was
needed as a dependency. On 18.04 gcc-multilib was also needed.
* Workaround for libelf headers being used instead of elfutils.
Due to Kitware API changes, default ANTs builds were failing, presumably for all versions (https://github.com/ANTsX/ANTs/issues/1236).
This commit defaults BUILD_TESTING to OFF, preventing calls against
these APIs and fixing all versions.
Note that the ANTs test suite was not clean anyway (e.g. ANTs/#842).
* Python tests: allow importing weirdly-named modules
e.g. with dashes in name
* SIP tests: allow importing weirdly-named modules
* Skip modules with invalid names
* Changes from review
* Update from review
* Update from review
* Cleanup
* New version: py-lz4 3.1.3; use external lz4 instead of bundled one
* Update var/spack/repos/builtin/packages/py-lz4/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Changes from review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add spack package py-ytopt-team-ytopt and required dependencies.
* Removed old ytop package.
* Added author as maintainer.
* Fix style.
* Update var/spack/repos/builtin/packages/py-config-space/package.py
Update python dependency to 3.7
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-config-space/package.py
Remove run dependency from py-cython.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-config-space/package.py
Added run dependency type for py-pyparsing.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Updated description of py-dh-scikit-optimize.
* Source py-dh-scikit-optimize from PyPI.
* Added latest py-dh-scikit-optimize version.
* Made plots option False by default for py-dh-scikit-optimize.
* Removed 0.9.4 as it needs additional dependencies.
* Added version dependencies.
* Added missing py-joblib dependencies.
* Added run dependency type.
* Added python 2.7+ as supported for py-pyaml.
* Change py-config-space to py-configspace.
* Added dependency on python 3.6+.
* Fix py-configspace package naming.
* Changed py-autotune to py-ytopt-autotune.
* Update var/spack/repos/builtin/packages/py-pyaml/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Added debug variant with py-ray dependency.
* Added missing py-mpi4py missing dependency.
* Removed erroneous variant.
* Added debug variant to py-ray.
* Fix indentation.
* Removed debug variant of py-ray.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Noting that missing numeric_limits was the cause of the compile issues
with gcc-11, I tested adding -include limits fixing @5.9:5.14%gcc@11.
Therefore, we can replace the conflicts('%gcc@11:', when='@5.9:5.14').
Co-authored-by: Bernhard Kaindl <bernhard.kaindl@ait.ac.at>
* Prevent additional properties to be in the answer set when reusing specs
fixes#27237
The mechanism to reuse concrete specs relies on imposing
the set of constraints stemming from the concrete spec
being reused.
We also need to prevent that other constraints get added
to this set.
See #25249 and https://github.com/spack/spack/pull/27159#issuecomment-958163679.
This adds `spack load --list` as an alias for `spack find --loaded`. The new command is
not as powerful as `spack find --loaded`, as you can't combine it with all the queries or
formats that `spack find` provides. However, it is more intuitively located in the command
structure in that it appears in the output of `spack load --help`.
The idea here is that people can use `spack load --list` for simple stuff but fall back to
`spack find --loaded` if they need more.
- add help to `spack load --list` that references `spack find`
- factor some parts of `spack find` out to be called from `spack load`
- add shell tests
- update docs
Co-authored-by: Peter Josef Scheibel <scheibel1@llnl.gov>
Co-authored-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>
Reformulate variant rules so that we minimize both
1. The number of non-default values being used
2. The number of default values not-being used
This is crucial for MV variants where we may have
more than one default value
In our tests, we use concrete specs generated from mock packages,
which *only* occur as inputs to the solver. This fixes two problems:
1. We weren't previously adding facts to encode the necessary
`depends_on()` relationships, and specs were unsatisfiable on
reachability.
2. Our hash lookup for reconstructing the DAG does not
consider that a hash may have come from the inputs.
Concrete specs that are already installed or that come from a buildcache
may have compilers and variant settings that we do not recognize, but that
shouldn't prevent reuse (at least not until we have a more detailed compiler
model).
- [x] make sure compiler and variant consistency rules only apply to
built specs
- [x] don't validate concrete specs on input, either -- they're concrete
and we shouldn't apply today's rules to yesterday's build
In switching to hash facts for concrete specs, we lost the transitive facts
from dependencies. This was fine for solves, because they were implied by
the imposed constraints from every hash. However, for `spack diff`, we want
to see what the hashes mean, so we need another mode for `spec_clauses()` to
show that.
This adds a `expand_hashes` argument to `spec_clauses()` that allows us to
output *both* the hashes and their implications on dependencies. We use
this mode in `spack diff`.
- [x] Get rid of forgotten maximize directive.
- [x] Simplify variant handling
- [x] Fix bug in treatment of defaults on externals (don't count
non-default variants on externals against them)
Variants in concrete specs are "always" correct -- or at least we assume
them to be b/c they were concretized before. Their variants need not match
the current version of the package.
Multi-valued variants previously maximized default values to handle
cases where the default contained two values, e.g.:
variant("foo", default="bar,baz")
This is because previously we were minimizing non-default values, and
`foo=bar`, `foo=baz`, and `foo=bar,baz` all had the same score, as
none of them had any "non-default" values.
This commit changes the approach and considers a non-default value
to be either a value set to something not default *or* the absence
of a default value from the set value. This allows multi- and
single-valued variants to be handled the same way, with the same
minimization criterion. It alse means that the "best" value for every
optimization criterion is now zero, which allows us to make useful
assumptions about the optimization criteria.
Minimizing builds is tricky. We want a minimizing criterion because
we want to reuse the avaialble installs, but we also want things that
have to be built to stick to *default preferences* from the package
and from the user. We therefore treat built specs differently and
apply a different set of optimization criteria to them. Spack's *first*
priority is to reuse what it can, but if it builds something, the built
specs will respect defaults and preferences.
This is implemented by bumping the priority of optimization criteria
for built specs -- so that they take precedence over the otherwise
topmost-priority criterion to reuse what is installed.
The scheme relies on all of our optimization criteria being minimizations.
That is, we need the case where all specs are reused to be better than
any built spec could be. Basically, if nothing is built, all the build
criteria are zero (the best possible) and the number of built packages
dominates. If something *has* to be built, it must be strictly worse
than full reuse, because:
1. it increases the number of built specs
2. it must have either zero or some positive number for all criteria
Our optimziation criteria effectively sum into two buckets at once to
accomplish this. We use a `build_priority()` number to shift the
priority of optimization criteria for built specs higher.
The constraints in the `spack diff` test were very specific and assumed
a lot about the structure of what was being diffed. Relax them a bit to
make them more resilient to changes.
Make the first minimization conditional on whether `--reuse` is enabled in the solve.
If `--reuse` is not enabled, there will be nothing in the set to minimize and the
objective function (for this criterion) will be 0 for every answer set.
Many of the integrity constraints in the concretizer are there to restrict how solves are done, but
they ignore that past solves may have had different initial conditions. For example, for things
we're building, we want the allowed variants to be restricted to those currently in Spack packages,
but if we are reusing a concrete spec, we need to be flexible about names that may have existed in
old packages.
Similarly, restrictions around compatibility of OS's, compiler versions, compiler OS support, etc.
are really only about what is supported by the *current* set of compilers/build tools known to
Spack, not about what we may get from concrete specs.
- [x] restrict certain integrity constraints to only apply to packages that we need to build, and
omit concrete specs from consideration.
The OS logic in the concretizer is still the way it was in the first version.
Defaults are implemented in a fairly inflexible way using straight logic. Most
of the other sections have been reworked to leave these kinds of decisions to
optimization. This commit does that for OS's as well.
As with targets, we optimize for target matches. We also try to optimize for
OS matches between nodes. Additionally, this commit adds the notion of
"OS compatibility" where we allow for builds to depend on binaries for certain
other OS's. e.g, for macos, a bigsur build can depend on an already installed
(concrete) catalina build. One cool thing about this is that we can declare
additional compatible OS's later, e.g. CentOS and RHEL.
If we don't rename Spack will fail with:
```
ImportError: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %gcc target=x86_64" due to the following failures:
'spack-install' raised ValueError: Invalid config scope: 'bootstrap'. Must be one of odict_keys(['_builtin', 'defaults', 'defaults/cray', 'bootstrap/cray', 'disable_modules', 'overrides-0'])
Please run `spack -d spec zlib` for more verbose error messages
```
in case bootstrapping from binaries fails and we are
falling back to bootstrapping from sources.
ensure that none of ^intel-mkl, ^intel-mpi, and ^mkl are used, unless
the compiler is intel.
Fix bad logic in the src/src_xs/m_makespectrum.f90 file in the oxygen version.
Add the -fallow-argument-mismatch for gcc >= 10.
* scr: 3.0rc2 release, variants and deps updates
This adds 3.0rc2 release for end users to aid in testing scr for
upcoming 3.0 release.
Included in this change:
- Require most recent component versions for this release
- Add a variant for PDSH as it is now an optional dependency with
this release
- Add bbapi and datawarp (dw) variants
- bbapi_fallback variant now requires bbapi variant with latest
release
- Add variants to enable/disable examples and tests
- Add shared variant and current conflicts with ~shared
- Update cmake_args to account for added variants where needed
Additional updates:
- Add maintainers
- Use lists and for loops to clean up repetitive code involving all
components
- Use self.define and self.define_from_variant to clean up cmake_args
- Use consistent quoting throughout package
* Un-deprecate v2 and legacy
* Use new conditional variants
* trilinos: fix @13.0.1+tpetra^cuda@11
* Mark CUDA conflict with old versions and always define TPL
* trilinos: patch doesn't build so just mark as conflict
A common question from users has been how to model variants
that are new in new versions of a package, or variants that are
dependent on other variants. Our stock answer so far has been
an unsatisfying combination of "just have it do nothing in the old
version" and "tell Spack it conflicts".
This PR enables conditional variants, on any spec condition. The
syntax is straightforward, and matches that of previous features.
* GnuPG: allow bootstrapping from buildcache and sources
* Add a test to bootstrap GnuPG from binaries
* Disable bootstrapping in tests
* Add e2e test to bootstrap GnuPG from sources on Ubuntu
* Add e2e test to bootstrap GnuPG on macOS
* trilinos: add @13.2.0, and switch default to cxxstd=14
* trilinos: fix python dependency when using +ifpack or +ifpack2
* trilinos: add conflict for ~epetra +ml when @13.2.0:
* trilinos: keep 13.0.1 as the preferred version
* Update var/spack/repos/builtin/packages/trilinos/package.py
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* update
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
This PR adds error message sentinels to the clingo solve, attached to each of the rules that could fail a solve. The unsat core is then restricted to these messages, which makes the minimization problem tractable. Errors that can only be generated by a bug in the logic program or generating code are prefaced with "Internal error" to make clear to users that something has gone wrong on the Spack side of things.
* minimize unsat cores manually
* only errors messages are choices/assumptions for performance
* pre-check for unreachable nodes
* update tests for new error message
* make clingo concretization errors show up in cdash reports fully
* clingo: make import of clingo.ast parsing routines robust to clingo version
Older `clingo` has `parse_string`; newer `clingo` has `parse_files`. Make the
code work wtih both.
* make AST access functions backward-compatible with clingo 5.4.0
Clingo AST API has changed since 5.4.0; make some functions to help us
handle both versions of the AST.
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
This removes `-lpytrilinos` from Makefile.export.Trilinos so that C code
trying to link against a Trilinos built with PyTrilinos does not fail
due to undefined references to python routines (libpytrilinos is only
used when importing PyTrilinos in python, in which case those references
are already defined by Python).
There was already a bit of code to do something similar for C codes
importing Trilinos via a CMake mechanism, this extends that to a basic
Makefile mechanism as well. This patch also updates the comments to
remove a stale link discussing this issue, and replacing with links to
the some Trilinos issue reports related to the matter.
After #26608 I got a report about missing rpaths when building a
downstream package independently using a spack-installed toolchain
(@tmdelellis). This occurred because the spack-installed libraries were
being linked into the downstream app, but the rpaths were not being
manually added. Prior to #26608 autotools-installed libs would retain
their hard-coded path and would thus propagate their link information
into the downstream library on mac.
We could solve this problem *if* the mac linker (ld) respected
`LD_RUN_PATH` like it does on GNU systems, i.e. adding `rpath` entries
to each item in the environment variable. However on mac we would have
to manually add rpaths either using spack's compiler wrapper scripts or
manually (e.g. using `CMAKE_BUILD_RPATH` and pointing to the libraries of
all the autotools-installed spack libraries).
The easier and safer thing to do for now is to simply stop changing the
dylib IDs.
gconf depends on gettext and libintl (dep: intltool)
glibmm, gtkmm, libcanberra and cups need pkgconfig
glibmm needs libsigc++ < 2.9x(which are 3.x pre-releases)
libsigc++@:2.9 depends on m4 for the build
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
- add version 6.0.5
- add patch to allow fsl to use newer gcc versions
- add patch to allow fsl to use newer cuda versions
- remove constraints on gcc and cuda versions
- add filters to prevent using system headers and libraries
- clean up the installed tree
The `--generic` argument allows printing the best generic target for the
current machine. This can be quite handy when wanting to find the
generic architecture to use when building a shared software stack for
multiple machines.
This PR adds a "spack tags" command to output package tags or
(available) packages with those tags. It also ensures each package
is listed in the tag cache ONLY ONCE per tag.
glib has a few tests which have external dependencies or
try to access the X server. We cannot run those.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* ci: Enable more packages in the DVSDK CI pipeline
* doxygen: Add conflicts for gcc bugs
* dray: Add version constraints for api breakage with newer deps
5.14.2 fails with %gcc@11 with Error: 'numeric_limits' is not a class template
5.8.0 has multiple compile failures as well: Extend the conflict to those too.
- Fix also the confgigure of @5.6.3 (tested with %gcc@11)
* New versions: py-gitdb 4.0.7, 4.0.8, 4.0.9
* Update var/spack/repos/builtin/packages/py-gitdb/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New versions of py-google-auth and py-google-auth-oauthlib
* Update var/spack/repos/builtin/packages/py-google-auth/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New verion: py-html5lib 1.1
* Update var/spack/repos/builtin/packages/py-html5lib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-html5lib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update package.py
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-html5lib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-html5lib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* New version: py-importlib-resources 5.3.0
* Update var/spack/repos/builtin/packages/py-importlib-resources/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-slurm-pipeline: Add 4.0.4, Fix base.py: import six for @:3
Before 4.0.0, slurm_pipeline/base.py has: `from six import string_types`
* Added depends_on('py-pytest@6.2.2:', type='build') as requested by Adam
* remove comment requested to be removed
- [x] Allow dding enumerated types and types whose default value is forbidden by the schema
- [x] Add a test for using enumerated types in the tests for `spack config add`
- [x] Make `config add` tests use the `mutable_config` fixture so they do not
affect other tests
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
The build-time testsuite which would be run when building
with tests needs docker. Check that it exists before
attempting to execute the tests.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
If you don't format `spack.yaml` correctly, `spack config edit` still fails and
you have to edit your `spack.yaml` manually.
- [x] Add some code to `_main()` to defer `ConfigFormatError` when loading the
environment, until we know what command is being run.
- [x] Make `spack config edit` use `SPACK_ENV` instead of the config scope
object to find `spack.yaml`, so it can work even if the environment is bad.
Co-authored-by: scheibelp <scheibel1@llnl.gov>
`spack config get <section>` was erroneously returning just the `spack.yaml`
for the environment.
It should return the combined configuration for that section (including
anything from `spack.yaml`), even in an environment.
- [x] reorder conditions in `cmd/config.py` to fix
`spack --debug config edit` was not working properly -- it would not do show a
stack trace for configuration errors.
- [x] Rework `_main()` and add some notes for maintainers on where things need
to go for configuration to work properly.
- [x] Move config setup to *after* command-line parsing is done.
Co-authored-by: scheibelp <scheibel1@llnl.gov>
`main()` has grown, and in some cases code that can generate errors has gotten
outside the top-level try/catch in there. This means that simple errors like
config issues give you large stack traces, which shouldn't happen without
`--debug`.
- [x] Split `main()` into `main()` for the top-level error handling and
`_main()` with all logic.
There were some loose ends left in ##26735 that cause errors when
using `SPACK_DISABLE_LOCAL_CONFIG`.
- [x] Fix hard-coded `~/.spack` references in `install_test.py` and `monitor.py`
Also, if `SPACK_DISABLE_LOCAL_CONFIG` is used, there is the issue that
`$user_config_path`, when used in configuration files, makes no sense,
because there is no user config scope.
Since we already have `$user_cache_path` in configuration files, and since there
really shouldn't be *any* data stored in a configuration scope (which is what
you'd configure in `config.yaml`/`bootstrap.yaml`/etc., this just removes
`$user_config_path`.
There will *always* be a `$user_cache_path`, as Spack needs to write files, but
we shouldn't rely on the existence of a particular configuration scope in the
Spack code, as scopes are configurable, both in number and location.
- [x] Remove `$user_config_path` substitution.
- [x] Fix reference to `$user_config_path` in `etc/spack/deaults/bootstrap.yaml`
to refer to `$user_cache_path`, which is where it was intended to be.
* Deactivate previous env before activating new one
Currently on develop you can run `spack env activate` multiple times to switch
between environments, but they leave traces, even though Spack only supports
one active environment at a time.
Currently:
```console
$ spack env create a
$ spack env create b
$ spack env activate -p a
[a] $ spack env activate -p b
[b] [a] $ spack env activate -p b
[a] [b] [a] $ spack env activate -p a
[a] [b] [c] $ echo $MANPATH | tr ":" "\n"
/path/to/environments/a/.spack-env/view/share/man
/path/to/environments/a/.spack-env/view/man
/path/to/environments/b/.spack-env/view/share/man
/path/to/environments/b/.spack-env/view/man
```
This PR fixes that:
```console
$ spack env activate -p a
[a] $ spack env activate -p b
[b] $ spack env activate -p a
[a] $ echo $MANPATH | tr ":" "\n"
/path/to/environments/a/.spack-env/view/share/man
/path/to/environments/a/.spack-env/view/man
```
Currently spack is a bit of a bad actor as a zsh plugin, and it was my
fault. The autoload and compinit should really be handled by the user,
as was made abundantly clear when I found spack was doing completion
initialization for *all* of my plugins due to a deferred setup that was
getting messed up by it.
Making this conditional took spack load time from 1.5 seconds (with
module loading disabled) to 0.029 seconds. I can actually afford to load
spack by default with this change in.
Hopefully someday we'll do proper zsh completion support, but for now
this helps a lot.
* use zsh hist expansion in place of dirname
* only run (bash)compinit if compdef/complete missing
* add zsh compiled files to .gitignore
* move changes to .in file, because spack
* Drastically improve YamlFilesystemView file removal via batching
The `remove_file` routine has to check if the file is owned by multiple packages, so it doesn't
remove necessary files. This is done by the `get_all_specs` routine, which walks the entire
package tree. With large numbers of packages on shared file systems, this can take seconds
per file tree traversal, which adds up extremely quickly. For example, a single deactivate
of a largish python package in our software stack on GPFS took approximately 40 minutes.
This patch simply replaces `remove_file` with a batch `remove_files` routine. This routine
removes a list of files rather than a single file, requiring only one traversal per batch. In
practice this means a package can be removed in seconds time, rather than potentially hours,
essentially a ~100x speedup (ignoring initial deactivation logic, which takes about 3 minutes
in our test setup).
* Fix sbang hook for non-writable files
PR #26793 seems to have broken the sbang hook for files with missing
write permissions. Installing perl now breaks with the following error:
```
==> [2021-10-28-12:09:26.832759] Error: PermissionError: [Errno 13] Permission denied: '$SPACK/opt/spack/linux-fedora34-zen2/gcc-11.2.1/perl-5.34.0-afuweplnhphcojcowsc2mb5ngncmczk4/bin/cpanm'
```
Temporarily add write permissions to the original file so it can be
overwritten with the patched one.
And test that file permissions are preserved in sbang even for non-writable files
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
When relocating a binary distribution, Spack only checks files to see
if they are a link that needs to be relocated. Directories can be
such links as well, however, and need to undergo the same checks
and potential relocation.
`spack list` tests are not using mock packages for some reason, and many
are marked as potentially slow. This isn't really necessary; we don't need
6,000 packages to test the command.
- [x] update tests to use `mock_packages` fixture
- [x] remove `maybeslow` annotations
Currently Spack reads full files containing shebangs to memory as
strings, meaning Spack would have to guess their encoding. Currently
Spack has a fixed guess of UTF-8.
This is unnecessary, since e.g. the Linux kernel does not assume an
encoding on paths at all, it's just bytes and some delimiters on the
byte level.
This commit does the following:
1. Shebangs are treated as bytes, so that e.g. latin1 encoded files do
not throw UnicodeEncoding errors, and adds a test for this.
2. No more bytes than necessary are read to memory, we only have to read
until the first newline, and from there on we an copy the file byte by
bytes instead of decoding and re-encoding text.
3. We cap the number of bytes read to 4096, if no newline is found
before that, we don't attempt to patch it.
4. Add support for luajit too.
This should make Spack both more efficient and usable for non-UTF8
files.
Spack's `system` and `user` scopes provide ways for administrators and
users to set global defaults for all Spack instances, but for use cases
where one wants a clean Spack installation, these scopes can be undesirable.
For example, users may want to opt out of global system configuration, or
they may want to ignore their own home directory settings when running in
a continuous integration environment.
Spack also, by default, keeps various caches and user data in `~/.spack`,
but users may want to override these locations.
Spack provides three environment variables that allow you to override or
opt out of configuration locations:
* `SPACK_USER_CONFIG_PATH`: Override the path to use for the
`user` (`~/.spack`) scope.
* `SPACK_SYSTEM_CONFIG_PATH`: Override the path to use for the
`system` (`/etc/spack`) scope.
* `SPACK_DISABLE_LOCAL_CONFIG`: set this environment variable to completely
disable *both* the system and user configuration directories. Spack will
only consider its own defaults and `site` configuration locations.
And one that allows you to move the default cache location:
* `SPACK_USER_CACHE_PATH`: Override the default path to use for user data
(misc_cache, tests, reports, etc.)
With these settings, if you want to isolate Spack in a CI environment, you can do this:
export SPACK_DISABLE_LOCAL_CONFIG=true
export SPACK_USER_CACHE_PATH=/tmp/spack
This is a stop-gap approach until we have figured out how to deal with
the system and user config scopes more generally, as there are plans to
potentially / eventually get rid of them.
**User config**
Spack is a bit of a pain when you have:
- a shared $HOME folder across different systems.
- multiple Spack versions on the same system.
**System config**
- On shared systems with a versioned programming environment / toolkit,
system administrators want to provide config for each version (e.g.
21.09, 21.10) of the programming environment, and the user Spack
instance should be able to pick this up without a steep learning
curve.
- On shared systems the user should be able to opt out of the
hard-coded config scope in /etc/spack, since it may be incompatible
with their particular instance. Currently Spack can only opt out of all
config scopes through overrides with `"config:":`, `"packages:":`, but that
also drops the defaults config, which would have to be repeated, which
is undesirable, especially the lengthy packages.yaml.
An example use case is: having config in this folder:
```
/path/to/programming/environment/{version}/{compilers,packages}.yaml
```
and have `module load spack-system-config` set the variable
```
SPACK_SYSTEM_CONFIG_PATH=/path/to/programming/environment/{version}
```
where the user no longer has to worry about what `{version}` they are
on.
**Continuous integration**
Finally, there is the use case of continuous integration, which may
clone an arbitrary Spack version, which optimally should not pick up
system or user config from the previous run (like may happen in
classical bare metal non-containerized filesystem side effect ridden
jenkins pipelines). In fact this is very similar to how spack itself
tries to avoid picking up system dependencies during builds...
**But environments solve this?**
- You could do `include`s in environment files to get similar behavior
to the spack_system_config_path example, but environments require you
to:
1) require paths to individual config files, not directories.
2) fail if the listed config file does not exist
- They allow you to override config scopes, but this is generally too
rigurous, as it requires you to repeat the default config, in
particular packages.yaml, and just defies the point of layered config.
Co-authored-by: Tom Scogland <tscogland@llnl.gov>
Co-authored-by: Tim Fuller <tjfulle@sandia.gov>
Co-authored-by: Steve Leak <sleak@lbl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Any spec satisfying a default will be symlinked to `default`
If multiple specs have modulefiles in the same directory and satisfy
configured module defaults, then whichever was written last will be
default.
Use of `-R` flag to CTest command causes "empty-14" test to run,
by matching "empty", before the empty-14 target is built.
Patch CTest command in buildscript to match name exactly.
* [geant4] depends_on vecgeom@1.1.8:1.1 range
While previous versions were unclear, the [geant4.10.7 release notes](https://geant4-data.web.cern.ch/ReleaseNotes/ReleaseNotes4.10.7.html) indicate that vecgeom@1.1.8 is a minimum required version, not an exact required version ("Set VecGeom-1.1.8 as minimum required version for optional build with VecGeom."). This will allow some more freedom on the concretizer solutions while allowing geant4 to take advantage of bugfixes and improvements in vecgeom.
* [vecgeom] new version 1.1.17
* For py-torch: Also update dependencies: many version constraints
with an upper bound of @1.9 are now open (e.g. `@1.8.0:1.9` is
converted to `@1.8.0:`).
* For py-torchvision: Also add 0.11.0 and update ^pil constraint
to avoid building with 8.3.0
This PR permits to specify the `url` and `ref` of the Spack instance used in a container recipe simply by expanding the YAML schema as outlined in #20442:
```yaml
container:
images:
os: amazonlinux:2
spack:
ref: develop
resolve_sha: true
```
The `resolve_sha` option, if true, verifies the `ref` by cloning the Spack repository in a temporary directory and transforming any tag or branch name to a commit sha. When this new ability is leveraged an additional "bootstrap" stage is added, which builds an image with Spack setup and ready to install software. The Spack repository to be used can be customized with the `url` keyword under `spack`.
Modifications:
- [x] Permit to pin the version of Spack, either by branch or tag or sha
- [x] Added a few new OSes (centos:8, amazonlinux:2, ubuntu:20.04, alpine:3, cuda:11.2.1)
- [x] Permit to print the bootstrap image as a standalone
- [x] Add documentation on the new part of the schema
- [x] Add unit tests for different use cases
* [pkg][new version] Provide eospac@6.5.0 and mark it as default.
* Merge in changes found in #21629
* Mark all alpha/beta versions as deprecated.
- Addresses @sethrj's recommendation
- Also add a note indicated why these versions are marked this way.
1. Currently it prints not just the spec name, but the dependencies +
their variants + their compilers + their architectures + ...
2. It's clear from the context what spec the message applies to, so,
let's not print the spec at all.
These three rules in `concretize.lp` are overly complex:
```prolog
:- not provider(Package, Virtual),
provides_virtual(Package, Virtual),
virtual_node(Virtual).
```
```prolog
:- provides_virtual(Package, V1), provides_virtual(Package, V2), V1 != V2,
provider(Package, V1), not provider(Package, V2),
virtual_node(V1), virtual_node(V2).
```
```prolog
provider(Package, Virtual) :- root(Package), provides_virtual(Package, Virtual).
```
and they can be simplified to just:
```prolog
provider(Package, Virtual) :- node(Package), provides_virtual(Package, Virtual).
```
- [x] simplify virtual rules to just one implication
- [x] rename `provides_virtual` to `virtual_condition_holds`
fixes#26866
This semantics fits with the way Spack currently treats providers of
virtual dependencies. It needs to be revisited when #15569 is reworked
with a new syntax.
* gcc: support runtime ability to not install spack rpaths
Fixes#26582 .
* gcc: Fix malformed specs file and add docs
The updated docs point out that the spack-modified GCC does *not*
follow the usual behavior of LD_RUN_PATH!
* gcc: fix bad rpath on macOS
This bug has been around since the beginning of the GCC package file:
the rpath command it generates for macOS writes a single (invalid)
rpath entry.
* gcc: only write rpaths for directories with shared libraries
The original lib64+lib was just a hack for "in case either has one" but
it's easy to tell whether either has libraries that can be directly
referenced.
* py-vermin: add latest version 1.3.1
* Exclude line from Vermin since version is already being checked for
Vermin 1.3.1 finds that `encoding` kwarg of builtin `open()` requires Python 3+.
* Update py-aiohttp to 3.7.4 and py-chartdet to 4.0
* Changes from review
* Update package.py
* Update var/spack/repos/builtin/packages/py-chardet/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The OS should only interpret shebangs, if a file is executable.
Thus, there should be no need to modify files where no execute bit is set.
This solves issues that are e.g. encountered while packaging software as
COVISE (https://github.com/hlrs-vis/covise), which includes example data
in Tecplot format. The sbang post-install hook is applied to every installed
file that starts with the two characters #!, but this fails on the binary Tecplot
files, as they happen to start with #!TDV. Decoding them with UTF-8 fails
and an exception is thrown during post_install.
Co-authored-by: Martin Aumüller <aumuell@reserv.at>
This commit contains changes to support Google Cloud Storage
buckets as mirrors, meant for hosting Spack build-caches. This
feature is beneficial for folks that are running infrastructure on
Google Cloud Platform. On public cloud systems, resources are
ephemeral and in many cases, installing compilers, MPI flavors,
and user packages from scratch takes up considerable time.
Giving users the ability to host a Spack mirror that can store build
caches in GCS buckets offers a clean solution for reducing
application rebuilds for Google Cloud infrastructure.
Co-authored-by: Joe Schoonover <joe@fluidnumerics.com>
* scr/veloc: component releases
Update the ECP-VeloC component packages in preparation for an
upcoming scr@3.0rc2 release.
All
- Add new release versions
- Add new `shared` variant for all components
- Add zlib link dependency to packages that were missing it
- Add maintainers
- Use self.define and self.define_from_variant to clean up cmake_args
axl
- Add independent vendor async support variants
rankstr
- Update older version sha that fails checksum on install
* Fix scr build error
Lock dependencies for scr@3.0rc1 to the versions released at the same
time.
* py-neurokit2: add 0.1.4.1
* Update var/spack/repos/builtin/packages/py-neurokit2/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update cray architecture detection for milan
Update the cray architecture module table with x86-milan -> zen3
Make cray architecture more robust to back off from frontend
architecture to a recent ancestor if necessary. This should make
future cray updates less paingful for users.
Co-authored-by: Gregory Becker <becker33.llnl.gov>
Co-authored-by: Todd Gamblin <gamblin2@llnl.gov>
1. Don't use 16 digits of precision for the seconds, round to 2 digits after the comma
2. Don't print if we don't concretize (i.e. `spack concretize` without `-f` doesn't have to tell me it did nothing in `0.00` seconds)
* Speed-up environment concretization with a process pool
We can exploit the fact that the environment is concretized
separately and use a pool of processes to concretize it.
* Add module spack.util.parallel
Module includes `pool` and `parallel_map` abstractions,
along with implementation details for both.
* Add a new hash type to pass specs across processes
* Add tty msg with concretization time
We use POSIX `patch` to apply patches to files when building, but
`patch` by default prompts the user when it looks like a patch
has already been applied. This means that:
1. If a patch lands in upstream and we don't disable it
in a package, the build will start failing.
2. `spack develop` builds (which keep the stage around) will
fail the second time you try to use them.
To avoid that, we can run `patch` with `-N` (also called
`--forward`, but the long option is not in POSIX). `-N` causes
`patch` to just ignore patches that have already been applied.
This *almost* makes `patch` idempotent, except that it returns 1
when it detects already applied patches with `-N`, so we have to
look at the output of the command to see if it's safe to ignore
the error.
- [x] Remove non-POSIX `-s` option from `patch` call
- [x] Add `-N` option to `patch`
- [x] Ignore error status when `patch` returns 1 due to `-N`
- [x] Add tests for applying a patch twice and applying a bad patch
- [x] Tweak `spack.util.executable` so that it saves the error that
*would have been* raised with `fail_on_error=True`. This lets
us easily re-raise it.
Co-authored-by: Greg Becker <becker33@llnl.gov>
* py-magic: delete redundant package
This package is actually named py-python-magic (since the project itself
is "python-magic").
* New package: libmagic
* Py-python-magic: add required runtime dependency on libmagic and new version
* Py-filemagic: add required runtime dependency
* py-magic: restore and mark as redundant
This reverts commit 4cab7fb69e.
* file: add implicit dependencies and static variant
Replaces redundant libmagic that I added. Compression headers were previously
being picked up from the system.
* Fix py-python-magic dependency
* Update python version requirements
* relocate: call install_name_tool less
* zstd: fix race condition
Multiple times on my mac, trying to install in parallel led to failures
from multiple tasks trying to simultaneously create `$PREFIX/lib`.
* PackageMeta: simplify callback flush
* Relocate: use spack.platforms instead of platform
* Relocate: code improvements
* fix zstd
* Automatically fix rpaths for packages on macOS
* Only change library IDs when the path is already in the rpath
This restores the hardcoded library path for GCC.
* Delete nonexistent rpaths and add more testing
* Relocate: Allow @executable_path and @loader_path
* downgrade_docutils_version
* invalid version
* Update requirements.txt
* Improve spelling and shorten the reference link
* Update spack.yaml
* update version requirement
* update version to maximum of 0.16
Co-authored-by: bernhardkaindl <43588962+bernhardkaindl@users.noreply.github.com>
* py-jupytext: add new package
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* update jupytext dependencies
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-jupytext: remove py-jupyerlab dependency
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-gevent: add version 1.5
* py-gevent: update dependencies for v1.5.0
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-gevent/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* phist: Prefer 1.9.5 (1.9.6 uses mpi_f08, but not available in CI)
* phist: remove dupe of 1.9.5, missing preferred=True
Also, for 1.9.6, patch the (most, one does not work) tests to use
Currently Spack keeps track of the origin in the code of any
modification to the environment variables. This is very slow
and enabled unconditionally even in code paths where the
origin of the modification is never queried.
The only place where we inspect the origins of environment
modifications is before we start a build, If there's an override
of the type `e.set(...)` after incremental changes like
`e.append_path(..)`, which is a "suspicious" change.
This is very rare though.
If an override like this ever happens, it might mean a package is
broken. If that leads to build errors, we can just ask the user to run
`spack -d install ...` and check the warnings issued by Spack to find
the origins of the problem.
UPP and ncep-post are the same package, so this PR
removes the duplication.
ncep-post was originally named after the upstream repo
that now changed its name to UPP.
It can be frustrating to successfully run `spack test run --alias <name>` only to find you cannot get the results because you already use `<name>` in some previous stand-alone test execution. This PR prevents that from happening.
Primary fix:
Due to a typo in a version range, overlapping PR merges resulted
in a build failure of the latest version:
Don't attempt to remove a non-existing file for version 1.9.6.
Secondary fixes:
update_tpetra_gotypes.patch was mentioned twice, and the version
range has to exclude @1.4.2, to which it cannot be applied.
Add depend_on() py-pytest, py-numpy and pkgconfig with type='test'
@:1.9.0 fail with 'Rank mismatch with gfortran@10:, add a conflicts().
raise InstallError('~mpi not possible with kernel_lib=builtin!')
when applicable.
Fixes for spack install --test=root phist:
mpiexec -n12 puts a lot of stress on a pod and gets stuck in a loop
very often: Reduce the mpiexec procs and the number of threads.
Remove @run_after('build') @on_package_attributes(run_tests=True):
from 'def check()': fixes it from getting called twice
The build script of 'make test_install' for the installcheck expects
the examples to be copied to self.stage.path: Provide them.
* updating the recipe for betterment
* addressing the suggesions received from reviewers
* adding package helper macros
Co-authored-by: mohan002 <mohbabul@amd.com>
Using the Spec.constrain method doesn't work since it might
trigger a repository lookup which could break our directives
and triggers a circular import error.
To fix that we introduce a function to merge abstract anonymous
specs, based only on package names, which does not perform any
lookup in the repository.
Add missing pkgconfig to openslide and its dep perl-alien-libxml2.
Fix shared-mime-info to be a runtime dependency of gdk-pixbuf,
Otherwise, configure cannot detect use gdk-pixbuf without error.
* SEACAS: add a Faodel variant
* Use safer CMake and variant packages instead of directly adding parameters
Add a "+faodel ~mpi" dependency to balance "+faodel +mpi"
The buildcache is now extracted in a temporary folder within the current store,
moved to its final place and relocated.
"spack clean -s" has been extended to also clean the temporary extraction directory.
Add hardlinks with absolute paths for libraries in the corge, garply and quux packages
to detect incorrect handling of hardlinks in tests.
Problem: Flux expects the `FLUX_PMI_LIBRARY_PATH` to point directly at
the `libpmi.so` installed by Flux. When the env var is unset,
prepending to it results in this behavior. In the rare case that the
env var is already set, then the spack `libpmi.so` gets prepended with a
`:`, which Flux then attempts to interpret as a single path.
Solution: don't prepend to the path, instead set the path to point to
the `libpmi.so` (which will be undone when Flux is unloaded).
* flux-core: remove deprecated environment variables
The earliest checksummed version in this package is 0.15.0. As of
0.12.0, wreck (and its associated paths) no longer exist in Flux. As of
0.13.0, the `FLUX_RCX_PATH` variables are no longer used. So clean up
these env vars from the `setup_run_environment`.
gromacs@2018:2020.6 is fixed to build with gcc@11.2.0
by adding #include <limits> to a few header files.
Thanks to Maciej Wójcik <w8jcik@gmail.com> for testing versions.
The `find` command was missing for the examples forcing colorized output. Without this (or another suitable) command, spack produces output that is not using any color. Thus, without the `find` command one does not see any difference between forced colorized and non-colorized output.
There was a bug in 2.36.* of missing Makefile dependencies. The
previous workaround was to require 2.36 to be built serially. This is
now fixed upstream in 2.37 and this PR adds the patch to restore
parallel make to 2.36.
* py-niworkflows: add new package
* Update var/spack/repos/builtin/packages/py-niworkflows/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* remove unnecessary comment
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-nistats: add new package
* Update var/spack/repos/builtin/packages/py-nistats/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* remove `conflicts`
* remove test dependencies
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
when deployed on kubernetes, the server sends back permanent redirect responses.
This is elegantly handled by the requests library, but not urllib that we have
to use here, so I have to manually handle it by parsing the exception to
get the Location header, and then retrying the request there.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
* packages/phist, re #26002: force phist to use MPI compiler wrappers (copied from trilinos package)
* packages/phist re #26002, use cmake-provded FindMPI module only
* packages/phist source code formatting
* packages/phist: set MPI_HOME rather than MPI_BASE_DIR, thanks @sethri.
* phist: delete own FindMPI.cmake for older versions (rather than patching it away)
* packages/phist: remove blank line
* phist: adjust sorting of imports
* phist: change order of imports
The ASP-based solver maximizes the number of values in multi-valued
variants (if other higher order constraints are met), to avoid cases
where only a subset of the values that have been specified on the
command line or imposed by another constraint are selected.
Here we swap the priority of this optimization target with the
selection of the default providers, to avoid unexpected results
like the one in #26598
Seems like https://bugs.python.org/issue29699 is relevant. Better to
just ignore errors when removing them tmpdir. The OS will remove it
anyways.
Errors are happening randomly from tests that are using this fixture.
TL;DR: there are matching groups trying to match 1 or more occurrences of
something. We don't use the matching group. Therefore it's sufficient to test
for 1 occurrence. This reduce quadratic complexity to linear time.
---
When parsing logs of an mpich build, I'm getting a 4 minute (!!) wait
with 16 threads for regexes to run:
```
In [1]: %time p.parse("mpich.log")
Wall time: 4min 14s
```
That's really unacceptably slow...
After some digging, it seems a few regexes tend to have `O(n^2)` scaling
where `n` is the string / log line length. I don't think they *necessarily*
should scale like that, but it seems that way. The common pattern is this
```
([^:]+): error
```
which matches `: error` literally, and then one or more non-colons before that. So
for a log line like this:
```
abcdefghijklmnopqrstuvwxyz: error etc etc
```
Any of these are potential group matches when using `search` in Python:
```
abcdefghijklmnopqrstuvwxyz
bcdefghijklmnopqrstuvwxyz
cdefghijklmnopqrstuvwxyz
⋮
yz
z
```
but clearly the capture group should return the longest match.
My hypothesis is that Python has a very bad implementation of `search`
that somehow considers all of these, even though it can be implemented
in linear time by scanning for `: error` first, and then greedily expanding
the longest possible `[^:]+` match to the left. If Python indeed considers
all possible matches, then with `n` matches of length `1 .. n` you
see the `O(n^2)` slowness (i verified this by replacing + with {1,k}
and doubling `k`, it doubles the execution time indeed).
This PR fixes this by removing the `+`, so effectively changing the
O(n^2) into a O(n) worst case.
The reason we are fine with dropping `+` is that we don't use the
capture group anywhere, so, we just ensure `:: error` is not a match
but `x: error` is.
After going from O(n^2) to O(n), the 15MB mpich build log is parsed
in `1.288s`, so about 200x faster.
Just to be sure I've also updated `^CMake Error.*:` to `^CMake Error`,
so that it does not match with all the possible `:`'s in the line.
Another option is to use `.*?` there to make it quit scanning as soon as
possible, but what line that starts with `CMake Error` that does not have
a colon is really a false positive...
Installing packages with a lot of dependencies does not have an easy way
of judging the current progress (apart from running `spack spec -I pkg`
in another terminal). This change allows Spack to update the terminal's
title with status information, including its current progress as well as
information about the current and total number of packages.
* kahip: update to cmake for v3.11, retain scons for older versions
* kahip: update build system to cmake for v3.11, retain SCons for older versions
* address PR comments and add maintainer
* address PR comments - correct version to 2.10, add deprecated and url, and remove scons version
- Do not store the full list of environment variables in
<prefix>/.spack/spack-build-env.txt because it may contain user secrets.
- Only store environment variable modifications upon installation.
- Variables like PATH may still contain user and system paths to make
spack-build-env.txt sourceable. Variables containing paths are
modified through prepending/appending, and if we don't apply these
to the current environment variable, we end up with statements like
`export PATH=/path/to/spack/bin` with system paths missing, meaning
no system binaries are in the path, which is a bad user experience.
- Do write the full environment to spack-build-env.txt in the staging dir,
but ensure it is readonly for the current user, to make it a bit safer
on shared systems.
Creates an environment in a temporary directory and activates it, which
is useful for a quick ephemeral environment:
```
$ spack env activate -p --temp
[spack-1a203lyg] $ spack add zlib
==> Adding zlib to environment /tmp/spack-1a203lyg
==> Updating view at /tmp/spack-1a203lyg/.spack-env/view
```
PR #25904 moved the `--with-tcl` option to only older versions. However,
without this option, the build breaks:
```
checking for Tcl configuration... configure: error: Can't find Tcl configuration definitions. Use --with-tcl to specify a directory containing tclConfig.sh
```
The DB should be what is trusted for certain operations.
If it is not present when read we should assume the
corresponding store is empty, rather than trying a
write operation during a read.
* Add a unit test
* Document what needs to be there in tests
* py-twisted,py-storm: dep on zope.interface, bump storm version
py-twisted and py-storm's import tests need zope.interface.
py-storm: Use pypi and add version 0.25. It didn't change reqs.
zope.infterface@4.5.0 imports removed Feature: Use setuptools@:45
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-storm: all deps updated with type=('build', 'run')
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-bcrypt, py-bleach, py-decorator, py-pygdal: fix python dependency
* Update var/spack/repos/builtin/packages/py-bleach/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-matplotlib: fix 3.4.3
* Update var/spack/repos/builtin/packages/py-matplotlib/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-keras-preprocessing: Add missing deps: six@1.9.0: and numpy@1.9.1:
Add deps: pip download --no-binary :all: keras-preprocessing==1.1.2
Collecting numpy>=1.9.1
Installing build dependencies: started
Collecting six>=1.9.0
* Update var/spack/repos/builtin/packages/py-keras-preprocessing/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
When a symlink to a license file exists but is broken, the license symlink post-install hook fails
because os.path.exists() checks the existence of the target not the symlink itself.
os.path.lexists() is the proper function to use.
* visit: add an external find function (determine_version)
* visit: correct too long comment line
* visit: forgot to set executables
* visit: external find uses signgle dash version
* visit: found as external asking visit version
* fish: adding version 3.3.1
* adding maintainer
* Update var/spack/repos/builtin/packages/fish/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The most recent release of numactl (2.0.14) fails to build on riscv64
because of a missing "-latomic". This patch from upstream resolves this
issue. It can be dropped once the next version of numactl is released.
Environments push/pop scopes upon activation. If some lazily
evaluated value depending on the current configuration was
computed and cached before the scopes are pushed / popped
there will be an inconsistency in the current state.
This PR fixes the issue for stores, but it would be better
to move away from global state.
The `spack.architecture` module contains an `Arch` class that is very similar to `spack.spec.ArchSpec` but points to platform, operating system and target objects rather than "names". There's a TODO in the class since 2016:
abb0f6e27c/lib/spack/spack/architecture.py (L70-L75)
and this PR basically addresses that. Since there are just a few places where the `Arch` class was used, here we query the relevant platform objects where they are needed directly from `spack.platforms`. This permits to clean the code from vestigial logic.
Modifications:
- [x] Remove the `spack.architecture` module and replace its use by `spack.platforms`
- [x] Remove unneeded tests
* Use gnuconfig package for config file replacement for RISC-V.
This extends the changes in #26035 to handle RISC-V. Before this change,
many packages fail to configure on riscv64 due to config.guess being too
old to know about RISC-V. This is seen out of the box when clingo fails
to build from source due to pkgconfig failing to configure, throwing
error: "configure: error: cannot guess build type; you must specify one".
* Add riscv64 architecture
* Update vendored archspec from upstream project.
These archspec updates include changes needed to support riscv64.
* Update archspec's __init__.py to reflect the commit hash of archspec being used.
Cherry-picked from #25564 so this is standalone.
With this PR we can activate an environment in Spack itself, without computing changes to environment variables only necessary for "shell aware" env activation.
1. Activating an environment:
```python
spack.environment.activate(Environment(xyz)) -> None
```
this basically just sets `_active_environment` and modifies some config scopes.
2. Activating an environment **and** getting environment variable modifications for the shell:
```python
spack.environment.shell.activate(Environment(xyz)) -> EnvironmentModifications
```
This should make it easier/faster to do unit tests and scripting with spack, without the cli interface.
* Isolate bootstrap configuration from user configuration
* Search for build dependencies automatically if bootstrapping from sources
The bootstrapping logic will search for build dependencies
automatically if bootstrapping anything form sources. Any
external spec, if found, is written in a scope that is specific
to bootstrapping.
* Don't clean the bootstrap store with "spack clean -a"
* Copy bootstrap.yaml and config.yaml in the bootstrap area
Reverting from CMake to Make install caused
`-install_path=/usr/local/lib/libzstd.1.dylib` to be hardcoded into the
zstd. Now we explicitly pass the PREFIX into the build command so that
the correct spack install path is saved.
Fixes#26438 and also the ROOT install issue I had :)
- [x] Our wrapper error messages are sometimes hard to differentiate from other build
output, so prefix all errors from `die()` with '[spack cc] ERROR:'
- [x] The error we raise when running, say, `fc` without a Fortran compiler was not
clear enough. Clarify the message and the comment.
This converts everything in cc to POSIX sh, except for the parts currently
handled with bash arrays. Tests are still passing.
This version tries to be as straightforward as possible. Specifically, most conversions
are kept simple -- convert ifs to ifs, handle indirect expansion the way we do in
`setup-env.sh`, only mess with the logic in `cc`, and don't mess with the python code at
all.
The big refactor is for arrays. We can't rely on bash's nice arrays and be ignorant of
separators anymore. So:
1. To avoid complicated separator logic, there are three types of lists. They are:
* `$lsep`-separated lists, which end with `_list`. `lsep` is customizable, but we
picked `^G` (alarm bell) for `$lsep` because it's ASCII and it's unlikely that it
would actually appear in any arguments. If we need to get fancier (and I will lose
faith in the world if we do) then we could consider XON or XOFF.
* `:`-separated directory lists, which end with `_dirs`, `_DIRS`, `PATH`, or `PATHS`
* Whitespace-separated lists (like flags), which can have any other name.
Whitespace and colon-separated lists come with the territory with PATHs from env
vars and lists of flags. `^G` separated lists are what we use for most internal
variables, b/c it's more likely to work.
2. To avoid subshells, use a bunch of functions that do dirty `eval` stuff instead. This
adds 3 functions to deal with lists:
* `append LISTNAME ELEMENT [SEP]` will put `ELEMENT` at the end of the list called
`LISTNAME`. You can optionally say what separator you expect to use. Note that we
are taking advantage of everything being global and passing lists by name.
* `prepend LISTNAME ELEMENT [SEP]` like append, but puts `ELEMENT` at the start of
`LISTNAME`
* `extend LISTNAME1 LISTNAME2 [PREFIX]` appends everything in LISTNAME2 to
LISTNAME1, and optionally prepends `PREFIX` to every element (this is useful for
things like `-I`, `-isystem `, etc.
* `preextend LISTNAME1 LISTNAME2 [PREFIX]` prepends everything in LISTNAME2 to
LISTNAME1 in order, and optionally prepends `PREFIX` to every element.
The routines determine the separator for each argument by its name, so we don't have to
pass around separators everywhere. Amazingly, as long as you do not expand variables'
values within an `eval` environment, you can do all this and still preserve quoting.
When iterating over lists, the user of this API still has to set and unset `IFS`
properly.
We ended up having to ignore shellcheck SC2034 (unused variable), because using evals
all over the place means that shellcheck doesn't notice that our list variables are
actually used.
So far this is looking pretty good. I took the most complex unit test I could find
(which runs a sample link line) and ran the same command line 200 times in a shell
script. Times are roughly as follows:
For this invocation:
```console
$ bash -c 'time (for i in `seq 1 200`; do ~/test_cc.sh > /dev/null; done)'
```
I get the following performance numbers (the listed shells are what I put in `cc`'s
shebang):
**Original**
* Old version of `cc` with arrays and `bash v3.2.57` (macOS builtin): `4.462s` (`.022s` / call)
* Old version of `cc` with arrays and `bash v5.1.8` (Homebrew): `3.267s` (`.016s` / call)
**Using many subshells (#26408)**
* with `bash v3.2.57`: `25.302s` (`.127s` / call)
* with `bash v5.1.8`: `27.801s` (`.139s` / call)
* with `dash`: `15.302s` (`.077s` / call)
This version didn't seem to work with zsh.
**This PR (no subshells)**
* with `bash v3.2.57`: `4.973s` (`.025s` / call)
* with `bash v5.1.8`: `4.984s` (`.025s` / call)
* with `zsh`: `2.995s` (`.015s` / call)
* with `dash`: `1.890s` (`.0095s` / call)
Dash, with the new posix design, is easily the winner.
So there are several interesting things to note here:
1. Running the posix version in `bash` is slower than using `bash` arrays. That is to be
expected because it's doing a bunch of string processing where it likely did not have
to before, at least in `bash`.
2. `zsh`, at least on macOS, is significantly faster than the ancient `bash` they ship
with the system. Using `zsh` with the new version also makes the posix wrappers
faster than `develop`. So it's worth preferring `zsh` if we have it. I suppose we
should also try this with newer `bash` on Linux.
3. `bash v5.1.8` seems to be significantly faster than the old system `bash v3.2.57` for
arrays. For straight POSIX stuff, it's a little slower. It did not seem to matter
whether `--posix` was used.
4. `dash` is way faster than `bash` or `zsh`, so the real payoff just comes from being
able to use it. I am not sure if that is mostly startup time, but it's significant.
`dash` is ~2.4x faster than the original `bash` with arrays.
So, doing a lot of string stuff is slower than arrays, but converting to posix seems
worth it to be able to exploit `dash`.
- [x] Convert all but array-related portions to sh
- [x] Fix basic shellcheck issues.
- [x] Convert arrays to use a few convenience functions: `append` and `extend`
- [x] Get `cc` tests passing.
- [x] Add `cc` tests where needed passing.
- [x] Benchmarking.
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
* Add version 0.12.1
* Add variant to build with C++11 standard
build with c++11 standard requires boost threads, and needs explicit setting of
CMAKE_CXX_STANDARD
* intel-tbb: install pkgconfig file
* intel-tbb: install pkgconfig file when @:2021.2.0
* intel-tbb: add blank line
* intel-tbb: fix library name to refer
* intel-tbb: fix library name to refer again
* intel-tbb: use self.prefix.lib.pkgconfig
From the gnupg.org website:
> GnuPG 1.4 is the old, single binary version which still support the
> unsafe PGP-2 keys. However, it lacks many modern features and will
> receive only important updates.
I'm starting to appreciate gpg1 more, because it is relocatable (gng2
has hard-coded paths to gpg-agent and other tools) and it does not
require gpg-agent at all.
* graph500: added option -fcommon for gcc@10.2:, otherwise failed to build with "multiple definition of `column'"
* graph500: moved setting cflag to flag_handler
Work around issues in older hdf5 build and overzealous build flags:
```
>> 1420 /var/folders/j4/fznvdyhx4875h6fhkqjn2kdr4jvyqd/T/9te/spack-stage/spack-stage-hdf5-1.10.4-feyl6tz6hpx5kl7m33avpuacwje2ubul/spack-src/src/H5Odeprec.c:141:8: error: implicit decl
aration of function 'H5CX_set_apl' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
```
The older patch does not apply so the build ends up failing:
```
1539 In file included from /private/var/folders/fy/x2xtwh1n7fn0_0q2kk29xkv9vvmbqb/T/s3j/spack-stage/spack-stage-python-3.8.11
-6jyb6sxztfs6fw26xdbc3ktmbtut3ypr/spack-src/Modules/_tkinter.c:48:
>> 1540 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/tk.h:86:11: f
atal error: 'X11/Xlib.h' file not found
1541 # include <X11/Xlib.h>
1542 ^~~~~~~~~~~~
1543 1 error generated.
```
* Explicitly set path to Kokkos for ArborX testing
* Improve formatting
* Update var/spack/repos/builtin/packages/arborx/package.py
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* Remove blank line
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
* -fallow-argument-mismatch flag added when compiling with GCC to avoid a compilation error when using a GCC version > 10.0.
Co-authored-by: Haz99 <jsalamerosanz@gmail.com>
* Filtered every occurrence of "!$OMP SIMD SAFELEN(LVEC2)" when compiling with nvhcp to avoid a compilation error.
Co-authored-by: Haz99 <jsalamerosanz@gmail.com>
* Line with more than 80 characters split into multiple lines.
Co-authored-by: Haz99 <jsalamerosanz@gmail.com>
When using modules for compiler (and/or external package), if a
package's `setup_[dependent_]build_environment` sets `PYTHONHOME`, it
can influence the python subprocess executed to gather module
information.
The error seen was:
```
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```
But the actual hidden error happened in the `python -c 'import
json...'` subprocess, which made it return an empty string as json:
```
ModuleNotFoundError: No module named 'encodings'
```
This fix uses `python -E` to ignore `PYTHONHOME` and
`PYTHONPATH`. Should be safe here because the python subprocess code
only use packages built-in python.
The python subprocess in `environment.py` was also patched to be safe
and consistent.
* Remove redundant preserve environment code in build environment
* Remove fix for a bug in a module
See https://github.com/spack/spack/issues/3153#issuecomment-280460041,
this shouldn't be part of core spack.
* Don't module unload cray-libsci on all platforms
Spack has logic to preserve an installation prefix when it is being
overwritten: if the new install fails, the old files are restored.
This PR adds error handling for when this backup restoration fails
(i.e. the new install fails, and then some unexpected error prevents
restoration from the backup).
* Remove vestigial code to be compatible with Spack v0.9.X
* ArchSpec: reworked __repr__ to be more adherent to common Python idioms
* ArchSpec: simplified __init__.py and copy()
closes #26354 and #26358
Previously we did not pass paths for GDB or GMP and ./configure would
get confused about which one to pull from. Be more specific.
Built with all variants enabled and fixed the fixable versions and variants:
@:8.1 were fixable by limiting python versions for these to @:3.6.
7.10.1 and 7.11(.1) were fixable to build with glibc-2.25 and newer
using two long patches.
gdb 7.8 and 7.9 weren't fixable as there is no backport if the fix
to build these with glibc-2.25 and newer:
http://lists.busybox.net/pipermail/buildroot/2017-March/188055.html
Co-authored-by: Bernhard Kaindl <bernhardkaindl7@gmail.com>
Modifications:
- Modify the workflow to build container images without pushing when the workflow file itself is modified
- Strip the leading ghcr.io/spack/ from env.container env.versioned to prepare pushing to multiple registries
- Fixed CentOS 7 and Amazon Linux builds
- Login and push to Docker Hub as well as Github Action
- Add a badge to README.md with the status of docker images
- Specify CMake minimum version more precisely
- Ensure rocBLAS is available at build time
- Limit workaround for missing rocblas include path
to the only affected version (4.1.0)
- Make hip a build and link dependency
- Remove hip's link dependencies
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
CMake 3.21.3 disables the broken hipcc-as-rocmclang detection again.
From the release notes:
> The AMD ROCm Platform hipcc compiler was identifed by CMake 3.21.0
> through 3.21.2 as a distinct compiler with id ROCMClang. This has been
> removed because it caused regressions. Instead:
> * hipcc may no longer be used as a HIP compiler because it interferes
> with flags CMake needs to pass to Clang. Use Clang directly.
> * hipcc may once again be used as a CXX compiler, and is treated as
> whatever compiler it selects underneath, as CMake 3.20 and below
> did.
* py-snappy: add patch to fix dependencies
* Update var/spack/repos/builtin/packages/py-snappy/req.patch
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-jupyter-packaging: add 0.7.12 and 0.10.6
* Update var/spack/repos/builtin/packages/py-jupyter-packaging/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-jupyter-packaging/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The format of the HPE/Cray supplied module for cray-mvapich2 on HPE apollo systems is
very different from the cray-mpich module supplied on Cray EX and XE
systems.
Recent changes to the cray-mpich package -
https://github.com/spack/spack/pull/23470
broke support for cray-mvapich2 and relies now on the structure of the
cray-mpich module to work properly.
Rather than try to support two very different vendor mpich modules
using the same spack package, just add another one specialized for
the cray-mvapich2 module.
Signed-off-by: Howard Pritchard <hppritcha@gmail.com>
1. Changes the variant of openssl to `certs=mozilla/system/none` so that
users can pick whether they want Spack or system certs, or if they
don't want certs at all.
2. Keeps the default behavior of openssl to use certs=systems.
3. Changes the curl configuration to not guess the ca path during
config, but rather fall back to whatever the tls provider is
configured with. If we don't do this, curl will still pick up system
certs if it finds them.
As a minor fix, it also adds the build dep `pkgconfig` to curl, since
that's being used during the configure phase to get openssl compilation
flags.
* py-mock: fix depends of `@:2.0.0` and bump version
fixes the build of `py-gsutil`, it depends on `'py-mock@:2.0.0'`.
* Update var/spack/repos/builtin/packages/py-mock/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-mock/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Apply the other requested changes
* Add requested change: Add the python@3.6 for newer versions
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* seacas: new release and fixes for metis/parmetis
* Update to add sha256 checksum for latest seacas release
* Updated the documentation strings with new applications
* Fixed the metis/parmetis variants and logic depending on whether mpi
is enabled/disabled. (There is still a zoltan issue I need to fix,
but this will at least allow seacas to be built without
metis/parmetis or with +mpi+parmetis. The ~mpi+metis still needs
work elsewhere.)
* Enable cpup, slice, zellij in +applications
NetCDF-4.8.1 has been released.
As discussed in https://github.com/Unidata/netcdf-c/issues/2110
(netcdf-c-4.8.1.tar.gz not on ftp site... #2110), the canonical
download site for netCDF releases has been changed and the previous
ftp site is no longer available.
This PR updates the `url` to point to the new recommended download
site and updates the sha256 checksums for the new tar files.
* py-anuga: add git main version to support build with python@3.5:
py-anuga's main branch has been converted to Python-3 recently.
* py-triangle: use pypi, py-anuga: Fixed depends and test suite works now
* py-numba: add 0.54.0 and restrict old dependencies
* Update var/spack/repos/builtin/packages/py-numba/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* OpenSSL 3.0.0
* Remove openssl constraint in e4s to test 3.0.0
* Restrict openssl
* Restrict openssl to @:1 in unifyfs
* Revert "Remove openssl constraint in e4s to test 3.0.0"
This reverts commit 0f0355609771764280ab1b6a523c80843a4f85d6.
* Prefer 1.x
The logic to perform detection of already installed
packages has been extracted from cmd/external.py
and put into the spack.detection package.
In this way it can be reused programmatically for
other purposes, like bootstrapping.
The new implementation accounts for cases where the
executables are placed in a subdirectory within <prefix>/bin
The build needs `pkgconfig` and `openssl`, `m4` is already added by `autoconf`.
Also add the current version of `libp11` to the list of versions.
Co-authored-by: Bernhard Kaindl <bernhard.kaindl@ait.ac.at>
* Use gnuconfig package for config file replacement
Currently the autotools build system tries to pick up config.sub and
config.guess files from the system (in /usr/share) on arm and power.
This is introduces an implicit system dependency which we can avoid by
distributing config.guess and config.sub files in a separate package,
such as the new `gnuconfig` package which is very lightweight/text only
(unlike automake where we previously pulled these files from as a
backup). This PR adds `gnuconfig` as an unconditional build dependency
for arm and power archs.
In case the user needs a system version of config.sub and config.guess,
they are free to mark `gnuconfig` as an external package with the prefix
pointing to the directory containing the config files:
```yaml
gnuconfig:
externals:
- spec: gnuconfig@master
prefix: /tmp/tmp.ooBlkyAKdw/lol
buildable: false
```
Apart from that, this PR gives some better instructions for users when
replacing config files goes wrong.
* Mock needs this package too now, because autotools adds a depends_on
* Add documentation
* Make patch_config_files a prop, fix the docs, add integrations tests
* Make macOS happy
* py-datalad: move datalad wtf test over from py-datalad-metalad
* Update var/spack/repos/builtin/packages/py-datalad/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* [vbfnlo] Add doc variant to toggle building of docs
* [openloops] Add scons to dependencies
Make sure that the build_processes does not accidentally pick up a
non-suitable scons version from the underlying system
* [openloops] Set OLPYTHON to make sure the right scons is picked
* [openloops] Fix Flake8 style complaints
Workaround this compile error for gcc by adding -Wno-narrowing for it:
spindle_logd.cc:65:76: error: narrowing conversion of '255' from 'int' to 'char'
spindle_logd.cc:65:76: error: narrowing conversion of '223' from 'int' to 'char'
spindle_logd.cc:65:76: error: narrowing conversion of '191' from 'int' to 'char'
spindle 0.8.1 wants to compile tests with mpi.h, newer versions need mpicc,
thus add: depends_on("mpi"). Spindle supports the --no-mpi to disable MPI.
Fix the perl test case bug Perl/perl5#15544
Variable PATH longer than 1000 characters (as is usual with spack) fails a perl test case
The fix is: Don't test PATH in testcase perlbug.t
Fixes `spack install --test=all` for specs triggering a build and test of perl!
"Long long" is the default type when building trilinos on its own, and
many downstream packages (both in and out of spack) rely on it. E4S
already sets this explicitly to long_long.
When using the ONNX package inside of an environment that specifies a
python3 executable, it will attempt to use a system installed
version. This can lead to a failure where the system python and the
environment python don't agree and the system python ends up with an
invalid environment. Forces ONNX to use the same version of python as
the rest of the spec.
Co-authored-by: Greg Becker <becker33@llnl.gov>
Fix solving/concretizing candle-benchmarks:
py-theano: The requested variant +gpu us now named +cuda
opencv: The requested variants +python and +zlib are now fixed deps
- Match failed autotest tests show the word "FAILED" near the end
- Match "FAIL: ", "FATAL: ", "failed ", "Failed test" of other suites
- autotest " ok"$ means the test passed, independend of text before.
- autoconf messages showing missing tools are fatal later, show them.
dropwatch is a network packet drop checker and it's make check starts
a daemon which does not terminate.
- Skip this test to not block builds.
- Add depends_on('pkgconfig', type='build')
It is needed in case the host does not have pkg-config installed.
- Remove the depends_on('m4', type='build'):
The depends_on('autoconf', type='build') pulls m4 as it needs it.
When using Ubuntu's gcc-8.4.0 on Ubuntu 18.04 to compile rivet-3.1.3,
compilation errors related to UnstableParticles(), "UFS" show up.
Compilation with this compiler is fixed in rivet-3.1.4, adding it.
Adding type='link' to the depends on 'hepmc' and 'hepmc' fixes
the tests to find libHepMC.so.4 in `spack install --tests=all`
Co-authored-by: Valentin Volkl <valentin.volkl@cern.ch>
The cairo test suite is huge, has many backends and the README states
that running and attempting to pass it is not a goal for normal users,
it has so many dependencies into the system, including fonts, that
passing it is not a goal realistically in reach soon:
Skip it, it takes far too long to be practical.
Despite the patch disabling installation of rules, meson's setup
stage looks up the udev package to get `/lib/udev/rules.d`, but as
spack has no `systemd/udev` package, it would fail to build.
Fix such builds by passing `-Dudevrulesdir` and bump version to 3.10.5
* autotoolspackage.rst: No depends_on('m4') with depends_on('autoconf')
- Remove `m4` from the example depends_on() lines for the autoreconf phase.
- Change the branch used as example from develop to master as it is
far more common in the packages of spack's builtin repo.
- Fix the wrong info that libtoolize and aclocal are run explicitly
in the autoreconf phase by default. autoreconf calls these internally
as needed, thus autotools.py also does not call them directly.
- Add that autoreconf() also adds -I<aclocal-prefix>/share/aclocal.
- Add an example how to set autoreconf_extra_args.
- Add an example of a custom autoreconf phase for running autogen.sh.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Based on the original script of R. Mijakovic further improvements of
GPI-2 installation, in particular different official versions,
configuration setups and even testing. Importantly, the non-autotools
way of installation for older versions is also considered, which is
relevant for some packages using GPI-2.
Co-authored-by: Arcesio Castaneda Medina <arcesio.castaneda.medina@itwm.fraunhofer.de>
xload failed with unresolved referenced to libintl functions:
Disabled it's use of gettext calls and added the last "new" version.
Co-authored-by: Bernhard Kaindl <bernhard.kaindl@ait.ac.at>
The hand-written `configure` script of this package does not handle
--without-<feature> at all. The source wants to use `lhapdf` headers
even if support of lhapdf is not indicated using `--with-lhapdf`.
Enable the variant `lhapdf` by default: It fixes the build of the
current package and provides the TauSpinner feature as well.
Fix the build for normal non-root/non-system-user builds, as we cannot
know that we'd have to uninstall these files even if installed as root.
Also add `pkgconfig` and remove not explicitly needed `depends_on('m4')`
* update the Tau package to use the correct ROCm dependencies and prefixes
1st:
When the rocm variant is selected, tau defaults to look for rocm in /opt/rocm
which is not guarenteed to be the correct location -- this has been fixed
to provide the prefix for hsa-rocr-dev (which is now a dependency when +rocm is
selected).
2nd:
the rocprofiler dependency package was not specified correctly, it should
be called rocprofiler-dev, also rocprofiler-dev is a dependency when
+rocprofiler is selected.
added roctracer support
w3m's build fails with `undefined reference to `RAND_egd'` which
is an deprecated insecure feature and from building japanese messages.
Disabling both makes the build of `w3m` work.
This commit shows a template for cut-and-paste into the package to fix it:
```py
==> fast-global-file-status: Executing phase: 'autoreconf'
==> Error: RuntimeError: Cannot generate configure: missing dependencies autoconf, automake, libtool.
Please add the following lines to the package:
depends_on('autoconf', type='build', when='@master')
depends_on('automake', type='build', when='@master')
depends_on('libtool', type='build', when='@master')
Update the version (when='@master') as needed.
```
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
The build needs pkgconfig and lzma, m4 is already added by autoconf.
Disable generation of kmod manpages as spack does not have xsltproc yet.
Co-authored-by: Bernhard Kaindl <bernhard.kaindl@ait.ac.at>
Assimp searches for zlib (or builds its own version). When it searches, it can find a system install that is not provided by spack. Ref: d286aadbdf/CMakeLists.txt (L451)
Tumbleweed has been broken for a couple of days. The attempt
to fix it in #26170 didn't really work. Let's try to move to
a more stable release series for OpenSuse.
* Make libunwind optional
* Add support for sized_delete and debugalloc
Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Fix the build of pango and it's 20 dependents: Only provide the versions which
support the build using autotools (conversion to MesonPackage didn't progress)
This only restores the list of versions of August 10, before the build broke.
This adds lockfile tracking to Spack's lock mechanism, so that we ensure that there
is only one open file descriptor per inode.
The `fcntl` locks that Spack uses are associated with an inode and a process.
This is convenient, because if a process exits, it releases its locks.
Unfortunately, this also means that if you close a file, *all* locks associated
with that file's inode are released, regardless of whether the process has any
other open file descriptors on it.
Because of this, we need to track open lock files so that we only close them when
a process no longer needs them. We do this by tracking each lockfile by its
inode and process id. This has several nice properties:
1. Tracking by pid ensures that, if we fork, we don't inadvertently track the parent
process's lockfiles. `fcntl` locks are not inherited across forks, so we'll
just track new lockfiles in the child.
2. Tracking by inode ensures that referencs are counted per inode, and that we don't
inadvertently close a file whose inode still has open locks.
3. Tracking by both pid and inode ensures that we only open lockfiles the minimum
number of times necessary for the locks we have.
Note: as mentioned elsewhere, these locks aren't thread safe -- they're designed to
work in Python and assume the GIL.
Tasks:
- [x] Introduce an `OpenFileTracker` class to track open file descriptors by inode.
- [x] Reference-count open file descriptors and only close them if they're no longer
needed (this avoids inadvertently releasing locks that should not be released).
Spack's source mirror was previously in a plain old S3 bucket. That will still
work, but we can do better. This switches to AWS's CloudFront CDN for hosting
the mirror.
CloudFront is 16x faster (or more) than the old bucket.
- [x] change mirror to https://mirror.spack.io
This PR fixes two problems with clang/llvm's version detection. clang's
version output looks like this:
```
clang version 11.0.0
Target: x86_64-unknown-linux-gnu
```
This caused clang's version to be misdetected as:
```
clang@11.0.0
Target:
```
This resulted in errors when trying to actually use it as a compiler.
When using `spack external find`, we couldn't determine the compiler
version, resulting in errors like this:
```
==> Warning: "llvm@11.0.0+clang+lld+lldb" has been detected on the system but will not be added to packages.yaml [reason=c compiler not found for llvm@11.0.0+clang+lld+lldb]
```
Changing the regex to only match until the end of the line fixes these
problems.
Fixes: #19473
clean_environment(): Unset three more environment variables:
MAKEFLAGS: Affects make, can eg indirectly inhibit enabling parallel build
DISPLAY: Tests of GUI widget libraries might try to connect to an X server
TERM: Could make testsuites attempt to color their output
* Switch Umpire to CMakeCachedPackage
* Fix missing import
* Correct tests option in Umpire
* Switch RAJA to CachedCMakePackage
* Convert CHAI to CachedCMakePackage
* Corrections in RAJA
* Patches in Umpire & RAJA for BLT target export
* Fixup style
* Fixup incorrect use of cmake_cache_string
fixes#25992
Currently the bootstrapping process may need a compiler.
When bootstrapping from sources the need is obvious, while
when bootstrapping from binaries it's currently needed in
case patchelf is not on the system (since it will be then
bootstrapped from sources).
Before this PR we were searching for compilers as the
first operation, in case they were not declared in
the configuration. This fails in case we start
bootstrapping from within an environment.
The fix is to defer the search until we have swapped
configuration.
While debugging #24508, I noticed that we call `basename` in `cc`. The
same can be achieved by using Bash's parameter expansion, saving one
external process per call.
Parameter expansion cannot replace basename for directories in some
cases, but is guaranteed to work for executables.
Git 2.24 introduced a feature flag for repositories with many files, see:
https://github.blog/2019-11-03-highlights-from-git-2-24/#feature-macros
Since Spack's Git repository contains roughly 8,500 files, it can be
worthwhile to enable this, especially on slow file systems such as NFS:
```
$ hyperfine --warmup 3 'cd spack-default; git status' 'cd spack-manyfiles; git status'
Benchmark #1: cd spack-default; git status
Time (mean ± σ): 3.388 s ± 0.095 s [User: 256.2 ms, System: 625.8 ms]
Range (min … max): 3.168 s … 3.535 s 10 runs
Benchmark #2: cd spack-manyfiles; git status
Time (mean ± σ): 168.7 ms ± 10.9 ms [User: 98.6 ms, System: 126.1 ms]
Range (min … max): 144.8 ms … 188.0 ms 19 runs
Summary
'cd spack-manyfiles; git status' ran
20.09 ± 1.42 times faster than 'cd spack-default; git status'
```
* Add support for C++20 to HPX package
* Enable unity builds in HPX package when available
* Add support for HIP/ROCm to HPX package
* Rearrange and update required versions for HPX package
* Add C++20 option to asio package
Modifications:
- [x] Change `defaults/config.yaml`
- [x] Add a fix for bootstrapping patchelf from sources if `compilers.yaml` is empty
- [x] Make `SPACK_TEST_SOLVER=clingo` the default for unit-tests
- [x] Fix package failures in the e4s pipeline
Caveats:
1. CentOS 6 still uses the original concretizer as it can't connect to the buildcache due to issues with `ssl` (bootstrapping from sources requires a C++14 capable compiler)
1. I had to update the image tag for GitlabCI in e699f14.
1. libtool v2.4.2 has been deprecated and other packages received some update
This will allow a user to (from anywhere a Spec is parsed including both name and version) refer to a git commit in lieu of
a package version, and be able to make comparisons with releases in the history based on commits (or with other commits). We do this by way of:
- Adding a property, is_commit, to a version, meaning I can always check if a version is a commit and then change some action.
- Adding an attribute to the Version object which can lookup commits from a git repo and find the last known version before that commit, and the distance
- Construct new Version comparators, which are tuples. For normal versions, they are unchanged. For commits with a previous version x.y.z, d commits away, the comparator is (x, y, z, '', d). For commits with no previous version, the comparator is ('', d) where d is the distance from the first commit in the repo.
- Metadata on git commits is cached in the misc_cache, for quick lookup later.
- Git repos are cached as bare repos in `~/.spack/git_repos`
- In both caches, git repo urls are turned into file paths within the cache
If a commit cannot be found in the cached git repo, we fetch from the repo. If a commit is found in the cached metadata, we do not recompare to newly downloaded tags (assuming repo structure does not change). The cached metadata may be thrown out by using the `spack clean -m` option if you know the repo structure has changed in a way that invalidates existing entries. Future work will include automatic updates.
# Finding previous versions
Spack will search the repo for any tags that match the string of a version given by the `version` directive. Spack will also search for any tags that match `v + string` for any version string. Beyond that, Spack will search for tags that match a SEMVER regex (i.e., tags of the form x.y.z) and interpret those tags as valid versions as well. Future work will increase the breadth of tags understood by Spack
For each tag, Spack queries git to determine whether the tag is an ancestor of the commit in question or not. Spack then sorts the tags that are ancestors of the commit by commit-distance in the repo, and takes the nearest ancestor. The version represented by that tag is listed as the previous version for the commit.
Not all commits will find a previous version, depending on the package workflow. Future work may enable more tangential relationships between commits and versions to be discovered, but many commits in real world git repos require human knowledge to associate with a most recent previous version. Future work will also allow packages to specify commit/tag/version relationships manually for such situations.
# Version comparisons.
The empty string is a valid component of a Spack version tuple, and is in fact the lowest-valued component. It cannot be generated as part of any valid version. These two characteristics make it perfect for delineating previous versions from distances. For any version x.y.z, (x, y, z, '', _) will be less than any "real" version beginning x.y.z. This ensures that no distance from a release will cause the commit to be interpreted as "greater than" a version which is not an ancestor of it.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: Gregory Becker <becker33@llnl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
This PR coincides with tiny changes to spack to support spack monitor using the new spec
the corresponding spack monitor PR is at https://github.com/spack/spack-monitor/pull/31.
Since there are no changes to the database we can actually update the current server
fairly easily, so either someone can test locally or we can just update and then
test from that (and update as needed).
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
* ESMF and NEMSIO changes.
- Updating ESMF to set the COMM correctly when using Intel oneapi.
- Explicitly setting the CMake MPI Fortran compiler for NEMSIO.
* Update UFS utils CMake to use MPI_<lang>_COMPILER.
#22845 revealed a long-standing bug that had never been triggered before, because the
hashing algorithm had been stable for multiple years while the bug was in production. The
bug was that when reading a concretized environment, Spack did not properly read in the
build hashes associated with the specs in the environment. Those hashes were recomputed
(and as long as we didn't change the algorithm, were recomputed identically). Spack's
policy, though, is never to recompute a hash. Once something is installed, we respect its
metadata hash forever -- even if internally Spack changes the hashing method. Put
differently, once something is concretized, it has a concrete hash, and that's it -- forever.
When we changed the hashing algorithm for performance in #22845 we exposed the bug.
This PR fixes the bug at its source, but properly reading in the cached build hash attributes
associated with the specs. I've also renamed some variables in the Environment class
methods to make a mistake of this sort more difficult to make in the future.
* ensure environment build hashes are never recomputed
* add comment clarifying reattachment of env build hashes
* bump lockfile version and include specfile version in env meta
* Fix unit-test for v1 to v2 conversion
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* Refactor platform etc. to avoid circular dependencies
All the base classes in spack.architecture have been
moved to the corresponding specialized subpackages,
e.g. Platform is now defined within spack.platforms.
This resolves a circular dependency where spack.architecture
was both:
- Defining the base classes for spack.platforms, etc.
- Collecting derived classes from spack.platforms, etc.
Now it dopes only the latter.
* Move a few platform related functions to "spack.platforms"
* Removed spack.architecture.sys_type()
* Fixup for docs
* Rename Python modules according to review
* dvsdk: Turn off variants by default
This allows an install to more easily be explicit about which pieces to
turn on as more variants are added
* dvsdk: effectively disable the broken variants
* Switch http to https where latter exists
* Hopefully restore original permissions
* Add URL updates after include the -L curl option
* Manual corrections to select URL format strings
* Tell gtk-doc where the XML catalog is
The gtk-doc configure script has an option for specifying the path to
the XML catalog. If this is not set the configure script will search
a defined set of directories for a catalog file and will set
`with_xml_catalog` based on that. Only if no system catalog is found will
the XML_CATALOG_FILES be looked at. In order to make sure that the spack
provided catalog is used, pass the `--with-xml-catalog` option.
* Use the property from docbook-xml
Currently as part of installing a package, we lock a prefix, check if
it exists, and create it if not; the logic for creating the prefix
included a check for the existence of that prefix (and raised an
exception if it did), which was redundant.
This also includes removal of tests which were not verifying
anything (they pass with or without the modifications in this PR).
- Parallel install was failing to generate a config file.
- OpenSSH has an extensive test suite, run it if requested.
- 'executables' wrongly had 'rsh', replaced the openssh tools.
There are two ways to build SQLite: With the Autotools setup or the
so-called "amalgamation" which is a single large C file containing the
SQLite implementation. The amalgamation build is controlled by
pre-processor flags and the Spack setup was using an amalgamation
pre-processor flag for a feature that is controlled by an option of the
configure script. As a consequence, until now Spack has always built
SQLite with the rtree feature enabled.
Knowing that spack has patched the code and organized the build is potentially valuable information for GROMACS users and developers troubleshooting their builds.
PLUMED does further patches to GROMACS, so that is expressed directly also.
Modifications:
- Export platforms from spack.platforms directly, so that client modules don't have to import submodules
- Use only plain imports in test/architecture.py
- Parametrized test in test/architecture.py and put most of the setup/teardown in fixtures
This is a major rework of Spack's core core `spec.yaml` metadata format. It moves from `spec.yaml` to `spec.json` for speed, and it changes the format in several ways. Specifically:
1. The spec format now has a `_meta` section with a version (now set to version `2`). This will simplify major changes like this one in the future.
2. The node list in spec dictionaries is no longer keyed by name. Instead, it is a list of records with no required key. The name, hash, etc. are fields in the dictionary records like any other.
3. Dependencies can be keyed by any hash (`hash`, `full_hash`, `build_hash`).
4. `build_spec` provenance from #20262 is included in the spec format. This means that, for spliced specs, we preserve the *full* provenance of how to build, and we can reproduce a spliced spec from the original builds that produced it.
**NOTE**: Because we have switched the spec format, this PR changes Spack's hashing algorithm. This means that after this commit, Spack will think a lot of things need rebuilds.
There are two major benefits this PR provides:
* The switch to JSON format speeds up Spack significantly, as Python's builtin JSON implementation is orders of magnitude faster than YAML.
* The new Spec format will soon allow us to represent DAGs with potentially multiple versions of the same dependency -- e.g., for build dependencies or for compilers-as-dependencies. This PR lays the necessary groundwork for those features.
The old `spec.yaml` format continues to be supported, but is now considered a legacy format, and Spack will opportunistically convert these to the new `spec.json` format.
* Added spackage to build Sina (https://github.com/LLNL/Sina).
* Improvements to sina/package.py
Made numerous simplifications and improvements to sina/package.py
based on PR feedback.
* Added licence info
* Added maintainers
* Changed maintainers to be Github IDs.
Added a dependency for mpip@3.5: when the libunwind is set to true (which is the default)
and '~setjmp' is set to False (which is also the default) to avoid a configure
time error from not finding libunwind.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
This modification accounts for:
1. Bootstrapping from sources using system, non-standard Python
2. Using later an ABI compatible standard Python interpreter
* tests: make `spack url [stats|summary]` work on mock packages
Mock packages have historically had mock hashes, but this means they're also invalid
as far as Spack's hash detection is concerned.
- [x] convert all hashes in mock package to md5 or sha256
- [x] ensure that all mock packages have a URL
- [x] ignore some special cases with multiple VCS fetchers
* url stats: add `--show-issues` option
`spack url stats` tells us how many URLs are using what protocol, type of checksum,
etc., but it previously did not tell us which packages and URLs had the issues. This
adds a `--show-issues` option to show URLs with insecure (`http`) URLs or `md5` hashes
(which are now deprecated by NIST).
This allows to fix the compilation of gcc versions less than 11.1.0,
due to the remove of cyclades of libsanitizer as it is described in
the patch:
The Linux kernel has removed the interface to cyclades from the latest
kernel headers due to them being orphaned for the past 13
years. libsanitizer uses this header when compiling against glibc, but
glibcs itself doesn't seem to have any references to cyclades. Further
more it seems that the driver is broken in the kernel and the firmware
doesn't seem to be available anymore. As such since this is breaking
the build of libsanitizer (and so the GCC bootstrap) it is proposed to
remove this.
Co-authored-by: Arcesio Castaneda Medina <arcesio.castaneda.medina@itwm.fraunhofer.de>
By changing return values from C #defines to enums, gdbm-1.20 breaks a kludge:
#ifndef GDBM_ITEM_NOT_FOUND
# define GDBM_ITEM_NOT_FOUND GDBM_NO_ERROR
#endif
The absence of the #define causes perl to #define GDBM_ITEM_NOT_FOUND
as GDBM_NO_ERROR which incorrect for gdbm@1.20:
* Optionally enable ccmake in cmake
Renames ncurses variant to `ccmake` since that's how users know it, and
explicitly enable/disable `BUILD_CursesDialog`.
* Make cmake locate its dependencies with CMAKE_PREFIX_PATH, and set rpath flags too
* Undo variant name & defaults change
Fixes removal of SPACK_ENV_PATH from PATH in the presence of trailing
slashes in the elements of PATH:
The compiler wrapper has to ensure that it is not called nested like
it would happen when gcc's collect2 uses PATH to call the linker ld,
or else the compilation fails.
To prevent nested calls, the compiler wrapper removes the elements
of SPACK_ENV_PATH from PATH.
Sadly, the autotest framework appends a slash to each element
of PATH when adding AUTOTEST_PATH to the PATH for the tests,
and some tests like those of GNU bison run cc inside the test.
Thus, ensure that PATH cleanup works even with trailing slashes.
This fixes the autotest suite of bison, compiling hundreds of
bison-generated test cases in a autotest-generated testsuite.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
netlib-lapack: Version 3.9.0 and above no longer builds with the IBM XL
compiler (#25447). Ported some fixes from the old ibm-xl.patch and added
logic for detection of XL's -qrecur flag.
Apply stable-release fixes from 2017 to older autoconf releses:
- Fix the scripts autoheader and autoscan to pass the test suite
- Fix test case to passing when libtool 2.4.3+ is in use
autoconf-2.13 dates back to 1999. The build wasn't possible since
4 years: Since 2017, we patch autom4te which didn't exist in 2.13,
failing the build of it. 4 years of not being able to build 2.13
is a crystal clear indication that we can remove it safely.
* amrex: support sundials variant in newer amrex versions
* propagate cuda_arch to sundials
* change to old string formatting
* require sundials+rocm when amrex+rocm
Ensure that testsuite has py-anytree and py-parameterized
and finds gtk-doc's gitdocize.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This PR will add a new audit, specifically for spack package homepage urls (and eventually
other kinds I suspect) to see if there is an http address that can be changed to https.
Usage is as follows:
```bash
$ spack audit packages-https <package>
```
And in list view:
```bash
$ spack audit list
generic:
Generic checks relying on global variables
configs:
Sanity checks on compilers.yaml
Sanity checks on packages.yaml
packages:
Sanity checks on specs used in directives
packages-https:
Sanity checks on https checks of package urls, etc.
```
I think it would be unwise to include with packages, because when run for all, since we do requests it takes a long time. I also like the idea of more well scoped checks - likely there will be other addresses for http/https within a package that we eventually check. For now, there are two error cases - one is when an https url is tried but there is some SSL error (or other error that means we cannot update to https):
```bash
$ spack audit packages-https zoltan
PKG-HTTPS-DIRECTIVES: 1 issue found
1. Error with attempting https for "zoltan":
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'www.cs.sandia.gov'. (_ssl.c:1125)>
```
This is either not fixable, or could be fixed with a change to the url or (better) contacting the site owners to ask about some certificate or similar.
The second case is when there is an http that needs to be https, which is a huge issue now, but hopefully not after this spack PR.
```bash
$ spack audit packages-https xman
Package "xman" uses http but has a valid https endpoint.
```
And then when a package is fixed:
```bash
$ spack audit packages-https zlib
PKG-HTTPS-DIRECTIVES: 0 issues found.
```
And that's mostly it. :)
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
* py-jupyterhub: add version: 1.4.1
* dont need mako for latest release
* sort dependencies
* notebook isnt used for 1.4.1+
* add dependency on py-jupyter-telemetry; create new package py-jupyter-telemetry
* py-jupyter-telemetry: declare missing dependencies
* py-jupyterhub: need more specific depends_on before less specific
* add py-json-logger; py-jupyter-telemetry: add depends_on for py-json-logger
* Update var/spack/repos/builtin/packages/py-jupyter-telemetry/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* remove py-json-logger erroneously and duplicatively added
* Update var/spack/repos/builtin/packages/py-jupyterhub/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* need py-alembic@1.4: for newest py-jupyterhub
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add a __reduce__ method to Spec
fixes#23892
The recursion limit seems to be due to the default
way in which a Spec is serialized, following all
the attributes. It's still not clear to me why this
is related to being in an environment, but in any
case we already have methods to serialize Specs to
disk in JSON and YAML format. Here we use them to
pickle a Spec instance too.
* Downgrade to build-hash
Hopefully nothing will change the package in
between serializing the spec and sending it
to the child process.
* Add support for Python 2
* Make sure PackageInstaller does not remove the just-restored
install dir after failure in spack install --overwrite
* Remove cryptic error message and rethrow actual error
By default, figlet looks for fonts in `/usr/local/share/figlet`, and if
it doesn't exist you get `figlet: standard: Unable to open font file`.
This fix changes the default font dir to the one installed in the
install prefix.
The gcc compiler can be configured to use `ld.gold` by default. It will
then call `ld.gold` explicitly when linking. When so, spack need to have
a ld.gold wrapper in PATH to inject rpaths link flags etc...
Also I wouldn't be surprised to see some package calling `ld.gold`
directly.
As for ld.gold, the argument could be made that we want to support any
package that could call ld.lld.
* Add a __reduce__ method to SpecBuildInterface
This class was confusing pickle when being serialized,
due to its scary nature of being an object that disguise
as another type.
* Add more MacOS tests, switch them to clingo
* Fix condition syntax
* Remove Python v3.6 and v3.9 with macOS
some of these are not resolvable in that there is only an http page
available, or a page reported as broken is actually ok, or a page has
an SSL error that does not prevent one from visiting (and no good replacement)
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
* Add intel-tbb-oneapi package that does the cmake configure and build.
Compare too the intel-oneapi-tbb package which only downloads a script that contains prebuilt binaries.
* Rename package intel-tbb-cmake
* Incorporate intel-tbb-cmake into intel-tbb package
* Conditionally remove 'context' from kwargs in _urlopen
Previously, 'context' is purged from kwargs in _urlopen to
conform to varying support for 'context' in different versions
of urllib. This fix tries to use 'context', and then removes
it if an exception is thrown and tries again.
* Specify error type in try statement in _urlopen
Specify TypeError when checking if 'context' is in kwargs
for _urlopen. Also, if try fails, check that 'context' is
in the error message before removing from kwargs.
This is a direct followup to #13557 which caches additional attributes that were added in #24095 that are expensive to compute. I had to reopen#25556 in another PR to invalidate the GitLab CI cache, but see #25556 for prior discussion.
### Before
```console
$ time spack env activate .
real 2m13.037s
user 1m25.584s
sys 0m43.654s
$ time spack env view regenerate
==> Updating view at /Users/Adam/.spack/.spack-env/view
real 16m3.541s
user 10m28.892s
sys 4m57.816s
$ time spack env deactivate
real 2m30.974s
user 1m38.090s
sys 0m49.781s
```
### After
```console
$ time spack env activate .
real 0m8.937s
user 0m7.323s
sys 0m1.074s
$ time spack env view regenerate
==> Updating view at /Users/Adam/.spack/.spack-env/view
real 2m22.024s
user 1m44.739s
sys 0m30.717s
$ time spack env deactivate
real 0m10.398s
user 0m8.414s
sys 0m1.630s
```
Fixes#25555Fixes#25541
* Speedup environment activation, part 2
* Only query distutils a single time
* Fix KeyError bug
* Make vermin happy
* Manual memoize
* Add comment on cross-compiling
* Use platform-specific include directory
* Fix multiple bugs
* Fix python_inc discrepancy
* Fix import tests
Most of these are perl packages that need to point to the meta docs site,
and then a fair amount of http addresses that need to be https, and then
the rest are usually documentation sites that no longer exist or were
otherwise changes
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
* Set pubkey trust to ultimate during `gpg trust`
Tries to solve the same problem as #24760 without surpressing stderr
from gpg commands.
This PR makes every imported key trusted in the gpg database.
Note: I've outlined
[here](https://github.com/spack/spack/pull/24760#issuecomment-883183175)
that gpg's trust model makes sense, since how can we trust a random
public key we download from a binary cache?
* Fix test
Fixes#25603
This commit adds a new context manager to temporarily
deactivate active environments. This context manager
is used when setting up bootstrapping configuration to
make sure that the current environment is not affected
by operations on the bootstrap store.
* Preserve exit code 1 if nothing is found
* Use context manager for the environment
- remove unneeded dependency on blas
- create external-lapack variant
- patch makefile to not build lapack if `+external-lapack`
Also:
- fix homepage link
- set parallel = False
- make references to `spec` consistent
- remove unneeded `build` method
This commit adds a regression test for version selection
with preferences in `packages.yaml`. Before PR 25585 we
used negative weights in a minimization to select the
optimal version. This may lead to situations where a
dependency may make the version score of dependents
"better" if it is preferred in packages.yaml.
PackageInstaller and Package.installed disagree over what it means
for a package to be installed: PackageInstaller believes it should be
enough for a database entry to exist, whereas Package.installed
requires a database entry & a prefix directory.
This leads to the following niche issue:
* a develop spec in an environment is successfully installed
* then somehow its install prefix is removed (e.g. through a bug fixed
in #25583)
* you modify the sources and reinstall the environment
1. spack checks pkg.installed and realizes the develop spec is NOT
installed, therefore it doesn't need to have 'overwrite: true'
2. the installer gets the build task and checks the database and
realizes the spec IS installed, hence it doesn't have to install it.
3. the develop spec is not rebuilt.
The solution is to make PackageInstaller and pkg.installed agree over
what it means to be installed, and this PR does that by dropping the
prefix directory check from pkg.installed, so that it only checks the
database.
As a result, spack will create a build task with overwrite: true for
the develop spec, and the installer in fact handles overwrite requests
fine even if the install prefix doesn't exist (it just does a normal
install).
By default the number of parellel compiler processes launched by
py-grpcio equals the number of threads. This commit limit it to
spack config build_jobs.
* Provide new version of eospac.
+ Provide version 6.5.0beta.
+ Make version 6.4.2 the default
+ Also increment
* volunteer to be the maintainer (for now).
see #25563
When we have a concrete environment and we ask to install a
concrete spec from a file, currently Spack returns a list of
specs that are all the one that match the argument DAG hash.
Instead we want to compare build hashes, which also account
for build-only dependencies.
* Added py-meshio package
* Added setuptools dependency to py-meshio package
* Update var/spack/repos/builtin/packages/py-meshio/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-meshio/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-meshio/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-meshio/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-meshio/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* added missing py-importlib-metadata dependency in py-meshio
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
#25303 filtered padding from build output, but it's still there in binary install/relocate output,
so our CI logs are still quite long and frequently hit the limit.
- [x] add context handler from #25303 to buildcache installation as well
This allows you to run `spack graph --installed` from within an environment and get a dot graph of
its concrete specs.
- [x] make `spack graph -i` environment-aware
- [x] add code to the generated dot graph to ensure roots have min rank (i.e., they're all at the
top or left of the DAG)
As of cray-mpich version 8.1.7, conventional MPI compiler wrappers are included in cray-mpich.
Co-authored-by: Luke Roskop <lroskop@cedar.head.cm.us.cray.com>
Bootstrapping clingo on macOS on `develop` gives errors like this:
```
==> Error: RuntimeError: Unable to locate python command in /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/bin
/Users/gamblin2/Workspace/spack/var/spack/repos/builtin/packages/python/package.py:662, in command:
659 return Executable(path)
660 else:
661 msg = 'Unable to locate {0} command in {1}'
>> 662 raise RuntimeError(msg.format(self.name, self.prefix.bin))
```
On macOS, `python` is laid out differently. In particular, `sys.executable` is here:
```console
Python 2.7.16 (default, May 8 2021, 11:48:02)
[GCC Apple LLVM 12.0.5 (clang-1205.0.19.59.6) [+internal-os, ptrauth-isa=deploy on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python'
```
Based on that, you'd think that
`/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents` would be
where you'd look for a `bin` directory, but you (and Spack) would be wrong:
```console
$ ls /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/
Info.plist MacOS/ PkgInfo Resources/ _CodeSignature/ version.plist
```
You need to look in `sys.exec_prefix`
```
>>> sys.exec_prefix
'/System/Library/Frameworks/Python.framework/Versions/2.7'
```
Which looks much more like a standard prefix, with understandable `bin`, `lib`, and `include`
directories:
```console
$ ls /System/Library/Frameworks/Python.framework/Versions/2.7
Extras/ Mac/ Resources/ bin/ lib/
Headers@ Python* _CodeSignature/ include/
$ ls -l /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python
lrwxr-xr-x 1 root wheel 7B Jan 1 2020 /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python@ -> python2
```
- [x] change `bootstrap.py` to use the `sys.exec_prefix` as the external prefix, instead of just
getting the parent directory of the executable.
This adds lockfile tracking to Spack's lock mechanism, so that we ensure that there
is only one open file descriptor per inode.
The `fcntl` locks that Spack uses are associated with an inode and a process.
This is convenient, because if a process exits, it releases its locks.
Unfortunately, this also means that if you close a file, *all* locks associated
with that file's inode are released, regardless of whether the process has any
other open file descriptors on it.
Because of this, we need to track open lock files so that we only close them when
a process no longer needs them. We do this by tracking each lockfile by its
inode and process id. This has several nice properties:
1. Tracking by pid ensures that, if we fork, we don't inadvertently track the parent
process's lockfiles. `fcntl` locks are not inherited across forks, so we'll
just track new lockfiles in the child.
2. Tracking by inode ensures that referencs are counted per inode, and that we don't
inadvertently close a file whose inode still has open locks.
3. Tracking by both pid and inode ensures that we only open lockfiles the minimum
number of times necessary for the locks we have.
Note: as mentioned elsewhere, these locks aren't thread safe -- they're designed to
work in Python and assume the GIL.
Tasks:
- [x] Introduce an `OpenFileTracker` class to track open file descriptors by inode.
- [x] Reference-count open file descriptors and only close them if they're no longer
needed (this avoids inadvertently releasing locks that should not be released).
This commit rework version facts so that:
1. All the information on versions is collected
before emitting the facts
2. The same kind of atom is emitted for versions
stemming from different origins (package.py
vs. packages.yaml)
In the end all the possible versions for a given
package are totally ordered and they are given
different and increasing weights staring from zero.
This refactor allow us to avoid using negative
weights, which in some configurations may make
parent node score "better" and lead to unexpected
"optimal" results.
Add HPDDM, MMG, ParMMG and Tetgen to PETSc.
Add mmg version 5.5.2 (compatible with PETSc).
Add parmmg, depending on mmg.
Add pic variant to tetgen for PETSc.
* Adding a heap of NOAA packages for UFS.
Adding the Unified Forecast System (UFS) and all of the packages
it depends on.
* Fixing style tests.
* Removing the package CMAKE_BUILD_TYPE override.
* Removing compiler specs from `cmake_args()`.
- provides the site packages fix
- excludes the hdf5 linking changes (which are fixed in conduit@develop's build system)
- relaxes constraints to allows building static ascent against shared python
Once PR binary graduation is deployed, the shared PR mirror will
contain binaries just built by a merged PR, before the subsequent
develop pipeline has had time to finish. Using the shared PR mirror
as a source of binaries will reduce the number of times we have to
rebuild the same full hash.
* Refactor active environment getters
- Make `spack.environment.active_environment` a trivial getter for the active
environment, replacing `spack.environment.get_env` when the arguments are
not needed
- New method `spack.cmd.require_active_environment(cmd_name)` for
commands that require an environment (rather than abusing
get_env/active_environment)
- Clean up calling code to call spack.environment.active_environment or
spack.cmd.require_active_environment as appropriate
- Remove the `-e` parsing from `active_environment`, because `main.py` is
responsible for processing `-e` and already activates the environment.
- Move `spack.environment.find_environment` to
`spack.cmd.find_environment`, to avoid having spack.environment aware
of argparse.
- Refactor `spack install` command so argument parsing is all handled in the
command, no argparse in spack.environment or spack.installer
- Update documentation
* Python 2: toplevel import errors only with 'as ev'
In two files, `import spack.environment as ev` leads to errors
These errors are not well understood ("'module' object has no attribute
'environment'"). All other files standardize on the above syntax.
* Bootstrap clingo from binaries
* Move information on clingo binaries to a JSON file
* Add support to bootstrap on Cray
Bootstrapping on Cray requires, at the moment, to
swap the platform when looking for binaries - due
to #22800.
* Add SHA256 verification for bootstrapped software
Use sha256 verification for binaries necessary to bootstrap
the concretizer and gpg for signature verification
* patchelf: use Spec._old_concretize() to bootstrap
As noted in #24450 we may happen to need the
concretizer when bootstrapping clingo. In that case
only the old concretizer is available.
* Add a schema for bootstrapping methods
Two fields have been added to bootstrap.yaml:
"sources" which lists the methods available for
bootstrapping software
"trusted" which records if a source is trusted or not
A subcommand has been added to "spack bootstrap" to list
the sources currently available.
* Methods used for bootstrapping are configurable from bootstrap:sources
The function that tries to ensure a given Python module
is importable now tries bootstrapping methods in the same
order as they are defined in `bootstrap.yaml`
* Permit to trust/untrust bootstrapping methods
* Add binary tests for MacOS, Ubuntu
* Add documentation
* Add a note on bash
Spack is internally using a patched version of `argparse` mainly to backport Python 3 functionality
into Python 2. This PR makes it such that for the supported Python 3 versions we use `argparse`
from the standard Python library. This PR has been extracted from #25371 where it was needed
to be able to use recent versions of `pytest`.
* Fixed formatting issues when using a pristine argparse.py
* Fix error message for Python 3.X when missing positional arguments
* Account for the change of API in Python 3.7
* Layout multi-valued args into columns in error messages
* Seamless transition in develop if argparse.pyc is in external
* Be more defensive in case we can't remove the file.
Add link type to spack.yaml format
Add tests to verify link behavior is correct for installed files
for all three view types
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
* ampl: Add missing ampl_lic install and improve look of resources
* ampl: Add myself as maintainer
* ampl: Remove unused variable and delete extra lines
Co-authored-by: Rob Groner <rug262@psu.edu>
* rct: new packages (core packages and some dependencies)
* rct: new packages (core packages and some dependencies)
* radical-entk: updated dependencies (according to comments)
* radical-gtod: updated version name
* radical-pilot: updated dependencies (according to comments)
* radical-saga: updated dependencies (according to comments)
* radical-utils: updated dependencies and set old versions deprecated
* saga-python: removed due to absence of packages (in PyPI, GitHub), this project was replaced by `radical-saga` and corresponding package `py-radical-saga` should be used
* saga-python: rolled back, but with deprecation status
* ntplib: removed maintainer
* pika: removed maintainer
The commands have been deprecated in #7098, and have
been failing with an error message since then.
Cleaning the code since it is unlikely that somebody
is still using them.
* Fix for building shared lib when enabling ROCm, for STRUMPACK 5.1.1.
* Update patch for shared lib with STRUMPACK 5.1.1 and ROCm, also update FindHIP.cmake
* update patch for shared libs with ROCm
Preferred providers had a non-zero weight because in an earlier formulation of the logic program that was needed to prefer external providers over default providers. With the current formulation for externals this is not needed anymore, so we can give a weight of zero to both default choices and providers that are externals. _Using zero ensures that we don't introduce any drift towards having less providers, which was happening when minimizing positive weights_.
Modifications:
- [x] Default weight for providers starts at 0 (instead of 10, needed before to prefer externals)
- [x] Rules to compute the `provider_weight` have been refactored. There are multiple possible weights for a given `Virtual`. Only one gets selected by the solver (the one that minimizes the objective function).
- [x] `provider_weight` are now accounting for each different `Virtual`. Before there was a single weight per provider, even if the package was providing multiple virtuals.
* Give preferred providers a weight of zero
Preferred providers had a non-zero weight because in an earlier
formulation of the logic program that was needed to prefer
external providers over default providers.
With the current formulation for externals this is not needed anymore,
so we can give a weight of zero to default choices. Using zero
ensures that we don't introduce any drift towards having
less providers, which was happening when minimizing positive weights.
* Simplify how we compute weights for providers
Rewrite rules so that specific events (i.e. being
an external) unlock the possibility to use certain
weights. The weight being considered is then selected
by the minimization process to be the one that gives
the best score.
* Allow providers to have different weights for different virtuals
Before this change we didn't differentiate providers based on
the virtual they provide, which meant that packages providing
more than one virtual had nonetheless a single weight.
With this change there will be a weight per virtual.
* elk package updated to handle 3 latest versions support for older
versions is dropped
* fixed typos
* openmp dependency handling added
* and for blis too
* Retain support for elk 3, deprecate
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* cp2k: fix build with GCC-10+ and MPICH
* cp2k: update SIRIUS and ELPA dependencies
* elpa: add version 2021.05.001, add ROCm support, include SVE flags
This is both a bugfix and a generalization of #25168. In #25168, we attempted to filter padding
*just* from the debug output of `spack.util.executable.Executable` objects. It turns out we got it
wrong -- filtering the command line string instead of the arg list resulted in output like this:
```
==> [2021-08-05-21:34:19.918576] ["'", '/', 'b', 'i', 'n', '/', 't', 'a', 'r', "'", ' ', "'", '-', 'o', 'x', 'f', "'", ' ', "'", '/', 't', 'm', 'p', '/', 'r', 'o', 'o', 't', '/', 's', 'p', 'a', 'c', 'k', '-', 's', 't', 'a', 'g', 'e', '/', 's', 'p', 'a', 'c', 'k', '-', 's', 't', 'a', 'g', 'e', '-', 'p', 'a', 't', 'c', 'h', 'e', 'l', 'f', '-', '0', '.', '1', '3', '-', 'w', 'p', 'h', 'p', 't', 'l', 'h', 'w', 'u', 's', 'e', 'i', 'a', '4', 'k', 'p', 'g', 'y', 'd', 'q', 'l', 'l', 'i', '2', '4', 'q', 'b', '5', '5', 'q', 'u', '4', '/', 'p', 'a', 't', 'c', 'h', 'e', 'l', 'f', '-', '0', '.', '1', '3', '.', 't', 'a', 'r', '.', 'b', 'z', '2', "'"]
```
Additionally, plenty of builds output padded paths in other plcaes -- e.g., not just command
arguments, but in other `tty` messages via `llnl.util.filesystem` and other places. `Executable`
isn't really the right place for this.
This PR reverts the changes to `Executable` and moves the filtering into `llnl.util.tty`. There is
now a context manager there that you can use to install a filter for all output.
`spack.installer.build_process()` now uses this context manager to make `tty` do path filtering
when padding is enabled.
- [x] revert filtering in `Executable`
- [x] add ability for `tty` to filter output
- [x] install output filter in `build_process()`
- [x] tests
These versions can cause weird concretizations, and it looks like the
old version of xsdk may not even work because of xsdktrilinos being
disabled. The hypre version tagged for xsdk@0.2 no longer exists at the
described location.
With the previous naming scheme, `trilinos@:10` concretizes to
`trilinos@xsdk-0.2.0`. Now, it's clear what the xsdk version is closest
to. Changed from tag to the corresponding commit SHA for safety.
* Do not allow cray build system patch for later version of otf2
* Modify flag_handler logic in the trilinos package
Modify flag_handler logic in the trilinos package to work better with compilers
other than CCE
Run CTest at build time with:
```
spack install --test=root openpmd-api@<version>
```
and run smoke-tests after install and loading of the package via
```
spack load -r /<spec>
spack test run /<spec>
```
This pull request adds a new workflow to build and deploy Spack Docker containers
from GitHub Actions. In comparison with our current system where we use Dockerhub's
CI to build our Docker containers, this workflow will allow us to now build for multiple
architectures and deploy to multiple registries. (At the moment x86_64 and Arm64 because
ppc64le is throwing an error within archspec.)
As currently set up, the PR will build all of the current containers (minus Centos6 because
those yum repositories are no longer available?) as both x86_64 and Arm64 variants. The
workflow is currently setup to build and deploy containers nightly from develop as well as
on tagged releases. The workflow will also build, but NOT deploy containers on a pull request
for the purposes of testing this PR. At the moment it is setup to deploy the built containers to
GitHub's Container Registry although, support for also uploading to Dockerhub/Quay can be
included easily if we decide to keep releasing on Dockerhub/want to begin releasing on Quay.
This is an attempt to fix "Missing base commit" messages in the codecov UI. Because we do not run
full tests on package PRs, package PRs' merge commits on `develop` don't have coverage info. It
appears that codecov will give you an error if the pseudo-base's coverage data doesn't all apply
properly to the real PR base, unless the `allow_coverage_offsets` option is set.
* See here for docs:
https://docs.codecov.com/docs/comparing-commits#pseudo-comparison
* See here for another potential solution:
https://community.codecov.com/t/2480/15
`compare_specs()` had a `colorful` keyword argument, but everything else in
spack uses `color` for this.
- [x] rename the argument
- [x] make the default follow spack's `--color=always/never/auto` setting
* Bump py-boto3, add python constraints, bump deps
* Update var/spack/repos/builtin/packages/py-boto3/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-boto3/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-boto3/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add py-os-service-types
* Update var/spack/repos/builtin/packages/py-os-service-types/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add py-oslo-i18n
* Update var/spack/repos/builtin/packages/py-oslo-i18n/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Bump py-botocore and add python constraints
* Update var/spack/repos/builtin/packages/py-botocore/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Add a workflow to test bootstrapping clingo on
different platforms so that we can detect changes
that break it.
Compute `site_packages_dir` in `bootstrap.py` as it was
before #24095, until we figure a better way to override
that attribute.
These are the versions tested (and successfully patched) against
intel-tbb.nvhpc-remove-flags.2017.patch: @2017, @2017.8, @2018, @2018.6
intel-tbb.nvhpc-remove-flags.2019.patch: @2019
intel-tbb.nvhpc-remove-flags.2019.1.patch: @2019.[1-6]
intel-tbb.nvhpc-remove-flags.2019.7.patch: @2019.[7-8]
intel-tbb.nvhpc-remove-flags.2019.9.patch: @2019.9, 2020.[0-3]
The intel-tbb.nvhpc-version-script-fix.2017.patch was tested and
applied successfully against all of the versions above.
Long, padded install paths can get to be very long in the verbose install
output. This has to be filtered out by the Executable class, as it
generates these debug messages.
- [x] add ability to filter paths from Executable output.
- [x] add a context manager that can enable path filtering
- [x] make `build_process` in `installer.py`
This should hopefully allow us to see most of the build output in
Gitlab pipeline builds again.
`build_process` has been around a long time but it's become a very large,
unwieldy method. It's hard to work with because it has a lot of local
variables that need to persist across all of the code.
- [x] To address this, convert it its own `BuildInfoProcess` class.
- [x] Start breaking the method apart by factoring out the main
installation logic into its own function.
When context managers are used to save and restore values, we need to remember
to use try/finally around the yield in case an exception is thrown. Otherwise,
the cleanup will be skipped.
* rnpletal: New package
RNPL is an old package that is still used today by my collaborators, but doesn't see any development any more. I'm creating a Spack package merely to make it easier to install it on various systems. The code is not modern (C without prototypes – yes, that used to be a thing), and a large diff modernizes the code to make it palatable to modern C and Fortran compilers.
RNPL contains several sub-package. The current Spack package builds only the main one.
* rnpletal: Remove unused import
* Convert into AutotoolsPackage
* Don't check for "shared" variant
* rnpletal: Change "version" to `develop`
* rnpletal: Use existing `configure` function
* adjust for erroneous detection of nvc as gcc
adjust for erroneous detection of nvc as gcc when it is built with gcc
* add missing parenthesis :/
* fix trailing whitespace
* re-work hdf5 patch for nvc to make it more general
* flake8 fixes
* Render as comment
Render intended note as a comment rather than logical constraint
Co-authored-by: Frank Willmore <willmore@anl.gov>
- Change config from the undocumented `use_curl: true/false` to `url_fetch_method: urllib/curl`.
- Documentation of `url_fetch_method` in `defaults/config.yaml`
- Default fetch option explicitly set to `urllib` for users who may not have curl on their system
To upgrade from `use_curl` to `url_fetch_method`, run `spack config update config`
* kadath: New package
* Update var/spack/repos/builtin/packages/kadath/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/kadath/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/kadath/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/kadath/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* kadath: Add description to MPI variant
* kadath: Add empty line
* kadath: Add variant "codes=none" to avoid empty default
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* add variant with compiler optimization
Update package.py to include variant with compiler optimization, benchmarked at A-HUG hackaton to improve major kernel time by roughly 3%.
* fix style
* Update var/spack/repos/builtin/packages/laghos/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The output order for `spack diff` is nondeterministic for larger diffs -- if you
ran it several times it will not put the fields in the spec in the same order on
successive invocations.
This makes a few fixes to `spack diff`:
- [x] Implement the change discussed in https://github.com/spack/spack/pull/22283#discussion_r598337448
to make `AspFunction` comparable in and of itself and to eliminate the need for `to_tuple()`
- [x] Sort the lists of diff properties so that the output is always in the same order.
- [x] Make the output for different fields the same as what we use in the solver. Previously, we
would use `Type(value)` for non-string values and `value` for strings. Now we just use
the value. So the output looks a little cleaner:
```
== Old ========================== == New ====================
@@ node_target @@ @@ node_target @@
- gdbm Target(x86_64) - gdbm x86_64
+ zlib Target(skylake) + zlib skylake
@@ variant_value @@ @@ variant_value @@
- ncurses symlinks bool(False) - ncurses symlinks False
+ zlib optimize bool(True) + zlib optimize True
@@ version @@ @@ version @@
- gdbm Version(1.18.1) - gdbm 1.18.1
+ zlib Version(1.2.11) + zlib 1.2.11
@@ node_os @@ @@ node_os @@
- gdbm catalina - gdbm catalina
+ zlib catalina + zlib catalina
```
I suppose if we want to use `repr()` in the output we could do that and could be
consistent but we don't do that elsewhere -- the types of things in Specs are
all stringifiable so the string and the name of the attribute (`version`, `node_os`,
etc.) are sufficient to know what they are.
* lorene: Install only executables, not unrelated files in the same directory
* lorene: Don't determine compile dependencies
The current way doesn't work (cpp misses C++ include paths), and we don't need dependencies anyway.
* lorene: Correct BLAS library names
* lorene: Remove comment
Gitlab truncates job trace output (even the complete raw output) at 4MB,
so this change captures it to a file under "user_data" artifacts as well,
to make sure we can debug output from the end of the rebuild job.
When a spec fails to build on `develop`, instead of storing an empty file as the entry in the broken specs list, this change stores the full spec yaml as well as links to the failing pipeline and job.
A `spack diff` will take two specs, and then use the spack.solver.asp.SpackSolverSetup to generate
lists of facts about each (e.g., nodes, variants, etc.) and then take a set difference between the
two to show the user the differences.
Example output:
$ spack diff python@2.7.8 python@3.8.11
==> Warning: This interface is subject to change.
--- python@2.7.8/tsxdi6gl4lihp25qrm4d6nys3nypufbf
+++ python@3.8.11/yjtseru4nbpllbaxb46q7wfkyxbuvzxx
@@ variant_value @@
- python patches a8c52415a8b03c0e5f28b5d52ae498f7a7e602007db2b9554df28cd5685839b8
+ python patches 0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87
@@ version @@
- openssl Version(1.0.2u)
+ openssl Version(1.1.1k)
- python Version(2.7.8)
+ python Version(3.8.11)
Currently this uses diff-like output but we will attempt to improve on this in the future.
One use case for `spack diff` is whenever a user has a disambiguate situation and cannot
remember how two different installs are different. The command can also output `--json` in
the case of a more analysis type use case where we want to save complete data with all
diffs and the intersection. However, the command is really more intended for a command
line use case, and we likely will have an analyzer more suited to saving data
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
* Catch ConnectionError from CDash reporter
Catch ConnectionError when attempting to upload the results of `spack install`
to CDash. This follows in the spirit of #24299. We do not want `spack install`
to exit with a non-zero status when something goes wrong while attempting to
report results to CDash.
* Catch HTTP Error 400 (Bad Request) in relate_cdash_builds()
* sst-elements: add optional support for flashdimmsim, dramsim3 and
add new packages for each
* sst-dumpi: add version 7.1.0
* sst-core: autotools dependencies are required for all versions
* new package: dtc
* add error message redirect for +dumpi, otf, and otf2: these are not
currently supported
Modifications:
- Remove the "build tests" workflow from GitHub Actions
- Setup a similar e2e test on Gitlab
In this way we'll reduce load on GitHub Actions workflows and for e2e tests will
benefit from the buildcache reuse granted by pipelines.
ENABLE_SPLASH configuration has been removed entirely after 21.06 so
patch is no longer necessary after #24931. (Versions between 0.90.1 and
21.06 will likely still need a patch, and while it's not clear if this
patch is the right one, seems better to leave something in.)
- add version 9.1.2
- set a license file
- set the license environment variable
- remove the download and license information out of the description so
it does not show up in environment modules
- extend python and set python version constraints
- build gurobipy to be used in any compatible python, used for more
extensive computations than the gurobi shell
- remove preexisting PYTHONPATH from gurobi.sh as the shell uses a
built-in python, which will likely be different from "system" python
- add maintainer
`spack style` previously used a Travis CI variable to figure out
what the base branch of a PR was, and this was apparently also set
on `develop`. We switched to `GITHUB_BASE_REF` to support GitHub
Actions, but it looks like this is set to `""` in pushes to develop,
so `spack style` breaks there.
This PR does two things:
- [x] Remove `GITHUB_BASE_REF` knowledge from `spack style` entirely
- [x] Handle `GITHUB_BASE_REF` in style scripts instead, and explicitly
pass the base ref if it is present, but don't otherwise.
This makes `spack style` *not* dependent on the environment and fixes
handling of the base branch in the right place.
This adds a `--root` option so that `spack style` can check style for
a spack instance other than its own.
We also change the inner workings of `spack style` so that `--config FILE`
(and similar options for the various tools) options are used. This ensures
that when `spack style` runs, it always uses the config from the running spack,
and does *not* pick up configuration from the external root.
- [x] add `--root` option to `spack style`
- [x] add `--config` (or similar) option when invoking style tools
- [x] add a test that verifies we can check an external instance
* [py-lmfit] fixed py-asteval dependency requirements
* [py-lmfit] added version 1.0.2
* [py-lmfit] flake8
* [py-lmfit] 1.0.2 reqires python 3.6
* [py-lmfit] removed newer dependency requirements to be in line with setup.py not requirements.txt
* pbs: new virtual package
Some of our clusters have an older installation of
libtorque and tm.h that are *not* from OpenPBS. Using the current
openpbs dependency for openmpi causes concretization errors due to
restrictions on older python and hwloc requirements that don't apply,
even with an external non-buildable installation.
The new 'torque' bundle package allows users to point to that external
installation without problems.
Detailed description of torque by Sergey Kosukhin <skosukhin@gmail.com>
Intel oneAPI installs maintain a lock file in XDG_RUNTIME_DIR,
which by default exists in /tmp (and is shared by all component
installs). This prevented multiple oneAPI components from being
installed in parallel. This commit sets XDG_RUNTIME_DIR to exist
within Spack's installation Stage, so allows multiple components
to be installed at the same time.
* aws-parallelcluster: update maintainers list
Signed-off-by: Tim Lane <tilne@amazon.com>
* aws-parallelcluster: add v2.11.1
Signed-off-by: Tim Lane <tilne@amazon.com>
This PR fixes the tesseract package
- add missing dependencies
- build documentation
- build and install java component
- build and install training component
This uses our bootstrapping logic to automatically install dependencies for
`spack style`. Users should no longer have to pre-install all of the tools
(`isort`, `mypy`, `black`, `flake8`). The command will do it for them.
- [x] add logic to bootstrap specs with specific version requirements in `spack style`
- [x] remove style tools from CI requirements (to ensure we test bootstrapping)
- [x] rework dependencies for `mypy` and `py-typed-ast`
- `py-typed-ast` needs to be a link dependency
- it needs to be at 1.4.1 or higher to work with python 3.9
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
* Adding package for omegaconf
* Update var/spack/repos/builtin/packages/py-omegaconf/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Changing py-omegaconf to use github source URL instead of pypi
* Style fix
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Worked with flecsi developers to tighten, relax, and clarify
constraints and better understand how the flecsi project uses
legion. In the process, discovered that flecsi@1.4 cannot be
built with legion without heavy changes/reverts to the legion
and gasnet spackages.
Also, most importantly, fixed branding as to how flecsi is spelled
We add compilation flags when using %nvhpc to suppress warnings
(which due to global -Werror flag in the build get promoted to
errors) for the following:
Diagnostic 111: statement is unreachable
Diagnostic 177: variable "foo" was declared but never referenced
Diagnostic 188: enumerated type mixed with another type
Diagnostic 550: variable "foo" was set but never used
#24095 introduced a couple of bugs, which are fixed here:
1. The module path is computed incorrectly for bootstrapped clingo
2. We remove too many paths for `sys.path` in case of failures
z3 is a dependency of llvm and llvm-amdgpu, and when z3 python bindings
are enabled it depends on py-setuptools as a run dependency. That's
fine, except that py-setuptools now influences the hash of
llvm/llvm-amdgpu, which can be very annoying when another package
restricts the py-setuptools version -- you'll end up recompiling llvm
for no good reason :(.
* Updated the lbann package to not enabled OpenMP in BLAS package when
working on Darwin systems.
* Add the Sphinx RTD theme as an explicit dependency when building documentation
In some cases the FindHDF5.cmake returnd a wrong value for the HDF5 library names and path. For example it returns hdf5-shared as library name without a search path or checking if this is really an existing shared library. By HDF5_NO_FIND_PACKAGE_CONFIG_FILE=True/ON to the cmake options, the FindHDF5 module does not rely on a properly install hdf5-config.cmake and thus searches for the library and its paths. This results in a usable return value and fenics works afterwards.
* Updates for dependencies in main branch
* Add more depends
* Make CMake available at runtime for fenics-dolfinx
* Add maintainer
Co-authored-by: Garth N. Wells <gnw20@cam.ac.uk>
Although `cpio` is present in many environments, it may not be always
available.
The failure to build this package can be reproduced in a fresh Docker
image `debian:10`.
* trilinos: rename basker variant
The Basker solver is part of amesos2 but is clearer without the extra
scoping.
* trilinos: automatically enable teuchos and remove variant
Basically everything in trilinos needs teuchos
* trilinos: group top-level dependencies
* trilinos: update dependencies, removing unused
- GLM, X11 are unused (x11 lacks dependency specs too)
- Python variant is more like a TPL so rearrange that
- Gtest internal package shouldn't be compiled or exported
- Add MPI4PY requirement for pytrilinos
* trilinos: remove package meta-options
- XSDK settings and "all opt packages" are not used anywhere
- all optional packages are dangerous
* trilinos: Use hwloc iff kokkos
See #19119, also the HWLOC tpl name was misspelled so this was being ignored before.
* Flake
* Fix trilinos +netcdf~mpi
* trilinos: default to disabling external dependencies
* Remove teuchos from downstream dependencies
* fixup! trilinos: Use hwloc iff kokkos
* Add netcdf requirements to packages with ^trilinos+exodus
* trilinos: disable exodus by default
* fixup! Add netcdf requirements to packages with ^trilinos+exodus
* trilinos: only enable hwloc when @13: +kokkos
* xyce: propagate trilinos dependencies more simply
* dtk: fix missing boost dependency
* trilinos: remove explicit metis dependency
* trilinos: require metis/parmetis for zoltan
Disable zoltan by default to minimize default dependencies
* trilinos: mark mesquite disabled and fix kokkos arch
* xsdk: fix trilinos to also list zoltan [with zoltan2]
* ci: remove nonexistent variant from trilinos
* trilinos: add missing boost dependency
Co-authored-by: Satish Balay <balay@mcs.anl.gov>
Third-party Python libraries may be installed in one of several directories:
1. `lib/pythonX.Y/site-packages` for Spack-installed Python
2. `lib64/pythonX.Y/site-packages` for system Python on RHEL/CentOS/Fedora
3. `lib/pythonX/dist-packages` for system Python on Debian/Ubuntu
Previously, Spack packages were hard-coded to use the (1). Now, we query the Python installation itself and ask it which to use. Ever since #21446 this is how we've been determining where to install Python libraries anyway.
Note: there are still many packages that are hard-coded to use (1). I can change them in this PR, but I don't have the bandwidth to test all of them.
* Python: handle dist-packages and site-packages
* Query Python to find site-packages directory
* Add try-except statements for when distutils isn't installed
* Catch more errors
* Fix root directory used in import tests
* Rely on site_packages_dir property
* Change url and checksums for libpng to official sourceforge archives
* Update url scheme from http to https
* switch to .xz archives
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
* Add py-h5py version 3.3.0
The mpi4py dependency was bumped to 3.0.2 in setup.py. I'm not sure if that's actually required or not, but nothing lower is still tested.
* Use environment variable to stop h5py using setuptools setup_requires feature
* Add myself as a maintainer for py-h5py
* [py-transformers] can now use newer versions of tokenizers
* [py-transformers] Added version 4.6.1
* [py-transformers] removing old patch
* [py-transformers] boto3 no longer needed
* first build of py-torchmeta
* updated versions for torchvision and torch
* [py-torchmeta] using pil provider
Co-authored-by: Sid Pendelberry <sid@rit.edu>
The Makefile for the MAGMA smoke tests uses pkg-config to find
the MAGMA compile flags, but the test() routine in the spack
package was not configured to provide the location of the
pkg-config file. This modification sets PKG_CONFIG_PATH correctly
to allow the smoketests to successfully compile. It also removes
the *_dir variables which were unused by the magma
examples/Makefile.
Using the original concretizer, trying to concretize py-jupyterlab fails
with
```
==> Error: Invalid Version range: 6.1.0:6.1
```
because py-tornado does not have a 6.1.0 version but only a 6.1 one.
Makefiles for libtirpc have hardcoded the -pipe flag to the compiler
nvhpc compilers do not recognize that flag.
This PR provides a patch to remove the -pipe flag from the Makefile.
Patch should work with libtirpc@1.2.6 and @1.1.4
jupyterlab was looking for its application directory inside the python
prefix instead its own one. This was fixed by setting the according
environment variable.
* Permit to enable/disable bootstrapping and customize store location
This PR adds configuration handles to allow enabling
and disabling bootstrapping, and to customize the store
location.
* Move bootstrap related configuration into its own YAML file
* Add a bootstrap command to manage configuration
Spack allows users to set `padded_length` to pad out the installation path in
build farms so that any binaries created are more easily relocatable. The issue
with this is that the padding dominates installation output and makes it
difficult to see what is going on. The padding also causes logs to easily
exceed size limits for things like GitLab artifacts.
This PR fixes this by adding a filter in the logger daemon. If you use a
setting like this:
config:
install_tree:
padded_length: 512
Then lines like this in the output:
==> [2021-06-23-15:59:05.020387] './configure' '--prefix=/Users/gamblin2/padding-log-test/opt/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_pla/darwin-bigsur-skylake/apple-clang-12.0.5/zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga
will be replaced with the much more readable:
==> [2021-06-23-15:59:05.020387] './configure' '--prefix=/Users/gamblin2/padding-log-test/opt/[padded-to-512-chars]/darwin-bigsur-skylake/apple-clang-12.0.5/zlib-1.2.11-74mwnxgn6nujehpyyalhwizwojwn5zga
You can see that the padding has been replaced with `[padded-to-512-chars]` to
indicate the total number of characters in the padded prefix. Over a long log
file, this should save a lot of space and allow us to see error messages in
GitHub/GitLab log output.
The *actual* build logs still have full paths in them. Also lines that are
output by Spack and not by a package build are not filtered and will still
display the fully padded path. There aren't that many of these, so the change
should still help reduce file size and readability quite a bit.
015e29efe1 that introduced this section to the
documentation said “two” here instead of the actual count, three.
9f54cea5c5 then added a fourth, BLAS/LAPACK.
Rather than trying to keep this leading count in sync, this change just replaces
the wording with something more generic/stable.
Getting rid of another top-level file.
`coverage.py` has supported `pyproject.toml` since version 5.0, and
all versions of coverage so far work with python 2.7. We just need to
ensure that a version of coverage with the `toml` extra is installed
in the test environment.
I tested this with `coverage run`, `coverage report`, and `coverage html`.
* openPMD-api: rename develop
Rename to match known Spack version comparison schemes:
```
develop>main>master>head>trunk>9999>0>z>a
```
Currently, the hdf5 patch that is pre-0.14.0 is also applied to
`dev`, which naturally fails (already applied).
* fix dev in warpx
* py-markupsafe: add 2.0.1
* Update var/spack/repos/builtin/packages/py-markupsafe/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This moves our `mypy` configuration from `.mypy.ini` to `.pyproject.toml`
and increases the minimum `mypy` version in the tests.
- [x] move `mypy` configuration to `pyproject.toml`
- [x] remove `.mypy.ini`
- [x] ensure that `mypy` version .900 or higher is used in tests
Ideally a test-only dependency won't be in the build, but until then
mark the requirement of gtest up to 1.10.
See e4s job failure at https://gitlab.spack.io/spack/spack/-/jobs/349959 .
Looks like 1.11 introduces some breaking incompatibilities, so perhaps
we should transition later.
* fix remaining flake8 errors
* imports: sort imports everywhere in Spack
We enabled import order checking in #23947, but fixing things manually drives
people crazy. This used `spack style --fix --all` from #24071 to automatically
sort everything in Spack so PR submitters won't have to deal with it.
This should go in after #24071, as it assumes we're using `isort`, not
`flake8-import-order` to order things. `isort` seems to be more flexible and
allows `llnl` mports to be in their own group before `spack` ones, so this
seems like a good switch.
* Fix compiler test
Use `self.spec.satisfies` on compiler to determine if a flag should be
applied or not. This approach avoids issues with the strings `gcc`
or `clang` appearing in the full path to the compiler executables, as
happens with spack-installed compilers (e.g. `nvhpc%gcc`).
* Limit compiler name search to last path component
@skosukhin pointed out that the cflag modification should happen for any
clang or gcc compiler, regardless of what compiler spec provides them.
This commit reverts to searching for a compiler name containing "gcc"
or "clang", but limits the search to the last path component, which
avoids matching spack-installed compilers built with gcc (e.g.
`nvhpc%gcc`), which will have "gcc" in the compiler path.
* Use `os.path` rather than `pathlib`
Co-authored-by: Paul Henning <phenning@lanl.gov>
`dateutil.parser` was an optional dependency for CVS tests. It was failing on macOS
beacuse the dateutil types were not being installed, and mypy was failing *even when the
CVS tests were skipped*. This seems like it was an oversight on macOS --
`types-dateutil-parser` was not installed there, though it was on Linux unit tests.
It takes 6 lines of YAML and some weird test-skipping logic to get `python-dateutil` and
`types-python-dateutil` installed in all the tests where we need them, but it only takes
4 lines of code to write the date parser we need for CVS, so I just did that instead.
Note that CVS date format can vary from system to system, but it seems like it's always
pretty similar for the parts we care about.
- [x] Replace dateutil.parser with a simpler date regex
- [x] Lose the dependency on `dateutil.parser`
Previous tests of `spack style` didn't really run the tools --
they just ensure that the commands worked enough to get coverage.
This adds several real tests and ensures that we hit the corner
cases in `spack style`. This also tests sucess as well as failure
cases.
This consolidates code across tools in `spack style` so that each
`run_<tool>` function can be called indirecty through a dictionary
of handlers, and os that checks like finding the executable for the
tool can be shared across commands.
- [x] rework `spack style` to use decorators to register tools
- [x] define tool order in one place in `spack style`
- [x] fix python 2/3 issues to Get `isort` checks working
- [x] make isort error regex more robust across versions
- [x] remove unused output option
- [x] change vestigial `TRAVIS_BRANCH` to `GITHUB_BASE_REF`
- [x] update completion
This PR configures the spack docbook packages
- docbook-xsl
- docbook-xml
The public entities are now mapped to the locally installed files of the
respective packages. The example catalogs are left in place and
XML_CATALOG_FILES points to the newly created catalogs.
Perl keeps copies of the bzip2 and zlib source code in its own source
tree and by default uses them in favor of outside libraries. Instead,
put these dependencies under control of spack and tell perl to use the
spack-built versions.
* py-keyring: fix installation on linux
* Update var/spack/repos/builtin/packages/py-keyring/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-keyring/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
We should not fail the generate stage simply due to the presence of
a broken-spec somewhere in the DAG. Only fail if the known broken
spec needs to be rebuilt.
This PR adds a context manager that permit to group the common part of a `when=` argument and add that to the context:
```python
class Gcc(AutotoolsPackage):
with when('+nvptx'):
depends_on('cuda')
conflicts('@:6', msg='NVPTX only supported in gcc 7 and above')
conflicts('languages=ada')
conflicts('languages=brig')
conflicts('languages=go')
```
The above snippet is equivalent to:
```python
class Gcc(AutotoolsPackage):
depends_on('cuda', when='+nvptx')
conflicts('@:6', when='+nvptx', msg='NVPTX only supported in gcc 7 and above')
conflicts('languages=ada', when='+nvptx')
conflicts('languages=brig', when='+nvptx')
conflicts('languages=go', when='+nvptx')
```
which needs a repetition of the `when='+nvptx'` argument. The context manager might help improving readability and permits to group together directives related to the same semantic aspect (e.g. all the directives needed to model the behavior of `gcc` when `+nvptx` is active).
Modifications:
- [x] Added a `when` context manager to be used with package directives
- [x] Add unit tests and documentation for the new feature
- [x] Modified `cp2k` and `gcc` to show the use of the context manager
I installed curl on my mac and it picked up a homebrew (I think?)
installation of gsasl. A later system update broke git because of the
implicitly added dependency. Explicitly disabling libraries that *might*
exist on the system is the safe approach here.
```
dyld: Library not loaded: /usr/local/opt/gsasl/lib/libgsasl.7.dylib
Referenced from: /rnsdhpc/code/spack/opt/spack/apple-clang/curl/gag5v3c/lib/libcurl.4.dylib
Reason: image not found
error: git-remote-https died of signal 6
```
* Added Perl workaround for CUDA <= 8
* Re-wrapped comment
* Proofreading corrections
* Added a reference
* Do not override Perl include path
* Retrieve shell once
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
* trilinos: add teko conflict
* trilinos: improve gotype variant
Instead of 'none' and 'long' typically being the same (but not for older
trilinos versions), add an explicit 'all' variant that only works for
older trilinos which supports multiple simultaneous tpetra
instantiations.
* trilinos: add self as maintainer
* trilinos: disable vendored gtest by default
This changes several conflicting variants to a single
multi-value variant, and uses conflicts instead of raising InstallError.
(With clingo, requesting +gui automatically selects features=huge!)
I have also rearranged the dependencies for clarity and simplified the
conifgure args.
ci: only write to broken-specs list on SpackError
Only write to the broken-specs list when `spack install` raises a SpackError,
instead of writing to this list unnecessarily when infrastructure-related problems
prevent a develop job from completing successfully.
If two Specs have the same hash (and prefix) but are not equal, Spack
originally had logic to detect this and raise an error (since both
cannot be installed in the same place). Recently this has eroded and
the check no-longer works; moreover, when defining projections (which
may truncate the hash or other distinguishing properties from the
prefix) Spack was also failing to detect collisions (in both of these
cases, Spack would overwrite the old prefix with the new Spec).
This PR maintains a list of all "taken" prefixes: if a hash is not
registered (i.e. recorded as installed in the database) but the prefix
is occupied, that is a collision. This can detect collisions created
by defining projections (specifically when they omit the hash).
The PR does not detect collisions where specs have the same hash
(and prefix) but are not equal.
Fix syntax of conflict between numpy 1.21.0 and gcc11 to that the clingo
concretizer recognizes it.
In addition the upstream master branch was renamed to main.
* Switch hdf5 package from autotools to cmake.
* Add variant for building with zlib, default to ON.
* Update for format requirements.
* Format change.
* Fix breakage from last merge from develop.
Switch szip to use libaec (unrestricted encryption).
Remove 'static' variant: static libs will only be installed when
~shared.
* Improve args based on suggestions from pull request.
* Update code URL to github.com
Add/modify 4 depends_on lines to fix running "spack graph --deptype=link hdf5".
* Remove trailing whitespace.
* Remove dependencies added solely to make "spack greph --type=link" work.
* Add new version HDF5 1.8.22.
* Remove unnecessary java_check.
* Fix whitespace for style checks.
* Reverted zlib version dependency to 1.1.2:.
zlib variant removed.
api version default renamed "default".
* Remove blank line.
* Whitespace corrections.
* iRemoved unnecessary 'debug' variant.
* Fix typo in version number in conflict for '+szip'.
* Set default for tools variant to True.
Remove patch functions dependent on 'libtool' file that cmake doesn't
produce.
* Remove line to set ONLY_SHARED_LIBS to true.
Add post_install code to install only one version of tools with shared
linkage and original tool names.
* Remove trailing white space and import of glob package not used.
* Leave BUILD_TESTING set to default which is ON.
* Remove post_install code to install only one version of tools because
some dependent packages running tests in e4s testing are using
h5diff-shared. Keep both tools versions for now.
* No longer need to import os.
Instead of refusing to build +mpi with gcc10, add what I guess is now
the standard workaround, ie., `-fallow-argument-mismatch`.
Getting this into pfunit's cmake-based but kinda non-standard build isi
a bit ugly, but you gotta do what you gotta do...
Version 1.17 of DD4hep was renamed from "01-17-00" to "01-17", in line
with the naming conventions of previous releases. Since release archives
contain a subdirectory with the version string in it, this changes the contents
of the tarball ever so slightly, so the SHA-256 checksum must change as well.
Fix url to find newer versions, add newest version 4.0.2 and add
variants for
- cxxstd: To use a specific c++ standard
- static: Enable or disable build of static libraries
- boost: Boost support
- sqlite: SQLite support
- postgresql: PostgreSQL support
When having a few packages loaded, installing go-bootstrap will fail
because the `PATH` variable is truncated at 4096 bytes. Increase the
limit to 128 KiB to make longer paths fit.
1. "+simplex" conflicts with "dealii@:9.2" [The interface to simplex is supported from version 9.3.0 onwards. Please explicitly disable this variant via ~simplex]
2. "+arborx" conflicts with "dealii@:9.2" [The interface to arborx is supported from version 9.3.0 onwards. Please explicitly disable this variant via ~arborx]
Prior to any Spack build, Spack modifies PATH etc. to help the build
find the dependencies it needs. It also allows any package to define
custom environment modifications (and furthermore a package can
specify environment modifications to apply when it is used as a
dependency). If an external package defines custom environment
modifications that alter PATH, and the external package is in a merged
or system prefix, then that prefix could "override" the Spack-built
packages.
This commit reorders environment modifications so that PrependPath
actions which expose Spack-built packages override PrependPath actions
for custom environment modifications of external packages.
In more detail, the original order of environment modifications is:
* Modules
* Compiler flag variables
* PATH, CMAKE_PREFIX_PATH, and PKG_CONFIG_PATH for dependencies
* Custom package.py modifications in the following order:
* dependencies
* root
This commit changes the order:
* Modules
* Compiler flag variables
* For each external dependency
* PATH, CMAKE_PREFIX_PATH, and PKG_CONFIG_PATH modifications
* Custom modifications
* For each Spack-built dependency
* PATH, CMAKE_PREFIX_PATH, and PKG_CONFIG_PATH modifications
* Custom modifications
Spack pipelines need to take specific actions internally that depend
on whether the pipeline is being run on a PR to spack or a merge to
the develop branch. Pipelines can also run in other repositories,
which represents other possible use cases than just the two mentioned
above. This PR creates a "SPACK_PIPELINE_TYPE" gitlab variable which
is propagated to rebuild jobs, and is also used internally to determine
which pipeline-specific tasks to run.
One goal of the PR is fix an issue where rebuild jobs which failed on
develop pipelines did not properly report the broken full hash to the
"broken-specs-url".
* Add Externally Findable section to info command
* Use comma delimited detection attributes in addition to boolean value
* Unit test externally detectable part of spack info
yes I know this name isn't popular but that's the way it is right now.
master and the upcoming v5.0.x release branch use git submodules.
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
* [py-xxhash] created template
* [py-xxhash] working on dependencies
* [py-xxhash] set version for xxhash
* [py-xxhash] Final cleanup
- added homepage
- added description
- removed fixmes
* Force the Python interpreter with an env variable
This commit forces the Python interpreter with an
environment variable, to ensure that the Python set
by the "setup-python" action is the one being used.
Due to the policy adopted by Spack to prefer python3
over python we may end up picking a Python 3.X
interpreter where Python 2.7 was meant to be used.
* Revert "Update conftest.py (#24473)"
This reverts commit 477c8ce820.
* Make python-dateutil a soft dependency for unit tests
Before #23212 people could clone spack and run
```
spack unit-tests
```
while now this is not possible, since python-dateutil is
a required but not vendored dependency. This change makes
it not a hard requirement, i.e. it will be used if found
in the current interpreter.
* Workaround mypy complaint
This commit fixes a subtle bug that may occur when
a package is a "possible_provider" of a virtual but
no "provides_virtual" can be deduced. In that case
the cardinality constraint on "provides_virtual"
may arbitrarily assign a package the role of provider
even if the constraints for it to be one are not fulfilled.
The fix reworks the logic around three concepts:
- "possible_provider": a package may provide a virtual if some constraints are met
- "provides_virtual": a package meet the constraints to provide a virtual
- "provider": a package selected to provide a virtual
Spack packages can now fetch versions from CVS repositories. Note
this fetch mechanism is unsafe unless using :extssh:. Most public
CVS repositories use an insecure protocol implemented as part of CVS.
Here we are adding an install_times.json into the spack install metadata folder.
We record a total, global time, along with the times for each phase. The type
of phase or install start / end is included (e.g., build or fail)
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
The original implementation of `flag_handler` searched the
`self.compiler.cc` string for `clang` or `gcc` in order to add a flag
for those compilers. This approach fails when using a spack-installed
compiler that was itself built with gcc or clang, as those strings will
appear in the fully-qualified compiler executable paths. This commit
switches to searching for `%gcc` or `%clang` in `self.spec`.
Co-authored-by: Paul Henning <phenning@lanl.gov>
the 4.1.1 release has fixes for problems that kept 4.1.0 from
being the default open mpi version to build using spack.
related to #24396
Signed-off-by: Howard Pritchard <hppritcha@gmail.com>
* remove blueos check on cuda variant, fix typo
* restore necessary compiler guard
* remove axom+cuda from testing because it only partially works outside ppc systems
This PR does the following:
- adds version corresponding to commit at 08/03/2020
- adds missing get_DE_events.py script
- adds dependencies needed by get_DE_events.py
- removes REDItoolDenovo.py.patch and python2to3.patch in favor of
running 2to3 and reindent pre-build
- add batch_sort.patch to handle differences in string/char handling
betweeen python2 and python3
- adds a variant for the Nature Protocol
- adds dependencies for the nature_protocol variant
- added myself as maintainer
This PR adds a new version of reditools from git.
This PR fixes a couple of issues with the opencv package, mostly in
relation to cuda. This is only focused on cuda, not any of the other
variants.
- Added versions to the contrib_vers list. Added for all that can be
retrieved from github. The one for the latest version was missing.
- Added a cmake patch for v3.2.0.
- Deprecated versions 3.1.0 and 3.2.0 as neither of those could be
built, with or without cuda.
- Adjusted constraints on applying initial cmake patch.
- Added cudnn dependency when +cuda.
- Set constraints for cudnn and cuda for older versions of opencv.
Add a new "spack audit" command. This command can check for issues
with configuration or with packages and is intended to help a
user debug a failed Spack build.
In some cases the reported issues are always errors but are too
costly to check for (e.g. packages that specify missing variants on
dependencies). In other cases the issues may be legitimate but
uncommon usage of Spack and we want to be sure the user intended the
behavior (e.g. duplicate compiler definitions).
Audits are grouped by theme, and for now the two themes are packages
and configuration. For example you can run all available audits
on packages with "spack audit packages". It is intended that in
the future users will be able to define their own audits.
The package audits are good candidates for running in package_sanity
(i.e. they could catch bugs in user-submitted packages before they
are merged) but that is left for a later PR.
Building magma has been failing consistently and is currently
blocking PRs from being merged. Disable that spec while we
investigate the failure and work on a fix.
This should get us most of the way there to support using monitor during a spack container build, for both Singularity and Docker. Some quick notes:
### Docker
Docker works by way of BUILDKIT and being able to specify --secret. What this means is that you can prefix a line with a mount of type secret as follows:
```bash
# Install the software, remove unnecessary deps
RUN --mount=type=secret,id=su --mount=type=secret,id=st cd /opt/spack-environment && spack env activate . && export SPACKMON_USER=$(cat /run/secrets/su) && export SPACKMON_TOKEN=$(cat /run/secrets/st) && spack install --monitor --fail-fast && spack gc -y
```
Where the id for one or more secrets corresponds to the file mounted at `/run/secrets/<name>`. So, for example, to build this container with su (spackmon user) and sv (spackmon token) defined I would export them on my host and do:
```bash
$ DOCKER_BUILDKIT=1 docker build --network="host" --secret id=st,env=SPACKMON_TOKEN --secret id=su,env=SPACKMON_USER -t spack/container .
```
And when we add `env` to the secret definition that tells the build to look for the secret with id "st" in the environment variable `SPACKMON_TOKEN` for example.
If the user is building locally with a local spack monitor, we also need to set the `--network` to be the host, otherwise you can't connect to it (a la isolation of course.)
## Singularity
Singularity doesn't have as nice an ability to clearly specify secrets, so (hoping this eventually gets implemented) what I'm doing now is providing the user instructions to write the credentials to a file, add it to the container to source, and remove when done.
## Tags
Note that the tags PR https://github.com/spack/spack/pull/23712 will need to be merged before `--monitor-tags` will actually work because I'm checking for the attribute (that doesn't exist yet):
```bash
"tags": getattr(args, "monitor_tags", None)
```
So when that PR is merged to update the argument group, it will work here, and I can either update the PR here to not check if the attribute is there (it will be) or open another one in the case this PR is already merged.
Finally, I added a bunch of documetation for how to use monitor with containerize. I say "mostly working" because I can't do a full test run with this new version until the container base is built with the updated spack (the request to the monitor server for an env install was missing so I had to add it here).
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Inline codecov annotations make the code hard to read, and they add annotations
in files that seemingly have nothing to do with the PR. Sadly, they add a whole
lot of noise and not a lot of benefit over looking at the PR on codecov. We
should just have people look at the coverage on codecov itself.
* New package: py-pyusb
Change-Id: I606127858b961b5841c60befc5a8353df0f9f38c
* fixup dependencies
Change-Id: I0c9b0ccee693d2c4e847717950d4ce64cb319794
* fixup 2
Change-Id: Ibaccbdafd865e363564f491054e4e4ceb778727b
* Update var/spack/repos/builtin/packages/py-pyusb/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
A patch no longer applies cleanly as its fixed in v4.0.6 - fix it here
==> Installing openmpi-4.0.6-in47f6rxspbnyibkdx6x4ekg6piujobd
==> No binary for openmpi-4.0.6-in47f6rxspbnyibkdx6x4ekg6piujobd found: installing from source
==> Fetching https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-4.0.6.tar.bz2
Reversed (or previously applied) patch detected! Assume -R? [n]
Apply anyway? [n]
2 out of 2 hunks ignored -- saving rejects to file opal/include/opal/sys/gcc_builtin/atomic.h.rej
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
When running executables from build dependencies, we want to avoid that
`LD_PRELOAD` and `DYLD_INSERT_LIBRARIES` any of their shared libs build
by spack with system libraries.
The Z3 solver provides a Z3Config.cmake file when built using the CMake build
system. This submission changes the package build system to inherit the
CMakePackage type. In addition to changing the build system, this submission:
- Adds the GMP variant
- Removes v4.4.0 and v4.4.1 as CMake was implemented starting with v4.5.0
This adds a package for `irep`, a tool for reading `lua` input decks from
Fortran, C, and C++.
`irep` can be built with either `lua` or `luajit`. To address this, we also add
a virtual package for lua called `lua-lang`. `luajit` isn't, by default, a drop-in
replacement for `lua`, but we add a `+lualinks` variant to it that adds symlinks
that make it behave like `lua@5.1`. With this variant enabled, it provides the
`lua-lang` virtual. `lua` always provides `lua-lang`.
- [x] add `irep` package
- [x] add `+lualinks` variant to `lua-luajit`
- [x] create `lua-lang` virtual, provided by `lua` and `luajit+lualinks`
Co-authored-by: Kayla Richarda Butler <butler59@quartz1148.llnl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
* libdrm: fix one configure error and require libpciaccess
Failure with `LIBS`: the linker can't find `-lrt` so configure fails on
darwin-bigsur %apple-clang@12.0.5
```
>> 22 configure: error: in `/private/var/folders/gy/mrg1ffts2h945qj9k29s1l1dvvmbqb/T/s3j/spack-s
tage/spack-stage-libdrm-2.4.100-ofhk6m25n2pi427ihnxmvjkfmgyzlrqc/spack-src':
>> 23 configure: error: C compiler cannot create executables
24 See `config.log' for more details
See build log for details:
/var/folders/gy/mrg1ffts2h945qj9k29s1l1dvvmbqb/T/s3j/spack-stage/spack-stage-libdrm-2.4.100-ofhk6m25n2pi427ihnxmvjkfmgyzlrqc/spack-build-out.txt
```
* libpciaccess: Mark conflict with darwin
```
make[2]: *** [common_init.lo] Error 1
make[2]: *** Waiting for unfinished jobs....
common_interface.c:75:10: fatal error: 'sys/endian.h' file not found
^~~~~~~~~~~~~~
```
and
```
common_init.c:73:3: error: "Unsupported OS"
```
and others
* extending example for buildcaches
I was attempting to create a local build cache from a directory, and I found the
docs for both buildcaches and mirrors, but did not connect the docs that the
url variable could be the local filesystem variable. I am extending the docs for
buildcaches with an example of creating and interacting with one on the filesystem
because I suspect other users will run into this need and possibly not find what
they are looking for.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
* adding as follows to spack mirror list
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* update url, add all new versions and fix installation
* add wxparaver package and set the old paraver package as deprecated
* remove update of deprecated package
* remove old version from new wxparaver
* Update url
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
It is currently kind of confusing to the reader to distinguish spack buildcache install
and spack install, and it is not clear how to use a build cache once a mirror is added.
Hopefully this little big of description can help (and I hope I got it right!)
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Use the 'version_yearlike' attribute instead of 'version' to
check if the SPACK_COMPILER_EXTRA_RPATHS should be set to include
the built-in 'libfabrics'.
When using the bare 'version', the comparison is wrong when
building with 'intel-parallel-studio', which has the version
format '<edition>.YYYY.Nupdate', due to the leading '<edition>'.
xfsprogs currently does not install with error message:
FATAL ERROR: could not find a valid ini.h header.
Adding this package libinih, and including it as
a dependency for xfsprogs seems to fix the issue. It could be
that we only need to add it for newer versions (if it worked before)
and maybe a maintainer can comment on that.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Pagination on Github prevent spack from easily parse all available
versions. Also, due to recent migration to GitHub, tarballs for
versions up to 3.12.13 have be regenerated, changing the hash.
The current URL will apparently be supported, so we keep it, and give
the alternative one as a comment.
This should fix#24278
$INSTALLDIR/lib/python3.7/site-packages/IPython/core/events.py contains an
import from backcall even in @7.3.0, so dependency on py-backcall needs
to start earlier.
Restrict poppler version for texlive to poppler@:0.84
Should fix#19946
See also https://github.com/NixOS/nixpkgs/issues/79170
Looks like poppler@0.84 upgraded their header files to use the C++ cstdio
instead of the C stdio.h. Since TeX is using C, not C++. this causes problems.
* zfp: several package improvements
- add variants for build targets, language bindings, backends
- ensure selected variants are compatible with zfp version
- point to GitHub (not LLNL) tar balls
- add dependencies
- update link to homepage
- add maintainers
* zfp: address suggestions by Spack team
- use conflicts() instead of raising exceptions
- use define() and define_from_variant() where applicable
* Apply suggestions from code review
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* Fix ZFP OpenMP build.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* [py-keyboard] created template
* [py-keyboard]
- updated homepage
- added dependency for OSX
- added description
- removed fixmes
* [py-keyboard] Until py-pyobjc can be created, specifying conflict with platform=darwin
* [py-keyboard] is verb
* Update of Flecsi Spackage
Update of flecsi spackage to reconcile differences between flecsi@1:1.9
and flecsi@2: for future support purposes
* Removing Unnecessary Conditional
Removing unused conditional. Initially the plan was to switch based on
version in `cmake_args` but this was not necessary as build system
variable names remained mostly the same and conflicts prevent the rest.
For the most part, if a variant is there it does not need to check
against what version of the code is being built.
* Updated CI To Reconcile Flecsi Changes
Updated CI to target flecsi@1.4.2 which best matches the previous
release version and reconciled change in variant name
The common.inc script in TBB uses the environ var 'OS' to determine
the platform it's on. On Linux, this is normally empty and TBB falls
back to uname. But some systems set this to 'CentOS Linux 8' which is
descriptive, but not exactly what common.inc is looking for.
Instead, take the value from python and explicitly set OS to what TBB
expects to avoid this problem.
Since the two packages share a common history, the installation
procedure has been factored into a common base class.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
* Tcl: fix TCLLIBPATH
* Fix TCL|TK|TIX_LIBRARY paths
* Fix TCL_LIBRARY, no tcl8.6 subdir
* Don't rely on os.listdirs sorting
For tcl and tk, we also install the source directory, so there are
two init.tcl and tk.tcl locations. We want the one in lib/lib64,
which should come before the one in share.
* Add more patches
* Fix dylib on macOS
* Tk: add smoke tests
* Tix: add smoke test
Extracting specs for the result of a solve has been factored
as a method into the asp.Result class. The method account for
virtual specs being passed as initial requests.
Minimizing compiler mismatches in the DAG and preferring newer
versions of packages are now higher priority than trying to use as
many default values as possible in multi-valued variants.
According to the docs, r is needed for plotting, but plotting is
untested. In addition, the specific version requirement of java for gatk
could lead to multiple installations of r being triggered in an
environment. That might cause people to have to be deliberate about
java in a deployment. All in all, it seems that r is better as a
variant for gatk.
* Set job_id for SGE in darshan-runtime package
* Use a multi value variant for scheduler
Only one scheduler can be selected so make the variant multi valued and
set multi=False.
* hdf-eos5: Fix issue when linking against hdf5+szip (#23411)
Should fix issue #23411 when linking against hdf5+szip
Also fix bug if hdf5 does not depend on zlib
Reluctantly added payerle as a maintainer
Added version 1.1.13
Fixed versions for dependencies based on README.md for package
In particular:
* versions 1.1.x require python@3, at least 3.4 and for 1.1.13 at least 3.6
* py-osqp had been pinned to version 0.4.1, but README.md either shows
no version restriction, of 0.4.1 and higher
* @1.1.13 requires at least 1.1.6 of py-scs
* I am assuming since 1.1.x is python@3 only, py-six no longer required
(it was not explicitly showing up in README.md for these versions)
Since the module roots were removed from the config file,
`--print-shell-vars` cannot find the module roots anymore. Fix it by
using the new `root_path` function. Moreover, the roots for lmod and
modules seems to have been flipped by accident.
* add versions 2.2.0.2 and 2.2.1.1
* Add maintainer
Added Ishaan as additional maintainer as he is also maintainer of the Python bindings
* add new major precice version as dependency
The VALID_VERSION regex didn't check that the version string was
completely valid, only that a prefix of it was. This version ensures
the entire string represents a valid version.
This makes a few related changes.
1. Make the SEGMENT_REGEX identify *which* arm it matches by what groups
are populated, including whether it's a string or int component or a
separator all at once.
2. Use the updated regex to parse the input once with a findall rather
than twice, once with findall and once with split, since the version
components and separators can be distinguished by their group status.
3. Rather than "convert to int, on exception stay string," if the int
group is set then convert to int, if not then construct an instance
of the VersionStrComponent class
4. VersionStrComponent now implements all of the special string
comparison logic as part of its __lt__ and __eq__ methods to deal
with infinity versions and also overloads comparison with integers.
5. Version now uses direct tuple comparison since it has no per-element
special logic outside the VersionStrComponent class.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* New Package:py-haphpipe@1.0.3
* removed llvm restrict. & changed freebayes
* Style fix
* Removed pip, wheel, added url for deps list
* used proper gsutil naming
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* url src for deps, samtools fix
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* petsc: add hip variant
* libceed: add 0.8, disable occa by default, and let autodetect AVX
Disabling OCCA because backend updates did not make this release and
there are some known bugs so most users won't have reason to use OCCA.
https://github.com/CEED/libCEED/pull/688
* WIP: ceed: 4.0 release
* MFEM package updates (#19748)
* MFEM package updates
* mfem: flake8
* [mfem] Various fixes and tweaks.
[arpack-ng] Add a patch to fix building with IBM XL Fortran.
[libceed] Fix building with IBM XL C/C++.
[pumi] Add C++11 flag for version 2.2.3.
* [mfem] Fix the shared CUDA build.
Reported by: @MPhysXDev
* [mfem] Fix a TODO item
* [mfem] Tweak the AmgX dependencies
* [suite-sparse] Fix the version of the mpfr dependency
* MFEM: add initial HIP support using the ROCmPackage.
* MFEM: add 'slepc' variant.
* MFEM: update the patch for v4.2 for SLEPc.
* mfem: apply 'mfem-4.2-slepc.patch' just to v4.2.
* ceed: apply 'spack style'
* [mfem] Add a patch for mfem v4.2 to work with petsc v3.15.0.
[laghos] Add laghos version 3.1 based on the latest commit in
the repository; this version works with mfem v4.2.
[ceed] For ceed v4.0 use laghos v3.1.
* [libceed] Explicitly set 'CC_VENDOR=icc' when using 'intel'
compiler.
* [mfem] Allow pumi >= 2.2.3 with mfem >= 4.2.0.
[ceed] Use pumi v2.2.5 with ceed v4.0.0.
* [ceed] Explicitly use occa v1.1.0 with ceed v4.0.0.
Use mfem@4.2.0+rocm with ceed@4.0.0+mfem+hip.
* [ceed] Add NekRS v21 as a dependency for ceed v4.0.0.
* [ceed] Fix NekRS version: 21 --> 21.0
* [ceed] Propagate +cuda variant to petsc for ceed v4.0.
* [mfem] Propagate '+rocm' variant to some other packages.
* [ceed] Use +rocm variant of nekrs instead of +hip.
* [ceed] Do not enable magma with ceed@4.0.0+hip.
* [libceed] Fix hip build with libceed@0.8.
* [laghos] For v3.1, use the release .tar.gz file instead of commit.
* Remove cuda & hip variants as they are inherited
* [ceed] Remove comments and FIXMEs about 'magma+hip'.
* [ceed] [libceed] Remove TODOs about occa + hip.
* libceed: use ROCmPackage and +rocm
* petsc: use ROCmPackage for HIP
* libceed, petsc: use CudaPackage
* ceed: forward cuda_arch and amdgpu_target
* [mfem] Use Spack's CudaPackage as a base class; as a result,
'cuda_arch' values should not include the 'sm_' prefix.
Also, propagate 'cuda_arch' and 'amdgpu_target' variants
to enabled dependencies.
* petsc: variant is +rocm, package name is hip
Co-authored-by: Jed Brown <jed@jedbrown.org>
Co-authored-by: Thilina Rathnayake <thilinarmtb@gmail.com>
Passing absolute paths from pipeline generate job to downstream rebuild jobs
causes problems when the CI_PROJECT_DIR is not the same for the generate and
rebuild jobs. This has happened, for example, when gitlab checks out the
project into a runner-specific directory and different runners are chosen
for the generate and rebuild jobs.
* ensure that the stage root exists for `spack stage -p <PATH>`
* add test to verify `spack stage -p <PATH>` works!
* move out shared tmp staging path setup to a fixture to fix the test
* Simplified the spack.util.gpg implementation
All the classes defined in this Python module,
which were previously used to construct singleton
instances, have been removed in favor of four
global variables. These variables are initialized
lazily, like before.
The API of the module has been unchanged for the
most part. A few tests have been modified to use
the new global names.
1. add version 2021.05.15.
2. add patch to build old revs with gcc 11.x, version 2021.15.05
already has patch integrated, fixes#23667.
3. add variant +debug to build unoptimized, debug version.
4. add variant +viewer to include hpcviewer and add viewer path to
hpctoolkit module.
5. add dependency on memkind to workaround a glibc problem found on
some Cray platforms.
For me the buildcache force overwrite option does not work. It tries to
delete a file, but errors with a key error, apparently because the
leading / has to be removed.
* util.tty.log: read up to 100 lines if ready
Rework to read up to 100 lines from the captured stdin as long as data
is ready to be read immediately. Adds a helper function to poll with
`select` for ready data. This showed a roughly 5-10x perf improvement
for high-rate writes through the logger with relatively short lines.
* util.tty.log: Defer flushes to end of ready reads
Rather than flush per line, flush per set of reads. Since this is a
non-blocking loop, the total perceived wait is short.
* util.tty.log: only scan each line once, usually
Rather than always find all control characters then substitute them all,
use `subn` to count the number of control characters replaced. Only if
control characters exist find out what they are. This could be made
truly single pass with sub with a function, but it's a more intrusive
change and this got 99%ish of the performance improvement (roughly
another 2x in some cases).
* util.tty.log: remove check for `readable`
Python < 3 does not support a readable check on streams, should not be
necessary here since we control the only use and it's explicitly a
stream to be read.
* e4s ci: enable full e4s
* add llvm-amdgpu to list of specs needing an xlarge tagged runner
* comment out qt and qwt because of intermittent build failures
* remove +rocm specs because rocblas job consistently fails due to infrastructure
* qt: skip multimedia when ~opengl
On 5.9 on macOS the multimedia option causes build errors; on other
platforms and versions it should probably be assumed inoperative anyway.
* qt: Omit flags when disabling multimedia
```
ERROR: Unknown command line option '-no-pulseaudio'.
```
* Work around another qt@5.9 error
* qt: Fix build error on darwin
This PR allows users to `--export`, `--export-secret`, or both to export GPG keys
from Spack. The docs are updated that include a warning that this usually does not
need to be done.
This addresses an issue brought up in slack, and also represented in #14721.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment).
This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention.
As part of this change, the module roots configuration moved from the config section to inside each module configuration.
Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec.
* New Package:py-ucsf-pyem
* Dep additions, eun env deletion
* extraction step change
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
### Overview
The goal of this PR is to make gitlab pipeline builds (especially build failures) more reproducible outside of the pipeline environment. The two key changes here which aim to improve reproducibility are:
1. Produce a `spack.lock` during pipeline generation which is passed to child jobs via artifacts. This concretized environment is used both by generated child jobs as well as uploaded as an artifact to be used when reproducing the build locally.
2. In the `spack ci rebuild` command, if a spec needs to be rebuilt from source, do this by generating and running an `install.sh` shell script which is then also uploaded as a job artifact to be run during local reproduction.
To make it easier to take advantage of improved build reproducibility, this PR also adds a new subcommand, `spack ci reproduce-build`, which, given a url to job artifacts:
- fetches and unzips the job artifacts to a local directory
- looks for the generated pipeline yaml and parses it to find details about the job to reproduce
- attempts to provide a copy of the same version of spack used in the ci build
- if the ci build used a docker image, the command prints a `docker run` command you can run to get an interactive shell for reproducing the build
#### Some highlights
One consequence of this change will be much smaller pipeline yaml files. By encoding the concrete environment in a `spack.lock` and passing to child jobs via artifacts, we will no longer need to encode the concrete root of each spec and write it into the job variables, greatly reducing the size of the generated pipeline yaml.
Additionally `spack ci rebuild` output (stdout/stderr) is no longer internally redirected to a log file, so job output will appear directly in the gitlab job trace. With debug logging turned on, this often results in log files getting truncated because they exceed the maximum amount of log output gitlab allows. If this is a problem, you still have the option to `tee` command output to a file in the within the artifacts directory, as now each generated job exposes a `user_data` directory as an artifact, which you can fill with whatever you want in your custom job scripts.
There are some changes to be aware of in how pipelines should be set up after this PR:
#### Pipeline generation
Because the pipeline generation job now writes a `spack.lock` artifact to be consumed by generated downstream jobs, `spack ci generate` takes a new option `--artifacts-root`, inside which it creates a `concrete_env` directory to place the lockfile. This artifacts root directory is also where the `user_data` directory will live, in case you want to generate any custom artifacts. If you do not provide `--artifacts-root`, the default is for it to create a `jobs_scratch_dir` within your `CI_PROJECT_DIR` (a gitlab predefined environment variable) or whatever is your current working directory if that variable isn't set. Here's the diff of the PR testing `.gitlab-ci.yml` taking advantage of the new option:
```
$ git diff develop..pipelines-reproducible-builds share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml
diff --git a/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml b/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml
index 579d7b56f3..0247803a30 100644
--- a/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml
+++ b/share/spack/gitlab/cloud_pipelines/.gitlab-ci.yml
@@ -28,10 +28,11 @@ default:
- cd share/spack/gitlab/cloud_pipelines/stacks/${SPACK_CI_STACK_NAME}
- spack env activate --without-view .
- spack ci generate --check-index-only
+ --artifacts-root "${CI_PROJECT_DIR}/jobs_scratch_dir"
--output-file "${CI_PROJECT_DIR}/jobs_scratch_dir/cloud-ci-pipeline.yml"
artifacts:
paths:
- - "${CI_PROJECT_DIR}/jobs_scratch_dir/cloud-ci-pipeline.yml"
+ - "${CI_PROJECT_DIR}/jobs_scratch_dir"
tags: ["spack", "public", "medium", "x86_64"]
interruptible: true
```
Notice how we replaced the specific pointer to the generated pipeline file with its containing folder, the same folder we passed as `--artifacts-root`. This way anything in that directory (the generated pipeline yaml, as well as the concrete environment directory containing the `spack.lock`) will be uploaded as an artifact and available to the downstream jobs.
#### Rebuild jobs
Rebuild jobs now must activate the concrete environment created by `spack ci generate` and provided via artifacts. When the pipeline is generated, a directory called `concrete_environment` is created within the artifacts root directory, and this is where the `spack.lock` file is written to be passed to the generated rebuild jobs. The artifacts root directory can be specified using the `--artifacts-root` option to `spack ci generate`, otherwise, it is assumed to be `$CI_PROJECT_DIR`. The directory containing the concrete environment files (`spack.yaml` and `spack.lock`) is then passed to generated child jobs via the `SPACK_CONCRETE_ENV_DIR` variable in the generated pipeline yaml file.
When you don't provide custom `script` sections in your `mappings` within the `gitlab-ci` section of your `spack.yaml`, the default behavior of rebuild jobs is now to change into `SPACK_CONCRETE_ENV_DIR` and activate that environment. If you do provide custom rebuild scripts in your `spack.yaml`, be aware those scripts should do the same thing: assume `SPACK_CONCRETE_ENV_DIR` contains the concretized environment to activate. No other changes to existing custom rebuild scripts should be required as a result of this PR.
As mentioned above, one key change made in this PR is the generation of the `install.sh` script by the rebuild jobs, as that same script is both run by the CI rebuild job as well as exported as an artifact to aid in subsequent attempts to reproduce the build outside of CI. The generated `install.sh` script contains only a single `spack install` command with arguments computed by `spack ci rebuild`. If the install fails, the job trace in gitlab will contain instructions on how to reproduce the build locally:
```
To reproduce this build locally, run:
spack ci reproduce-build https://gitlab.next.spack.io/api/v4/projects/7/jobs/240607/artifacts [--working-dir <dir>]
If this project does not have public pipelines, you will need to first:
export GITLAB_PRIVATE_TOKEN=<generated_token>
... then follow the printed instructions.
```
When run locally, the `spack ci reproduce-build` command shown above will download and process the job artifacts from gitlab, then print out instructions you can copy-paste to run a local reproducer of the CI job.
This PR includes a few other changes to the way pipelines work, see the documentation on pipelines for more details.
This PR erelies on
~- [ ] #23194 to be able to refer to uninstalled specs by DAG hash~
EDIT: that is going to take longer to come to fruition, so for now, we will continue to install specs represented by a concrete `spec.yaml` file on disk.
- [x] #22657 to support install a single spec already present in the active, concrete environment
* embree: allow for compiling with gcc 7.3
strip out unsupported -mprefer-vector-width=256
* embree: fix build on AMD CPUs
The ISAs that embree is compiled for have to match the CPU
features enabled by the compiler, as embree derives theISA
that it compiles for from the latter.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Spack's source mirror was previously in a plain old S3 bucket. That will still
work, but we can do better. This switches to AWS's CloudFront CDN for hosting
the mirror.
CloudFront is 16x faster (or more) than the old bucket.
- [x] change mirror to https://mirror.spack.io
* New package:py-coveralls
* dep fixes
* added python constraint
* pyyaml version constraint
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
- [x] add `in_buildcache` field to DB records to indicate what parts of an index,
which includes roots and dependencies, are in the buildcache.
- [x] add `mark()` method to DB for setting values on single nodes of the DAG.
This also fixes the build with %gcc@11:. According to upstream, the
proper solution is to disable -Werror=array-bounds since the stable
branch will not receive a patch for newer compilers.
* Update py-pint and fix runtime dependency on setuptools
Without the runtime dependency on setuptools, importing pint yields:
0.11:
ModuleNotFoundError: No module named 'pkg_resources'
0.17:
ModuleNotFoundError: No module named 'packaging'
* Fix
* Address comments
I would like to be able to export (and save and then load programatically)
spack blame metadata, so this commit adds a spack blame --json argument,
along with developer docs for it
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
This work will come in two phases. The first here is to allow saving of a local result
with spack monitor, and the second will add a spack monitor command so the user can
do spack monitor upload.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Currently if one package does `depends_on('pkg default_library=shared')`
and another does `depends_on('pkg default_library=both')`, you'd get a
concretization error.
With this PR one package can do `depends_on('pkg default_library=shared')`
and another depends_on('default_library=static'), and it would concretize to
`pkg default_library=shared,static`
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* Package update to version 1.0.2
* switched submodule boolean to string
* switched from string to bools
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
- Changed to cmake package with backward compatibility with older
makefile
- Removed unused cmake variable 'blas_blas_libs'
- Added new version 5.2.2 which change to external blas variable
- Remove unused tcsh dependency
- Change URL to use git repository for current and future versions
- Add older 4.2 version
- Add conflict for older versions with apple-clang
This adds RHEL8's `/usr/libexec/platform-python` to Spack's list of preferred
pythons. It will only be used if no other `python` is available in the `PATH`.
We have been testing with this python for a while now, and it seems to do all
that we need. If Spack one day isn't able to work with it, we'll take it out,
but for now it is useful to allow Spack to be used on RHEL8 without a dedicated
`python` installation.
Spack doesn't require users to manually index their repos; it reindexes the indexes automatically when things change. To determine when to do this, it has to `stat()` all package files in each repository to make sure that indexes up to date with packages. We currently index virtual providers, patches by sha256, and tags on packages.
When this was originally implemented, we ran the checker all the time, at startup, but that was slow (see #7587). But we didn't go far enough -- it still consults the checker and does all the stat operations just to see if a package exists (`Repo.exists()`). That might've been a wash in 2018, but as the number of packages has grown, it's gotten slower -- checking 5k packages is expensive and users see this for small operations. It's a win now to make `Repo.exists()` check files directly.
**Fix:**
This PR does a number of things to speed up `spack load`, `spack info`, and other commands:
- [x] Make `Repo.exists()` check files directly again with `os.path.exists()` (this is the big one)
- [x] Refactor `Spec.satisfies()` so that a checking for virtual packages only happens if needed
(avoids some calls to exists())
- [x] Avoid calling `Repo.exists(spec)` in `Repo.get()`. `Repo.get()` will ultimately try to load
a `package.py` file anyway; we can let the failure to load it indicate that the package doesn't
exist, and avoid another call to exists().
- [x] Fix up some comments in spec parsing
- [x] Call `UnknownPackageError` more consistently in `repo.py`
The ASP-based solver can natively manage cases where more than one root spec is given, and is able to concretize all the roots together (ensuring one spec per package at most).
Modifications:
- [x] When concretising together an environment the ASP-based solver calls directly its `solve` method rather than constructing a temporary fake root package.
The function we coded in Spack to load Python modules with arbitrary
names from a file seem to have issues with local imports. For
loading hooks though it is unnecessary to use such functions, since
we don't care to bind a custom name to a module nor we have to load
it from an unknown location.
This PR thus modifies spack.hook in the following ways:
- Use __import__ instead of spack.util.imp.load_source (this
addresses #20005)
- Sync module docstring with all the hooks we have
- Avoid using memoization in a module function
- Marked with a leading underscore all the names that are supposed
to stay local
fixes#22786
Trying to get optimization flags for a specific target from
a compiler may trigger warnings. In the context of constructing
facts for the ASP-based solver we don't want to show these
warnings to the user, so here we simply ignore them.
We remove system paths from search variables like PATH and
from -L options because they may contain many packages and
could interfere with Spack-built packages. External packages
may be installed to prefixes that are not actually system paths
but are still "merged" in the sense that many other packages are
installed there. To avoid conflicts, this PR places all external
packages at the end of search paths.
If you install packages using spack install in an environment with
complex spec constraints, and the install fails, you may want to
test out the build using spack build-env; one issue (particularly
if you use concretize: together) is that it may be hard to pass
the appropriate spec that matches what the environment is
attempting to install.
This updates the build-env command to default to pulling a matching
spec from the environment rather than concretizing what the user
provides on the command line independently.
This makes a similar change to spack cd.
If the user-provided spec matches multiple specs in the environment,
then these commands will now report an error and display all
matching specs (to help the user specify).
Co-authored-by: Gregory Becker <becker33@llnl.gov>
fixes#22294
A combination of the swapping order for global variables and
the fact that most of them are lazily evaluated resulted in
custom install tree not being taken into account if clingo
had to be bootstrapped.
This commit fixes that particular issue, but a broader refactor
may be needed to ensure that similar situations won't affect us
in the future.
fixes#22565
This change enforces the uniqueness of the version_weight
atom per node(Package) in the DAG. It does so by applying
FTSE and adding an extra layer of indirection with the
possible_version_weight/2 atom.
Before this change it may have happened that for the same
node two different version_weight/2 were in the answer set,
each of which referred to a different spec with the same
version, and their weights would sum up.
This lead to unexpected result like preferring to build a
new version of an external if the external version was
older.
* clingo: modify recipe for bootstrapping
Modifications:
- clingo builds with shared Python only if ^python+shared
- avoid building the clingo app for bootstrapping
- don't link to libpython when bootstrapping
* Remove option that breaks on linux
* Give more hints for the current Python
* Disable CLINGO_BUILD_PY_SHARED for bootstrapping
* bootstrapping: try to detect the current python from std library
This is much faster than calling external executables
* Fix compatibility with Python 2.6
* Give hints on which compiler and OS to use when bootstrapping
This change hints which compiler to use for bootstrapping clingo
(either GCC or Apple Clang on MacOS). On Cray platforms it also
hints to build for the frontend system, where software is meant
to be installed.
* Use spec_for_current_python to constrain module requirement
(cherry picked from commit d5fa509b07)
* ASP-based solver: avoid adding values to variants when they're set
fixes#22533fixes#21911
Added a rule that prevents any value to slip in a variant when the
variant is set explicitly. This is relevant for multi-valued variants,
in particular for those that have disjoint sets of values.
* Ensure disjoint sets have a clear semantics for external packages
fixes#22547
SingleFileScope was not able to repopulate its cache before this
change. This was affecting the configuration seen by environments
using clingo bootstrapped from sources, since the bootstrapping
operation involved a few cache invalidation for config files.
In most cases, we want condition_holds(ID) to imply any imposed
constraints associated with the ID. However, the dependency relationship
in Spack is special because it's "extra" conditional -- a dependency
*condition* may hold, but we have decided that externals will not have
dependencies, so we need a way to avoid having imposed constraints appear
for nodes that don't exist.
This introduces a new rule that says that constraints are imposed
*unless* we define `do_not_impose(ID)`. This allows rules like
dependencies, which rely on more than just spec conditions, to cancel
imposed constraints.
We add one special case for this: dependencies of externals.
We only consider test dependencies some of the time. Some packages are
*only* test dependencies. Spack's algorithm was previously generating
dependency conditions that could hold, *even* if there was no potential
dependency type.
- [x] change asp.py so that this can't happen -- we now only generate
dependency types for possible dependencies.
This builds on #20638 by unifying all the places in the concretizer where
things are conditional on specs. Previously, we duplicated a common spec
conditional pattern for dependencies, virtual providers, conflicts, and
externals. That was introduced in #20423 and refined in #20507, and
roughly looked as follows.
Given some directives in a package like:
```python
depends_on("foo@1.0+bar", when="@2.0+variant")
provides("mpi@2:", when="@1.9:")
```
We handled the `@2.0+variant` and `@1.9:` parts by generating generated
`dependency_condition()`, `required_dependency_condition()`, and
`imposed_dependency_condition()` facts to trigger rules like this:
```prolog
dependency_conditions_hold(ID, Parent, Dependency) :-
attr(Name, Arg1) : required_dependency_condition(ID, Name, Arg1);
attr(Name, Arg1, Arg2) : required_dependency_condition(ID, Name, Arg1, Arg2);
attr(Name, Arg1, Arg2, Arg3) : required_dependency_condition(ID, Name, Arg1, Arg2, Arg3);
dependency_condition(ID, Parent, Dependency);
node(Parent).
```
And we handled `foo@1.0+bar` and `mpi@2:` parts ("imposed constraints")
like this:
```prolog
attr(Name, Arg1, Arg2) :-
dependency_conditions_hold(ID, Package, Dependency),
imposed_dependency_condition(ID, Name, Arg1, Arg2).
attr(Name, Arg1, Arg2, Arg3) :-
dependency_conditions_hold(ID, Package, Dependency),
imposed_dependency_condition(ID, Name, Arg1, Arg2, Arg3).
```
These rules were repeated with different input predicates for
requirements (e.g., `required_dependency_condition`) and imposed
constraints (e.g., `imposed_dependency_condition`) throughout
`concretize.lp`. In #20638 it got to be a bit confusing, because we used
the same `dependency_condition_holds` predicate to impose constraints on
conditional dependencies and virtual providers. So, even though the
pattern was repeated, some of the conditional rules were conjoined in a
weird way.
Instead of repeating this pattern everywhere, we now have *one* set of
consolidated rules for conditions:
```prolog
condition_holds(ID) :-
condition(ID);
attr(Name, A1) : condition_requirement(ID, Name, A1);
attr(Name, A1, A2) : condition_requirement(ID, Name, A1, A2);
attr(Name, A1, A2, A3) : condition_requirement(ID, Name, A1, A2, A3).
attr(Name, A1) :- condition_holds(ID), imposed_constraint(ID, Name, A1).
attr(Name, A1, A2) :- condition_holds(ID), imposed_constraint(ID, Name, A1, A2).
attr(Name, A1, A2, A3) :- condition_holds(ID), imposed_constraint(ID, Name, A1, A2, A3).
```
this allows us to use `condition(ID)` and `condition_holds(ID)` to
encapsulate the conditional logic on specs in all the scenarios where we
need it. Instead of defining predicates for the requirements and imposed
constraints, we generate the condition inputs with generic facts, and
define predicates to associate the condition ID with a particular
scenario. So, now, the generated facts for a condition look like this:
```prolog
condition(121).
condition_requirement(121,"node","cairo").
condition_requirement(121,"variant_value","cairo","fc","True").
imposed_constraint(121,"version_satisfies","fontconfig","2.10.91:").
dependency_condition(121,"cairo","fontconfig").
dependency_type(121,"build").
dependency_type(121,"link").
```
The requirements and imposed constraints are generic, and we associate
them with their meaning via the id. Here, `dependency_condition(121,
"cairo", "fontconfig")` tells us that condition 121 has to do with the
dependency of `cairo` on `fontconfig`, and the conditional dependency
rules just become:
```prolog
dependency_holds(Package, Dependency, Type) :-
dependency_condition(ID, Package, Dependency),
dependency_type(ID, Type),
condition_holds(ID).
```
Dependencies, virtuals, conflicts, and externals all now use similar
patterns, and the logic for generating condition facts is common to all
of them on the python side, as well. The more specific routines like
`package_dependencies_rules` just call `self.condition(...)` to get an id
and generate requirements and imposed constraints, then they generate
their extra facts with the returned id, like this:
```python
def package_dependencies_rules(self, pkg, tests):
"""Translate 'depends_on' directives into ASP logic."""
for _, conditions in sorted(pkg.dependencies.items()):
for cond, dep in sorted(conditions.items()):
condition_id = self.condition(cond, dep.spec, pkg.name) # create a condition and get its id
self.gen.fact(fn.dependency_condition( # associate specifics about the dependency w/the id
condition_id, pkg.name, dep.spec.name
))
# etc.
```
- [x] unify generation and logic for conditions
- [x] use unified logic for dependencies
- [x] use unified logic for virtuals
- [x] use unified logic for conflicts
- [x] use unified logic for externals
LocalWords: concretizer mpi attr Arg concretize lp cairo fc fontconfig
LocalWords: virtuals def pkg cond dep fn refactor github py
This change accounts for platform specific configuration scopes,
like ~/.spack/linux, during bootstrapping. These scopes were
previously not accounted for and that was causing issues e.g.
when searching for compilers.
(cherry picked from commit 413c422e53)
* Allow the bootstrapping of clingo from sources
Allow python builds with system python as external
for MacOS
* Ensure consistent configuration when bootstrapping clingo
This commit uses context managers to ensure we can
bootstrap clingo using a consistent configuration
regardless of the use case being managed.
* Github actions: test clingo with bootstrapping from sources
* Add command to inspect and clean the bootstrap store
Prevent users to set the install tree root to the bootstrap store
* clingo: documented how to bootstrap from sources
Co-authored-by: Gregory Becker <becker33@llnl.gov>
(cherry picked from commit 10e9e142b7)
Bash has a builtin `fc` that will override the compiler if you use "fc",
so it's better to use the full spack-supplied compiler path.
Additionally, the filter regex in the docs was wrong: it replaced the
entire assignment operation with the RHS.
* py-kubernetes: add new package
* Apply suggestions from code review
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-kubernetes: remove alpha/beta versions, fix dependency types
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This PR updates the abinit package. The underlying build system has
several changes from previous versions, which are reflected in the
package recipe.
- added version 9.4.2
- removed commented out code
- add new libxml2 variant, with dependency and conflicts
- add dependency on atompaw
- depend on fftw-api when ~openmp
This allows other fftw implementations to be used. This PR adds MKL.
- depend on netcdf explicitly
- remove hdf5 variant as hdf5 is required
- only use wannier90 if +mpi as the wannier90 spack package is MPI only
- allow newer versions of libxc for abinit 9
- split configure options for versions before and after abinit 9
- always use MPI compiler wrappers
- add patch to remove march settings for version 9
- Set conflict for fftw~openmp if abinit+openmp
This allows the virtual fftw-api to be used for the dependency. If fftw
is the fftw-api provider then bail if fftw~openmp is set when
abinit+openmp is used.
- Set conflicts for +openmp and mkl
- Be explicit about +mkl for intel-parallel-studio
- Add TODO entry for switching conflicts/depends_on logic
* clingo/clingo-bootstrap: added a package with option for bootstrapping clingo
package builds in Release mode
uses GCC options to link libstdc++ and libgcc statically
* clingo-bootstrap: apple-clang options to bootstrap statically on darwin
* clingo: fix the path of the Python interpreter
In case multiple Python versions are in the same prefix
(e.g. when clingo is built against an external Python),
it may happen that the Python used by CMake does not
match the corresponding node in the current spec.
This is fixed here by defining "Python_EXECUTABLE"
properly as a hint to CMake.
* clingo: the commit for "spack" version has been updated.
Most people installing `clingo` with Spack are going to be doing it to
use the new concretizer, and that requires the `master` branch.
- [x] make `master` the default so we don't have to keep telling people
to install `clingo@master`. We'll update the preferred version when
there's a new release.
* make `spack fetch` work with environments
* previously: `spack fetch` required the explicit statement of
the specs to be fetched, even when in an environment
* now: if there is no spec(s) provided to `spack fetch` we check
if an environment is active and if yes we fetch all
uninstalled specs.
* Update pylint to 2.8.2
* Update var/spack/repos/builtin/packages/py-pylint/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Address comments
* Update
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This solves a few FIXMEs in conftest.py, where
we were manipulating globals and seeing side
effects prior to registering fixtures.
This commit solves the FIXMEs, but introduces
a performance regression on tests that may need
to be investigated
(cherry picked from commit 4558dc06e2)
The context manager can be used to swap the current
configuration temporarily, for any use case that may need it.
(cherry picked from commit 553d37a6d6)
The method is now called "use_repositories" and
makes it clear in the docstring that it accepts
as arguments either Repo objects or paths.
Since there was some duplication between this
contextmanager and "use_repo" in the testing framework,
remove the latter and use spack.repo.use_repositories
across the entire code base.
Make a few adjustment to MockPackageMultiRepo, since it was
stating in the docstring that it was supposed to mock
spack.repo.Repo and was instead mocking spack.repo.RepoPath.
(cherry picked from commit 1a8963b0f4)
There clingo-cffi job has two issues to be solved:
1. It uses the default concretizer
2. It requires a package from https://test.pypi.org/simple/
The former can be fixed by setting the SPACK_TEST_SOLVER
environment variable to "clingo".
The latter though requires clingo-cffi to be pushed to a
more stable package index (since https://test.pypi.org/simple/
is meant as a scratch version of PyPI that can be wiped at
any time).
For the time being run the tests in a container. Switch back to
PyPI whenever a new official version of clingo will be released.
* Support clingo when used with cffi
Clingo recently merged in a new Python module option based on cffi.
Compatibility with this module requires a few changes to spack - it does not automatically convert strings/ints/etc to Symbol and clingo.Symbol.string throws on failure.
manually convert str/int to clingo.Symbol types
catch stringify exceptions
add job for clingo-cffi to Spack CI
switch to potassco-vendored wheel for clingo-cffi CI
on_unsat argument when cffi
(cherry picked from commit 93ed1a410c)
* Improve error message for inconsistencies in package.py
Sometimes directives refer to variants that do not exist.
Make it such that:
1. The name of the variant
2. The name of the package which is supposed to have
such variant
3. The name of the package making this assumption
are all printed in the error message for easier debugging.
* Add unit tests
(cherry picked from commit 7226bd64dc)
The "fact" method before was dealing with multiple facts
registered per call, which was used when we were emitting
grounded rules from knowledge of the problem instance.
Now that the encoding is changed we can simplify the method
to deal only with a single fact per call.
(cherry picked from commit ba42c36f00)
* Modification to R environment
This PR modifies how the R environmnet is presented, and fixes
installing the standalone Rmath library.
- The Rmath build and install methods are combined into one
- Set parallel=False when installing Rmath
- remove the run environment that set up variables for libraries and
headers that are not really needed, and pollute the environment.
* Add setup_run_environment back
- Add back the setup_run_environment with LD_LIBRARY_PATH and
PKG_CONFIG_PATH.
- Adjust documentation to reflect the current code.
The previous `gasnet` spack package was not vetted/approved by the GASNet library maintainers. This one is.
Notably adds build-time testing and smoke-testing.
Convert network variants into a multi-valued `conduits` variant has the minor advantage of enabling a concise `conduits=none` spec, but the major drawback that it degrades the `spack info gasnet` output.
* py-lazyarray: add new version 0.3.2
Change-Id: Ie8a40f3ff1fe7477e27f6085b9ad6673395258b2
* fixup dependencies
Change-Id: I4b2fb7a0abb462f8df74c383c67517065cd95b67
* Update var/spack/repos/builtin/packages/py-lazyarray/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* new package: py-batchspawner
Change-Id: I508bad7ba7f1fc32c2f6c0bfccf35d864cf47ced
* fixup
Change-Id: If183933ce40a8d12214ea24acc683cb046fcfbcb
* fix broken version
Change-Id: Ie4dd8d18465877cd8f9cb862112af37d85b1c30f
* fixup license
Change-Id: I51d92a6d229f6a6b56eea6e53c65ed31fe59f6af
* Update var/spack/repos/builtin/packages/py-batchspawner/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Example replacement:
```
'-D(\w+)(:BOOL)?=\{0\}'\.\s*format\s*\(\s*'(ON|YES|true|TRUE)' if '\+(\w+)' in (self\.)?spec else '(OFF|NO|false|FALSE)'\)
```
with
```
self.define_from_variant('\1', '\4')
```
This will cause failures if any variants were misspelled: I have already caught two packages with nonexistent variants.
Spack uses curl to fetch URL resources. For locally-stored resources
it uses curl's file protocol; when using this protocol, curl expects
that the URL encoding conforms to RFC 3986 (which reserves characters
like '?' and '=' for special use).
We were not performing this encoding, and found a resource where
curl was interpreting this in an unfavorable way (succeeding, but
producing an empty file). This commit properly encodes URLs when
using curl's file protocol.
This error did not likely come up before because in most contexts
Spack was either fetching via http or it was using URLs without
offending characters (for example, the sha-based URLs in mirrors
never contain these characters).
* Add versions 1.9.4 and 1.9.4.1 for cbtf-* packages
* Add versions 2.4.2 and 2.4.2.1 for openspeedshop packages
* Remove older versions
* Switch from generic dependency on elf to a dependency on the
elfutils implementation for cbtf-* and openspeedshop packages
* For llvm-openmp-ompt, relax dependency on libelf to elf (cbtf-krell
now depends on elfutils, and llvm-openmp-ompt, so unless this
dependency is relaxed there would be a conflict)
* Update CMake build_type to support Debug, Release, RelWithDebInfo
in cbtf-* and openspeedshop packages
* Update libmonitor patches when building as a dependency of
cbtf-krell
Pass -ef to the cce fortran compiler, fix the build system to use the correct openmp flag for CCE
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
This also changes the checksum for 1.22.1 because I switched the package
to use the proper upstream tarballs to get rid of the autotools
dependencies. Moreover, a few dependencies were missing. netdata also
requires a few directories to be created in its prefix to actually work.
Spack doesn't require users to manually index their repos; it reindexes the indexes automatically when things change. To determine when to do this, it has to `stat()` all package files in each repository to make sure that indexes up to date with packages. We currently index virtual providers, patches by sha256, and tags on packages.
When this was originally implemented, we ran the checker all the time, at startup, but that was slow (see #7587). But we didn't go far enough -- it still consults the checker and does all the stat operations just to see if a package exists (`Repo.exists()`). That might've been a wash in 2018, but as the number of packages has grown, it's gotten slower -- checking 5k packages is expensive and users see this for small operations. It's a win now to make `Repo.exists()` check files directly.
**Fix:**
This PR does a number of things to speed up `spack load`, `spack info`, and other commands:
- [x] Make `Repo.exists()` check files directly again with `os.path.exists()` (this is the big one)
- [x] Refactor `Spec.satisfies()` so that a checking for virtual packages only happens if needed
(avoids some calls to exists())
- [x] Avoid calling `Repo.exists(spec)` in `Repo.get()`. `Repo.get()` will ultimately try to load
a `package.py` file anyway; we can let the failure to load it indicate that the package doesn't
exist, and avoid another call to exists().
- [x] Fix up some comments in spec parsing
- [x] Call `UnknownPackageError` more consistently in `repo.py`
- [x] `analyze` isn't commonly used; move it to long help
(`spack -H` vs `spack -h`). Give it its own section.
- [x] make it clear from `spack -h` that `spack module` can generate
module files
- [x] shorten help for `spack style`
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment).
This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention.
As part of this change, the module roots configuration moved from the `config` section to inside each module configuration.
Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec.
TODO:
- [x] code changes to support multiple module sets
- [x] code changes to support modules relative to a view
- [x] Tests for multiple module configurations
- [x] Tests for modules relative to a view
- [x] Backwards compatibility for module roots from config section
- [x] Backwards compatibility for default module set without the name specified
- [x] Tests for backwards compatibility
Simplify logic by just enabling or disabling fsync as user specified
(default to off currently). Also remove the 4.1 version check, since
that version isn't actually supported in here.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
The implementation for __str__ has been simplified to traverse the spec directly,
and doesn't call anymore the flat_dependencies method. Dead code has been
removed.
For configure (e.g. for hdf5) to pass, this option needs to be pulled out when invoked in ccld mode.
I thought it had fixed the issue but I still saw it after that. After some digging, my guess is that I was able
to get hdf5 to build with ifort instead of ifx. Lot of overlapping changes occurring at the time, as it were.
There are still outstanding issues building hdf5 with ifx, and Intel is looking into what appears to be a
compiler bug, but this manifests during build and is likely a separate issue.
I have verified that the making the edit in 'ccld' mode removes the -loopopt=0 and enables hdf5 to pass
configure. It should be fine to make the edit in 'ld' mode as well, but I have not tested that and didn't
include an -or- condition for it.
Add new release of SEACAS.
Update netcdf-c version to recent release which fixes some issues that have caused problems in past
Use release version of CGNS instead of develop
* Update Nalu-Wind to remove SuperLU from Trilinos requirement. Also simplify Nalu-Wind package.
* Leave boost option in nalu-wind.
* Add git branches into TPL requirements. Update OpenFAST for change to main branch.
Currently, environment views blink out of existence during the view regeneration, and are slowly built back up to their new and improved state. This is not good if other processes attempt to access the view -- they can see it in an inconsistent state.
This PR fixes makes environment view updates atomic. This requires a level of indirection (via symlink, similar to nix or guix) from the view root to the underlying implementation on the filesystem.
Now, an environment view at `/path/to/foo` is a symlink to `/path/to/._foo/<hash>`, where `<hash>` is a hash of the contents of the view. We construct the view in its content-keyed hash directory, create a new symlink to this directory, and atomically replace the symlink with one to the new view.
This PR has a couple of other benefits:
* It future-proofs environment views so that we can implement rollback.
* It ensures that we don't leave users in an inconsistent state if building a new view fails for some reason.
For background:
* there is no atomic operation in posix that allows for a non-empty directory to be replaced.
* There is an atomic `renameat2` in the linux kernel starting in version 3.15, but many filesystems don't support the system call, including NFS3 and NFS4, which makes it a poor implementation choice for an HPC tool, so we use the symlink approach that others tools like nix and guix have used successfully.
* Added the option to use high performance linkers: gold and lld, for
LBANN. Including them as build flags causes unnecessary propagation
to all dependent packages, reducing package reuse.
fixes#22351
The ASP-based solver now accounts for the presence
in the DAG of deprecated versions and tries to minimize
their number at highest priority.
* gobject-introspection: fix for Python 3.9.
* Fixes the too long line formatting issue.
* gobject-introspection: limits the scope of the patch
Co-authored-by: Robert Mijakovic <robert.mijakovic@lxp.lu>
Variants explicitly set in an abstract root spec are considered
as defaults for the package they refer to, and they override
what is in packages.yaml and in package.py. This is relevant
only for multi-valued variants, where a constraint may extend
an already default value.
* Fixes to flex
- Prefer the version that doesn't need all the patches and extra build
tools
- Make dependency on gettext optional under the nls variant (off by
default)
- Drop the dependency on help2man if we don't have to regenerate the man
pages (when no patches are necessary)
* Bring back gettext dep as it is used during autoconf
The code for guessing cpu archtype based on craype modules names got confused,
at least on LLNL RZ prototype systems. In particular a (L) or (D) at the end of a craype-x86-xxx or other
cpu architecture module was geting the logic confused.
With this patch, any white space + remaining characters in the moduel name are removed.
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
There have been a lot of questions and some confusion recently surrounding Spack installation test capabilities so this PR is intended to clean up and refine the documentation for "Checking an installation".
It aims to better distinguish between checks that are performed during an installation (i.e., build-time tests) and those that can be done days and weeks after the software has been installed (i.e., install (or smoke) tests).
* Enhancing package gmsh to more options, new version
* Enhancing package gmsh, url from https
* Enhancing package gmsh, following reviewer 1
* Improving package gmsh from reviewer
* Adding MED dependency
* Removing env variables and unused dependency (netgen/tetgen)
`flag_handler` currently passes all flags via injection. This makes it
impossible to override the default flags provided by autotools (for
instance, `binutils cflags='-O2'` will still build with `-O2 -g`).
Instead, use injection for our workaround flags and pass other flags to
the build system.
When we first merged the ASP-based solver, unit-tests
were run in a Docker container with root permissions
and that was preventing a few tests to succeed.
Since some time though, clingo is tested as a regular
user within Github Actions VMs, so we should start to
run checks again.
* geos: Fix config issues with python bindings using python3 (#23479)
This should fix some config issues when building geos with python
bindings and using python3 --- the geos configuration scripts had
a few python2-isms.
I only tested (lightly; geos built and I can import geos in python3)
on 3.8.1, but I did check that the patch can at least be applied
in 3.5.
I belatedly discovered that geos dropped all the SWIG bindings
in @3.9, so I also added some conflicts on the +python and +ruby
options to note that they are not supported in 3.9.
* geos: adding omitted patch file
In an active concretize environment, support installing one or more
cli specs only if they are already present in the environment. The
`--no-add` option is the default for root specs, but optional for
dependency specs. I.e. if you `spack install <depspec>` in an
environment, the dependency-only spec `depspec` will be added as a
root of the environment before being installed. In addition,
`spack install --no-add <spec>` fails if it does not find an
unambiguous match for `spec`.
Added the checksum for 4.1.2 and 4.2.0
The `parallel` variant did the exact same behavior as the `mpi` variant, but they had different default values than each other. Both variants set the value of `-DCGNS_ENABLE_PARALLEL`, so it was unclear which variant was "winning" and could definitely result in a non-intuitive build. Did a grep of the spack packages and none of them where using the `parallel` variant to control the cgns options. Retained the `mpi` variant as that one is being used by multiple packages.
One issue that remains to be solved is that the default integer size has changed from 32-bit to 64-bit for the 4.2.0 release. This is controlled by the `int64` variant which currently defaults to `OFF`. There should maybe be some thought about changing the default to match the default of the current release, or maybe having a version-specific default... For now, left the behavior as it has been for previous versions.
The patch available in spack does not patch
cleanly for the 4.1.1 and presumably later releases.
See Open MPI commit b8a8096a3f153380f95af8f285f48e926eb18bf1
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
SILO has optional support for compression libraries that require
C++ (hzip and fpzip). This patch exposes those options as variants
to enable configuration of SILO without the C++ libraries for C
applications. hzip and fpzip are enabled by default to preserve
current behavior.
Like compilers targets now try to minimize
mismatches, instead of maximizing matches.
Deduction of mismatches is reworked to be
the opposite of a match, since computing
that is faster.
The ASP-based solver can natively manage cases where more than one root spec is given, and is able to concretize all the roots together (ensuring one spec per package at most).
Modifications:
- [x] When concretising together an environment the ASP-based solver calls directly its `solve` method rather than constructing a temporary fake root package.
* py-keras: new version
* Adds missing dependencies.
* Removes the newline which is against formatting rules.
* py-keras: limits some dependencies to older versions
* py-keras: restricts dependencies
* pykeras: fixes dependency ranges :)
Co-authored-by: Robert Mijakovic <robert.mijakovic@lrz.de>
Co-authored-by: Robert Mijakovic <robert.mijakovic@lxp.lu>
The loading protocol mandates that the the module we are going
to import needs to be already in sys.modules before its code is
executed, so to prevent unbounded recursions and multiple loading.
Loading a module from file exits early if the module is already
in sys.modules
When installing OneAPI packages as root (e.g. in a container), the
installer places cache files in /var/intel/installercache that
interfere with future Spack installs. This ensures that when
running an installation as a root user that this is removed.
* Adding hip support
* Added new blaspp version and rocm support. Fixed error in mesa18 package.
* Correcting variant name.
* Code style fixes
* Change of name of library
* Change "make check" to correctly run from the build directory.
* Upgraded version to fix testing errors
* Fixed testing directory
* Removed unnecessary variant entry (already inherited from CudaPackage)
* Generalization of version matching logic
* Code style
* Corrected version requirement
SCR moved to a component version some time ago, but never had a
release associated with these changes. SCR v2 is a legacy version
that is no longer being developed/supported. In preparation for an
upcoming SCR v3 release, there is now a 3.0rc1 release available to
users.
This adds the 3.0rc1 release to the spack package and deprecates the
older versions.
Additional changes include:
- Enforce using the main branches of the components when installing
scr@develop
- Enforce SCR v3 uses at least the recently released versions of each
of the components
- Use a simple `detect_scheduler()` function in an attempt to be
smarter about setting the default resource manager and not require
users to always manually provide the variant
- Add/update variants that were recently added to AXL and KVTree
components
- Fix cmake arg naming bug of setting `SCR_CONFIG_FILE`
- `SCR_ASYNC_API` is now being handled by a component and is only
needed by the legacy versions.
* Added checksum for recently released 4.8.0
* Added `enable-fsync` variant. The `fsync` flag was added to the configuration as of version 4.1.0 and later. Originally, it defaulted to `on`, but at version 4.3.0 and later, it was changed to default to `off` and a `enable-fsync` configuration flag was added to enable it.
The spack package has the `--enable-fsync` specified with no way to disable for all builds of netcdf-c 4.1.0 and later. This can cause horrendously slow I/O for certain use cases (e.g. 7 seconds with no-fsync versus 2300 seconds with fsync enabled). With the new variant, the default build behavior matches the default of non-spack netCDF.
* Metall: add version 0.2
* Add Metall v0.3
* Update Metall package to v0.4 and v0.5.
* Metall package: add v0.6
* Metall package: add v0.7
* Metall package: add v0.8 and v0.9
* Add Metall package v0.10
* Metall package: set run_environment METALL_ROOT
* Metall package: removed blanks
* Metall package: add v0.11 and v0.12
* Metall package: change required cmake version
* Metall package: support build test
* Metall package: add v0.13
* Metall package: change to use setup_build_environment
gettext uses a test with <libxml2/libxml/someheader.h> to locate a header,
and libxml2 itself includes <libxml/otherheader.h>, so both have to be
in the include path.
* Building binutils with gold implies building ld
* add +ld to llvm to make the old concretizer happy and add +gas to gcc since that's used in the package.py
* Remove sys
* Metall: add version 0.2
* Add Metall v0.3
* Update Metall package to v0.4 and v0.5.
* Metall package: add v0.6
* Metall package: add v0.7
* Metall package: add v0.8 and v0.9
* Add Metall package v0.10
* Metall package: set run_environment METALL_ROOT
* Metall package: removed blanks
* Metall package: add v0.11 and v0.12
* Metall package: change required cmake version
* qt: update versions and URLs
- Add LTS releases of 5.12.10, 5.9.9, 5.6.3
- Mark other minor versions of 5 as deprecated
- Use https
- The URL for older QT versions changed recently to "new_archive"
- Prefer xz instead of gz for >=5.6 because 5.6.3 isn't available as
gz. This invalidates the SHA of 5.7-5.8.
* mxnet: new version 1.8.0
use submodules on master
introduce constraints on cuda versions supported
handle USE_MKLDNN->USE_ONEDNN conversion
* * use define for USE_CUTENSOR
* fix up dependencies for 2.0.0+
libtirpc puts its header files under prefix/include/tirpc, but
spack was returning just prefix/include for location of headers.
This will cause spack to return both prefix/include and
prefix/include/tirpc for headers, so both
include <rpc/xdr.h>
or
include <tirpc/rpc/xdr.h>
should work.
Help dependents find libraries/headers. Like intel-oneapi-mkl, this
package offers several different versions of libraries that conflict.
This PR chooses one of those versions. When
https://github.com/spack/spack/discussions/22749 is resolved, this
package should be updated to choose which libraries to use.
Previously the tau package got the cxx and cc names from
os.path.basename(self.compiler.cxx), however if the path to the compiler
looks like "/usr/bin/g++-10.2.0" then tau's custom build system doesn't
recognize it. What we want instead is something that looks like "g++"
which is exactly what cxx_names[0] gives us. We already did this for
fortran, so I am not sure why we didn't do it here. Not doing this
causes a build failure when tau tries to use a polyfill (vector.h,
iostream.h) that doesn't seem to be packaged with tau.
Additionally, tau needs some help finding mpi include directories when
building with MPI, so we provide them. Unfortunately, we can't just say
that the compilers are mpicc and mpicxx in the previous fix to have
these things found automatically. This is because tau assumes we always
need the polyfill when the compilers are set to these values which again
causes a build failure.
The function we coded in Spack to load Python modules with arbitrary
names from a file seem to have issues with local imports. For
loading hooks though it is unnecessary to use such functions, since
we don't care to bind a custom name to a module nor we have to load
it from an unknown location.
This PR thus modifies spack.hook in the following ways:
- Use __import__ instead of spack.util.imp.load_source (this
addresses #20005)
- Sync module docstring with all the hooks we have
- Avoid using memoization in a module function
- Marked with a leading underscore all the names that are supposed
to stay local
Complete overhaul of the Legion package to better capture a more
up-to-date set of configuration options and variants. This update
adds additional flexibility and features that were requested by
users.
* Add version 21.03.0 and "stable" branch
* Remove all older numeric versions
* Add support for CUDA, Python, PAPI support and more
* Add maintainer
* This no longer uses the Spack `gasnet` package: it defaults to
using an embedded gasnet or can be pointed to an external
* MUMPS: Use GEMMT BLAS extension when possible.
This should improve the performance and is recommanded by the developers.
* MUMPS: Add a new "openmp" variant.
* MUMPS: Add a "blr_mt" variant.
This improves performance when using OpenMP but might not be compatible with all multithreaded BLAS.
Set the path to javah via the JAVAH environment variable. If it is
a version of java that does not have javah it will fall back to `javac
-h`. Without specifying this the build could pick up a javah from the
system.
- add version 3.4.0
- add patch for bam2wig when version 3.4.0
- url format changed again, hopefully stable now
- added missing python dependency when version >3.3.1
- have older version compile with htslib, samtools ,bcftools
- new dependencies for version 3.4.0
- sqlite
- mysql-client
- mysqlpp
- lp-solve
- suite-sparse
- refactored filtering code
- set python interpreter in scripts
This is as much a question as it is a minor fine-tuning of the docs. I've been known to add things to an environment by editing the `spack.yaml` file directly. When I read the previous version of this sentence, I was afraid that `spack add` was actually doing *two* things, modifying the `spack.yaml` and updating something else that defined the roots of the Environment. A bit of experimentation suggests that editing the `spack.yaml` file is sufficient to change the roots.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This adds that new version to the package, updates the url, and
updates the hash of v0.0.3 for the new url.
This also updates the KVTree dependency as MPI is required to be
enabled in KVTree for er to work.
rankstr is now also required by er for recently added tests.
PR #22864 added a patch to hpctoolkit to fix an issue with gcc 10.x, and the patch was applied to all revs unconditionally. But this was fixed in hpctoolkit master on Aug 11, 2020, so the patch should only apply to old revs.
Fixes#22951.
Update package with 4.1 sha keys.
Use variant to disable openmp in the build of llvm-amdgpu.
Set CPATH, LIBRARY_PATH so that clang knows to look in the rocm-openmp-extras for headers/libraries.
Disable flang warnings as Spack thinks they are errors.
In ROCm 4.1, the plugin changed names from hsa -> amdgpu.
Update HSA_INCLUDE for 4.1.0.
Clingo has been released on PyPI, so there
are no more concerns on our CI depending
on pypy.test for installing the wheel.
Apparently we have parts of Spack which
are not compatible with kcov > 3.4
UnifyFS has been integrated with updated versions of its mochi-margo
dependency (and mochi-margo's mercury and libfabric dependencies).
This removes support for version 0.9.0
fixes#22786
Trying to get optimization flags for a specific target from
a compiler may trigger warnings. In the context of constructing
facts for the ASP-based solver we don't want to show these
warnings to the user, so here we simply ignore them.
These were deprecated when the custom cuda_arch list was
removed. Also fixed up the Aluminum dependencies for Hydrogen and
DiHydrogen. Turns out that Aluminum v0.6.0 didn't have a correct
version in CMake and thus the interaction with older versions of
Hydrogen and DiHydrogen needed to be corrected.
This isn't a significant issue, but I noticed that the docstring incorrectly references "tty.fail" and I wanted to quickly fix it to reflect the correct command, tty.die. I also wanted to fix the docstrings to not be large clumps, to what @tgamblin suggested after I wrote this - having one line at the top that is a quick summary, and more verbose after that.
* New package: py-pymumps
Python bindings for MUMPS, a parallel sparse direct solver
* py-pymumps: fixing flake issues
* py-pymumps: fix dependency types
Following suggestion of @adamjstewart
* Update var/spack/repos/builtin/packages/py-pymumps/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pymumps/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pymumps/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pymumps/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pymumps/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-pymumps/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This has been checked with gcc on ubuntu 16.04, which ships binutils 2.26 by
default, using spack's binutils 2.36. Only the combination +gas and ~ld
seems to trigger this incompatibility with debug symbols (gcc -g -O2
main.c fails with the error in the comment above the conflict)
- Add dependency on eigen package
- Add last version known to work with ROOT 6.16.00. Until recently GenFit lacked
any tagged versions, therefore, we use a commit hash
FFTW:
(1) Condition to ensure Quad precision is not supported in MPI under FFTW base class
AMDFFTW:
(1) Support for debug and quad precision for aocc compiler
(2) Dedicated variant for threads for enabling SMP threads
(3) Restricted simd features to 'sse2', 'avx' and 'avx2'
(4) Removed float simd features
(5) If debug option is enabled, configure option will be appended with --enable-debug option
(6) Condition to ensure amd-fast-planner is supported from 3.0 onwards under amdfftw derived class
(7) New variant amd-fast-planner - This option will reduce the planning time without much tradeoff in the performance. It is supported for single and double precisions.
(8) Removed following flags for amdfftw - '--enable-threads', '--enable-fma' and '--enable-sse'
MDSplus is a set of software tools for data acquisition and storage and
a methodology for management of complex scientific data.
https://www.mdsplus.org
Co-authored-by: Marijn van Vliet <marijn.vanvliet@aalto.fi>
This provides initial support for [spack monitor](https://github.com/spack/spack-monitor), a web application that stores information and analysis about Spack installations. Spack can now contact a monitor server and upload analysis -- even after a build is already done.
Specifically, this adds:
- [x] monitor options for `spack install`
- [x] `spack analyze` command
- [x] hook architecture for analyzers
- [x] separate build logs (in addition to the existing combined log)
- [x] docs for spack analyze
- [x] reworked developer docs, with hook docs
- [x] analyzers for:
- [x] config args
- [x] environment variables
- [x] installed files
- [x] libabigail
There is a lot more information in the docs contained in this PR, so consult those for full details on this feature.
Additional tests will be added in a future PR.
In debug mode, processes taking an exclusive lock write out their node name to
the lock file. We were using `getfqdn()` for this, but it seems to produce
inconsistent results when used from within some github actions containers.
We get this error because getfqdn() seems to return a short name in one place
and a fully qualified name in another:
```
File "/home/runner/work/spack/spack/lib/spack/spack/test/llnl/util/lock.py", line 1211, in p1
assert lock.host == self.host
AssertionError: assert 'fv-az290-764....cloudapp.net' == 'fv-az290-764'
- fv-az290-764.internal.cloudapp.net
+ fv-az290-764
!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!
== 1 failed, 2547 passed, 7 skipped, 22 xfailed, 2 xpassed in 1238.67 seconds ==
```
This seems to stem from https://bugs.python.org/issue5004.
We don't really need to get a fully qualified hostname for debugging, so use
`gethostname()` because its results are more consistent. This seems to fix the
issue.
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
New version has new dependencies (which are also added here as new
packages):
* perl-mce
* perl-threads
* perl-thread-queue
The new version of genemark-et also has a different URL scheme.
* Add a +gui variant (default off) which adds dependencies on
qt, paraview, and qwt
* Backport upstream patch when installing version 8.4 (this patch
is already applied for versions >= 9.0)
Both binary packages would otherwise require X11 and Mesa libraries to
be installed on the host to run. Make sure they use the Spack-provided
libraries by patching the `rpath` via `patchelf`.
* Clarify stub compiler definition in compilers.yaml
* Update explanation of why stub compiler definition is needed
* Add note about required module definition when using Spack-installed
intel-parallel-studio as intel-compiler
* Add suggestion about updating package config preferences based on
choice of variants when installing intel-parallel-studio to avoid
reinstallation
on multilib distros with lib/lib64 (rather than lib32/lib) the library ends up in a dir lib64/ instead of lib/, breaking the libs property (and the cp2k+spglib build)
We remove system paths from search variables like PATH and
from -L options because they may contain many packages and
could interfere with Spack-built packages. External packages
may be installed to prefixes that are not actually system paths
but are still "merged" in the sense that many other packages are
installed there. To avoid conflicts, this PR places all external
packages at the end of search paths.
See #17270.
```
make[2]: Entering directory `/tmp/vavolkl/spack-stage/spack-stage-qt-5.14.2-63dapppjbq6vqh3le7pazsprijls7cfl/spack-src/qtwebengine/src'
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `echo Modules will not be built. Python version 2 (2.7.5 or later) is required to build QtWebEngine.'
make[2]: *** [errorbuild] Error 1
```
We set LC_ALL=C to encourage a build process to generate ASCII
output (so our logger daemon can decode it). Most packages
respect this but it appears that intel-oneapi-compilers does
not in some cases (see #22813). This reads the output of the build
process as UTF-8, which still works if the build process respects
LC_ALL=C but also works if the process generates UTF-8 output.
For Python >= 3.7 all files are opened with UTF-8 encoding by
default. Python 2 does not support the encoding argument on
'open', so to support Python 2 the files would have to be
opened in byte mode and explicitly decoded (as a side note,
this would be the only way to handle other encodings without
being informed of them in advance).
* bugfix: fix representation of null in spack_yaml output
Nulls were previously printed differently by `spack config blame config`
and `spack config get config`. Fix this in the `spack_yaml` dumpers.
* bugfix: `spack config blame` should print all lines of config
`spack config blame` was not printing all lines of configuration because
there were no annotations for empty lines in the YAML dump output. Fix
this by removing empty lines.
Fixed previously unspecified python dependency and ensured that spack's
python is what exodus@v2016 uses. Also, in the process, identified many
missing versions
* new package: gatetools
This PR adds the gatetools package and dependencies. The gatetools
package is a set of command line tools for gate. Since it is primarily a
CLI, although python modules can be loaded, it is named gatetools as
opposed to py-gatetools.
* Fix quote characterss to avoid test error
* Found another UTF8 character that was tripping up tests
* Another UTF-8 character to replace
* Remove py-python-box dependency and package file
* Make numpy a variant
- py-setuptools needs to be a run dependendency
This was masked by py-numpy having py-setuptools as a run dependency.
* Add missing build depency on py-pytest-runner
- set constraint for geant4 to version 10.6 as gate does not work with
geant-10.7+
- set GATE_USE_ITK: Although RTK is built under ITK, there are some ITK
macros that need to be set explicitly.
As pointed out in https://github.com/STEllAR-GROUP/hpx/issues/5239,
there is an issues in OTF2 <=2.2 where a variable is not properly
initialized. As currently no release of OTF2 is available fixing this,
the patch should be applied.
* [py-scikit-image] Added py-setuptools back into depends_on. Otherwise it is putting skimage in scikit_image-version-pyX.Y-arch.egg dir under site-packages
* [py-scikit-image] Added latest version
* [py-scikit-image] Added py-numpy version dependency when package version greater than 0.18
* [py-scikit-image] Updates to python dependency
* Fix typo
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
- Use debugoptimized as default build type, just like RelWithDebInfo for cmake
- Do not strip by default, and add a default_library variant which conveniently support both shared and static
- Add a maintainer
- Help libtool to find the correct paths to libraries
- Handle externals from system directories
- Enable eccodes for older versions
* The fltk package can build libraries with opengl support. By default, the configure script looks for opengl headers in the sytem include paths. If 'devel' packages have not been installed on the system it omits the 'ftlk_gl.so' library. This can break packages like 'octave' which expects 'fltk' to have opengl support and looks for the library 'fltk_gl'.
Make opengl support explicit in fltk by adding a dependency on 'gl' and adding a new variant of the same name 'gl' (default On).
With these modifications 'fltk_gl' and 'octave' build successfully on CentOS8.
The default behavior is to always enable opengl.
https://www.fltk.org/doc-1.3/intro.html
* Add patch for latest hwloc@:1 to locate ncurses
This way we don't have to depend on ncurses~termlib, which may run into
issues when another package explicitly depends on ncurses+termlib
* Move termcap to the back, cause it's a system symlink on macos and isn't set by spack
- add new version, 4.09.1
- use github url
- convert to autotools package
- deprecate version 4.07b: This version requires manual download and is
a binary only installation.
- version 4.0.7 was not building
- version 4.0.9 was not setting search correctly due to an extra "return"
in config
- added version 4.1.2-p1
- new version needs py-h5py
- new version does not need utf8 patch
- url format changed
Add a conflict for CUDA and shared libraries in Ascent.
The new concretizer will automatically change the default for
Ascent in that case. Until then, dependencies like WarpX need
to hint the `~shared` wish explicitly.
This initial package recipe uses a custom-built wrapper to drive an internal CMake file. Since nekRS also includes built-in copies of several dependencies such as BLAS and HYPRE, it cannot be linked with other such dependencies. However, to work with the `ceed` metapackage, we cannot add `^blas` conflicts to nekRS.
See https://github.com/spack/spack/pull/22519 for discussion.
By default, clingo doesn't show any optimization criteria (maximized or
minimized sums) if the set they aggregate is empty. Per the clingo
mailing list, we can get around that by adding, e.g.:
```
#minimize{ 0@2 : #true }.
```
for the 2nd criterion. This forces clingo to print out the criterion but
does not affect the optimization.
This PR adds directives as above for all of our optimization criteria, as
well as facts with descriptions of each criterion,like this:
```
opt_criterion(2, "number of non-default variants")
```
We use facts in `concretize.lp` rather than hard-coding these in `asp.py`
so that the names can be maintained in the same place as the other
optimization criteria.
The now-displayed weights and the names are used to display optimization
output like this:
```console
(spackle):solver> spack solve --show opt zlib
==> Best of 0 answers.
==> Optimization Criteria:
Priority Criterion Value
1 version weight 0
2 number of non-default variants (roots) 0
3 multi-valued variants + preferred providers for roots 0
4 number of non-default variants (non-roots) 0
5 number of non-default providers (non-roots) 0
6 count of non-root multi-valued variants 0
7 compiler matches + number of nodes 1
8 version badness 0
9 non-preferred compilers 0
10 target matches 0
11 non-preferred targets 0
zlib@1.2.11%apple-clang@12.0.0+optimize+pic+shared arch=darwin-catalina-skylake
```
Note that this is all hidden behind a `--show opt` option to `spack
solve`. Optimization weights are no longer shown by default, but you can
at least inspect them and more easily understand what is going on.
- [x] always show optimization criteria in `clingo` output
- [x] add `opt_criterion()` facts for all optimizationc criteria
- [x] make display of opt criteria optional in `spack solve`
- [x] rework how optimization criteria are displayed, and add a `--show opt`
optiong to `spack solve`
CachedCMakePackage is a CMakePackage subclass for using CMake initial
cache. This feature of CMake allows packages to increase reproducibility,
especially between spack builds and manual builds. It also allows
packages to sidestep certain parsing bugs in extremely long cmake
commands, and to avoid system limits on the length of the command line.
Co-authored by: Chris White <white238@llnl.gov>
* Add patch for Intel C++ compiler
- On some machines (in particular MacOSX Catalina), the icpc in some way
utilizes the preprocessor of the associated "developer tools" used by
icpc. This leads to, in some cases, a preprocessor claiming support for
__tuple_element_packs, even though icpc (as of version 21.1) can't
actually parse such code. Just use the MPARK_TUPLE_ELEMENT_PACK impl
with __icc until icpc supports it, to avoid issues with developer tools
that are untested.
- The same patch has been PRed against mpark-variant
In the face of two consecutive spaces in the command line, the compiler wrapper would skip all remaining arguments, causing problems building py-scipy with Intel compiler. This PR solves the problem.
* Fixed compiler wrapper in the face of extra spaces between arguments
Co-authored-by: Elizabeth Fischer <elizabeth.fischer@alaska.edu>
Backwards incompatible cleanup to target single-tarball-per-arch builds
going forwards.
* Replace per-distro versions with new per-arch builds, and add
url_for_version to avoid specifying per tarball.
* Customise environment setup to avoid adding lib to LD_LIBRARY_PATH.
* Update homepage and licensing URLs.
* Avoid shell interpretation when running textinstall.sh.
* Added NickRF as maintainer.
Use `conflicts` directive whenever possible.
This allows failing early when conflicting variants are used.
Do not silently ignore `+parmetis` variant when `~metis`.
Instead throw an error during concretization.
Simplify the "Makefile.inc" generation.
This will make easier to add new variants in the future.
* Added version patch for 1.4.0 tag on mpark-variant
Redirected urls to git and github tags.
* Updated to commit hashes
* Update var/spack/repos/builtin/packages/mpark-variant/package.py
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
* Update var/spack/repos/builtin/packages/mpark-variant/package.py
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
* Update var/spack/repos/builtin/packages/mpark-variant/package.py
Co-authored-by: Anthony J Zukaitis <zukaitis@lanl.gov>
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
Original commit message:
This feature of CMake allows packages to increase reproducibility, especially between
Spack- and manual builds. It also allows packages to sidestep certain parsing bugs in
extremely long ``cmake`` commands, and to avoid system limits on the length of the
command line.
Adding:
Co-authored by: Chris White <white238@llnl.gov>
This reverts commit c4f0a3cf6c.
CachedCMakePackage is a specialized class for packages built using CMake initial cache.
This feature of CMake allows packages to increase reproducibility, especially between
Spack- and manual builds. It also allows packages to sidestep certain parsing bugs in
extremely long ``cmake`` commands, and to avoid system limits on the length of the
command line.
Autoconf before 2.70 will erroneously pass ifx's -loopopt argument to the
linker, requiring all packages to use autoconf 2.70 or newer to use ifx.
This is a hotfix enabling ifx to be used in Spack. Instead of bothering
to upgrade autoconf for every package, we'll just strip out the
problematic flag if we're in `ld` mode.
- [x] Add a conditional to the `cc` wrapper to skip `-loopopt` in `ld`
mode. This can probably be generalized in the future to strip more
things (e.g., via an environment variable we can constrol from
Spack) but it's good enough for now.
- [x] Add a test ensuring that `-loopopt` arguments are stripped in link
mode, but not in compile mode.
Since `lazy_lexicographic_ordering` handles `None` comparison for us, we
don't need to adjust the spec comparators to return empty strings or
other type-specific empty types. We can just leverage the None-awareness
of `lazy_lexicographic_ordering`.
- [x] remove "or ''" from `_cmp_iter` in `Spec`
- [x] remove setting of `self.namespace` to `''` in `MockPackage`
We have been using the `@llnl.util.lang.key_ordering` decorator for specs
and most of their components. This leverages the fact that in Python,
tuple comparison is lexicographic. It allows you to implement a
`_cmp_key` method on your class, and have `__eq__`, `__lt__`, etc.
implemented automatically using that key. For example, you might use
tuple keys to implement comparison, e.g.:
```python
class Widget:
# author implements this
def _cmp_key(self):
return (
self.a,
self.b,
(self.c, self.d),
self.e
)
# operators are generated by @key_ordering
def __eq__(self, other):
return self._cmp_key() == other._cmp_key()
def __lt__(self):
return self._cmp_key() < other._cmp_key()
# etc.
```
The issue there for simple comparators is that we have to bulid the
tuples *and* we have to generate all the values in them up front. When
implementing comparisons for large data structures, this can be costly.
This PR replaces `@key_ordering` with a new decorator,
`@lazy_lexicographic_ordering`. Lazy lexicographic comparison maps the
tuple comparison shown above to generator functions. Instead of comparing
based on pre-constructed tuple keys, users of this decorator can compare
using elements from a generator. So, you'd write:
```python
@lazy_lexicographic_ordering
class Widget:
def _cmp_iter(self):
yield a
yield b
def cd_fun():
yield c
yield d
yield cd_fun
yield e
# operators are added by decorator (but are a bit more complex)
There are no tuples that have to be pre-constructed, and the generator
does not have to complete. Instead of tuples, we simply make functions
that lazily yield what would've been in the tuple. If a yielded value is
a `callable`, the comparison functions will call it and recursively
compar it. The comparator just walks the data structure like you'd expect
it to.
The ``@lazy_lexicographic_ordering`` decorator handles the details of
implementing comparison operators, and the ``Widget`` implementor only
has to worry about writing ``_cmp_iter``, and making sure the elements in
it are also comparable.
Using this PR shaves another 1.5 sec off the runtime of `spack buildcache
list`, and it also speeds up Spec comparison by about 30%. The runtime
improvement comes mostly from *not* calling `hash()` `_cmp_iter()`.
* New package py-argh
* Fixed deps
* Changed setuptools type
* Update var/spack/repos/builtin/packages/py-argh/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Make -j flag less exceptional
The -j flag in spack behaves differently from make, ctest, ninja, etc,
because it caps the number of jobs to an arbitrary number 16.
Spack will behave like other tools if `spack install` uses a reasonable
default, and `spack install -j <num>` *overrides* that default.
This will be particularly useful for Spack usage outside of a traditional
HPC context and for HPC centers that encourage users to compile on
login nodes with many cores instead of on compute nodes, which has
become increasingly common as individual nodes have more cores.
This maintains the existing default value of min(num_cpus, 16). However,
as it is right now, Spack does a poor job at determining the number of
cpus on linux, since it doesn't take cgroups into account. This is
particularly problematic when using distributed builds with slurm. This PR
also introduces `spack.util.cpus.cpus_available()` to consolidate
knowledge on determining the number of available cores, and improves
core detection for linux. This should also improve core detection for Docker/
Kubernetes, which also use cgroups.
This commit extends the API of the __call__ method of the
SpackCommand class to permit passing global arguments
like those interposed between the main "spack" command
and the subsequent subcommand.
The functionality is used to fix an issue where running
```spack -e . location -b some_package```
ends up printing the name of the environment instead of
the build directory of the package, because the location arg
parser also stores this value as `arg.env`.
fixes#22294
A combination of the swapping order for global variables and
the fact that most of them are lazily evaluated resulted in
custom install tree not being taken into account if clingo
had to be bootstrapped.
This commit fixes that particular issue, but a broader refactor
may be needed to ensure that similar situations won't affect us
in the future.
* Fixed a bug in the DiHydrogen package where the variant legacy was
changed to distconv and wasn't fully propagated. Cleaned up the
openmp variants on the blas library packages in DiHydrogen and
Elemental. Extended support for Aluminum v1.0 in LBANN, Hydrogen, and
DiHydrogen. Fixed a when clause in the LBANN dependencies.
* Removed the upper range limit for the Aluminum library dependence
* Update var/spack/repos/builtin/packages/dihydrogen/package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Remote buildcache indices need to be stored in a place that does not
require writing to the Spack prefix. Move them from the install_tree to
the misc_cache.
fixes#22565
This change enforces the uniqueness of the version_weight
atom per node(Package) in the DAG. It does so by applying
FTSE and adding an extra layer of indirection with the
possible_version_weight/2 atom.
Before this change it may have happened that for the same
node two different version_weight/2 were in the answer set,
each of which referred to a different spec with the same
version, and their weights would sum up.
This lead to unexpected result like preferring to build a
new version of an external if the external version was
older.
* Make stage use concrete specs from environment
Same as in https://github.com/spack/spack/pull/21642, the idea is that
we want to easily stage a package that fails to build in a complex
environment. Instead of making the user create a spec by hand (basically
transforming all the rules in the environment manifest into a spec,
defying the purpose of the environment...), use the provided spec as a
filter for the already concretized specs. This also speeds up things,
cause we don't have to reconcretize.
This adds MPICC=/path/to/intel-oneapi/mpicc etc to he dependents build stage enabling the use of the compiler wrappers.
Co-authored-by: Stephen Sachs <stesachs@amazon.com>
* clingo: modify recipe for bootstrapping
Modifications:
- clingo builds with shared Python only if ^python+shared
- avoid building the clingo app for bootstrapping
- don't link to libpython when bootstrapping
* Remove option that breaks on linux
* Give more hints for the current Python
* Disable CLINGO_BUILD_PY_SHARED for bootstrapping
* bootstrapping: try to detect the current python from std library
This is much faster than calling external executables
* Fix compatibility with Python 2.6
* Give hints on which compiler and OS to use when bootstrapping
This change hints which compiler to use for bootstrapping clingo
(either GCC or Apple Clang on MacOS). On Cray platforms it also
hints to build for the frontend system, where software is meant
to be installed.
* Use spec_for_current_python to constrain module requirement
* ASP-based solver: avoid adding values to variants when they're set
fixes#22533fixes#21911
Added a rule that prevents any value to slip in a variant when the
variant is set explicitly. This is relevant for multi-valued variants,
in particular for those that have disjoint sets of values.
* Ensure disjoint sets have a clear semantics for external packages
fixes#22547
SingleFileScope was not able to repopulate its cache before this
change. This was affecting the configuration seen by environments
using clingo bootstrapped from sources, since the bootstrapping
operation involved a few cache invalidation for config files.
* py-dask-glm: Push again for testing with git.
* py-dask-glm: Fixed the pointed out OSS dependency setting to type=build.
* py-dask-glm: Set depends_on to type=build in the OSS to be built when building the document.
* py-dask-glm: Fix type of depends_on (py-scikit-learn)
Co-authored-by: miura <miura@fx7-pg01.cm.cluster>
about: Report a bug in the core of Spack (command not working as expected, etc.)
labels: "bug,triage"
---
<!-- Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
Example: "I ran `spack find` to list all the installed packages and ..." -->
### Steps to reproduce the issue
```console
$ spack <command1> <spec>
$ spack <command2> <spec>
...
```
### Error Message
<!-- If Spack reported an error, provide the error message. If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect. -->
```console
$ spack --debug --stacktrace <command>
```
### Information on your system
<!-- Please include the output of `spack debug report` -->
<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->
### Additional information
<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [ ] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [ ] I have searched the issues of this repo and believe this is not a duplicate
- [ ] I have run the failing commands in debug mode and reported the output
<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!
If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.
Other than that, thanks for taking the time to contribute to Spack! -->
description:Report a bug in the core of Spack (command not working as expected, etc.)
labels:[bug, triage]
body:
- type:textarea
id:reproduce
attributes:
label:Steps to reproduce
description:|
Explain, in a clear and concise way, the command you ran and the result you were trying to achieve.
Example: "I ran `spack find` to list all the installed packages and ..."
placeholder:|
```console
$ spack <command1> <spec>
$ spack <command2> <spec>
...
```
validations:
required:true
- type:textarea
id:error
attributes:
label:Error message
description:|
If Spack reported an error, provide the error message. If it did not report an error but the output appears incorrect, provide the incorrect output. If there was no error message and no output but the result is incorrect, describe how it does not match what you expect.
placeholder:|
```console
$ spack --debug --stacktrace <command>
```
- type:textarea
id:information
attributes:
label:Information on your system
description:Please include the output of `spack debug report`
validations:
required:true
- type:markdown
attributes:
value:|
If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well.
- type:checkboxes
id:checks
attributes:
label:General information
options:
- label:I have run `spack debug report` and reported the version of Spack/Python/Platform
required:true
- label:I have searched the issues of this repo and believe this is not a duplicate
required:true
- label:I have run the failing commands in debug mode and reported the output
required:true
- type:markdown
attributes:
value:|
We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!
If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on [our Slack](https://slack.spack.io/) first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.
Other than that, thanks for taking the time to contribute to Spack!
about: Some package in Spack didn't build correctly
title: "Installation issue: "
labels: "build-error"
---
<!-- Thanks for taking the time to report this build failure. To proceed with the report please:
1. Title the issue "Installation issue: <name-of-the-package>".
2. Provide the information required below.
We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively! -->
### Steps to reproduce the issue
<!-- Fill in the exact spec you are trying to build and the relevant part of the error message -->
```console
$ spack install <spec>
...
```
### Information on your system
<!-- Please include the output of `spack debug report` -->
<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->
### Additional information
<!-- Please upload the following files. They should be present in the stage directory of the failing build. Also upload any config.log or similar file if one exists. -->
* [spack-build-out.txt]()
* [spack-build-env.txt]()
<!-- Some packages have maintainers who have volunteered to debug build failures. Run `spack maintainers <name-of-the-package>` and @mention them here if they exist. -->
### General information
<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [ ] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [ ] I have run `spack maintainers <name-of-the-package>` and @mentioned any maintainers
- [ ] I have uploaded the build log and environment files
- [ ] I have searched the issues of this repo and believe this is not a duplicate
description:Some package in Spack didn't build correctly
title:"Installation issue: "
labels:[build-error]
body:
- type:markdown
attributes:
value:|
Thanks for taking the time to report this build failure. To proceed with the report please:
1. Title the issue `Installation issue: <name-of-the-package>`.
2. Provide the information required below.
We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!
- type:textarea
id:reproduce
attributes:
label:Steps to reproduce the issue
description:|
Fill in the console output from the exact spec you are trying to build.
value:|
```console
$ spack spec -I <spec>
...
```
- type:textarea
id:error
attributes:
label:Error message
description:|
Please post the error message from spack inside the `<details>` tag below:
value:|
<details><summary>Error message</summary><pre>
...
</pre></details>
validations:
required:true
- type:textarea
id:information
attributes:
label:Information on your system
description:Please include the output of `spack debug report`.
validations:
required:true
- type:markdown
attributes:
value:|
If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well.
- type:textarea
id:additional_information
attributes:
label:Additional information
description:|
Please upload the following files:
* **`spack-build-out.txt`**
* **`spack-build-env.txt`**
They should be present in the stage directory of the failing build. Also upload any `config.log` or similar file if one exists.
- type:markdown
attributes:
value:|
Some packages have maintainers who have volunteered to debug build failures. Run `spack maintainers <name-of-the-package>` and **@mention** them here if they exist.
- type:checkboxes
id:checks
attributes:
label:General information
options:
- label:I have run `spack debug report` and reported the version of Spack/Python/Platform
required:true
- label:I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
required:true
- label:I have uploaded the build log and environment files
required:true
- label:I have searched the issues of this repo and believe this is not a duplicate
about: Suggest adding a feature that is not yet in Spack
labels: feature
---
<!--*Please add a concise summary of your suggestion here.*-->
### Rationale
<!--*Is your feature request related to a problem? Please describe it!*-->
### Description
<!--*Describe the solution you'd like and the alternatives you have considered.*-->
### Additional information
<!--*Add any other context about the feature request here.*-->
### General information
- [ ] I have run `spack --version` and reported the version of Spack
- [ ] I have searched the issues of this repo and believe this is not a duplicate
<!--If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.
Other than that, thanks for taking the time to contribute to Spack!
description:Suggest adding a feature that is not yet in Spack
labels:[feature]
body:
- type:textarea
id:summary
attributes:
label:Summary
description:Please add a concise summary of your suggestion here.
validations:
required:true
- type:textarea
id:rationale
attributes:
label:Rationale
description:Is your feature request related to a problem? Please describe it!
- type:textarea
id:description
attributes:
label:Description
description:Describe the solution you'd like and the alternatives you have considered.
- type:textarea
id:additional_information
attributes:
label:Additional information
description:Add any other context about the feature request here.
- type:checkboxes
id:checks
attributes:
label:General information
options:
- label:I have run `spack --version` and reported the version of Spack
required:true
- label:I have searched the issues of this repo and believe this is not a duplicate
required:true
- type:markdown
attributes:
value:|
If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on [our Slack](https://slack.spack.io/) first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.
Other than that, thanks for taking the time to contribute to Spack!
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
