diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d17e49d596..4f18c386cb7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,365 @@ +# v0.23.0 (2024-11-13) + +`v0.23.0` is a major feature release. + +We are planning to make this the last major release before Spack `v1.0` +in June 2025. Alongside `v0.23`, we will be making pre-releases (alpha, +beta, etc.) of `v1.0`, and we encourage users to try them and send us +feedback, either on GitHub or on Slack. You can track the road to +`v1.0` here: + + * https://github.com/spack/spack/releases + * https://github.com/spack/spack/discussions/30634 + +## Features in this Release + +1. **Language virtuals** + + Your packages can now explicitly depend on the languages they require. + Historically, Spack has considered C, C++, and Fortran compiler + dependencies to be implicit. In `v0.23`, you should ensure that + new packages add relevant C, C++, and Fortran dependencies like this: + + ```python + depends_on("c", type="build") + depends_on("cxx", type="build") + depends_on("fortran", type="build") + ``` + + We encourage you to add these annotations to your packages now, to prepare + for Spack `v1.0.0`. In `v1.0.0`, these annotations will be necessary for + your package to use C, C++, and Fortran compilers. Note that you should + *not* add language dependencies to packages that don't need them, e.g., + pure python packages. + + We have already auto-generated these dependencies for packages in the + `builtin` repository (see #45217), based on the types of source files + present in each package's source code. We *may* have added too many or too + few language dependencies, so please submit pull requests to correct + packages if you find that the language dependencies are incorrect. + + Note that we have also backported support for these dependencies to + `v0.21.3` and `v0.22.2`, to make all of them forward-compatible with + `v0.23`. This should allow you to move easily between older and newer Spack + releases without breaking your packages. + +2. **Spec splicing** + + We are working to make binary installation more seamless in Spack. `v0.23` + introduces "splicing", which allows users to deploy binaries using local, + optimized versions of a binary interface, even if they were not built with + that interface. For example, this would allow you to build binaries in the + cloud using `mpich` and install them on a system using a local, optimized + version of `mvapich2` *without rebuilding*. Spack preserves full provenance + for the installed packages and knows that they were built one way but + deployed another. + + Our intent is to leverage this across many key HPC binary packages, + e.g. MPI, CUDA, ROCm, and libfabric. + + Fundamentally, splicing allows Spack to redeploy an existing spec with + different dependencies than how it was built. There are two interfaces to + splicing. + + a. Explicit Splicing + + #39136 introduced the explicit splicing interface. In the + concretizer config, you can specify a target spec and a replacement + by hash. + + ```yaml + concretizer: + splice: + explicit: + - target: mpi + replacement: mpich/abcdef + ``` + + Here, every installation that would normally use the target spec will + instead use its replacement. Above, any spec using *any* `mpi` will be + spliced to depend on the specific `mpich` installation requested. This + *can* go wrong if you try to replace something built with, e.g., + `openmpi` with `mpich`, and it is on the user to ensure ABI + compatibility between target and replacement specs. This currently + requires some expertise to use, but it will allow users to reuse the + binaries they create across more machines and environments. + + b. Automatic Splicing (experimental) + + #46729 introduced automatic splicing. In the concretizer config, enable + automatic splicing: + + ```yaml + concretizer: + splice: + automatic: true + ``` + + or run: + + ```console + spack config add concretizer:splice:automatic:true + ``` + + The concretizer will select splices for ABI compatibility to maximize + package reuse. Packages can denote ABI compatibility using the + `can_splice` directive. No packages in Spack yet use this directive, so + if you want to use this feature you will need to add `can_splice` + annotations to your packages. We are working on ways to add more ABI + compatibility information to the Spack package repository, and this + directive may change in the future. + + See the documentation for more details: + * https://spack.readthedocs.io/en/latest/build_settings.html#splicing + * https://spack.readthedocs.io/en/latest/packaging_guide.html#specifying-abi-compatibility + +3. Broader variant propagation + + Since #42931, you can specify propagated variants like `hdf5 + build_type==RelWithDebInfo` or `trilinos ++openmp` to propagate a variant + to all dependencies for which it is relevant. This is valid *even* if the + variant does not exist on the package or its dependencies. + + See https://spack.readthedocs.io/en/latest/basic_usage.html#variants. + +4. Query specs by namespace + + #45416 allows a package's namespace (indicating the repository it came from) + to be treated like a variant. You can request packages from particular repos + like this: + + ```console + spack find zlib namespace=builtin + spack find zlib namespace=myrepo + ``` + + Previously, the spec syntax only allowed namespaces to be prefixes of spec + names, e.g. `builtin.zlib`. The previous syntax still works. + +5. `spack spec` respects environment settings and `unify:true` + + `spack spec` did not previously respect environment lockfiles or + unification settings, which made it difficult to see exactly how a spec + would concretize within an environment. Now it does, so the output you get + with `spack spec` will be *the same* as what your environment will + concretize to when you run `spack concretize`. Similarly, if you provide + multiple specs on the command line with `spack spec`, it will concretize + them together if `unify:true` is set. + + See #47556 and #44843. + +6. Less noisy `spack spec` output + + `spack spec` previously showed output like this: + + ```console + > spack spec /v5fn6xo + Input spec + -------------------------------- + - /v5fn6xo + + Concretized + -------------------------------- + [+] openssl@3.3.1%apple-clang@16.0.0~docs+shared arch=darwin-sequoia-m1 + ... + ``` + + But the input spec is redundant, and we know we run `spack spec` to concretize + the input spec. `spack spec` now *only* shows the concretized spec. See #47574. + +7. Better output for `spack find -c` + + In an environmnet, `spack find -c` lets you search the concretized, but not + yet installed, specs, just as you would the installed ones. As with `spack + spec`, this should make it easier for you to see what *will* be built + before building and installing it. See #44713. + +8. `spack -C `: use an environment's configuration without activation + + Spack environments allow you to associate: + 1. a set of (possibly concretized) specs, and + 2. configuration + + When you activate an environment, you're using both of these. Previously, we + supported: + * `spack -e ` to run spack in the context of a specific environment, and + * `spack -C ` to run spack using a directory with configuration files. + + You can now also pass an environment to `spack -C` to use *only* the environment's + configuration, but not the specs or lockfile. See #45046. + +## New commands, options, and directives + +* The new `spack env track` command (#41897) takes a non-managed Spack + environment and adds a symlink to Spack's `$environments_root` directory, so + that it will be included for reference counting for commands like `spack + uninstall` and `spack gc`. If you use free-standing directory environments, + this is useful for preventing Spack from removing things required by your + environments. You can undo this tracking with the `spack env untrack` + command. + +* Add `-t` short option for `spack --backtrace` (#47227) + + `spack -d / --debug` enables backtraces on error, but it can be very + verbose, and sometimes you just want the backtrace. `spack -t / --backtrace` + provides that option. + +* `gc`: restrict to specific specs (#46790) + + If you only want to garbage-collect specific packages, you can now provide + them on the command line. This gives users finer-grained control over what + is uninstalled. + +* oci buildcaches now support `--only=package`. You can now push *just* a + package and not its dependencies to an OCI registry. This allows dependents + of non-redistributable specs to be stored in OCI registries without an + error. See #45775. + +## Notable refactors +* Variants are now fully conditional + + The `variants` dictionary on packages was previously keyed by variant name, + and allowed only one definition of any given variant. Spack is now smart + enough to understand that variants may have different values and defaults + for different versions. For example, `warpx` prior to `23.06` only supported + builds for one dimensionality, and newer `warpx` versions could be built + with support for many different dimensions: + + ```python + variant( + "dims", + default="3", + values=("1", "2", "3", "rz"), + multi=False, + description="Number of spatial dimensions", + when="@:23.05", + ) + variant( + "dims", + default="1,2,rz,3", + values=("1", "2", "3", "rz"), + multi=True, + description="Number of spatial dimensions", + when="@23.06:", + ) + ``` + + Previously, the default for the old version of `warpx` was not respected and + had to be specified manually. Now, Spack will select the right variant + definition for each version at concretization time. This allows variants to + evolve more smoothly over time. See #44425 for details. + +## Highlighted bugfixes + +1. Externals no longer override the preferred provider (#45025). + + External definitions could interfere with package preferences. Now, if + `openmpi` is the preferred `mpi`, and an external `mpich` is defined, a new + `openmpi` *will* be built if building it is possible. Previously we would + prefer `mpich` despite the preference. + +2. Composable `cflags` (#41049). + + This release fixes a longstanding bug that concretization would fail if + there were different `cflags` specified in `packages.yaml`, + `compilers.yaml`, or on `the` CLI. Flags and their ordering are now tracked + in the concretizer and flags from multiple sources will be merged. + +3. Fix concretizer Unification for included environments (#45139). + +## Deprecations, removals, and syntax changes + +1. The old concretizer has been removed from Spack, along with the + `config:concretizer` config option. Spack will emit a warning if the option + is present in user configuration, since it now has no effect. Spack now + uses a simpler bootstrapping mechanism, where a JSON prototype is tweaked + slightly to get an initial concrete spec to download. See #45215. + +2. Best-effort expansion of spec matrices has been removed. This feature did + not work with the "new" ASP-based concretizer, and did not work with + `unify: True` or `unify: when_possible`. Use the + [exclude key](https://spack.readthedocs.io/en/latest/environments.html#spec-matrices) + for the environment to exclude invalid components, or use multiple spec + matrices to combine the list of specs for which the constraint is valid and + the list of specs for which it is not. See #40792. + +3. The old Cray `platform` (based on Cray PE modules) has been removed, and + `platform=cray` is no longer supported. Since `v0.19`, Spack has handled + Cray machines like Linux clusters with extra packages, and we have + encouraged using this option to support Cray. The new approach allows us to + correctly handle Cray machines with non-SLES operating systems, and it is + much more reliable than making assumptions about Cray modules. See the + `v0.19` release notes and #43796 for more details. + +4. The `config:install_missing_compilers` config option has been deprecated, + and it is a no-op when set in `v0.23`. Our new compiler dependency model + will replace it with a much more reliable and robust mechanism in `v1.0`. + See #46237. + +5. Config options that deprecated in `v0.21` have been removed in `v0.23`. You + can now only specify preferences for `compilers`, `targets`, and + `providers` globally via the `packages:all:` section. Similarly, you can + only specify `versions:` locally for a specific package. See #44061 and + #31261 for details. + +6. Spack's old test interface has been removed (#45752), having been + deprecated in `v0.22.0` (#34236). All `builtin` packages have been updated + to use the new interface. See the [stand-alone test documentation]( + https://spack.readthedocs.io/en/latest/packaging_guide.html#stand-alone-tests) + +7. The `spack versions --safe-only` option, deprecated since `v0.21.0`, has + been removed. See #45765. + +* The `--dependencies` and `--optimize` arguments to `spack ci` have been + deprecated. See #45005. + +## Binary caches +1. Public binary caches now include an ML stack for Linux/aarch64 (#39666)We + now build an ML stack for Linux/aarch64 for all pull requests and on + develop. The ML stack includes both CPU-only and CUDA builds for Horovod, + Hugging Face, JAX, Keras, PyTorch,scikit-learn, TensorBoard, and + TensorFlow, and related packages. The CPU-only stack also includes XGBoost. + See https://cache.spack.io/tag/develop/?stack=ml-linux-aarch64-cuda. + +2. There is also now an stack of developer tools for macOS (#46910), which is + analogous to the Linux devtools stack. You can use this to avoid building + many common build dependencies. See + https://cache.spack.io/tag/develop/?stack=developer-tools-darwin. + +## Architecture support +* archspec has been updated to `v0.2.5`, with support for `zen5` +* Spack's CUDA package now supports the Grace Hopper `9.0a` compute capability (#45540) + +## Windows +* Windows bootstrapping: `file` and `gpg` (#41810) +* `scripts` directory added to PATH on Windows for python extensions (#45427) +* Fix `spack load --list` and `spack unload` on Windows (#35720) + +## Other notable changes +* Bugfix: `spack find -x` in environments (#46798) +* Spec splices are now robust to duplicate nodes with the same name in a spec (#46382) +* Cache per-compiler libc calculations for performance (#47213) +* Fixed a bug in external detection for openmpi (#47541) +* Mirror configuration allows username/password as environment variables (#46549) +* Default library search caps maximum depth (#41945) +* Unify interface for `spack spec` and `spack solve` commands (#47182) +* Spack no longer RPATHs directories in the default library search path (#44686) +* Improved performance of Spack database (#46554) +* Enable package reuse for packages with versions from git refs (#43859) +* Improved handling for `uuid` virtual on macos (#43002) +* Improved tracking of task queueing/requeueing in the installer (#46293) + +## Spack community stats + +* Over 2,000 pull requests updated package recipes +* 8,307 total packages, 329 new since `v0.22.0` + * 140 new Python packages + * 14 new R packages +* 373 people contributed to this release + * 357 committers to packages + * 60 committers to core + + # v0.22.2 (2024-09-21) ## Bugfixes @@ -419,7 +781,7 @@ - spack graph: fix coloring with environments (#41240) - spack info: sort variants in --variants-by-name (#41389) - Spec.format: error on old style format strings (#41934) -- ASP-based solver: +- ASP-based solver: - fix infinite recursion when computing concretization errors (#41061) - don't error for type mismatch on preferences (#41138) - don't emit spurious debug output (#41218)