Compare commits
1 Commits
e4s-24.02
...
refactor-s
Author | SHA1 | Date | |
---|---|---|---|
![]() |
39c074ff79 |
2
.github/workflows/build-containers.yml
vendored
2
.github/workflows/build-containers.yml
vendored
@@ -57,7 +57,7 @@ jobs:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # @v2
|
||||
|
||||
- uses: docker/metadata-action@dbef88086f6cef02e264edb7dbf63250c17cef6c
|
||||
- uses: docker/metadata-action@31cebacef4805868f9ce9a0cb03ee36c32df2ac4
|
||||
id: docker_meta
|
||||
with:
|
||||
images: |
|
||||
|
8
.github/workflows/style/requirements.txt
vendored
8
.github/workflows/style/requirements.txt
vendored
@@ -1,7 +1,7 @@
|
||||
black==23.12.1
|
||||
black==23.11.0
|
||||
clingo==5.6.2
|
||||
flake8==7.0.0
|
||||
isort==5.13.2
|
||||
mypy==1.8.0
|
||||
flake8==6.1.0
|
||||
isort==5.12.0
|
||||
mypy==1.7.1
|
||||
types-six==1.16.21.9
|
||||
vermin==1.6.0
|
||||
|
20
CITATION.cff
20
CITATION.cff
@@ -31,17 +31,13 @@ type: software
|
||||
message: "If you are referencing Spack in a publication, please cite the paper below."
|
||||
title: "The Spack Package Manager: Bringing Order to HPC Software Chaos"
|
||||
abstract: >-
|
||||
Large HPC centers spend considerable time supporting software for thousands of users, but the
|
||||
complexity of HPC software is quickly outpacing the capabilities of existing software management
|
||||
tools. Scientific applications require specific versions of compilers, MPI, and other dependency
|
||||
libraries, so using a single, standard software stack is infeasible. However, managing many
|
||||
configurations is difficult because the configuration space is combinatorial in size. We
|
||||
introduce Spack, a tool used at Lawrence Livermore National Laboratory to manage this complexity.
|
||||
Spack provides a novel, re- cursive specification syntax to invoke parametric builds of packages
|
||||
and dependencies. It allows any number of builds to coexist on the same system, and it ensures
|
||||
that installed packages can find their dependencies, regardless of the environment. We show
|
||||
through real-world use cases that Spack supports diverse and demanding applications, bringing
|
||||
order to HPC software chaos.
|
||||
Large HPC centers spend considerable time supporting software for thousands of users, but the complexity of HPC software is quickly outpacing the capabilities of existing software management tools.
|
||||
Scientific applications require specific versions of compilers, MPI, and other dependency libraries, so using a single, standard software stack is infeasible.
|
||||
However, managing many configurations is difficult because the configuration space is combinatorial in size.
|
||||
We introduce Spack, a tool used at Lawrence Livermore National Laboratory to manage this complexity.
|
||||
Spack provides a novel, re- cursive specification syntax to invoke parametric builds of packages and dependencies.
|
||||
It allows any number of builds to coexist on the same system, and it ensures that installed packages can find their dependencies, regardless of the environment.
|
||||
We show through real-world use cases that Spack supports diverse and demanding applications, bringing order to HPC software chaos.
|
||||
preferred-citation:
|
||||
title: "The Spack Package Manager: Bringing Order to HPC Software Chaos"
|
||||
type: conference-paper
|
||||
@@ -75,7 +71,7 @@ preferred-citation:
|
||||
type: doi
|
||||
value: 10.1145/2807591.2807623
|
||||
- description: "The DOE Document Release Number of the work"
|
||||
type: other
|
||||
type: other
|
||||
value: "LLNL-CONF-669890"
|
||||
authors:
|
||||
- family-names: "Gamblin"
|
||||
|
@@ -1,6 +1,6 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2013-2024 LLNS, LLC and other Spack Project Developers.
|
||||
Copyright (c) 2013-2023 LLNS, LLC and other Spack Project Developers.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
|
39
README.md
39
README.md
@@ -1,34 +1,13 @@
|
||||
<div align="left">
|
||||
# <img src="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo.svg" width="64" valign="middle" alt="Spack"/> Spack
|
||||
|
||||
<h2>
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo-white-text.svg" width="250">
|
||||
<source media="(prefers-color-scheme: light)" srcset="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo-text.svg" width="250">
|
||||
<img alt="Spack" src="https://cdn.rawgit.com/spack/spack/develop/share/spack/logo/spack-logo-text.svg" width="250">
|
||||
</picture>
|
||||
|
||||
<br>
|
||||
<br clear="all">
|
||||
|
||||
<a href="https://github.com/spack/spack/actions/workflows/ci.yml"><img src="https://github.com/spack/spack/workflows/ci/badge.svg" alt="CI Status"></a>
|
||||
<a href="https://github.com/spack/spack/actions/workflows/bootstrapping.yml"><img src="https://github.com/spack/spack/actions/workflows/bootstrap.yml/badge.svg" alt="Bootstrap Status"></a>
|
||||
<a href="https://github.com/spack/spack/actions/workflows/build-containers.yml"><img src="https://github.com/spack/spack/actions/workflows/build-containers.yml/badge.svg" alt="Containers Status"></a>
|
||||
<a href="https://spack.readthedocs.io"><img src="https://readthedocs.org/projects/spack/badge/?version=latest" alt="Documentation Status"></a>
|
||||
<a href="https://codecov.io/gh/spack/spack"><img src="https://codecov.io/gh/spack/spack/branch/develop/graph/badge.svg" alt="Code coverage"/></a>
|
||||
<a href="https://slack.spack.io"><img src="https://slack.spack.io/badge.svg" alt="Slack"/></a>
|
||||
<a href="https://matrix.to/#/#spack-space:matrix.org"><img src="https://img.shields.io/matrix/spack-space%3Amatrix.org?label=matrix" alt="Matrix"/></a>
|
||||
|
||||
</h2>
|
||||
|
||||
**[Getting Started] • [Config] • [Community] • [Contributing] • [Packaging Guide]**
|
||||
|
||||
[Getting Started]: https://spack.readthedocs.io/en/latest/getting_started.html
|
||||
[Config]: https://spack.readthedocs.io/en/latest/configuration.html
|
||||
[Community]: #community
|
||||
[Contributing]: https://spack.readthedocs.io/en/latest/contribution_guide.html
|
||||
[Packaging Guide]: https://spack.readthedocs.io/en/latest/packaging_guide.html
|
||||
|
||||
</div>
|
||||
[](https://github.com/spack/spack/actions)
|
||||
[](https://github.com/spack/spack/actions/workflows/bootstrap.yml)
|
||||
[](https://codecov.io/gh/spack/spack)
|
||||
[](https://github.com/spack/spack/actions/workflows/build-containers.yml)
|
||||
[](https://spack.readthedocs.io)
|
||||
[](https://github.com/psf/black)
|
||||
[](https://slack.spack.io)
|
||||
[](https://matrix.to/#/#spack-space:matrix.org)
|
||||
|
||||
Spack is a multi-platform package manager that builds and installs
|
||||
multiple versions and configurations of software. It works on Linux,
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,6 +1,6 @@
|
||||
#!/bin/sh
|
||||
#
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# sbang project developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,7 +1,7 @@
|
||||
#!/bin/sh
|
||||
# -*- python -*-
|
||||
#
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,6 +1,6 @@
|
||||
#!/bin/sh
|
||||
#
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
:: Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
:: Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
:: Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
::
|
||||
:: SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -90,7 +90,7 @@ and optimizers do require a paid license. In Spack, they are packaged as:
|
||||
TODO: Confirm and possible change(!) the scope of MPI components (runtime
|
||||
vs. devel) in current (and previous?) *cluster/professional/composer*
|
||||
editions, i.e., presence in downloads, possibly subject to license
|
||||
coverage(!); see `discussion in PR #4300
|
||||
coverage(!); see `disussion in PR #4300
|
||||
<https://github.com/spack/spack/pull/4300#issuecomment-305582898>`_. [NB:
|
||||
An "mpi" subdirectory is not indicative of the full MPI SDK being present
|
||||
(i.e., ``mpicc``, ..., and header files). The directory may just as well
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -88,13 +88,13 @@ command-line. However, Makefiles that use ``?=`` for assignment honor
|
||||
environment variables. Since Spack already sets ``CC``, ``CXX``, ``F77``,
|
||||
and ``FC``, you won't need to worry about setting these variables. If
|
||||
there are any other variables you need to set, you can do this in the
|
||||
``setup_build_environment`` method:
|
||||
``edit`` method:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def setup_build_environment(self, env):
|
||||
env.set("PREFIX", prefix)
|
||||
env.set("BLASLIB", spec["blas"].libs.ld_flags)
|
||||
def edit(self, spec, prefix):
|
||||
env["PREFIX"] = prefix
|
||||
env["BLASLIB"] = spec["blas"].libs.ld_flags
|
||||
|
||||
|
||||
`cbench <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cbench/package.py>`_
|
||||
@@ -140,7 +140,7 @@ Edit Makefile
|
||||
Some Makefiles are just plain stubborn and will ignore command-line
|
||||
variables. The only way to ensure that these packages build correctly
|
||||
is to directly edit the Makefile. Spack provides a ``FileFilter`` class
|
||||
and a ``filter`` method to help with this. For example:
|
||||
and a ``filter_file`` method to help with this. For example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -215,7 +215,6 @@ def setup(sphinx):
|
||||
("py:class", "spack.spec.InstallStatus"),
|
||||
("py:class", "spack.spec.SpecfileReaderBase"),
|
||||
("py:class", "spack.install_test.Pb"),
|
||||
("py:class", "spack.filesystem_view.SimpleFilesystemView"),
|
||||
]
|
||||
|
||||
# The reST default role (used for this markup: `text`) to use for all documents.
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -9,96 +9,34 @@
|
||||
Container Images
|
||||
================
|
||||
|
||||
Spack :ref:`environments` can easily be turned into container images. This page
|
||||
outlines two ways in which this can be done:
|
||||
|
||||
1. By installing the environment on the host system, and copying the installations
|
||||
into the container image. This approach does not require any tools like Docker
|
||||
or Singularity to be installed.
|
||||
2. By generating a Docker or Singularity recipe that can be used to build the
|
||||
container image. In this approach, Spack builds the software inside the
|
||||
container runtime, not on the host system.
|
||||
|
||||
The first approach is easiest if you already have an installed environment,
|
||||
the second approach gives more control over the container image.
|
||||
|
||||
---------------------------
|
||||
From existing installations
|
||||
---------------------------
|
||||
|
||||
If you already have a Spack environment installed on your system, you can
|
||||
share the binaries as an OCI compatible container image. To get started you
|
||||
just have to configure and OCI registry and run ``spack buildcache push``.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# Create and install an environment in the current directory
|
||||
spack env create -d .
|
||||
spack -e . add pkg-a pkg-b
|
||||
spack -e . install
|
||||
|
||||
# Configure the registry
|
||||
spack -e . mirror add --oci-username ... --oci-password ... container-registry oci://example.com/name/image
|
||||
|
||||
# Push the image
|
||||
spack -e . buildcache push --update-index --base-image ubuntu:22.04 --tag my_env container-registry
|
||||
|
||||
The resulting container image can then be run as follows:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ docker run -it example.com/name/image:my_env
|
||||
|
||||
The image generated by Spack consists of the specified base image with each package from the
|
||||
environment as a separate layer on top. The image is minimal by construction, it only contains the
|
||||
environment roots and its runtime dependencies.
|
||||
|
||||
.. note::
|
||||
|
||||
When using registries like GHCR and Docker Hub, the ``--oci-password`` flag is not
|
||||
the password for your account, but a personal access token you need to generate separately.
|
||||
|
||||
The specified ``--base-image`` should have a libc that is compatible with the host system.
|
||||
For example if your host system is Ubuntu 20.04, you can use ``ubuntu:20.04``, ``ubuntu:22.04``
|
||||
or newer: the libc in the container image must be at least the version of the host system,
|
||||
assuming ABI compatibility. It is also perfectly fine to use a completely different
|
||||
Linux distribution as long as the libc is compatible.
|
||||
|
||||
For convenience, Spack also turns the OCI registry into a :ref:`build cache <binary_caches_oci>`,
|
||||
so that future ``spack install`` of the environment will simply pull the binaries from the
|
||||
registry instead of doing source builds. The flag ``--update-index`` is needed to make Spack
|
||||
take the build cache into account when concretizing.
|
||||
|
||||
.. note::
|
||||
|
||||
When generating container images in CI, the approach above is recommended when CI jobs
|
||||
already run in a sandboxed environment. You can simply use ``spack`` directly
|
||||
in the CI job and push the resulting image to a registry. Subsequent CI jobs should
|
||||
run faster because Spack can install from the same registry instead of rebuilding from
|
||||
sources.
|
||||
|
||||
---------------------------------------------
|
||||
Generating recipes for Docker and Singularity
|
||||
---------------------------------------------
|
||||
|
||||
Apart from copying existing installations into container images, Spack can also
|
||||
generate recipes for container images. This is useful if you want to run Spack
|
||||
itself in a sandboxed environment instead of on the host system.
|
||||
|
||||
Since recipes need a little bit more boilerplate than
|
||||
Spack :ref:`environments` are a great tool to create container images, but
|
||||
preparing one that is suitable for production requires some more boilerplate
|
||||
than just:
|
||||
|
||||
.. code-block:: docker
|
||||
|
||||
COPY spack.yaml /environment
|
||||
RUN spack -e /environment install
|
||||
|
||||
Spack provides a command to generate customizable recipes for container images. Customizations
|
||||
include minimizing the size of the image, installing packages in the base image using the system
|
||||
package manager, and setting up a proper entrypoint to run the image.
|
||||
Additional actions may be needed to minimize the size of the
|
||||
container, or to update the system software that is installed in the base
|
||||
image, or to set up a proper entrypoint to run the image. These tasks are
|
||||
usually both necessary and repetitive, so Spack comes with a command
|
||||
to generate recipes for container images starting from a ``spack.yaml``.
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
.. seealso::
|
||||
|
||||
This page is a reference for generating recipes to build container images.
|
||||
It means that your environment is built from scratch inside the container
|
||||
runtime.
|
||||
|
||||
Since v0.21, Spack can also create container images from existing package installations
|
||||
on your host system. See :ref:`binary_caches_oci` for more information on
|
||||
that topic.
|
||||
|
||||
--------------------
|
||||
A Quick Introduction
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
--------------------
|
||||
|
||||
Consider having a Spack environment like the following:
|
||||
|
||||
@@ -109,8 +47,8 @@ Consider having a Spack environment like the following:
|
||||
- gromacs+mpi
|
||||
- mpich
|
||||
|
||||
Producing a ``Dockerfile`` from it is as simple as changing directories to
|
||||
where the ``spack.yaml`` file is stored and running the following command:
|
||||
Producing a ``Dockerfile`` from it is as simple as moving to the directory
|
||||
where the ``spack.yaml`` file is stored and giving the following command:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
@@ -176,9 +114,9 @@ configuration are discussed in details in the sections below.
|
||||
|
||||
.. _container_spack_images:
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
--------------------------
|
||||
Spack Images on Docker Hub
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
--------------------------
|
||||
|
||||
Docker images with Spack preinstalled and ready to be used are
|
||||
built when a release is tagged, or nightly on ``develop``. The images
|
||||
@@ -248,9 +186,9 @@ by Spack use them as default base images for their ``build`` stage,
|
||||
even though handles to use custom base images provided by users are
|
||||
available to accommodate complex use cases.
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
Configuring the Container Recipe
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
---------------------------------
|
||||
Creating Images From Environments
|
||||
---------------------------------
|
||||
|
||||
Any Spack Environment can be used for the automatic generation of container
|
||||
recipes. Sensible defaults are provided for things like the base image or the
|
||||
@@ -291,18 +229,18 @@ under the ``container`` attribute of environments:
|
||||
|
||||
A detailed description of the options available can be found in the :ref:`container_config_options` section.
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
-------------------
|
||||
Setting Base Images
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
-------------------
|
||||
|
||||
The ``images`` subsection is used to select both the image where
|
||||
Spack builds the software and the image where the built software
|
||||
is installed. This attribute can be set in different ways and
|
||||
which one to use depends on the use case at hand.
|
||||
|
||||
""""""""""""""""""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Use Official Spack Images From Dockerhub
|
||||
""""""""""""""""""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To generate a recipe that uses an official Docker image from the
|
||||
Spack organization to build the software and the corresponding official OS image
|
||||
@@ -507,9 +445,9 @@ responsibility to ensure that:
|
||||
Therefore we don't recommend its use in cases that can be otherwise
|
||||
covered by the simplified mode shown first.
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
----------------------------
|
||||
Singularity Definition Files
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
----------------------------
|
||||
|
||||
In addition to producing recipes in ``Dockerfile`` format Spack can produce
|
||||
Singularity Definition Files by just changing the value of the ``format``
|
||||
@@ -530,9 +468,9 @@ attribute:
|
||||
The minimum version of Singularity required to build a SIF (Singularity Image Format)
|
||||
image from the recipes generated by Spack is ``3.5.3``.
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
------------------------------
|
||||
Extending the Jinja2 Templates
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
------------------------------
|
||||
|
||||
The Dockerfile and the Singularity definition file that Spack can generate are based on
|
||||
a few Jinja2 templates that are rendered according to the environment being containerized.
|
||||
@@ -653,9 +591,9 @@ The recipe that gets generated contains the two extra instruction that we added
|
||||
|
||||
.. _container_config_options:
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
-----------------------
|
||||
Configuration Reference
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
-----------------------
|
||||
|
||||
The tables below describe all the configuration options that are currently supported
|
||||
to customize the generation of container recipes:
|
||||
@@ -752,13 +690,13 @@ to customize the generation of container recipes:
|
||||
- Description string
|
||||
- No
|
||||
|
||||
~~~~~~~~~~~~~~
|
||||
--------------
|
||||
Best Practices
|
||||
~~~~~~~~~~~~~~
|
||||
--------------
|
||||
|
||||
"""
|
||||
^^^
|
||||
MPI
|
||||
"""
|
||||
^^^
|
||||
Due to the dependency on Fortran for OpenMPI, which is the spack default
|
||||
implementation, consider adding ``gfortran`` to the ``apt-get install`` list.
|
||||
|
||||
@@ -769,9 +707,9 @@ For execution on HPC clusters, it can be helpful to import the docker
|
||||
image into Singularity in order to start a program with an *external*
|
||||
MPI. Otherwise, also add ``openssh-server`` to the ``apt-get install`` list.
|
||||
|
||||
""""
|
||||
^^^^
|
||||
CUDA
|
||||
""""
|
||||
^^^^
|
||||
Starting from CUDA 9.0, Nvidia provides minimal CUDA images based on
|
||||
Ubuntu. Please see `their instructions <https://hub.docker.com/r/nvidia/cuda/>`_.
|
||||
Avoid double-installing CUDA by adding, e.g.
|
||||
@@ -790,9 +728,9 @@ to your ``spack.yaml``.
|
||||
Users will either need ``nvidia-docker`` or e.g. Singularity to *execute*
|
||||
device kernels.
|
||||
|
||||
"""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Docker on Windows and OSX
|
||||
"""""""""""""""""""""""""
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
On Mac OS and Windows, docker runs on a hypervisor that is not allocated much
|
||||
memory by default, and some spack packages may fail to build due to lack of
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -142,21 +142,6 @@ user's prompt to begin with the environment name in brackets.
|
||||
$ spack env activate -p myenv
|
||||
[myenv] $ ...
|
||||
|
||||
The ``activate`` command can also be used to create a new environment, if it is
|
||||
not already defined, by adding the ``--create`` flag. Managed and anonymous
|
||||
environments, anonymous environments are explained in the next section,
|
||||
can both be created using the same flags that `spack env create` accepts.
|
||||
If an environment already exists then spack will simply activate it and ignore the
|
||||
create specific flags.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack env activate --create -p myenv
|
||||
# ...
|
||||
# [creates if myenv does not exist yet]
|
||||
# ...
|
||||
[myenv] $ ...
|
||||
|
||||
To deactivate an environment, use the command:
|
||||
|
||||
.. code-block:: console
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -97,35 +97,6 @@ Each package version and compiler listed in an external should
|
||||
have entries in Spack's packages and compiler configuration, even
|
||||
though the package and compiler may not ever be built.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Extra attributes for external packages
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Sometimes external packages require additional attributes to be used
|
||||
effectively. This information can be defined on a per-package basis
|
||||
and stored in the ``extra_attributes`` section of the external package
|
||||
configuration. In addition to per-package information, this section
|
||||
can be used to define environment modifications to be performed
|
||||
whenever the package is used. For example, if an external package is
|
||||
built without ``rpath`` support, it may require ``LD_LIBRARY_PATH``
|
||||
settings to find its dependencies. This could be configured as
|
||||
follows:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
packages:
|
||||
mpich:
|
||||
externals:
|
||||
- spec: mpich@3.3 %clang@12.0.0 +hwloc
|
||||
prefix: /path/to/mpich
|
||||
extra_attributes:
|
||||
environment:
|
||||
prepend_path:
|
||||
LD_LIBRARY_PATH: /path/to/hwloc/lib64
|
||||
|
||||
See :ref:`configuration_environment_variables` for more information on
|
||||
how to configure environment modifications in Spack config files.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Prevent packages from being built from sources
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@@ -487,56 +458,6 @@ present. For instance with a configuration like:
|
||||
|
||||
you will use ``mvapich2~cuda %gcc`` as an ``mpi`` provider.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Conflicts and strong preferences
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If the semantic of requirements is too strong, you can also express "strong preferences" and "conflicts"
|
||||
from configuration files:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
packages:
|
||||
all:
|
||||
prefer:
|
||||
- '%clang'
|
||||
conflict:
|
||||
- '+shared'
|
||||
|
||||
The ``prefer`` and ``conflict`` sections can be used whenever a ``require`` section is allowed.
|
||||
The argument is always a list of constraints, and each constraint can be either a simple string,
|
||||
or a more complex object:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
packages:
|
||||
all:
|
||||
conflict:
|
||||
- spec: '%clang'
|
||||
when: 'target=x86_64_v3'
|
||||
message: 'reason why clang cannot be used'
|
||||
|
||||
The ``spec`` attribute is mandatory, while both ``when`` and ``message`` are optional.
|
||||
|
||||
.. note::
|
||||
|
||||
Requirements allow for expressing both "strong preferences" and "conflicts".
|
||||
The syntax for doing so, though, may not be immediately clear. For
|
||||
instance, if we want to prevent any package from using ``%clang``, we can set:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
packages:
|
||||
all:
|
||||
require:
|
||||
- one_of: ['%clang', '@:']
|
||||
|
||||
Since only one of the requirements must hold, and ``@:`` is always true, the rule above is
|
||||
equivalent to a conflict. For "strong preferences" we need to substitute the ``one_of`` policy
|
||||
with ``any_of``.
|
||||
|
||||
|
||||
|
||||
.. _package-preferences:
|
||||
|
||||
-------------------
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -237,7 +237,7 @@ for details):
|
||||
.. code-block:: python
|
||||
:linenos:
|
||||
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -4379,16 +4379,10 @@ implementation was selected for this build:
|
||||
elif "mvapich" in spec:
|
||||
configure_args.append("--with-mvapich")
|
||||
|
||||
It's also a bit more concise than satisfies.
|
||||
|
||||
.. note::
|
||||
|
||||
The ``satisfies()`` method tests whether this spec has, at least, all the constraints of the argument spec,
|
||||
while ``in`` tests whether a spec or any of its dependencies satisfy the provided spec.
|
||||
|
||||
If the provided spec is anonymous (e.g., ":1.2:", "+shared") or has the
|
||||
same name as the spec being checked, then ``in`` works the same as
|
||||
``satisfies()``; however, use of ``satisfies()`` is more intuitive.
|
||||
It's also a bit more concise than satisfies. The difference between
|
||||
the two functions is that ``satisfies()`` tests whether spec
|
||||
constraints overlap at all, while ``in`` tests whether a spec or any
|
||||
of its dependencies satisfy the provided spec.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Architecture specifiers
|
||||
@@ -5290,7 +5284,7 @@ installed example.
|
||||
example = which(self.prefix.bin.example)
|
||||
example()
|
||||
|
||||
Output showing the identification of each test part after running the tests
|
||||
Output showing the identification of each test part after runnig the tests
|
||||
is illustrated below.
|
||||
|
||||
.. code-block:: console
|
||||
@@ -5787,7 +5781,7 @@ with those implemented in the package itself.
|
||||
* - `Cxx
|
||||
<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cxx>`_
|
||||
- Compiles and runs several ``hello`` programs
|
||||
* - `Fortran
|
||||
* - `Fortan
|
||||
<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/fortran>`_
|
||||
- Compiles and runs ``hello`` programs (``F`` and ``f90``)
|
||||
* - `Mpi
|
||||
@@ -6979,18 +6973,3 @@ you probably care most about are:
|
||||
You may also care about `license exceptions
|
||||
<https://spdx.org/licenses/exceptions-index.html>`_ that use the ``WITH`` operator,
|
||||
e.g. ``Apache-2.0 WITH LLVM-exception``.
|
||||
|
||||
Many of the licenses that are currently in the spack repositories have been
|
||||
automatically determined. While this is great for bulk adding license
|
||||
information and is most likely correct, there are sometimes edge cases that
|
||||
require manual intervention. To determine which licenses are validated and
|
||||
which are not, there is the `checked_by` parameter in the license directive:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
license("<license>", when="<when>", checked_by="<github username>")
|
||||
|
||||
When you have validated a github license, either when doing so explicitly or
|
||||
as part of packaging a new package, please set the `checked_by` parameter
|
||||
to your Github username to signal that the license has been manually
|
||||
verified.
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -6,8 +6,8 @@ python-levenshtein==0.23.0
|
||||
docutils==0.20.1
|
||||
pygments==2.17.2
|
||||
urllib3==2.1.0
|
||||
pytest==7.4.4
|
||||
isort==5.13.2
|
||||
black==23.12.1
|
||||
flake8==7.0.0
|
||||
mypy==1.8.0
|
||||
pytest==7.4.3
|
||||
isort==5.12.0
|
||||
black==23.11.0
|
||||
flake8==6.1.0
|
||||
mypy==1.7.1
|
||||
|
@@ -1,4 +1,4 @@
|
||||
.. Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
.. Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
|
||||
Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
|
||||
SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -142,7 +142,7 @@ Reputational Key
|
||||
----------------
|
||||
|
||||
The Reputational Key is the public facing key used to sign complete groups of
|
||||
development and release packages. Only one key pair exists in this class of
|
||||
development and release packages. Only one key pair exsits in this class of
|
||||
keys. In contrast to the Intermediate CI Key the Reputational Key *should* be
|
||||
used to verify package integrity. At the end of develop and release pipeline a
|
||||
final pipeline job pulls down all signed package metadata built by the pipeline,
|
||||
@@ -272,7 +272,7 @@ Internal Implementation
|
||||
|
||||
The technical implementation of the pipeline signing process includes components
|
||||
defined in Amazon Web Services, the Kubernetes cluster, at affilicated
|
||||
institutions, and the GitLab/GitLab Runner deployment. We present the technical
|
||||
institutions, and the GitLab/GitLab Runner deployment. We present the techincal
|
||||
implementation in two interdependent sections. The first addresses how secrets
|
||||
are managed through the lifecycle of a develop or release pipeline. The second
|
||||
section describes how Gitlab Runner and pipelines are configured and managed to
|
||||
@@ -295,7 +295,7 @@ infrastructure.
|
||||
-----------------------
|
||||
|
||||
Multiple intermediate CI signing keys exist, one Intermediate CI Key for jobs
|
||||
run in AWS, and one key for each affiliated institution (e.g. University of
|
||||
run in AWS, and one key for each affiliated institution (e.g. Univerity of
|
||||
Oregon). Here we describe how the Intermediate CI Key is managed in AWS:
|
||||
|
||||
The Intermediate CI Key (including the Signing Intermediate CI Private Key is
|
||||
@@ -305,7 +305,7 @@ contains an ASCII-armored export of just the *public* components of the
|
||||
Reputational Key. This secret also contains the *public* components of each of
|
||||
the affiliated institutions' Intermediate CI Key. These are potentially needed
|
||||
to verify dependent packages which may have been found in the public mirror or
|
||||
built by a protected job running on an affiliated institution's infrastructure
|
||||
built by a protected job running on an affiliated institution's infrastrcuture
|
||||
in an earlier stage of the pipeline.
|
||||
|
||||
Procedurally the ``spack-intermediate-ci-signing-key`` secret is used in
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
2
lib/spack/env/cc
vendored
2
lib/spack/env/cc
vendored
@@ -1,7 +1,7 @@
|
||||
#!/bin/sh -f
|
||||
# shellcheck disable=SC2034 # evals in this script fool shellcheck
|
||||
#
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
2
lib/spack/external/__init__.py
vendored
2
lib/spack/external/__init__.py
vendored
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -204,23 +204,17 @@ def color_when(value):
|
||||
|
||||
|
||||
class match_to_ansi:
|
||||
def __init__(self, color=True, enclose=False, zsh=False):
|
||||
def __init__(self, color=True, enclose=False):
|
||||
self.color = _color_when_value(color)
|
||||
self.enclose = enclose
|
||||
self.zsh = zsh
|
||||
|
||||
def escape(self, s):
|
||||
"""Returns a TTY escape sequence for a color"""
|
||||
if self.color:
|
||||
if self.zsh:
|
||||
result = rf"\e[0;{s}m"
|
||||
else:
|
||||
result = f"\033[{s}m"
|
||||
|
||||
if self.enclose:
|
||||
result = rf"\[{result}\]"
|
||||
|
||||
return result
|
||||
return r"\[\033[%sm\]" % s
|
||||
else:
|
||||
return "\033[%sm" % s
|
||||
else:
|
||||
return ""
|
||||
|
||||
@@ -267,11 +261,9 @@ def colorize(string, **kwargs):
|
||||
codes, for output to non-console devices.
|
||||
enclose (bool): If True, enclose ansi color sequences with
|
||||
square brackets to prevent misestimation of terminal width.
|
||||
zsh (bool): If True, use zsh ansi codes instead of bash ones (for variables like PS1)
|
||||
"""
|
||||
color = _color_when_value(kwargs.get("color", get_color_when()))
|
||||
zsh = kwargs.get("zsh", False)
|
||||
string = re.sub(color_re, match_to_ansi(color, kwargs.get("enclose")), string, zsh)
|
||||
string = re.sub(color_re, match_to_ansi(color, kwargs.get("enclose")), string)
|
||||
string = string.replace("}}", "}")
|
||||
return string
|
||||
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -667,8 +667,8 @@ def _unknown_variants_in_directives(pkgs, error_cls):
|
||||
pkg_cls = spack.repo.PATH.get_pkg_class(pkg_name)
|
||||
|
||||
# Check "conflicts" directive
|
||||
for trigger, conflicts in pkg_cls.conflicts.items():
|
||||
for conflict, _ in conflicts:
|
||||
for conflict, triggers in pkg_cls.conflicts.items():
|
||||
for trigger, _ in triggers:
|
||||
vrn = spack.spec.Spec(conflict)
|
||||
try:
|
||||
vrn.constrain(trigger)
|
||||
@@ -694,21 +694,25 @@ def _unknown_variants_in_directives(pkgs, error_cls):
|
||||
)
|
||||
|
||||
# Check "depends_on" directive
|
||||
for trigger in pkg_cls.dependencies:
|
||||
vrn = spack.spec.Spec(trigger)
|
||||
errors.extend(
|
||||
_analyze_variants_in_directive(
|
||||
pkg_cls, vrn, directive="depends_on", error_cls=error_cls
|
||||
for _, triggers in pkg_cls.dependencies.items():
|
||||
triggers = list(triggers)
|
||||
for trigger in list(triggers):
|
||||
vrn = spack.spec.Spec(trigger)
|
||||
errors.extend(
|
||||
_analyze_variants_in_directive(
|
||||
pkg_cls, vrn, directive="depends_on", error_cls=error_cls
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
# Check "provides" directive
|
||||
for when_spec in pkg_cls.provided:
|
||||
errors.extend(
|
||||
_analyze_variants_in_directive(
|
||||
pkg_cls, when_spec, directive="provides", error_cls=error_cls
|
||||
# Check "patch" directive
|
||||
for _, triggers in pkg_cls.provided.items():
|
||||
triggers = [spack.spec.Spec(x) for x in triggers]
|
||||
for vrn in triggers:
|
||||
errors.extend(
|
||||
_analyze_variants_in_directive(
|
||||
pkg_cls, vrn, directive="patch", error_cls=error_cls
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
# Check "resource" directive
|
||||
for vrn in pkg_cls.resources:
|
||||
@@ -732,72 +736,61 @@ def _issues_in_depends_on_directive(pkgs, error_cls):
|
||||
for pkg_name in pkgs:
|
||||
pkg_cls = spack.repo.PATH.get_pkg_class(pkg_name)
|
||||
filename = spack.repo.PATH.filename_for_package_name(pkg_name)
|
||||
|
||||
for when, deps_by_name in pkg_cls.dependencies.items():
|
||||
for dep_name, dep in deps_by_name.items():
|
||||
# Check if there are nested dependencies declared. We don't want directives like:
|
||||
#
|
||||
# depends_on('foo+bar ^fee+baz')
|
||||
#
|
||||
# but we'd like to have two dependencies listed instead.
|
||||
nested_dependencies = dep.spec.dependencies()
|
||||
for dependency_name, dependency_data in pkg_cls.dependencies.items():
|
||||
# Check if there are nested dependencies declared. We don't want directives like:
|
||||
#
|
||||
# depends_on('foo+bar ^fee+baz')
|
||||
#
|
||||
# but we'd like to have two dependencies listed instead.
|
||||
for when, dependency_edge in dependency_data.items():
|
||||
dependency_spec = dependency_edge.spec
|
||||
nested_dependencies = dependency_spec.dependencies()
|
||||
if nested_dependencies:
|
||||
summary = f"{pkg_name}: nested dependency declaration '{dep.spec}'"
|
||||
ndir = len(nested_dependencies) + 1
|
||||
summary = (
|
||||
f"{pkg_name}: invalid nested dependency "
|
||||
f"declaration '{str(dependency_spec)}'"
|
||||
)
|
||||
details = [
|
||||
f"split depends_on('{dep.spec}', when='{when}') into {ndir} directives",
|
||||
f"split depends_on('{str(dependency_spec)}', when='{str(when)}') "
|
||||
f"into {len(nested_dependencies) + 1} directives",
|
||||
f"in {filename}",
|
||||
]
|
||||
errors.append(error_cls(summary=summary, details=details))
|
||||
|
||||
def check_virtual_with_variants(spec, msg):
|
||||
if not spec.virtual or not spec.variants:
|
||||
return
|
||||
error = error_cls(
|
||||
f"{pkg_name}: {msg}",
|
||||
f"remove variants from '{spec}' in depends_on directive in {filename}",
|
||||
)
|
||||
errors.append(error)
|
||||
# No need to analyze virtual packages
|
||||
if spack.repo.PATH.is_virtual(dependency_name):
|
||||
continue
|
||||
|
||||
check_virtual_with_variants(dep.spec, "virtual dependency cannot have variants")
|
||||
check_virtual_with_variants(dep.spec, "virtual when= spec cannot have variants")
|
||||
try:
|
||||
dependency_pkg_cls = spack.repo.PATH.get_pkg_class(dependency_name)
|
||||
except spack.repo.UnknownPackageError:
|
||||
# This dependency is completely missing, so report
|
||||
# and continue the analysis
|
||||
summary = pkg_name + ": unknown package '{0}' in " "'depends_on' directive".format(
|
||||
dependency_name
|
||||
)
|
||||
details = [" in " + filename]
|
||||
errors.append(error_cls(summary=summary, details=details))
|
||||
continue
|
||||
|
||||
# No need to analyze virtual packages
|
||||
if spack.repo.PATH.is_virtual(dep_name):
|
||||
continue
|
||||
|
||||
# check for unknown dependencies
|
||||
try:
|
||||
dependency_pkg_cls = spack.repo.PATH.get_pkg_class(dep_name)
|
||||
except spack.repo.UnknownPackageError:
|
||||
# This dependency is completely missing, so report
|
||||
# and continue the analysis
|
||||
summary = (
|
||||
f"{pkg_name}: unknown package '{dep_name}' in " "'depends_on' directive"
|
||||
)
|
||||
details = [f" in {filename}"]
|
||||
errors.append(error_cls(summary=summary, details=details))
|
||||
continue
|
||||
|
||||
# check variants
|
||||
dependency_variants = dep.spec.variants
|
||||
for _, dependency_edge in dependency_data.items():
|
||||
dependency_variants = dependency_edge.spec.variants
|
||||
for name, value in dependency_variants.items():
|
||||
try:
|
||||
v, _ = dependency_pkg_cls.variants[name]
|
||||
v.validate_or_raise(value, pkg_cls=dependency_pkg_cls)
|
||||
except Exception as e:
|
||||
summary = (
|
||||
f"{pkg_name}: wrong variant used for dependency in 'depends_on()'"
|
||||
pkg_name + ": wrong variant used for a "
|
||||
"dependency in a 'depends_on' directive"
|
||||
)
|
||||
|
||||
error_msg = str(e).strip()
|
||||
if isinstance(e, KeyError):
|
||||
error_msg = (
|
||||
f"variant {str(e).strip()} does not exist in package {dep_name}"
|
||||
)
|
||||
error_msg += f" in package '{dep_name}'"
|
||||
error_msg = "the variant {0} does not " "exist".format(error_msg)
|
||||
error_msg += " in package '" + dependency_name + "'"
|
||||
|
||||
errors.append(
|
||||
error_cls(summary=summary, details=[error_msg, f"in {filename}"])
|
||||
error_cls(summary=summary, details=[error_msg, "in " + filename])
|
||||
)
|
||||
|
||||
return errors
|
||||
@@ -864,17 +857,14 @@ def _version_constraints_are_satisfiable_by_some_version_in_repo(pkgs, error_cls
|
||||
for pkg_name in pkgs:
|
||||
pkg_cls = spack.repo.PATH.get_pkg_class(pkg_name)
|
||||
filename = spack.repo.PATH.filename_for_package_name(pkg_name)
|
||||
|
||||
dependencies_to_check = []
|
||||
for dependency_name, dependency_data in pkg_cls.dependencies.items():
|
||||
# Skip virtual dependencies for the time being, check on
|
||||
# their versions can be added later
|
||||
if spack.repo.PATH.is_virtual(dependency_name):
|
||||
continue
|
||||
|
||||
for _, deps_by_name in pkg_cls.dependencies.items():
|
||||
for dep_name, dep in deps_by_name.items():
|
||||
# Skip virtual dependencies for the time being, check on
|
||||
# their versions can be added later
|
||||
if spack.repo.PATH.is_virtual(dep_name):
|
||||
continue
|
||||
|
||||
dependencies_to_check.append(dep.spec)
|
||||
dependencies_to_check.extend([edge.spec for edge in dependency_data.values()])
|
||||
|
||||
host_architecture = spack.spec.ArchSpec.default_arch()
|
||||
for s in dependencies_to_check:
|
||||
@@ -946,53 +936,39 @@ def _named_specs_in_when_arguments(pkgs, error_cls):
|
||||
for pkg_name in pkgs:
|
||||
pkg_cls = spack.repo.PATH.get_pkg_class(pkg_name)
|
||||
|
||||
def _refers_to_pkg(when):
|
||||
when_spec = spack.spec.Spec(when)
|
||||
return when_spec.name is None or when_spec.name == pkg_name
|
||||
|
||||
def _error_items(when_dict):
|
||||
for when, elts in when_dict.items():
|
||||
if not _refers_to_pkg(when):
|
||||
yield when, elts, [f"using '{when}', should be '^{when}'"]
|
||||
|
||||
def _extracts_errors(triggers, summary):
|
||||
_errors = []
|
||||
for trigger in list(triggers):
|
||||
if not _refers_to_pkg(trigger):
|
||||
when_spec = spack.spec.Spec(trigger)
|
||||
if when_spec.name is not None and when_spec.name != pkg_name:
|
||||
details = [f"using '{trigger}', should be '^{trigger}'"]
|
||||
_errors.append(error_cls(summary=summary, details=details))
|
||||
return _errors
|
||||
|
||||
for when, dnames, details in _error_items(pkg_cls.dependencies):
|
||||
errors.extend(
|
||||
error_cls(f"{pkg_name}: wrong 'when=' condition for '{dname}' dependency", details)
|
||||
for dname in dnames
|
||||
)
|
||||
for dname, triggers in pkg_cls.dependencies.items():
|
||||
summary = f"{pkg_name}: wrong 'when=' condition for the '{dname}' dependency"
|
||||
errors.extend(_extracts_errors(triggers, summary))
|
||||
|
||||
for vname, (variant, triggers) in pkg_cls.variants.items():
|
||||
summary = f"{pkg_name}: wrong 'when=' condition for the '{vname}' variant"
|
||||
errors.extend(_extracts_errors(triggers, summary))
|
||||
|
||||
for when, providers, details in _error_items(pkg_cls.provided):
|
||||
errors.extend(
|
||||
error_cls(f"{pkg_name}: wrong 'when=' condition for '{provided}' virtual", details)
|
||||
for provided in providers
|
||||
)
|
||||
for provided, triggers in pkg_cls.provided.items():
|
||||
summary = f"{pkg_name}: wrong 'when=' condition for the '{provided}' virtual"
|
||||
errors.extend(_extracts_errors(triggers, summary))
|
||||
|
||||
for when, requirements, details in _error_items(pkg_cls.requirements):
|
||||
errors.append(
|
||||
error_cls(f"{pkg_name}: wrong 'when=' condition in 'requires' directive", details)
|
||||
)
|
||||
for _, triggers in pkg_cls.requirements.items():
|
||||
triggers = [when_spec for when_spec, _, _ in triggers]
|
||||
summary = f"{pkg_name}: wrong 'when=' condition in 'requires' directive"
|
||||
errors.extend(_extracts_errors(triggers, summary))
|
||||
|
||||
for when, _, details in _error_items(pkg_cls.patches):
|
||||
errors.append(
|
||||
error_cls(f"{pkg_name}: wrong 'when=' condition in 'patch' directives", details)
|
||||
)
|
||||
triggers = list(pkg_cls.patches)
|
||||
summary = f"{pkg_name}: wrong 'when=' condition in 'patch' directives"
|
||||
errors.extend(_extracts_errors(triggers, summary))
|
||||
|
||||
for when, _, details in _error_items(pkg_cls.resources):
|
||||
errors.append(
|
||||
error_cls(f"{pkg_name}: wrong 'when=' condition in 'resource' directives", details)
|
||||
)
|
||||
triggers = list(pkg_cls.resources)
|
||||
summary = f"{pkg_name}: wrong 'when=' condition in 'resource' directives"
|
||||
errors.extend(_extracts_errors(triggers, summary))
|
||||
|
||||
return llnl.util.lang.dedupe(errors)
|
||||
|
||||
|
@@ -1,10 +1,11 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
||||
import codecs
|
||||
import collections
|
||||
import errno
|
||||
import hashlib
|
||||
import io
|
||||
import itertools
|
||||
@@ -22,7 +23,8 @@
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
import warnings
|
||||
from contextlib import closing
|
||||
from contextlib import closing, contextmanager
|
||||
from gzip import GzipFile
|
||||
from typing import Dict, Iterable, List, NamedTuple, Optional, Set, Tuple
|
||||
from urllib.error import HTTPError, URLError
|
||||
|
||||
@@ -48,7 +50,6 @@
|
||||
import spack.stage
|
||||
import spack.store
|
||||
import spack.traverse as traverse
|
||||
import spack.util.archive
|
||||
import spack.util.crypto
|
||||
import spack.util.file_cache as file_cache
|
||||
import spack.util.gpg
|
||||
@@ -67,10 +68,7 @@
|
||||
|
||||
BUILD_CACHE_RELATIVE_PATH = "build_cache"
|
||||
BUILD_CACHE_KEYS_RELATIVE_PATH = "_pgp"
|
||||
|
||||
#: The build cache layout version that this version of Spack creates.
|
||||
#: Version 2: includes parent directories of the package prefix in the tarball
|
||||
CURRENT_BUILD_CACHE_LAYOUT_VERSION = 2
|
||||
CURRENT_BUILD_CACHE_LAYOUT_VERSION = 1
|
||||
|
||||
|
||||
class BuildCacheDatabase(spack_db.Database):
|
||||
@@ -1132,46 +1130,195 @@ def generate_key_index(key_prefix, tmpdir=None):
|
||||
shutil.rmtree(tmpdir)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def gzip_compressed_tarfile(path):
|
||||
"""Create a reproducible, compressed tarfile"""
|
||||
# Create gzip compressed tarball of the install prefix
|
||||
# 1) Use explicit empty filename and mtime 0 for gzip header reproducibility.
|
||||
# If the filename="" is dropped, Python will use fileobj.name instead.
|
||||
# This should effectively mimick `gzip --no-name`.
|
||||
# 2) On AMD Ryzen 3700X and an SSD disk, we have the following on compression speed:
|
||||
# compresslevel=6 gzip default: llvm takes 4mins, roughly 2.1GB
|
||||
# compresslevel=9 python default: llvm takes 12mins, roughly 2.1GB
|
||||
# So we follow gzip.
|
||||
with open(path, "wb") as f, ChecksumWriter(f) as inner_checksum, closing(
|
||||
GzipFile(filename="", mode="wb", compresslevel=6, mtime=0, fileobj=inner_checksum)
|
||||
) as gzip_file, ChecksumWriter(gzip_file) as outer_checksum, tarfile.TarFile(
|
||||
name="", mode="w", fileobj=outer_checksum
|
||||
) as tar:
|
||||
yield tar, inner_checksum, outer_checksum
|
||||
|
||||
|
||||
def _tarinfo_name(absolute_path: str, *, _path=pathlib.PurePath) -> str:
|
||||
"""Compute tarfile entry name as the relative path from the (system) root."""
|
||||
return _path(*_path(absolute_path).parts[1:]).as_posix()
|
||||
|
||||
|
||||
def tarfile_of_spec_prefix(tar: tarfile.TarFile, prefix: str) -> None:
|
||||
"""Create a tarfile of an install prefix of a spec. Skips existing buildinfo file.
|
||||
Only adds regular files, symlinks and dirs. Skips devices, fifos. Preserves hardlinks.
|
||||
Normalizes permissions like git. Tar entries are added in depth-first pre-order, with
|
||||
dir entries partitioned by file | dir, and sorted alphabetically, for reproducibility.
|
||||
Partitioning ensures only one dir is in memory at a time, and sorting improves compression.
|
||||
|
||||
Args:
|
||||
tar: tarfile object to add files to
|
||||
prefix: absolute install prefix of spec"""
|
||||
if not os.path.isabs(prefix) or not os.path.isdir(prefix):
|
||||
raise ValueError(f"prefix '{prefix}' must be an absolute path to a directory")
|
||||
hardlink_to_tarinfo_name: Dict[Tuple[int, int], str] = dict()
|
||||
stat_key = lambda stat: (stat.st_dev, stat.st_ino)
|
||||
|
||||
try: # skip buildinfo file if it exists
|
||||
files_to_skip = [stat_key(os.lstat(buildinfo_file_name(prefix)))]
|
||||
skip = lambda entry: stat_key(entry.stat(follow_symlinks=False)) in files_to_skip
|
||||
except OSError:
|
||||
skip = lambda entry: False
|
||||
files_to_skip = []
|
||||
|
||||
spack.util.archive.reproducible_tarfile_from_prefix(
|
||||
tar,
|
||||
prefix,
|
||||
# Spack <= 0.21 did not include parent directories, leading to issues when tarballs are
|
||||
# used in runtimes like AWS lambda.
|
||||
include_parent_directories=True,
|
||||
skip=skip,
|
||||
)
|
||||
dir_stack = [prefix]
|
||||
while dir_stack:
|
||||
dir = dir_stack.pop()
|
||||
|
||||
# Add the dir before its contents
|
||||
dir_info = tarfile.TarInfo(_tarinfo_name(dir))
|
||||
dir_info.type = tarfile.DIRTYPE
|
||||
dir_info.mode = 0o755
|
||||
tar.addfile(dir_info)
|
||||
|
||||
# Sort by name: reproducible & improves compression
|
||||
with os.scandir(dir) as it:
|
||||
entries = sorted(it, key=lambda entry: entry.name)
|
||||
|
||||
new_dirs = []
|
||||
for entry in entries:
|
||||
if entry.is_dir(follow_symlinks=False):
|
||||
new_dirs.append(entry.path)
|
||||
continue
|
||||
|
||||
file_info = tarfile.TarInfo(_tarinfo_name(entry.path))
|
||||
|
||||
s = entry.stat(follow_symlinks=False)
|
||||
|
||||
# Skip existing binary distribution files.
|
||||
id = stat_key(s)
|
||||
if id in files_to_skip:
|
||||
continue
|
||||
|
||||
# Normalize the mode
|
||||
file_info.mode = 0o644 if s.st_mode & 0o100 == 0 else 0o755
|
||||
|
||||
if entry.is_symlink():
|
||||
file_info.type = tarfile.SYMTYPE
|
||||
file_info.linkname = os.readlink(entry.path)
|
||||
tar.addfile(file_info)
|
||||
|
||||
elif entry.is_file(follow_symlinks=False):
|
||||
# Deduplicate hardlinks
|
||||
if s.st_nlink > 1:
|
||||
if id in hardlink_to_tarinfo_name:
|
||||
file_info.type = tarfile.LNKTYPE
|
||||
file_info.linkname = hardlink_to_tarinfo_name[id]
|
||||
tar.addfile(file_info)
|
||||
continue
|
||||
hardlink_to_tarinfo_name[id] = file_info.name
|
||||
|
||||
# If file not yet seen, copy it.
|
||||
file_info.type = tarfile.REGTYPE
|
||||
file_info.size = s.st_size
|
||||
|
||||
with open(entry.path, "rb") as f:
|
||||
tar.addfile(file_info, f)
|
||||
|
||||
dir_stack.extend(reversed(new_dirs)) # we pop, so reverse to stay alphabetical
|
||||
|
||||
|
||||
class ChecksumWriter(io.BufferedIOBase):
|
||||
"""Checksum writer computes a checksum while writing to a file."""
|
||||
|
||||
myfileobj = None
|
||||
|
||||
def __init__(self, fileobj, algorithm=hashlib.sha256):
|
||||
self.fileobj = fileobj
|
||||
self.hasher = algorithm()
|
||||
self.length = 0
|
||||
|
||||
def hexdigest(self):
|
||||
return self.hasher.hexdigest()
|
||||
|
||||
def write(self, data):
|
||||
if isinstance(data, (bytes, bytearray)):
|
||||
length = len(data)
|
||||
else:
|
||||
data = memoryview(data)
|
||||
length = data.nbytes
|
||||
|
||||
if length > 0:
|
||||
self.fileobj.write(data)
|
||||
self.hasher.update(data)
|
||||
|
||||
self.length += length
|
||||
|
||||
return length
|
||||
|
||||
def read(self, size=-1):
|
||||
raise OSError(errno.EBADF, "read() on write-only object")
|
||||
|
||||
def read1(self, size=-1):
|
||||
raise OSError(errno.EBADF, "read1() on write-only object")
|
||||
|
||||
def peek(self, n):
|
||||
raise OSError(errno.EBADF, "peek() on write-only object")
|
||||
|
||||
@property
|
||||
def closed(self):
|
||||
return self.fileobj is None
|
||||
|
||||
def close(self):
|
||||
fileobj = self.fileobj
|
||||
if fileobj is None:
|
||||
return
|
||||
self.fileobj.close()
|
||||
self.fileobj = None
|
||||
|
||||
def flush(self):
|
||||
self.fileobj.flush()
|
||||
|
||||
def fileno(self):
|
||||
return self.fileobj.fileno()
|
||||
|
||||
def rewind(self):
|
||||
raise OSError("Can't rewind while computing checksum")
|
||||
|
||||
def readable(self):
|
||||
return False
|
||||
|
||||
def writable(self):
|
||||
return True
|
||||
|
||||
def seekable(self):
|
||||
return True
|
||||
|
||||
def tell(self):
|
||||
return self.fileobj.tell()
|
||||
|
||||
def seek(self, offset, whence=io.SEEK_SET):
|
||||
# In principle forward seek is possible with b"0" padding,
|
||||
# but this is not implemented.
|
||||
if offset == 0 and whence == io.SEEK_CUR:
|
||||
return
|
||||
raise OSError("Can't seek while computing checksum")
|
||||
|
||||
def readline(self, size=-1):
|
||||
raise OSError(errno.EBADF, "readline() on write-only object")
|
||||
|
||||
|
||||
def _do_create_tarball(tarfile_path: str, binaries_dir: str, buildinfo: dict):
|
||||
with spack.util.archive.gzip_compressed_tarfile(tarfile_path) as (
|
||||
tar,
|
||||
inner_checksum,
|
||||
outer_checksum,
|
||||
):
|
||||
with gzip_compressed_tarfile(tarfile_path) as (tar, inner_checksum, outer_checksum):
|
||||
# Tarball the install prefix
|
||||
tarfile_of_spec_prefix(tar, binaries_dir)
|
||||
|
||||
# Serialize buildinfo for the tarball
|
||||
bstring = syaml.dump(buildinfo, default_flow_style=True).encode("utf-8")
|
||||
tarinfo = tarfile.TarInfo(
|
||||
name=spack.util.archive.default_path_to_name(buildinfo_file_name(binaries_dir))
|
||||
)
|
||||
tarinfo = tarfile.TarInfo(name=_tarinfo_name(buildinfo_file_name(binaries_dir)))
|
||||
tarinfo.type = tarfile.REGTYPE
|
||||
tarinfo.size = len(bstring)
|
||||
tarinfo.mode = 0o644
|
||||
@@ -1900,12 +2047,11 @@ def _extract_inner_tarball(spec, filename, extract_to, signature_required: bool,
|
||||
|
||||
|
||||
def _tar_strip_component(tar: tarfile.TarFile, prefix: str):
|
||||
"""Yield all members of tarfile that start with given prefix, and strip that prefix (including
|
||||
symlinks)"""
|
||||
"""Strip the top-level directory `prefix` from the member names in a tarfile."""
|
||||
# Including trailing /, otherwise we end up with absolute paths.
|
||||
regex = re.compile(re.escape(prefix) + "/*")
|
||||
|
||||
# Only yield members in the package prefix.
|
||||
# Remove the top-level directory from the member (link)names.
|
||||
# Note: when a tarfile is created, relative in-prefix symlinks are
|
||||
# expanded to matching member names of tarfile entries. So, we have
|
||||
# to ensure that those are updated too.
|
||||
@@ -1913,14 +2059,12 @@ def _tar_strip_component(tar: tarfile.TarFile, prefix: str):
|
||||
# them.
|
||||
for m in tar.getmembers():
|
||||
result = regex.match(m.name)
|
||||
if not result:
|
||||
continue
|
||||
assert result is not None
|
||||
m.name = m.name[result.end() :]
|
||||
if m.linkname:
|
||||
result = regex.match(m.linkname)
|
||||
if result:
|
||||
m.linkname = m.linkname[result.end() :]
|
||||
yield m
|
||||
|
||||
|
||||
def extract_tarball(spec, download_result, force=False, timer=timer.NULL_TIMER):
|
||||
@@ -1966,7 +2110,7 @@ def extract_tarball(spec, download_result, force=False, timer=timer.NULL_TIMER):
|
||||
_delete_staged_downloads(download_result)
|
||||
shutil.rmtree(tmpdir)
|
||||
raise e
|
||||
elif 1 <= layout_version <= 2:
|
||||
elif layout_version == 1:
|
||||
# Newer buildcache layout: the .spack file contains just
|
||||
# in the install tree, the signature, if it exists, is
|
||||
# wrapped around the spec.json at the root. If sig verify
|
||||
@@ -1994,10 +2138,8 @@ def extract_tarball(spec, download_result, force=False, timer=timer.NULL_TIMER):
|
||||
try:
|
||||
with closing(tarfile.open(tarfile_path, "r")) as tar:
|
||||
# Remove install prefix from tarfil to extract directly into spec.prefix
|
||||
tar.extractall(
|
||||
path=spec.prefix,
|
||||
members=_tar_strip_component(tar, prefix=_ensure_common_prefix(tar)),
|
||||
)
|
||||
_tar_strip_component(tar, prefix=_ensure_common_prefix(tar))
|
||||
tar.extractall(path=spec.prefix)
|
||||
except Exception:
|
||||
shutil.rmtree(spec.prefix, ignore_errors=True)
|
||||
_delete_staged_downloads(download_result)
|
||||
@@ -2032,47 +2174,20 @@ def extract_tarball(spec, download_result, force=False, timer=timer.NULL_TIMER):
|
||||
|
||||
|
||||
def _ensure_common_prefix(tar: tarfile.TarFile) -> str:
|
||||
# Find the lowest `binary_distribution` file (hard-coded forward slash is on purpose).
|
||||
binary_distribution = min(
|
||||
(
|
||||
e.name
|
||||
for e in tar.getmembers()
|
||||
if e.isfile() and e.name.endswith(".spack/binary_distribution")
|
||||
),
|
||||
key=len,
|
||||
default=None,
|
||||
)
|
||||
# Get the shortest length directory.
|
||||
common_prefix = min((e.name for e in tar.getmembers() if e.isdir()), key=len, default=None)
|
||||
|
||||
if binary_distribution is None:
|
||||
raise ValueError("Tarball is not a Spack package, missing binary_distribution file")
|
||||
if common_prefix is None:
|
||||
raise ValueError("Tarball does not contain a common prefix")
|
||||
|
||||
pkg_path = pathlib.PurePosixPath(binary_distribution).parent.parent
|
||||
|
||||
# Even the most ancient Spack version has required to list the dir of the package itself, so
|
||||
# guard against broken tarballs where `path.parent.parent` is empty.
|
||||
if pkg_path == pathlib.PurePosixPath():
|
||||
raise ValueError("Invalid tarball, missing package prefix dir")
|
||||
|
||||
pkg_prefix = str(pkg_path)
|
||||
|
||||
# Ensure all tar entries are in the pkg_prefix dir, and if they're not, they should be parent
|
||||
# dirs of it.
|
||||
has_prefix = False
|
||||
# Validate that each file starts with the prefix
|
||||
for member in tar.getmembers():
|
||||
stripped = member.name.rstrip("/")
|
||||
if not (
|
||||
stripped.startswith(pkg_prefix) or member.isdir() and pkg_prefix.startswith(stripped)
|
||||
):
|
||||
raise ValueError(f"Tarball contains file {stripped} outside of prefix {pkg_prefix}")
|
||||
if member.isdir() and stripped == pkg_prefix:
|
||||
has_prefix = True
|
||||
if not member.name.startswith(common_prefix):
|
||||
raise ValueError(
|
||||
f"Tarball contains file {member.name} outside of prefix {common_prefix}"
|
||||
)
|
||||
|
||||
# This is technically not required, but let's be defensive about the existence of the package
|
||||
# prefix dir.
|
||||
if not has_prefix:
|
||||
raise ValueError(f"Tarball does not contain a common prefix {pkg_prefix}")
|
||||
|
||||
return pkg_prefix
|
||||
return common_prefix
|
||||
|
||||
|
||||
def install_root_node(spec, unsigned=False, force=False, sha256=None):
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
@@ -217,9 +217,6 @@ def clean_environment():
|
||||
env.unset("R_HOME")
|
||||
env.unset("R_ENVIRON")
|
||||
|
||||
env.unset("LUA_PATH")
|
||||
env.unset("LUA_CPATH")
|
||||
|
||||
# Affects GNU make, can e.g. indirectly inhibit enabling parallel build
|
||||
# env.unset('MAKEFLAGS')
|
||||
|
||||
@@ -555,55 +552,58 @@ def set_package_py_globals(pkg, context: Context = Context.BUILD):
|
||||
"""
|
||||
module = ModuleChangePropagator(pkg)
|
||||
|
||||
m = module
|
||||
|
||||
if context == Context.BUILD:
|
||||
module.std_cmake_args = spack.build_systems.cmake.CMakeBuilder.std_args(pkg)
|
||||
module.std_meson_args = spack.build_systems.meson.MesonBuilder.std_args(pkg)
|
||||
module.std_pip_args = spack.build_systems.python.PythonPipBuilder.std_args(pkg)
|
||||
jobs = determine_number_of_jobs(parallel=pkg.parallel)
|
||||
m.make_jobs = jobs
|
||||
|
||||
jobs = determine_number_of_jobs(parallel=pkg.parallel)
|
||||
module.make_jobs = jobs
|
||||
# TODO: make these build deps that can be installed if not found.
|
||||
m.make = MakeExecutable("make", jobs)
|
||||
m.gmake = MakeExecutable("gmake", jobs)
|
||||
m.ninja = MakeExecutable("ninja", jobs, supports_jobserver=False)
|
||||
# TODO: johnwparent: add package or builder support to define these build tools
|
||||
# for now there is no entrypoint for builders to define these on their
|
||||
# own
|
||||
if sys.platform == "win32":
|
||||
m.nmake = Executable("nmake")
|
||||
m.msbuild = Executable("msbuild")
|
||||
# analog to configure for win32
|
||||
m.cscript = Executable("cscript")
|
||||
|
||||
# TODO: make these build deps that can be installed if not found.
|
||||
module.make = MakeExecutable("make", jobs)
|
||||
module.gmake = MakeExecutable("gmake", jobs)
|
||||
module.ninja = MakeExecutable("ninja", jobs, supports_jobserver=False)
|
||||
# TODO: johnwparent: add package or builder support to define these build tools
|
||||
# for now there is no entrypoint for builders to define these on their
|
||||
# own
|
||||
if sys.platform == "win32":
|
||||
module.nmake = Executable("nmake")
|
||||
module.msbuild = Executable("msbuild")
|
||||
# analog to configure for win32
|
||||
module.cscript = Executable("cscript")
|
||||
# Find the configure script in the archive path
|
||||
# Don't use which for this; we want to find it in the current dir.
|
||||
m.configure = Executable("./configure")
|
||||
|
||||
# Find the configure script in the archive path
|
||||
# Don't use which for this; we want to find it in the current dir.
|
||||
module.configure = Executable("./configure")
|
||||
# Standard CMake arguments
|
||||
m.std_cmake_args = spack.build_systems.cmake.CMakeBuilder.std_args(pkg)
|
||||
m.std_meson_args = spack.build_systems.meson.MesonBuilder.std_args(pkg)
|
||||
m.std_pip_args = spack.build_systems.python.PythonPipBuilder.std_args(pkg)
|
||||
|
||||
# Put spack compiler paths in module scope. (Some packages use it
|
||||
# in setup_run_environment etc, so don't put it context == build)
|
||||
link_dir = spack.paths.build_env_path
|
||||
module.spack_cc = os.path.join(link_dir, pkg.compiler.link_paths["cc"])
|
||||
module.spack_cxx = os.path.join(link_dir, pkg.compiler.link_paths["cxx"])
|
||||
module.spack_f77 = os.path.join(link_dir, pkg.compiler.link_paths["f77"])
|
||||
module.spack_fc = os.path.join(link_dir, pkg.compiler.link_paths["fc"])
|
||||
m.spack_cc = os.path.join(link_dir, pkg.compiler.link_paths["cc"])
|
||||
m.spack_cxx = os.path.join(link_dir, pkg.compiler.link_paths["cxx"])
|
||||
m.spack_f77 = os.path.join(link_dir, pkg.compiler.link_paths["f77"])
|
||||
m.spack_fc = os.path.join(link_dir, pkg.compiler.link_paths["fc"])
|
||||
|
||||
# Useful directories within the prefix are encapsulated in
|
||||
# a Prefix object.
|
||||
module.prefix = pkg.prefix
|
||||
m.prefix = pkg.prefix
|
||||
|
||||
# Platform-specific library suffix.
|
||||
module.dso_suffix = dso_suffix
|
||||
m.dso_suffix = dso_suffix
|
||||
|
||||
def static_to_shared_library(static_lib, shared_lib=None, **kwargs):
|
||||
compiler_path = kwargs.get("compiler", module.spack_cc)
|
||||
compiler_path = kwargs.get("compiler", m.spack_cc)
|
||||
compiler = Executable(compiler_path)
|
||||
|
||||
return _static_to_shared_library(
|
||||
pkg.spec.architecture, compiler, static_lib, shared_lib, **kwargs
|
||||
)
|
||||
|
||||
module.static_to_shared_library = static_to_shared_library
|
||||
m.static_to_shared_library = static_to_shared_library
|
||||
|
||||
module.propagate_changes_to_mro()
|
||||
|
||||
@@ -972,8 +972,8 @@ def __init__(self, *specs: spack.spec.Spec, context: Context) -> None:
|
||||
self.should_set_package_py_globals = (
|
||||
self.should_setup_dependent_build_env | self.should_setup_run_env | UseMode.ROOT
|
||||
)
|
||||
# In a build context, the root needs build-specific globals set.
|
||||
self.needs_build_context = UseMode.ROOT
|
||||
# In a build context, the root and direct build deps need build-specific globals set.
|
||||
self.needs_build_context = UseMode.ROOT | UseMode.BUILDTIME_DIRECT
|
||||
|
||||
def set_all_package_py_globals(self):
|
||||
"""Set the globals in modules of package.py files."""
|
||||
@@ -1032,11 +1032,6 @@ def get_env_modifications(self) -> EnvironmentModifications:
|
||||
if id(spec) in self.nodes_in_subdag:
|
||||
pkg.setup_dependent_run_environment(run_env_mods, spec)
|
||||
pkg.setup_run_environment(run_env_mods)
|
||||
|
||||
external_env = (dspec.extra_attributes or {}).get("environment", {})
|
||||
if external_env:
|
||||
run_env_mods.extend(spack.schema.environment.parse(external_env))
|
||||
|
||||
if self.context == Context.BUILD:
|
||||
# Don't let the runtime environment of comiler like dependencies leak into the
|
||||
# build env
|
||||
@@ -1338,7 +1333,7 @@ def make_stack(tb, stack=None):
|
||||
# don't provide context if the code is actually in the base classes.
|
||||
obj = frame.f_locals["self"]
|
||||
func = getattr(obj, tb.tb_frame.f_code.co_name, "")
|
||||
if func and hasattr(func, "__qualname__"):
|
||||
if func:
|
||||
typename, *_ = func.__qualname__.partition(".")
|
||||
if isinstance(obj, CONTEXT_BASES) and typename not in basenames:
|
||||
break
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
@@ -1,4 +1,4 @@
|
||||
# Copyright 2013-2024 Lawrence Livermore National Security, LLC and other
|
||||
# Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user