spack/lib/spack/docs/binary_caches.rst

.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other
   Spack Project Developers. See the top-level COPYRIGHT file for details.

   SPDX-License-Identifier: (Apache-2.0 OR MIT)

.. _binary_caches:

============
Build Caches
============

Some sites may encourage users to set up their own test environments
before carrying out central installations, or some users may prefer to set
up these environments on their own motivation. To reduce the load of
recompiling otherwise identical package specs in different installations,
installed packages can be put into build cache tarballs, pushed to
your Spack mirror and then downloaded and installed by others.

Whenever a mirror provides prebuilt packages, Spack will take these packages
into account during concretization and installation, making ``spack install``
significantly faster.


.. note::

    We use the terms "build cache" and "mirror" often interchangeably. Mirrors
    are used during installation both for sources and prebuilt packages. Build
    caches refer to mirrors that provide prebuilt packages.


----------------------
Creating a build cache
----------------------

Build caches are created via:

.. code-block:: console

    $ spack buildcache push <path/url/mirror name> <spec>

This command takes the locally installed spec and its dependencies, and
creates tarballs of their install prefixes. It also generates metadata files,
signed with GPG. These tarballs and metadata files are then pushed to the
provided binary cache, which can be a local directory or a remote URL.

Here is an example where a build cache is created in a local directory named
"spack-cache", to which we push the "ninja" spec:

.. code-block:: console

    $ spack buildcache push ./spack-cache ninja
    ==> Pushing binary packages to file:///home/spackuser/spack/spack-cache/build_cache

Note that ``ninja`` must be installed locally for this to work.

Once you have a build cache, you can add it as a mirror, discussed next.

---------------------------------------
Finding or installing build cache files
---------------------------------------

To find build caches or install build caches, a Spack mirror must be configured
with:

.. code-block:: console

    $ spack mirror add <name> <url or path>


Both web URLs and local paths on the filesystem can be specified. In the previous
example, you might add the directory "spack-cache" and call it ``mymirror``:


.. code-block:: console

    $ spack mirror add mymirror ./spack-cache


You can see that the mirror is added with ``spack mirror list`` as follows:

.. code-block:: console


    $ spack mirror list
    mymirror           file:///home/spackuser/spack/spack-cache
    spack-public       https://spack-llnl-mirror.s3-us-west-2.amazonaws.com/


At this point, you've create a buildcache, but spack hasn't indexed it, so if
you run ``spack buildcache list`` you won't see any results. You need to index
this new build cache as follows:

.. code-block:: console

    $ spack buildcache update-index ./spack-cache

Now you can use list:

.. code-block:: console

    $  spack buildcache list
    ==> 1 cached build.
    -- linux-ubuntu20.04-skylake / gcc@9.3.0 ------------------------
    ninja@1.10.2

With ``mymirror`` configured and an index available, Spack will automatically
use it during concretization and installation. That means that you can expect
``spack install ninja`` to fetch prebuilt packages from the mirror. Let's
verify by re-installing ninja:

.. code-block:: console

    $ spack uninstall ninja
    $ spack install ninja
    ==> Installing ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz
    ==> Fetching file:///home/spackuser/spack/spack-cache/build_cache/linux-ubuntu20.04-skylake-gcc-9.3.0-ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spec.json.sig
    gpg: Signature made Do 12 Jan 2023 16:01:04 CET
    gpg:                using RSA key 61B82B2B2350E171BD17A1744E3A689061D57BF6
    gpg: Good signature from "example (GPG created for Spack) <example@example.com>" [ultimate]
    ==> Fetching file:///home/spackuser/spack/spack-cache/build_cache/linux-ubuntu20.04-skylake/gcc-9.3.0/ninja-1.10.2/linux-ubuntu20.04-skylake-gcc-9.3.0-ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spack
    ==> Extracting ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz from binary cache
    ==> ninja: Successfully installed ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz
    Search: 0.00s.  Fetch: 0.17s.  Install: 0.12s.  Total: 0.29s
    [+] /home/harmen/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz


It worked! You've just completed a full example of creating a build cache with
a spec of interest, adding it as a mirror, updating its index, listing the contents,
and finally, installing from it.

By default Spack falls back to building from sources when the mirror is not available
or when the package is simply not already available. To force Spack to only install
prebuilt packages, you can use

.. code-block:: console

   $ spack install --use-buildcache only <package>

For example, to combine all of the commands above to add the E4S build cache
and then install from it exclusively, you would do:

.. code-block:: console

    $ spack mirror add E4S https://cache.e4s.io
    $ spack buildcache keys --install --trust
    $ spack install --use-buildcache only <package>

We use ``--install`` and ``--trust`` to say that we are installing keys to our
keyring, and trusting all downloaded keys.


^^^^^^^^^^^^^^^^^^^^^^^^^^^^
List of popular build caches
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* `Extreme-scale Scientific Software Stack (E4S) <https://e4s-project.github.io/>`_: `build cache <https://oaciss.uoregon.edu/e4s/inventory.html>`_


-----------------------------------------
OCI / Docker V2 registries as build cache
-----------------------------------------

Spack can also use OCI or Docker V2 registries such as Dockerhub, Quay.io,
Github Packages, GitLab Container Registry, JFrog Artifactory, and others
as build caches. This is a convenient way to share binaries using public
infrastructure, or to cache Spack built binaries in Github Actions and
GitLab CI.

To get started, configure an OCI mirror using ``oci://`` as the scheme,
and optionally specify a username and password (or personal access token):

.. code-block:: console

    $ spack mirror add --oci-username username --oci-password password my_registry oci://example.com/my_image

Spack follows the naming conventions of Docker, with Dockerhub as the default
registry. To use Dockerhub, you can omit the registry domain:

.. code-block:: console

    $ spack mirror add --oci-username username --oci-password password my_registry oci://username/my_image

From here, you can use the mirror as any other build cache:

.. code-block:: console

    $ spack buildcache push my_registry <specs...>  # push to the registry
    $ spack install <specs...> # install from the registry

A unique feature of buildcaches on top of OCI registries is that it's incredibly
easy to generate get a runnable container image with the binaries installed. This
is a great way to make applications available to users without requiring them to
install Spack -- all you need is Docker, Podman or any other OCI-compatible container
runtime.

To produce container images, all you need to do is add the ``--base-image`` flag
when pushing to the build cache:

.. code-block:: console

    $ spack buildcache push --base-image ubuntu:20.04 my_registry ninja
    Pushed to example.com/my_image:ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spack

    $ docker run -it example.com/my_image:ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spack
    root@e4c2b6f6b3f4:/# ninja --version
    1.11.1

If ``--base-image`` is not specified, distroless images are produced. In practice,
you won't be able to run these as containers, since they don't come with libc and
other system dependencies. However, they are still compatible with tools like
``skopeo``, ``podman``, and ``docker`` for pulling and pushing.

.. note::
    The docker ``overlayfs2`` storage driver is limited to 128 layers, above which a
    ``max depth exceeded`` error may be produced when pulling the image. There
    are `alternative drivers <https://docs.docker.com/storage/storagedriver/>`_.

------------------------------------
Using a buildcache in GitHub Actions
------------------------------------

GitHub Actions is a popular CI/CD platform for building and testing software,
but each CI job has limited resources, making from source builds too slow for
many applications. Spack build caches can be used to share binaries between CI
runs, speeding up CI significantly.

A typical workflow is to include a ``spack.yaml`` environment in your repository
that specifies the packages to install:

.. code-block:: yaml

    spack:
      specs: [pkg-x, pkg-y]
    packages:
      all:
        require: target=x86_64_v2
    mirrors:
      github_packages: oci://ghcr.io/<user>/<repo>

And a GitHub action that sets up Spack, installs packages from the build cache
or from sources, and pushes newly built binaries to the build cache:

.. code-block:: yaml

    name: Install Spack packages

    on: push

    env:
      SPACK_COLOR: always

    jobs:
      example:
        runs-on: ubuntu-22.04
        steps:
        - name: Checkout
          uses: actions/checkout@v3

        - name: Install Spack
          run: |
            git clone --depth=1 https://github.com/spack/spack.git
            echo "$PWD/spack/bin/" >> "$GITHUB_PATH"

        - name: Concretize
          run: spack -e . concretize

        - name: Install
          run: spack -e . install --no-check-signature --fail-fast

        - name: Push to buildcache
          run: |
            spack -e . mirror set --oci-username <user> --oci-password "${{ secrets.GITHUB_TOKEN }}" github_packages
            spack -e . buildcache push --base-image ubuntu:22.04 --unsigned --update-index github_packages
            if: always()

The first time this action runs, it will build the packages from source and
push them to the build cache. Subsequent runs will pull the binaries from the
build cache. The concretizer will ensure that prebuilt binaries are favored
over source builds.

The build cache entries appear in the GitHub Packages section of your repository,
and contain instructions for pulling and running them with ``docker`` or ``podman``.

----------
Relocation
----------

Initial build and later installation do not necessarily happen at the same
location. Spack provides a relocation capability and corrects for RPATHs and
non-relocatable scripts. However, many packages compile paths into binary
artifacts directly. In such cases, the build instructions of this package would
need to be adjusted for better re-locatability.

.. _cmd-spack-buildcache:

--------------------
``spack buildcache``
--------------------

^^^^^^^^^^^^^^^^^^^^^^^^^^^
``spack buildcache push``
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Create tarball of installed Spack package and all dependencies.
Tarballs are checksummed and signed if gpg2 is available.
Places them in a directory ``build_cache`` that can be copied to a mirror.
Commands like ``spack buildcache install`` will search Spack mirrors for build_cache to get the list of build caches.

==============  ========================================================================================================================
Arguments       Description
==============  ========================================================================================================================
``<specs>``     list of partial specs or hashes with a leading ``/`` to match from installed packages and used for creating build caches
``-d <path>``   directory in which ``build_cache`` directory is created, defaults to ``.``
``-f``          overwrite ``.spack`` file in ``build_cache`` directory if it exists
``-k <key>``    the key to sign package with. In the case where multiple keys exist, the package will be unsigned unless ``-k`` is used.
``-r``          make paths in binaries relative before creating tarball
``-y``          answer yes to all create unsigned ``build_cache`` questions
==============  ========================================================================================================================

^^^^^^^^^^^^^^^^^^^^^^^^^
``spack buildcache list``
^^^^^^^^^^^^^^^^^^^^^^^^^

Retrieves all specs for build caches available on a Spack mirror.

==============  =====================================================================================
Arguments       Description
==============  =====================================================================================
``<specs>``     list of partial package specs to be matched against specs downloaded for build caches
==============  =====================================================================================

E.g. ``spack buildcache list gcc`` with print only commands to install ``gcc`` package(s)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``spack buildcache install``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Retrieves all specs for build caches available on a Spack mirror and installs build caches
with specs matching the specs input.

==============  ==============================================================================================
Arguments       Description
==============  ==============================================================================================
``<specs>``     list of partial package specs or hashes with a leading ``/`` to be installed from build caches
``-f``          remove install directory if it exists before unpacking tarball
``-y``          answer yes to all to don't verify package with gpg questions
==============  ==============================================================================================

^^^^^^^^^^^^^^^^^^^^^^^^^
``spack buildcache keys``
^^^^^^^^^^^^^^^^^^^^^^^^^

List public keys available on Spack mirror.

=========  ==============================================
Arguments  Description
=========  ==============================================
``-i``     trust the keys downloaded with prompt for each
``-y``     answer yes to all trust all keys downloaded
=========  ==============================================