Docs: Update pipeline ci rebuild to add --tests (plus fixed typos) (#32048)
This commit is contained in:
Tamara Dahlgren
@@ -5,9 +5,9 @@
|
||||
|
||||
.. _pipelines:
|
||||
|
||||
=========
|
||||
Pipelines
|
||||
=========
|
||||
============
|
||||
CI Pipelines
|
||||
============
|
||||
|
||||
Spack provides commands that support generating and running automated build
|
||||
pipelines designed for Gitlab CI. At the highest level it works like this:
|
||||
@@ -168,7 +168,7 @@ which specs are up to date and which need to be rebuilt (it's a good idea for ot
|
||||
reasons as well, but those are out of scope for this discussion). In this case we
|
||||
have disabled it (using ``rebuild-index: False``) because the index would only be
|
||||
generated in the artifacts mirror anyway, and consequently would not be available
|
||||
during subesequent pipeline runs.
|
||||
during subsequent pipeline runs.
|
||||
|
||||
.. note::
|
||||
With the addition of reproducible builds (#22887) a previously working
|
||||
@@ -267,24 +267,64 @@ generated by jobs in the pipeline.
|
||||
``spack ci rebuild``
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The purpose of the ``spack ci rebuild`` is straightforward: take its assigned
|
||||
spec job, check whether the target mirror already has a binary for that spec,
|
||||
and if not, build the spec from source and push the binary to the mirror. To
|
||||
accomplish this in a reproducible way, the sub-command prepares a ``spack install``
|
||||
command line to build a single spec in the DAG, saves that command in a
|
||||
shell script, ``install.sh``, in the current working directory, and then runs
|
||||
it to install the spec. The shell script is also exported as an artifact to
|
||||
aid in reproducing the build outside of the CI environment.
|
||||
The purpose of ``spack ci rebuild`` is straightforward: take its assigned
|
||||
spec and ensure a binary of a successful build exists on the target mirror.
|
||||
If the binary does not already exist, it is built from source and pushed
|
||||
to the mirror. The associated stand-alone tests are optionally run against
|
||||
the new build. Additionally, files for reproducing the build outside of the
|
||||
CI environment are created to facilitate debugging.
|
||||
|
||||
If it was necessary to install the spec from source, ``spack ci rebuild`` will
|
||||
also subsequently create a binary package for the spec and try to push it to the
|
||||
mirror.
|
||||
If a binary for the spec does not exist on the target mirror, an install
|
||||
shell script, ``install.sh``, is created and saved in the current working
|
||||
directory. The script is run in a job to install the spec from source. The
|
||||
resulting binary package is pushed to the mirror. If ``cdash`` is configured
|
||||
for the environment, then the build results will be uploaded to the site.
|
||||
|
||||
The ``spack ci rebuild`` sub-command mainly expects its "input" to come either
|
||||
from environment variables or from the ``gitlab-ci`` section of the ``spack.yaml``
|
||||
environment file. There are two main sources of the environment variables, some
|
||||
are written into ``.gitlab-ci.yml`` by ``spack ci generate``, and some are
|
||||
provided by the GitLab CI runtime.
|
||||
Environment variables and values in the ``gitlab-ci`` section of the
|
||||
``spack.yaml`` environment file provide inputs to this process. The
|
||||
two main sources of environment variables are variables written into
|
||||
``.gitlab-ci.yml`` by ``spack ci generate`` and the GitLab CI runtime.
|
||||
Several key CI pipeline variables are described in
|
||||
:ref:`ci_environment_variables`.
|
||||
|
||||
If the ``--tests`` option is provided, stand-alone tests are performed but
|
||||
only if the build was successful *and* the package does not appear in the
|
||||
list of ``broken-tests-packages``. A shell script, ``test.sh``, is created
|
||||
and run to perform the tests. On completion, test logs are exported as job
|
||||
artifacts for review and to facilitate debugging. If `cdash` is configured,
|
||||
test results are also uploaded to the site.
|
||||
|
||||
A snippet from an example ``spack.yaml`` file illustrating use of this
|
||||
option *and* specification of a package with broken tests is given below.
|
||||
The inclusion of a spec for building ``gptune`` is not shown here. Note
|
||||
that ``--tests`` is passed to ``spack ci rebuild`` as part of the
|
||||
``gitlab-ci`` script.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
gitlab-ci:
|
||||
script:
|
||||
- . "./share/spack/setup-env.sh"
|
||||
- spack --version
|
||||
- cd ${SPACK_CONCRETE_ENV_DIR}
|
||||
- spack env activate --without-view .
|
||||
- spack config add "config:install_tree:projections:${SPACK_JOB_SPEC_PKG_NAME}:'morepadding/{architecture}/{compiler.name}-{compiler.version}/{name}-{version}-{hash}'"
|
||||
- mkdir -p ${SPACK_ARTIFACTS_ROOT}/user_data
|
||||
- if [[ -r /mnt/key/intermediate_ci_signing_key.gpg ]]; then spack gpg trust /mnt/key/intermediate_ci_signing_key.gpg; fi
|
||||
- if [[ -r /mnt/key/spack_public_key.gpg ]]; then spack gpg trust /mnt/key/spack_public_key.gpg; fi
|
||||
- spack -d ci rebuild --tests > >(tee ${SPACK_ARTIFACTS_ROOT}/user_data/pipeline_out.txt) 2> >(tee ${SPACK_ARTIFACTS_ROOT}/user_data/pipeline_err.txt >&2)
|
||||
|
||||
broken-tests-packages:
|
||||
- gptune
|
||||
|
||||
In this case, even if ``gptune`` is successfully built from source, the
|
||||
pipeline will *not* run its stand-alone tests since the package is listed
|
||||
under ``broken-tests-packages``.
|
||||
|
||||
Spack's cloud pipelines provide actual, up-to-date examples of the CI/CD
|
||||
configuration and environment files used by Spack. You can find them
|
||||
under Spack's `stacks
|
||||
<https://github.com/spack/spack/tree/develop/share/spack/gitlab/cloud_pipelines/stacks>`_ repository directory.
|
||||
|
||||
.. _cmd-spack-ci-rebuild-index:
|
||||
|
||||
@@ -447,7 +487,7 @@ Note about "no-op" jobs
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If no specs in an environment need to be rebuilt during a given pipeline run
|
||||
(meaning all are already up to date on the mirror), a single succesful job
|
||||
(meaning all are already up to date on the mirror), a single successful job
|
||||
(a NO-OP) is still generated to avoid an empty pipeline (which GitLab
|
||||
considers to be an error). An optional ``service-job-attributes`` section
|
||||
can be added to your ``spack.yaml`` where you can provide ``tags`` and
|
||||
@@ -725,7 +765,7 @@ above with ``git checkout ${SPACK_CHECKOUT_VERSION}``.
|
||||
On the other hand, if you're pointing to a spack repository and branch under your
|
||||
control, there may be no benefit in using the captured ``SPACK_CHECKOUT_VERSION``,
|
||||
and you can instead just clone using the variables you define (``SPACK_REPO``
|
||||
and ``SPACK_REF`` in the example aboves).
|
||||
and ``SPACK_REF`` in the example above).
|
||||
|
||||
.. _custom_workflow:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user