zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs: Update pipeline ci rebuild to add --tests (plus fixed typos) (#32048)

Browse Source
This commit is contained in:
Tamara Dahlgren 2022-09-21 05:23:58 -07:00 committed by GitHub
parent 1313bc99a6
commit f12ececee5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 62 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

84
lib/spack/docs/pipelines.rst
View File

@ -5,9 +5,9 @@
.. _pipelines: .. _pipelines:
========= ============
Pipelines CI Pipelines
========= ============
Spack provides commands that support generating and running automated build Spack provides commands that support generating and running automated build
pipelines designed for Gitlab CI. At the highest level it works like this: pipelines designed for Gitlab CI. At the highest level it works like this:
@ -168,7 +168,7 @@ which specs are up to date and which need to be rebuilt (it's a good idea for ot
reasons as well, but those are out of scope for this discussion). In this case we reasons as well, but those are out of scope for this discussion). In this case we
have disabled it (using ``rebuild-index: False``) because the index would only be have disabled it (using ``rebuild-index: False``) because the index would only be
generated in the artifacts mirror anyway, and consequently would not be available generated in the artifacts mirror anyway, and consequently would not be available
during subesequent pipeline runs. during subsequent pipeline runs.
.. note:: .. note::
With the addition of reproducible builds (#22887) a previously working With the addition of reproducible builds (#22887) a previously working
@ -267,24 +267,64 @@ generated by jobs in the pipeline.
``spack ci rebuild`` ``spack ci rebuild``
^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^
The purpose of the ``spack ci rebuild`` is straightforward: take its assigned The purpose of ``spack ci rebuild`` is straightforward: take its assigned
spec job, check whether the target mirror already has a binary for that spec, spec and ensure a binary of a successful build exists on the target mirror.
and if not, build the spec from source and push the binary to the mirror. To If the binary does not already exist, it is built from source and pushed
accomplish this in a reproducible way, the sub-command prepares a ``spack install`` to the mirror. The associated stand-alone tests are optionally run against
command line to build a single spec in the DAG, saves that command in a the new build. Additionally, files for reproducing the build outside of the
shell script, ``install.sh``, in the current working directory, and then runs CI environment are created to facilitate debugging.
it to install the spec. The shell script is also exported as an artifact to
aid in reproducing the build outside of the CI environment.
If it was necessary to install the spec from source, ``spack ci rebuild`` will If a binary for the spec does not exist on the target mirror, an install
also subsequently create a binary package for the spec and try to push it to the shell script, ``install.sh``, is created and saved in the current working
mirror. directory. The script is run in a job to install the spec from source. The
resulting binary package is pushed to the mirror. If ``cdash`` is configured
for the environment, then the build results will be uploaded to the site.
The ``spack ci rebuild`` sub-command mainly expects its "input" to come either Environment variables and values in the ``gitlab-ci`` section of the
from environment variables or from the ``gitlab-ci`` section of the ``spack.yaml`` ``spack.yaml`` environment file provide inputs to this process. The
environment file. There are two main sources of the environment variables, some two main sources of environment variables are variables written into
are written into ``.gitlab-ci.yml`` by ``spack ci generate``, and some are ``.gitlab-ci.yml`` by ``spack ci generate`` and the GitLab CI runtime.
provided by the GitLab CI runtime. Several key CI pipeline variables are described in
:ref:`ci_environment_variables`.
If the ``--tests`` option is provided, stand-alone tests are performed but
only if the build was successful *and* the package does not appear in the
list of ``broken-tests-packages``. A shell script, ``test.sh``, is created
and run to perform the tests. On completion, test logs are exported as job
artifacts for review and to facilitate debugging. If `cdash` is configured,
test results are also uploaded to the site.
A snippet from an example ``spack.yaml`` file illustrating use of this
option *and* specification of a package with broken tests is given below.
The inclusion of a spec for building ``gptune`` is not shown here. Note
that ``--tests`` is passed to ``spack ci rebuild`` as part of the
``gitlab-ci`` script.
.. code-block:: yaml
gitlab-ci:
script:
- . "./share/spack/setup-env.sh"
- spack --version
- cd ${SPACK_CONCRETE_ENV_DIR}
- spack env activate --without-view .
- spack config add "config:install_tree:projections:${SPACK_JOB_SPEC_PKG_NAME}:'morepadding/{architecture}/{compiler.name}-{compiler.version}/{name}-{version}-{hash}'"
- mkdir -p ${SPACK_ARTIFACTS_ROOT}/user_data
- if [[ -r /mnt/key/intermediate_ci_signing_key.gpg ]]; then spack gpg trust /mnt/key/intermediate_ci_signing_key.gpg; fi
- if [[ -r /mnt/key/spack_public_key.gpg ]]; then spack gpg trust /mnt/key/spack_public_key.gpg; fi
- spack -d ci rebuild --tests > >(tee ${SPACK_ARTIFACTS_ROOT}/user_data/pipeline_out.txt) 2> >(tee ${SPACK_ARTIFACTS_ROOT}/user_data/pipeline_err.txt >&2)
broken-tests-packages:
- gptune
In this case, even if ``gptune`` is successfully built from source, the
pipeline will *not* run its stand-alone tests since the package is listed
under ``broken-tests-packages``.
Spack's cloud pipelines provide actual, up-to-date examples of the CI/CD
configuration and environment files used by Spack. You can find them
under Spack's `stacks
<https://github.com/spack/spack/tree/develop/share/spack/gitlab/cloud_pipelines/stacks>`_ repository directory.
.. _cmd-spack-ci-rebuild-index: .. _cmd-spack-ci-rebuild-index:
@ -447,7 +487,7 @@ Note about "no-op" jobs
^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^
If no specs in an environment need to be rebuilt during a given pipeline run If no specs in an environment need to be rebuilt during a given pipeline run
(meaning all are already up to date on the mirror), a single succesful job (meaning all are already up to date on the mirror), a single successful job
(a NO-OP) is still generated to avoid an empty pipeline (which GitLab (a NO-OP) is still generated to avoid an empty pipeline (which GitLab
considers to be an error). An optional ``service-job-attributes`` section considers to be an error). An optional ``service-job-attributes`` section
can be added to your ``spack.yaml`` where you can provide ``tags`` and can be added to your ``spack.yaml`` where you can provide ``tags`` and
@ -725,7 +765,7 @@ above with ``git checkout ${SPACK_CHECKOUT_VERSION}``.
On the other hand, if you're pointing to a spack repository and branch under your On the other hand, if you're pointing to a spack repository and branch under your
control, there may be no benefit in using the captured ``SPACK_CHECKOUT_VERSION``, control, there may be no benefit in using the captured ``SPACK_CHECKOUT_VERSION``,
and you can instead just clone using the variables you define (``SPACK_REPO`` and you can instead just clone using the variables you define (``SPACK_REPO``
and ``SPACK_REF`` in the example aboves). and ``SPACK_REF`` in the example above).
.. _custom_workflow: .. _custom_workflow: