zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

spack test (#15702)

Browse Source 
Users can add test() methods to their packages to run smoke tests on
installations with the new `spack test` command (the old `spack test` is
now `spack unit-test`). spack test is environment-aware, so you can
`spack install` an environment and then run `spack test run` to run smoke
tests on all of its packages. Historical test logs can be perused with
`spack test results`. Generic smoke tests for MPI implementations, C,
C++, and Fortran compilers as well as specific smoke tests for 18
packages.

Inside the test method, individual tests can be run separately (and
continue to run best-effort after a test failure) using the `run_test`
method. The `run_test` method encapsulates finding test executables,
running and checking return codes, checking output, and error handling.

This handles the following trickier aspects of testing with direct
support in Spack's package API:

- [x] Caching source or intermediate build files at build time for
      use at test time.
- [x] Test dependencies,
- [x] packages that require a compiler for testing (such as library only
      packages).

See the packaging guide for more details on using Spack testing support.
Included is support for package.py files for virtual packages. This does
not change the Spack interface, but is a major change in internals.

Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
Co-authored-by: wspear <wjspear@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This commit is contained in:
Greg Becker
2020-11-18 02:39:02 -08:00
committed by GitHub
parent b81bbfb6e9
commit 77b2e578ec
131 changed files with 3567 additions and 644 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
lib/spack/docs/contribution_guide.rst
View File

@@ -74,7 +74,7 @@ locally to speed up the review process.
We currently test against Python 2.6, 2.7, and 3.5-3.7 on both macOS and Linux and
perform 3 types of tests:
.. _cmd-spack-test:
.. _cmd-spack-unit-test:
^^^^^^^^^^
Unit Tests
@@ -96,7 +96,7 @@ To run *all* of the unit tests, use:
.. code-block:: console
$ spack test
$ spack unit-test
These tests may take several minutes to complete. If you know you are
only modifying a single Spack feature, you can run subsets of tests at a
@@ -105,13 +105,13 @@ time. For example, this would run all the tests in
.. code-block:: console
$ spack test lib/spack/spack/test/architecture.py
$ spack unit-test lib/spack/spack/test/architecture.py
And this would run the ``test_platform`` test from that file:
.. code-block:: console
$ spack test lib/spack/spack/test/architecture.py::test_platform
$ spack unit-test lib/spack/spack/test/architecture.py::test_platform
This allows you to develop iteratively: make a change, test that change,
make another change, test that change, etc. We use `pytest
@@ -121,29 +121,29 @@ pytest docs
<http://doc.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`_
for more details on test selection syntax.
``spack test`` has a few special options that can help you understand
what tests are available. To get a list of all available unit test
files, run:
``spack unit-test`` has a few special options that can help you
understand what tests are available. To get a list of all available
unit test files, run:
.. command-output:: spack test --list
.. command-output:: spack unit-test --list
:ellipsis: 5
To see a more detailed list of available unit tests, use ``spack test
--list-long``:
To see a more detailed list of available unit tests, use ``spack
unit-test --list-long``:
.. command-output:: spack test --list-long
.. command-output:: spack unit-test --list-long
:ellipsis: 10
And to see the fully qualified names of all tests, use ``--list-names``:
.. command-output:: spack test --list-names
.. command-output:: spack unit-test --list-names
:ellipsis: 5
You can combine these with ``pytest`` arguments to restrict which tests
you want to know about. For example, to see just the tests in
``architecture.py``:
.. command-output:: spack test --list-long lib/spack/spack/test/architecture.py
.. command-output:: spack unit-test --list-long lib/spack/spack/test/architecture.py
You can also combine any of these options with a ``pytest`` keyword
search. See the `pytest usage docs
@@ -151,7 +151,7 @@ search. See the `pytest usage docs
for more details on test selection syntax. For example, to see the names of all tests that have "spec"
or "concretize" somewhere in their names:
.. command-output:: spack test --list-names -k "spec and concretize"
.. command-output:: spack unit-test --list-names -k "spec and concretize"
By default, ``pytest`` captures the output of all unit tests, and it will
print any captured output for failed tests. Sometimes it's helpful to see
@@ -161,7 +161,7 @@ argument to ``pytest``:
.. code-block:: console
$ spack test -s --list-long lib/spack/spack/test/architecture.py::test_platform
$ spack unit-test -s --list-long lib/spack/spack/test/architecture.py::test_platform
Unit tests are crucial to making sure bugs aren't introduced into
Spack. If you are modifying core Spack libraries or adding new
@@ -176,7 +176,7 @@ how to write tests!
You may notice the ``share/spack/qa/run-unit-tests`` script in the
repository. This script is designed for CI. It runs the unit
tests and reports coverage statistics back to Codecov. If you want to
run the unit tests yourself, we suggest you use ``spack test``.
run the unit tests yourself, we suggest you use ``spack unit-test``.
^^^^^^^^^^^^
Flake8 Tests

9
lib/spack/docs/developer_guide.rst
View File

@@ -363,11 +363,12 @@ Developer commands
``spack doc``
^^^^^^^^^^^^^
^^^^^^^^^^^^^^
``spack test``
^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^
``spack unit-test``
^^^^^^^^^^^^^^^^^^^
See the :ref:`contributor guide section <cmd-spack-test>` on ``spack test``.
See the :ref:`contributor guide section <cmd-spack-unit-test>` on
``spack unit-test``.
.. _cmd-spack-python:

5
lib/spack/docs/extensions.rst
View File

@@ -87,11 +87,12 @@ will be available from the command line:
--implicit select specs that are not installed or were installed implicitly
--output OUTPUT where to dump the result
The corresponding unit tests can be run giving the appropriate options to ``spack test``:
The corresponding unit tests can be run giving the appropriate options
to ``spack unit-test``:
.. code-block:: console
$ spack test --extension=scripting
$ spack unit-test --extension=scripting
============================================================== test session starts ===============================================================
platform linux2 -- Python 2.7.15rc1, pytest-3.2.5, py-1.4.34, pluggy-0.4.0

112
lib/spack/docs/packaging_guide.rst
View File

@@ -3948,6 +3948,118 @@ using the ``run_before`` decorator.
.. _file-manipulation:
^^^^^^^^^^^^^
Install Tests
^^^^^^^^^^^^^
.. warning::
The API for adding and running install tests is not yet considered
stable and may change drastically in future releases. Packages with
upstreamed tests will be refactored to match changes to the API.
While build-tests are integrated with the build system, install tests
may be added to Spack packages to be run independently of the install
method.
Install tests may be added by defining a ``test`` method with the following signature:
.. code-block:: python
def test(self):
These tests will be run in an environment set up to provide access to
this package and all of its dependencies, including ``test``-type
dependencies. Inside the ``test`` method, standard python ``assert``
statements and other error reporting mechanisms can be used. Spack
will report any errors as a test failure.
Inside the test method, individual tests can be run separately (and
continue transparently after a test failure) using the ``run_test``
method. The signature for the ``run_test`` method is:
.. code-block:: python
def run_test(self, exe, options=[], expected=[], status=0, installed=False,
purpose='', skip_missing=False, work_dir=None):
This method will operate in ``work_dir`` if one is specified. It will
search for an executable in the ``PATH`` variable named ``exe``, and
if ``installed=True`` it will fail if that executable does not come
from the prefix of the package being tested. If the executable is not
found, it will fail the test unless ``skip_missing`` is set to
``True``. The executable will be run with the options specified, and
the return code will be checked against the ``status`` argument, which
can be an integer or list of integers. Spack will also check that
every string in ``expected`` is a regex matching part of the output of
the executable. The ``purpose`` argument is recorded in the test log
for debugging purposes.
""""""""""""""""""""""""""""""""""""""
Install tests that require compilation
""""""""""""""""""""""""""""""""""""""
Some tests may require access to the compiler with which the package
was built, especially to test library-only packages. To ensure the
compiler is configured as part of the test environment, set the
attribute ``tests_require_compiler = True`` on the package. The
compiler will be available through the canonical environment variables
(``CC``, ``CXX``, ``FC``, ``F77``) in the test environment.
""""""""""""""""""""""""""""""""""""""""""""""""
Install tests that require build-time components
""""""""""""""""""""""""""""""""""""""""""""""""
Some packages cannot be easily tested without components from the
build-time test suite. For those packages, the
``cache_extra_test_sources`` method can be used.
.. code-block:: python
@run_after('install')
def cache_test_sources(self):
srcs = ['./tests/foo.c', './tests/bar.c']
self.cache_extra_test_sources(srcs)
This method will copy the listed methods into the metadata directory
of the package at the end of the install phase of the build. They will
be available to the test method in the directory
``self._extra_tests_path``.
While source files are generally recommended, for many packages
binaries may also technically be cached in this way for later testing.
"""""""""""""""""""""
Running install tests
"""""""""""""""""""""
Install tests can be run using the ``spack test run`` command. The
``spack test run`` command will create a ``test suite`` out of the
specs provided to it, or if no specs are provided it will test all
specs in the active environment, or all specs installed in Spack if no
environment is active. Test suites can be named using the ``--alias``
option; test suites not aliased will use the content hash of their
specs as their name.
Packages to install test can be queried using the ``spack test list``
command, which outputs all installed packages with defined ``test``
methods.
Test suites can be found using the ``spack test find`` command. It
will list all test suites that have been run and have not been removed
using the ``spack test remove`` command. The ``spack test remove``
command will remove tests to declutter the test stage. The ``spack
test results`` command will show results for completed test suites.
The test stage is the working directory for all install tests run with
Spack. By default, Spack uses ``~/.spack/test`` as the test stage. The
test stage can be set in the high-level config:
.. code-block:: yaml
config:
test_stage: /path/to/stage
---------------------------
File manipulation functions
---------------------------