Docs: Update stand-alone test information (#44755)
Update and slightly reorganize stand-alone test information to include new and improved examples and more links that can be used in PR feedback.
This commit is contained in:
Tamara Dahlgren
GitHub
parent
884a0a392d
commit
baf82c0245
@ -5194,12 +5194,6 @@ installed executable. The check is implemented as follows:
|
||||
reframe = Executable(self.prefix.bin.reframe)
|
||||
reframe("-l")
|
||||
|
||||
.. warning::
|
||||
|
||||
The API for adding tests is not yet considered stable and may change
|
||||
in future releases.
|
||||
|
||||
|
||||
""""""""""""""""""""""""""""""""
|
||||
Checking build-time test results
|
||||
""""""""""""""""""""""""""""""""
|
||||
@ -5237,38 +5231,42 @@ be left in the build stage directory as illustrated below:
|
||||
Stand-alone tests
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
While build-time tests are integrated with the build process, stand-alone
|
||||
While build-time tests are integrated with the installation process, stand-alone
|
||||
tests are expected to run days, weeks, even months after the software is
|
||||
installed. The goal is to provide a mechanism for gaining confidence that
|
||||
packages work as installed **and** *continue* to work as the underlying
|
||||
software evolves. Packages can add and inherit stand-alone tests. The
|
||||
`spack test`` command is used to manage stand-alone testing.
|
||||
``spack test`` command is used for stand-alone testing.
|
||||
|
||||
.. note::
|
||||
.. admonition:: Stand-alone test methods should complete within a few minutes.
|
||||
|
||||
Execution speed is important since these tests are intended to quickly
|
||||
assess whether installed specs work on the system. Consequently, they
|
||||
should run relatively quickly -- as in on the order of at most a few
|
||||
minutes -- while ideally executing all, or at least key aspects of the
|
||||
installed software.
|
||||
assess whether installed specs work on the system. Spack cannot spare
|
||||
resources for more extensive testing of packages included in CI stacks.
|
||||
|
||||
.. note::
|
||||
|
||||
Failing stand-alone tests indicate problems with the installation and,
|
||||
therefore, there is no reason to proceed with more resource-intensive
|
||||
tests until those have been investigated.
|
||||
|
||||
Passing stand-alone tests indicate that more thorough testing, such
|
||||
as running extensive unit or regression tests, or tests that run at
|
||||
scale can proceed without wasting resources on a problematic installation.
|
||||
Consequently, stand-alone tests should run relatively quickly -- as in
|
||||
on the order of at most a few minutes -- while testing at least key aspects
|
||||
of the installed software. Save more extensive testing for other tools.
|
||||
|
||||
Tests are defined in the package using methods with names beginning ``test_``.
|
||||
This allows Spack to support multiple independent checks, or parts. Files
|
||||
needed for testing, such as source, data, and expected outputs, may be saved
|
||||
from the build and or stored with the package in the repository. Regardless
|
||||
of origin, these files are automatically copied to the spec's test stage
|
||||
directory prior to execution of the test method(s). Spack also provides some
|
||||
helper functions to facilitate processing.
|
||||
directory prior to execution of the test method(s). Spack also provides helper
|
||||
functions to facilitate common processing.
|
||||
|
||||
.. tip::
|
||||
|
||||
**The status of stand-alone tests can be used to guide follow-up testing efforts.**
|
||||
|
||||
Passing stand-alone tests justify performing more thorough testing, such
|
||||
as running extensive unit or regression tests or tests that run at scale,
|
||||
when available. These tests are outside of the scope of Spack packaging.
|
||||
|
||||
Failing stand-alone tests indicate problems with the installation and,
|
||||
therefore, no reason to proceed with more resource-intensive tests until
|
||||
the failures have been investigated.
|
||||
|
||||
.. _configure-test-stage:
|
||||
|
||||
@ -5276,30 +5274,26 @@ helper functions to facilitate processing.
|
||||
Configuring the test stage directory
|
||||
""""""""""""""""""""""""""""""""""""
|
||||
|
||||
Stand-alone tests utilize a test stage directory for building, running,
|
||||
and tracking results in the same way Spack uses a build stage directory.
|
||||
The default test stage root directory, ``~/.spack/test``, is defined in
|
||||
:ref:`etc/spack/defaults/config.yaml <config-yaml>`. This location is
|
||||
customizable by adding or changing the ``test_stage`` path in the high-level
|
||||
``config`` of the appropriate ``config.yaml`` file such that:
|
||||
Stand-alone tests utilize a test stage directory to build, run, and track
|
||||
tests in the same way Spack uses a build stage directory to install software.
|
||||
The default test stage root directory, ``$HOME/.spack/test``, is defined in
|
||||
:ref:`config.yaml <config-yaml>`. This location is customizable by adding or
|
||||
changing the ``test_stage`` path such that:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
config:
|
||||
test_stage: /path/to/test/stage
|
||||
|
||||
Packages can use the ``self.test_suite.stage`` property to access this setting.
|
||||
Other package properties that provide access to spec-specific subdirectories
|
||||
and files are described in :ref:`accessing staged files <accessing-files>`.
|
||||
Packages can use the ``self.test_suite.stage`` property to access the path.
|
||||
|
||||
.. note::
|
||||
.. admonition:: Each spec being tested has its own test stage directory.
|
||||
|
||||
The test stage path is the root directory for the **entire suite**.
|
||||
In other words, it is the root directory for **all specs** being
|
||||
tested by the ``spack test run`` command. Each spec gets its own
|
||||
stage subdirectory. Use ``self.test_suite.test_dir_for_spec(self.spec)``
|
||||
to access the spec-specific test stage directory.
|
||||
The ``config:test_stage`` option is the path to the root of a
|
||||
**test suite**'s stage directories.
|
||||
|
||||
Other package properties that provide paths to spec-specific subdirectories
|
||||
and files are described in :ref:`accessing-files`.
|
||||
|
||||
.. _adding-standalone-tests:
|
||||
|
||||
@ -5312,61 +5306,144 @@ Test recipes are defined in the package using methods with names beginning
|
||||
Each method has access to the information Spack tracks on the package, such
|
||||
as options, compilers, and dependencies, supporting the customization of tests
|
||||
to the build. Standard python ``assert`` statements and other error reporting
|
||||
mechanisms are available. Such exceptions are automatically caught and reported
|
||||
mechanisms can be used. These exceptions are automatically caught and reported
|
||||
as test failures.
|
||||
|
||||
Each test method is an implicit test part named by the method and whose
|
||||
purpose is the method's docstring. Providing a purpose gives context for
|
||||
aiding debugging. A test method may contain embedded test parts. Spack
|
||||
outputs the test name and purpose prior to running each test method and
|
||||
any embedded test parts. For example, ``MyPackage`` below provides two basic
|
||||
examples of installation tests: ``test_always_fails`` and ``test_example``.
|
||||
As the name indicates, the first always fails. The second simply runs the
|
||||
installed example.
|
||||
Each test method is an *implicit test part* named by the method. Its purpose
|
||||
is the method's docstring. Providing a meaningful purpose for the test gives
|
||||
context that can aid debugging. Spack outputs both the name and purpose at the
|
||||
start of test execution so it's also important that the docstring/purpose be
|
||||
brief.
|
||||
|
||||
.. tip::
|
||||
|
||||
We recommend naming test methods so it is clear *what* is being tested.
|
||||
For example, if a test method is building and or running an executable
|
||||
called ``example``, then call the method ``test_example``. This, together
|
||||
with a similarly meaningful test purpose, will aid test comprehension,
|
||||
debugging, and maintainability.
|
||||
|
||||
Stand-alone tests run in an environment that provides access to information
|
||||
on the installed software, such as build options, dependencies, and compilers.
|
||||
Build options and dependencies are accessed using the same spec checks used
|
||||
by build recipes. Examples of checking :ref:`variant settings <variants>` and
|
||||
:ref:`spec constraints <testing-specs>` can be found at the provided links.
|
||||
|
||||
.. admonition:: Spack automatically sets up the test stage directory and environment.
|
||||
|
||||
Spack automatically creates the test stage directory and copies
|
||||
relevant files *prior to* running tests. It can also ensure build
|
||||
dependencies are available **if** necessary.
|
||||
|
||||
The path to the test stage is configurable (see :ref:`configure-test-stage`).
|
||||
|
||||
Files that Spack knows to copy are those saved from the build (see
|
||||
:ref:`cache_extra_test_sources`) and those added to the package repository
|
||||
(see :ref:`cache_custom_files`).
|
||||
|
||||
Spack will use the value of the ``test_requires_compiler`` property to
|
||||
determine whether it needs to also set up build dependencies (see
|
||||
:ref:`test-build-tests`).
|
||||
|
||||
The ``MyPackage`` package below provides two basic test examples:
|
||||
``test_example`` and ``test_example2``. The first runs the installed
|
||||
``example`` and ensures its output contains an expected string. The second
|
||||
runs ``example2`` without checking output so is only concerned with confirming
|
||||
the executable runs successfully. If the installed spec is not expected to have
|
||||
``example2``, then the check at the top of the method will raise a special
|
||||
``SkipTest`` exception, which is captured to facilitate reporting skipped test
|
||||
parts to tools like CDash.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
class MyPackage(Package):
|
||||
...
|
||||
|
||||
def test_always_fails(self):
|
||||
"""use assert to always fail"""
|
||||
assert False
|
||||
|
||||
def test_example(self):
|
||||
"""run installed example"""
|
||||
"""ensure installed example works"""
|
||||
expected = "Done."
|
||||
example = which(self.prefix.bin.example)
|
||||
example()
|
||||
|
||||
# Capture stdout and stderr from running the Executable
|
||||
# and check that the expected output was produced.
|
||||
out = example(output=str.split, error=str.split)
|
||||
assert expected in out, f"Expected '{expected}' in the output"
|
||||
|
||||
def test_example2(self):
|
||||
"""run installed example2"""
|
||||
if self.spec.satisfies("@:1.0"):
|
||||
# Raise SkipTest to ensure flagging the test as skipped for
|
||||
# test reporting purposes.
|
||||
raise SkipTest("Test is only available for v1.1 on")
|
||||
|
||||
example2 = which(self.prefix.bin.example2)
|
||||
example2()
|
||||
|
||||
Output showing the identification of each test part after running the tests
|
||||
is illustrated below.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack test run --alias mypackage mypackage@1.0
|
||||
$ spack test run --alias mypackage mypackage@2.0
|
||||
==> Spack test mypackage
|
||||
...
|
||||
$ spack test results -l mypackage
|
||||
==> Results for test suite 'mypackage':
|
||||
...
|
||||
==> [2023-03-10-16:03:56.625204] test: test_always_fails: use assert to always fail
|
||||
==> [2024-03-10-16:03:56.625439] test: test_example: ensure installed example works
|
||||
...
|
||||
FAILED
|
||||
==> [2023-03-10-16:03:56.625439] test: test_example: run installed example
|
||||
PASSED: MyPackage::test_example
|
||||
==> [2024-03-10-16:03:56.625439] test: test_example2: run installed example2
|
||||
...
|
||||
PASSED
|
||||
PASSED: MyPackage::test_example2
|
||||
|
||||
.. admonition:: Do NOT implement tests that must run in the installation prefix.
|
||||
|
||||
.. note::
|
||||
Use of the package spec's installation prefix for building and running
|
||||
tests is **strongly discouraged**. Doing so causes permission errors for
|
||||
shared spack instances *and* facilities that install the software in
|
||||
read-only file systems or directories.
|
||||
|
||||
If ``MyPackage`` were a recipe for a library, the tests should build
|
||||
an example or test program that is then executed.
|
||||
Instead, start these test methods by explicitly copying the needed files
|
||||
from the installation prefix to the test stage directory. Note the test
|
||||
stage directory is the current directory when the test is executed with
|
||||
the ``spack test run`` command.
|
||||
|
||||
A test method can include test parts using the ``test_part`` context manager.
|
||||
Each part is treated as an independent check to allow subsequent test parts
|
||||
to execute even after a test part fails.
|
||||
.. admonition:: Test methods for library packages should build test executables.
|
||||
|
||||
.. _test-part:
|
||||
Stand-alone tests for library packages *should* build test executables
|
||||
that utilize the *installed* library. Doing so ensures the tests follow
|
||||
a similar build process that users of the library would follow.
|
||||
|
||||
For more information on how to do this, see :ref:`test-build-tests`.
|
||||
|
||||
.. tip::
|
||||
|
||||
If you want to see more examples from packages with stand-alone tests, run
|
||||
``spack pkg grep "def\stest" | sed "s/\/package.py.*//g" | sort -u``
|
||||
from the command line to get a list of the packages.
|
||||
|
||||
.. _adding-standalone-test-parts:
|
||||
|
||||
"""""""""""""""""""""""""""""
|
||||
Adding stand-alone test parts
|
||||
"""""""""""""""""""""""""""""
|
||||
|
||||
Sometimes dependencies between steps of a test lend themselves to being
|
||||
broken into parts. Tracking the pass/fail status of each part may aid
|
||||
debugging. Spack provides a ``test_part`` context manager for use within
|
||||
test methods.
|
||||
|
||||
Each test part is independently run, tracked, and reported. Test parts are
|
||||
executed in the order they appear. If one fails, subsequent test parts are
|
||||
still performed even if they would also fail. This allows tools like CDash
|
||||
to track and report the status of test parts across runs. The pass/fail status
|
||||
of the enclosing test is derived from the statuses of the embedded test parts.
|
||||
|
||||
.. admonition:: Test method and test part names **must** be unique.
|
||||
|
||||
Test results reporting requires that test methods and embedded test parts
|
||||
within a package have unique names.
|
||||
|
||||
The signature for ``test_part`` is:
|
||||
|
||||
@ -5388,40 +5465,68 @@ where each argument has the following meaning:
|
||||
* ``work_dir`` is the path to the directory in which the test will run.
|
||||
|
||||
The default of ``None``, or ``"."``, corresponds to the the spec's test
|
||||
stage (i.e., ``self.test_suite.test_dir_for_spec(self.spec)``.
|
||||
stage (i.e., ``self.test_suite.test_dir_for_spec(self.spec)``).
|
||||
|
||||
.. admonition:: Tests should **not** run under the installation directory.
|
||||
.. admonition:: Start test part names with the name of the enclosing test.
|
||||
|
||||
Use of the package spec's installation directory for building and running
|
||||
tests is **strongly** discouraged. Doing so causes permission errors for
|
||||
shared spack instances *and* facilities that install the software in
|
||||
read-only file systems or directories.
|
||||
We **highly recommend** starting the names of test parts with the name
|
||||
of the enclosing test. Doing so helps with the comprehension, readability
|
||||
and debugging of test results.
|
||||
|
||||
Suppose ``MyPackage`` actually installs two examples we want to use for tests.
|
||||
These checks can be implemented as separate checks or, as illustrated below,
|
||||
embedded test parts.
|
||||
Suppose ``MyPackage`` installs multiple executables that need to run in a
|
||||
specific order since the outputs from one are inputs of others. Further suppose
|
||||
we want to add an integration test that runs the executables in order. We can
|
||||
accomplish this goal by implementing a stand-alone test method consisting of
|
||||
test parts for each executable as follows:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
class MyPackage(Package):
|
||||
...
|
||||
|
||||
def test_example(self):
|
||||
"""run installed examples"""
|
||||
for example in ["ex1", "ex2"]:
|
||||
with test_part(
|
||||
self,
|
||||
f"test_example_{example}",
|
||||
purpose=f"run installed {example}",
|
||||
):
|
||||
exe = which(join_path(self.prefix.bin, example))
|
||||
exe()
|
||||
def test_series(self):
|
||||
"""run setup, perform, and report"""
|
||||
|
||||
In this case, there will be an implicit test part for ``test_example``
|
||||
and separate sub-parts for ``ex1`` and ``ex2``. The second sub-part
|
||||
will be executed regardless of whether the first passes. The test
|
||||
log for a run where the first executable fails and the second passes
|
||||
is illustrated below.
|
||||
with test_part(self, "test_series_setup", purpose="setup operation"):
|
||||
exe = which(self.prefix.bin.setup))
|
||||
exe()
|
||||
|
||||
with test_part(self, "test_series_run", purpose="perform operation"):
|
||||
exe = which(self.prefix.bin.run))
|
||||
exe()
|
||||
|
||||
with test_part(self, "test_series_report", purpose="generate report"):
|
||||
exe = which(self.prefix.bin.report))
|
||||
exe()
|
||||
|
||||
The result is ``test_series`` runs the following executable in order: ``setup``,
|
||||
``run``, and ``report``. In this case no options are passed to any of the
|
||||
executables and no outputs from running them are checked. Consequently, the
|
||||
implementation could be simplified with a for-loop as follows:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
class MyPackage(Package):
|
||||
...
|
||||
|
||||
def test_series(self):
|
||||
"""execute series setup, run, and report"""
|
||||
|
||||
for exe, reason in [
|
||||
("setup", "setup operation"),
|
||||
("run", "perform operation"),
|
||||
("report", "generate report")
|
||||
]:
|
||||
with test_part(self, f"test_series_{exe}", purpose=reason):
|
||||
exe = which(self.prefix.bin.join(exe))
|
||||
exe()
|
||||
|
||||
In both cases, since we're using a context manager, each test part in
|
||||
``test_series`` will execute regardless of the status of the other test
|
||||
parts.
|
||||
|
||||
Now let's look at the output from running the stand-alone tests where
|
||||
the second test part, ``test_series_run``, fails.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
@ -5431,50 +5536,67 @@ is illustrated below.
|
||||
$ spack test results -l mypackage
|
||||
==> Results for test suite 'mypackage':
|
||||
...
|
||||
==> [2023-03-10-16:03:56.625204] test: test_example: run installed examples
|
||||
==> [2023-03-10-16:03:56.625439] test: test_example_ex1: run installed ex1
|
||||
==> [2024-03-10-16:03:56.625204] test: test_series: execute series setup, run, and report
|
||||
==> [2024-03-10-16:03:56.625439] test: test_series_setup: setup operation
|
||||
...
|
||||
FAILED
|
||||
==> [2023-03-10-16:03:56.625555] test: test_example_ex2: run installed ex2
|
||||
PASSED: MyPackage::test_series_setup
|
||||
==> [2024-03-10-16:03:56.625555] test: test_series_run: perform operation
|
||||
...
|
||||
PASSED
|
||||
FAILED: MyPackage::test_series_run
|
||||
==> [2024-03-10-16:03:57.003456] test: test_series_report: generate report
|
||||
...
|
||||
FAILED: MyPackage::test_series_report
|
||||
FAILED: MyPackage::test_series
|
||||
...
|
||||
|
||||
.. warning::
|
||||
Since test parts depended on the success of previous parts, we see that the
|
||||
failure of one results in the failure of subsequent checks and the overall
|
||||
result of the test method, ``test_series``, is failure.
|
||||
|
||||
Test results reporting requires that each test method and embedded
|
||||
test part for a package have a unique name.
|
||||
.. tip::
|
||||
|
||||
Stand-alone tests run in an environment that provides access to information
|
||||
Spack has on how the software was built, such as build options, dependencies,
|
||||
and compilers. Build options and dependencies are accessed with the normal
|
||||
spec checks. Examples of checking :ref:`variant settings <variants>` and
|
||||
:ref:`spec constraints <testing-specs>` can be found at the provided links.
|
||||
Accessing compilers in stand-alone tests that are used by the build requires
|
||||
setting a package property as described :ref:`below <test-compilation>`.
|
||||
If you want to see more examples from packages using ``test_part``, run
|
||||
``spack pkg grep "test_part(" | sed "s/\/package.py.*//g" | sort -u``
|
||||
from the command line to get a list of the packages.
|
||||
|
||||
.. _test-build-tests:
|
||||
|
||||
.. _test-compilation:
|
||||
"""""""""""""""""""""""""""""""""""""
|
||||
Building and running test executables
|
||||
"""""""""""""""""""""""""""""""""""""
|
||||
|
||||
"""""""""""""""""""""""""
|
||||
Enabling test compilation
|
||||
"""""""""""""""""""""""""
|
||||
.. admonition:: Re-use build-time sources and (small) input data sets when possible.
|
||||
|
||||
If you want to build and run binaries in tests, then you'll need to tell
|
||||
Spack to load the package's compiler configuration. This is accomplished
|
||||
by setting the package's ``test_requires_compiler`` property to ``True``.
|
||||
We **highly recommend** re-using build-time test sources and pared down
|
||||
input files for testing installed software. These files are easier
|
||||
to keep synchronized with software capabilities when they reside
|
||||
within the software's repository. More information on saving files from
|
||||
the installation process can be found at :ref:`cache_extra_test_sources`.
|
||||
|
||||
Setting the property to ``True`` ensures access to the compiler through
|
||||
canonical environment variables (e.g., ``CC``, ``CXX``, ``FC``, ``F77``).
|
||||
It also gives access to build dependencies like ``cmake`` through their
|
||||
``spec objects`` (e.g., ``self.spec["cmake"].prefix.bin.cmake``).
|
||||
If that is not possible, you can add test-related files to the package
|
||||
repository (see :ref:`cache_custom_files`). It will be important to
|
||||
remember to maintain them so they work across listed or supported versions
|
||||
of the package.
|
||||
|
||||
.. note::
|
||||
Packages that build libraries are good examples of cases where you'll want
|
||||
to build test executables from the installed software before running them.
|
||||
Doing so requires you to let Spack know it needs to load the package's
|
||||
compiler configuration. This is accomplished by setting the package's
|
||||
``test_requires_compiler`` property to ``True``.
|
||||
|
||||
The ``test_requires_compiler`` property should be added at the top of
|
||||
the package near other attributes, such as the ``homepage`` and ``url``.
|
||||
.. admonition:: ``test_requires_compiler = True`` is required to build test executables.
|
||||
|
||||
Below illustrates using this feature to compile an example.
|
||||
Setting the property to ``True`` ensures access to the compiler through
|
||||
canonical environment variables (e.g., ``CC``, ``CXX``, ``FC``, ``F77``).
|
||||
It also gives access to build dependencies like ``cmake`` through their
|
||||
``spec objects`` (e.g., ``self.spec["cmake"].prefix.bin.cmake``).
|
||||
|
||||
Be sure to add the property at the top of the package class under other
|
||||
properties like the ``homepage``.
|
||||
|
||||
The example below, which ignores how ``cxx-example.cpp`` is acquired,
|
||||
illustrates the basic process of compiling a test executable using the
|
||||
installed library before running it.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@ -5498,28 +5620,22 @@ Below illustrates using this feature to compile an example.
|
||||
cxx_example = which(exe)
|
||||
cxx_example()
|
||||
|
||||
Typically the files used to build and or run test executables are either
|
||||
cached from the installation (see :ref:`cache_extra_test_sources`) or added
|
||||
to the package repository (see :ref:`cache_custom_files`). There is nothing
|
||||
preventing the use of both.
|
||||
|
||||
.. _cache_extra_test_sources:
|
||||
|
||||
"""""""""""""""""""""""
|
||||
Saving build-time files
|
||||
"""""""""""""""""""""""
|
||||
""""""""""""""""""""""""""""""""""""
|
||||
Saving build- and install-time files
|
||||
""""""""""""""""""""""""""""""""""""
|
||||
|
||||
.. note::
|
||||
|
||||
We highly recommend re-using build-time test sources and pared down
|
||||
input files for testing installed software. These files are easier
|
||||
to keep synchronized with software capabilities since they reside
|
||||
within the software's repository.
|
||||
|
||||
If that is not possible, you can add test-related files to the package
|
||||
repository (see :ref:`adding custom files <cache_custom_files>`). It
|
||||
will be important to maintain them so they work across listed or supported
|
||||
versions of the package.
|
||||
|
||||
You can use the ``cache_extra_test_sources`` helper to copy directories
|
||||
and or files from the source build stage directory to the package's
|
||||
installation directory.
|
||||
You can use the ``cache_extra_test_sources`` helper routine to copy
|
||||
directories and or files from the source build stage directory to the
|
||||
package's installation directory. Spack will automatically copy these
|
||||
files for you when it sets up the test stage directory and before it
|
||||
begins running the tests.
|
||||
|
||||
The signature for ``cache_extra_test_sources`` is:
|
||||
|
||||
@ -5534,46 +5650,69 @@ where each argument has the following meaning:
|
||||
* ``srcs`` is a string *or* a list of strings corresponding to the
|
||||
paths of subdirectories and or files needed for stand-alone testing.
|
||||
|
||||
The paths must be relative to the staged source directory. Contents of
|
||||
subdirectories and files are copied to a special test cache subdirectory
|
||||
of the installation prefix. They are automatically copied to the appropriate
|
||||
relative paths under the test stage directory prior to executing stand-alone
|
||||
tests.
|
||||
.. warning::
|
||||
|
||||
For example, a package method for copying everything in the ``tests``
|
||||
subdirectory plus the ``foo.c`` and ``bar.c`` files from ``examples``
|
||||
and using ``foo.c`` in a test method is illustrated below.
|
||||
Paths provided in the ``srcs`` argument **must be relative** to the
|
||||
staged source directory. They will be copied to the equivalent relative
|
||||
location under the test stage directory prior to test execution.
|
||||
|
||||
Contents of subdirectories and files are copied to a special test cache
|
||||
subdirectory of the installation prefix. They are automatically copied to
|
||||
the appropriate relative paths under the test stage directory prior to
|
||||
executing stand-alone tests.
|
||||
|
||||
.. tip::
|
||||
|
||||
*Perform test-related conversions once when copying files.*
|
||||
|
||||
If one or more of the copied files needs to be modified to reference
|
||||
the installed software, it is recommended that those changes be made
|
||||
to the cached files **once** in the post-``install`` copy method
|
||||
**after** the call to ``cache_extra_test_sources``. This will reduce
|
||||
the amount of unnecessary work in the test method **and** avoid problems
|
||||
running stand-alone tests in shared instances and facility deployments.
|
||||
|
||||
The ``filter_file`` function can be quite useful for such changes
|
||||
(see :ref:`file-filtering`).
|
||||
|
||||
Below is a basic example of a test that relies on files from the installation.
|
||||
This package method re-uses the contents of the ``examples`` subdirectory,
|
||||
which is assumed to have all of the files implemented to allow ``make`` to
|
||||
compile and link ``foo.c`` and ``bar.c`` against the package's installed
|
||||
library.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
class MyLibPackage(Package):
|
||||
class MyLibPackage(MakefilePackage):
|
||||
...
|
||||
|
||||
@run_after("install")
|
||||
def copy_test_files(self):
|
||||
srcs = ["tests",
|
||||
join_path("examples", "foo.c"),
|
||||
join_path("examples", "bar.c")]
|
||||
cache_extra_test_sources(self, srcs)
|
||||
cache_extra_test_sources(self, "examples")
|
||||
|
||||
def test_foo(self):
|
||||
exe = "foo"
|
||||
src_dir = self.test_suite.current_test_cache_dir.examples
|
||||
with working_dir(src_dir):
|
||||
cc = which(os.environ["CC"])
|
||||
cc(
|
||||
f"-L{self.prefix.lib}",
|
||||
f"-I{self.prefix.include}",
|
||||
f"{exe}.c",
|
||||
"-o", exe
|
||||
)
|
||||
foo = which(exe)
|
||||
foo()
|
||||
def test_example(self):
|
||||
"""build and run the examples"""
|
||||
examples_dir = self.test_suite.current_test_cache_dir.examples
|
||||
with working_dir(examples_dir):
|
||||
make = which("make")
|
||||
make()
|
||||
|
||||
In this case, the method copies the associated files from the build
|
||||
stage, **after** the software is installed, to the package's test
|
||||
cache directory. Then ``test_foo`` builds ``foo`` using ``foo.c``
|
||||
before running the program.
|
||||
for program in ["foo", "bar"]:
|
||||
with test_part(
|
||||
self,
|
||||
f"test_example_{program}",
|
||||
purpose=f"ensure {program} runs"
|
||||
):
|
||||
exe = Executable(program)
|
||||
exe()
|
||||
|
||||
In this case, ``copy_test_files`` copies the associated files from the
|
||||
build stage to the package's test cache directory under the installation
|
||||
prefix. Running ``spack test run`` for the package results in Spack copying
|
||||
the directory and its contents to the the test stage directory. The
|
||||
``working_dir`` context manager ensures the commands within it are executed
|
||||
from the ``examples_dir``. The test builds the software using ``make`` before
|
||||
running each executable, ``foo`` and ``bar``, as independent test parts.
|
||||
|
||||
.. note::
|
||||
|
||||
@ -5582,43 +5721,18 @@ before running the program.
|
||||
|
||||
The key to copying files for stand-alone testing at build time is use
|
||||
of the ``run_after`` directive, which ensures the associated files are
|
||||
copied **after** the provided build stage where the files **and**
|
||||
installation prefix are available.
|
||||
copied **after** the provided build stage (``install``) when the installation
|
||||
prefix **and** files are available.
|
||||
|
||||
These paths are **automatically copied** from cache to the test stage
|
||||
directory prior to the execution of any stand-alone tests. Tests access
|
||||
the files using the ``self.test_suite.current_test_cache_dir`` property.
|
||||
In our example above, test methods can use the following paths to reference
|
||||
the copy of each entry listed in ``srcs``, respectively:
|
||||
The test method uses the path contained in the package's
|
||||
``self.test_suite.current_test_cache_dir`` property for the root directory
|
||||
of the copied files. In this case, that's the ``examples`` subdirectory.
|
||||
|
||||
* ``self.test_suite.current_test_cache_dir.tests``
|
||||
* ``join_path(self.test_suite.current_test_cache_dir.examples, "foo.c")``
|
||||
* ``join_path(self.test_suite.current_test_cache_dir.examples, "bar.c")``
|
||||
|
||||
.. admonition:: Library packages should build stand-alone tests
|
||||
|
||||
Library developers will want to build the associated tests
|
||||
against their **installed** libraries before running them.
|
||||
|
||||
.. note::
|
||||
|
||||
While source and input files are generally recommended, binaries
|
||||
**may** also be cached by the build process. Only you, as the package
|
||||
writer or maintainer, know whether these files would be appropriate
|
||||
for testing the installed software weeks to months later.
|
||||
|
||||
.. note::
|
||||
|
||||
If one or more of the copied files needs to be modified to reference
|
||||
the installed software, it is recommended that those changes be made
|
||||
to the cached files **once** in the ``copy_test_sources`` method and
|
||||
***after** the call to ``cache_extra_test_sources()``. This will
|
||||
reduce the amount of unnecessary work in the test method **and** avoid
|
||||
problems testing in shared instances and facility deployments.
|
||||
|
||||
The ``filter_file`` function can be quite useful for such changes.
|
||||
See :ref:`file manipulation <file-manipulation>`.
|
||||
.. tip::
|
||||
|
||||
If you want to see more examples from packages that cache build files, run
|
||||
``spack pkg grep cache_extra_test_sources | sed "s/\/package.py.*//g" | sort -u``
|
||||
from the command line to get a list of the packages.
|
||||
|
||||
.. _cache_custom_files:
|
||||
|
||||
@ -5626,8 +5740,9 @@ the copy of each entry listed in ``srcs``, respectively:
|
||||
Adding custom files
|
||||
"""""""""""""""""""
|
||||
|
||||
In some cases it can be useful to have files that can be used to build or
|
||||
check the results of tests. Examples include:
|
||||
Sometimes it is helpful or necessary to include custom files for building and
|
||||
or checking the results of tests as part of the package. Examples of the types
|
||||
of files that might be useful are:
|
||||
|
||||
- test source files
|
||||
- test input files
|
||||
@ -5635,17 +5750,15 @@ check the results of tests. Examples include:
|
||||
- expected test outputs
|
||||
|
||||
While obtaining such files from the software repository is preferred (see
|
||||
:ref:`adding build-time files <cache_extra_test_sources>`), there are
|
||||
circumstances where that is not feasible (e.g., the software is not being
|
||||
actively maintained). When test files can't be obtained from the repository
|
||||
or as a supplement to files that can, Spack supports the inclusion of
|
||||
additional files under the ``test`` subdirectory of the package in the
|
||||
Spack repository.
|
||||
:ref:`cache_extra_test_sources`), there are circumstances where doing so is not
|
||||
feasible such as when the software is not being actively maintained. When test
|
||||
files cannot be obtained from the repository or there is a need to supplement
|
||||
files that can, Spack supports the inclusion of additional files under the
|
||||
``test`` subdirectory of the package in the Spack repository.
|
||||
|
||||
Spack **automatically copies** the contents of that directory to the
|
||||
test staging directory prior to running stand-alone tests. Test methods
|
||||
access those files using the ``self.test_suite.current_test_data_dir``
|
||||
property as shown below.
|
||||
The following example assumes a ``custom-example.c`` is saved in ``MyLibary``
|
||||
package's ``test`` subdirectory. It also assumes the program simply needs to
|
||||
be compiled and linked against the installed ``MyLibrary`` software.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@ -5655,17 +5768,29 @@ property as shown below.
|
||||
test_requires_compiler = True
|
||||
...
|
||||
|
||||
def test_example(self):
|
||||
def test_custom_example(self):
|
||||
"""build and run custom-example"""
|
||||
data_dir = self.test_suite.current_test_data_dir
|
||||
src_dir = self.test_suite.current_test_data_dir
|
||||
exe = "custom-example"
|
||||
src = datadir.join(f"{exe}.cpp")
|
||||
...
|
||||
# TODO: Build custom-example using src and exe
|
||||
...
|
||||
custom_example = which(exe)
|
||||
custom_example()
|
||||
|
||||
with working_dir(src_dir):
|
||||
cc = which(os.environ["CC"])
|
||||
cc(
|
||||
f"-L{self.prefix.lib}",
|
||||
f"-I{self.prefix.include}",
|
||||
f"{exe}.cpp",
|
||||
"-o", exe
|
||||
)
|
||||
|
||||
custom_example = Executable(exe)
|
||||
custom_example()
|
||||
|
||||
In this case, ``spack test run`` for the package results in Spack copying
|
||||
the contents of the ``test`` subdirectory to the test stage directory path
|
||||
in ``self.test_suite.current_test_data_dir`` before calling
|
||||
``test_custom_example``. Use of the ``working_dir`` context manager
|
||||
ensures the commands to build and run the program are performed from
|
||||
within the appropriate subdirectory of the test stage.
|
||||
|
||||
.. _expected_test_output_from_file:
|
||||
|
||||
@ -5674,9 +5799,8 @@ Reading expected output from a file
|
||||
"""""""""""""""""""""""""""""""""""
|
||||
|
||||
The helper function ``get_escaped_text_output`` is available for packages
|
||||
to retrieve and properly format the text from a file that contains the
|
||||
expected output from running an executable that may contain special
|
||||
characters.
|
||||
to retrieve properly formatted text from a file potentially containing
|
||||
special characters.
|
||||
|
||||
The signature for ``get_escaped_text_output`` is:
|
||||
|
||||
@ -5686,10 +5810,13 @@ The signature for ``get_escaped_text_output`` is:
|
||||
|
||||
where ``filename`` is the path to the file containing the expected output.
|
||||
|
||||
The ``filename`` for a :ref:`custom file <cache_custom_files>` can be
|
||||
accessed by tests using the ``self.test_suite.current_test_data_dir``
|
||||
property. The example below illustrates how to read a file that was
|
||||
added to the package's ``test`` subdirectory.
|
||||
The path provided to ``filename`` for one of the copied custom files
|
||||
(:ref:`custom file <cache_custom_files>`) is in the path rooted at
|
||||
``self.test_suite.current_test_data_dir``.
|
||||
|
||||
The example below shows how to reference both the custom database
|
||||
(``packages.db``) and expected output (``dump.out``) files Spack copies
|
||||
to the test stage:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@ -5711,8 +5838,9 @@ added to the package's ``test`` subdirectory.
|
||||
for exp in expected:
|
||||
assert re.search(exp, out), f"Expected '{exp}' in output"
|
||||
|
||||
If the file was instead copied from the ``tests`` subdirectory of the staged
|
||||
source code, the path would be obtained as shown below.
|
||||
If the files were instead cached from installing the software, the paths to the
|
||||
two files would be found under the ``self.test_suite.current_test_cache_dir``
|
||||
directory as shown below:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@ -5720,17 +5848,24 @@ source code, the path would be obtained as shown below.
|
||||
"""check example table dump"""
|
||||
test_cache_dir = self.test_suite.current_test_cache_dir
|
||||
db_filename = test_cache_dir.join("packages.db")
|
||||
..
|
||||
expected = get_escaped_text_output(test_cache_dir.join("dump.out"))
|
||||
...
|
||||
|
||||
Alternatively, if the file was copied to the ``share/tests`` subdirectory
|
||||
as part of the installation process, the test could access the path as
|
||||
follows:
|
||||
Alternatively, if both files had been installed by the software into the
|
||||
``share/tests`` subdirectory of the installation prefix, the paths to the
|
||||
two files would be referenced as follows:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def test_example(self):
|
||||
"""check example table dump"""
|
||||
db_filename = join_path(self.prefix.share.tests, "packages.db")
|
||||
|
||||
db_filename = self.prefix.share.tests.join("packages.db")
|
||||
..
|
||||
expected = get_escaped_text_output(
|
||||
self.prefix.share.tests.join("dump.out")
|
||||
)
|
||||
...
|
||||
|
||||
.. _check_outputs:
|
||||
|
||||
@ -5738,9 +5873,9 @@ follows:
|
||||
Comparing expected to actual outputs
|
||||
""""""""""""""""""""""""""""""""""""
|
||||
|
||||
The helper function ``check_outputs`` is available for packages to ensure
|
||||
the expected outputs from running an executable are contained within the
|
||||
actual outputs.
|
||||
The ``check_outputs`` helper routine is available for packages to ensure
|
||||
multiple expected outputs from running an executable are contained within
|
||||
the actual outputs.
|
||||
|
||||
The signature for ``check_outputs`` is:
|
||||
|
||||
@ -5766,11 +5901,17 @@ Invoking the method is the equivalent of:
|
||||
if errors:
|
||||
raise RuntimeError("\n ".join(errors))
|
||||
|
||||
.. tip::
|
||||
|
||||
If you want to see more examples from packages that use this helper, run
|
||||
``spack pkg grep check_outputs | sed "s/\/package.py.*//g" | sort -u``
|
||||
from the command line to get a list of the packages.
|
||||
|
||||
|
||||
.. _accessing-files:
|
||||
|
||||
"""""""""""""""""""""""""""""""""""""""""
|
||||
Accessing package- and test-related files
|
||||
Finding package- and test-related files
|
||||
"""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
You may need to access files from one or more locations when writing
|
||||
@ -5779,8 +5920,7 @@ include test source files or includes them but has no way to build the
|
||||
executables using the installed headers and libraries. In these cases
|
||||
you may need to reference the files relative to one or more root directory.
|
||||
The table below lists relevant path properties and provides additional
|
||||
examples of their use.
|
||||
:ref:`Reading expected output <expected_test_output_from_file>` provides
|
||||
examples of their use. See :ref:`expected_test_output_from_file` for
|
||||
examples of accessing files saved from the software repository, package
|
||||
repository, and installation.
|
||||
|
||||
@ -5809,7 +5949,6 @@ repository, and installation.
|
||||
- ``self.test_suite.current_test_data_dir``
|
||||
- ``join_path(self.test_suite.current_test_data_dir, "hello.f90")``
|
||||
|
||||
|
||||
.. _inheriting-tests:
|
||||
|
||||
""""""""""""""""""""""""""""
|
||||
@ -5852,7 +5991,7 @@ maintainers provide additional stand-alone tests customized to the package.
|
||||
.. warning::
|
||||
|
||||
Any package that implements a test method with the same name as an
|
||||
inherited method overrides the inherited method. If that is not the
|
||||
inherited method will override the inherited method. If that is not the
|
||||
goal and you are not explicitly calling and adding functionality to
|
||||
the inherited method for the test, then make sure that all test methods
|
||||
and embedded test parts have unique test names.
|
||||
@ -6017,6 +6156,8 @@ running:
|
||||
This is already part of the boilerplate for packages created with
|
||||
``spack create``.
|
||||
|
||||
.. _file-filtering:
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
Filtering functions
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Loading…
Reference in New Issue
Block a user