zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Simplify the detection protocol for packages

Browse Source 
Packages can implement “detect_version” to support detection
of external instances of a package. This is generally easier
than implementing “determine_spec_details”. The API for
determine_version is similar: for example you can return
“None” to indicate that an executable is not an instance
of a package.

Users may implement a “determine_variants” method for a package.
When doing external detection, executables are grouped by version
and each group results in a single invocation of “determine_variants”
for the associated spec. The method returns a string specifying
the variants for the package. The method may additionally return
a dictionary representing extra attributes for the package.

These will be stored in the spec yaml and can be retrieved
from self.spec.extra_attributes

The Spack GCC package has been updated with an implementation
of “determine_variants” which adds the following extra
attributes to the package: c, cxx, fortran
This commit is contained in:
Massimiliano Culpo
2020-07-31 13:07:48 +02:00
committed by Peter Scheibel
parent 193e8333fa
commit c0d490ffbe
11 changed files with 576 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files

239
lib/spack/docs/packaging_guide.rst
View File

@@ -4054,21 +4054,223 @@ File functions
Making a package discoverable with ``spack external find``
----------------------------------------------------------
To make a package discoverable with
:ref:`spack external find <cmd-spack-external-find>` you must
define one or more executables associated with the package and must
implement a method to generate a Spec when given an executable.
The simplest way to make a package discoverable with
:ref:`spack external find <cmd-spack-external-find>` is to:
The executables are specified as a package level ``executables``
attribute which is a list of strings (see example below); each string
is treated as a regular expression (e.g. 'gcc' would match 'gcc', 'gcc-8.3',
'my-weird-gcc', etc.).
1. Define the executables associated with the package
2. Implement a method to determine the versions of these executables
The method ``determine_spec_details`` has the following signature:
^^^^^^^^^^^^^^^^^
Minimal detection
^^^^^^^^^^^^^^^^^
The first step is fairly simple, as it requires only to
specify a package level ``executables`` attribute:
.. code-block:: python
def determine_spec_details(prefix, exes_in_prefix):
class Foo(Package):
# Each string provided here is treated as a regular expression, and
# would match for example 'foo', 'foobar', and 'bazfoo'.
executables = ['foo']
This attribute must be a list of strings. Each string is a regular
expression (e.g. 'gcc' would match 'gcc', 'gcc-8.3', 'my-weird-gcc', etc.) to
determine a set of system executables that might be part or this package. Note
that to match only executables named 'gcc' the regular expression ``'^gcc$'``
must be used.
Finally to determine the version of each executable the ``determine_version``
method must be implemented:
.. code-block:: python
@classmethod
def determine_version(cls, exe):
"""Return either the version of the executable passed as argument
or ``None`` if the version cannot be determined.
Args:
exe (str): absolute path to the executable being examined
"""
This method receives as input the path to a single executable and must return
as output its version as a string; if the user cannot determine the version
or determines that the executable is not an instance of the package, they can
return None and the exe will be discarded as a candidate.
Implementing the two steps above is mandatory, and gives the package the
basic ability to detect if a spec is present on the system at a given version.
.. note::
Any executable for which the ``determine_version`` method returns ``None``
will be discarded and won't appear in later stages of the workflow described below.
^^^^^^^^^^^^^^^^^^^^^^^^
Additional functionality
^^^^^^^^^^^^^^^^^^^^^^^^
Besides the two mandatory steps described above, there are also optional
methods that can be implemented to either increase the amount of details
being detected or improve the robustness of the detection logic in a package.
""""""""""""""""""""""""""""""
Variants and custom attributes
""""""""""""""""""""""""""""""
The ``determine_variants`` method can be optionally implemented in a package
to detect additional details of the spec:
.. code-block:: python
@classmethod
def determine_variants(cls, exes, version_str):
"""Return either a variant string, a tuple of a variant string
and a dictionary of extra attributes that will be recorded in
packages.yaml or a list of those items.
Args:
exes (list of str): list of executables (absolute paths) that
live in the same prefix and share the same version
version_str (str): version associated with the list of
executables, as detected by ``determine_version``
"""
This method takes as input a list of executables that live in the same prefix and
share the same version string, and returns either:
1. A variant string
2. A tuple of a variant string and a dictionary of extra attributes
3. A list of items matching either 1 or 2 (if multiple specs are detected
from the set of executables)
If extra attributes are returned, they will be recorded in ``packages.yaml``
and be available for later reuse. As an example, the ``gcc`` package will record
by default the different compilers found and an entry in ``packages.yaml``
would look like:
.. code-block:: yaml
packages:
gcc:
externals:
- spec: 'gcc@9.0.1 languages=c,c++,fortran'
prefix: /usr
extra_attributes:
compilers:
c: /usr/bin/x86_64-linux-gnu-gcc-9
c++: /usr/bin/x86_64-linux-gnu-g++-9
fortran: /usr/bin/x86_64-linux-gnu-gfortran-9
This allows us, for instance, to keep track of executables that would be named
differently if built by Spack (e.g. ``x86_64-linux-gnu-gcc-9``
instead of just ``gcc``).
.. TODO: we need to gather some more experience on overriding 'prefix'
and other special keywords in extra attributes, but as soon as we are
confident that this is the way to go we should document the process.
See https://github.com/spack/spack/pull/16526#issuecomment-653783204
"""""""""""""""""""""""""""
Filter matching executables
"""""""""""""""""""""""""""
Sometimes defining the appropriate regex for the ``executables``
attribute might prove to be difficult, especially if one has to
deal with corner cases or exclude "red herrings". To help keeping
the regular expressions as simple as possible, each package can
optionally implement a ``filter_executables`` method:
.. code-block:: python
@classmethod
def filter_detected_exes(cls, prefix, exes_in_prefix):
"""Return a filtered list of the executables in prefix"""
which takes as input a prefix and a list of matching executables and
returns a filtered list of said executables.
Using this method has the advantage of allowing custom logic for
filtering, and does not restrict the user to regular expressions
only. Consider the case of detecting the GNU C++ compiler. If we
try to search for executables that match ``g++``, that would have
the unwanted side effect of selecting also ``clang++`` - which is
a C++ compiler provided by another package - if present on the system.
Trying to select executables that contain ``g++`` but not ``clang``
would be quite complicated to do using regex only. Employing the
``filter_detected_exes`` method it becomes:
.. code-block:: python
class Gcc(Package):
executables = ['g++']
def filter_detected_exes(cls, prefix, exes_in_prefix):
return [x for x in exes_in_prefix if 'clang' not in x]
Another possibility that this method opens is to apply certain
filtering logic when specific conditions are met (e.g. take some
decisions on an OS and not on another).
^^^^^^^^^^^^^^^^^^
Validate detection
^^^^^^^^^^^^^^^^^^
To increase detection robustness, packagers may also implement a method
to validate the detected Spec objects:
.. code-block:: python
@classmethod
def validate_detected_spec(cls, spec, extra_attributes):
"""Validate a detected spec. Raise an exception if validation fails."""
This method receives a detected spec along with its extra attributes and can be
used to check that certain conditions are met by the spec. Packagers can either
use assertions or raise an ``InvalidSpecDetected`` exception when the check fails.
In case the conditions are not honored the spec will be discarded and any message
associated with the assertion or the exception will be logged as the reason for
discarding it.
As an example, a package that wants to check that the ``compilers`` attribute is
in the extra attributes can implement this method like this:
.. code-block:: python
@classmethod
def validate_detected_spec(cls, spec, extra_attributes):
"""Check that 'compilers' is in the extra attributes."""
msg = ('the extra attribute "compilers" must be set for '
'the detected spec "{0}"'.format(spec))
assert 'compilers' in extra_attributes, msg
or like this:
.. code-block:: python
@classmethod
def validate_detected_spec(cls, spec, extra_attributes):
"""Check that 'compilers' is in the extra attributes."""
if 'compilers' not in extra_attributes:
msg = ('the extra attribute "compilers" must be set for '
'the detected spec "{0}"'.format(spec))
raise InvalidSpecDetected(msg)
.. _determine_spec_details:
^^^^^^^^^^^^^^^^^^^^^^^^^
Custom detection workflow
^^^^^^^^^^^^^^^^^^^^^^^^^
In the rare case when the mechanisms described so far don't fit the
detection of a package, the implementation of all the methods above
can be disregarded and instead a custom ``determine_spec_details``
method can be implemented directly in the package class (note that
the definition of the ``executables`` attribute is still required):
.. code-block:: python
@classmethod
def determine_spec_details(cls, prefix, exes_in_prefix):
# exes_in_prefix = a set of paths, each path is an executable
# prefix = a prefix that is common to each path in exes_in_prefix
@@ -4076,14 +4278,13 @@ The method ``determine_spec_details`` has the following signature:
# the package. Return one or more Specs for each instance of the
# package which is thought to be installed in the provided prefix
``determine_spec_details`` takes as parameters a set of discovered
executables (which match those specified by the user) as well as a
common prefix shared by all of those executables. The function must
return one or more Specs associated with the executables (it can also
return ``None`` to indicate that no provided executables are associated
with the package).
This method takes as input a set of discovered executables (which match
those specified by the user) as well as a common prefix shared by all
of those executables. The function must return one or more :py:class:`spack.spec.Spec` associated
with the executables (it can also return ``None`` to indicate that no
provided executables are associated with the package).
Say for example we have a package called ``foo-package`` which
As an example, consider a made-up package called ``foo-package`` which
builds an executable called ``foo``. ``FooPackage`` would appear as
follows:
@@ -4110,7 +4311,9 @@ follows:
exe = spack.util.executable.Executable(exe_path)
output = exe('--version')
version_str = ... # parse output for version string
return Spec('foo-package@{0}'.format(version_str))
return Spec.from_detection(
'foo-package@{0}'.format(version_str)
)
.. _package-lifecycle: