openmpi: add opa-psm2 dependency (#16873)

Also document with_or_without and enable_or_disable, (which are used
to configure the opa-psm2 dependency).

lib/spack/docs/build_systems/autotoolspackage.rst

@@ -233,7 +233,124 @@ You may have noticed that most of the Autotools flags are of the form
 ``--without-baz``. Since these flags are so common, Spack provides a
 couple of helper functions to make your life easier.
 
-TODO: document ``with_or_without`` and ``enable_or_disable``.
+"""""""""""""""""
+enable_or_disable
+"""""""""""""""""
+
+Autotools flags for simple boolean variants can be automatically
+generated by calling the ``enable_or_disable`` method. This is
+typically used to enable or disable some feature within the package.
+
+.. code-block:: python
+
+   variant(
+       'memchecker',
+       default=False,
+       description='Memchecker support for debugging [degrades performance]'
+   )
+   config_args.extend(self.enable_or_disable('memchecker'))
+
+In this example, specifying the variant ``+memchecker`` will generate
+the following configuration options:
+
+.. code-block:: console
+
+   --enable-memchecker
+
+"""""""""""""""
+with_or_without
+"""""""""""""""
+
+Autotools flags for more complex variants, including boolean variants
+and multi-valued variants, can be automatically generated by calling
+the ``with_or_without`` method.
+
+.. code-block:: python
+
+   variant(
+       'schedulers',
+       values=disjoint_sets(
+           ('auto',), ('alps', 'lsf', 'tm', 'slurm', 'sge', 'loadleveler')
+       ).with_non_feature_values('auto', 'none'),
+       description="List of schedulers for which support is enabled; "
+       "'auto' lets openmpi determine",
+   )
+   if 'schedulers=auto' not in spec:
+       config_args.extend(self.with_or_without('schedulers'))
+
+In this example, specifying the variant ``schedulers=slurm,sge`` will
+generate the following configuration options:
+
+.. code-block:: console
+
+   --with-slurm --with-sge
+
+``enable_or_disable`` is actually functionally equivalent with
+``with_or_without``, and accepts the same arguments and variant types;
+but idiomatic autotools packages often follow these naming
+conventions.
+
+""""""""""""""""
+activation_value
+""""""""""""""""
+
+Autotools parameters that require an option can still be automatically
+generated, using the ``activation_value`` argument to
+``with_or_without`` (or, rarely, ``enable_or_disable``).
+
+.. code-block:: python
+
+   variant(
+      'fabrics',
+       values=disjoint_sets(
+           ('auto',), ('psm', 'psm2', 'verbs', 'mxm', 'ucx', 'libfabric')
+       ).with_non_feature_values('auto', 'none'),
+       description="List of fabrics that are enabled; "
+       "'auto' lets openmpi determine",
+   )
+   if 'fabrics=auto' not in spec:
+       config_args.extend(self.with_or_without('fabrics',
+           activation_value='prefix'))
+
+``activation_value`` accepts a callable that generates the configure
+parameter value given the variant value; but the special value
+``prefix`` tells Spack to automatically use the dependenency's
+installation prefix, which is the most common use for such
+parameters. In this example, specifying the variant
+``fabrics=libfabric`` will generate the following configuration
+options:
+
+.. code-block:: console
+
+   --with-libfabric=</path/to/libfabric>
+
+""""""""""""""""""""
+activation overrides
+""""""""""""""""""""
+
+Finally, the behavior of either ``with_or_without`` or
+``enable_or_disable`` can be overridden for specific variant
+values. This is most useful for multi-values variants where some of
+the variant values require atypical behavior.
+
+.. code-block:: python
+
+   def with_or_without_verbs(self, activated):
+       # Up through version 1.6, this option was named --with-openib.
+       # In version 1.7, it was renamed to be --with-verbs.
+       opt = 'verbs' if self.spec.satisfies('@1.7:') else 'openib'
+       if not activated:
+           return '--without-{0}'.format(opt)
+       return '--with-{0}={1}'.format(opt, self.spec['rdma-core'].prefix)
+
+Defining ``with_or_without_verbs`` overrides the behavior of a
+``fabrics=verbs`` variant, changing the configure-time option to
+``--with-openib`` for older versions of the package and specifying an
+alternative dependency name:
+
+.. code-block::
+
+   --with-openib=</path/to/rdma-core>
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Configure script in a sub-directory

var/spack/repos/builtin/packages/openmpi/package.py

@@ -247,6 +247,7 @@ class Openmpi(AutotoolsPackage):
     depends_on('ucx +thread_multiple', when='fabrics=ucx +thread_multiple')
     depends_on('ucx +thread_multiple', when='@3.0.0: fabrics=ucx')
     depends_on('libfabric', when='fabrics=libfabric')
+    depends_on('opa-psm2', when='fabrics=psm2')
     depends_on('mxm', when='fabrics=mxm')
     depends_on('binutils+libiberty', when='fabrics=mxm')
     depends_on('rdma-core', when='fabrics=verbs')
@@ -330,6 +331,11 @@ def with_or_without_tm(self, activated):
             return '--without-tm'
         return '--with-tm={0}'.format(self.spec['openpbs'].prefix)
 
+    def with_or_without_psm2(self, activated):
+        if not activated:
+            return '--without-psm2'
+        return '--with-psm2={0}'.format(self.spec['opa-psm2'].prefix)
+
     @run_before('autoreconf')
     def die_without_fortran(self):
         # Until we can pass variants such as +fortran through virtual