Compare commits

..

10 Commits

Author SHA1 Message Date
Wouter Deconinck
3cd1968cb6 RPackage: switch verify to phase_callback 2025-04-29 13:15:15 -05:00
Wouter Deconinck
56e087ad77 RPackage: fix style 2025-04-29 13:15:15 -05:00
Wouter Deconinck
0f47d68dd7 RPackage: fix line length 2025-04-29 13:15:15 -05:00
wdconinc
924cc93a02 [@spackbot] updating style on behalf of wdconinc 2025-04-29 13:15:14 -05:00
Wouter Deconinck
99bfe06ae9 RPackage:move verification to separate method, only when testing 2025-04-29 13:14:21 -05:00
Wouter Deconinck
7314fce7e3 RPackage: include our own parser (from pycran) 2025-04-29 13:14:21 -05:00
Wouter Deconinck
cd3d2791f9 RPackage: allow empty Depends/Imports 2025-04-29 13:14:21 -05:00
Wouter Deconinck
ee7025755a RPackage: fix assert 2025-04-29 13:14:21 -05:00
Wouter Deconinck
d564985fb2 RPackage: allow for missing Imports/Depends 2025-04-29 13:14:21 -05:00
Wouter Deconinck
0cd97d584e RPackage: parse description for incomplete dependencies 2025-04-29 13:14:19 -05:00
11054 changed files with 1151 additions and 1913 deletions

View File

@@ -28,7 +28,7 @@ max-line-length = 99
# - F821: undefined name `name`
#
per-file-ignores =
var/spack/*/package.py:F403,F405,F821
var/spack/repos/*/package.py:F403,F405,F821
*-ci-package.py:F403,F405,F821
# exclude things we usually do not want linting for.

View File

@@ -42,17 +42,17 @@ jobs:
# built-in repository or documentation
filters: |
bootstrap:
- 'var/spack/repos/spack_repo/builtin/packages/clingo-bootstrap/**'
- 'var/spack/repos/spack_repo/builtin/packages/clingo/**'
- 'var/spack/repos/spack_repo/builtin/packages/python/**'
- 'var/spack/repos/spack_repo/builtin/packages/re2c/**'
- 'var/spack/repos/spack_repo/builtin/packages/gnupg/**'
- 'var/spack/repos/spack_repo/builtin/packages/libassuan/**'
- 'var/spack/repos/spack_repo/builtin/packages/libgcrypt/**'
- 'var/spack/repos/spack_repo/builtin/packages/libgpg-error/**'
- 'var/spack/repos/spack_repo/builtin/packages/libksba/**'
- 'var/spack/repos/spack_repo/builtin/packages/npth/**'
- 'var/spack/repos/spack_repo/builtin/packages/pinentry/**'
- 'var/spack/repos/builtin/packages/clingo-bootstrap/**'
- 'var/spack/repos/builtin/packages/clingo/**'
- 'var/spack/repos/builtin/packages/python/**'
- 'var/spack/repos/builtin/packages/re2c/**'
- 'var/spack/repos/builtin/packages/gnupg/**'
- 'var/spack/repos/builtin/packages/libassuan/**'
- 'var/spack/repos/builtin/packages/libgcrypt/**'
- 'var/spack/repos/builtin/packages/libgpg-error/**'
- 'var/spack/repos/builtin/packages/libksba/**'
- 'var/spack/repos/builtin/packages/npth/**'
- 'var/spack/repos/builtin/packages/pinentry/**'
- 'lib/spack/**'
- 'share/spack/**'
- '.github/workflows/bootstrap.yml'

View File

@@ -34,7 +34,7 @@ jobs:
vermin --backport importlib --backport argparse --violations --backport typing -t=3.6- -vvv lib/spack/spack/ lib/spack/llnl/ bin/
- name: vermin (Repositories)
run: |
vermin --backport importlib --backport argparse --violations --backport typing -t=3.6- -vvv var/spack/repos var/spack/test_repos
vermin --backport importlib --backport argparse --violations --backport typing -t=3.6- -vvv var/spack/repos
# Run style checks on the files that have been changed
style:
@@ -65,11 +65,7 @@ jobs:
python_version: '3.13'
verify-checksums:
# do not run if the commit message or PR description contains [skip-verify-checksums]
if: >-
${{ inputs.with_packages == 'true' &&
!contains(github.event.pull_request.body, '[skip-verify-checksums]') &&
!contains(github.event.head_commit.message, '[skip-verify-checksums]') }}
if: ${{ inputs.with_packages == 'true' }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29

View File

@@ -5,4 +5,4 @@ isort==6.0.1
mypy==1.15.0
types-six==1.17.0.20250403
vermin==1.6.0
pylint==3.3.7
pylint==3.3.6

View File

@@ -11,4 +11,4 @@
# ~/.spack/repos.yaml
# -------------------------------------------------------------------------
repos:
- $spack/var/spack/repos/spack_repo/builtin
- $spack/var/spack/repos/builtin

View File

@@ -1916,7 +1916,7 @@ diagnostics. Issues, if found, are reported to stdout:
PKG-DIRECTIVES: 1 issue found
1. lammps: wrong variant in "conflicts" directive
the variant 'adios' does not exist
in /home/spack/spack/var/spack/repos/spack_repo/builtin/packages/lammps/package.py
in /home/spack/spack/var/spack/repos/builtin/packages/lammps/package.py
------------

View File

@@ -83,7 +83,7 @@ packages. You can quickly find examples by running:
.. code-block:: console
$ cd var/spack/repos/spack_repo/builtin/packages
$ cd var/spack/repos/builtin/packages
$ grep -l QMakePackage */package.py

View File

@@ -27,10 +27,10 @@ it could use the ``require`` directive as follows:
Spack has a number of built-in bundle packages, such as:
* `AmdAocl <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/amd_aocl/package.py>`_
* `EcpProxyApps <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/ecp_proxy_apps/package.py>`_
* `Libc <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/libc/package.py>`_
* `Xsdk <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/xsdk/package.py>`_
* `AmdAocl <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/amd-aocl/package.py>`_
* `EcpProxyApps <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/ecp-proxy-apps/package.py>`_
* `Libc <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/libc/package.py>`_
* `Xsdk <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/xsdk/package.py>`_
where ``Xsdk`` also inherits from ``CudaPackage`` and ``RocmPackage`` and
``Libc`` is a virtual bundle package for the C standard library.

View File

@@ -199,7 +199,7 @@ a variant to control this:
However, not every CMake package accepts all four of these options.
Grep the ``CMakeLists.txt`` file to see if the default values are
missing or replaced. For example, the
`dealii <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/dealii/package.py>`_
`dealii <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/dealii/package.py>`_
package overrides the default variant with:
.. code-block:: python

View File

@@ -20,8 +20,8 @@ start is to look at the definitions of other build systems. This guide
focuses mostly on how Spack's build systems work.
In this guide, we will be using the
`perl <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/perl/package.py>`_ and
`cmake <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/cmake/package.py>`_
`perl <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/perl/package.py>`_ and
`cmake <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cmake/package.py>`_
packages as examples. ``perl``'s build system is a hand-written
``Configure`` shell script, while ``cmake`` bootstraps itself during
installation. Both of these packages require custom build systems.

View File

@@ -96,9 +96,9 @@ there are any other variables you need to set, you can do this in the
env.set("BLASLIB", spec["blas"].libs.ld_flags)
`cbench <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/cbench/package.py>`_
`cbench <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cbench/package.py>`_
is a good example of a simple package that does this, while
`esmf <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/esmf/package.py>`_
`esmf <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/esmf/package.py>`_
is a good example of a more complex package.
""""""""""""""""""""""
@@ -129,7 +129,7 @@ If you do need access to the spec, you can create a property like so:
]
`cloverleaf <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/cloverleaf/package.py>`_
`cloverleaf <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cloverleaf/package.py>`_
is a good example of a package that uses this strategy.
"""""""""""""
@@ -152,7 +152,7 @@ and a ``filter`` method to help with this. For example:
makefile.filter(r"^\s*FC\s*=.*", f"FC = {spack_fc}")
`stream <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/stream/package.py>`_
`stream <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/stream/package.py>`_
is a good example of a package that involves editing a Makefile to set
the appropriate variables.
@@ -192,7 +192,7 @@ well for storing variables:
inc.write(f"{key} = {config[key]}\n")
`elk <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/elk/package.py>`_
`elk <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/elk/package.py>`_
is a good example of a package that uses a dictionary to store
configuration variables.
@@ -213,7 +213,7 @@ them in a list:
inc.write(f"{var}\n")
`hpl <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/hpl/package.py>`_
`hpl <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/hpl/package.py>`_
is a good example of a package that uses a list to store
configuration variables.

View File

@@ -39,7 +39,7 @@ for "CRAN <package-name>" and you should quickly find what you want.
If it isn't on CRAN, try Bioconductor, another common R repository.
For the purposes of this tutorial, we will be walking through
`r-caret <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/r_caret/package.py>`_
`r-caret <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/r-caret/package.py>`_
as an example. If you search for "CRAN caret", you will quickly find what
you are looking for at https://cran.r-project.org/package=caret.
https://cran.r-project.org is the main CRAN website. However, CRAN also
@@ -337,7 +337,7 @@ Non-R dependencies
^^^^^^^^^^^^^^^^^^
Some packages depend on non-R libraries for linking. Check out the
`r-stringi <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/r_stringi/package.py>`_
`r-stringi <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/r-stringi/package.py>`_
package for an example: https://cloud.r-project.org/package=stringi.
If you search for the text "SystemRequirements", you will see:
@@ -352,7 +352,7 @@ Passing arguments to the installation
Some R packages provide additional flags that can be passed to
``R CMD INSTALL``, often to locate non-R dependencies.
`r-rmpi <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/r_rmpi/package.py>`_
`r-rmpi <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/r-rmpi/package.py>`_
is an example of this, and flags for linking to an MPI library. To pass
these to the installation command, you can override ``configure_args``
like so:

View File

@@ -104,10 +104,10 @@ Finding available options
The first place to start when looking for a list of valid options to
build a package is ``scons --help``. Some packages like
`kahip <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/kahip/package.py>`_
`kahip <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/kahip/package.py>`_
don't bother overwriting the default SCons help message, so this isn't
very useful, but other packages like
`serf <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/serf/package.py>`_
`serf <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/serf/package.py>`_
print a list of valid command-line variables:
.. code-block:: console
@@ -177,7 +177,7 @@ print a list of valid command-line variables:
More advanced packages like
`cantera <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/cantera/package.py>`_
`cantera <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cantera/package.py>`_
use ``scons --help`` to print a list of subcommands:
.. code-block:: console

View File

@@ -225,14 +225,10 @@ def setup(sphinx):
("py:class", "llnl.util.lang.T"),
("py:class", "llnl.util.lang.KT"),
("py:class", "llnl.util.lang.VT"),
("py:class", "llnl.util.lang.K"),
("py:class", "llnl.util.lang.V"),
("py:class", "llnl.util.lang.ClassPropertyType"),
("py:obj", "llnl.util.lang.KT"),
("py:obj", "llnl.util.lang.VT"),
("py:obj", "llnl.util.lang.ClassPropertyType"),
("py:obj", "llnl.util.lang.K"),
("py:obj", "llnl.util.lang.V"),
]
# The reST default role (used for this markup: `text`) to use for all documents.

View File

@@ -226,9 +226,9 @@ If all is well, you'll see something like this:
Modified files:
var/spack/repos/spack_repo/builtin/packages/hdf5/package.py
var/spack/repos/spack_repo/builtin/packages/hdf/package.py
var/spack/repos/spack_repo/builtin/packages/netcdf/package.py
var/spack/repos/builtin/packages/hdf5/package.py
var/spack/repos/builtin/packages/hdf/package.py
var/spack/repos/builtin/packages/netcdf/package.py
=======================================================
Flake8 checks were clean.
@@ -236,9 +236,9 @@ However, if you aren't compliant with PEP 8, flake8 will complain:
.. code-block:: console
var/spack/repos/spack_repo/builtin/packages/netcdf/package.py:26: [F401] 'os' imported but unused
var/spack/repos/spack_repo/builtin/packages/netcdf/package.py:61: [E303] too many blank lines (2)
var/spack/repos/spack_repo/builtin/packages/netcdf/package.py:106: [E501] line too long (92 > 79 characters)
var/spack/repos/builtin/packages/netcdf/package.py:26: [F401] 'os' imported but unused
var/spack/repos/builtin/packages/netcdf/package.py:61: [E303] too many blank lines (2)
var/spack/repos/builtin/packages/netcdf/package.py:106: [E501] line too long (92 > 79 characters)
Flake8 found errors.
Most of the error messages are straightforward, but if you don't understand what
@@ -280,7 +280,7 @@ All of these can be installed with Spack, e.g.
.. warning::
Sphinx has `several required dependencies <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/py-sphinx/package.py>`_.
Sphinx has `several required dependencies <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/py-sphinx/package.py>`_.
If you're using a ``python`` from Spack and you installed
``py-sphinx`` and friends, you need to make them available to your
``python``. The easiest way to do this is to run:

View File

@@ -154,7 +154,9 @@ Package-related modules
:mod:`spack.util.naming`
Contains functions for mapping between Spack package names,
Python module names, and Python class names.
Python module names, and Python class names. Functions like
:func:`~spack.util.naming.mod_to_class` handle mapping package
module names to class names.
:mod:`spack.directives`
*Directives* are functions that can be called inside a package definition

View File

@@ -131,7 +131,7 @@ creates a simple python file:
It doesn't take much python coding to get from there to a working
package:
.. literalinclude:: _spack_root/var/spack/repos/spack_repo/builtin/packages/libelf/package.py
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/libelf/package.py
:lines: 5-
Spack also provides wrapper functions around common commands like

View File

@@ -369,9 +369,9 @@ If you have a collection of software expected to work well together with
no source code of its own, you can create a :ref:`BundlePackage <bundlepackage>`.
Examples where bundle packages can be useful include defining suites of
applications (e.g, `EcpProxyApps
<https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/ecp_proxy_apps/package.py>`_), commonly used libraries
(e.g., `AmdAocl <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/amd_aocl/package.py>`_),
and software development kits (e.g., `EcpDataVisSdk <https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/ecp_data_vis_sdk/package.py>`_).
<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/ecp-proxy-apps/package.py>`_), commonly used libraries
(e.g., `AmdAocl <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/amd-aocl/package.py>`_),
and software development kits (e.g., `EcpDataVisSdk <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/ecp-data-vis-sdk/package.py>`_).
These versioned packages primarily consist of dependencies on the associated
software packages. They can include :ref:`variants <variants>` to ensure
@@ -443,7 +443,7 @@ lives in:
.. code-block:: console
$ spack location -p gmp
${SPACK_ROOT}/var/spack/repos/spack_repo/builtin/packages/gmp/package.py
${SPACK_ROOT}/var/spack/repos/builtin/packages/gmp/package.py
but ``spack edit`` provides a much simpler shortcut and saves you the
trouble of typing the full path.
@@ -457,19 +457,19 @@ live in Spack's directory structure. In general, :ref:`cmd-spack-create`
handles creating package files for you, so you can skip most of the
details here.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``var/spack/repos/spack_repo/builtin/packages``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
``var/spack/repos/builtin/packages``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A Spack installation directory is structured like a standard UNIX
install prefix (``bin``, ``lib``, ``include``, ``var``, ``opt``,
etc.). Most of the code for Spack lives in ``$SPACK_ROOT/lib/spack``.
Packages themselves live in ``$SPACK_ROOT/var/spack/repos/spack_repo/builtin/packages``.
Packages themselves live in ``$SPACK_ROOT/var/spack/repos/builtin/packages``.
If you ``cd`` to that directory, you will see directories for each
package:
.. command-output:: cd $SPACK_ROOT/var/spack/repos/spack_repo/builtin/packages && ls
.. command-output:: cd $SPACK_ROOT/var/spack/repos/builtin/packages && ls
:shell:
:ellipsis: 10
@@ -479,7 +479,7 @@ package lives in:
.. code-block:: none
$SPACK_ROOT/var/spack/repos/spack_repo/builtin/packages/libelf/package.py
$SPACK_ROOT/var/spack/repos/builtin/packages/libelf/package.py
Alongside the ``package.py`` file, a package may contain extra
directories or files (like patches) that it needs to build.
@@ -492,7 +492,7 @@ Packages are named after the directory containing ``package.py``. So,
``libelf``'s ``package.py`` lives in a directory called ``libelf``.
The ``package.py`` file defines a class called ``Libelf``, which
extends Spack's ``Package`` class. For example, here is
``$SPACK_ROOT/var/spack/repos/spack_repo/builtin/packages/libelf/package.py``:
``$SPACK_ROOT/var/spack/repos/builtin/packages/libelf/package.py``:
.. code-block:: python
:linenos:
@@ -520,7 +520,7 @@ these:
$ spack install libelf@0.8.13
Spack sees the package name in the spec and looks for
``libelf/package.py`` in ``var/spack/repos/spack_repo/builtin/packages``.
``libelf/package.py`` in ``var/spack/repos/builtin/packages``.
Likewise, if you run ``spack install py-numpy``, Spack looks for
``py-numpy/package.py``.
@@ -686,7 +686,7 @@ https://www.open-mpi.org/software/ompi/v2.1/downloads/openmpi-2.1.1.tar.bz2
In order to handle this, you can define a ``url_for_version()`` function
like so:
.. literalinclude:: _spack_root/var/spack/repos/spack_repo/builtin/packages/openmpi/package.py
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/openmpi/package.py
:pyobject: Openmpi.url_for_version
With the use of this ``url_for_version()``, Spack knows to download OpenMPI ``2.1.1``
@@ -787,7 +787,7 @@ of GNU. For that, Spack goes a step further and defines a mixin class that
takes care of all of the plumbing and requires packagers to just define a proper
``gnu_mirror_path`` attribute:
.. literalinclude:: _spack_root/var/spack/repos/spack_repo/builtin/packages/autoconf/package.py
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/autoconf/package.py
:lines: 9-18
^^^^^^^^^^^^^^^^^^^^^^^^
@@ -1995,7 +1995,7 @@ structure like this:
.. code-block:: none
$SPACK_ROOT/var/spack/repos/spack_repo/builtin/packages/
$SPACK_ROOT/var/spack/repos/builtin/packages/
mvapich2/
package.py
ad_lustre_rwcontig_open_source.patch
@@ -2133,7 +2133,7 @@ handles ``RPATH``:
.. _pyside-patch:
.. literalinclude:: _spack_root/var/spack/repos/spack_repo/builtin/packages/py_pyside/package.py
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/py-pyside/package.py
:pyobject: PyPyside.patch
:linenos:
@@ -2201,7 +2201,7 @@ using the ``spack resource show`` command::
$ spack resource show 3877ab54
3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00
path: /home/spackuser/src/spack/var/spack/repos/spack_repo/builtin/packages/m4/gnulib-pgi.patch
path: /home/spackuser/src/spack/var/spack/repos/builtin/packages/m4/gnulib-pgi.patch
applies to: builtin.m4
``spack resource show`` looks up downloadable resources from package
@@ -2219,7 +2219,7 @@ wonder where the extra boost patches are coming from::
^boost@1.68.0%apple-clang@9.0.0+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199,b37164268f34f7133cbc9a4066ae98fda08adf51e1172223f6a969909216870f ~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=darwin-highsierra-x86_64
$ spack resource show b37164268
b37164268f34f7133cbc9a4066ae98fda08adf51e1172223f6a969909216870f
path: /home/spackuser/src/spack/var/spack/repos/spack_repo/builtin/packages/dealii/boost_1.68.0.patch
path: /home/spackuser/src/spack/var/spack/repos/builtin/packages/dealii/boost_1.68.0.patch
applies to: builtin.boost
patched by: builtin.dealii
@@ -2930,7 +2930,7 @@ this, Spack provides four different methods that can be overridden in a package:
The Qt package, for instance, uses this call:
.. literalinclude:: _spack_root/var/spack/repos/spack_repo/builtin/packages/qt/package.py
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/qt/package.py
:pyobject: Qt.setup_dependent_build_environment
:linenos:
@@ -2958,7 +2958,7 @@ variables to be used by the dependent. This is done by implementing
:meth:`setup_dependent_package <spack.package_base.PackageBase.setup_dependent_package>`. An
example of this can be found in the ``Python`` package:
.. literalinclude:: _spack_root/var/spack/repos/spack_repo/builtin/packages/python/package.py
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/python/package.py
:pyobject: Python.setup_dependent_package
:linenos:
@@ -3785,7 +3785,7 @@ It is usually sufficient for a packager to override a few
build system specific helper methods or attributes to provide, for instance,
configure arguments:
.. literalinclude:: _spack_root/var/spack/repos/spack_repo/builtin/packages/m4/package.py
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/m4/package.py
:pyobject: M4.configure_args
:linenos:
@@ -4110,7 +4110,7 @@ Shell command functions
Recall the install method from ``libelf``:
.. literalinclude:: _spack_root/var/spack/repos/spack_repo/builtin/packages/libelf/package.py
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/libelf/package.py
:pyobject: Libelf.install
:linenos:
@@ -4901,7 +4901,7 @@ the one passed to install, only the MPI implementations all set some
additional properties on it to help you out. E.g., in openmpi, you'll
find this:
.. literalinclude:: _spack_root/var/spack/repos/spack_repo/builtin/packages/openmpi/package.py
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/openmpi/package.py
:pyobject: Openmpi.setup_dependent_package
That code allows the ``openmpi`` package to associate an ``mpicc`` property
@@ -6001,16 +6001,16 @@ with those implemented in the package itself.
* - Parent/Provider Package
- Stand-alone Tests
* - `C
<https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/c>`_
<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/c>`_
- Compiles ``hello.c`` and runs it
* - `Cxx
<https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/cxx>`_
<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/cxx>`_
- Compiles and runs several ``hello`` programs
* - `Fortran
<https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/fortran>`_
<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/fortran>`_
- Compiles and runs ``hello`` programs (``F`` and ``f90``)
* - `Mpi
<https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/mpi>`_
<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/mpi>`_
- Compiles and runs ``mpi_hello`` (``c``, ``fortran``)
* - :ref:`PythonPackage <pythonpackage>`
- Imports modules listed in the ``self.import_modules`` property with defaults derived from the tarball
@@ -6031,7 +6031,7 @@ maintainers provide additional stand-alone tests customized to the package.
One example of a package that adds its own stand-alone tests to those
"inherited" by the virtual package it provides an implementation for is
the `Openmpi package
<https://github.com/spack/spack/blob/develop/var/spack/repos/spack_repo/builtin/packages/openmpi/package.py>`_.
<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/openmpi/package.py>`_.
Below are snippets from running and viewing the stand-alone test results
for ``openmpi``:

View File

@@ -9,7 +9,7 @@ Package Repositories (repos.yaml)
=================================
Spack comes with thousands of built-in package recipes in
``var/spack/repos/spack_repo/builtin/``. This is a **package repository** -- a
``var/spack/repos/builtin/``. This is a **package repository** -- a
directory that Spack searches when it needs to find a package by name.
You may need to maintain packages for restricted, proprietary or
experimental software separately from the built-in repository. Spack
@@ -69,7 +69,7 @@ The default ``etc/spack/defaults/repos.yaml`` file looks like this:
.. code-block:: yaml
repos:
- $spack/var/spack/repos/spack_repo/builtin
- $spack/var/spack/repos/builtin
The file starts with ``repos:`` and contains a single ordered list of
paths to repositories. Each path is on a separate line starting with
@@ -78,16 +78,16 @@ paths to repositories. Each path is on a separate line starting with
.. code-block:: yaml
repos:
- /opt/repos/spack_repo/local_repo
- $spack/var/spack/repos/spack_repo/builtin
- /opt/local-repo
- $spack/var/spack/repos/builtin
When Spack interprets a spec, e.g., ``mpich`` in ``spack install mpich``,
it searches these repositories in order (first to last) to resolve each
package name. In this example, Spack will look for the following
packages and use the first valid file:
1. ``/opt/repos/spack_repo/local_repo/packages/mpich/package.py``
2. ``$spack/var/spack/repos/spack_repo/builtin/packages/mpich/package.py``
1. ``/opt/local-repo/packages/mpich/package.py``
2. ``$spack/var/spack/repos/builtin/packages/mpich/package.py``
.. note::
@@ -101,15 +101,14 @@ Namespaces
Every repository in Spack has an associated **namespace** defined in its
top-level ``repo.yaml`` file. If you look at
``var/spack/repos/spack_repo/builtin/repo.yaml`` in the built-in repository, you'll
``var/spack/repos/builtin/repo.yaml`` in the built-in repository, you'll
see that its namespace is ``builtin``:
.. code-block:: console
$ cat var/spack/repos/spack_repo/builtin/repo.yaml
$ cat var/spack/repos/builtin/repo.yaml
repo:
namespace: builtin
api: v2.0
Spack records the repository namespace of each installed package. For
example, if you install the ``mpich`` package from the ``builtin`` repo,
@@ -218,15 +217,15 @@ Suppose you have three repositories: the builtin Spack repo
repo containing your own prototype packages (``proto``). Suppose they
contain packages as follows:
+--------------+-----------------------------------------------+-----------------------------+
| Namespace | Path to repo | Packages |
+==============+===============================================+=============================+
| ``proto`` | ``~/my_spack_repos/spack_repo/proto`` | ``mpich`` |
+--------------+-----------------------------------------------+-----------------------------+
| ``llnl`` | ``/usr/local/repos/spack_repo/llnl`` | ``hdf5`` |
+--------------+-----------------------------------------------+-----------------------------+
| ``builtin`` | ``$spack/var/spack/repos/spack_repo/builtin`` | ``mpich``, ``hdf5``, others |
+--------------+-----------------------------------------------+-----------------------------+
+--------------+------------------------------------+-----------------------------+
| Namespace | Path to repo | Packages |
+==============+====================================+=============================+
| ``proto`` | ``~/proto`` | ``mpich`` |
+--------------+------------------------------------+-----------------------------+
| ``llnl`` | ``/usr/local/llnl`` | ``hdf5`` |
+--------------+------------------------------------+-----------------------------+
| ``builtin`` | ``$spack/var/spack/repos/builtin`` | ``mpich``, ``hdf5``, others |
+--------------+------------------------------------+-----------------------------+
Suppose that ``hdf5`` depends on ``mpich``. You can override the
built-in ``hdf5`` by adding the ``llnl`` repo to ``repos.yaml``:
@@ -234,8 +233,8 @@ built-in ``hdf5`` by adding the ``llnl`` repo to ``repos.yaml``:
.. code-block:: yaml
repos:
- /usr/local/repos/spack_repo/llnl
- $spack/var/spack/repos/spack_repo/builtin
- /usr/local/llnl
- $spack/var/spack/repos/builtin
``spack install hdf5`` will install ``llnl.hdf5 ^builtin.mpich``.
@@ -244,9 +243,9 @@ If, instead, ``repos.yaml`` looks like this:
.. code-block:: yaml
repos:
- ~/my_spack_repos/spack_repo/proto
- /usr/local/repos/spack_repo/llnl
- $spack/var/spack/repos/spack_repo/builtin
- ~/proto
- /usr/local/llnl
- $spack/var/spack/repos/builtin
``spack install hdf5`` will install ``llnl.hdf5 ^proto.mpich``.
@@ -327,8 +326,8 @@ files, use ``spack repo list``.
$ spack repo list
==> 2 package repositories.
myrepo v2.0 ~/my_spack_repos/spack_repo/myrepo
builtin v2.0 ~/spack/var/spack/repos/spack_repo/builtin
myrepo ~/myrepo
builtin ~/spack/var/spack/repos/builtin
Each repository is listed with its associated namespace. To get the raw,
merged YAML from all configuration files, use ``spack config get repos``:
@@ -336,9 +335,9 @@ merged YAML from all configuration files, use ``spack config get repos``:
.. code-block:: console
$ spack config get repos
repos:
- ~/my_spack_repos/spack_repo/myrepo
- $spack/var/spack/repos/spack_repo/builtin
repos:srepos:
- ~/myrepo
- $spack/var/spack/repos/builtin
Note that, unlike ``spack repo list``, this does not include the
namespace, which is read from each repo's ``repo.yaml``.
@@ -352,54 +351,66 @@ yourself; you can use the ``spack repo create`` command.
.. code-block:: console
$ spack repo create ~/my_spack_repos myrepo
$ spack repo create myrepo
==> Created repo with namespace 'myrepo'.
==> To register it with spack, run this command:
spack repo add ~/my_spack_repos/spack_repo/myrepo
spack repo add ~/myrepo
$ ls ~/my_spack_repos/spack_repo/myrepo
$ ls myrepo
packages/ repo.yaml
$ cat ~/my_spack_repos/spack_repo/myrepo/repo.yaml
$ cat myrepo/repo.yaml
repo:
namespace: 'myrepo'
api: v2.0
Namespaces can also be nested, which can be useful if you have
multiple package repositories for an organization. Spack will
create the corresponding directory structure for you:
By default, the namespace of a new repo matches its directory's name.
You can supply a custom namespace with a second argument, e.g.:
.. code-block:: console
$ spack repo create ~/my_spack_repos llnl.comp
$ spack repo create myrepo llnl.comp
==> Created repo with namespace 'llnl.comp'.
==> To register it with spack, run this command:
spack repo add ~/my_spack_repos/spack_repo/llnl/comp
spack repo add ~/myrepo
$ cat ~/my_spack_repos/spack_repo/llnl/comp/repo.yaml
$ cat myrepo/repo.yaml
repo:
namespace: 'llnl.comp'
api: v2.0
You can also create repositories with custom structure with the ``-d/--subdirectory``
argument, e.g.:
.. code-block:: console
$ spack repo create -d applications myrepo apps
==> Created repo with namespace 'apps'.
==> To register it with Spack, run this command:
spack repo add ~/myrepo
$ ls myrepo
applications/ repo.yaml
$ cat myrepo/repo.yaml
repo:
namespace: apps
subdirectory: applications
^^^^^^^^^^^^^^^^^^
``spack repo add``
^^^^^^^^^^^^^^^^^^
Once your repository is created, you can register it with Spack with
``spack repo add``. You nee to specify the path to the directory that
contains the ``repo.yaml`` file.
``spack repo add``:
.. code-block:: console
$ spack repo add ~/my_spack_repos/spack_repo/llnl/comp
$ spack repo add ./myrepo
==> Added repo with namespace 'llnl.comp'.
$ spack repo list
==> 2 package repositories.
llnl.comp v2.0 ~/my_spack_repos/spack_repo/llnl/comp
builtin v2.0 ~/spack/var/spack/repos/spack_repo/builtin
llnl.comp ~/myrepo
builtin ~/spack/var/spack/repos/builtin
This simply adds the repo to your ``repos.yaml`` file.
@@ -421,43 +432,46 @@ By namespace:
.. code-block:: console
$ spack repo rm llnl.comp
==> Removed repository ~/my_spack_repos/spack_repo/llnl/comp with namespace 'llnl.comp'.
==> Removed repository ~/myrepo with namespace 'llnl.comp'.
$ spack repo list
==> 1 package repository.
builtin ~/spack/var/spack/repos/spack_repo/builtin
builtin ~/spack/var/spack/repos/builtin
By path:
.. code-block:: console
$ spack repo rm ~/my_spack_repos/spack_repo/llnl/comp
==> Removed repository ~/my_spack_repos/spack_repo/llnl/comp
$ spack repo rm ~/myrepo
==> Removed repository ~/myrepo
$ spack repo list
==> 1 package repository.
builtin ~/spack/var/spack/repos/spack_repo/builtin
builtin ~/spack/var/spack/repos/builtin
--------------------------------
Repo namespaces and Python
--------------------------------
Package repositories are implemented as Python packages. To be precise,
they are `namespace packages
<https://packaging.python.org/en/latest/guides/packaging-namespace-packages/>`_
with ``spack_repo`` the top-level namespace, followed by the repository
namespace as submodules. For example, the builtin repository corresponds
to the Python module ``spack_repo.builtin.packages``.
You may have noticed that namespace notation for repositories is similar
to the notation for namespaces in Python. As it turns out, you *can*
treat Spack repositories like Python packages; this is how they are
implemented.
This structure allows you to extend a ``builtin`` package in your own
You could, for example, extend a ``builtin`` package in your own
repository:
.. code-block:: python
from spack_repo.builtin.packages.mpich.package import Mpich
from spack.pkg.builtin.mpich import Mpich
class MyPackage(Mpich):
...
Spack populates ``sys.path`` at runtime with the path to the root of your
package repository's ``spack_repo`` directory.
Spack repo namespaces are actually Python namespaces tacked on under
``spack.pkg``. The search semantics of ``repos.yaml`` are actually
implemented using Python's built-in `sys.path
<https://docs.python.org/2/library/sys.html#sys.path>`_ search. The
:py:mod:`spack.repo` module implements a custom `Python importer
<https://docs.python.org/2/library/imp.html>`_.

View File

@@ -21,7 +21,6 @@
Dict,
Generic,
Iterable,
Iterator,
List,
Mapping,
Optional,
@@ -437,39 +436,46 @@ def add_func_to_class(name, func):
return cls
K = TypeVar("K")
V = TypeVar("V")
@lazy_lexicographic_ordering
class HashableMap(typing.MutableMapping[K, V]):
class HashableMap(collections.abc.MutableMapping):
"""This is a hashable, comparable dictionary. Hash is performed on
a tuple of the values in the dictionary."""
__slots__ = ("dict",)
def __init__(self):
self.dict: Dict[K, V] = {}
self.dict = {}
def __getitem__(self, key: K) -> V:
def __getitem__(self, key):
return self.dict[key]
def __setitem__(self, key: K, value: V) -> None:
def __setitem__(self, key, value):
self.dict[key] = value
def __iter__(self) -> Iterator[K]:
def __iter__(self):
return iter(self.dict)
def __len__(self) -> int:
def __len__(self):
return len(self.dict)
def __delitem__(self, key: K) -> None:
def __delitem__(self, key):
del self.dict[key]
def _cmp_iter(self):
for _, v in sorted(self.items()):
yield v
def copy(self):
"""Type-agnostic clone method. Preserves subclass type."""
# Construct a new dict of my type
self_type = type(self)
clone = self_type()
# Copy everything from this dict into it.
for key in self:
clone[key] = self[key].copy()
return clone
def match_predicate(*args):
"""Utility function for making string matching predicates.

View File

@@ -18,7 +18,7 @@
#: version is incremented when the package API is extended in a backwards-compatible way. The major
#: version is incremented upon breaking changes. This version is changed independently from the
#: Spack version.
package_api_version = (2, 0)
package_api_version = (1, 0)
#: The minimum Package API version that this version of Spack is compatible with. This should
#: always be a tuple of the form ``(major, 0)``, since compatibility with vX.Y implies

View File

@@ -1055,8 +1055,8 @@ def setup_dependent_build_environment(
) -> None:
# NB: This function is overwritten by 'mpi' provider packages:
#
# var/spack/repos/spack_repo/builtin/packages/intel_mpi/package.py
# var/spack/repos/spack_repo/builtin/packages/intel_parallel_studio/package.py
# var/spack/repos/builtin/packages/intel-mpi/package.py
# var/spack/repos/builtin/packages/intel-parallel-studio/package.py
#
# They call _setup_dependent_env_callback() as well, but with the
# dictionary kwarg compilers_of_client{} present and populated.

View File

@@ -1,12 +1,22 @@
# Copyright Spack Project Developers. See COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from typing import Optional, Tuple
import re
from typing import Dict, Generator, Optional, Set, Tuple, Union
import llnl.util.filesystem as fs
import llnl.util.lang as lang
import llnl.util.tty as tty
from llnl.util.filesystem import mkdirp
from llnl.util.lang import ClassProperty, classproperty
import spack.builder
import spack.deptypes as dt
import spack.phase_callbacks
from spack.dependency import Dependency
from spack.directives import extends
from spack.error import SpackError
from spack.spec import Spec
from .generic import GenericBuilder, Package
@@ -34,6 +44,169 @@ def configure_vars(self):
"""Arguments to pass to install via ``--configure-vars``."""
return []
@staticmethod
def parse_description(data: str) -> Generator:
"""Parses CRAN package metadata from
https://cran.r-project.org/src/contrib/PACKAGES
and returns the list of dictionaries.
Args:
data (str): raw text from the package list
Returns:
(Generator): each entry from packages as dictionary
Note: based on PyPI pycran v0.2.0 under Apache-2.0 license.
"""
fields: Set = set()
package: Dict = {}
def append(field_value: Union[bytes, str]):
pairs = list(package.items())
if pairs:
last_field = pairs[-1][0]
package[last_field] += field_value
# We want to iterate over each line and accumulate
# keys in dictionary, once we meet the same key
# in our dictionary we have a single package
# metadata parsed so we yield and repeat again.
for line in data.splitlines():
if not line.strip():
continue
if ":" in line:
parts = line.split(":", maxsplit=1)
field = parts[0].strip()
value = str("".join(parts[1:]).strip())
if not field[0].isalpha():
field = ""
value = line
if field and field in fields:
fields = {field}
result = {**package}
package = {field: value}
if result:
yield result
else:
# Here we want to parse dangling lines
# like the ones with long dependency
# list, `R (>= 2.15.0), xtable, pbapply ... \n and more`
if field:
package[field] = value.strip()
fields.add(field)
else:
append(f" {value.strip()}")
else:
append(f" {line.strip()}")
# We also need to return the metadata for
# the last parsed package.
if package:
yield package
@spack.phase_callbacks.run_before("install")
def _verify_package(self):
if not self.pkg.run_tests:
return
# Read DESCRIPTION file with dependency information
r_deps = []
with open(fs.join_path(self.stage.source_path, "DESCRIPTION")) as file:
for desc in RBuilder.parse_description(file.read()):
for field in [f for f in ["Depends", "Imports", "LinkingTo"] if f in desc]:
r_deps.extend([d.strip() for d in desc[field].split(",") if d != ""])
tty.debug(f"DESCRIPTION: {r_deps}")
# Convert to spack dependencies format for comparison
deps = {}
r_core = [
"r-compiler",
"r-graphics",
"r-grdevices",
"r-grid",
"r-methods",
"r-parallel",
"r-splines",
"r-stats",
"r-stats4",
"r-tcltk",
"r-tools",
"r-utils",
]
for r_dep in r_deps:
p = re.search(r"^[\w_.-]+", r_dep) # first word, incl. underscore, dot, or dash
v = re.search("(?<=[(]).*(?=[)])", r_dep) # everything between parentheses
# require valid package
assert p, f"Unable to find package name in {r_dep}"
r_spec = f"r-{p[0].strip().lower()}" if p[0].lower() != "r" else "r"
r_spec = re.sub(r"\.", "-", r_spec) # dot to dash
# filter R core packages
if r_spec in r_core:
r_spec = "r"
# allow minimum or pinned versions
if v:
v = re.sub(r">=\s([\d.-]+)", r"@\1:", v[0]) # >=
v = re.sub(r">\s([\d.-]+)", r"@\1.1:", v) # >
v = re.sub(r"==\s([\d.-]+)", r"@\1", v) # ==
else:
v = ""
# merge dependencies as they are added
if r_spec in deps:
deps[r_spec].merge(Dependency(self.pkg, Spec(r_spec + v), dt.BUILD | dt.RUN))
else:
deps[r_spec] = Dependency(self.pkg, Spec(r_spec + v), dt.BUILD | dt.RUN)
tty.debug(f"Converted: {deps}")
# Retrieve dependencies for current spack package and version
spack_dependencies = []
for when, dep in self.pkg.dependencies.items():
if self.spec.satisfies(when):
spack_dependencies.append(dep)
tty.debug(f"Spack as read: {spack_dependencies}")
merged_dependencies = {}
for dep in spack_dependencies:
for n, d in dep.items():
if n in merged_dependencies:
merged_dependencies[n].merge(d)
else:
merged_dependencies[n] = d
tty.debug(f"Spack merged: {merged_dependencies}")
# For each R dependency, ensure Spack dependency is at least as strong
missing_deps = []
for dep in sorted(deps.keys()):
if dep in list(merged_dependencies.keys()):
# Spack dependency must satisfy R dependency
if not merged_dependencies[dep].spec.satisfies(deps[dep].spec):
missing_deps.append(
f' depends_on("{deps[dep].spec}",'
+ ' type=("build", "run"),'
+ f' when="@{self.pkg.version}:")'
)
# Remove from dict
del merged_dependencies[dep]
else:
missing_deps.append(
f' depends_on("{deps[dep].spec}",'
+ ' type=("build", "run"),'
+ f' when="@{self.pkg.version}:")'
)
for dep in merged_dependencies:
if re.match("^r-.*", dep):
missing_deps.append(
f' #depends_on("{merged_dependencies[dep].spec}") not needed anymore'
)
# Raise exception
if len(missing_deps) > 0:
raise SpackError(
"This package requires stricter dependencies than specified:\n\n"
+ "\n".join(missing_deps)
)
def install(self, pkg, spec, prefix):
"""Installs an R package."""
mkdirp(pkg.module.r_lib_dir)

View File

@@ -59,7 +59,7 @@ def __call__(self, spec, prefix):
def get_builder_class(pkg, name: str) -> Optional[Type["Builder"]]:
"""Return the builder class if a package module defines it."""
cls = getattr(pkg.module, name, None)
if cls and spack.repo.is_package_module(cls.__module__):
if cls and cls.__module__.startswith(spack.repo.ROOT_PYTHON_NAMESPACE):
return cls
return None
@@ -121,7 +121,6 @@ def __init__(self, wrapped_pkg_object, root_builder):
new_cls_name,
bases,
{
"__module__": package_cls.__module__,
"run_tests": property(lambda x: x.wrapped_package_object.run_tests),
"test_requires_compiler": property(
lambda x: x.wrapped_package_object.test_requires_compiler
@@ -130,6 +129,7 @@ def __init__(self, wrapped_pkg_object, root_builder):
"tester": property(lambda x: x.wrapped_package_object.tester),
},
)
new_cls.__module__ = package_cls.__module__
self.__class__ = new_cls
self.__dict__.update(wrapped_pkg_object.__dict__)

View File

@@ -150,10 +150,10 @@ def get_stack_changed(env_path, rev1="HEAD^", rev2="HEAD"):
return False
def compute_affected_packages(rev1: str = "HEAD^", rev2: str = "HEAD") -> Set[str]:
def compute_affected_packages(rev1="HEAD^", rev2="HEAD"):
"""Determine which packages were added, removed or changed
between rev1 and rev2, and return the names as a set"""
return spack.repo.get_all_package_diffs("ARC", spack.repo.builtin_repo(), rev1=rev1, rev2=rev2)
return spack.repo.get_all_package_diffs("ARC", rev1=rev1, rev2=rev2)
def get_spec_filter_list(env, affected_pkgs, dependent_traverse_depth=None):

View File

@@ -791,9 +791,7 @@ def ci_verify_versions(args):
"""
# Get a list of all packages that have been changed or added
# between from_ref and to_ref
pkgs = spack.repo.get_all_package_diffs(
"AC", spack.repo.builtin_repo(), args.from_ref, args.to_ref
)
pkgs = spack.repo.get_all_package_diffs("AC", args.from_ref, args.to_ref)
failed_version = False
for pkg_name in pkgs:

View File

@@ -23,7 +23,7 @@
from spack.util.editor import editor
from spack.util.executable import which
from spack.util.format import get_version_lines
from spack.util.naming import pkg_name_to_class_name, simplify_name
from spack.util.naming import mod_to_class, simplify_name, valid_fully_qualified_module_name
description = "create a new package file"
section = "packaging"
@@ -95,7 +95,7 @@ class BundlePackageTemplate:
def __init__(self, name: str, versions, languages: List[str]):
self.name = name
self.class_name = pkg_name_to_class_name(name)
self.class_name = mod_to_class(name)
self.versions = versions
self.languages = languages
@@ -874,7 +874,7 @@ def get_name(name, url):
result = simplify_name(result)
if not re.match(r"^[a-z0-9-]+$", result):
if not valid_fully_qualified_module_name(result):
tty.die("Package name can only contain a-z, 0-9, and '-'")
return result

View File

@@ -10,13 +10,11 @@
import re
import sys
from html import escape
from typing import Type
import llnl.util.tty as tty
from llnl.util.tty.colify import colify
import spack.deptypes as dt
import spack.package_base
import spack.repo
from spack.cmd.common import arguments
from spack.version import VersionList
@@ -141,10 +139,10 @@ def name_only(pkgs, out):
tty.msg("%d packages" % len(pkgs))
def github_url(pkg: Type[spack.package_base.PackageBase]) -> str:
def github_url(pkg):
"""Link to a package file on github."""
mod_path = pkg.__module__.replace(".", "/")
return f"https://github.com/spack/spack/blob/develop/var/spack/{mod_path}.py"
url = "https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/{0}/package.py"
return url.format(pkg.name)
def rows_for_ncols(elts, ncols):

View File

@@ -89,17 +89,17 @@ def setup_parser(subparser):
def pkg_add(args):
"""add a package to the git stage with `git add`"""
spack.repo.add_package_to_git_stage(args.packages, spack.repo.builtin_repo())
spack.repo.add_package_to_git_stage(args.packages)
def pkg_list(args):
"""list packages associated with a particular spack git revision"""
colify(spack.repo.list_packages(args.rev, spack.repo.builtin_repo()))
colify(spack.repo.list_packages(args.rev))
def pkg_diff(args):
"""compare packages available in two different git revisions"""
u1, u2 = spack.repo.diff_packages(args.rev1, args.rev2, spack.repo.builtin_repo())
u1, u2 = spack.repo.diff_packages(args.rev1, args.rev2)
if u1:
print("%s:" % args.rev1)
@@ -114,23 +114,21 @@ def pkg_diff(args):
def pkg_removed(args):
"""show packages removed since a commit"""
u1, u2 = spack.repo.diff_packages(args.rev1, args.rev2, spack.repo.builtin_repo())
u1, u2 = spack.repo.diff_packages(args.rev1, args.rev2)
if u1:
colify(sorted(u1))
def pkg_added(args):
"""show packages added since a commit"""
u1, u2 = spack.repo.diff_packages(args.rev1, args.rev2, spack.repo.builtin_repo())
u1, u2 = spack.repo.diff_packages(args.rev1, args.rev2)
if u2:
colify(sorted(u2))
def pkg_changed(args):
"""show packages changed since a commit"""
packages = spack.repo.get_all_package_diffs(
args.type, spack.repo.builtin_repo(), args.rev1, args.rev2
)
packages = spack.repo.get_all_package_diffs(args.type, args.rev1, args.rev2)
if packages:
colify(sorted(packages))

View File

@@ -4,7 +4,6 @@
import os
import sys
from typing import List
import llnl.util.tty as tty
@@ -25,7 +24,9 @@ def setup_parser(subparser):
create_parser = sp.add_parser("create", help=repo_create.__doc__)
create_parser.add_argument("directory", help="directory to create the repo in")
create_parser.add_argument(
"namespace", help="name or namespace to identify packages in the repository"
"namespace",
help="namespace to identify packages in the repository (defaults to the directory name)",
nargs="?",
)
create_parser.add_argument(
"-d",
@@ -137,7 +138,7 @@ def repo_remove(args):
def repo_list(args):
"""show registered repositories and their namespaces"""
roots = spack.config.get("repos", scope=args.scope)
repos: List[spack.repo.Repo] = []
repos = []
for r in roots:
try:
repos.append(spack.repo.from_path(r))
@@ -145,14 +146,17 @@ def repo_list(args):
continue
if sys.stdout.isatty():
tty.msg(f"{len(repos)} package repositor" + ("y." if len(repos) == 1 else "ies."))
msg = "%d package repositor" % len(repos)
msg += "y." if len(repos) == 1 else "ies."
tty.msg(msg)
if not repos:
return
max_ns_len = max(len(r.namespace) for r in repos)
for repo in repos:
print(f"{repo.namespace:<{max_ns_len + 4}}{repo.package_api_str:<8}{repo.root}")
fmt = "%%-%ds%%s" % (max_ns_len + 4)
print(fmt % (repo.namespace, repo.root))
def repo(parser, args):

View File

@@ -136,7 +136,20 @@ def solve(parser, args):
setup_only = set(show) == {"asp"}
unify = spack.config.get("concretizer:unify")
allow_deprecated = spack.config.get("config:deprecated", False)
if unify == "when_possible":
if unify != "when_possible":
# set up solver parameters
# Note: reuse and other concretizer prefs are passed as configuration
result = solver.solve(
specs,
out=output,
timers=args.timers,
stats=args.stats,
setup_only=setup_only,
allow_deprecated=allow_deprecated,
)
if not setup_only:
_process_result(result, show, required_format, kwargs)
else:
for idx, result in enumerate(
solver.solve_in_rounds(
specs,
@@ -153,29 +166,3 @@ def solve(parser, args):
print("% END ROUND {0}\n".format(idx))
if not setup_only:
_process_result(result, show, required_format, kwargs)
elif unify:
# set up solver parameters
# Note: reuse and other concretizer prefs are passed as configuration
result = solver.solve(
specs,
out=output,
timers=args.timers,
stats=args.stats,
setup_only=setup_only,
allow_deprecated=allow_deprecated,
)
if not setup_only:
_process_result(result, show, required_format, kwargs)
else:
for spec in specs:
tty.msg("SOLVING SPEC:", spec)
result = solver.solve(
[spec],
out=output,
timers=args.timers,
stats=args.stats,
setup_only=setup_only,
allow_deprecated=allow_deprecated,
)
if not setup_only:
_process_result(result, show, required_format, kwargs)

View File

@@ -59,7 +59,7 @@ def is_package(f):
packages, since we allow `from spack import *` and poking globals
into packages.
"""
return f.startswith("var/spack/") and f.endswith("package.py")
return f.startswith("var/spack/repos/") and f.endswith("package.py")
#: decorator for adding tools to the list
@@ -380,7 +380,7 @@ def run_black(black_cmd, file_list, args):
def _module_part(root: str, expr: str):
parts = expr.split(".")
# spack.pkg is for repositories, don't try to resolve it here.
if expr.startswith(spack.repo.PKG_MODULE_PREFIX_V1) or expr == "spack.pkg":
if ".".join(parts[:2]) == spack.repo.ROOT_PYTHON_NAMESPACE:
return None
while parts:
f1 = os.path.join(root, "lib", "spack", *parts) + ".py"

View File

@@ -65,7 +65,7 @@ def __init__(cls: "DirectiveMeta", name: str, bases: tuple, attr_dict: dict):
# The instance is being initialized: if it is a package we must ensure
# that the directives are called to set it up.
if spack.repo.is_package_module(cls.__module__):
if cls.__module__.startswith(spack.repo.ROOT_PYTHON_NAMESPACE):
# Ensure the presence of the dictionaries associated with the directives.
# All dictionaries are defaultdicts that create lists for missing keys.
for d in DirectiveMeta._directive_dict_names:

View File

@@ -2312,12 +2312,8 @@ def update_environment_repository(self) -> None:
def _add_to_environment_repository(self, spec_node: Spec) -> None:
"""Add the root node of the spec to the environment repository"""
namespace: str = spec_node.namespace
repository = spack.repo.create_or_construct(
root=os.path.join(self.repos_path, namespace),
namespace=namespace,
package_api=spack.repo.PATH.get_repo(namespace).package_api,
)
repository_dir = os.path.join(self.repos_path, spec_node.namespace)
repository = spack.repo.create_or_construct(repository_dir, spec_node.namespace)
pkg_dir = repository.dirname_for_package_name(spec_node.name)
fs.mkdirp(pkg_dir)
spack.repo.PATH.dump_provenance(spec_node, pkg_dir)

View File

@@ -566,11 +566,10 @@ def dump_packages(spec: "spack.spec.Spec", path: str) -> None:
tty.warn(f"Warning: Couldn't copy in provenance for {node.name}")
# Create a destination repository
pkg_api = spack.repo.PATH.get_repo(node.namespace).package_api
repo_root = os.path.join(path, node.namespace) if pkg_api < (2, 0) else path
repo = spack.repo.create_or_construct(
repo_root, namespace=node.namespace, package_api=pkg_api
)
dest_repo_root = os.path.join(path, node.namespace)
if not os.path.exists(dest_repo_root):
spack.repo.create_repo(dest_repo_root)
repo = spack.repo.from_path(dest_repo_root)
# Get the location of the package in the dest repo.
dest_pkg_dir = repo.dirname_for_package_name(node.name)

View File

@@ -47,7 +47,6 @@
import spack.url
import spack.util.environment
import spack.util.executable
import spack.util.naming
import spack.util.path
import spack.util.web
import spack.variant
@@ -839,36 +838,26 @@ def fullname(cls):
def fullnames(cls):
"""Fullnames for this package and any packages from which it inherits."""
fullnames = []
for base in cls.__mro__:
if not spack.repo.is_package_module(base.__module__):
for cls in cls.__mro__:
namespace = getattr(cls, "namespace", None)
if namespace:
fullnames.append("%s.%s" % (namespace, cls.name))
if namespace == "builtin":
# builtin packages cannot inherit from other repos
break
fullnames.append(base.fullname)
return fullnames
@classproperty
def name(cls):
"""The name of this package."""
"""The name of this package.
The name of a package is the name of its Python module, without
the containing module names.
"""
if cls._name is None:
# We cannot know the exact package API version, but we can distinguish between v1
# v2 based on the module. We don't want to figure out the exact package API version
# since it requires parsing the repo.yaml.
module = cls.__module__
if module.startswith(spack.repo.PKG_MODULE_PREFIX_V1):
version = (1, 0)
elif module.startswith(spack.repo.PKG_MODULE_PREFIX_V2):
version = (2, 0)
else:
raise ValueError(f"Package {cls.__qualname__} is not a known Spack package")
if version < (2, 0):
# spack.pkg.builtin.package_name.
_, _, pkg_module = module.rpartition(".")
else:
# spack_repo.builtin.packages.package_name.package
pkg_module = module.rsplit(".", 2)[-2]
cls._name = spack.util.naming.pkg_dir_to_pkg_name(pkg_module, version)
cls._name = cls.module.__name__
if "." in cls._name:
cls._name = cls._name[cls._name.rindex(".") + 1 :]
return cls._name
@classproperty

View File

@@ -56,9 +56,8 @@
# read-only things in $spack/var/spack
repos_path = os.path.join(var_path, "repos")
test_repos_path = os.path.join(var_path, "test_repos")
packages_path = os.path.join(repos_path, "spack_repo", "builtin")
mock_packages_path = os.path.join(test_repos_path, "builtin.mock")
packages_path = os.path.join(repos_path, "builtin")
mock_packages_path = os.path.join(repos_path, "builtin.mock")
#
# Writable things in $spack/var/spack

View File

@@ -47,34 +47,40 @@
import spack.util.path
import spack.util.spack_yaml as syaml
PKG_MODULE_PREFIX_V1 = "spack.pkg."
PKG_MODULE_PREFIX_V2 = "spack_repo."
#: Package modules are imported as spack.pkg.<repo-namespace>.<pkg-name>
ROOT_PYTHON_NAMESPACE = "spack.pkg"
_API_REGEX = re.compile(r"^v(\d+)\.(\d+)$")
def is_package_module(fullname: str) -> bool:
"""Check if the given module is a package module."""
return fullname.startswith(PKG_MODULE_PREFIX_V1) or fullname.startswith(PKG_MODULE_PREFIX_V2)
def python_package_for_repo(namespace):
"""Returns the full namespace of a repository, given its relative one
For instance:
python_package_for_repo('builtin') == 'spack.pkg.builtin'
Args:
namespace (str): repo namespace
"""
return "{0}.{1}".format(ROOT_PYTHON_NAMESPACE, namespace)
def namespace_from_fullname(fullname: str) -> str:
def namespace_from_fullname(fullname):
"""Return the repository namespace only for the full module name.
For instance:
namespace_from_fullname("spack.pkg.builtin.hdf5") == "builtin"
namespace_from_fullname("spack_repo.x.y.z.packages.pkg_name.package") == "x.y.z"
namespace_from_fullname('spack.pkg.builtin.hdf5') == 'builtin'
Args:
fullname: full name for the Python module
fullname (str): full name for the Python module
"""
if fullname.startswith(PKG_MODULE_PREFIX_V1):
namespace, _, _ = fullname.rpartition(".")
return namespace[len(PKG_MODULE_PREFIX_V1) :]
elif fullname.startswith(PKG_MODULE_PREFIX_V2) and fullname.endswith(".package"):
return ".".join(fullname.split(".")[1:-3])
return fullname
namespace, dot, module = fullname.rpartition(".")
prefix_and_dot = "{0}.".format(ROOT_PYTHON_NAMESPACE)
if namespace.startswith(prefix_and_dot):
namespace = namespace[len(prefix_and_dot) :]
return namespace
class SpackNamespaceLoader:
@@ -86,14 +92,14 @@ def exec_module(self, module):
class ReposFinder:
"""MetaPathFinder class that loads a Python module corresponding to an API v1 Spack package.
"""MetaPathFinder class that loads a Python module corresponding to a Spack package.
Returns a loader based on the inspection of the current repository list.
"""
def __init__(self):
self._repo_init = _path
self._repo: Optional[RepoType] = None
self._repo = None
@property
def current_repository(self):
@@ -121,7 +127,7 @@ def find_spec(self, fullname, python_path, target=None):
raise RuntimeError('cannot reload module "{0}"'.format(fullname))
# Preferred API from https://peps.python.org/pep-0451/
if not fullname.startswith(PKG_MODULE_PREFIX_V1) and fullname != "spack.pkg":
if not fullname.startswith(ROOT_PYTHON_NAMESPACE):
return None
loader = self.compute_loader(fullname)
@@ -129,17 +135,16 @@ def find_spec(self, fullname, python_path, target=None):
return None
return importlib.util.spec_from_loader(fullname, loader)
def compute_loader(self, fullname: str):
def compute_loader(self, fullname):
# namespaces are added to repo, and package modules are leaves.
namespace, dot, module_name = fullname.rpartition(".")
# If it's a module in some repo, or if it is the repo's namespace, let the repo handle it.
current_repo = self.current_repository
is_repo_path = isinstance(current_repo, RepoPath)
is_repo_path = isinstance(self.current_repository, RepoPath)
if is_repo_path:
repos = current_repo.repos
repos = self.current_repository.repos
else:
repos = [current_repo]
repos = [self.current_repository]
for repo in repos:
# We are using the namespace of the repo and the repo contains the package
@@ -156,9 +161,7 @@ def compute_loader(self, fullname: str):
# No repo provides the namespace, but it is a valid prefix of
# something in the RepoPath.
if is_repo_path and current_repo.by_namespace.is_prefix(
fullname[len(PKG_MODULE_PREFIX_V1) :]
):
if is_repo_path and self.current_repository.by_namespace.is_prefix(fullname):
return SpackNamespaceLoader()
return None
@@ -176,12 +179,12 @@ def compute_loader(self, fullname: str):
NOT_PROVIDED = object()
def builtin_repo() -> "Repo":
def packages_path():
"""Get the test repo if it is active, otherwise the builtin repo."""
try:
return PATH.get_repo("builtin.mock")
return PATH.get_repo("builtin.mock").packages_path
except UnknownNamespaceError:
return PATH.get_repo("builtin")
return PATH.get_repo("builtin").packages_path
class GitExe:
@@ -189,25 +192,24 @@ class GitExe:
# invocations.
#
# Not using -C as that is not supported for git < 1.8.5.
def __init__(self, packages_path: str):
def __init__(self):
self._git_cmd = spack.util.git.git(required=True)
self.packages_dir = packages_path
def __call__(self, *args, **kwargs) -> str:
with working_dir(self.packages_dir):
return self._git_cmd(*args, **kwargs, output=str)
def __call__(self, *args, **kwargs):
with working_dir(packages_path()):
return self._git_cmd(*args, **kwargs)
def list_packages(rev: str, repo: "Repo") -> List[str]:
def list_packages(rev):
"""List all packages associated with the given revision"""
git = GitExe(repo.packages_path)
git = GitExe()
# git ls-tree does not support ... merge-base syntax, so do it manually
if rev.endswith("..."):
ref = rev.replace("...", "")
rev = git("merge-base", ref, "HEAD").strip()
rev = git("merge-base", ref, "HEAD", output=str).strip()
output = git("ls-tree", "-r", "--name-only", rev)
output = git("ls-tree", "-r", "--name-only", rev, output=str)
# recursively list the packages directory
package_paths = [
@@ -215,54 +217,54 @@ def list_packages(rev: str, repo: "Repo") -> List[str]:
]
# take the directory names with one-level-deep package files
package_names = [
nm.pkg_dir_to_pkg_name(line[0], repo.package_api)
for line in package_paths
if len(line) == 2
]
package_names = sorted(set([line[0] for line in package_paths if len(line) == 2]))
return sorted(set(package_names))
return package_names
def diff_packages(rev1: str, rev2: str, repo: "Repo") -> Tuple[Set[str], Set[str]]:
def diff_packages(rev1, rev2):
"""Compute packages lists for the two revisions and return a tuple
containing all the packages in rev1 but not in rev2 and all the
packages in rev2 but not in rev1."""
p1 = set(list_packages(rev1, repo))
p2 = set(list_packages(rev2, repo))
p1 = set(list_packages(rev1))
p2 = set(list_packages(rev2))
return p1.difference(p2), p2.difference(p1)
def get_all_package_diffs(type: str, repo: "Repo", rev1="HEAD^1", rev2="HEAD") -> Set[str]:
"""Get packages changed, added, or removed (or any combination of those) since a commit.
def get_all_package_diffs(type, rev1="HEAD^1", rev2="HEAD"):
"""Show packages changed, added, or removed (or any combination of those)
since a commit.
Arguments:
type: String containing one or more of 'A', 'R', 'C'
rev1: Revision to compare against, default is 'HEAD^'
rev2: Revision to compare to rev1, default is 'HEAD'
type (str): String containing one or more of 'A', 'R', 'C'
rev1 (str): Revision to compare against, default is 'HEAD^'
rev2 (str): Revision to compare to rev1, default is 'HEAD'
Returns:
A set contain names of affected packages.
"""
lower_type = type.lower()
if not re.match("^[arc]*$", lower_type):
tty.die(
f"Invalid change type: '{type}'. "
"Can contain only A (added), R (removed), or C (changed)"
"Invald change type: '%s'." % type,
"Can contain only A (added), R (removed), or C (changed)",
)
removed, added = diff_packages(rev1, rev2, repo)
removed, added = diff_packages(rev1, rev2)
git = GitExe(repo.packages_path)
out = git("diff", "--relative", "--name-only", rev1, rev2).strip()
git = GitExe()
out = git("diff", "--relative", "--name-only", rev1, rev2, output=str).strip()
lines = [] if not out else re.split(r"\s+", out)
changed: Set[str] = set()
changed = set()
for path in lines:
dir_name, _, _ = path.partition("/")
pkg_name = nm.pkg_dir_to_pkg_name(dir_name, repo.package_api)
pkg_name, _, _ = path.partition("/")
if pkg_name not in added and pkg_name not in removed:
changed.add(pkg_name)
packages: Set[str] = set()
packages = set()
if "a" in lower_type:
packages |= added
if "r" in lower_type:
@@ -273,14 +275,14 @@ def get_all_package_diffs(type: str, repo: "Repo", rev1="HEAD^1", rev2="HEAD") -
return packages
def add_package_to_git_stage(packages: List[str], repo: "Repo") -> None:
def add_package_to_git_stage(packages):
"""add a package to the git stage with `git add`"""
git = GitExe(repo.packages_path)
git = GitExe()
for pkg_name in packages:
filename = PATH.filename_for_package_name(pkg_name)
if not os.path.isfile(filename):
tty.die(f"No such package: {pkg_name}. Path does not exist:", filename)
tty.die("No such package: %s. Path does not exist:" % pkg_name, filename)
git("add", filename)
@@ -350,10 +352,9 @@ class FastPackageChecker(collections.abc.Mapping):
#: Global cache, reused by every instance
_paths_cache: Dict[str, Dict[str, os.stat_result]] = {}
def __init__(self, packages_path: str, package_api: Tuple[int, int]):
def __init__(self, packages_path):
# The path of the repository managed by this instance
self.packages_path = packages_path
self.package_api = package_api
# If the cache we need is not there yet, then build it appropriately
if packages_path not in self._paths_cache:
@@ -378,38 +379,41 @@ def _create_new_cache(self) -> Dict[str, os.stat_result]:
# Create a dictionary that will store the mapping between a
# package name and its stat info
cache: Dict[str, os.stat_result] = {}
with os.scandir(self.packages_path) as entries:
for entry in entries:
# Construct the file name from the directory
pkg_file = os.path.join(entry.path, package_file_name)
for pkg_name in os.listdir(self.packages_path):
# Skip non-directories in the package root.
pkg_dir = os.path.join(self.packages_path, pkg_name)
try:
sinfo = os.stat(pkg_file)
except OSError as e:
if e.errno in (errno.ENOENT, errno.ENOTDIR):
# No package.py file here.
continue
elif e.errno == errno.EACCES:
tty.warn(f"Can't read package file {pkg_file}.")
continue
raise e
# If it's not a file, skip it.
if not stat.S_ISREG(sinfo.st_mode):
continue
# Only consider package.py files in directories that are valid module names under
# the current package API
if not nm.valid_module_name(entry.name, self.package_api):
x, y = self.package_api
# Warn about invalid names that look like packages.
if not nm.valid_module_name(pkg_name):
if not pkg_name.startswith(".") and pkg_name != "repo.yaml":
tty.warn(
f"Package {pkg_file} cannot be used because `{entry.name}` is not a valid "
f"Spack package module name for Package API v{x}.{y}."
'Skipping package at {0}. "{1}" is not '
"a valid Spack module name.".format(pkg_dir, pkg_name)
)
continue
continue
# Store the stat info by package name.
cache[nm.pkg_dir_to_pkg_name(entry.name, self.package_api)] = sinfo
# Construct the file name from the directory
pkg_file = os.path.join(self.packages_path, pkg_name, package_file_name)
# Use stat here to avoid lots of calls to the filesystem.
try:
sinfo = os.stat(pkg_file)
except OSError as e:
if e.errno == errno.ENOENT:
# No package.py file here.
continue
elif e.errno == errno.EACCES:
tty.warn("Can't read package file %s." % pkg_file)
continue
raise e
# If it's not a file, skip it.
if stat.S_ISDIR(sinfo.st_mode):
continue
# If it is a file, then save the stats under the
# appropriate key
cache[pkg_name] = sinfo
return cache
@@ -684,7 +688,7 @@ def put_first(self, repo: "Repo") -> None:
return
self.repos.insert(0, repo)
self.by_namespace[repo.namespace] = repo
self.by_namespace[repo.full_namespace] = repo
def put_last(self, repo):
"""Add repo last in the search path."""
@@ -696,8 +700,8 @@ def put_last(self, repo):
self.repos.append(repo)
# don't mask any higher-precedence repos with same namespace
if repo.namespace not in self.by_namespace:
self.by_namespace[repo.namespace] = repo
if repo.full_namespace not in self.by_namespace:
self.by_namespace[repo.full_namespace] = repo
def remove(self, repo):
"""Remove a repo from the search path."""
@@ -706,9 +710,10 @@ def remove(self, repo):
def get_repo(self, namespace: str) -> "Repo":
"""Get a repository by namespace."""
if namespace not in self.by_namespace:
full_namespace = python_package_for_repo(namespace)
if full_namespace not in self.by_namespace:
raise UnknownNamespaceError(namespace)
return self.by_namespace[namespace]
return self.by_namespace[full_namespace]
def first_repo(self) -> Optional["Repo"]:
"""Get the first repo in precedence order."""
@@ -816,9 +821,10 @@ def repo_for_pkg(self, spec: Union[str, "spack.spec.Spec"]) -> "Repo":
# If the spec already has a namespace, then return the
# corresponding repo if we know about it.
if namespace:
if namespace not in self.by_namespace:
fullspace = python_package_for_repo(namespace)
if fullspace not in self.by_namespace:
raise UnknownNamespaceError(namespace, name=name)
return self.by_namespace[namespace]
return self.by_namespace[fullspace]
# If there's no namespace, search in the RepoPath.
for repo in self.repos:
@@ -839,15 +845,8 @@ def get(self, spec: "spack.spec.Spec") -> "spack.package_base.PackageBase":
assert isinstance(spec, spack.spec.Spec) and spec.concrete, msg
return self.repo_for_pkg(spec).get(spec)
def python_paths(self) -> List[str]:
"""Return a list of all the Python paths in the repos."""
return [repo.python_path for repo in self.repos if repo.python_path]
def get_pkg_class(self, pkg_name: str) -> Type["spack.package_base.PackageBase"]:
"""Find a class for the spec's package and return the class object."""
for p in self.python_paths():
if p not in sys.path:
sys.path.insert(0, p)
return self.repo_for_pkg(pkg_name).get_pkg_class(pkg_name)
@autospec
@@ -943,30 +942,6 @@ def _parse_package_api_version(
)
def _validate_and_normalize_subdir(subdir: Any, root: str, package_api: Tuple[int, int]) -> str:
if not isinstance(subdir, str):
raise BadRepoError(f"Invalid subdirectory '{subdir}' in '{root}'. Must be a string")
if package_api < (2, 0):
return subdir # In v1.x we did not validate subdir names
if subdir in (".", ""):
raise BadRepoError(
f"Invalid subdirectory '{subdir}' in '{root}'. Use a symlink packages -> . instead"
)
# Otherwise we expect a directory name (not path) that can be used as a Python module.
if os.sep in subdir:
raise BadRepoError(
f"Invalid subdirectory '{subdir}' in '{root}'. Expected a directory name, not a path"
)
if not nm.valid_module_name(subdir, package_api):
raise BadRepoError(
f"Invalid subdirectory '{subdir}' in '{root}'. Must be a valid Python module name"
)
return subdir
class Repo:
"""Class representing a package repository in the filesystem.
@@ -987,8 +962,6 @@ class Repo:
:py:data:`spack.package_api_version`.
"""
namespace: str
def __init__(
self,
root: str,
@@ -1018,79 +991,32 @@ def check(condition, msg):
# Read configuration and validate namespace
config = self._read_config()
self.package_api = _parse_package_api_version(config)
self.subdirectory = _validate_and_normalize_subdir(
config.get("subdirectory", packages_dir_name), root, self.package_api
)
self.packages_path = os.path.join(self.root, self.subdirectory)
check(
os.path.isdir(self.packages_path),
f"No directory '{self.subdirectory}' found in '{root}'",
"namespace" in config,
f"{os.path.join(root, repo_config_name)} must define a namespace.",
)
# The parent dir of spack_repo/ which should be added to sys.path for api v2.x
self.python_path: Optional[str] = None
if self.package_api < (2, 0):
check(
"namespace" in config,
f"{os.path.join(root, repo_config_name)} must define a namespace.",
)
self.namespace = config["namespace"]
# Note: for Package API v1.x the namespace validation always had bugs, which won't be
# fixed for compatibility reasons. The regex is missing "$" at the end, and it claims
# to test for valid identifiers, but fails to split on `.` first.
check(
isinstance(self.namespace, str)
and re.match(r"[a-zA-Z][a-zA-Z0-9_.]+", self.namespace),
f"Invalid namespace '{self.namespace}' in repo '{self.root}'. "
"Namespaces must be valid python identifiers separated by '.'",
)
else:
# From Package API v2.0 the namespace follows from the directory structure.
check(
f"{os.sep}spack_repo{os.sep}" in self.root,
f"Invalid repository path '{self.root}'. "
f"Path must contain 'spack_repo{os.sep}'",
)
derived_namespace = self.root.rpartition(f"spack_repo{os.sep}")[2].replace(os.sep, ".")
if "namespace" in config:
self.namespace = config["namespace"]
check(
isinstance(self.namespace, str) and self.namespace == derived_namespace,
f"Namespace '{self.namespace}' should be {derived_namespace} or omitted in "
f"{os.path.join(root, repo_config_name)}",
)
else:
self.namespace = derived_namespace
# strip the namespace directories from the root path to get the python path
# e.g. /my/pythonpath/spack_repo/x/y/z -> /my/pythonpath
python_path = self.root
for _ in self.namespace.split("."):
python_path = os.path.dirname(python_path)
self.python_path = os.path.dirname(python_path)
# check that all subdirectories are valid module names
check(
all(nm.valid_module_name(x, self.package_api) for x in self.namespace.split(".")),
f"Invalid namespace '{self.namespace}' in repo '{self.root}'",
)
self.namespace: str = config["namespace"]
check(
re.match(r"[a-zA-Z][a-zA-Z0-9_.]+", self.namespace),
f"Invalid namespace '{self.namespace}' in repo '{self.root}'. "
"Namespaces must be valid python identifiers separated by '.'",
)
# Set up 'full_namespace' to include the super-namespace
if self.package_api < (2, 0):
self.full_namespace = f"{PKG_MODULE_PREFIX_V1}{self.namespace}"
elif self.subdirectory == ".":
self.full_namespace = f"{PKG_MODULE_PREFIX_V2}{self.namespace}"
else:
self.full_namespace = f"{PKG_MODULE_PREFIX_V2}{self.namespace}.{self.subdirectory}"
self.full_namespace = python_package_for_repo(self.namespace)
# Keep name components around for checking prefixes.
self._names = self.full_namespace.split(".")
packages_dir: str = config.get("subdirectory", packages_dir_name)
self.packages_path = os.path.join(self.root, packages_dir)
check(
os.path.isdir(self.packages_path), f"No directory '{packages_dir}' found in '{root}'"
)
self.package_api = _parse_package_api_version(config)
# Class attribute overrides by package name
self.overrides = overrides or {}
@@ -1104,36 +1030,27 @@ def check(condition, msg):
self._repo_index: Optional[RepoIndex] = None
self._cache = cache
@property
def package_api_str(self) -> str:
return f"v{self.package_api[0]}.{self.package_api[1]}"
def finder(self, value: RepoPath) -> None:
self._finder = value
def real_name(self, import_name: str) -> Optional[str]:
"""Allow users to import Spack packages using Python identifiers.
In Package API v1.x, there was no canonical module name for a package, and package's dir
was not necessarily a valid Python module name. For that case we have to guess the actual
package directory. From Package API v2.0 there is a one-to-one mapping between Spack
package names and Python module names, so there is no guessing.
A python identifier might map to many different Spack package
names due to hyphen/underscore ambiguity.
For Packge API v1.x we support the following one-to-many mappings:
num3proxy -> 3proxy
Easy example:
num3proxy -> 3proxy
Ambiguous:
foo_bar -> foo_bar, foo-bar
More ambiguous:
foo_bar_baz -> foo_bar_baz, foo-bar-baz, foo_bar-baz, foo-bar_baz
"""
if self.package_api >= (2, 0):
if nm.pkg_dir_to_pkg_name(import_name, package_api=self.package_api) in self:
return import_name
return None
if import_name in self:
return import_name
# For v1 generate the possible package names from a module name, and return the first
# package name that exists in this repo.
options = nm.possible_spack_module_names(import_name)
try:
options.remove(import_name)
@@ -1266,9 +1183,7 @@ def extensions_for(
def dirname_for_package_name(self, pkg_name: str) -> str:
"""Given a package name, get the directory containing its package.py file."""
_, unqualified_name = self.partition_package_name(pkg_name)
return os.path.join(
self.packages_path, nm.pkg_name_to_pkg_dir(unqualified_name, self.package_api)
)
return os.path.join(self.packages_path, unqualified_name)
def filename_for_package_name(self, pkg_name: str) -> str:
"""Get the filename for the module we should load for a particular
@@ -1285,7 +1200,7 @@ def filename_for_package_name(self, pkg_name: str) -> str:
@property
def _pkg_checker(self) -> FastPackageChecker:
if self._fast_package_checker is None:
self._fast_package_checker = FastPackageChecker(self.packages_path, self.package_api)
self._fast_package_checker = FastPackageChecker(self.packages_path)
return self._fast_package_checker
def all_package_names(self, include_virtuals: bool = False) -> List[str]:
@@ -1297,9 +1212,7 @@ def all_package_names(self, include_virtuals: bool = False) -> List[str]:
def package_path(self, name: str) -> str:
"""Get path to package.py file for this repo."""
return os.path.join(
self.packages_path, nm.pkg_name_to_pkg_dir(name, self.package_api), package_file_name
)
return os.path.join(self.packages_path, name, package_file_name)
def all_package_paths(self) -> Generator[str, None, None]:
for name in self.all_package_names():
@@ -1357,19 +1270,15 @@ def get_pkg_class(self, pkg_name: str) -> Type["spack.package_base.PackageBase"]
package. Then extracts the package class from the module
according to Spack's naming convention.
"""
_, pkg_name = self.partition_package_name(pkg_name)
fullname = f"{self.full_namespace}.{nm.pkg_name_to_pkg_dir(pkg_name, self.package_api)}"
if self.package_api >= (2, 0):
fullname += ".package"
namespace, pkg_name = self.partition_package_name(pkg_name)
class_name = nm.mod_to_class(pkg_name)
fullname = f"{self.full_namespace}.{pkg_name}"
class_name = nm.pkg_name_to_class_name(pkg_name)
if self.python_path and self.python_path not in sys.path:
sys.path.insert(0, self.python_path)
try:
with REPOS_FINDER.switch_repo(self._finder or self):
module = importlib.import_module(fullname)
except ImportError as e:
raise UnknownPackageError(fullname) from e
except ImportError:
raise UnknownPackageError(fullname)
except Exception as e:
msg = f"cannot load package '{pkg_name}' from the '{self.namespace}' repository: {e}"
raise RepoError(msg) from e
@@ -1460,71 +1369,46 @@ def partition_package_name(pkg_name: str) -> Tuple[str, str]:
return namespace, pkg_name
def get_repo_yaml_dir(
root: str, namespace: Optional[str], package_api: Tuple[int, int]
) -> Tuple[str, str]:
"""Returns the directory where repo.yaml is located and the effective namespace."""
if package_api < (2, 0):
namespace = namespace or os.path.basename(root)
# This ad-hoc regex is left for historical reasons, and should not have a breaking change.
if not re.match(r"\w[\.\w-]*", namespace):
raise InvalidNamespaceError(f"'{namespace}' is not a valid namespace.")
return root, namespace
# Package API v2 has <root>/spack_repo/<namespace>/<subdir> structure and requires a namespace
if namespace is None:
raise InvalidNamespaceError("Namespace must be provided.")
# if namespace has dots those translate to subdirs of further namespace packages.
namespace_components = namespace.split(".")
if not all(nm.valid_module_name(n, package_api=package_api) for n in namespace_components):
raise InvalidNamespaceError(f"'{namespace}' is not a valid namespace." % namespace)
return os.path.join(root, "spack_repo", *namespace_components), namespace
def create_repo(
root,
namespace: Optional[str] = None,
subdir: str = packages_dir_name,
package_api: Tuple[int, int] = spack.package_api_version,
) -> Tuple[str, str]:
def create_repo(root, namespace=None, subdir=packages_dir_name):
"""Create a new repository in root with the specified namespace.
If the namespace is not provided, use basename of root.
Return the canonicalized path and namespace of the created repository.
"""
root = spack.util.path.canonicalize_path(root)
repo_yaml_dir, namespace = get_repo_yaml_dir(os.path.abspath(root), namespace, package_api)
if not namespace:
namespace = os.path.basename(root)
existed = True
try:
dir_entry = next(os.scandir(repo_yaml_dir), None)
except OSError as e:
if e.errno == errno.ENOENT:
existed = False
dir_entry = None
else:
raise BadRepoError(f"Cannot create new repo in {root}: {e}")
if not re.match(r"\w[\.\w-]*", namespace):
raise InvalidNamespaceError("'%s' is not a valid namespace." % namespace)
if dir_entry is not None:
raise BadRepoError(f"Cannot create new repo in {root}: directory is not empty.")
existed = False
if os.path.exists(root):
if os.path.isfile(root):
raise BadRepoError("File %s already exists and is not a directory" % root)
elif os.path.isdir(root):
if not os.access(root, os.R_OK | os.W_OK):
raise BadRepoError("Cannot create new repo in %s: cannot access directory." % root)
if os.listdir(root):
raise BadRepoError("Cannot create new repo in %s: directory is not empty." % root)
existed = True
config_path = os.path.join(repo_yaml_dir, repo_config_name)
subdir = _validate_and_normalize_subdir(subdir, root, package_api)
packages_path = os.path.join(repo_yaml_dir, subdir)
full_path = os.path.realpath(root)
parent = os.path.dirname(full_path)
if not os.access(parent, os.R_OK | os.W_OK):
raise BadRepoError("Cannot create repository in %s: can't access parent!" % root)
try:
config_path = os.path.join(root, repo_config_name)
packages_path = os.path.join(root, subdir)
fs.mkdirp(packages_path)
with open(config_path, "w", encoding="utf-8") as config:
config.write("repo:\n")
config.write(f" namespace: '{namespace}'\n")
if subdir != packages_dir_name:
config.write(f" subdirectory: '{subdir}'\n")
x, y = package_api
x, y = spack.package_api_version
config.write(f" api: v{x}.{y}\n")
except OSError as e:
@@ -1537,27 +1421,22 @@ def create_repo(
raise BadRepoError(
"Failed to create new repository in %s." % root, "Caused by %s: %s" % (type(e), e)
) from e
)
return repo_yaml_dir, namespace
return full_path, namespace
def from_path(path: str) -> Repo:
def from_path(path: str) -> "Repo":
"""Returns a repository from the path passed as input. Injects the global misc cache."""
return Repo(path, cache=spack.caches.MISC_CACHE)
def create_or_construct(
root: str,
namespace: Optional[str] = None,
package_api: Tuple[int, int] = spack.package_api_version,
) -> Repo:
def create_or_construct(path, namespace=None):
"""Create a repository, or just return a Repo if it already exists."""
repo_yaml_dir, _ = get_repo_yaml_dir(root, namespace, package_api)
if not os.path.exists(repo_yaml_dir):
fs.mkdirp(root)
create_repo(root, namespace=namespace, package_api=package_api)
return from_path(repo_yaml_dir)
if not os.path.exists(path):
fs.mkdirp(path)
create_repo(path, namespace)
return from_path(path)
def _path(configuration=None):
@@ -1635,10 +1514,8 @@ class MockRepositoryBuilder:
"""Build a mock repository in a directory"""
def __init__(self, root_directory, namespace=None):
namespace = namespace or "".join(random.choice(string.ascii_lowercase) for _ in range(10))
repo_root = os.path.join(root_directory, namespace)
os.mkdir(repo_root)
self.root, self.namespace = create_repo(repo_root, namespace)
namespace = namespace or "".join(random.choice(string.ascii_uppercase) for _ in range(10))
self.root, self.namespace = create_repo(str(root_directory), namespace)
def add_package(self, name, dependencies=None):
"""Create a mock package in the repository, using a Jinja2 template.
@@ -1650,7 +1527,7 @@ def add_package(self, name, dependencies=None):
``spack.dependency.default_deptype`` and ``spack.spec.Spec()`` are used.
"""
dependencies = dependencies or []
context = {"cls_name": nm.pkg_name_to_class_name(name), "dependencies": dependencies}
context = {"cls_name": nm.mod_to_class(name), "dependencies": dependencies}
template = spack.tengine.make_environment().get_template("mock-repository/package.pyt")
text = template.render(context)
package_py = self.recipe_filename(name)
@@ -1662,10 +1539,8 @@ def remove(self, name):
package_py = self.recipe_filename(name)
shutil.rmtree(os.path.dirname(package_py))
def recipe_filename(self, name: str):
return os.path.join(
self.root, "packages", nm.pkg_name_to_pkg_dir(name, package_api=(2, 0)), "package.py"
)
def recipe_filename(self, name):
return os.path.join(self.root, "packages", name, "package.py")
class RepoError(spack.error.SpackError):
@@ -1715,10 +1590,7 @@ def __init__(self, name, repo=None):
# We need to compare the base package name
pkg_name = name.rsplit(".", 1)[-1]
try:
similar = difflib.get_close_matches(pkg_name, repo.all_package_names())
except Exception:
similar = []
similar = difflib.get_close_matches(pkg_name, repo.all_package_names())
if 1 <= len(similar) <= 5:
long_msg += "\n\nDid you mean one of the following packages?\n "

View File

@@ -2492,7 +2492,7 @@ def _spec_clauses(
# TODO: variant="*" means 'variant is defined to something', which used to
# be meaningless in concretization, as all variants had to be defined. But
# now that variants can be conditional, it should force a variant to exist.
if not variant.values:
if variant.value == ("*",):
continue
for value in variant.values:

View File

@@ -837,7 +837,7 @@ def _shared_subset_pair_iterate(container1, container2):
b_idx += 1
class FlagMap(lang.HashableMap[str, List[CompilerFlag]]):
class FlagMap(lang.HashableMap):
__slots__ = ("spec",)
def __init__(self, spec):
@@ -1861,7 +1861,9 @@ def add_dependency_edge(
@property
def fullname(self):
return (
f"{self.namespace}.{self.name}" if self.namespace else (self.name if self.name else "")
("%s.%s" % (self.namespace, self.name))
if self.namespace
else (self.name if self.name else "")
)
@property
@@ -4488,7 +4490,7 @@ def has_virtual_dependency(self, virtual: str) -> bool:
return bool(self.dependencies(virtuals=(virtual,)))
class VariantMap(lang.HashableMap[str, vt.VariantValue]):
class VariantMap(lang.HashableMap):
"""Map containing variant instances. New values can be added only
if the key is not already present."""

View File

@@ -93,7 +93,6 @@ def test_package_audits(packages, expected_errors, mock_packages):
]
# TODO/RepoSplit: Should this not rely on mock packages post split?
@pytest.mark.parametrize(
"config_section,data,failing_check",
[
@@ -114,7 +113,7 @@ def test_package_audits(packages, expected_errors, mock_packages):
),
],
)
def test_config_audits(config_section, data, failing_check, mock_packages):
def test_config_audits(config_section, data, failing_check):
with spack.config.override(config_section, data):
reports = spack.audit.run_group("configs")
assert any((check == failing_check) and errors for check, errors in reports)

View File

@@ -15,7 +15,7 @@
@pytest.fixture()
def builder_test_repository(config):
builder_test_path = os.path.join(spack.paths.test_repos_path, "builder.test")
builder_test_path = os.path.join(spack.paths.repos_path, "builder.test")
with spack.repo.use_repositories(builder_test_path) as mock_repo:
yield mock_repo

View File

@@ -216,13 +216,16 @@ def check_args_contents(cc, args, must_contain, must_not_contain):
assert a not in cc_modified_args
def check_wrapper_var(exe, *args, var, expected):
"""Check variables set by the compiler wrapper. This works by setting SPACK_TEST_COMMAND to
dump-var-<variable-to-debug>, which will print the variable and exit."""
executable = Executable(str(exe))
with set_env(SPACK_TEST_COMMAND=f"dump-var-{var}"):
output = executable(*args, output=str).strip()
assert f"{executable.path}: {var}: {expected}" == output
def check_env_var(executable, var, expected):
"""Check environment variables updated by the passed compiler wrapper
This assumes that cc will print debug output when it's environment
contains SPACK_TEST_COMMAND=dump-env-<variable-to-debug>
"""
executable = Executable(str(executable))
with set_env(SPACK_TEST_COMMAND="dump-env-" + var):
output = executable(*test_args, output=str).strip()
assert executable.path + ": " + var + ": " + expected == output
def dump_mode(cc, args):
@@ -746,22 +749,11 @@ def test_system_path_cleanup(wrapper_environment, wrapper_dir):
"""
cc = wrapper_dir / "cc"
system_path = "/bin:/usr/bin:/usr/local/bin"
with set_env(SPACK_COMPILER_WRAPPER_PATH=str(wrapper_dir)):
with set_env(SPACK_COMPILER_WRAPPER_PATH=str(wrapper_dir), SPACK_CC="true"):
with set_env(PATH=str(wrapper_dir) + ":" + system_path):
check_wrapper_var(cc, *test_args, var="PATH", expected=system_path)
check_env_var(cc, "PATH", system_path)
with set_env(PATH=str(wrapper_dir) + "/:" + system_path):
check_wrapper_var(cc, *test_args, var="PATH", expected=system_path)
def test_language_from_flags(wrapper_environment, wrapper_dir):
"""Test that the compiler language mode is determined by -x/--language flags if present"""
cc = wrapper_dir / "cc"
for flag_value, lang in [("c", "CC"), ("c++", "CXX"), ("f77", "F77"), ("f95", "FC")]:
check_wrapper_var(cc, "-c", "file", "-x", flag_value, var="comp", expected=lang)
check_wrapper_var(cc, "-c", "file", f"-x{flag_value}", var="comp", expected=lang)
check_wrapper_var(cc, "-c", "file", f"--language={flag_value}", var="comp", expected=lang)
check_wrapper_var(cc, "-c", "file", "--language", flag_value, var="comp", expected=lang)
check_env_var(cc, "PATH", system_path)
def test_ld_deps_partial(wrapper_environment, wrapper_dir):

View File

@@ -2032,7 +2032,7 @@ def test_ci_verify_versions_valid(
repo, _, commits = mock_git_package_changes
spack.repo.PATH.put_first(repo)
monkeypatch.setattr(spack.repo, "builtin_repo", lambda: repo)
monkeypatch.setattr(spack.repo, "packages_path", mock_packages_path(repo.packages_path))
out = ci_cmd("verify-versions", commits[-1], commits[-3])
assert "Validated diff-test@2.1.5" in out
@@ -2049,7 +2049,7 @@ def test_ci_verify_versions_standard_invalid(
repo, _, commits = mock_git_package_changes
spack.repo.PATH.put_first(repo)
monkeypatch.setattr(spack.repo, "builtin_repo", lambda: repo)
monkeypatch.setattr(spack.repo, "packages_path", mock_packages_path(repo.packages_path))
out = ci_cmd("verify-versions", commits[-1], commits[-3], fail_on_error=False)
assert "Invalid checksum found diff-test@2.1.5" in out
@@ -2060,7 +2060,7 @@ def test_ci_verify_versions_manual_package(monkeypatch, mock_packages, mock_git_
repo, _, commits = mock_git_package_changes
spack.repo.PATH.put_first(repo)
monkeypatch.setattr(spack.repo, "builtin_repo", lambda: repo)
monkeypatch.setattr(spack.repo, "packages_path", mock_packages_path(repo.packages_path))
pkg_class = spack.spec.Spec("diff-test").package_class
monkeypatch.setattr(pkg_class, "manual_download", True)

View File

@@ -20,8 +20,6 @@
config = spack.main.SpackCommand("config")
env = spack.main.SpackCommand("env")
pytestmark = pytest.mark.usefixtures("mock_packages")
def _create_config(scope=None, data={}, section="packages"):
scope = scope or spack.config.default_modify_scope()

View File

@@ -1829,7 +1829,7 @@ def test_indirect_build_dep(tmp_path):
build-only dep. Make sure this concrete DAG is preserved when writing the
environment out and reading it back.
"""
builder = spack.repo.MockRepositoryBuilder(tmp_path)
builder = spack.repo.MockRepositoryBuilder(tmp_path / "repo")
builder.add_package("z")
builder.add_package("y", dependencies=[("z", "build", None)])
builder.add_package("x", dependencies=[("y", None, None)])
@@ -1862,7 +1862,7 @@ def test_store_different_build_deps(tmp_path):
z1
"""
builder = spack.repo.MockRepositoryBuilder(tmp_path)
builder = spack.repo.MockRepositoryBuilder(tmp_path / "mirror")
builder.add_package("z")
builder.add_package("y", dependencies=[("z", "build", None)])
builder.add_package("x", dependencies=[("y", None, None), ("z", "build", None)])

View File

@@ -9,8 +9,6 @@
import spack.cmd.info
from spack.main import SpackCommand
pytestmark = [pytest.mark.usefixtures("mock_packages")]
info = SpackCommand("info")
@@ -33,12 +31,15 @@ def _print(*args, **kwargs):
return buffer
@pytest.mark.parametrize(
"pkg", ["openmpi", "trilinos", "boost", "python", "dealii", "xsdk", "gasnet", "warpx"]
)
@pytest.mark.parametrize("extra_args", [[], ["--variants-by-name"]])
def test_it_just_runs(extra_args):
info("vtk-m", *extra_args)
def test_it_just_runs(pkg, extra_args):
info(pkg, *extra_args)
def test_info_noversion(print_buffer):
def test_info_noversion(mock_packages, print_buffer):
"""Check that a mock package with no versions outputs None."""
info("noversion")
@@ -57,7 +58,7 @@ def test_info_noversion(print_buffer):
@pytest.mark.parametrize(
"pkg_query,expected", [("zlib", "False"), ("find-externals1", "True (version)")]
)
def test_is_externally_detectable(pkg_query, expected, parser, print_buffer):
def test_is_externally_detectable(mock_packages, pkg_query, expected, parser, print_buffer):
args = parser.parse_args(["--detectable", pkg_query])
spack.cmd.info.info(parser, args)
@@ -69,7 +70,13 @@ def test_is_externally_detectable(pkg_query, expected, parser, print_buffer):
@pytest.mark.parametrize(
"pkg_query", ["vtk-m", "gcc"] # This should ensure --test's c_names processing loop covered
"pkg_query",
[
"hdf5",
"cloverleaf3d",
"trilinos",
"gcc", # This should ensure --test's c_names processing loop covered
],
)
@pytest.mark.parametrize("extra_args", [[], ["--variants-by-name"]])
def test_info_fields(pkg_query, extra_args, parser, print_buffer):

View File

@@ -6,20 +6,16 @@
import sys
from textwrap import dedent
import pytest
import spack.paths
import spack.repo
from spack.main import SpackCommand
pytestmark = [pytest.mark.usefixtures("mock_packages")]
list = SpackCommand("list")
def test_list():
output = list()
assert "bzip2" in output
assert "cloverleaf3d" in output
assert "hdf5" in output
@@ -45,7 +41,7 @@ def test_list_cli_output_format(mock_tty_stdout):
assert out == out_str
def test_list_filter():
def test_list_filter(mock_packages):
output = list("py-*")
assert "py-extension1" in output
assert "py-extension2" in output
@@ -61,18 +57,18 @@ def test_list_filter():
assert "mpich" not in output
def test_list_search_description():
def test_list_search_description(mock_packages):
output = list("--search-description", "one build dependency")
assert "depb" in output
def test_list_format_name_only():
def test_list_format_name_only(mock_packages):
output = list("--format", "name_only")
assert "zmpi" in output
assert "hdf5" in output
def test_list_format_version_json():
def test_list_format_version_json(mock_packages):
output = list("--format", "version_json")
assert '{"name": "zmpi",' in output
assert '{"name": "dyninst",' in output
@@ -81,7 +77,7 @@ def test_list_format_version_json():
json.loads(output)
def test_list_format_html():
def test_list_format_html(mock_packages):
output = list("--format", "html")
assert '<div class="section" id="zmpi">' in output
assert "<h1>zmpi" in output
@@ -90,7 +86,7 @@ def test_list_format_html():
assert "<h1>hdf5" in output
def test_list_update(tmpdir):
def test_list_update(tmpdir, mock_packages):
update_file = tmpdir.join("output")
# not yet created when list is run
@@ -117,7 +113,7 @@ def test_list_update(tmpdir):
assert f.read() == "empty\n"
def test_list_tags():
def test_list_tags(mock_packages):
output = list("--tag", "tag1")
assert "mpich" in output
assert "mpich2" in output
@@ -131,7 +127,7 @@ def test_list_tags():
assert "mpich2" in output
def test_list_count():
def test_list_count(mock_packages):
output = list("--count")
assert int(output.strip()) == len(spack.repo.all_package_names())
@@ -141,10 +137,11 @@ def test_list_count():
)
# def test_list_repos(mock_packages, builder_test_repository):
def test_list_repos():
with spack.repo.use_repositories(
os.path.join(spack.paths.test_repos_path, "builtin.mock"),
os.path.join(spack.paths.test_repos_path, "builder.test"),
os.path.join(spack.paths.repos_path, "builtin.mock"),
os.path.join(spack.paths.repos_path, "builder.test"),
):
total_pkgs = len(list().strip().split())
mock_pkgs = len(list("-r", "builtin.mock").strip().split())

View File

@@ -111,12 +111,12 @@ def split(output):
pkg = spack.main.SpackCommand("pkg")
def test_builtin_repo():
assert spack.repo.builtin_repo() is spack.repo.PATH.get_repo("builtin")
def test_packages_path():
assert spack.repo.packages_path() == spack.repo.PATH.get_repo("builtin").packages_path
def test_mock_builtin_repo(mock_packages):
assert spack.repo.builtin_repo() is spack.repo.PATH.get_repo("builtin.mock")
def test_mock_packages_path(mock_packages):
assert spack.repo.packages_path() == spack.repo.PATH.get_repo("builtin.mock").packages_path
def test_pkg_add(git, mock_pkg_git_repo):

View File

@@ -2,7 +2,6 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import os
import pathlib
import pytest
@@ -23,19 +22,19 @@ def test_help_option():
assert repo.returncode in (None, 0)
def test_create_add_list_remove(mutable_config, tmp_path: pathlib.Path):
def test_create_add_list_remove(mutable_config, tmpdir):
# Create a new repository and check that the expected
# files are there
repo("create", str(tmp_path), "mockrepo")
assert (tmp_path / "spack_repo" / "mockrepo" / "repo.yaml").exists()
repo("create", str(tmpdir), "mockrepo")
assert os.path.exists(os.path.join(str(tmpdir), "repo.yaml"))
# Add the new repository and check it appears in the list output
repo("add", "--scope=site", str(tmp_path / "spack_repo" / "mockrepo"))
repo("add", "--scope=site", str(tmpdir))
output = repo("list", "--scope=site", output=str)
assert "mockrepo" in output
# Then remove it and check it's not there
repo("remove", "--scope=site", str(tmp_path / "spack_repo" / "mockrepo"))
repo("remove", "--scope=site", str(tmpdir))
output = repo("list", "--scope=site", output=str)
assert "mockrepo" not in output

View File

@@ -116,7 +116,7 @@ def test_changed_no_base(git, tmpdir, capfd):
assert "This repository does not have a 'foobar'" in err
def test_changed_files_all_files(mock_packages):
def test_changed_files_all_files():
# it's hard to guarantee "all files", so do some sanity checks.
files = set(
[
@@ -159,12 +159,8 @@ def test_bad_root(tmpdir):
def test_style_is_package(tmpdir):
"""Ensure the is_package() function works."""
assert spack.cmd.style.is_package(
"var/spack/repos/spack_repo/builtin/packages/hdf5/package.py"
)
assert spack.cmd.style.is_package(
"var/spack/repos/spack_repo/builtin/packages/zlib/package.py"
)
assert spack.cmd.style.is_package("var/spack/repos/builtin/packages/hdf5/package.py")
assert spack.cmd.style.is_package("var/spack/repos/builtin/packages/zlib/package.py")
assert not spack.cmd.style.is_package("lib/spack/spack/spec.py")
assert not spack.cmd.style.is_package("lib/spack/external/pytest.py")

View File

@@ -10,9 +10,6 @@
versions = SpackCommand("versions")
pytestmark = [pytest.mark.usefixtures("mock_packages")]
def test_safe_versions():
"""Only test the safe versions of a package."""
@@ -73,11 +70,11 @@ def test_no_unchecksummed_versions():
def test_versions_no_url():
"""Test a package with versions but without a ``url`` attribute."""
versions("attributes-foo-app")
versions("graphviz")
@pytest.mark.maybeslow
def test_no_versions_no_url():
"""Test a package without versions or a ``url`` attribute."""
versions("no-url-or-version")
versions("opengl")

View File

@@ -29,7 +29,7 @@ def _concretize_with_reuse(*, root_str, reused_str):
@pytest.fixture
def runtime_repo(mutable_config):
repo = os.path.join(spack.paths.test_repos_path, "compiler_runtime.test")
repo = os.path.join(spack.paths.repos_path, "compiler_runtime.test")
with spack.repo.use_repositories(repo) as mock_repo:
yield mock_repo

View File

@@ -1719,7 +1719,7 @@ def test_reuse_with_unknown_namespace_dont_raise(
@pytest.mark.regression("45538")
def test_reuse_from_other_namespace_no_raise(self, tmpdir, temporary_store, monkeypatch):
myrepo = spack.repo.MockRepositoryBuilder(tmpdir, namespace="mock_repo")
myrepo = spack.repo.MockRepositoryBuilder(tmpdir.mkdir("mock.repo"), namespace="myrepo")
myrepo.add_package("zlib")
builtin = spack.concretize.concretize_one("zlib")
@@ -1727,19 +1727,21 @@ def test_reuse_from_other_namespace_no_raise(self, tmpdir, temporary_store, monk
with spack.repo.use_repositories(myrepo.root, override=False):
with spack.config.override("concretizer:reuse", True):
myrepo = spack.concretize.concretize_one("mock_repo.zlib")
myrepo = spack.concretize.concretize_one("myrepo.zlib")
assert myrepo.namespace == "mock_repo"
assert myrepo.namespace == "myrepo"
@pytest.mark.regression("28259")
def test_reuse_with_unknown_package_dont_raise(self, tmpdir, temporary_store, monkeypatch):
builder = spack.repo.MockRepositoryBuilder(str(tmpdir), namespace="myrepo")
builder = spack.repo.MockRepositoryBuilder(tmpdir.mkdir("mock.repo"), namespace="myrepo")
builder.add_package("pkg-c")
with spack.repo.use_repositories(builder.root, override=False):
s = spack.concretize.concretize_one("pkg-c")
assert s.namespace == "myrepo"
PackageInstaller([s.package], fake=True, explicit=True).install()
del sys.modules["spack_repo.myrepo.packages.pkg_c"]
del sys.modules["spack.pkg.myrepo.pkg-c"]
del sys.modules["spack.pkg.myrepo"]
builder.remove("pkg-c")
with spack.repo.use_repositories(builder.root, override=False) as repos:
# TODO (INJECT CONFIGURATION): unclear why the cache needs to be invalidated explicitly
@@ -2332,7 +2334,7 @@ def test_select_lower_priority_package_from_repository_stack(
from cli.
"""
# 'builtin.mock" and "duplicates.test" share a 'gmake' package
additional_repo = os.path.join(spack.paths.test_repos_path, "duplicates.test")
additional_repo = os.path.join(spack.paths.repos_path, "duplicates.test")
with spack.repo.use_repositories(additional_repo, override=False):
s = spack.concretize.concretize_one(spec_str)
@@ -2576,7 +2578,7 @@ def test_correct_external_is_selected_from_packages_yaml(self, mutable_config):
@pytest.fixture()
def duplicates_test_repository():
repository_path = os.path.join(spack.paths.test_repos_path, "duplicates.test")
repository_path = os.path.join(spack.paths.repos_path, "duplicates.test")
with spack.repo.use_repositories(repository_path) as mock_repo:
yield mock_repo
@@ -2811,7 +2813,7 @@ def test_adding_specs(self, input_specs, default_mock_concretization):
@pytest.fixture()
def edges_test_repository():
repository_path = os.path.join(spack.paths.test_repos_path, "edges.test")
repository_path = os.path.join(spack.paths.repos_path, "edges.test")
with spack.repo.use_repositories(repository_path) as mock_repo:
yield mock_repo
@@ -3105,9 +3107,7 @@ def test_spec_unification(unify, mutable_config, mock_packages):
_ = spack.cmd.parse_specs([a_restricted, b], concretize=True)
def test_concretization_cache_roundtrip(
mock_packages, use_concretization_cache, monkeypatch, mutable_config
):
def test_concretization_cache_roundtrip(use_concretization_cache, monkeypatch, mutable_config):
"""Tests whether we can write the results of a clingo solve to the cache
and load the same spec request from the cache to produce identical specs"""
# Force determinism:

View File

@@ -46,7 +46,7 @@
@pytest.fixture
def test_repo(mutable_config, monkeypatch, mock_stage):
repo_dir = pathlib.Path(spack.paths.test_repos_path) / "flags.test"
repo_dir = pathlib.Path(spack.paths.repos_path) / "flags.test"
with spack.repo.use_repositories(str(repo_dir)) as mock_repo_path:
yield mock_repo_path

View File

@@ -28,7 +28,7 @@ def update_packages_config(conf_str):
@pytest.fixture
def test_repo(mutable_config, monkeypatch, mock_stage):
repo_dir = pathlib.Path(spack.paths.test_repos_path) / "requirements.test"
repo_dir = pathlib.Path(spack.paths.repos_path) / "requirements.test"
with spack.repo.use_repositories(str(repo_dir)) as mock_repo_path:
yield mock_repo_path

View File

@@ -914,7 +914,7 @@ def test_single_file_scope(config, env_yaml):
assert spack.config.get("config:checksum") is True
assert spack.config.get("config:checksum") is True
assert spack.config.get("packages:externalmodule:buildable") is False
assert spack.config.get("repos") == ["/x/y/z", "$spack/var/spack/repos/spack_repo/builtin"]
assert spack.config.get("repos") == ["/x/y/z", "$spack/var/spack/repos/builtin"]
def test_single_file_scope_section_override(tmpdir, config):
@@ -950,7 +950,7 @@ def test_single_file_scope_section_override(tmpdir, config):
# from the lower config scopes
assert spack.config.get("config:checksum") is True
assert not spack.config.get("packages:externalmodule")
assert spack.config.get("repos") == ["/x/y/z", "$spack/var/spack/repos/spack_repo/builtin"]
assert spack.config.get("repos") == ["/x/y/z", "$spack/var/spack/repos/builtin"]
def test_write_empty_single_file_scope(tmpdir):

View File

@@ -259,9 +259,9 @@ def mock_git_package_changes(git, tmpdir, override_git_repos_cache_path, monkeyp
Important attributes of the repo for test coverage are: multiple package
versions are added with some coming from a tarball and some from git refs.
"""
filename = "diff_test/package.py"
filename = "diff-test/package.py"
repo_path, _ = spack.repo.create_repo(str(tmpdir), namespace="myrepo")
repo_path, _ = spack.repo.create_repo(str(tmpdir.mkdir("myrepo")))
repo_cache = spack.util.file_cache.FileCache(str(tmpdir.mkdir("cache")))
repo = spack.repo.Repo(repo_path, cache=repo_cache)

View File

@@ -1,2 +1,2 @@
repos:
- $spack/var/spack/repos/spack_repo/builtin
- $spack/var/spack/repos/builtin

View File

@@ -246,29 +246,22 @@ class BadDetectablePackage(spack.package.Package):
def test_package_url_and_urls():
UrlsPackage = type(
"URLsPackage",
(spack.package.Package,),
{
"__module__": "spack.pkg.builtin.urls_package",
"url": "https://www.example.com/url-package-1.0.tgz",
"urls": ["https://www.example.com/archive"],
},
)
class URLsPackage(spack.package.Package):
url = "https://www.example.com/url-package-1.0.tgz"
urls = ["https://www.example.com/archive"]
s = spack.spec.Spec("urls-package")
s = spack.spec.Spec("pkg-a")
with pytest.raises(ValueError, match="defines both"):
UrlsPackage(s)
URLsPackage(s)
def test_package_license():
LicensedPackage = type(
"LicensedPackage",
(spack.package.Package,),
{"__module__": "spack.pkg.builtin.licensed_package"},
)
class LicensedPackage(spack.package.Package):
extendees = None # currently a required attribute for is_extension()
license_files = None
pkg = LicensedPackage(spack.spec.Spec("licensed-package"))
s = spack.spec.Spec("pkg-a")
pkg = LicensedPackage(s)
assert pkg.global_license_file is None
pkg.license_files = ["license.txt"]

View File

@@ -15,7 +15,7 @@
import spack.repo
from spack.paths import mock_packages_path
from spack.spec import Spec
from spack.util.naming import pkg_name_to_class_name
from spack.util.naming import mod_to_class
from spack.version import VersionChecksumError
@@ -47,15 +47,11 @@ def test_nonexisting_package_filename(self):
)
def test_package_class_names(self):
assert "Mpich" == pkg_name_to_class_name("mpich")
assert "PmgrCollective" == pkg_name_to_class_name("pmgr_collective")
assert "PmgrCollective" == pkg_name_to_class_name("pmgr-collective")
assert "Pmgrcollective" == pkg_name_to_class_name("PmgrCollective")
assert "_3db" == pkg_name_to_class_name("3db")
assert "_True" == pkg_name_to_class_name("true") # reserved keyword
assert "_False" == pkg_name_to_class_name("false") # reserved keyword
assert "_None" == pkg_name_to_class_name("none") # reserved keyword
assert "Finally" == pkg_name_to_class_name("finally") # `Finally` is not reserved
assert "Mpich" == mod_to_class("mpich")
assert "PmgrCollective" == mod_to_class("pmgr_collective")
assert "PmgrCollective" == mod_to_class("pmgr-collective")
assert "Pmgrcollective" == mod_to_class("PmgrCollective")
assert "_3db" == mod_to_class("3db")
# Below tests target direct imports of spack packages from the
# spack.pkg namespace
@@ -337,16 +333,3 @@ def test_package_can_have_sparse_checkout_properties(mock_packages, mock_fetch,
assert isinstance(fetcher, spack.fetch_strategy.GitFetchStrategy)
assert hasattr(fetcher, "git_sparse_paths")
assert fetcher.git_sparse_paths == pkg_cls.git_sparse_paths
def test_pkg_name_can_only_be_derived_when_package_module():
"""When the module prefix is not spack_repo (or legacy spack.pkg) we cannot derive
a package name."""
ExamplePackage = type(
"ExamplePackage",
(spack.package_base.PackageBase,),
{"__module__": "not.a.spack.repo.packages.example_package.package"},
)
with pytest.raises(ValueError, match="Package ExamplePackage is not a known Spack package"):
ExamplePackage.name

View File

@@ -6,14 +6,11 @@
import pytest
import spack
import spack.package_base
import spack.paths
import spack.repo
import spack.spec
import spack.util.file_cache
import spack.util.naming
from spack.util.naming import valid_module_name
@pytest.fixture(params=["packages", "", "foo"])
@@ -187,12 +184,12 @@ def _repo_paths(repos):
repo_paths.append(spack.paths.mock_packages_path)
namespaces.append("builtin.mock")
if entry == "extra":
name = "extra_mock"
name = "extra.mock"
repo_dir = tmp_path / name
repo_dir.mkdir()
repo = spack.repo.MockRepositoryBuilder(repo_dir, name)
repo_paths.append(repo.root)
namespaces.append(repo.namespace)
_ = spack.repo.MockRepositoryBuilder(repo_dir, name)
repo_paths.append(str(repo_dir))
namespaces.append(name)
return repo_paths, namespaces
repo_paths, namespaces = _repo_paths(repos)
@@ -219,7 +216,7 @@ def test_use_repositories_and_import():
"""Tests that use_repositories changes the import search too"""
import spack.paths
repo_dir = pathlib.Path(spack.paths.test_repos_path)
repo_dir = pathlib.Path(spack.paths.repos_path)
with spack.repo.use_repositories(str(repo_dir / "compiler_runtime.test")):
import spack.pkg.compiler_runtime.test.gcc_runtime
@@ -313,7 +310,7 @@ def test_creation_from_string(self, mock_test_cache):
repo = spack.repo.RepoPath(spack.paths.mock_packages_path, cache=mock_test_cache)
assert len(repo.repos) == 1
assert repo.repos[0]._finder is repo
assert repo.by_namespace["builtin.mock"] is repo.repos[0]
assert repo.by_namespace["spack.pkg.builtin.mock"] is repo.repos[0]
def test_get_repo(self, mock_test_cache):
repo = spack.repo.RepoPath(spack.paths.mock_packages_path, cache=mock_test_cache)
@@ -367,160 +364,3 @@ def test_repo_package_api_version(tmp_path: pathlib.Path):
)
cache = spack.util.file_cache.FileCache(tmp_path / "cache")
assert spack.repo.Repo(str(tmp_path / "example"), cache=cache).package_api == (1, 0)
def test_mod_to_pkg_name_and_reverse():
# In repo v1 the dirname/module name is the package name
assert spack.util.naming.pkg_dir_to_pkg_name("zlib_ng", package_api=(1, 0)) == "zlib_ng"
assert (
spack.util.naming.pkg_dir_to_pkg_name("_3example_4", package_api=(1, 0)) == "_3example_4"
)
assert spack.util.naming.pkg_name_to_pkg_dir("zlib_ng", package_api=(1, 0)) == "zlib_ng"
assert (
spack.util.naming.pkg_name_to_pkg_dir("_3example_4", package_api=(1, 0)) == "_3example_4"
)
# In repo v2 there is a 1-1 mapping between module and package names
assert spack.util.naming.pkg_dir_to_pkg_name("_3example_4", package_api=(2, 0)) == "3example-4"
assert spack.util.naming.pkg_dir_to_pkg_name("zlib_ng", package_api=(2, 0)) == "zlib-ng"
assert spack.util.naming.pkg_name_to_pkg_dir("zlib-ng", package_api=(2, 0)) == "zlib_ng"
assert spack.util.naming.pkg_name_to_pkg_dir("3example-4", package_api=(2, 0)) == "_3example_4"
# reserved names need an underscore
assert spack.util.naming.pkg_dir_to_pkg_name("_finally", package_api=(2, 0)) == "finally"
assert spack.util.naming.pkg_dir_to_pkg_name("_assert", package_api=(2, 0)) == "assert"
assert spack.util.naming.pkg_name_to_pkg_dir("finally", package_api=(2, 0)) == "_finally"
assert spack.util.naming.pkg_name_to_pkg_dir("assert", package_api=(2, 0)) == "_assert"
# reserved names are case sensitive, so true/false/none are ok
assert spack.util.naming.pkg_dir_to_pkg_name("true", package_api=(2, 0)) == "true"
assert spack.util.naming.pkg_dir_to_pkg_name("none", package_api=(2, 0)) == "none"
assert spack.util.naming.pkg_name_to_pkg_dir("true", package_api=(2, 0)) == "true"
assert spack.util.naming.pkg_name_to_pkg_dir("none", package_api=(2, 0)) == "none"
def test_repo_v2_invalid_module_name(tmp_path: pathlib.Path, capsys):
# Create a repo with a v2 structure
root, _ = spack.repo.create_repo(str(tmp_path), namespace="repo_1", package_api=(2, 0))
repo_dir = pathlib.Path(root)
# Create two invalid module names
(repo_dir / "packages" / "zlib-ng").mkdir()
(repo_dir / "packages" / "zlib-ng" / "package.py").write_text(
"""
from spack.package import Package
class ZlibNg(Package):
pass
"""
)
(repo_dir / "packages" / "UPPERCASE").mkdir()
(repo_dir / "packages" / "UPPERCASE" / "package.py").write_text(
"""
from spack.package import Package
class Uppercase(Package):
pass
"""
)
with spack.repo.use_repositories(str(repo_dir)) as repo:
assert len(repo.all_package_names()) == 0
stderr = capsys.readouterr().err
assert "cannot be used because `zlib-ng` is not a valid Spack package module name" in stderr
assert "cannot be used because `UPPERCASE` is not a valid Spack package module name" in stderr
def test_repo_v2_module_and_class_to_package_name(tmp_path: pathlib.Path, capsys):
# Create a repo with a v2 structure
root, _ = spack.repo.create_repo(str(tmp_path), namespace="repo_2", package_api=(2, 0))
repo_dir = pathlib.Path(root)
# Create an invalid module name
(repo_dir / "packages" / "_1example_2_test").mkdir()
(repo_dir / "packages" / "_1example_2_test" / "package.py").write_text(
"""
from spack.package import Package
class _1example2Test(Package):
pass
"""
)
with spack.repo.use_repositories(str(repo_dir)) as repo:
assert repo.exists("1example-2-test")
pkg_cls = repo.get_pkg_class("1example-2-test")
assert pkg_cls.name == "1example-2-test"
assert pkg_cls.module.__name__ == "spack_repo.repo_2.packages._1example_2_test.package"
def test_valid_module_name_v2():
api = (2, 0)
# no hyphens
assert not valid_module_name("zlib-ng", api)
# cannot start with a number
assert not valid_module_name("7zip", api)
# no consecutive underscores
assert not valid_module_name("zlib__ng", api)
# reserved names
assert not valid_module_name("finally", api)
assert not valid_module_name("assert", api)
# cannot contain uppercase
assert not valid_module_name("False", api)
assert not valid_module_name("zlib_NG", api)
# reserved names are allowed when preceded by underscore
assert valid_module_name("_finally", api)
assert valid_module_name("_assert", api)
# digits are allowed when preceded by underscore
assert valid_module_name("_1example_2_test", api)
# underscore is not allowed unless followed by reserved name or digit
assert not valid_module_name("_zlib", api)
assert not valid_module_name("_false", api)
def test_namespace_is_optional_in_v2(tmp_path: pathlib.Path):
"""Test that a repo without a namespace is valid in v2."""
repo_yaml_dir = tmp_path / "spack_repo" / "foo" / "bar" / "baz"
(repo_yaml_dir / "packages").mkdir(parents=True)
(repo_yaml_dir / "repo.yaml").write_text(
"""\
repo:
api: v2.0
"""
)
cache = spack.util.file_cache.FileCache(tmp_path / "cache")
repo = spack.repo.Repo(str(repo_yaml_dir), cache=cache)
assert repo.namespace == "foo.bar.baz"
assert repo.full_namespace == "spack_repo.foo.bar.baz.packages"
assert repo.root == str(repo_yaml_dir)
assert repo.packages_path == str(repo_yaml_dir / "packages")
assert repo.python_path == str(tmp_path)
assert repo.package_api == (2, 0)
def test_subdir_in_v2():
"""subdir cannot be . or empty in v2, because otherwise we cannot statically distinguish
between namespace and subdir."""
with pytest.raises(spack.repo.BadRepoError, match="Use a symlink packages -> . instead"):
spack.repo._validate_and_normalize_subdir(subdir="", root="root", package_api=(2, 0))
with pytest.raises(spack.repo.BadRepoError, match="Use a symlink packages -> . instead"):
spack.repo._validate_and_normalize_subdir(subdir=".", root="root", package_api=(2, 0))
with pytest.raises(spack.repo.BadRepoError, match="Expected a directory name, not a path"):
subdir = os.path.join("a", "b")
spack.repo._validate_and_normalize_subdir(subdir=subdir, root="root", package_api=(2, 0))
with pytest.raises(spack.repo.BadRepoError, match="Must be a valid Python module name"):
spack.repo._validate_and_normalize_subdir(subdir="123", root="root", package_api=(2, 0))

View File

@@ -6,107 +6,72 @@
import itertools
import re
import string
from typing import List, Tuple
import spack.error
__all__ = [
"pkg_name_to_class_name",
"mod_to_class",
"spack_module_to_python_module",
"valid_module_name",
"valid_fully_qualified_module_name",
"validate_fully_qualified_module_name",
"validate_module_name",
"possible_spack_module_names",
"simplify_name",
"NamespaceTrie",
]
#: see keyword.kwlist: https://github.com/python/cpython/blob/main/Lib/keyword.py
RESERVED_NAMES_ONLY_LOWERCASE = frozenset(
(
"and",
"as",
"assert",
"async",
"await",
"break",
"class",
"continue",
"def",
"del",
"elif",
"else",
"except",
"finally",
"for",
"from",
"global",
"if",
"import",
"in",
"is",
"lambda",
"nonlocal",
"not",
"or",
"pass",
"raise",
"return",
"try",
"while",
"with",
"yield",
)
)
RESERVED_NAMES_LIST_MIXED_CASE = ("False", "None", "True")
# Valid module names can contain '-' but can't start with it.
_valid_module_re = r"^\w[\w-]*$"
# Valid module names can contain '-' but can't start with it.
_VALID_MODULE_RE_V1 = re.compile(r"^\w[\w-]*$")
_VALID_MODULE_RE_V2 = re.compile(r"^[a-z_][a-z0-9_]*$")
_valid_fully_qualified_module_re = r"^(\w[\w-]*)(\.\w[\w-]*)*$"
def pkg_name_to_class_name(pkg_name: str):
"""Convert a Spack package name to a class name, based on
`PEP-8 <http://legacy.python.org/dev/peps/pep-0008/>`_:
def mod_to_class(mod_name):
"""Convert a name from module style to class name style. Spack mostly
follows `PEP-8 <http://legacy.python.org/dev/peps/pep-0008/>`_:
* Module and package names use lowercase_with_underscores.
* Class names use the CapWords convention.
Not all package names are valid Python identifiers:
Regular source code follows these convetions. Spack is a bit
more liberal with its Package names and Compiler names:
* They can contain '-', but cannot start with '-'.
* They can contain '-' as well as '_', but cannot start with '-'.
* They can start with numbers, e.g. "3proxy".
This function converts from the package name to the class convention by removing _ and - and
converting surrounding lowercase text to CapWords. If package name starts with a number, the
class name returned will be prepended with '_' to make a valid Python identifier.
This function converts from the module convention to the class
convention by removing _ and - and converting surrounding
lowercase text to CapWords. If mod_name starts with a number,
the class name returned will be prepended with '_' to make a
valid Python identifier.
"""
class_name = re.sub(r"[-_]+", "-", pkg_name)
validate_module_name(mod_name)
class_name = re.sub(r"[-_]+", "-", mod_name)
class_name = string.capwords(class_name, "-")
class_name = class_name.replace("-", "")
# Ensure that the class name is a valid Python identifier
if re.match(r"^[0-9]", class_name) or class_name in RESERVED_NAMES_LIST_MIXED_CASE:
class_name = f"_{class_name}"
# If a class starts with a number, prefix it with Number_ to make it
# a valid Python class name.
if re.match(r"^[0-9]", class_name):
class_name = "_%s" % class_name
return class_name
def pkg_dir_to_pkg_name(dirname: str, package_api: Tuple[int, int]) -> str:
"""Translate a package dir (pkg_dir/package.py) to its corresponding package name"""
if package_api < (2, 0):
return dirname
return dirname.lstrip("_").replace("_", "-")
def spack_module_to_python_module(mod_name):
"""Given a Spack module name, returns the name by which it can be
imported in Python.
"""
if re.match(r"[0-9]", mod_name):
mod_name = "num" + mod_name
return mod_name.replace("-", "_")
def pkg_name_to_pkg_dir(name: str, package_api: Tuple[int, int]) -> str:
"""Translate a package name to its corresponding package dir (pkg_dir/package.py)"""
if package_api < (2, 0):
return name
name = name.replace("-", "_")
if re.match(r"^[0-9]", name) or name in RESERVED_NAMES_ONLY_LOWERCASE:
name = f"_{name}"
return name
def possible_spack_module_names(python_mod_name: str) -> List[str]:
def possible_spack_module_names(python_mod_name):
"""Given a Python module name, return a list of all possible spack module
names that could correspond to it."""
mod_name = re.sub(r"^num(\d)", r"\1", python_mod_name)
@@ -114,7 +79,7 @@ def possible_spack_module_names(python_mod_name: str) -> List[str]:
parts = re.split(r"(_)", mod_name)
options = [["_", "-"]] * mod_name.count("_")
results: List[str] = []
results = []
for subs in itertools.product(*options):
s = list(parts)
s[1::2] = subs
@@ -123,7 +88,7 @@ def possible_spack_module_names(python_mod_name: str) -> List[str]:
return results
def simplify_name(name: str) -> str:
def simplify_name(name):
"""Simplify package name to only lowercase, digits, and dashes.
Simplifies a name which may include uppercase letters, periods,
@@ -171,17 +136,42 @@ def simplify_name(name: str) -> str:
return name
def valid_module_name(mod_name: str, package_api: Tuple[int, int]) -> bool:
def valid_module_name(mod_name):
"""Return whether mod_name is valid for use in Spack."""
if package_api < (2, 0):
return bool(_VALID_MODULE_RE_V1.match(mod_name))
elif not _VALID_MODULE_RE_V2.match(mod_name) or "__" in mod_name:
return False
elif mod_name.startswith("_"):
# it can only start with an underscore if followed by digit or reserved name
return mod_name[1:] in RESERVED_NAMES_ONLY_LOWERCASE or mod_name[1].isdigit()
else:
return mod_name not in RESERVED_NAMES_ONLY_LOWERCASE
return bool(re.match(_valid_module_re, mod_name))
def valid_fully_qualified_module_name(mod_name):
"""Return whether mod_name is a valid namespaced module name."""
return bool(re.match(_valid_fully_qualified_module_re, mod_name))
def validate_module_name(mod_name):
"""Raise an exception if mod_name is not valid."""
if not valid_module_name(mod_name):
raise InvalidModuleNameError(mod_name)
def validate_fully_qualified_module_name(mod_name):
"""Raise an exception if mod_name is not a valid namespaced module name."""
if not valid_fully_qualified_module_name(mod_name):
raise InvalidFullyQualifiedModuleNameError(mod_name)
class InvalidModuleNameError(spack.error.SpackError):
"""Raised when we encounter a bad module name."""
def __init__(self, name):
super().__init__("Invalid module name: " + name)
self.name = name
class InvalidFullyQualifiedModuleNameError(spack.error.SpackError):
"""Raised when we encounter a bad full package name."""
def __init__(self, name):
super().__init__("Invalid fully qualified package name: " + name)
self.name = name
class NamespaceTrie:

View File

@@ -99,12 +99,12 @@ archspec = ["archspec"]
llnl = ["llnl"]
[tool.ruff.lint.per-file-ignores]
"var/spack/*/package.py" = ["F403", "F405", "F811", "F821"]
"var/spack/repos/*/package.py" = ["F403", "F405", "F811", "F821"]
"*-ci-package.py" = ["F403", "F405", "F821"]
[tool.black]
line-length = 99
include = '(lib/spack|var/spack/repos|var/spack/test_repos)/.*\.pyi?$|bin/spack$'
include = '(lib/spack|var/spack/repos)/.*\.pyi?$|bin/spack$'
extend-exclude = 'lib/spack/external'
skip_magic_trailing_comma = true
@@ -126,16 +126,8 @@ src_paths = "lib"
honor_noqa = true
[tool.mypy]
files = [
"lib/spack/llnl/**/*.py",
"lib/spack/spack/**/*.py",
"var/spack/repos/spack_repo/builtin/packages/*/package.py"
]
mypy_path = [
"lib/spack",
"lib/spack/external",
"var/spack/repos",
]
files = ['lib/spack/llnl/**/*.py', 'lib/spack/spack/**/*.py', './var/spack/repos/builtin/packages/*/package.py']
mypy_path = ['bin', 'lib/spack', 'lib/spack/external', 'var/spack/repos/builtin']
allow_redefinition = true
# This and a generated import file allows supporting packages
@@ -152,7 +144,7 @@ ignore_missing_imports = true
ignore_missing_imports = false
[[tool.mypy.overrides]]
module = 'spack_repo.*'
module = 'packages.*'
ignore_errors = false
ignore_missing_imports = false
# we can't do this here, not a module scope option, in spack style instead
@@ -205,7 +197,7 @@ ignore_missing_imports = true
useLibraryCodeForTypes = true
reportMissingImports = true
reportWildcardImportFromLibrary = false
include = ['lib/spack', 'var/spack/repos', 'var/spack/test_repos']
include = ['lib/spack', 'var/spack/repos']
ignore = ['lib/spack/external']
extraPaths = ['lib/spack', 'lib/spack/external']

View File

@@ -5,6 +5,7 @@ ci:
- Write-Output "Done"
before_script::
- git config core.autocrlf true
- fsutil 8dot3name set C:\ 0
- . .\share\spack\setup-env.ps1
- If (Test-Path -path C:\\key\intermediate_ci_signing_key.gpg) { spack.ps1 gpg trust C:\\key\intermediate_ci_signing_key.gpg }

View File

@@ -173,11 +173,11 @@ spack:
- vtk-m ~openmp # +openmp: https://github.com/spack/spack/issues/31830
- zfp
# --
# - paraview +qt # concretize: paraview: Qt support requires GLX on non Windows; llvm-17.0.6: https://github.com/spack/spack/issues/49625
# - py-cinemasci # py-maturin-1.8.3: rust-lld: error: undefined symbol: _intel_fast_memcpy
# - visit # vtk-9.2.6: ??
# - paraview +qt # llvm-17.0.6: https://github.com/spack/spack/issues/49625
# - py-cinemasci # llvm-14.0.6: https://github.com/spack/spack/issues/49625
# - visit # llvm-17.0.6: https://github.com/spack/spack/issues/49625
# --
# - chapel ~cuda ~rocm # chapel-2.4.0: KeyError: 'intel-oneapi-compilers': /builds/spack/spack/var/spack/repos/builtin/packages/chapel/package.py:602, in setup_chpl_compilers: env.set("CHPL_HOST_COMPILER", self.compiler_map[self.spec.compiler.name])
# - chapel ~cuda ~rocm # llvm-19.1.7: https://github.com/spack/spack/issues/49625
# - cp2k +mpi # dbcsr-2.8.0: FAILED: src/CMakeFiles/dbcsr.dir/dbcsr_api.F-pp.f src/CMakeFiles/dbcsr.dir/dbcsr_api.F.o.ddi:
# - dealii # taskflow@3.7.0: cmake: Taskflow currently supports the following compilers: g++ v7.0 or above, clang++ v6.0 or above
# - exago +mpi ~ipopt +hiop ~python +raja ^hiop+raja~sparse # raja-0.14.0: RAJA/pattern/kernel/Tile.hpp:174:30: error: no member named 'block_id' in 'IterableTiler<Iterable>'
@@ -185,8 +185,8 @@ spack:
# - fftx # fftx-1.2.0: https://github.com/spack/spack/issues/49621
# - fpm # fpm-0.10.0: /tmp/ifx1305151083OkWTRB/ifxqBG60i.i90: error #6405: The same named entity from different modules and/or program units cannot be referenced. [TOML_TABLE]; fpm.F90(32048): error #7002: Error in opening the compiled module file. Check INCLUDE paths. [FPM_MANIFEST_PREPROCESS]
# - geopm-runtime # concretize: c-blosc2: conflicts with '%oneapi';
- glvis # llvm-17.0.6: https://github.com/spack/spack/issues/49625
# - gptune ~mpispawn # py-maturin-1.8.3: rust-lld: error: undefined symbol: __intel_cpu_feature_indicator_x
# - glvis # llvm-17.0.6: https://github.com/spack/spack/issues/49625
# - gptune ~mpispawn # llvm-14.0.6: https://github.com/spack/spack/issues/49625
# - lbann # lbann-0.104: https://github.com/spack/spack/issues/49619
# - libpressio +bitgrooming +bzip2 ~cuda ~cusz +fpzip +hdf5 +libdistributed +lua +openmp +python +sz +sz3 +unix +zfp # concretize: c-blosc2: conflicts with '%oneapi';
# - nek5000 +mpi ~visit # nek5000-19.0: RuntimeError: Cannot build example: short_tests/eddy.
@@ -209,8 +209,8 @@ spack:
# --
# - py-jupyterlab # py-maturin: rust-lld: error: undefined symbol: _intel_fast_memcpy
# - py-notebook # py-maturin: rust-lld: error: undefined symbol: _intel_fast_memcpy
- py-numba # llvm-14.0.6: https://github.com/spack/spack/issues/49625
- py-pandas # llvm-14.0.6: https://github.com/spack/spack/issues/49625
# - py-numba # llvm-14.0.6: https://github.com/spack/spack/issues/49625
# - py-pandas # llvm-14.0.6: https://github.com/spack/spack/issues/49625
# - py-plotly # py-maturin: rust-lld: error: undefined symbol: _intel_fast_memcpy
- aml +level_zero

View File

@@ -40,9 +40,7 @@ spack -p --lines 20 spec mpileaks%gcc
$coverage_run $(which spack) bootstrap status --dev --optional
# Check that we can import Spack packages directly as a first import
# TODO: this check is disabled, because sys.path is only updated once
# spack.repo.PATH.get_pkg_class is called.
# $coverage_run $(which spack) python -c "import spack.pkg.builtin.mpileaks; repr(spack.pkg.builtin.mpileaks.Mpileaks)"
$coverage_run $(which spack) python -c "import spack.pkg.builtin.mpileaks; repr(spack.pkg.builtin.mpileaks.Mpileaks)"
#-----------------------------------------------------------
# Run unit tests with code coverage

Some files were not shown because too many files have changed in this diff Show More