zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove support for generating dotkit files (#11986)

Browse Source 
Dotkit is being used only at a few sites and has been deprecated on new
machines. This commit removes all the code that provide support for the
generation of dotkit module files.

A new validator named "deprecatedProperties" has been added to the
jsonschema validators. It permits to prompt a warning message or exit
with an error if a property that has been marked as deprecated is
encountered.

* Removed references to dotkit in the docs
* Removed references to dotkit in setup-env-test.sh
* Added a unit test for the 'deprecatedProperties' schema validator
This commit is contained in:
Massimiliano Culpo
2019-10-03 07:15:01 +02:00
committed by Todd Gamblin
parent b11984767b
commit 76b9c56110
34 changed files with 121 additions and 426 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
lib/spack/docs/basic_usage.rst
View File

@@ -1190,24 +1190,15 @@ Using Extensions
^^^^^^^^^^^^^^^^
There are three ways to get ``numpy`` working in Python. The first is
to use :ref:`shell-support`. You can simply ``use`` or ``load`` the
to use :ref:`shell-support`. You can simply ``load`` the
module for the extension, and it will be added to the ``PYTHONPATH``
in your current shell.
For tcl modules:
in your current shell:
.. code-block:: console
$ spack load python
$ spack load py-numpy
or, for dotkit:
.. code-block:: console
$ spack use python
$ spack use py-numpy
Now ``import numpy`` will succeed for as long as you keep your current
session open.

1
lib/spack/docs/config_yaml.rst
View File

@@ -74,7 +74,6 @@ the location for each type of module. e.g.:
module_roots:
tcl: $spack/share/spack/modules
lmod: $spack/share/spack/lmod
dotkit: $spack/share/spack/dotkit
See :ref:`modules` for details.

2
lib/spack/docs/configuration.rst
View File

@@ -463,7 +463,6 @@ account all scopes. For example, to see the fully merged
module_roots:
tcl: $spack/share/spack/modules
lmod: $spack/share/spack/lmod
dotkit: $spack/share/spack/dotkit
build_stage:
- $tempdir/$user/spack-stage
- ~/.spack/stage
@@ -514,7 +513,6 @@ down the problem:
/home/myuser/spack/etc/spack/defaults/config.yaml:32 module_roots:
/home/myuser/spack/etc/spack/defaults/config.yaml:33 tcl: $spack/share/spack/modules
/home/myuser/spack/etc/spack/defaults/config.yaml:34 lmod: $spack/share/spack/lmod
/home/myuser/spack/etc/spack/defaults/config.yaml:35 dotkit: $spack/share/spack/dotkit
/home/myuser/spack/etc/spack/defaults/config.yaml:49 build_stage:
/home/myuser/spack/etc/spack/defaults/config.yaml:50 - $tempdir/$user/spack-stage
/home/myuser/spack/etc/spack/defaults/config.yaml:51 - ~/.spack/stage

2
lib/spack/docs/getting_started.rst
View File

@@ -71,7 +71,7 @@ This automatically adds Spack to your ``PATH`` and allows the ``spack``
command to be used to execute spack :ref:`commands <shell-support>` and
:ref:`useful packaging commands <packaging-shell-support>`.
If :ref:`environment-modules or dotkit <InstallEnvironmentModules>` is
If :ref:`environment-modules <InstallEnvironmentModules>` is
installed and available, the ``spack`` command can also load and unload
:ref:`modules <modules>`.

50
lib/spack/docs/module_file_support.rst
View File

@@ -13,8 +13,8 @@ The use of module systems to manage user environment in a controlled way
is a common practice at HPC centers that is often embraced also by individual
programmers on their development machines. To support this common practice
Spack integrates with `Environment Modules
<http://modules.sourceforge.net/>`_ , `LMod
<http://lmod.readthedocs.io/en/latest/>`_ and `Dotkit <https://computing.llnl.gov/?set=jobs&page=dotkit>`_ by
<http://modules.sourceforge.net/>`_ and `LMod
<http://lmod.readthedocs.io/en/latest/>`_ by
providing post-install hooks that generate module files and commands to manipulate them.
.. note::
@@ -67,7 +67,7 @@ to load the ``cmake`` module:
$ module load cmake-3.7.2-gcc-6.3.0-fowuuby
Neither of these is particularly pretty, easy to remember, or
easy to type. Luckily, Spack has its own interface for using modules and dotkits.
easy to type. Luckily, Spack has its own interface for using modules.
^^^^^^^^^^^^^
Shell support
@@ -108,20 +108,10 @@ that the startup time may be slightly increased because of that.
^^^^^^^^^^^^^^^^^^^^^^^
Once you have shell support enabled you can use the same spec syntax
you're used to:
you're used to and you can use the same shortened names you use
everywhere else in Spack.
========================= ==========================
Modules Dotkit
========================= ==========================
``spack load <spec>`` ``spack use <spec>``
``spack unload <spec>`` ``spack unuse <spec>``
========================= ==========================
And you can use the same shortened names you use everywhere else in
Spack.
For example, if you are using dotkit, this will add the ``mpich``
package built with ``gcc`` to your path:
For example this will add the ``mpich`` package built with ``gcc`` to your path:
.. code-block:: console
@@ -129,17 +119,10 @@ package built with ``gcc`` to your path:
# ... wait for install ...
$ spack use mpich %gcc@4.4.7 # dotkit
Prepending: mpich@3.0.4%gcc@4.4.7 (ok)
$ spack load mpich %gcc@4.4.7 # modules
$ which mpicc
~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpich@3.0.4/bin/mpicc
Or, similarly if you are using modules, you could type:
.. code-block:: console
$ spack load mpich %gcc@4.4.7 # modules
These commands will add appropriate directories to your ``PATH``,
``MANPATH``, ``CPATH``, and ``LD_LIBRARY_PATH``. When you no longer
want to use a package, you can type unload or unuse similarly:
@@ -147,13 +130,12 @@ want to use a package, you can type unload or unuse similarly:
.. code-block:: console
$ spack unload mpich %gcc@4.4.7 # modules
$ spack unuse mpich %gcc@4.4.7 # dotkit
.. note::
These ``use``, ``unuse``, ``load``, and ``unload`` subcommands are
The ``load`` and ``unload`` subcommands are
only available if you have enabled Spack's shell support *and* you
have dotkit or modules installed on your machine.
have environment-modules installed on your machine.
^^^^^^^^^^^^^^^^^^^^^^
Ambiguous module names
@@ -292,8 +274,6 @@ that can be generated by Spack:
+-----------------------------+--------------------+-------------------------------+----------------------------------------------+----------------------+
| | **Hook name** | **Default root directory** | **Default template file** | **Compatible tools** |
+=============================+====================+===============================+==============================================+======================+
| **Dotkit** | ``dotkit`` | share/spack/dotkit | share/spack/templates/modules/modulefile.dk | DotKit |
+-----------------------------+--------------------+-------------------------------+----------------------------------------------+----------------------+
| **TCL - Non-Hierarchical** | ``tcl`` | share/spack/modules | share/spack/templates/modules/modulefile.tcl | Env. Modules/LMod |
+-----------------------------+--------------------+-------------------------------+----------------------------------------------+----------------------+
| **Lua - Hierarchical** | ``lmod`` | share/spack/lmod | share/spack/templates/modules/modulefile.lua | LMod |
@@ -377,7 +357,7 @@ are named ``modules.yaml``. The default configuration:
.. literalinclude:: _spack_root/etc/spack/defaults/modules.yaml
:language: yaml
activates the hooks to generate ``tcl`` and ``dotkit`` module files and inspects
activates the hooks to generate ``tcl`` module files and inspects
the installation folder of each package for the presence of a set of subdirectories
(``bin``, ``man``, ``share/man``, etc.). If any is found its full path is prepended
to the environment variables listed below the folder name.
@@ -399,12 +379,9 @@ to the generator being customized:
modules:
enable:
- tcl
- dotkit
- lmod
tcl:
# contains environment modules specific customizations
dotkit:
# contains dotkit specific customizations
lmod:
# contains lmod specific customizations
@@ -590,15 +567,14 @@ do so by using the environment blacklist:
.. code-block:: yaml
modules:
dotkit:
tcl:
all:
filter:
# Exclude changes to any of these variables
environment_blacklist: ['CPATH', 'LIBRARY_PATH']
The configuration above will generate dotkit module files that will not contain
modifications to either ``CPATH`` or ``LIBRARY_PATH`` and environment module
files that instead will contain these modifications.
The configuration above will generate module files that will not contain
modifications to either ``CPATH`` or ``LIBRARY_PATH``.
.. _autoloading-dependencies:

2
lib/spack/spack/cmd/module.py
View File

@@ -7,7 +7,6 @@
import llnl.util.tty as tty
import spack.cmd.modules.dotkit
import spack.cmd.modules.lmod
import spack.cmd.modules.tcl
@@ -23,7 +22,6 @@
def setup_parser(subparser):
sp = subparser.add_subparsers(metavar='SUBCOMMAND', dest='module_command')
spack.cmd.modules.dotkit.add_command(sp, _subcommands)
spack.cmd.modules.lmod.add_command(sp, _subcommands)
spack.cmd.modules.tcl.add_command(sp, _subcommands)

1
lib/spack/spack/cmd/modules/__init__.py
View File

@@ -137,7 +137,6 @@ def loads(module_type, specs, args, out=sys.stdout):
module_commands = {
'tcl': 'module load ',
'lmod': 'module load ',
'dotkit': 'use '
}
d = {

19
lib/spack/spack/cmd/modules/dotkit.py
View File

@@ -1,19 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import functools
import spack.cmd.modules
def add_command(parser, command_dict):
dotkit_parser = parser.add_parser(
'dotkit', help='manipulate dotkit module files'
)
spack.cmd.modules.setup_parser(dotkit_parser)
command_dict['dotkit'] = functools.partial(
spack.cmd.modules.modules_cmd, module_type='dotkit'
)

23
lib/spack/spack/cmd/unuse.py
View File

@@ -1,23 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import argparse
from spack.cmd.common import print_module_placeholder_help
description = "remove package from environment using dotkit"
section = "modules"
level = "long"
def setup_parser(subparser):
"""Parser is only constructed so that this prints a nice help
message with -h. """
subparser.add_argument(
'spec', nargs=argparse.REMAINDER,
help='spec of package to unuse with dotkit')
def unuse(parser, args):
print_module_placeholder_help()

24
lib/spack/spack/cmd/use.py
View File

@@ -1,24 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import argparse
from spack.cmd.common import print_module_placeholder_help, arguments
description = "add package to environment using dotkit"
section = "modules"
level = "long"
def setup_parser(subparser):
"""Parser is only constructed so that this prints a nice help
message with -h. """
subparser.add_argument(
'spec', nargs=argparse.REMAINDER,
help='spec of package to use with dotkit')
arguments.add_common_arguments(subparser, ['recurse_dependencies'])
def use(parser, args):
print_module_placeholder_help()

2
lib/spack/spack/hooks/__init__.py
View File

@@ -19,7 +19,7 @@
* post_uninstall(spec)
This can be used to implement support for things like module
systems (e.g. modules, dotkit, etc.) or to add other custom
systems (e.g. modules, lmod, etc.) or to add other custom
features.
"""
import os.path

1
lib/spack/spack/main.py
View File

@@ -591,7 +591,6 @@ def shell_set(var, value):
module_roots = spack.config.get('config:module_roots')
module_to_roots = {
'tcl': list(),
'dotkit': list(),
'lmod': list()
}
for name, path in module_roots.items():

6
lib/spack/spack/modules/__init__.py
View File

@@ -4,24 +4,20 @@
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""This package contains code for creating environment modules, which can
include dotkits, TCL non-hierarchical modules, LUA hierarchical modules, and
others.
include TCL non-hierarchical modules, LUA hierarchical modules, and others.
"""
from __future__ import absolute_import
from .dotkit import DotkitModulefileWriter
from .tcl import TclModulefileWriter
from .lmod import LmodModulefileWriter
__all__ = [
'DotkitModulefileWriter',
'TclModulefileWriter',
'LmodModulefileWriter'
]
module_types = {
'dotkit': DotkitModulefileWriter,
'tcl': TclModulefileWriter,
'lmod': LmodModulefileWriter
}

60
lib/spack/spack/modules/dotkit.py
View File

@@ -1,60 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""This module implements the classes necessary to generate dotkit modules."""
import os.path
import spack.config
from .common import BaseConfiguration, BaseFileLayout
from .common import BaseContext, BaseModuleFileWriter
#: Dotkit specific part of the configuration
configuration = spack.config.get('modules:dotkit', {})
#: Caches the configuration {spec_hash: configuration}
configuration_registry = {}
def make_configuration(spec):
"""Returns the dotkit configuration for spec"""
key = spec.dag_hash()
try:
return configuration_registry[key]
except KeyError:
return configuration_registry.setdefault(
key, DotkitConfiguration(spec)
)
def make_layout(spec):
"""Returns the layout information for spec """
conf = make_configuration(spec)
return DotkitFileLayout(conf)
def make_context(spec):
"""Returns the context information for spec"""
conf = make_configuration(spec)
return DotkitContext(conf)
class DotkitConfiguration(BaseConfiguration):
"""Configuration class for dotkit module files."""
class DotkitFileLayout(BaseFileLayout):
"""File layout for dotkit module files."""
#: file extension of dotkit module files
extension = 'dk'
class DotkitContext(BaseContext):
"""Context class for dotkit module files."""
class DotkitModulefileWriter(BaseModuleFileWriter):
"""Writer class for dotkit module files."""
default_template = os.path.join('modules', 'modulefile.dk')

27
lib/spack/spack/schema/__init__.py
View File

@@ -11,6 +11,7 @@
import six
import llnl.util.lang
import llnl.util.tty
import spack.spec
@@ -72,11 +73,35 @@ def _validate_spec(validator, is_spec, instance, schema):
'"{0}" is an invalid spec [{1}]'.format(spec_str, str(e))
)
def _deprecated_properties(validator, deprecated, instance, schema):
if not (validator.is_type(instance, "object") or
validator.is_type(instance, "array")):
return
# Get a list of the deprecated properties, return if there is none
deprecated_properties = [
x for x in instance if x in deprecated['properties']
]
if not deprecated_properties:
return
# Retrieve the template message
msg = deprecated['message']
is_error = deprecated['error']
if not is_error:
for entry in deprecated_properties:
llnl.util.tty.warn(msg.format(property=entry))
else:
import jsonschema
for entry in deprecated_properties:
yield jsonschema.ValidationError(msg.format(property=entry))
return jsonschema.validators.extend(
jsonschema.Draft4Validator, {
"validate_spec": _validate_spec,
"properties": _set_defaults,
"patternProperties": _set_pp_defaults
"patternProperties": _set_pp_defaults,
"deprecatedProperties": _deprecated_properties
}
)

7
lib/spack/spack/schema/config.py
View File

@@ -41,6 +41,13 @@
'lmod': {'type': 'string'},
'dotkit': {'type': 'string'},
},
'deprecatedProperties': {
'properties': ['dotkit'],
'message': 'specifying a "{property}" module root has no '
'effect [support for {property} module files'
' has been dropped]',
'error': False
},
},
'source_cache': {'type': 'string'},
'misc_cache': {'type': 'string'},

18
lib/spack/spack/schema/modules.py
View File

@@ -135,7 +135,16 @@
'default': [],
'items': {
'type': 'string',
'enum': ['tcl', 'dotkit', 'lmod']}},
'enum': ['tcl', 'dotkit', 'lmod']
},
'deprecatedProperties': {
'properties': ['dotkit'],
'message': 'cannot enable "{property}" in modules.yaml '
'[support for {property} module files has been'
' dropped]',
'error': False
},
},
'lmod': {
'allOf': [
# Base configuration
@@ -164,6 +173,13 @@
]
},
},
'deprecatedProperties': {
'properties': ['dotkit'],
'message': 'the section "{property}" in modules.yaml has no effect'
' [support for {property} module files has been '
'dropped]',
'error': False
},
},
}

3
lib/spack/spack/schema/upstreams.py
View File

@@ -20,8 +20,7 @@
'type': 'object',
'properties': {
'tcl': {'type': 'string'},
'lmod': {'type': 'string'},
'dotkit': {'type': 'string'}
'lmod': {'type': 'string'}
}
}
}

3
lib/spack/spack/test/cmd/module.py
View File

@@ -21,7 +21,6 @@ def _module_files(module_type, *specs):
@pytest.fixture(scope='module', autouse=True)
def ensure_module_files_are_there(database):
module('dotkit', 'refresh', '-y')
module('tcl', 'refresh', '-y')
@@ -39,7 +38,7 @@ def failure_args(request):
@pytest.fixture(
params=['dotkit', 'tcl', 'lmod']
params=['tcl', 'lmod']
)
def module_type(request):
return request.param

1
lib/spack/spack/test/data/config/modules.yaml
View File

@@ -16,7 +16,6 @@
modules:
enable:
- tcl
- dotkit
prefix_inspections:
bin:
- PATH

5
lib/spack/spack/test/data/modules/dotkit/autoload_direct.yaml
View File

@@ -1,5 +0,0 @@
enable:
- dotkit
dotkit:
all:
autoload: 'direct'

5
lib/spack/spack/test/data/modules/dotkit/override_template.yaml
View File

@@ -1,5 +0,0 @@
enable:
- dotkit
dotkit:
all:
template: 'override_from_modules.txt'

51
lib/spack/spack/test/modules/dotkit.py
View File

@@ -1,51 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import pytest
import spack.modules.dotkit
#: Class of the writer tested in this module
writer_cls = spack.modules.dotkit.DotkitModulefileWriter
@pytest.mark.usefixtures('config', 'mock_packages')
class TestDotkit(object):
def test_dotkit(self, modulefile_content, module_configuration):
"""Tests the generation of a dotkit file that loads dependencies
automatically.
"""
module_configuration('autoload_direct')
content = modulefile_content('mpileaks arch=x86-linux')
assert '#c spack' in content
assert '#d mpileaks @2.3' in content
assert len([x for x in content if 'dk_op' in x]) == 2
def test_override_template_in_package(
self, modulefile_content, module_configuration
):
"""Tests overriding a template from and attribute in the package."""
module_configuration('autoload_direct')
content = modulefile_content('override-module-templates')
assert 'Override successful!' in content
def test_override_template_in_modules_yaml(
self, modulefile_content, module_configuration
):
"""Tests overriding a template from `modules.yaml`"""
module_configuration('override_template')
# Check that this takes precedence over an attribute in the package
content = modulefile_content('override-module-templates')
assert 'Override even better!' in content
content = modulefile_content('mpileaks arch=x86-linux')
assert 'Override even better!' in content

25
lib/spack/spack/test/schema.py
View File

@@ -114,3 +114,28 @@ def test_schema_validation(meta_schema, config_name):
# If this validation throws the test won't pass
jsonschema.validate(schema, meta_schema)
def test_deprecated_properties(module_suffixes_schema):
# Test that an error is reported when 'error: True'
module_suffixes_schema['deprecatedProperties'] = {
'properties': ['tcl'],
'message': '{property} not allowed',
'error': True
}
v = spack.schema.Validator(module_suffixes_schema)
data = {'tcl': {'all': {'suffixes': {'^python': 'py'}}}}
with pytest.raises(jsonschema.ValidationError, match='tcl not allowed'):
v.validate(data)
# Test that just a warning is reported when 'error: False'
module_suffixes_schema['deprecatedProperties'] = {
'properties': ['tcl'],
'message': '{property} not allowed',
'error': False
}
v = spack.schema.Validator(module_suffixes_schema)
data = {'tcl': {'all': {'suffixes': {'^python': 'py'}}}}
# The next validation doesn't raise anymore
v.validate(data)