zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Allow combinatorial projections in views (#9679)

Browse Source 
Allow customizing views with Spec-formatted directory structure

Allow views to specify projections that are more complicated than
merging every package into a single shared prefix. This will allow
sites to configure a view for the way they want to present packages
to their users; for example this can be used to create a prefix for
each package but omit the DAG hash from the path.

This includes a new YAML format file for specifying the simplified
prefix for a spec in a view. This configuration allows the use of
different prefix formats for different specs (i.e. specs depending
on MPI can include the MPI implementation in the prefix).
Documentation on usage of the view projection configuration is
included.

Depending on the projection configuration, paths are not guaranteed
to be unique and it may not be possible to add multiple installs of
a package to a view.
This commit is contained in:
Greg Becker 2019-01-09 17:39:35 -08:00 committed by Peter Scheibel
parent f5bb93c75b
commit 450b0e3059
19 changed files with 619 additions and 151 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

68
lib/spack/docs/workflows.rst
View File

@ -437,11 +437,23 @@ Filesystem views offer an alternative to environment modules, another
way to assemble packages in a useful way and load them into a user's
environment.
A filesystem view is a single directory tree that is the union of the
directory hierarchies of a number of installed packages; it is similar
to the directory hiearchy that might exist under ``/usr/local``. The
files of the view's installed packages are brought into the view by
symbolic or hard links, referencing the original Spack installation.
A single-prefix filesystem view is a single directory tree that is the
union of the directory hierarchies of a number of installed packages;
it is similar to the directory hiearchy that might exist under
``/usr/local``. The files of the view's installed packages are
brought into the view by symbolic or hard links, referencing the
original Spack installation.
A combinatorial filesystem view can contain more software than a
single-prefix view. Combinatorial filesystem views are created by
defining a projection for each spec or set of specs. The syntax for
this will be discussed in the section for the ``spack view`` command
under `adding_projections_to_views`_.
The projection for a spec or set of specs specifies the naming scheme
for the directory structure under the root of the view into which the
package will be linked. For example, the spec ``zlib@1.2.8%gcc@4.4.7``
could be projected to ``MYVIEW/zlib-1.2.8-gcc``.
When software is built and installed, absolute paths are frequently
"baked into" the software, making it non-relocatable. This happens
@ -507,6 +519,51 @@ files in the ``cmake`` package while retaining its dependencies.
When packages are removed from a view, empty directories are
purged.
.. _adding_projections_to_views:
""""""""""""""""""""""""""""
Controlling View Projections
""""""""""""""""""""""""""""
The default projection into a view is to link every package into the
root of the view. This can be changed by adding a ``projections.yaml``
configuration file to the view. The projection configuration file for
a view located at ``/my/view`` is stored in
``/my/view/.spack/projections.yaml``.
When creating a view, the projection configuration file can also be
specified from the command line using the ``--projection-file`` option
to the ``spack view`` command.
The projections configuration file is a mapping of partial specs to
spec format strings, as shown in the example below.
.. code-block:: yaml
projections:
zlib: ${PACKAGE}-${VERSION}
^mpi: ${PACKAGE}-${VERSION}/${DEP:mpi:PACKAGE}-${DEP:mpi:VERSION}-${COMPILERNAME}-${COMPILERVER}
all: ${PACKAGE}-${VERSION}/${COMPILERNAME}-${COMPILERVER}
The entries in the projections configuration file must all be either
specs or the keyword ``all``. For each spec, the projection used will
be the first non-``all`` entry that the spec satisfies, or ``all`` if
there is an entry for ``all`` and no other entry is satisfied by the
spec. Where the keyword ``all`` appears in the file does not
matter. Given the example above, any spec satisfying ``zlib@1.2.8``
will be linked into ``/my/view/zlib-1.2.8/``, any spec satisfying
``hdf5@1.8.10+mpi %gcc@4.9.3 ^mvapich2@2.2`` will be linked into
``/my/view/hdf5-1.8.10/mvapich2-2.2-gcc-4.9.3``, and any spec
satisfying ``hdf5@1.8.10~mpi %gcc@4.9.3`` will be linked into
``/my/view/hdf5-1.8.10/gcc-4.9.3``.
If the keyword ``all`` does not appear in the projections
configuration file, any spec that does not satisfy any entry in the
file will be linked into the root of the view as in a single-prefix
view. Any entries that appear below the keyword ``all`` in the
projections configuration file will not be used, as all specs will use
the projection under ``all`` before reaching those entries.
""""""""""""""""""
Fine-Grain Control
""""""""""""""""""
@ -1437,4 +1494,3 @@ Disadvantages:
2. Although patches of a few lines work OK, large patch files can be
hard to create and maintain.

2
lib/spack/spack/build_systems/aspell_dict.py
View File

@ -26,7 +26,7 @@ class AspellDictPackage(AutotoolsPackage):
def view_destination(self, view):
aspell_spec = self.spec['aspell']
if view.root != aspell_spec.prefix:
if view.get_projection_for_spec(aspell_spec) != aspell_spec.prefix:
raise ExtensionError(
'aspell does not support non-global extensions')
aspell = aspell_spec.command

14
lib/spack/spack/build_systems/python.py
View File

@ -414,7 +414,9 @@ def view_file_conflicts(self, view, merge_map):
def add_files_to_view(self, view, merge_map):
bin_dir = self.spec.prefix.bin
python_prefix = self.extendee_spec.prefix
global_view = same_path(python_prefix, view.root)
global_view = same_path(python_prefix, view.get_projection_for_spec(
self.spec
))
for src, dst in merge_map.items():
if os.path.exists(dst):
continue
@ -424,7 +426,9 @@ def add_files_to_view(self, view, merge_map):
shutil.copy2(src, dst)
if 'script' in get_filetype(src):
filter_file(
python_prefix, os.path.abspath(view.root), dst)
python_prefix, os.path.abspath(
view.get_projection_for_spec(self.spec)), dst
)
else:
orig_link_target = os.path.realpath(src)
new_link_target = os.path.abspath(merge_map[orig_link_target])
@ -443,7 +447,11 @@ def remove_files_from_view(self, view, merge_map):
ignore_namespace = True
bin_dir = self.spec.prefix.bin
global_view = self.extendee_spec.prefix == view.root
global_view = (
self.extendee_spec.prefix == view.get_projection_for_spec(
self.spec
)
)
for src, dst in merge_map.items():
if ignore_namespace and namespace_init(dst):
continue

13
lib/spack/spack/cmd/__init__.py
View File

@ -8,6 +8,7 @@
import os
import re
import sys
import argparse
import llnl.util.tty as tty
from llnl.util.lang import attr_setdefault, index_by
@ -342,3 +343,15 @@ def spack_is_git_repo():
"""Ensure that this instance of Spack is a git clone."""
with working_dir(spack.paths.prefix):
return os.path.isdir('.git')
########################################
# argparse types for argument validation
########################################
def extant_file(f):
"""
Argparse type for files that exist.
"""
if not os.path.isfile(f):
raise argparse.ArgumentTypeError('%s does not exist' % f)
return f

58
lib/spack/spack/cmd/view.py
View File

@ -37,13 +37,17 @@
import llnl.util.tty as tty
from llnl.util.link_tree import MergeConflictError
from llnl.util.tty.color import colorize
import spack.environment as ev
import spack.cmd
import spack.store
import spack.schema.projections
from spack.config import validate
from spack.filesystem_view import YamlFilesystemView
from spack.util import spack_yaml as s_yaml
description = "produce a single-rooted directory view of packages"
description = "project packages to a compact naming scheme on the filesystem."
section = "environments"
level = "short"
@ -52,28 +56,28 @@
actions_status = ["statlink", "status", "check"]
def relaxed_disambiguate(specs, view):
def disambiguate_in_view(specs, view):
"""
When dealing with querying actions (remove/status) the name of the spec
is sufficient even though more versions of that name might be in the
database.
When dealing with querying actions (remove/status) we only need to
disambiguate among specs in the view
"""
name_to_spec = dict((s.name, s) for s in view.get_all_specs())
view_specs = set(view.get_all_specs())
def squash(matching_specs):
if not matching_specs:
tty.die("Spec matches no installed packages.")
elif len(matching_specs) == 1:
return matching_specs[0]
matching_in_view = [ms for ms in matching_specs if ms in view_specs]
elif matching_specs[0].name in name_to_spec:
return name_to_spec[matching_specs[0].name]
if len(matching_in_view) > 1:
args = ["Spec matches multiple packages.",
"Matching packages:"]
args += [colorize(" @K{%s} " % s.dag_hash(7)) +
s.cformat('$_$@$%@$=') for s in matching_in_view]
args += ["Use a more specific spec."]
tty.die(*args)
else:
# we just return the first matching spec, the error about the
# missing spec will be printed later on
return matching_specs[0]
return matching_in_view[0] if matching_in_view else matching_specs[0]
# make function always return a list to keep consistency between py2/3
return list(map(squash, map(spack.store.db.query, specs)))
@ -120,6 +124,13 @@ def setup_parser(sp):
act.add_argument('path', nargs=1,
help="path to file system view directory")
if cmd in ("symlink", "hardlink"):
# invalid for remove/statlink, for those commands the view needs to
# already know its own projections.
help_msg = "Initialize view using projections from file."
act.add_argument('--projection-file', dest='projection_file',
type=spack.cmd.extant_file, help=help_msg)
if cmd == "remove":
grp = act.add_mutually_exclusive_group(required=True)
act.add_argument(
@ -158,11 +169,20 @@ def view(parser, args):
specs = spack.cmd.parse_specs(args.specs)
path = args.path[0]
if args.action in actions_link and args.projection_file:
# argparse confirms file exists
with open(args.projection_file, 'r') as f:
projections_data = s_yaml.load(f)
validate(projections_data, spack.schema.projections.schema)
ordered_projections = projections_data['projections']
else:
ordered_projections = {}
view = YamlFilesystemView(
path, spack.store.layout,
projections=ordered_projections,
ignore_conflicts=getattr(args, "ignore_conflicts", False),
link=os.link if args.action in ["hardlink", "hard"]
else os.symlink,
link=os.link if args.action in ["hardlink", "hard"] else os.symlink,
verbose=args.verbose)
# Process common args and specs
@ -181,11 +201,11 @@ def view(parser, args):
if len(specs) == 0:
specs = view.get_all_specs()
else:
specs = relaxed_disambiguate(specs, view)
specs = disambiguate_in_view(specs, view)
else:
# status and remove can map the name to packages in view
specs = relaxed_disambiguate(specs, view)
# status and remove can map a partial spec to packages in view
specs = disambiguate_in_view(specs, view)
with_dependencies = args.dependencies.lower() in ['true', 'yes']

14
lib/spack/spack/config.py
View File

@ -142,12 +142,12 @@ def get_section(self, section):
def write_section(self, section):
filename = self.get_section_filename(section)
data = self.get_section(section)
_validate(data, section_schemas[section])
validate(data, section_schemas[section])
try:
mkdirp(self.path)
with open(filename, 'w') as f:
_validate(data, section_schemas[section])
validate(data, section_schemas[section])
syaml.dump(data, stream=f, default_flow_style=False)
except (yaml.YAMLError, IOError) as e:
raise ConfigFileError(
@ -233,7 +233,7 @@ def get_section(self, section):
return self.sections[section]
def write_section(self, section):
_validate(self.sections, self.schema)
validate(self.sections, self.schema)
try:
parent = os.path.dirname(self.path)
mkdirp(parent)
@ -277,7 +277,7 @@ def __init__(self, name, data=None):
if data:
for section in data:
dsec = data[section]
_validate({section: dsec}, section_schemas[section])
validate({section: dsec}, section_schemas[section])
self.sections[section] = _mark_internal(
syaml.syaml_dict({section: dsec}), name)
@ -295,7 +295,7 @@ def write_section(self, section):
"""This only validates, as the data is already in memory."""
data = self.get_section(section)
if data is not None:
_validate(data, section_schemas[section])
validate(data, section_schemas[section])
self.sections[section] = _mark_internal(data, self.name)
def __repr__(self):
@ -646,7 +646,7 @@ def _validate_section_name(section):
% (section, " ".join(section_schemas.keys())))
def _validate(data, schema, set_defaults=True):
def validate(data, schema, set_defaults=True):
"""Validate data read in from a Spack YAML file.
Arguments:
@ -683,7 +683,7 @@ def _read_config_file(filename, schema):
data = _mark_overrides(syaml.load(f))
if data:
_validate(data, schema)
validate(data, schema)
return data
except MarkedYAMLError as e:

19
lib/spack/spack/directory_layout.py
View File

@ -108,8 +108,8 @@ class ExtensionsLayout(object):
directly in the installation folder - or extensions activated in
filesystem views.
"""
def __init__(self, root, **kwargs):
self.root = root
def __init__(self, view, **kwargs):
self.view = view
def add_extension(self, spec, ext_spec):
"""Add to the list of currently installed extensions."""
@ -309,11 +309,11 @@ def specs_by_hash(self):
class YamlViewExtensionsLayout(ExtensionsLayout):
"""Maintain extensions within a view.
"""
def __init__(self, root, layout):
def __init__(self, view, layout):
"""layout is the corresponding YamlDirectoryLayout object for which
we implement extensions.
"""
super(YamlViewExtensionsLayout, self).__init__(root)
super(YamlViewExtensionsLayout, self).__init__(view)
self.layout = layout
self.extension_file_name = 'extensions.yaml'
@ -354,15 +354,18 @@ def extension_file_path(self, spec):
_check_concrete(spec)
normalize_path = lambda p: (
os.path.abspath(p).rstrip(os.path.sep))
if normalize_path(spec.prefix) == normalize_path(self.root):
# For backwards compatibility, when the root is the extended
view_prefix = self.view.get_projection_for_spec(spec)
if normalize_path(spec.prefix) == normalize_path(view_prefix):
# For backwards compatibility, when the view is the extended
# package's installation directory, do not include the spec name
# as a subdirectory.
components = [self.root, self.layout.metadata_dir,
components = [view_prefix, self.layout.metadata_dir,
self.extension_file_name]
else:
components = [self.root, self.layout.metadata_dir, spec.name,
components = [view_prefix, self.layout.metadata_dir, spec.name,
self.extension_file_name]
return os.path.join(*components)
def extension_map(self, spec):

159
lib/spack/spack/filesystem_view.py
View File

@ -12,13 +12,21 @@
from llnl.util.link_tree import LinkTree, MergeConflictError
from llnl.util import tty
from llnl.util.lang import match_predicate
from llnl.util.lang import match_predicate, index_by
from llnl.util.tty.color import colorize
from llnl.util.filesystem import mkdirp
import spack.util.spack_yaml as s_yaml
import spack.spec
import spack.store
import spack.schema.projections
import spack.config
from spack.error import SpackError
from spack.directory_layout import ExtensionAlreadyInstalledError
from spack.directory_layout import YamlViewExtensionsLayout
# compatability
if sys.version_info < (3, 0):
from itertools import imap as map
@ -28,6 +36,9 @@
__all__ = ["FilesystemView", "YamlFilesystemView"]
_projections_path = '.spack/projections.yaml'
class FilesystemView(object):
"""
Governs a filesystem view that is located at certain root-directory.
@ -48,9 +59,11 @@ def __init__(self, root, layout, **kwargs):
Files are linked by method `link` (os.symlink by default).
"""
self.root = root
self._root = root
self.layout = layout
self.projections = kwargs.get('projections', {})
self.ignore_conflicts = kwargs.get("ignore_conflicts", False)
self.link = kwargs.get("link", os.symlink)
self.verbose = kwargs.get("verbose", False)
@ -125,6 +138,12 @@ def remove_standalone(self, spec):
"""
raise NotImplementedError
def get_projection_for_spec(self, spec):
"""
Get the projection in this view for a spec.
"""
raise NotImplementedError
def get_all_specs(self):
"""
Get all specs currently active in this view.
@ -166,9 +185,37 @@ class YamlFilesystemView(FilesystemView):
def __init__(self, root, layout, **kwargs):
super(YamlFilesystemView, self).__init__(root, layout, **kwargs)
self.extensions_layout = YamlViewExtensionsLayout(root, layout)
# Super class gets projections from the kwargs
# YAML specific to get projections from YAML file
projections_path = os.path.join(self._root, _projections_path)
if not self.projections:
if os.path.exists(projections_path):
# Read projections file from view
with open(projections_path, 'r') as f:
projections_data = s_yaml.load(f)
spack.config.validate(projections_data,
spack.schema.projections.schema)
self.projections = projections_data['projections']
else:
# Write projections file to new view
# Not strictly necessary as the empty file is the empty
# projection but it makes sense for consistency
mkdirp(os.path.dirname(projections_path))
with open(projections_path, 'w') as f:
f.write(s_yaml.dump({'projections': self.projections}))
elif not os.path.exists(projections_path):
# Write projections file to new view
mkdirp(os.path.dirname(projections_path))
with open(projections_path, 'w') as f:
f.write(s_yaml.dump({'projections': self.projections}))
else:
msg = 'View at %s has projections file' % self._root
msg += ' and was passed projections manually.'
raise ConflictingProjectionsError(msg)
self._croot = colorize_root(self.root) + " "
self.extensions_layout = YamlViewExtensionsLayout(self, layout)
self._croot = colorize_root(self._root) + " "
def add_specs(self, *specs, **kwargs):
assert all((s.concrete for s in specs))
@ -371,8 +418,6 @@ def remove_extension(self, spec, with_dependents=True):
'Skipping package not linked in view: %s' % spec.name)
return
# The spec might have been deactivated as depdency of another package
# already
if spec.package.is_activated(self):
spec.package.do_deactivate(
self,
@ -395,13 +440,45 @@ def remove_standalone(self, spec):
if self.verbose:
tty.info(self._croot + 'Removed package: %s' % colorize_spec(spec))
def get_projection_for_spec(self, spec):
"""
Return the projection for a spec in this view.
Relies on the ordering of projections to avoid ambiguity.
"""
spec = spack.spec.Spec(spec)
# Extensions are placed by their extendee, not by their own spec
locator_spec = spec
if spec.package.extendee_spec:
locator_spec = spec.package.extendee_spec
all_fmt_str = None
for spec_like, fmt_str in self.projections.items():
if locator_spec.satisfies(spec_like, strict=True):
return os.path.join(self._root, locator_spec.format(fmt_str))
elif spec_like == 'all':
all_fmt_str = fmt_str
if all_fmt_str:
return os.path.join(self._root, locator_spec.format(all_fmt_str))
return self._root
def get_all_specs(self):
dotspack = os.path.join(self.root,
spack.store.layout.metadata_dir)
if os.path.exists(dotspack):
return list(filter(None, map(self.get_spec, os.listdir(dotspack))))
else:
return []
md_dirs = []
for root, dirs, files in os.walk(self._root):
if spack.store.layout.metadata_dir in dirs:
md_dirs.append(os.path.join(root,
spack.store.layout.metadata_dir))
specs = []
for md_dir in md_dirs:
if os.path.exists(md_dir):
for name_dir in os.listdir(md_dir):
filename = os.path.join(md_dir, name_dir,
spack.store.layout.spec_file_name)
spec = get_spec_from_file(filename)
if spec:
specs.append(spec)
return specs
def get_conflicts(self, *specs):
"""
@ -414,7 +491,7 @@ def get_conflicts(self, *specs):
def get_path_meta_folder(self, spec):
"Get path to meta folder for either spec or spec name."
return os.path.join(self.root,
return os.path.join(self.get_projection_for_spec(spec),
spack.store.layout.metadata_dir,
getattr(spec, "name", spec))
@ -423,11 +500,7 @@ def get_spec(self, spec):
filename = os.path.join(dotspack,
spack.store.layout.spec_file_name)
try:
with open(filename, "r") as f:
return spack.spec.Spec.from_yaml(f)
except IOError:
return None
return get_spec_from_file(filename)
def link_meta_folder(self, spec):
src = spack.store.layout.metadata_path(spec)
@ -466,10 +539,39 @@ def print_status(self, *specs, **kwargs):
if len(specs) > 0:
tty.msg("Packages linked in %s:" % self._croot[:-1])
# avoid circular dependency
import spack.cmd
spack.cmd.display_specs(in_view, flags=True, variants=True,
long=self.verbose)
# Make a dict with specs keyed by architecture and compiler.
index = index_by(specs, ('architecture', 'compiler'))
# Traverse the index and print out each package
for i, (architecture, compiler) in enumerate(sorted(index)):
if i > 0:
print()
header = "%s{%s} / %s{%s}" % (spack.spec.architecture_color,
architecture,
spack.spec.compiler_color,
compiler)
tty.hline(colorize(header), char='-')
specs = index[(architecture, compiler)]
specs.sort()
format_string = '$_$@$%@+$+'
abbreviated = [s.cformat(format_string) for s in specs]
# Print one spec per line along with prefix path
width = max(len(s) for s in abbreviated)
width += 2
format = " %%-%ds%%s" % width
for abbrv, s in zip(abbreviated, specs):
prefix = ''
if self.verbose:
prefix = colorize('@K{%s}' % s.dag_hash(7))
print(
prefix + (format % (abbrv,
self.get_projection_for_spec(s)))
)
else:
tty.warn(self._croot + "No packages found.")
@ -478,7 +580,7 @@ def purge_empty_directories(self):
Ascend up from the leaves accessible from `path`
and remove empty directories.
"""
for dirpath, subdirs, files in os.walk(self.root, topdown=False):
for dirpath, subdirs, files in os.walk(self._root, topdown=False):
for sd in subdirs:
sdp = os.path.join(dirpath, sd)
try:
@ -508,6 +610,13 @@ def _check_no_ext_conflicts(self, spec):
#####################
# utility functions #
#####################
def get_spec_from_file(filename):
try:
with open(filename, "r") as f:
return spack.spec.Spec.from_yaml(f)
except IOError:
return None
def colorize_root(root):
colorize = ft.partial(tty.color.colorize, color=sys.stdout.isatty())
@ -553,3 +662,7 @@ def get_dependencies(specs):
retval = set()
set(map(retval.update, (set(s.traverse()) for s in specs)))
return retval
class ConflictingProjectionsError(SpackError):
"""Raised when a view has a projections file and is given one manually."""

2
lib/spack/spack/package.py
View File

@ -284,7 +284,7 @@ def view_destination(self, view):
"""The target root directory: each file is added relative to this
directory.
"""
return view.root
return view.get_projection_for_spec(self.spec)
def view_file_conflicts(self, view, merge_map):
"""Report any files which prevent adding this package to the view. The

34
lib/spack/spack/schema/projections.py Normal file
View File

@ -0,0 +1,34 @@
# Copyright 2013-2018 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""Schema for projections.yaml configuration file.
.. literalinclude:: ../spack/schema/projections.py
:lines: 13-
"""
#: Properties for inclusion in other schemas
properties = {
'projections': {
'type': 'object',
'default': {},
'patternProperties': {
r'all|\w[\w-]*': {
'type': 'string'
},
},
},
}
#: Full schema with metadata
schema = {
'$schema': 'http://json-schema.org/schema#',
'title': 'Spack view projection configuration file schema',
'type': 'object',
'additionalProperties': False,
'properties': properties,
}

6
lib/spack/spack/spec.py
View File

@ -3025,6 +3025,8 @@ def format(self, format_string='$_$@$%@+$+$=', **kwargs):
${SHA1} Dependencies 8-char sha1 prefix
${HASH:len} DAG hash with optional length specifier
${DEP:name:OPTION} Evaluates as OPTION would for self['name']
${SPACK_ROOT} The spack root directory
${SPACK_INSTALL} The default spack install directory,
${SPACK_PREFIX}/opt
@ -3218,6 +3220,10 @@ def write(s, c=None):
out.write(fmt % (self.dag_hash(hashlen)))
elif named_str == 'NAMESPACE':
out.write(fmt % transform(self.namespace))
elif named_str.startswith('DEP:'):
_, dep_name, dep_option = named_str.lower().split(':', 2)
dep_spec = self[dep_name]
out.write(fmt % (dep_spec.format('${%s}' % dep_option)))
named = False

116
lib/spack/spack/test/cmd/view.py
View File

@ -7,12 +7,23 @@
import os.path
import pytest
import spack.util.spack_yaml as s_yaml
activate = SpackCommand('activate')
extensions = SpackCommand('extensions')
install = SpackCommand('install')
view = SpackCommand('view')
def create_projection_file(tmpdir, projection):
if 'projections' not in projection:
projection = {'projections': projection}
projection_file = tmpdir.mkdir('projection').join('projection.yaml')
projection_file.write(s_yaml.dump(projection))
return projection_file
@pytest.mark.parametrize('cmd', ['hardlink', 'symlink', 'hard', 'add'])
def test_view_link_type(
tmpdir, mock_packages, mock_archive, mock_fetch, config,
@ -25,6 +36,71 @@ def test_view_link_type(
assert os.path.islink(package_prefix) == (not cmd.startswith('hard'))
@pytest.mark.parametrize('cmd', ['hardlink', 'symlink', 'hard', 'add'])
def test_view_projections(
tmpdir, mock_packages, mock_archive, mock_fetch, config,
install_mockery, cmd):
install('libdwarf@20130207')
viewpath = str(tmpdir.mkdir('view_{0}'.format(cmd)))
view_projection = {
'projections': {
'all': '${PACKAGE}-${VERSION}'
}
}
projection_file = create_projection_file(tmpdir, view_projection)
view(cmd, viewpath, '--projection-file={0}'.format(projection_file),
'libdwarf')
package_prefix = os.path.join(viewpath, 'libdwarf-20130207/libdwarf')
assert os.path.exists(package_prefix)
assert os.path.islink(package_prefix) == (not cmd.startswith('hard'))
def test_view_multiple_projections(
tmpdir, mock_packages, mock_archive, mock_fetch, config,
install_mockery):
install('libdwarf@20130207')
install('extendee@1.0%gcc')
viewpath = str(tmpdir.mkdir('view'))
view_projection = s_yaml.syaml_dict(
[('extendee', '${PACKAGE}-${COMPILERNAME}'),
('all', '${PACKAGE}-${VERSION}')]
)
projection_file = create_projection_file(tmpdir, view_projection)
view('add', viewpath, '--projection-file={0}'.format(projection_file),
'libdwarf', 'extendee')
libdwarf_prefix = os.path.join(viewpath, 'libdwarf-20130207/libdwarf')
extendee_prefix = os.path.join(viewpath, 'extendee-gcc/bin')
assert os.path.exists(libdwarf_prefix)
assert os.path.exists(extendee_prefix)
def test_view_multiple_projections_all_first(
tmpdir, mock_packages, mock_archive, mock_fetch, config,
install_mockery):
install('libdwarf@20130207')
install('extendee@1.0%gcc')
viewpath = str(tmpdir.mkdir('view'))
view_projection = s_yaml.syaml_dict(
[('all', '${PACKAGE}-${VERSION}'),
('extendee', '${PACKAGE}-${COMPILERNAME}')]
)
projection_file = create_projection_file(tmpdir, view_projection)
view('add', viewpath, '--projection-file={0}'.format(projection_file),
'libdwarf', 'extendee')
libdwarf_prefix = os.path.join(viewpath, 'libdwarf-20130207/libdwarf')
extendee_prefix = os.path.join(viewpath, 'extendee-gcc/bin')
assert os.path.exists(libdwarf_prefix)
assert os.path.exists(extendee_prefix)
def test_view_external(
tmpdir, mock_packages, mock_archive, mock_fetch, config,
install_mockery):
@ -60,6 +136,39 @@ def test_view_extension(
assert os.path.exists(os.path.join(viewpath, 'bin', 'extension1'))
def test_view_extension_projection(
tmpdir, mock_packages, mock_archive, mock_fetch, config,
install_mockery):
install('extendee@1.0')
install('extension1@1.0')
install('extension1@2.0')
install('extension2@1.0')
viewpath = str(tmpdir.mkdir('view'))
view_projection = {'all': '${PACKAGE}-${VERSION}'}
projection_file = create_projection_file(tmpdir, view_projection)
view('symlink', viewpath, '--projection-file={0}'.format(projection_file),
'extension1@1.0')
all_installed = extensions('--show', 'installed', 'extendee')
assert 'extension1@1.0' in all_installed
assert 'extension1@2.0' in all_installed
assert 'extension2@1.0' in all_installed
global_activated = extensions('--show', 'activated', 'extendee')
assert 'extension1@1.0' not in global_activated
assert 'extension1@2.0' not in global_activated
assert 'extension2@1.0' not in global_activated
view_activated = extensions('--show', 'activated',
'-v', viewpath,
'extendee')
assert 'extension1@1.0' in view_activated
assert 'extension1@2.0' not in view_activated
assert 'extension2@1.0' not in view_activated
assert os.path.exists(os.path.join(viewpath, 'extendee-1.0',
'bin', 'extension1'))
def test_view_extension_remove(
tmpdir, mock_packages, mock_archive, mock_fetch, config,
install_mockery):
@ -144,3 +253,10 @@ def test_view_extendee_with_global_activations(
activate('extension1@2.0')
output = view('symlink', viewpath, 'extension1@1.0')
assert 'Error: Globally activated extensions cannot be used' in output
def test_view_fails_with_missing_projections_file(tmpdir):
viewpath = str(tmpdir.mkdir('view'))
projection_file = os.path.join(str(tmpdir), 'nonexistent')
with pytest.raises(SystemExit):
view('symlink', '--projection-file', projection_file, viewpath, 'foo')

2
lib/spack/spack/test/config.py
View File

@ -677,7 +677,7 @@ def check_schema(name, file_contents):
"""Check a Spack YAML schema against some data"""
f = StringIO(file_contents)
data = syaml.load(f)
spack.config._validate(data, name)
spack.config.validate(data, name)
def test_good_env_yaml(tmpdir):

142
lib/spack/spack/test/test_activations.py
View File

@ -5,69 +5,33 @@
import os
import pytest
import sys
import spack.spec
import spack.package
from llnl.util.link_tree import MergeConflictError
from spack.build_systems.python import PythonPackage
from spack.directory_layout import YamlDirectoryLayout
from spack.filesystem_view import YamlFilesystemView
from spack.util.prefix import Prefix
from spack.repo import RepoPath
"""This includes tests for customized activation logic for specific packages
(e.g. python and perl).
"""
class FakeExtensionPackage(spack.package.PackageViewMixin):
def __init__(self, name, prefix):
self.name = name
self.prefix = Prefix(prefix)
self.spec = FakeSpec(self)
def create_ext_pkg(name, prefix, extendee_spec):
ext_spec = spack.spec.Spec(name)
ext_spec._concrete = True
ext_spec.package.spec.prefix = prefix
ext_pkg = ext_spec.package
ext_pkg.extends_spec = extendee_spec
return ext_pkg
class FakeSpec(object):
def __init__(self, package):
self.name = package.name
self.prefix = package.prefix
self.hash = self.name
self.package = package
self.concrete = True
def dag_hash(self):
return self.hash
def __lt__(self, other):
return self.name < other.name
class FakePythonExtensionPackage(FakeExtensionPackage):
def __init__(self, name, prefix, py_namespace, python_spec):
self.py_namespace = py_namespace
self.extendee_spec = python_spec
super(FakePythonExtensionPackage, self).__init__(name, prefix)
def add_files_to_view(self, view, merge_map):
if sys.version_info >= (3, 0):
add_fn = PythonPackage.add_files_to_view
else:
add_fn = PythonPackage.add_files_to_view.im_func
return add_fn(self, view, merge_map)
def view_file_conflicts(self, view, merge_map):
if sys.version_info >= (3, 0):
conflicts_fn = PythonPackage.view_file_conflicts
else:
conflicts_fn = PythonPackage.view_file_conflicts.im_func
return conflicts_fn(self, view, merge_map)
def remove_files_from_view(self, view, merge_map):
if sys.version_info >= (3, 0):
remove_fn = PythonPackage.remove_files_from_view
else:
remove_fn = PythonPackage.remove_files_from_view.im_func
return remove_fn(self, view, merge_map)
def create_python_ext_pkg(name, prefix, python_spec, namespace=None):
ext_pkg = create_ext_pkg(name, prefix, python_spec)
ext_pkg.py_namespace = namespace
return ext_pkg
def create_dir_structure(tmpdir, dir_structure):
@ -78,7 +42,20 @@ def create_dir_structure(tmpdir, dir_structure):
@pytest.fixture()
def python_and_extension_dirs(tmpdir):
def builtin_and_mock_packages():
# These tests use mock_repo packages to test functionality of builtin
# packages for python and perl. To test this we put the mock repo at lower
# precedence than the builtin repo, so we test builtin.perl against
# builtin.mock.perl-extension.
repo_dirs = [spack.paths.packages_path, spack.paths.mock_packages_path]
path = RepoPath(*repo_dirs)
with spack.repo.swap(path):
yield
@pytest.fixture()
def python_and_extension_dirs(tmpdir, builtin_and_mock_packages):
python_dirs = {
'bin/': {
'python': None
@ -105,7 +82,7 @@ def python_and_extension_dirs(tmpdir):
'lib/': {
'python2.7/': {
'site-packages/': {
'py-extension/': {
'py-extension1/': {
'sample.py': None
}
}
@ -113,7 +90,7 @@ def python_and_extension_dirs(tmpdir):
}
}
ext_name = 'py-extension'
ext_name = 'py-extension1'
ext_prefix = tmpdir.join(ext_name)
create_dir_structure(ext_prefix, ext_dirs)
@ -126,7 +103,7 @@ def python_and_extension_dirs(tmpdir):
@pytest.fixture()
def namespace_extensions(tmpdir):
def namespace_extensions(tmpdir, builtin_and_mock_packages):
ext1_dirs = {
'bin/': {
'py-ext-tool1': None
@ -170,14 +147,15 @@ def namespace_extensions(tmpdir):
return str(ext1_prefix), str(ext2_prefix), 'examplenamespace'
def test_python_activation_with_files(tmpdir, python_and_extension_dirs):
def test_python_activation_with_files(tmpdir, python_and_extension_dirs,
builtin_and_mock_packages):
python_prefix, ext_prefix = python_and_extension_dirs
python_spec = spack.spec.Spec('python@2.7.12')
python_spec._concrete = True
python_spec.package.spec.prefix = python_prefix
ext_pkg = FakeExtensionPackage('py-extension', ext_prefix)
ext_pkg = create_python_ext_pkg('py-extension1', ext_prefix, python_spec)
python_pkg = python_spec.package
python_pkg.activate(ext_pkg, python_pkg.view())
@ -192,14 +170,15 @@ def test_python_activation_with_files(tmpdir, python_and_extension_dirs):
assert 'setuptools.egg' not in easy_install_contents
def test_python_activation_view(tmpdir, python_and_extension_dirs):
def test_python_activation_view(tmpdir, python_and_extension_dirs,
builtin_and_mock_packages):
python_prefix, ext_prefix = python_and_extension_dirs
python_spec = spack.spec.Spec('python@2.7.12')
python_spec._concrete = True
python_spec.package.spec.prefix = python_prefix
ext_pkg = FakeExtensionPackage('py-extension', ext_prefix)
ext_pkg = create_python_ext_pkg('py-extension1', ext_prefix, python_spec)
view_dir = str(tmpdir.join('view'))
layout = YamlDirectoryLayout(view_dir)
@ -213,7 +192,8 @@ def test_python_activation_view(tmpdir, python_and_extension_dirs):
assert os.path.exists(os.path.join(view_dir, 'bin/py-ext-tool'))
def test_python_ignore_namespace_init_conflict(tmpdir, namespace_extensions):
def test_python_ignore_namespace_init_conflict(tmpdir, namespace_extensions,
builtin_and_mock_packages):
"""Test the view update logic in PythonPackage ignores conflicting
instances of __init__ for packages which are in the same namespace.
"""
@ -222,10 +202,10 @@ def test_python_ignore_namespace_init_conflict(tmpdir, namespace_extensions):
python_spec = spack.spec.Spec('python@2.7.12')
python_spec._concrete = True
ext1_pkg = FakePythonExtensionPackage(
'py-extension1', ext1_prefix, py_namespace, python_spec)
ext2_pkg = FakePythonExtensionPackage(
'py-extension2', ext2_prefix, py_namespace, python_spec)
ext1_pkg = create_python_ext_pkg('py-extension1', ext1_prefix, python_spec,
py_namespace)
ext2_pkg = create_python_ext_pkg('py-extension2', ext2_prefix, python_spec,
py_namespace)
view_dir = str(tmpdir.join('view'))
layout = YamlDirectoryLayout(view_dir)
@ -246,7 +226,8 @@ def test_python_ignore_namespace_init_conflict(tmpdir, namespace_extensions):
assert os.path.exists(os.path.join(view_dir, init_file))
def test_python_keep_namespace_init(tmpdir, namespace_extensions):
def test_python_keep_namespace_init(tmpdir, namespace_extensions,
builtin_and_mock_packages):
"""Test the view update logic in PythonPackage keeps the namespace
__init__ file as long as one package in the namespace still
exists.
@ -256,10 +237,10 @@ def test_python_keep_namespace_init(tmpdir, namespace_extensions):
python_spec = spack.spec.Spec('python@2.7.12')
python_spec._concrete = True
ext1_pkg = FakePythonExtensionPackage(
'py-extension1', ext1_prefix, py_namespace, python_spec)
ext2_pkg = FakePythonExtensionPackage(
'py-extension2', ext2_prefix, py_namespace, python_spec)
ext1_pkg = create_python_ext_pkg('py-extension1', ext1_prefix, python_spec,
py_namespace)
ext2_pkg = create_python_ext_pkg('py-extension2', ext2_prefix, python_spec,
py_namespace)
view_dir = str(tmpdir.join('view'))
layout = YamlDirectoryLayout(view_dir)
@ -287,7 +268,8 @@ def test_python_keep_namespace_init(tmpdir, namespace_extensions):
assert not os.path.exists(os.path.join(view_dir, init_file))
def test_python_namespace_conflict(tmpdir, namespace_extensions):
def test_python_namespace_conflict(tmpdir, namespace_extensions,
builtin_and_mock_packages):
"""Test the view update logic in PythonPackage reports an error when two
python extensions with different namespaces have a conflicting __init__
file.
@ -298,10 +280,10 @@ def test_python_namespace_conflict(tmpdir, namespace_extensions):
python_spec = spack.spec.Spec('python@2.7.12')
python_spec._concrete = True
ext1_pkg = FakePythonExtensionPackage(
'py-extension1', ext1_prefix, py_namespace, python_spec)
ext2_pkg = FakePythonExtensionPackage(
'py-extension2', ext2_prefix, other_namespace, python_spec)
ext1_pkg = create_python_ext_pkg('py-extension1', ext1_prefix, python_spec,
py_namespace)
ext2_pkg = create_python_ext_pkg('py-extension2', ext2_prefix, python_spec,
other_namespace)
view_dir = str(tmpdir.join('view'))
layout = YamlDirectoryLayout(view_dir)
@ -315,7 +297,7 @@ def test_python_namespace_conflict(tmpdir, namespace_extensions):
@pytest.fixture()
def perl_and_extension_dirs(tmpdir):
def perl_and_extension_dirs(tmpdir, builtin_and_mock_packages):
perl_dirs = {
'bin/': {
'perl': None
@ -360,7 +342,7 @@ def perl_and_extension_dirs(tmpdir):
return str(perl_prefix), str(ext_prefix)
def test_perl_activation(tmpdir):
def test_perl_activation(tmpdir, builtin_and_mock_packages):
# Note the lib directory is based partly on the perl version
perl_spec = spack.spec.Spec('perl@5.24.1')
perl_spec._concrete = True
@ -375,20 +357,21 @@ def test_perl_activation(tmpdir):
ext_name = 'perl-extension'
tmpdir.ensure(ext_name, dir=True)
ext_pkg = FakeExtensionPackage(ext_name, str(tmpdir.join(ext_name)))
ext_pkg = create_ext_pkg(ext_name, str(tmpdir.join(ext_name)), perl_spec)
perl_pkg = perl_spec.package
perl_pkg.activate(ext_pkg, perl_pkg.view())
def test_perl_activation_with_files(tmpdir, perl_and_extension_dirs):
def test_perl_activation_with_files(tmpdir, perl_and_extension_dirs,
builtin_and_mock_packages):
perl_prefix, ext_prefix = perl_and_extension_dirs
perl_spec = spack.spec.Spec('perl@5.24.1')
perl_spec._concrete = True
perl_spec.package.spec.prefix = perl_prefix
ext_pkg = FakeExtensionPackage('perl-extension', ext_prefix)
ext_pkg = create_ext_pkg('perl-extension', ext_prefix, perl_spec)
perl_pkg = perl_spec.package
perl_pkg.activate(ext_pkg, perl_pkg.view())
@ -396,14 +379,15 @@ def test_perl_activation_with_files(tmpdir, perl_and_extension_dirs):
assert os.path.exists(os.path.join(perl_prefix, 'bin/perl-ext-tool'))
def test_perl_activation_view(tmpdir, perl_and_extension_dirs):
def test_perl_activation_view(tmpdir, perl_and_extension_dirs,
builtin_and_mock_packages):
perl_prefix, ext_prefix = perl_and_extension_dirs
perl_spec = spack.spec.Spec('perl@5.24.1')
perl_spec._concrete = True
perl_spec.package.spec.prefix = perl_prefix
ext_pkg = FakeExtensionPackage('perl-extension', ext_prefix)
ext_pkg = create_ext_pkg('perl-extension', ext_prefix, perl_spec)
view_dir = str(tmpdir.join('view'))
layout = YamlDirectoryLayout(view_dir)

28
var/spack/repos/builtin.mock/packages/perl-extension/package.py Normal file
View File

@ -0,0 +1,28 @@
# Copyright 2013-2018 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
import os.path
class PerlExtension(PerlPackage):
"""A package which extends perl"""
homepage = "http://www.example.com"
url = "http://www.example.com/extension1-1.0.tar.gz"
version('1.0', 'hash-extension-1.0')
version('2.0', 'hash-extension-2.0')
def install(self, spec, prefix):
mkdirp(prefix.bin)
with open(os.path.join(prefix.bin, 'perl-extension'), 'w+') as fout:
fout.write(str(spec.version))
# Give the package a hook to set the extendee spec
extends_spec = 'perl'
@property
def extendee_spec(self):
return self.extends_spec

18
var/spack/repos/builtin.mock/packages/perl/package.py Normal file
View File

@ -0,0 +1,18 @@
# Copyright 2013-2018 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Perl(Package):
"""Dummy Perl package to allow a dummy perl-extension in repo."""
homepage = "http://www.python.org"
url = "http://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz"
extendable = True
version('0.0.0', 'hash')
def install(self, spec, prefix):
pass

28
var/spack/repos/builtin.mock/packages/py-extension1/package.py Normal file
View File

@ -0,0 +1,28 @@
# Copyright 2013-2018 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
import os.path
class PyExtension1(PythonPackage):
"""A package which extends python"""
homepage = "http://www.example.com"
url = "http://www.example.com/extension1-1.0.tar.gz"
version('1.0', 'hash-extension1-1.0')
version('2.0', 'hash-extension1-2.0')
def install(self, spec, prefix):
mkdirp(prefix.bin)
with open(os.path.join(prefix.bin, 'py-extension1'), 'w+') as fout:
fout.write(str(spec.version))
# Give the package a hook to set the extendee spec
extends_spec = 'python'
@property
def extendee_spec(self):
return self.extends_spec

30
var/spack/repos/builtin.mock/packages/py-extension2/package.py Normal file
View File

@ -0,0 +1,30 @@
# Copyright 2013-2018 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
import os.path
class PyExtension2(PythonPackage):
"""A package which extends python. It also depends on another
package which extends the same package."""
homepage = "http://www.example.com"
url = "http://www.example.com/extension2-1.0.tar.gz"
depends_on('py-extension1', type=('build', 'run'))
version('1.0', 'hash-extension2-1.0')
def install(self, spec, prefix):
mkdirp(prefix.bin)
with open(os.path.join(prefix.bin, 'py-extension2'), 'w+') as fout:
fout.write(str(spec.version))
# Give the package a hook to set the extendee spec
extends_spec = 'python'
@property
def extendee_spec(self):
return self.extends_spec

17
var/spack/repos/builtin/packages/python/package.py
View File

@ -694,7 +694,9 @@ def activate(self, ext_pkg, view, **args):
exts = extensions_layout.extension_map(self.spec)
exts[ext_pkg.name] = ext_pkg.spec
self.write_easy_install_pth(exts, prefix=view.root)
self.write_easy_install_pth(exts, prefix=view.get_projection_for_spec(
self.spec
))
def deactivate(self, ext_pkg, view, **args):
args.update(ignore=self.python_ignore(ext_pkg, args))
@ -706,7 +708,10 @@ def deactivate(self, ext_pkg, view, **args):
# Make deactivate idempotent
if ext_pkg.name in exts:
del exts[ext_pkg.name]
self.write_easy_install_pth(exts, prefix=view.root)
self.write_easy_install_pth(exts,
prefix=view.get_projection_for_spec(
self.spec
))
def add_files_to_view(self, view, merge_map):
bin_dir = self.spec.prefix.bin
@ -717,7 +722,13 @@ def add_files_to_view(self, view, merge_map):
copy(src, dst)
if 'script' in get_filetype(src):
filter_file(
self.spec.prefix, os.path.abspath(view.root), dst)
self.spec.prefix,
os.path.abspath(
view.get_projection_for_spec(self.spec)
),
dst,
backup=False
)
else:
orig_link_target = os.path.realpath(src)
new_link_target = os.path.abspath(merge_map[orig_link_target])