Changed how architecture looks like in spec.yaml. Now it's only three strings, platform, os, target in spec.yaml. Also did some flake8 formatting changes

This commit is contained in:
Mario Melara 2016-06-03 15:57:54 -07:00
parent b14ba31250
commit 24d160e93e

View File

@ -29,8 +29,9 @@
On a multiple architecture machine, the architecture spec field can be set to On a multiple architecture machine, the architecture spec field can be set to
build a package against any target and operating system that is present on the build a package against any target and operating system that is present on the
platform. On Cray platforms or any other architecture that has different front and platform. On Cray platforms or any other architecture that has different front
back end environments, the operating system will determine the method of compiler and back end environments, the operating system will determine the method of
compiler
detection. detection.
There are two different types of compiler detection: There are two different types of compiler detection:
@ -40,44 +41,42 @@
Depending on which operating system is specified, the compiler will be detected Depending on which operating system is specified, the compiler will be detected
using one of those methods. using one of those methods.
For platforms such as linux and darwin, the operating system is autodetected and For platforms such as linux and darwin, the operating system is autodetected
the target is set to be x86_64. and the target is set to be x86_64.
The command line syntax for specifying an architecture is as follows: The command line syntax for specifying an architecture is as follows:
target=<Target name> os=<OperatingSystem name> target=<Target name> os=<OperatingSystem name>
If the user wishes to use the defaults, either target or os can be left out of If the user wishes to use the defaults, either target or os can be left out of
the command line and Spack will concretize using the default. These defaults are the command line and Spack will concretize using the default. These defaults
set in the 'platforms/' directory which contains the different subclasses for are set in the 'platforms/' directory which contains the different subclasses
platforms. If the machine has multiple architectures, the user can for platforms. If the machine has multiple architectures, the user can
also enter front-end, or fe or back-end or be. These settings will concretize also enter front-end, or fe or back-end or be. These settings will concretize
to their respective front-end and back-end targets and operating systems. to their respective front-end and back-end targets and operating systems.
Additional platforms can be added by creating a subclass of Platform Additional platforms can be added by creating a subclass of Platform
and adding it inside the platform directory. and adding it inside the platform directory.
Platforms are an abstract class that are extended by subclasses. If the user Platforms are an abstract class that are extended by subclasses. If the user
wants to add a new type of platform (such as cray_xe), they can create a subclass wants to add a new type of platform (such as cray_xe), they can create a
and set all the class attributes such as priority, front_target ,back_target, subclass and set all the class attributes such as priority, front_target,
front_os, back_os. Platforms also contain a priority class attribute. A lower back_target, front_os, back_os. Platforms also contain a priority class
number signifies higher priority. These numbers are arbitrarily set and can be attribute. A lower number signifies higher priority. These numbers are
changed though often there isn't much need unless a new platform is added and arbitrarily set and can be changed though often there isn't much need unless a
the user wants that to be detected first. new platform is added and the user wants that to be detected first.
Targets are created inside the platform subclasses. Most architecture (like linux, Targets are created inside the platform subclasses. Most architecture
and darwin) will have only one target (x86_64) but in the case of Cray machines, (like linux, and darwin) will have only one target (x86_64) but in the case of
there is both a frontend and backend processor. The user can specify which targets Cray machines, there is both a frontend and backend processor. The user can
are present on front-end and back-end architecture specify which targets are present on front-end and back-end architecture
Depending on the platform, operating systems are either auto-detected or are Depending on the platform, operating systems are either auto-detected or are
set. The user can set the front-end and back-end operating setting by the class set. The user can set the front-end and back-end operating setting by the class
attributes front_os and back_os. The operating system as described earlier, will attributes front_os and back_os. The operating system as described earlier,
be responsible for compiler detection. will be responsible for compiler detection.
""" """
import os import os
from collections import namedtuple
import imp import imp
import platform as py_platform
import inspect import inspect
from llnl.util.lang import memoized, list_modules, key_ordering from llnl.util.lang import memoized, list_modules, key_ordering
@ -91,6 +90,7 @@
from spack.util.multiproc import parmap from spack.util.multiproc import parmap
import spack.error as serr import spack.error as serr
class InvalidSysTypeError(serr.SpackError): class InvalidSysTypeError(serr.SpackError):
def __init__(self, sys_type): def __init__(self, sys_type):
super(InvalidSysTypeError, self).__init__( super(InvalidSysTypeError, self).__init__(
@ -128,12 +128,6 @@ def __repr__(self):
def __str__(self): def __str__(self):
return self.name return self.name
def to_dict(self):
d = {}
d['name'] = self.name
d['module_name'] = self.module_name
return d
@key_ordering @key_ordering
class Platform(object): class Platform(object):
@ -142,7 +136,7 @@ class Platform(object):
is returned is returned
""" """
priority = None # Subclass needs to set this number. This controls order in which platform is detected. priority = None # Subclass sets number. Controls detection order
front_end = None front_end = None
back_end = None back_end = None
default = None # The default back end target. On cray ivybridge default = None # The default back end target. On cray ivybridge
@ -156,19 +150,6 @@ def __init__(self, name):
self.operating_sys = {} self.operating_sys = {}
self.name = name self.name = name
def to_dict(self):
n = {}
n['targets'] = dict((name, target.to_dict()) for (name, target) in self.targets.items())
n['operating_systems'] = dict((name, os.to_dict()) for (name, os) in self.operating_sys.items())
n['priority'] = self.priority
n['default_front_end_target'] = self.front_end
n['default_back_end_target'] = self.back_end
n['default_target'] = self.default
n['default_front_end_os'] = self.front_os
n['default_back_end_os'] = self.back_os
n['default_os'] = self.default_os
return {self.name: n}
def add_target(self, name, target): def add_target(self, name, target):
"""Used by the platform specific subclass to list available targets. """Used by the platform specific subclass to list available targets.
Raises an error if the platform specifies a name Raises an error if the platform specifies a name
@ -215,7 +196,6 @@ def operating_system(self, name):
return self.operating_sys.get(name, None) return self.operating_sys.get(name, None)
@classmethod @classmethod
def detect(self): def detect(self):
""" Subclass is responsible for implementing this method. """ Subclass is responsible for implementing this method.
@ -232,8 +212,10 @@ def __str__(self):
return self.name return self.name
def _cmp_key(self): def _cmp_key(self):
t_keys = ''.join(str(t._cmp_key()) for t in sorted(self.targets.values())) t_keys = ''.join(str(t._cmp_key()) for t in
o_keys = ''.join(str(o._cmp_key()) for o in sorted(self.operating_sys.values())) sorted(self.targets.values()))
o_keys = ''.join(str(o._cmp_key()) for o in
sorted(self.operating_sys.values()))
return (self.name, return (self.name,
self.default, self.default,
self.front_end, self.front_end,
@ -244,6 +226,7 @@ def _cmp_key(self):
t_keys, t_keys,
o_keys) o_keys)
@key_ordering @key_ordering
class OperatingSystem(object): class OperatingSystem(object):
""" Operating System will be like a class similar to platform extended """ Operating System will be like a class similar to platform extended
@ -265,7 +248,6 @@ def __repr__(self):
def _cmp_key(self): def _cmp_key(self):
return (self.name, self.version) return (self.name, self.version)
def find_compilers(self, *paths): def find_compilers(self, *paths):
""" """
Return a list of compilers found in the suppied paths. Return a list of compilers found in the suppied paths.
@ -293,11 +275,13 @@ def find_compilers(self, *paths):
# compiler. We can spawn a bunch of parallel searches to reduce # compiler. We can spawn a bunch of parallel searches to reduce
# the overhead of spelunking all these directories. # the overhead of spelunking all these directories.
types = spack.compilers.all_compiler_types() types = spack.compilers.all_compiler_types()
compiler_lists = parmap(lambda cmp_cls: self.find_compiler(cmp_cls, *filtered_path), types) compiler_lists = parmap(lambda cmp_cls:
self.find_compiler(cmp_cls, *filtered_path),
types)
# ensure all the version calls we made are cached in the parent # ensure all the version calls we made are cached in the parent
# process, as well. This speeds up Spack a lot. # process, as well. This speeds up Spack a lot.
clist = reduce(lambda x,y: x+y, compiler_lists) clist = reduce(lambda x, y: x+y, compiler_lists)
return clist return clist
def find_compiler(self, cmp_cls, *path): def find_compiler(self, cmp_cls, *path):
@ -338,7 +322,7 @@ def find_compiler(self, cmp_cls, *path):
# prefer the one with more compilers. # prefer the one with more compilers.
prev_paths = [prev.cc, prev.cxx, prev.f77, prev.fc] prev_paths = [prev.cc, prev.cxx, prev.f77, prev.fc]
newcount = len([p for p in paths if p is not None]) newcount = len([p for p in paths if p is not None])
prevcount = len([p for p in prev_paths if p is not None]) prevcount = len([p for p in prev_paths if p is not None])
# Don't add if it's not an improvement over prev compiler. # Don't add if it's not an improvement over prev compiler.
@ -349,14 +333,7 @@ def find_compiler(self, cmp_cls, *path):
return list(compilers.values()) return list(compilers.values())
def to_dict(self):
d = {}
d['name'] = self.name
d['version'] = self.version
return d
#NOTE: Key error caused because Architecture has no comparison method
@key_ordering @key_ordering
class Arch(object): class Arch(object):
"Architecture is now a class to help with setting attributes" "Architecture is now a class to help with setting attributes"
@ -376,10 +353,11 @@ def __init__(self, platform=None, platform_os=None, target=None):
@property @property
def concrete(self): def concrete(self):
return all( (self.platform is not None, isinstance(self.platform, Platform), return all((self.platform is not None,
self.platform_os is not None, isinstance(self.platform_os, OperatingSystem), isinstance(self.platform, Platform),
self.target is not None, isinstance(self.target, Target) ) ) self.platform_os is not None,
isinstance(self.platform_os, OperatingSystem),
self.target is not None, isinstance(self.target, Target)))
def __str__(self): def __str__(self):
if self.platform or self.platform_os or self.target: if self.platform or self.platform_os or self.target:
@ -388,71 +366,75 @@ def __str__(self):
else: else:
os_name = str(self.platform_os) os_name = str(self.platform_os)
return (str(self.platform) +"-"+ return (str(self.platform) + "-" +
os_name + "-" + str(self.target)) os_name + "-" + str(self.target))
else: else:
return '' return ''
def _cmp_key(self): def _cmp_key(self):
platform = self.platform.name if isinstance(self.platform, Platform) else self.platform if isinstance(self.platform, Platform):
os = self.platform_os.name if isinstance(self.platform_os, OperatingSystem) else self.platform_os platform = self.platform.name
target = self.target.name if isinstance(self.target, Target) else self.target else:
return (platform, os, target) platform = self.platform
if isinstance(self.platform_os, OperatingSystem):
platform_os = self.platform_os.name
else:
platform_os = self.platform_os
if isinstance(self.target, Target):
target = self.target.name
else:
target = self.target
print (platform, platform_os, target)
return (platform, platform_os, target)
def to_dict(self): def to_dict(self):
d = {} d = {}
platform = self.platform d['platform'] = str(self.platform) if self.platform else None
platform_os = self.platform_os d['platform_os'] = str(self.platform_os) if self.platform_os else None
target = self.target d['target'] = str(self.target) if self.target else None
d['platform'] = self.platform.to_dict() if self.platform else None
d['platform_os'] = self.platform_os.to_dict() if self.platform_os else None
d['target'] = self.target.to_dict() if self.target else None
return d return d
def _target_from_dict(target_dict): def _target_from_dict(target_name, platform=None):
""" Creates new instance of target and assigns all the attributes of """ Creates new instance of target and assigns all the attributes of
that target from the dictionary that target from the dictionary
""" """
target = Target.__new__(Target) if not platform:
target.name = target_dict['name'] platform = sys_type()
target.module_name = target_dict['module_name'] return platform.target(target_name)
if 'platform_name' in target_dict:
target.platform_name = target_dict['platform_name']
return target
def _operating_system_from_dict(os_dict):
def _operating_system_from_dict(os_name, platform=None):
""" uses platform's operating system method to grab the constructed """ uses platform's operating system method to grab the constructed
operating systems that are valid on the platform. operating systems that are valid on the platform.
""" """
# NOTE: Might need a better way to create operating system objects if not platform:
operating_system = OperatingSystem.__new__(OperatingSystem) platform = sys_type()
operating_system.name = os_dict['name'] if isinstance(os_name, spack.util.spack_yaml.syaml_dict):
operating_system.version = os_dict['version'] name = os_name['name']
return operating_system version = os_name['version']
return platform.operating_system(name+version)
else:
return platform.operating_system(os_name)
def _platform_from_dict(platform_dict):
def _platform_from_dict(platform_name):
""" Constructs a platform from a dictionary. """ """ Constructs a platform from a dictionary. """
platform = Platform.__new__(Platform) platform_path = spack.platform_path
name, p_dict = platform_dict.items()[0] mod_string = "spack.platforms"
platform.name = name
platform.targets = {} for p in list_modules(platform_path):
for name, t_dict in p_dict['targets'].items(): if platform_name == p:
platform.add_target(name, _target_from_dict(t_dict)) mod_name = mod_string + platform_name
platform.operating_sys = {} path = join_path(platform_path, platform_name) + ".py"
for name, o_dict in p_dict['operating_systems'].items(): mod = imp.load_source(mod_name, path)
platform.add_operating_system(name, _operating_system_from_dict(o_dict)) platform_class = mod_to_class(platform_name)
platform.priority = p_dict['priority'] cls = getattr(mod, platform_class)
platform.front_end = p_dict['default_front_end_target'] platform = cls()
platform.back_end = p_dict['default_back_end_target'] return platform
platform.default = p_dict['default_target'] return None
platform.front_os = p_dict['default_front_end_os']
platform.back_os = p_dict['default_back_end_os']
platform.default_os = p_dict['default_os']
return platform
def arch_from_dict(d): def arch_from_dict(d):
""" Uses _platform_from_dict, _operating_system_from_dict, _target_from_dict """ Uses _platform_from_dict, _operating_system_from_dict, _target_from_dict
@ -472,19 +454,30 @@ def arch_from_dict(d):
else: else:
if d is None: if d is None:
return None return None
platform_dict = d['platform'] platform_name = d['platform']
os_dict = d['platform_os'] os_name = d['platform_os']
target_dict = d['target'] target_name = d['target']
arch.platform = _platform_from_dict(platform_dict) if platform_dict else None if platform_name:
arch.target = _target_from_dict(target_dict) if os_dict else None arch.platform = _platform_from_dict(platform_name)
arch.platform_os = _operating_system_from_dict(os_dict) if os_dict else None else:
arch.platform = None
if target_name:
arch.target = _target_from_dict(target_name, arch.platform)
else:
arch.target = None
if os_name:
arch.platform_os = _operating_system_from_dict(os_name,
arch.platform)
else:
arch.platform_os = None
arch.os_string = None arch.os_string = None
arch.target_string = None arch.target_string = None
return arch return arch
@memoized @memoized
def all_platforms(): def all_platforms():
modules = [] modules = []
@ -507,6 +500,7 @@ def all_platforms():
return modules return modules
@memoized @memoized
def sys_type(): def sys_type():
""" Gather a list of all available subclasses of platforms. """ Gather a list of all available subclasses of platforms.