zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Python command, libraries, and headers (#3367)

Browse Source 
## Motivation

Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically. 

I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.

Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends.

## Prefix

For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`.

Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.

To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable.

## Command

In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:

If the spec is for Python 3, try searching for the `python3` command.
If the spec is for Python 2, try searching for the `python2` command.
If neither are found, try searching for the `python` command.

## Libraries

Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work.

The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:
```
lib/libpython2.7.dylib
```
For Python 3.6 on CentOS 6, I'm seeing:
```
lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0
```
Notice the `m` after the version number. Yeah, that's a thing.

## Headers

In Python 2.7, I'm seeing:
```
include/python2.7/pyconfig.h
```
In Python 3.6, I'm seeing:
```
include/python3.6m/pyconfig.h
```
It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6

Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
This commit is contained in:
Adam J. Stewart
2017-04-29 19:24:13 -05:00
committed by Todd Gamblin
parent a32a0eacba
commit ce3ab503de
44 changed files with 883 additions and 378 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
var/spack/repos/builtin/packages/abinit/package.py
View File

@@ -33,9 +33,9 @@ class Abinit(AutotoolsPackage):
energy, charge density and electronic structure of systems made of
electrons and nuclei (molecules and periodic solids) within
Density Functional Theory (DFT), using pseudopotentials and a planewave
or wavelet basis.
ABINIT also includes options to optimize the geometry according to the
or wavelet basis.
ABINIT also includes options to optimize the geometry according to the
DFT forces and stresses, or to perform molecular dynamics
simulations using these forces, or to generate dynamical matrices,
Born effective charges, and dielectric tensors, based on Density-Functional
@@ -149,7 +149,7 @@ def configure_args(self):
# LibXC library
libxc = spec['libxc:fortran']
options.extend([
'with_libxc_incs={0}'.format(libxc.cppflags),
'with_libxc_incs={0}'.format(libxc.headers.cpp_flags),
'with_libxc_libs={0}'.format(libxc.libs.ld_flags + ' -lm')
])
@@ -161,7 +161,7 @@ def configure_args(self):
hdf5 = spec['hdf5:hl']
netcdff = spec['netcdf-fortran:shared']
options.extend([
'--with-netcdf-incs={0}'.format(netcdff.cppflags),
'--with-netcdf-incs={0}'.format(netcdff.headers.cpp_flags),
'--with-netcdf-libs={0}'.format(
netcdff.libs.ld_flags + ' ' + hdf5.libs.ld_flags
),
@@ -175,7 +175,7 @@ def configure_args(self):
def check(self):
"""This method is called after the build phase if tests have been
explicitly activated by user.
explicitly activated by user.
"""
make('check')
make('tests_in')

2
var/spack/repos/builtin/packages/ack/package.py
View File

@@ -45,7 +45,7 @@ def install(self, spec, prefix):
ack = 'ack-{0}-single-file'.format(self.version)
# rewrite the script's #! line to call the perl dependency
shbang = '#!' + join_path(spec['perl'].prefix.bin, 'perl')
shbang = '#!' + spec['perl'].command.path
filter_file(r'^#!/usr/bin/env perl', shbang, ack)
install(ack, join_path(prefix.bin, "ack"))

2
var/spack/repos/builtin/packages/blast-plus/package.py
View File

@@ -206,7 +206,7 @@ def configure_args(self):
if '+python' in spec:
config_args.append(
'--with-python={0}'.format(self.spec['python'].prefix)
'--with-python={0}'.format(self.spec['python'].home)
)
else:
config_args.append('--without-python')

23
var/spack/repos/builtin/packages/boost/package.py
View File

@@ -25,7 +25,6 @@
from spack import *
import sys
import os
from glob import glob
class Boost(Package):
@@ -173,23 +172,11 @@ def determine_toolset(self, spec):
return 'gcc'
def bjam_python_line(self, spec):
from os.path import dirname, splitext
pydir = 'python%s.%s*' % spec['python'].version.version[:2]
incs = join_path(spec['python'].prefix.include, pydir, "pyconfig.h")
incs = glob(incs)
incs = " ".join([dirname(u) for u in incs])
pylib = 'libpython%s.%s*' % spec['python'].version.version[:2]
all_libs = join_path(spec['python'].prefix.lib, pylib)
libs = [u for u in all_libs if splitext(u)[1] == dso_suffix]
if len(libs) == 0:
libs = [u for u in all_libs if splitext(u)[1] == '.a']
libs = " ".join(libs)
return 'using python : %s : %s : %s : %s ;\n' % (
return 'using python : {0} : {1} : {2} : {3} ;\n'.format(
spec['python'].version.up_to(2),
join_path(spec['python'].prefix.bin, 'python'),
incs, libs
spec['python'].command.path,
spec['python'].headers.directories[0],
spec['python'].libs[0]
)
def determine_bootstrap_options(self, spec, withLibs, options):
@@ -198,7 +185,7 @@ def determine_bootstrap_options(self, spec, withLibs, options):
options.append("--with-libraries=%s" % ','.join(withLibs))
if '+python' in spec:
options.append('--with-python=%s' % python_exe)
options.append('--with-python=%s' % spec['python'].command.path)
with open('user-config.jam', 'w') as f:
# Boost may end up using gcc even though clang+gfortran is set in

6
var/spack/repos/builtin/packages/cantera/package.py
View File

@@ -117,15 +117,13 @@ def install(self, spec, prefix):
if '+python' in spec:
options.extend([
'python_package=full',
'python_cmd={0}'.format(
join_path(spec['python'].prefix.bin, 'python')),
'python_cmd={0}'.format(spec['python'].command.path),
'python_array_home={0}'.format(spec['py-numpy'].prefix)
])
if spec['python'].satisfies('@3'):
options.extend([
'python3_package=y',
'python3_cmd={0}'.format(
join_path(spec['python'].prefix.bin, 'python')),
'python3_cmd={0}'.format(spec['python'].command.path),
'python3_array_home={0}'.format(spec['py-numpy'].prefix)
])
else:

18
var/spack/repos/builtin/packages/conduit/package.py
View File

@@ -174,13 +174,13 @@ def create_host_config(self, spec, prefix):
##############################################
if "+cmake" in spec:
cmake_exe = join_path(spec['cmake'].prefix.bin, "cmake")
cmake_exe = spec['cmake'].command.path
else:
cmake_exe = which("cmake")
if cmake_exe is None:
msg = 'failed to find CMake (and cmake variant is off)'
raise RuntimeError(msg)
cmake_exe = cmake_exe.command
cmake_exe = cmake_exe.path
host_cfg_fname = "%s-%s-%s.cmake" % (socket.gethostname(),
sys_type,
@@ -224,21 +224,15 @@ def create_host_config(self, spec, prefix):
cfg.write("# Python Support\n")
if "+python" in spec:
python_exe = join_path(spec['python'].prefix.bin, "python")
cfg.write("# Enable python module builds\n")
cfg.write(cmake_cache_entry("ENABLE_PYTHON", "ON"))
cfg.write("# python from spack \n")
cfg.write(cmake_cache_entry("PYTHON_EXECUTABLE", python_exe))
cfg.write(cmake_cache_entry("PYTHON_EXECUTABLE",
spec['python'].command.path))
# install module to standard style site packages dir
# so we can support spack activate
py_ver_short = "python{0}".format(spec["python"].version.up_to(2))
pym_prefix = join_path("${CMAKE_INSTALL_PREFIX}",
"lib",
py_ver_short,
"site-packages")
# use pym_prefix as the install path
cfg.write(cmake_cache_entry("PYTHON_MODULE_INSTALL_PREFIX",
pym_prefix))
site_packages_dir))
else:
cfg.write(cmake_cache_entry("ENABLE_PYTHON", "OFF"))
@@ -251,7 +245,7 @@ def create_host_config(self, spec, prefix):
cfg.write(cmake_cache_entry("SPHINX_EXECUTABLE", sphinx_build_exe))
cfg.write("# doxygen from uberenv\n")
doxygen_exe = join_path(spec['doxygen'].prefix.bin, "doxygen")
doxygen_exe = spec['doxygen'].command.path
cfg.write(cmake_cache_entry("DOXYGEN_EXECUTABLE", doxygen_exe))
else:
cfg.write(cmake_cache_entry("ENABLE_DOCS", "OFF"))

2
var/spack/repos/builtin/packages/cosmomc/package.py
View File

@@ -178,7 +178,7 @@ def check_install(self):
os.environ.pop('CLIKPATH', '')
os.environ.pop('PLANCKLIKE', '')
exe = join_path(prefix.bin, 'cosmomc')
exe = spec['cosmomc'].command.path
args = []
if '+mpi' in spec:
# Add mpirun prefix

8
var/spack/repos/builtin/packages/cp2k/package.py
View File

@@ -93,10 +93,10 @@ def install(self, spec, prefix):
cppflags = [
'-D__FFTW3',
'-D__LIBINT',
spec['fftw'].cppflags
spec['fftw'].headers.cpp_flags
]
fcflags = copy.deepcopy(optflags[self.spec.compiler.name])
fcflags.append(spec['fftw'].cppflags)
fcflags.append(spec['fftw'].headers.cpp_flags)
fftw = find_libraries('libfftw3', root=spec['fftw'].prefix.lib)
ldflags = [fftw.search_flags]
if 'superlu-dist@4.3' in spec:
@@ -161,14 +161,14 @@ def install(self, spec, prefix):
cppflags.append('-D__WANNIER90')
fcflags.extend([
# spec['elpa:fortran'].cppflags
# spec['elpa:fortran'].headers.cpp_flags
'-I' + join_path(
spec['elpa'].prefix,
'include',
'elpa-{0}'.format(str(spec['elpa'].version)),
'modules'
),
# spec[pexsi:fortran].cppflags
# spec[pexsi:fortran].headers.cpp_flags
'-I' + join_path(spec['pexsi'].prefix, 'fortran')
])
scalapack = spec['scalapack'].libs

4
var/spack/repos/builtin/packages/foam-extend/package.py
View File

@@ -356,8 +356,8 @@ def configure(self, spec, prefix):
'CMAKE_BIN_DIR': spec['cmake'].prefix.bin,
},
'python': {
'PYTHON_DIR': spec['python'].prefix,
'PYTHON_BIN_DIR': spec['python'].prefix.bin,
'PYTHON_DIR': spec['python'].home,
'PYTHON_BIN_DIR': spec['python'].home.bin,
},
'flex': {
'FLEX_SYSTEM': 1,

2
var/spack/repos/builtin/packages/gcc/package.py
View File

@@ -176,7 +176,7 @@ def write_rpath_specs(self):
self.spec.format('$_$@'))
return
gcc = Executable(join_path(self.prefix.bin, 'gcc'))
gcc = self.spec['gcc'].command
lines = gcc('-dumpspecs', output=str).strip().split("\n")
specs_file = join_path(self.spec_dir, 'specs')
with closing(open(specs_file, 'w')) as out:

2
var/spack/repos/builtin/packages/gdal/package.py
View File

@@ -72,7 +72,7 @@ def install(self, spec, prefix):
args.append("--prefix=%s" % prefix)
args.append("--with-liblzma=yes")
args.append("--with-zlib=%s" % spec['zlib'].prefix)
args.append("--with-python=%s" % spec['python'].prefix.bin + "/python")
args.append("--with-python=%s" % spec['python'].command.path)
args.append("--without-libtool")
if '+geos' in spec:

5
var/spack/repos/builtin/packages/geos/package.py
View File

@@ -60,9 +60,8 @@ class Geos(Package):
def install(self, spec, prefix):
args = ["--prefix=%s" % prefix]
# if '+python' in spec:
# os.environ['PYTHON'] = join_path(spec['python'].prefix, 'bin',
# 'python' if spec['python'].version[:1][0] <= 2 else 'python3')
# os.environ['SWIG'] = join_path(spec['swig'].prefix, 'bin', 'swig')
# os.environ['PYTHON'] = spec['python'].command.path
# os.environ['SWIG'] = spec['swig'].command.path
#
# args.append("--enable-python")

3
var/spack/repos/builtin/packages/git/package.py
View File

@@ -162,8 +162,7 @@ def configure_args(self):
'--with-iconv={0}'.format(spec['libiconv'].prefix),
'--with-libpcre={0}'.format(spec['pcre'].prefix),
'--with-openssl={0}'.format(spec['openssl'].prefix),
'--with-perl={0}'.format(
join_path(spec['perl'].prefix.bin, 'perl')),
'--with-perl={0}'.format(spec['perl'].command.path),
'--with-zlib={0}'.format(spec['zlib'].prefix),
]

2
var/spack/repos/builtin/packages/go/package.py
View File

@@ -117,7 +117,7 @@ def setup_dependent_package(self, module, dependent_spec):
shutil.copytree('bin', os.path.join(prefix, '/bin'))
"""
# Add a go command/compiler for extensions
module.go = Executable(join_path(self.spec.prefix.bin, 'go'))
module.go = self.spec['go'].command
def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
if os.environ.get('GOROOT', False):

2
var/spack/repos/builtin/packages/hdf5-blosc/package.py
View File

@@ -69,7 +69,7 @@ def install(self, spec, prefix):
# if sys.platform == "darwin":
# fix_darwin_install_name(prefix.lib)
libtool = Executable(join_path(spec["libtool"].prefix.bin, "libtool"))
libtool = spec["libtool"].command
# TODO: these vars are not used.
# if "+mpi" in spec["hdf5"]:

2
var/spack/repos/builtin/packages/hdf5/package.py
View File

@@ -257,7 +257,7 @@ def check_install(self):
cc = Executable(spec['mpi'].mpicc)
else:
cc = Executable(self.compiler.cc)
cc(*(['-c', "check.c"] + spec['hdf5'].cppflags.split()))
cc(*(['-c', "check.c"] + spec['hdf5'].headers.cpp_flags.split()))
cc(*(['-o', "check", "check.o"] +
spec['hdf5'].libs.ld_flags.split()))
try:

2
var/spack/repos/builtin/packages/hoomd-blue/package.py
View File

@@ -81,7 +81,7 @@ def cmake_args(self):
spec = self.spec
cmake_args = [
'-DPYTHON_EXECUTABLE={0}/python'.format(spec['python'].prefix.bin),
'-DPYTHON_EXECUTABLE={0}'.format(spec['python'].command.path),
]
# MPI support

4
var/spack/repos/builtin/packages/julia/package.py
View File

@@ -185,7 +185,7 @@ def install(self, spec, prefix):
juliarc.write('\n')
# Install some commonly used packages
julia = Executable(join_path(prefix.bin, "julia"))
julia = spec['julia'].command
julia("-e", 'Pkg.init(); Pkg.update()')
# Install HDF5
@@ -216,7 +216,7 @@ def install(self, spec, prefix):
with open(join_path(prefix, "etc", "julia", "juliarc.jl"),
"a") as juliarc:
juliarc.write('# Python\n')
juliarc.write('ENV["PYTHON"] = "%s"\n' % spec["python"].prefix)
juliarc.write('ENV["PYTHON"] = "%s"\n' % spec["python"].home)
juliarc.write('\n')
# Python's OpenSSL package installer complains:
# Error: PREFIX too long: 166 characters, but only 128 allowed

2
var/spack/repos/builtin/packages/libxml2/package.py
View File

@@ -49,7 +49,7 @@ def configure_args(self):
spec = self.spec
if '+python' in spec:
python_args = [
'--with-python={0}'.format(spec['python'].prefix),
'--with-python={0}'.format(spec['python'].home),
'--with-python-install-dir={0}'.format(site_packages_dir)
]
else:

2
var/spack/repos/builtin/packages/llvm/package.py
View File

@@ -339,7 +339,7 @@ def cmake_args(self):
cmake_args = [
'-DLLVM_REQUIRES_RTTI:BOOL=ON',
'-DCLANG_DEFAULT_OPENMP_RUNTIME:STRING=libomp',
'-DPYTHON_EXECUTABLE:PATH=%s/bin/python' % spec['python'].prefix
'-DPYTHON_EXECUTABLE:PATH={0}'.format(spec['python'].command.path),
]
if '+gold' in spec:

2
var/spack/repos/builtin/packages/nwchem/package.py
View File

@@ -88,7 +88,7 @@ def install(self, spec, prefix):
'MPI_LOC=%s' % spec['mpi'].prefix,
'USE_PYTHONCONFIG=y',
'PYTHONVERSION=%s' % spec['python'].version.up_to(2),
'PYTHONHOME=%s' % spec['python'].prefix,
'PYTHONHOME=%s' % spec['python'].home,
'BLASOPT=%s' % ((lapack + blas).ld_flags),
'BLAS_LIB=%s' % blas.ld_flags,
'LAPACK_LIB=%s' % lapack.ld_flags,

2
var/spack/repos/builtin/packages/openblas/package.py
View File

@@ -150,7 +150,7 @@ def check_install(self):
blessed_file = join_path(os.path.dirname(self.module.__file__),
'test_cblas_dgemm.output')
include_flags = spec['openblas'].cppflags
include_flags = spec['openblas'].headers.cpp_flags
link_flags = spec['openblas'].libs.ld_flags
if self.compiler.name == 'intel':
link_flags += ' -lifcore'

19
var/spack/repos/builtin/packages/opencv/package.py
View File

@@ -23,7 +23,6 @@
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
##############################################################################
from spack import *
from glob import glob
class Opencv(Package):
@@ -166,22 +165,11 @@ def install(self, spec, prefix):
# Python
if '+python' in spec:
python = spec['python']
try:
python_lib = glob(join_path(
python.prefix.lib, 'libpython*.{0}'.format(dso_suffix)))[0]
except KeyError:
raise InstallError('Cannot find libpython')
try:
python_include_dir = glob(join_path(python.prefix.include,
'python*'))[0]
except KeyError:
raise InstallError('Cannot find python include directory')
python_exe = spec['python'].command.path
python_lib = spec['python'].libs[0]
python_include_dir = spec['python'].headers.directories[0]
if '^python@3:' in spec:
python_exe = join_path(python.prefix.bin, 'python3')
cmake_options.extend([
'-DBUILD_opencv_python3=ON',
'-DPYTHON3_EXECUTABLE={0}'.format(python_exe),
@@ -190,7 +178,6 @@ def install(self, spec, prefix):
'-DBUILD_opencv_python2=OFF',
])
elif '^python@2:3' in spec:
python_exe = join_path(python.prefix.bin, 'python2')
cmake_options.extend([
'-DBUILD_opencv_python2=ON',
'-DPYTHON2_EXECUTABLE={0}'.format(python_exe),

18
var/spack/repos/builtin/packages/openspeedshop/package.py
View File

@@ -167,24 +167,18 @@ def set_defaultbase_cmakeOptions(self, spec, cmakeOptions):
# Appends to cmakeOptions the options that will enable
# the appropriate base level options to the openspeedshop
# cmake build.
python_vers = format(spec['python'].version.up_to(2))
python_pv = '/python' + python_vers
python_pvs = '/libpython' + python_vers + '.' + format(dso_suffix)
python_exe = spec['python'].command.path
python_library = spec['python'].libs[0]
python_include = spec['python'].headers.directories[0]
BaseOptions = []
BaseOptions.append('-DBINUTILS_DIR=%s' % spec['binutils'].prefix)
BaseOptions.append('-DLIBELF_DIR=%s' % spec['libelf'].prefix)
BaseOptions.append('-DLIBDWARF_DIR=%s' % spec['libdwarf'].prefix)
BaseOptions.append(
'-DPYTHON_EXECUTABLE=%s'
% join_path(spec['python'].prefix + '/bin/python'))
BaseOptions.append(
'-DPYTHON_INCLUDE_DIR=%s'
% join_path(spec['python'].prefix.include) + python_pv)
BaseOptions.append(
'-DPYTHON_LIBRARY=%s'
% join_path(spec['python'].prefix.lib) + python_pvs)
BaseOptions.append('-DPYTHON_EXECUTABLE=%s' % python_exe)
BaseOptions.append('-DPYTHON_INCLUDE_DIR=%s' % python_include)
BaseOptions.append('-DPYTHON_LIBRARY=%s' % python_library)
BaseOptions.append('-DBoost_NO_SYSTEM_PATHS=TRUE')
BaseOptions.append('-DBoost_NO_BOOST_CMAKE=TRUE')
BaseOptions.append('-DBOOST_ROOT=%s' % spec['boost'].prefix)

2
var/spack/repos/builtin/packages/pagmo/package.py
View File

@@ -110,7 +110,7 @@ def cmake_args(self):
if '+python' in spec:
args.extend([
# By default picks up the system python not the Spack build
'-DPYTHON_EXECUTABLE={0}'.format(python_exe),
'-DPYTHON_EXECUTABLE={0}'.format(spec['python'].command.path),
# By default installs to the python prefix not the pagmo prefix
'-DPYTHON_MODULES_DIR={0}'.format(site_packages_dir),
])

3
var/spack/repos/builtin/packages/paraview/package.py
View File

@@ -120,8 +120,7 @@ def nvariant_bool(feature):
if '+python' in spec:
cmake_args.extend([
'-DPARAVIEW_ENABLE_PYTHON:BOOL=ON',
'-DPYTHON_EXECUTABLE:FILEPATH=%s/bin/python'
% spec['python'].prefix
'-DPYTHON_EXECUTABLE:FILEPATH=%s' % spec['python'].command.path
])
if '+mpi' in spec:

3
var/spack/repos/builtin/packages/perl/package.py
View File

@@ -113,11 +113,10 @@ def install(self, spec, prefix):
@run_after('install')
def install_cpanm(self):
spec = self.spec
prefix = self.prefix
if '+cpanm' in spec:
with working_dir(join_path('cpanm', 'cpanm')):
perl = Executable(join_path(prefix.bin, 'perl'))
perl = spec['perl'].command
perl('Makefile.PL')
make()
make('install')

2
var/spack/repos/builtin/packages/plumed/package.py
View File

@@ -122,7 +122,7 @@ def apply_patch(self, other):
def setup_dependent_package(self, module, dependent_spec):
# Make plumed visible from dependent packages
module.plumed = Executable(join_path(self.spec.prefix.bin, 'plumed'))
module.plumed = self.spec['plumed'].command
@run_before('autoreconf')
def filter_gslcblas(self):

2
var/spack/repos/builtin/packages/pocl/package.py
View File

@@ -104,7 +104,7 @@ def check_install(self):
with working_dir(checkdir, create=True):
source = join_path(os.path.dirname(self.module.__file__),
"example1.c")
cflags = spec["pocl"].cppflags.split()
cflags = spec["pocl"].headers.cpp_flags.split()
# ldflags = spec["pocl"].libs.ld_flags.split()
ldflags = ["-L%s" % spec["pocl"].prefix.lib,
"-lOpenCL", "-lpoclu"]

3
var/spack/repos/builtin/packages/psi4/package.py
View File

@@ -115,8 +115,7 @@ def filter_compilers(self, spec, prefix):
' -I'.join([
os.path.join(spec['psi4'].prefix.include, 'psi4'),
os.path.join(spec['boost'].prefix.include, 'boost'),
os.path.join(spec['python'].prefix.include, 'python{0}'.format(
spec['python'].version.up_to(2))),
os.path.join(spec['python'].headers.directories[0]),
spec['lapack'].prefix.include,
spec['blas'].prefix.include,
'/usr/include'

258
var/spack/repos/builtin/packages/python/package.py
View File

@@ -24,7 +24,9 @@
##############################################################################
import ast
import os
import platform
import re
import sys
from contextlib import closing
import spack
@@ -33,10 +35,11 @@
from llnl.util.filesystem import force_remove
from spack import *
from spack.util.environment import *
from spack.util.prefix import Prefix
import spack.util.spack_json as sjson
class Python(Package):
class Python(AutotoolsPackage):
"""The Python programming language."""
homepage = "http://www.python.org"
@@ -62,6 +65,10 @@ class Python(Package):
extendable = True
# --enable-shared is known to cause problems for some users on macOS
# See http://bugs.python.org/issue29846
variant('shared', default=sys.platform != 'darwin',
description='Enable shared libraries')
variant('tk', default=False, description='Provide support for Tkinter')
variant('ucs4', default=False,
description='Enable UCS4 (wide) unicode strings')
@@ -82,13 +89,14 @@ class Python(Package):
depends_on("tk", when="+tk")
depends_on("tcl", when="+tk")
patch('ncurses.patch')
# Patch does not work for Python 3.1
patch('ncurses.patch', when='@:2.8,3.2:')
_DISTUTIL_VARS_TO_SAVE = ['LDSHARED']
_DISTUTIL_CACHE_FILENAME = 'sysconfig.json'
_distutil_vars = None
@when('@2.7,3.4:')
@when('@2.7:2.8,3.4:')
def patch(self):
# NOTE: Python's default installation procedure makes it possible for a
# user's local configurations to change the Spack installation. In
@@ -100,7 +108,10 @@ def patch(self):
r'\1setup.py\2 --no-user-cfg \3\6'
)
def install(self, spec, prefix):
def setup_environment(self, spack_env, run_env):
spec = self.spec
prefix = self.prefix
# TODO: The '--no-user-cfg' option for Python installation is only in
# Python v2.7 and v3.4+ (see https://bugs.python.org/issue1180) and
# adding support for ignoring user configuration will require
@@ -110,23 +121,27 @@ def install(self, spec, prefix):
'user configurations are present.').format(self.version))
# Need this to allow python build to find the Python installation.
env['PYTHONHOME'], env['PYTHONPATH'] = prefix, prefix
env['MACOSX_DEPLOYMENT_TARGET'] = '10.6'
spack_env.set('PYTHONHOME', prefix)
spack_env.set('PYTHONPATH', prefix)
spack_env.set('MACOSX_DEPLOYMENT_TARGET', platform.mac_ver()[0])
# Rest of install is pretty standard except setup.py needs to
# be able to read the CPPFLAGS and LDFLAGS as it scans for the
# library and headers to build
def configure_args(self):
spec = self.spec
# setup.py needs to be able to read the CPPFLAGS and LDFLAGS
# as it scans for the library and headers to build
dep_pfxs = [dspec.prefix for dspec in spec.dependencies('link')]
config_args = [
'--prefix={0}'.format(prefix),
'--with-threads',
'--enable-shared',
'CPPFLAGS=-I{0}'.format(' -I'.join(dp.include for dp in dep_pfxs)),
'LDFLAGS=-L{0}'.format(' -L'.join(dp.lib for dp in dep_pfxs)),
]
if spec.satisfies("platform=darwin") and ('%gcc' in spec):
if spec.satisfies('%gcc platform=darwin'):
config_args.append('--disable-toolbox-glue')
if '+shared' in spec:
config_args.append('--enable-shared')
if '+ucs4' in spec:
if spec.satisfies('@:2.7'):
config_args.append('--enable-unicode=ucs4')
@@ -140,18 +155,18 @@ def install(self, spec, prefix):
if spec.satisfies('@3:'):
config_args.append('--without-ensurepip')
configure(*config_args)
make()
make('install')
return config_args
@run_after('install')
def post_install(self):
spec = self.spec
prefix = self.prefix
self.sysconfigfilename = '_sysconfigdata.py'
if spec.satisfies('@3.6:'):
# Python 3.6.0 renamed the sys config file
python3 = os.path.join(prefix.bin,
'python{0}'.format(self.version.up_to(1)))
python = Executable(python3)
sc = 'import sysconfig; print(sysconfig._get_sysconfigdata_name())'
cf = python('-c', sc, output=str).strip('\n')
cf = self.command('-c', sc, output=str).strip()
self.sysconfigfilename = '{0}.py'.format(cf)
self._save_distutil_vars(prefix)
@@ -205,7 +220,7 @@ def _save_distutil_vars(self, prefix):
Spack partially covers this by setting environment variables that
are also accounted for by distutils. Currently there is one more known
variable that must be set, which is LDSHARED, so the method saves its
autogenerated value to pass it to the dependant package's setup script.
autogenerated value to pass it to the dependent package's setup script.
"""
self._distutil_vars = {}
@@ -234,20 +249,20 @@ def _save_distutil_vars(self, prefix):
pass
if not input_dict:
tty.warn('Failed to find \'build_time_vars\' dictionary in file '
'\'%s\'. This might cause the extensions that are '
'installed with distutils to call compilers directly '
'avoiding Spack\'s wrappers.' % input_filename)
tty.warn("Failed to find 'build_time_vars' dictionary in file "
"'%s'. This might cause the extensions that are "
"installed with distutils to call compilers directly "
"avoiding Spack's wrappers." % input_filename)
return
for var_name in Python._DISTUTIL_VARS_TO_SAVE:
if var_name in input_dict:
self._distutil_vars[var_name] = input_dict[var_name]
else:
tty.warn('Failed to find key \'%s\' in \'build_time_vars\' '
'dictionary in file \'%s\'. This might cause the '
'extensions that are installed with distutils to '
'call compilers directly avoiding Spack\'s wrappers.'
tty.warn("Failed to find key '%s' in 'build_time_vars' "
"dictionary in file '%s'. This might cause the "
"extensions that are installed with distutils to "
"call compilers directly avoiding Spack's wrappers."
% (var_name, input_filename))
if len(self._distutil_vars) > 0:
@@ -259,10 +274,10 @@ def _save_distutil_vars(self, prefix):
with open(output_filename, 'w') as output_file:
sjson.dump(self._distutil_vars, output_file)
except:
tty.warn('Failed to save metadata for distutils. This might '
'cause the extensions that are installed with '
'distutils to call compilers directly avoiding '
'Spack\'s wrappers.')
tty.warn("Failed to save metadata for distutils. This might "
"cause the extensions that are installed with "
"distutils to call compilers directly avoiding "
"Spack's wrappers.")
# We make the cache empty if we failed to save it to file
# to provide the same behaviour as in the case when the cache
# is initialized by the method load_distutils_data().
@@ -319,6 +334,144 @@ def filter_compilers(self, prefix):
# Set up environment to make install easy for python extensions.
# ========================================================================
@property
def command(self):
"""Returns the Python command, which may vary depending
on the version of Python and how it was installed.
In general, Python 2 comes with ``python`` and ``python2`` commands,
while Python 3 only comes with a ``python3`` command.
:returns: The Python command
:rtype: Executable
"""
# We need to be careful here. If the user is using an externally
# installed python, all 3 commands could be in the same directory.
# Search for `python2` iff using Python 2
if (self.spec.satisfies('@:2') and
os.path.exists(os.path.join(self.prefix.bin, 'python2'))):
command = 'python2'
# Search for `python3` iff using Python 3
elif (self.spec.satisfies('@3:') and
os.path.exists(os.path.join(self.prefix.bin, 'python3'))):
command = 'python3'
# If neither were found, try `python`
elif os.path.exists(os.path.join(self.prefix.bin, 'python')):
command = 'python'
else:
msg = 'Unable to locate {0} command in {1}'
raise RuntimeError(msg.format(self.name, self.prefix.bin))
# The python command may be a symlink if it was installed
# with Homebrew. Since some packages try to determine the
# location of libraries and headers based on the path,
# return the realpath
path = os.path.realpath(os.path.join(self.prefix.bin, command))
return Executable(path)
def print_string(self, string):
"""Returns the appropriate print string depending on the
version of Python.
Examples:
* Python 2
.. code-block:: python
>>> self.print_string('sys.prefix')
'print sys.prefix'
* Python 3
.. code-block:: python
>>> self.print_string('sys.prefix')
'print(sys.prefix)'
"""
if self.spec.satisfies('@:2'):
return 'print {0}'.format(string)
else:
return 'print({0})'.format(string)
def get_config_var(self, key):
"""Returns the value of a single variable. Wrapper around
``distutils.sysconfig.get_config_var()``."""
cmd = 'from distutils.sysconfig import get_config_var; '
cmd += self.print_string("get_config_var('{0}')".format(key))
return self.command('-c', cmd, output=str).strip()
def get_config_h_filename(self):
"""Returns the full path name of the configuration header.
Wrapper around ``distutils.sysconfig.get_config_h_filename()``."""
cmd = 'from distutils.sysconfig import get_config_h_filename; '
cmd += self.print_string('get_config_h_filename()')
return self.command('-c', cmd, output=str).strip()
@property
def home(self):
"""Most of the time, ``PYTHONHOME`` is simply
``spec['python'].prefix``. However, if the user is using an
externally installed python, it may be symlinked. For example,
Homebrew installs python in ``/usr/local/Cellar/python/2.7.12_2``
and symlinks it to ``/usr/local``. Users may not know the actual
installation directory and add ``/usr/local`` to their
``packages.yaml`` unknowingly. Query the python executable to
determine exactly where it is installed."""
prefix = self.get_config_var('prefix')
return Prefix(prefix)
@property
def libs(self):
# Spack installs libraries into lib, except on openSUSE where it
# installs them into lib64. If the user is using an externally
# installed package, it may be in either lib or lib64, so we need
# to ask Python where its LIBDIR is.
libdir = self.get_config_var('LIBDIR')
# The system Python installation on macOS and Homebrew installations
# install libraries into a Frameworks directory
frameworkprefix = self.get_config_var('PYTHONFRAMEWORKPREFIX')
if '+shared' in self.spec:
ldlibrary = self.get_config_var('LDLIBRARY')
if os.path.exists(os.path.join(libdir, ldlibrary)):
return LibraryList(os.path.join(libdir, ldlibrary))
elif os.path.exists(os.path.join(frameworkprefix, ldlibrary)):
return LibraryList(os.path.join(frameworkprefix, ldlibrary))
else:
msg = 'Unable to locate {0} libraries in {1}'
raise RuntimeError(msg.format(self.name, libdir))
else:
library = self.get_config_var('LIBRARY')
if os.path.exists(os.path.join(libdir, library)):
return LibraryList(os.path.join(libdir, library))
elif os.path.exists(os.path.join(frameworkprefix, library)):
return LibraryList(os.path.join(frameworkprefix, library))
else:
msg = 'Unable to locate {0} libraries in {1}'
raise RuntimeError(msg.format(self.name, libdir))
@property
def headers(self):
config_h = self.get_config_h_filename()
if os.path.exists(config_h):
return HeaderList(config_h)
else:
includepy = self.get_config_var('INCLUDEPY')
msg = 'Unable to locate {0} headers in {1}'
raise RuntimeError(msg.format(self.name, includepy))
@property
def python_lib_dir(self):
return join_path('lib', 'python{0}'.format(self.version.up_to(2)))
@@ -332,29 +485,10 @@ def site_packages_dir(self):
return join_path(self.python_lib_dir, 'site-packages')
def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
"""Set PYTHONPATH to include site-packages dir for the
"""Set PYTHONPATH to include the site-packages directory for the
extension and any other python extensions it depends on."""
# The python executable for version 3 may be python3 or python
# See https://github.com/LLNL/spack/pull/2173#issuecomment-257170199
pythonex = 'python{0}'.format('3' if self.spec.satisfies('@3') else '')
if os.path.isdir(self.prefix.bin):
base = self.prefix.bin
else:
base = self.prefix
if not os.path.isfile(os.path.join(base, pythonex)):
if self.spec.satisfies('@3'):
python = Executable(os.path.join(base, 'python'))
version = python('-c', 'import sys; print(sys.version)',
output=str)
if version.startswith('3'):
pythonex = 'python'
else:
raise RuntimeError('Cannot locate python executable')
else:
raise RuntimeError('Cannot locate python executable')
python = Executable(os.path.join(base, pythonex))
prefix = python('-c', 'import sys; print(sys.prefix)', output=str)
spack_env.set('PYTHONHOME', prefix.strip('\n'))
spack_env.set('PYTHONHOME', self.home)
python_paths = []
for d in dependent_spec.traverse(
@@ -378,19 +512,15 @@ def setup_dependent_package(self, module, dependent_spec):
In most cases, extensions will only need to have one line::
setup_py('install', '--prefix={0}'.format(prefix))"""
python_path = join_path(
self.spec.prefix.bin,
'python{0}'.format('3' if self.spec.satisfies('@3') else '')
)
module.python_exe = python_path
module.python = Executable(python_path)
module.setup_py = Executable(python_path + ' setup.py --no-user-cfg')
module.python = self.command
module.setup_py = Executable(
self.command.path + ' setup.py --no-user-cfg')
distutil_vars = self._load_distutil_vars()
if distutil_vars:
for key, value in distutil_vars.iteritems():
for key, value in distutil_vars.items():
module.setup_py.add_default_env(key, value)
# Add variables for lib/pythonX.Y and lib/pythonX.Y/site-packages dirs.
@@ -429,7 +559,7 @@ def python_ignore(self, ext_pkg, args):
if ext_pkg.name != 'py-pygments':
patterns.append(r'bin/pygmentize$')
if ext_pkg.name != 'py-numpy':
patterns.append(r'bin/f2py3?$')
patterns.append(r'bin/f2py[0-9.]*$')
return match_predicate(ignore_arg, patterns)
@@ -457,7 +587,7 @@ def write_easy_install_pth(self, exts):
paths.append(line)
site_packages = join_path(self.prefix, self.site_packages_dir)
site_packages = join_path(self.home, self.site_packages_dir)
main_pth = join_path(site_packages, "easy-install.pth")
if not paths:

5
var/spack/repos/builtin/packages/scotch/package.py
View File

@@ -22,7 +22,6 @@
# License along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
##############################################################################
import os
from spack import *
@@ -165,8 +164,8 @@ def configure(self):
# General Features #
flex_path = os.path.join(self.spec['flex'].prefix.bin, 'flex')
bison_path = os.path.join(self.spec['bison'].prefix.bin, 'bison')
flex_path = self.spec['flex'].command.path
bison_path = self.spec['bison'].command.path
makefile_inc.extend([
'EXE =',
'OBJ = .o',

4
var/spack/repos/builtin/packages/shiny-server/package.py
View File

@@ -54,9 +54,7 @@ def cmake_args(self):
spec = self.spec
options = []
options.extend([
"-DPYTHON=%s" % join_path(spec['python'].prefix.bin, 'python'),
])
options.append("-DPYTHON=%s" % spec['python'].command.path)
return options

5
var/spack/repos/builtin/packages/spark/package.py
View File

@@ -69,8 +69,7 @@ def setup_environment(self, spack_env, run_env):
env['JAVA_HOME'] = self.spec['jdk'].prefix
# spack_env.set('JAVA_HOME', self.spec['jdk'].prefix)
hadoop_bin_path = join_path(self.spec['hadoop'].prefix.bin, 'hadoop')
hadoop_bin = Executable(hadoop_bin_path)
hadoop_classpath = hadoop_bin('classpath', return_output=True)
hadoop = self.spec['hadoop'].command
hadoop_classpath = hadoop('classpath', return_output=True)
run_env.set('SPARK_DIST_CLASSPATH', hadoop_classpath)

2
var/spack/repos/builtin/packages/stat/package.py
View File

@@ -69,7 +69,7 @@ def configure_args(self):
"--with-graphlib=%s" % spec['graphlib'].prefix,
"--with-stackwalker=%s" % spec['dyninst'].prefix,
"--with-libdwarf=%s" % spec['libdwarf'].prefix,
"--with-python=%s/bin/python" % spec['python'].prefix
"--with-python=%s" % spec['python'].command.path,
]
if '+dysect' in spec:
args.append('--enable-dysectapi')

3
var/spack/repos/builtin/packages/subversion/package.py
View File

@@ -72,8 +72,7 @@ def install(self, spec, prefix):
if 'swig' in spec:
options.append('--with-swig=%s' % spec['swig'].prefix)
if 'perl' in spec:
options.append(
'PERL=%s' % join_path(spec['perl'].prefix.bin, 'perl'))
options.append('PERL=%s' % spec['perl'].command.path)
configure(*options)
make()

4
var/spack/repos/builtin/packages/superlu-dist/package.py
View File

@@ -72,8 +72,8 @@ def install(self, spec, prefix):
'RANLIB = true',
'CC = {0}'.format(self.spec['mpi'].mpicc),
'CFLAGS = -fPIC -std=c99 -O2 %s %s %s' % (
spec['parmetis'].cppflags,
spec['metis'].cppflags,
spec['parmetis'].headers.cpp_flags,
spec['metis'].headers.cpp_flags,
'-D_LONGINT' if '+int64' in spec else ''),
'NOOPTS = -fPIC -std=c99',
'FORTRAN = {0}'.format(self.spec['mpi'].mpif77),

2
var/spack/repos/builtin/packages/visit/package.py
View File

@@ -52,7 +52,7 @@ def install(self, spec, prefix):
'-DVTK_MINOR_VERSION=1',
'-DVISIT_USE_GLEW=OFF',
'-DVISIT_LOC_QMAKE_EXE:FILEPATH={0}/qmake-qt4'.format(qt_bin),
'-DPYTHON_DIR:PATH={0}'.format(spec['python'].prefix),
'-DPYTHON_DIR:PATH={0}'.format(spec['python'].home),
'-DVISIT_SILO_DIR:PATH={0}'.format(spec['silo'].prefix),
'-DVISIT_HDF5_DIR:PATH={0}'.format(spec['hdf5'].prefix),
'-DVISIT_VTK_DIR:PATH={0}'.format(spec['vtk'].prefix),