* Update the krell institute products to use the latest features of spack for building on cluster platforms.
* Address travis error messages and resubmit the pull request.
* Update the contents of openspeedshop package.py so it passes the flake8 tests.
* Fix flake8 error-whitespack issue in mrnet package.py file.
* Add updates based on spack reviewer feedback.
* More fixes based on comments from reviewers. Switch using extend to using append, remove additional setting of PATH and LD_LIBRARY_PATH that should not be required due to RPATH.
* More review related changes. Update MPIOption.append lines and take out xercesc references.
* Create a base options function for common openspeedshop base cmake options to reduce redundencies.
* Add libxml2+python depends on to get around issues with the libxml2 package file.
* Using boost over 1.60.0 causes compile errors. This is a known boost bug. Also, dyninst-9.2.0 is set to be the vesrion of dyninst to use with OSS, as of now. The newer version fails to build.
* Fix bad syntax in specifying the boost version range.
* Update the version numbers for the krell institute components and tools: cbtf and openspeedshop.
* Do not build glib for qt3, it is not needed and causes build problems at this time anyway.
* A fix was added for setting LD_LIBRARY_PATH in the qt3 build, but if LD_LIBRARY_PATH is not set the qt build fails. So so check and set LD_LIBRARY_PATH if not set, update if it is set.
* Update the fix for qt3 build by setting LD_LIBRARY_PATH instead of checking for whether it is set or not per Adams comment that spack clears LD_LIBRARY_PATH.
* A fix was added for setting LD_LIBRARY_PATH in the qt3 build, but if LD_LIBRARY_PATH is not set the qt build fails. So so check and set LD_LIBRARY_PATH if not set, update if it is set.
* Trim comments to fit more concisely.
* Fix tabs versus spaces and swap if and else clause check from a negative to a positive check.
* Fix issues with the cbtf-argonavis build, update to use dyninst-9.3.2, fixes to openspeedshop package build.
* Fix issues with the cbtf-argonavis package.py files related to comments.
* Add changes for changing the krell packages from Package to CMakePackage.
* Add better changes for changing the krell packages from Package to CMakePackage.
* Add more modifications for changing the krell packages from Package to CMakePackage.
* Add additional modifications for changing the krell packages from Package to CMakePackage and fixing Travis erros
* Fix new travis errors.
* Fix new travis errors.
* Add more changes for PR 4765.
* Add more refinements to the conversion from Package to CMakePackage.
* Fix new travis errors.
* Add dependencies for MPI to be passed to cbtf-krell, so it can build the MPI collectors requested by the builder of openspeedshop.
* Remove extra unnecessary routine to adjust build arguments. Fix if-else clause issue.
* Fix more flake issues caused by last changes.
* Fix a bug where openspeedshop will not build when no mpi variants are specified. Also switch to a multiple level variant for building the gui(s). Use none, qt3, and qt4 as the variants with qt3 being the default.
* Add fix for spack issue #4843, where LTDL include files were not found.
* Add the build_type variant back into the openspeedshop package file.
* Add universal build_type variant to CMakePackage
* Override build_type in some packages with different possible values
* Remove reference to no longer existent debug variant
* Update CBTF packages with new build_type variant
* Keep note on build size of LLVM
* Update the krell institute products to use the latest features of spack for building on cluster platforms.
* Address travis error messages and resubmit the pull request.
* Update the contents of openspeedshop package.py so it passes the flake8 tests.
* Fix flake8 error-whitespack issue in mrnet package.py file.
* Add updates based on spack reviewer feedback.
* More fixes based on comments from reviewers. Switch using extend to using append, remove additional setting of PATH and LD_LIBRARY_PATH that should not be required due to RPATH.
* More review related changes. Update MPIOption.append lines and take out xercesc references.
* Create a base options function for common openspeedshop base cmake options to reduce redundencies.
* Add libxml2+python depends on to get around issues with the libxml2 package file.
* Using boost over 1.60.0 causes compile errors. This is a known boost bug. Also, dyninst-9.2.0 is set to be the vesrion of dyninst to use with OSS, as of now. The newer version fails to build.
* Fix bad syntax in specifying the boost version range.
* Update the version numbers for the krell institute components and tools: cbtf and openspeedshop.
* Do not build glib for qt3, it is not needed and causes build problems at this time anyway.
* A fix was added for setting LD_LIBRARY_PATH in the qt3 build, but if LD_LIBRARY_PATH is not set the qt build fails. So so check and set LD_LIBRARY_PATH if not set, update if it is set.
* Update the fix for qt3 build by setting LD_LIBRARY_PATH instead of checking for whether it is set or not per Adams comment that spack clears LD_LIBRARY_PATH.
* A fix was added for setting LD_LIBRARY_PATH in the qt3 build, but if LD_LIBRARY_PATH is not set the qt build fails. So so check and set LD_LIBRARY_PATH if not set, update if it is set.
* Trim comments to fit more concisely.
* Fix tabs versus spaces and swap if and else clause check from a negative to a positive check.
* Fix issues with the cbtf-argonavis build, update to use dyninst-9.3.2, fixes to openspeedshop package build.
* Fix issues with the cbtf-argonavis package.py files related to comments.
* Add changes for changing the krell packages from Package to CMakePackage.
* Add better changes for changing the krell packages from Package to CMakePackage.
* Add more modifications for changing the krell packages from Package to CMakePackage.
* Add additional modifications for changing the krell packages from Package to CMakePackage and fixing Travis erros
* Fix new travis errors.
* Fix new travis errors.
* Add more changes for PR 4765.
* Add more refinements to the conversion from Package to CMakePackage.
* Fix new travis errors.
* Add dependencies for MPI to be passed to cbtf-krell, so it can build the MPI collectors requested by the builder of openspeedshop.
* Remove extra unnecessary routine to adjust build arguments. Fix if-else clause issue.
* Fix more flake issues caused by last changes.
## Motivation
Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically.
I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.
Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends.
## Prefix
For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`.
Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.
To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable.
## Command
In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:
If the spec is for Python 3, try searching for the `python3` command.
If the spec is for Python 2, try searching for the `python2` command.
If neither are found, try searching for the `python` command.
## Libraries
Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work.
The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:
```
lib/libpython2.7.dylib
```
For Python 3.6 on CentOS 6, I'm seeing:
```
lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0
```
Notice the `m` after the version number. Yeah, that's a thing.
## Headers
In Python 2.7, I'm seeing:
```
include/python2.7/pyconfig.h
```
In Python 3.6, I'm seeing:
```
include/python3.6m/pyconfig.h
```
It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6
Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
* Update the krell institute products to use the latest features of spack for building on cluster platforms.
* Address travis error messages and resubmit the pull request.
* Update the contents of openspeedshop package.py so it passes the flake8 tests.
* Fix flake8 error-whitespack issue in mrnet package.py file.
* Add updates based on spack reviewer feedback.
* More fixes based on comments from reviewers. Switch using extend to using append, remove additional setting of PATH and LD_LIBRARY_PATH that should not be required due to RPATH.
* More review related changes. Update MPIOption.append lines and take out xercesc references.
* Create a base options function for common openspeedshop base cmake options to reduce redundencies.
* Add libxml2+python depends on to get around issues with the libxml2 package file.
* Using boost over 1.60.0 causes compile errors. This is a known boost bug. Also, dyninst-9.2.0 is set to be the vesrion of dyninst to use with OSS, as of now. The newer version fails to build.
* Fix bad syntax in specifying the boost version range.
* Update the version numbers for the krell institute components and tools: cbtf and openspeedshop.
* Do not build glib for qt3, it is not needed and causes build problems at this time anyway.
* Update the krell institute products to use the latest features of spack for building on cluster platforms.
* Address travis error messages and resubmit the pull request.
* Update the contents of openspeedshop package.py so it passes the flake8 tests.
* Fix flake8 error-whitespack issue in mrnet package.py file.
* Add updates based on spack reviewer feedback.
* More fixes based on comments from reviewers. Switch using extend to using append, remove additional setting of PATH and LD_LIBRARY_PATH that should not be required due to RPATH.
* More review related changes. Update MPIOption.append lines and take out xercesc references.
* Create a base options function for common openspeedshop base cmake options to reduce redundencies.
* Add libxml2+python depends on to get around issues with the libxml2 package file.
* Update the krell institute products to use the latest features of spack for building on cluster platforms.
* Address travis error messages and resubmit the pull request.
* Update the contents of openspeedshop package.py so it passes the flake8 tests.
* Fix flake8 error-whitespack issue in mrnet package.py file.
* Add updates based on spack reviewer feedback.
* More fixes based on comments from reviewers. Switch using extend to using append, remove additional setting of PATH and LD_LIBRARY_PATH that should not be required due to RPATH.
* More review related changes. Update MPIOption.append lines and take out xercesc references.
* Create a base options function for common openspeedshop base cmake options to reduce redundencies.
