A build of python@3.X had the following in the logs:
```
The necessary bits to build these optional modules were not found:
_dbm _gdbm _tkinter
```
As Tkinter is already a variant, we adopt the same strategy for dbm.
fixes#7128
Before this PR packages that were indirectly dependent on python might
have failed due to inconsistency between the python found in the
environment and the standard libraries set in PYTHONHOME
See #6794
This fixes cases where test-only dependencies were omitted from
consideration when modifying the environment at build time. This
includes an update to the python package definition to add
testing-related python extensions to its specialized environment
setup.
* python: ensure that distutils sets valid compiler options for RPATH on cray
* python: add cray-rpath patches for python@2.3:3.0.1 and python@3.1:
* python: in patch(..., when="@a:b platform=c"), limit b must be specified
* python: assume that python@4 will break the cray patches
## Motivation
Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically.
I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.
Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends.
## Prefix
For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`.
Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.
To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable.
## Command
In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:
If the spec is for Python 3, try searching for the `python3` command.
If the spec is for Python 2, try searching for the `python2` command.
If neither are found, try searching for the `python` command.
## Libraries
Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work.
The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:
```
lib/libpython2.7.dylib
```
For Python 3.6 on CentOS 6, I'm seeing:
```
lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0
```
Notice the `m` after the version number. Yeah, that's a thing.
## Headers
In Python 2.7, I'm seeing:
```
include/python2.7/pyconfig.h
```
In Python 3.6, I'm seeing:
```
include/python3.6m/pyconfig.h
```
It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6
Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
- _spider in web.py was actually failing to spider deeper than a certain
point.
- Fixed multiprocessing pools to not use daemons and to allow recursive
spawning.
- Added detailed tests for spidering and for finding archive versions.
- left some xfail URL finding exercises for the reader.
- Fix noqa annotations for some @when decorators
* Add new package for PaGMO/PyGMO
* Rename py-pygmo to pagmo
* Fix concretization bug
* Correct Python detection, use correct flags
* Install PyGMO to the pagmo prefix, not the Python prefix
- Add a PythonPackage class with build system support.
- Support build phases in PythonPackage
- Add a custom sanity check for PythonPackages
- Get rid of nolink dependencies in python packages
- Update spack create to use new PythonPackage class
- Port most of Python packages to new PythonPackage class
- Conducted a massive install and activate of Python packages.
- Fixed bugs introduced by install and activate.
- Update API docs on PythonPackage
* Removing the nobuild, nolink, and alldeps dependency types in favor of being explicit.
* This will help with maintenance going forward, as adding more dependency types won't affect existing declared dependencies in weird ways.
* default deptype is still `('build', 'link')`
On MacOS, brew installs /usr/local/bin/python but the Python prefix is not /usr/local/bin
Use the python command sys.exec to get the correct directory, instead of the ad hoc self.prefix
previously used
This was a bear to debug; been driving me nuts since I started using spack.
Since spack passes PYTHONHOME down to package builds in the environment
it was passing PYTHONHOME of /usr/local/bin to the PETSc build that uses Python so
the PETSc Python ./configure errored immediately with
ImportError: No module named site
since python could find no python modules. Todd Gamblin pointed out that my first try to fix
this was wrong since it assumed the spack python was the same python used to run spack. Elizabeth Fischer
suggested how to get it to work also with python3
Funded-by: IDEAS
Project: IDEAS/xSDK
Time: 7 hours
Thanks-to: Todd Gamblin, Elizabeth Fischer
Python will look to link with libncursesw in preference to libncurses. Since
ncurses in spack is built without suffixes there is no libncursesw and
python will link to the system libncursesw for _curses.so and
_curses_panel.so, as well as libpanelw for _curses_panel.so.
This PR introduces a patch that simple removes the check for ncursesw in
setup.py and therefore sets `curses_library` to `ncurses`.
There are many python packages that depend on py-numpy. Each one of
those will have a copy of f2py that will need to be put in the ignore
list for activation. Thise PR adds f2py to the ignore list in the python
package.py file so that it does not have to be done for each package
that depends on py-numpy. This follows the model of what is done for
py-setuptools.
This PR also removes the f2py ignore expression for python packages
that were using it as it is no longer needed in the individual packages.
- Failed to catch all instances of modify_module when it was renamed
to setup_dependent_package.
- Refactored remaining modify_module calls.
- Also modified Python's setup_dependent_package slightly: only
creates empty site-packages directory for Python extensions now, not
for all dependents.
- Gave setup_environment and setup_dependent_environment more similar
signatures. They now allows editing the Spack env and the runtime
env for *this* package and dependents, respectively.
- modify_module renamed to setup_dependent_python_module for symmetry
with setup_dependent_environment and to avoid confusion with
environment modules.
- removed need for patching Package objects at runtime.
- adjust packages to reflect these changes.
- This moves var/spack/packages to var/spack/repos/builtin/packages.
- Packages that did not exist in the source branch, or were changed in
develop, were moved into var/spack/repos/builtin/packages as part of
the integration.
Conflicts:
lib/spack/spack/test/unit_install.py
var/spack/repos/builtin/packages/clang/package.py
Package repositories now look like this:
top-level-dir/
repo.yaml
packages/
libelf/
package.py
mpich/
package.py
...
This leaves room at the top level for additional metadata, source,
per-repo configs, indexes, etc., and it makes it easy to see that
something is a spack repo (just look for repo.yaml and packages).
