A flexible package manager that supports multiple versions, configurations, platforms, and compilers.
Go to file
Adam J. Stewart ce3ab503de Python command, libraries, and headers (#3367)
## Motivation

Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically. 

I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.

Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends.

## Prefix

For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`.

Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.

To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable.

## Command

In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:

If the spec is for Python 3, try searching for the `python3` command.
If the spec is for Python 2, try searching for the `python2` command.
If neither are found, try searching for the `python` command.

## Libraries

Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work.

The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:
```
lib/libpython2.7.dylib
```
For Python 3.6 on CentOS 6, I'm seeing:
```
lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0
```
Notice the `m` after the version number. Yeah, that's a thing.

## Headers

In Python 2.7, I'm seeing:
```
include/python2.7/pyconfig.h
```
In Python 3.6, I'm seeing:
```
include/python3.6m/pyconfig.h
```
It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6

Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
2017-04-29 17:24:13 -07:00
bin Spack works with Python 3 2017-03-31 13:40:41 -07:00
etc/spack/defaults Allow users to set parallel jobs in config.yaml (#3812) 2017-04-15 08:31:00 -07:00
lib/spack Python command, libraries, and headers (#3367) 2017-04-29 17:24:13 -07:00
share/spack Separate integration tests; simplify test scripts (#4006) 2017-04-27 11:47:56 -07:00
var/spack Python command, libraries, and headers (#3367) 2017-04-29 17:24:13 -07:00
.codecov.yml qa: adjust thresholds for acceptance (#3105) 2017-02-09 08:31:57 -08:00
.coveragerc unit tests: replace nose with pytest (#2502) 2016-12-29 07:48:48 -08:00
.flake8 Properly ignore flake8 F811 redefinition errors (#3932) 2017-04-25 11:01:25 -07:00
.gitignore unit tests: replace nose with pytest (#2502) 2016-12-29 07:48:48 -08:00
.mailmap Update mail map. So many email aliases. 2016-10-19 22:47:39 -07:00
.travis.yml Separate integration tests; simplify test scripts (#4006) 2017-04-27 11:47:56 -07:00
LICENSE Correct LLNL LGPL license template for clarity. 2016-05-11 21:22:25 -07:00
pytest.ini unit tests: replace nose with pytest (#2502) 2016-12-29 07:48:48 -08:00
README.md Spack works with Python 3 2017-03-31 13:40:41 -07:00

image

Build Status codecov

Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share common installations of software on clusters with exotic architectures, using libraries that do not have a standard ABI. Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system.

Most importantly, Spack is simple. It offers a simple spec syntax so that users can specify versions and configuration options concisely. Spack is also simple for package authors: package files are written in pure Python, and specs allow package authors to write a single build script for many different builds of the same package.

See the Feature Overview for examples and highlights.

To install spack and install your first package, make sure you have Python (2 or 3). Then:

$ git clone https://github.com/llnl/spack.git
$ cd spack/bin
$ ./spack install libelf

Documentation

Full documentation for Spack is the first place to look.

We've also got a Spack 101 Tutorial, so you can learn Spack yourself, or teach users at your own site.

See also:

Get Involved!

Spack is an open source project. Questions, discussion, and contributions are welcome. Contributions can be anything from new packages to bugfixes, or even new core features.

Mailing list

If you are interested in contributing to spack, the first step is to join the mailing list. We're using a Google Group for this, and you can join it here:

Contributions

Contributing to Spack is relatively easy. Just send us a pull request. When you send your request, make develop the destination branch on the Spack repository.

Your PR must pass Spack's unit tests and documentation tests, and must be PEP 8 compliant. We enforce these guidelines with Travis CI. To run these tests locally, and for helpful tips on git, see our Contribution Guide.

Spack uses a rough approximation of the Git Flow branching model. The develop branch contains the latest contributions, and master is always tagged and points to the latest stable release.

Authors

Many thanks go to Spack's contributors.

Spack was originally written by Todd Gamblin, tgamblin@llnl.gov.

Citing Spack

If you are referencing Spack in a publication, please cite the following paper:

Release

Spack is released under an LGPL license. For more details see the LICENSE file.

LLNL-CODE-647188