Replace use of `shutil.copytree` with `copy_tree` and `install_tree` functions in `llnl.util.filesystem`.
- `copy_tree` copies without setting permissions. It should be used to copy files around in the build directory.
- `install_tree` copies files and sets permissions. It should be used to copy files into the installation directory.
- `install` and `copy` are analogous single-file functions.
- add more extensive tests for these functions
- update packages to use these functions.
The go team released 1.9.2 which includes fixes for some things
that 1.9.1 broke:
> ... include fixes to the compiler, linker, runtime, documentation, go command, and the crypto/x509, database/sql, log, and net/smtp packages. They include a fix to a bug introduced in Go 1.9.1 and Go 1.8.4 that broke "go get" of non-Git repositories under certain conditions.
## Motivation
Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically.
I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.
Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends.
## Prefix
For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`.
Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.
To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable.
## Command
In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:
If the spec is for Python 3, try searching for the `python3` command.
If the spec is for Python 2, try searching for the `python2` command.
If neither are found, try searching for the `python` command.
## Libraries
Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work.
The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:
```
lib/libpython2.7.dylib
```
For Python 3.6 on CentOS 6, I'm seeing:
```
lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0
```
Notice the `m` after the version number. Yeah, that's a thing.
## Headers
In Python 2.7, I'm seeing:
```
include/python2.7/pyconfig.h
```
In Python 3.6, I'm seeing:
```
include/python3.6m/pyconfig.h
```
It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6
Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
* Remove fake URLs from Spack
* Ignore long lines for URLs that start with ftp:
* Preliminary changes to version regexes
* New redesign of version regexes
* Allow letters in version-only
* Fix detection of versions that end in Final
* Rearrange a few regexes and add examples
* Add tests for common download repositories
* Add test cases for common tarball naming schemes
* Finalize version regexes
* spack url test -> spack url summary
* Clean up comments
* Rearrange suffix checks
* Use query strings for name detection
* Remove no longer necessary url_for_version functions
* Strip off extraneous information after package name
* Add one more test
* Dot in square brackets does not need to be escaped
* Move renaming outside of parse_name_offset
* Fix versions for a couple more packages
* Fix flake8 and doc tests
* Correctly parse Python, Lua, and Bio++ package names
* Use effective URLs for mfem
* Add checksummed version to mitos
* Remove url_for_version from STAR-CCM+ package
* Revert changes to version numbers with underscores and dashes
* Fix name detection for tbb
* Correctly parse Ruby gems
* Reverted mfem back to shortened URLs.
* Updated instructions for better security
* Remove preferred=True from newest version
* Add tests for new `spack url list` flags
* Add tests for strip_name_suffixes
* Add unit tests for version separators
* Fix bugs related to parseable name but in parseable version
* Remove dead code, update docstring
* Ignore 'binary' at end of version string
* Remove platform from version
* Flip libedit version numbers
* Re-support weird NCO alpha/beta versions
* Rebase and remove one new fake URL
* Add / to beginning of regex to avoid picking up similarly named packages
* Ignore weird tar versions
* Fix bug in url parse --spider when no versions found
* Less strict version matching for spack versions
* Don't rename Python packages
* Be a little more selective, version must begin with a digit
* Re-add fake URLs
* Fix up several other packages
* Ignore more file endings
* Add parsing support for Miniconda
* Update tab completion
* XFAILS are now PASSES for 2 web tests
[The fix](https://github.com/golang/go/issues/17986) for the small buglet addressed by `misc-cgo-testcshared.patch` has been merged into the tree a while back. I was surprised to see that it wasn't in 1.7.5 and did a bit of digging. It is *has not* been merged into the 1.7 branch but it *has* been merged into 1.8 (and therefor the patch will no longer be necessary).
Figured I'd document my digging for the next person to come along.
Update the go package to v1.7.5.
- This release no longer needs the time-test patch (it's been merged upstream).
- This release still seems to need the cgo-testcshared patch.
- Also add a comment about environment set up that I need to build it successfully on a very large server.
Tested on CentOS 7.
* Removing the nobuild, nolink, and alldeps dependency types in favor of being explicit.
* This will help with maintenance going forward, as adding more dependency types won't affect existing declared dependencies in weird ways.
* default deptype is still `('build', 'link')`
* Update go to 1.7.4 and 1.6.4 to fix security issues
The go team recently rolled out two releases to address security
issues. Details available on the [go release
site](https://golang.org/doc/devel/release.html).
This commit updates our explicitly supported versions.
It also includes a comment about two CentOS requirements (enable
user_namespace and ensure that the static c library is installed) that
are required for the pacakges to pass their tests.
* Flake8 cleanup
* Update go-bootstrap package
The last C based Go src tree was the 1.4 series. For a while they
were cutting new releases so that people could bootstrap from a C only
system. Now they're recommending that you either use the release-1.4
branch or that you use a date-stamped tarball that they'll produce on
an as-needed basis.
There are several issues that keep 1.4.2 from building on a CentOS 7
system.
I've switched to the date based tarball.
The cgo bits were also mis-behaving, but they're not needed for the
bootstrapping task so I've set an environment variable that disables
them.
Details [on the install-from-source
page](https://golang.org/doc/install/source#go14) and these issues:
- https://github.com/golang/go/issues/17545
- https://github.com/golang/go/issues/16352.
* Update go package
Switched from pulling from the git repository to using the source
tarballs and added digest values.
Added support for 1.7.3, continued supporting 1.6.2, including patches
for a couple of problems (details in
[17545](https://github.com/golang/go/issues/17545) and
[17986](https://github.com/golang/go/issues/17986).
Dropped support for 1.5.4 and 1.4.2 because they no longer pass their
tests and the patches above to not apply.
* Fixed a few small bugs in the 'git-lfs' install script.
* Fixed a bug in the '(go|go-bootstrap)@1.4.2+test' install scripts.
* Fixing a minor style issue in the 'git-lfs' install script.
This commit includes:
* a new go package that uses gccgo to bootstrap the go toolchain
* env support added to Executable
* a new Go fetch strategy that uses `go get` to fetch a package and all
of its deps
* A platinum searcher package that leverages the new go package and
fetch strategy
