* Style: black 23, skip magic trailing commas
* isort should use same line length as black
* Fix unused import
* Update version of black used in CI
* Update new packages
* Update new packages
Many noqa's in the code are no longer necessary now that the column limit is 99
characters. Others can easily be eliminated, and still more can just be made more
specific if they do not have to do with line length.
The only E501's still in the code are in the tests for `spack.util.path` and the tests
for `spack style`.
Explicitly import package utilities in all packages, and corresponding fallout.
This includes:
* rename `spack.package` to `spack.package_base`
* rename `spack.pkgkit` to `spack.package`
* update all packages in builtin, builtin_mock and tutorials to include `from spack.package import *`
* update spack style
* ensure packages include the import
* automatically add the new import and remove any/all imports of `spack` and `spack.pkgkit`
from packages when using `--fix`
* add support for type-checking packages with mypy when SPACK_MYPY_CHECK_PACKAGES
is set in the environment
* fix all type checking errors in packages in spack upstream
* update spack create to include the new imports
* update spack repo to inject the new import, injection persists to allow for a deprecation period
Original message below:
As requested @adamjstewart, update all packages to use pkgkit. I ended up using isort to do this,
so repro is easy:
```console
$ isort -a 'from spack.pkgkit import *' --rm 'spack' ./var/spack/repos/builtin/packages/*/package.py
$ spack style --fix
```
There were several line spacing fixups caused either by space manipulation in isort or by packages
that haven't been touched since we added requirements, but there are no functional changes in here.
* [x] add config to isort to make sure this is maintained going forward
* fix remaining flake8 errors
* imports: sort imports everywhere in Spack
We enabled import order checking in #23947, but fixing things manually drives
people crazy. This used `spack style --fix --all` from #24071 to automatically
sort everything in Spack so PR submitters won't have to deal with it.
This should go in after #24071, as it assumes we're using `isort`, not
`flake8-import-order` to order things. `isort` seems to be more flexible and
allows `llnl` mports to be in their own group before `spack` ones, so this
seems like a good switch.
- [x] add `concretize.lp`, `spack.yaml`, etc. to licensed files
- [x] update all licensed files to say 2013-2021 using
`spack license update-copyright-year`
- [x] appease mypy with some additions to package.py that needed
for oneapi.py
Fixes#11781
* Rename build log to spack-build-log.txt
* Rename environment variables file to spack-build-env.txt
* The name of the log and env files is now the same during the build
and after the build completes
* Update packages which referred to the build log/env files
* For packages installed before this commit using older names for the
build and env files, search for the older names
The original packaging, with openfoam as a virtual and with
openfoam-com, openfoam-org and openfoam-extend as
providers, adds an obfuscation layer without any benefit.
- Rename `openfoam-com` to `openfoam`, and remove the
`openfoam` virtual package.
- `spack.util.environment` is the new home for routines that modify
environment variables.
- This is to make room for `spack.environment` to contain new routines
for dealing with spack environments
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
- core and most packages are (Apache-2.0 OR MIT)
- a very small number of remaining packages are LGPL-2.1-only
Fixes#9001#8289 added support for install_tree and copy_tree to merge into an existing
directory structure. However, it did not properly handle relative symlinks and
also removed support for the 'ignore' keyword. Additionally, some of the tests
were overly-strict when checking the permissions on the copied files.
This updates the install_tree/copy_tree methods and their tests:
* copy_tree/install_tree now preserve relative link targets (if the symlink in the
source directory structure is relative, the symlink created in the destination
will be relative)
* Added support for 'ignore' argument back to copy_tree/install_tree (removed
in #8289). It is no longer the object output by shutil.ignore_patterns: you pass a
function that accepts a path relative to the source and returns whether that
path should be copied.
* The openfoam packages (currently the only ones making use of the 'ignore'
argument) are updated for the new API
* When a symlink target is absolute, copy_tree and install_tree now rewrite the
source prefix to be the destination prefix
* copy_tree tests no longer check permissions: copy_tree doesn't enforce
anything about permissions so its tests don't check for that
* install_tree tests no longer check for exact permission matching since it can add
file permissions
- renamed develop version from 'plus' to 'develop'
- patches now prefixed by corresponding OpenFOAM version number.
This makes it easier to sort and see what old/junk exists.
- remove MPI_BUFFER_SIZEk env variable (for all openfoam variants).
The OpenFOAM shell setup addresses this and there is no reason
to pollute the module environment at this stage.
- The buggy flex-2.6.2 was blacklisted in the corresponding flex
package, but now also removed the md5sum to avoid suggesting that
this version should be revived.
The 2.6.3 has similar problems (at least for scotch), but 2.6.4
seems to work.
- Rejig flex restriction for scotch to exclude 2.6.2-2.6.3 only. Since
flex-2.6.4 appears to be okay again, we can remove the flex version
restriction that trickled through into the openfoam packages as a
consequent of an spack spec bug.
- Make flex a build dependency for the openfoam packages
(seems to have been an earlier oversight).
* Several improvements for the openfoam packages
--
Refactor openfoam packages by adding an OpenfoamArch class
Use separate configure, build, install phases.
Provide FOAM_PROJECT_DIR dependent env for openfoam packages
- easier way to locate
Eliminate intermediate installation directories
- unneeded clutter.
- makes it less than easy to find the etc/bashrc file
Add versioning for all openfoam patches
- no certainty which parts (if any) will be needed in future versions,
especially if we strive to ensure that the upstream version builds
well with spack to begin with.
Support build of develop branches
- helps track build regressions for future openfoam releases
STYLE: use common/ and assets/ to provide additional (build) resources ...
* - adjust OpenFOAM provider
Move openfoam-com up front since this is the one being used as a base
for the others
## Motivation
Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically.
I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.
Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends.
## Prefix
For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`.
Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.
To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable.
## Command
In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:
If the spec is for Python 3, try searching for the `python3` command.
If the spec is for Python 2, try searching for the `python2` command.
If neither are found, try searching for the `python` command.
## Libraries
Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work.
The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:
```
lib/libpython2.7.dylib
```
For Python 3.6 on CentOS 6, I'm seeing:
```
lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0
```
Notice the `m` after the version number. Yeah, that's a thing.
## Headers
In Python 2.7, I'm seeing:
```
include/python2.7/pyconfig.h
```
In Python 3.6, I'm seeing:
```
include/python3.6m/pyconfig.h
```
It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6
Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
* ENH: add package for building OpenFOAM (1612) from www.openfoam.com
- provide 'openfoam' as virtual package.
- package as openfoam-com to reflect the distribution point.
This initial spack packaging for OpenFOAM supports a number of possible
variants and should handle 64-bit labels properly now that the scotch
package has been updated accordingly.
* ENH: update package for foam-extend (extend-project.de)
- provide 'openfoam' as virtual package.
- much of the build is now aligned with how the openfoam-com package
looks, with the aim of future refactoring.
- avoid installing intermediate targets.
- contains its own environment sourcing script for the build, for more
flexibility and robustness (doesn't touch the python build environ)
* ENH: added package for building from openfoam.org
- provide 'openfoam' as a virtual package.
- this is largely a direct copy of the openfoam-com package.
It has been supplied as a courtesy for users and to ensure maximum
consistency in quality and naming between the foam-extend,
openfoam-com and openfoam-org packages.
* CONFIG: add openfoam into bash completion providers list
* ENH: have openfoam-com use spack as USERMPI
- also simplify the generation of mplib/compiler rules
* ENH: have openfoam-org use spack as SYSTEMMPI
- this setup requires more environment settings than USERMPI
(openfoam-com), but is currently the only means of integration
for openfoam-org
- simplify generation of mplib/compiler rules
* ENH: simplify generation of mplib/compiler rules (foam-extend)
- rename mpi rules from SPACK,SPACKMPI to USER,USERMPI for consistency
with openfoam-com and to generalize for any build system.
* STYLE: record spack tree as a log file (openfoam)
- can be useful for future diagnostics and general record keeping
* Ensure that every package has a license
Also fixes URLs with http://http:// doubled.
This is a continuation of #2656.
* Add license to every file in Spack
* Make sure Todd is the author of all packages
* Fix flake8 tests
* Don't license external Sphinx docs
* Don't display licenses in tutorial example packages
Also fixes typos and converts command-line examples
from tcsh to bash, which is more common
