Python depends on gettext. Packages that depend on gettext and Python
together will encounter a concretizer bug which incorrectly detects
a constraint conflict. This sets the default value of +libxml2 in
Python to be the same as gettext so that packages which depend on
both (like mesa) can successfully concretize without adding manual
constraints.
* Add a transaction around repeated calls to `spec.prefix` in the activation process
* cache the computation of home in the python package to speed up setting deps
* ensure that module-scope variables are only set *once* per module
* Methods setting the environment now do it separately for build and run
Before this commit the `*_environment` methods were setting
modifications to both the build-time and run-time environment
simultaneously. This might cause issues as the two environments
inherently rely on different preconditions:
1. The build-time environment is set before building a package, thus
the package prefix doesn't exist and can't be inspected
2. The run-time environment instead is set assuming the target package
has been already installed
Here we split each of these functions into two: one setting the
build-time environment, one the run-time.
We also adopt a fallback strategy that inspects for old methods and
executes them as before, but prints a deprecation warning to tty. This
permits to port packages to use the new methods in a distributed way,
rather than having to modify all the packages at once.
* Added a test that fails if any package uses the old API
Marked the test xfail for now as we have a lot of packages in that
state.
* Added a test to check that a package modified by a PR is up to date
This test can be used any time we deprecate a method call to ensure
that during the first modification of the package we update also
the deprecated calls.
* Updated documentation
We'd like to use a consistent checksum scheme everywhere so that we can:
a) incorporate archive checksums into our specs and have a
consistent hashing algorithm across all specs.
b) index mirrors with a consistent type of checksum, and not one that
is dependent on how spack packages are written.
- [x] convert existing md5, sha224, sha512, sha1 checksums to sha256
This updates the Spack QT package to enable building qt version 4 on
MacOS.
This includes the following changes to the qt package:
* add version 4.8.7
* add option to build with or without shared libs
* add options to disable tools, ssl, sql, and freetype support
* add qt4-tools patch when building qt@4+tools
* add option to build as a framework (only available on MacOS)
* replace qt4-el-capitan patch with qt4-mac patch (which includes the
edits from qt4-el-capitan)
* apply qt4-pcre-include-conflict.patch only for version 4.8.6
(rather than all 4.x versions)
* apply qt4-gcc-and-webkit.patch for 4.x versions before 4.8.7 and
create a separate qt4-gcc-and-webkit-487.patch for version 4.8.7
* update patch function for qt@4 on MacOS to update configure
variables relevant to Spack (e.g. PREFIX)
* add option to build freetype with Spack, as a vendored dependency
of QT, or not at all (default is to build with Spack)
This includes the following edits outside of the qt package:
* Update MacOS version utility function to return all parts of the
Mac version (rather than just the first two)
* gettext package: implement "libs"
* python package: add gettext as a dependency
* Remove libuuid dependency for python on Darwin
Python 2.7 includes Carbon, which requires a proprietary typedef defined in
the system `/usr/include/uuid/uuid.h`:
```
In file included from /rnsdhpc/code/spack/var/spack/stage/python-2.7.16-chgwvgls4ezgujsknn27vlmkhwbzpqik/Python-2.7.16/Python/mactoolboxglue.c:27:
In file included from /rnsdhpc/code/spack/var/spack/stage/python-2.7.16-chgwvgls4ezgujsknn27vlmkhwbzpqik/Python-2.7.16/Include/pymactoolbox.h:10:
In file included from /System/Library/Frameworks/Carbon.framework/Headers/Carbon.h:20:
In file included from /System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:23:
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:208:
In file included from /System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/HFSVolumes.h:25:
/usr/include/hfs/hfs_format.h:794:2: error: unknown type name 'uuid_string_t'; did you mean 'io_string_t'?
```
Similarly, even with Python 3, having Spack's libuuid in the toolchain
causes matplotlib to fail to build downstream because its `osx` backend
links against Carbon:
```
building 'matplotlib.backends._macosx' extension
/rnsdhpc/code/spack/lib/spack/env/clang/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include -I/usr/X11/include -I/opt/X11/include -I. -I/rnsdhpc/code/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/python-3.7.2-bipx24oyxqxowqc75o5vfih7dljiaohz/include/python3.7m -c src/_macosx.m -o build/temp.macosx-10.14.5-x86_64-3.7/src/_macosx.o
In file included from src/_macosx.m:1:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Cocoa.framework/Headers/Cocoa.h:12:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:87:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSURLError.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:24:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:208:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/HFSVolumes.h:25:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/hfs/hfs_format.h:794:2: error: unknown type name 'uuid_string_t'; did you mean 'io_string_t'?
uuid_string_t ext_jnl_uuid;
^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/device/device_types.h:89:33: note: 'io_string_t' declared here
typedef char io_string_t[512];
```
MacOS version 10.14.5
Xcode 10.2.1
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
* Mark libuuid as conflicting with Darwin due to framework
If Spack's libuuid is included by any other package that tests for or
requires the Cocoa or Carbon frameworks, the build will *fail* becuse it
appears that Apple's libraries require a special version of libuuid and
uuid.h.
```
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Cocoa.framework/Headers/Cocoa.h:12:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/Foundation.h:87:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSURLError.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:24:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/AE.framework/Headers/AE.h:20:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/CarbonCore.h:208:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/CarbonCore.framework/Headers/HFSVolumes.h:25:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/hfs/hfs_format.h:794:2: error: unknown type name 'uuid_string_t'; did you mean 'io_string_t'?
uuid_string_t ext_jnl_uuid;
^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/device/device_types.h:89:33: note: 'io_string_t' declared here
typedef char io_string_t[512];
```
* Revert "Mark libuuid as conflicting with Darwin due to framework"
This reverts commit 9a68b75202.
* Document libuuid exception in python package
Fixes#10769
This updates the .headers property to include header subdirectories
for Python and Eigen (as is recommended by these packages).
#10623 updated the default behavior of .headers.directories to
exclude subdirectories (since this can cause clashes with system
headers). This broke some packages which depended on the old behavior
of .headers.directories: for example if you had
<package-prefix>/include/subdir/ex1.h, .headers.directories would
include <package-prefix>/include/subdir.
Allow customizing views with Spec-formatted directory structure
Allow views to specify projections that are more complicated than
merging every package into a single shared prefix. This will allow
sites to configure a view for the way they want to present packages
to their users; for example this can be used to create a prefix for
each package but omit the DAG hash from the path.
This includes a new YAML format file for specifying the simplified
prefix for a spec in a view. This configuration allows the use of
different prefix formats for different specs (i.e. specs depending
on MPI can include the MPI implementation in the prefix).
Documentation on usage of the view projection configuration is
included.
Depending on the projection configuration, paths are not guaranteed
to be unique and it may not be possible to add multiple installs of
a package to a view.
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
- core and most packages are (Apache-2.0 OR MIT)
- a very small number of remaining packages are LGPL-2.1-only
Replace use of `shutil.copytree` with `copy_tree` and `install_tree` functions in `llnl.util.filesystem`.
- `copy_tree` copies without setting permissions. It should be used to copy files around in the build directory.
- `install_tree` copies files and sets permissions. It should be used to copy files into the installation directory.
- `install` and `copy` are analogous single-file functions.
- add more extensive tests for these functions
- update packages to use these functions.
Functional updates:
- `python` now creates a copy of the `python` binaries when it is added
to a view
- Python extensions (packages which subclass `PythonPackage`) rewrite
their shebang lines to refer to python in the view
- Python packages in the same namespace will not generate conflicts if
both have `...lib/site-packages/namespace-example/__init__.py`
- These `__init__` files will also remain when removing any package in
the namespace until the last package in the namespace is removed
Generally (Updated 2/16):
- Any package can define `add_files_to_view` to customize how it is added
to a view (and at the moment custom definitions are included for
`python` and `PythonPackage`)
- Likewise any package can define `remove_files_from_view` to customize
which files are removed (e.g. you don't always want to remove the
namespace `__init__`)
- Any package can define `view_file_conflicts` to customize what it
considers a merge conflict
- Global activations are handled like views (where the view root is the
spec prefix of the extendee)
- Benefit: filesystem-management aspects of activating extensions are
now placed in views (e.g. now one can hardlink a global activation)
- Benefit: overriding `Package.activate` is more straightforward (see
`Python.activate`)
- Complication: extension packages which have special-purpose logic
*only* when activated outside of the extendee prefix must check for
this in their `add_files_to_view` method (see `PythonPackage`)
- `LinkTree` is refactored to have separate methods for copying a
directory structure and for copying files (since it was found that
generally packages may want to alter how files are copied but still
wanted to copy directories in the same way)
TODOs (updated 2/20):
- [x] additional testing (there is some unit testing added at this point
but more would be useful)
- [x] refactor or reorganize `LinkTree` methods: currently there is a
separate set of methods for replicating just the directory structure
without the files, and a set for replicating everything
- [x] Right now external views (i.e. those not used for global
activations) call `view.add_extension`, but global activations do not
to avoid some extra work that goes into maintaining external views. I'm
not sure if addressing that needs to be done here but I'd like to
clarify it in the comments (UPDATE: for now I have added a TODO and in
my opinion this can be merged now and the refactor handled later)
- [x] Several method descriptions (e.g. for `Package.activate`) are out
of date and reference a distinction between global activations and
views, they need to be updated
- [x] Update aspell package activations
- simplify the singleton pattern across the codebase
- reduce lines of code needed for crufty initialization
- reduce functions that need to mess with a global
- Singletons whose semantics changed:
- spack.store.store() -> spack.store
- spack.repo.path() -> spack.repo.path
- spack.config.config() -> spack.config.config
- spack.caches.fetch_cache() -> spack.caches.fetch_cache
- spack.caches.misc_cache() -> spack.caches.misc_cache
- spack.store was previously initialized at the spack.store module level,
but this means the store has to be initialized on every spack call.
- this moves the state in spack.store to a singleton so that the store is
only initialized when needed.
