Fixes#9001#8289 added support for install_tree and copy_tree to merge into an existing
directory structure. However, it did not properly handle relative symlinks and
also removed support for the 'ignore' keyword. Additionally, some of the tests
were overly-strict when checking the permissions on the copied files.
This updates the install_tree/copy_tree methods and their tests:
* copy_tree/install_tree now preserve relative link targets (if the symlink in the
source directory structure is relative, the symlink created in the destination
will be relative)
* Added support for 'ignore' argument back to copy_tree/install_tree (removed
in #8289). It is no longer the object output by shutil.ignore_patterns: you pass a
function that accepts a path relative to the source and returns whether that
path should be copied.
* The openfoam packages (currently the only ones making use of the 'ignore'
argument) are updated for the new API
* When a symlink target is absolute, copy_tree and install_tree now rewrite the
source prefix to be the destination prefix
* copy_tree tests no longer check permissions: copy_tree doesn't enforce
anything about permissions so its tests don't check for that
* install_tree tests no longer check for exact permission matching since it can add
file permissions
Replace use of `shutil.copytree` with `copy_tree` and `install_tree` functions in `llnl.util.filesystem`.
- `copy_tree` copies without setting permissions. It should be used to copy files around in the build directory.
- `install_tree` copies files and sets permissions. It should be used to copy files into the installation directory.
- `install` and `copy` are analogous single-file functions.
- add more extensive tests for these functions
- update packages to use these functions.
* Fix performance issue when compiling.
Spack was doing active wait when compiling, spoiling one core.
My fix consists in not setting any timeout for select, instead of
the previous 0 second.
* Fix comments about select.select timeout
- This was a nasty workaround due to the way our compiler wrappers used
to work. We don't want to have to modify our elfutils installation to
install libdwarf.
- Since cd9691de5, we no longer need this because the package will always
come before dependencies in our include order.
- repo membership test was broken by the refactor of spack/__init__.py
- refactor singleton so that 'spec in repo' works again for `spack.repo.path`
- fix spec command and add basic tests for `spack spec` and `spack spec --yaml`
- Fixes a bug in `llnl.util.lock`
- Locks in the current directory would fail because the parent directory
was the empty string.
- Fix this and return '.' for the parent of locks in the current
directory.
- Clean up error messages for when a lock can't be created, or when an
exclusive (write) lock can't be taken on a file.
- Add a number of subclasses of LockError to distinguish timeouts from
permission issues.
- Add an explicit check to prevent the user from taking a write lock on a
read-only file.
- We had a check for this for when we try to *upgrade* a lock on an RO
file, but not for an initial write lock attempt.
- Add more tests for different lock permission scenarios.
- write locks previously wrote information about the lock holder (host
and pid), and read locks woudl read this in.
- This is really only for debugging, so only enable it then
- add some tests that target debug info, and improve multiproc lock test
output
Functional updates:
- `python` now creates a copy of the `python` binaries when it is added
to a view
- Python extensions (packages which subclass `PythonPackage`) rewrite
their shebang lines to refer to python in the view
- Python packages in the same namespace will not generate conflicts if
both have `...lib/site-packages/namespace-example/__init__.py`
- These `__init__` files will also remain when removing any package in
the namespace until the last package in the namespace is removed
Generally (Updated 2/16):
- Any package can define `add_files_to_view` to customize how it is added
to a view (and at the moment custom definitions are included for
`python` and `PythonPackage`)
- Likewise any package can define `remove_files_from_view` to customize
which files are removed (e.g. you don't always want to remove the
namespace `__init__`)
- Any package can define `view_file_conflicts` to customize what it
considers a merge conflict
- Global activations are handled like views (where the view root is the
spec prefix of the extendee)
- Benefit: filesystem-management aspects of activating extensions are
now placed in views (e.g. now one can hardlink a global activation)
- Benefit: overriding `Package.activate` is more straightforward (see
`Python.activate`)
- Complication: extension packages which have special-purpose logic
*only* when activated outside of the extendee prefix must check for
this in their `add_files_to_view` method (see `PythonPackage`)
- `LinkTree` is refactored to have separate methods for copying a
directory structure and for copying files (since it was found that
generally packages may want to alter how files are copied but still
wanted to copy directories in the same way)
TODOs (updated 2/20):
- [x] additional testing (there is some unit testing added at this point
but more would be useful)
- [x] refactor or reorganize `LinkTree` methods: currently there is a
separate set of methods for replicating just the directory structure
without the files, and a set for replicating everything
- [x] Right now external views (i.e. those not used for global
activations) call `view.add_extension`, but global activations do not
to avoid some extra work that goes into maintaining external views. I'm
not sure if addressing that needs to be done here but I'd like to
clarify it in the comments (UPDATE: for now I have added a TODO and in
my opinion this can be merged now and the refactor handled later)
- [x] Several method descriptions (e.g. for `Package.activate`) are out
of date and reference a distinction between global activations and
views, they need to be updated
- [x] Update aspell package activations
- Spack was assuming that a group with gid == current uid would always exist.
- This was breaking the travis build for macos.
- also fix issue with the DB tarball test finding coverage filesx
- spack.util.lock behaves the same as llnl.util.lock, but Lock._lock and
Lock._unlock do nothing.
- can be disabled with a control variable.
- configuration options can enable/disable locking:
- `locks` option in spack configuration controls whether Spack will use filesystem locks or not.
- `-l` and `-L` command-line options can force-disable or force-enable locking.
- Spack will check for group- and world-writability before disabling
locks, and it will not allow a group- or world-writable instance to
have locks disabled.
- update documentation
- simplify the singleton pattern across the codebase
- reduce lines of code needed for crufty initialization
- reduce functions that need to mess with a global
- Singletons whose semantics changed:
- spack.store.store() -> spack.store
- spack.repo.path() -> spack.repo.path
- spack.config.config() -> spack.config.config
- spack.caches.fetch_cache() -> spack.caches.fetch_cache
- spack.caches.misc_cache() -> spack.caches.misc_cache
* Added installation date and time to the database
Information on the date and time of installation of a spec is recorded
into the database. The information is retained on reindexing.
* Expose the possibility to query for installation date
The DB can now be queried for specs that have been installed in a given
time window. This query possibility is exposed to command line via two
new options of the `find` command.
* Extended docstring for Database._add
* Use timestamps since the epoch instead of formatted date in the DB
* Allow 'pretty date' formats from command line
* Substituted kwargs with explicit arguments
* Simplified regex for pretty date strings. Added unit tests.
This updates the fix_darwin_install_name function to use the Spack
Executable object to run install_name_tool, which ensures that
process output is formatted as a 'str' for python2 and python3.
Originally fix_darwin_install_name was invoking subprocess.Popen
directly.
Following the discussion with Todd and Adam, find has been modified to
accept glob expressions. This should not affect performance as every
glob implementation I inspected has 3 cases (no wildcard, wildcard but
no directories involved, wildcard and directories involved) and uses
fnmatch underneath.
Mixins have been changed to do by default a non-recursive search (but
a recursive search can still be triggered using the recursive keyword).
Following comments from Todd:
- the call to tty.debug has been moved deeper, to log the filtering of each file
- the shadowing on the name "kwargs" is avoided
- command reference now includes usage for all Spack commands as output
by `spack help`. Each command usage links to any related section in
the docs.
- added `spack commands` command which can list command names,
subcommands, and generate RST docs for commands.
- added `llnl.util.argparsewriter`, which analyzes an argparse parser and
calls hooks for description, usage, options, and subcommands
'spack install' can now reinstall a spec even if it has dependents, via
the --overwrite option. This option moves the current installation in a
temporary directory. If the reinstallation is successful the temporary
is removed, otherwise a rollback is performed.
- When you don't use wildcards, flake8 will find places where you used an
undefined name.
- This commit has all the bugfixes resulting from this static check.
I'm tracking down a problem with the perl package that's been
generating this error:
```
OSError: OSError: [Errno 2] No such file or directory: '/blah/blah/blah/lib/5.24.1/x86_64-linux/Config.pm~'
```
The real problem is upstream, but it's being masked by an exception
raised in `filter_file`s finally block.
In my case, `backup` is `False`.
The backup is created around line 127, the `re.sub()` calls
fails (working on that), the `except` block fires and moves the backup
file back, then the finally block tries to remove the non-existent
backup file.
This change just avoids trying to remove the non-existent file.
`spack blame` prints out the contributors to a package.
By modification time:
```
$ spack blame --time llvm
LAST_COMMIT LINES % AUTHOR EMAIL
3 days ago 2 0.6 Andrey Prokopenko <andrey.prok@gmail.com>
3 weeks ago 125 34.7 Massimiliano Culpo <massimiliano.culpo@epfl.ch>
3 weeks ago 3 0.8 Peter Scheibel <scheibel1@llnl.gov>
2 months ago 21 5.8 Adam J. Stewart <ajstewart426@gmail.com>
2 months ago 1 0.3 Gregory Becker <becker33@llnl.gov>
3 months ago 116 32.2 Todd Gamblin <tgamblin@llnl.gov>
5 months ago 2 0.6 Jimmy Tang <jcftang@gmail.com>
5 months ago 6 1.7 Jean-Paul Pelteret <jppelteret@gmail.com>
7 months ago 65 18.1 Tom Scogland <tscogland@llnl.gov>
11 months ago 13 3.6 Kelly (KT) Thompson <kgt@lanl.gov>
a year ago 1 0.3 Scott Pakin <pakin@lanl.gov>
a year ago 3 0.8 Erik Schnetter <schnetter@gmail.com>
3 years ago 2 0.6 David Beckingsale <davidbeckingsale@gmail.com>
3 days ago 360 100.0
```
Or by percent contribution:
```
$ spack blame --percent llvm
LAST_COMMIT LINES % AUTHOR EMAIL
3 weeks ago 125 34.7 Massimiliano Culpo <massimiliano.culpo@epfl.ch>
3 months ago 116 32.2 Todd Gamblin <tgamblin@llnl.gov>
7 months ago 65 18.1 Tom Scogland <tscogland@llnl.gov>
2 months ago 21 5.8 Adam J. Stewart <ajstewart426@gmail.com>
11 months ago 13 3.6 Kelly (KT) Thompson <kgt@lanl.gov>
5 months ago 6 1.7 Jean-Paul Pelteret <jppelteret@gmail.com>
3 weeks ago 3 0.8 Peter Scheibel <scheibel1@llnl.gov>
a year ago 3 0.8 Erik Schnetter <schnetter@gmail.com>
3 years ago 2 0.6 David Beckingsale <davidbeckingsale@gmail.com>
3 days ago 2 0.6 Andrey Prokopenko <andrey.prok@gmail.com>
5 months ago 2 0.6 Jimmy Tang <jcftang@gmail.com>
2 months ago 1 0.3 Gregory Becker <becker33@llnl.gov>
a year ago 1 0.3 Scott Pakin <pakin@lanl.gov>
3 days ago 360 100.0
```
- Python I/O would not properly interleave (or appear) with output from
subcommands.
- Add a flusing wrapper around sys.stdout and sys.stderr when
redirecting, so that Python output is synchronous with that of
subcommands.
- 'v' toggle was previously only good for the current install.
- subsequent installs needed user to press 'v' again.
- 'v' state is now preserved across dependency installs.
- Simplify interface to log_output. New interface requires only one
context handler instead of two. Before:
with log_output('logfile.txt') as log_redirection:
with log_redirection:
# do things ... output will be logged
After:
with log_output('logfile.txt'):
# do things ... output will be logged
If you also want the output to be echoed to ``stdout``, use the
`echo` parameter::
with log_output('logfile.txt', echo=True):
# do things ... output will be logged and printed out
And, if you just want to echo *some* stuff from the parent, use
``force_echo``:
with log_output('logfile.txt', echo=False) as logger:
# do things ... output will be logged
with logger.force_echo():
# things here will be echoed *and* logged
A key difference between this and the previous implementation is that
*everything* in the context handler is logged. Previously, things like
`Executing phase 'configure'` would not be logged, only output to the
screen, so understanding phases in the build log was difficult.
- The implementation of `log_output()` is different in two major ways:
1. This implementation avoids race cases by using only one pipe (before
we had a multiprocessing pipe and a unix pipe). The logger daemon
stops naturally when the input stream is closed, which avoids a race
in the previous implementation where we'd miss some lines of output
because the parent would shut the daemon down before it was done
with all output.
2. Instead of turning output redirection on and off, which prevented
some things from being logged, this version uses control characters
in the output stream to enable/disable forced echoing. We're using
the time-honored xon and xoff codes, which tell the daemon to echo
anything between them AND write it to the log. This is how
`logger.force_echo()` works.
- Fix places where output could get stuck in buffers by flushing more
aggressively. This makes the output printed to the terminal the same
as that which would be printed through a pipe to `cat` or to a file.
Previously these could be weirdly different, and some output would be
missing when redirecting Spack to a file or pipe.
- Simplify input and color handling in both `build_environment.fork()`
and `llnl.util.tty.log.log_output()`. Neither requires an input_stream
parameter anymore; we assume stdin will be forwarded if possible.
- remove `llnl.util.lang.duplicate_stream()` and remove associated
monkey-patching in tests, as these aren't needed if you just check
whether stdin is a tty and has a fileno attribute.
* Disable spec colorization when redirecting stdout and add command line flag to re-enable
* Add command line `--color` flag to control output colorization
* Add options to `llnl.util.tty.color` to allow color to be auto/always/never
* Add `Spec.cformat()` function to be used when `format()` should have auto-coloring
- Lock test can be run either as a node-local test or as an MPI test.
- Lock test is now parametrized by filesystem, so you can test the
locking capabilities of your NFS, Lustre, or GPFS filesystem. See docs
for details.
PR #3367 inadvertently changed the semantics of _find_recursive and
_find_non_recursive so that the returned list are not ordered as the
input search list. This commit restores the original semantic, and adds
tests to verify it.
Modifications:
- added support for multi-valued variants
- refactored code related to variants into variant.py
- added new generic features to AutotoolsPackage that leverage multi-valued variants
- modified openmpi to use new features
- added unit tests for the new semantics
## Motivation
Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically.
I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.
Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends.
## Prefix
For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`.
Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.
To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable.
## Command
In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:
If the spec is for Python 3, try searching for the `python3` command.
If the spec is for Python 2, try searching for the `python2` command.
If neither are found, try searching for the `python` command.
## Libraries
Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work.
The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:
```
lib/libpython2.7.dylib
```
For Python 3.6 on CentOS 6, I'm seeing:
```
lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0
```
Notice the `m` after the version number. Yeah, that's a thing.
## Headers
In Python 2.7, I'm seeing:
```
include/python2.7/pyconfig.h
```
In Python 3.6, I'm seeing:
```
include/python3.6m/pyconfig.h
```
It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6
Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
`set_executable` now checks if a user/group.other had read permission
on a file and if it does then it sets the corresponding executable
bit.
See #1483.
- Get rid of pkgsort() usage for preferred variants.
- Concretization is now entirely based on key-based sorting.
- Remove PreferredPackages class and various spec cmp() methods.
- Replace with PackagePrefs class that implements a key function for
sorting according to packages.yaml.
- Clear package pref caches on config test.
- Explicit compare methods instead of total_ordering in Version.
- Our total_ordering backport wasn't making Python 3 happy for some
reason.
- Python 3's functools.total_ordering and spelling the operators out
fixes the problem.
- Fix unicode issues with spec hashes, json, & YAML
- Try to use str everywhere and avoid unicode objects in python 2.
- convert print, StringIO, except as, octals, izip
- convert print statement to print function
- convert StringIO to six.StringIO
- remove usage of csv reader in Spec, in favor of simple regex
- csv reader only does byte strings
- convert 0755 octal literals to 0o755
- convert `except Foo, e` to `except Foo as e`
- fix a few places `str` is used.
- may need to switch everything to str later.
- convert iteritems usages to use six.iteritems
- fix urllib and HTMLParser
- port metaclasses to use six.with_metaclass
- More octal literal conversions for Python 2/3
- Fix a new octal literal.
- Convert `basestring` to `six.string_types`
- Convert xrange -> range
- Fix various issues with encoding, iteritems, and Python3 semantics.
- Convert contextlib.nested to explicitly nexted context managers.
- Convert use of filter() to list comprehensions.
- Replace reduce() with list comprehensions.
- Clean up composite: replace inspect.ismethod() with callable()
- Python 3 doesn't have "method" objects; inspect.ismethod returns False.
- Need to use callable in Composite to make it work.
- Update colify to use future division.
- Fix zip() usages that need to be lists.
- Python3: Use line-buffered logging instead of unbuffered.
- Python3 raises an error with unbuffered I/O
- See https://bugs.python.org/issue17404
- Added a new interface for Specs to pass build information
- Calls forwarded from Spec to Package are now explicit
- Added descriptor within Spec to manage forwarding
- Added state in Spec to maintain query information
- Modified a few packages (the one involved in spack install pexsi) to showcase changes
- This uses an object wrapper to `spec` to implement the `libs` sub-calls.
- wrapper is returned from `__getitem__` only if spec is concrete
- allows packagers to access build information easily
Previously, fix_darwin_install_name would only handle dependencies that have no path set, and it ignore dependencies that have the build directory as path baked in. Catch this, and replace it by the install directory.
* Porting: substitute nose with ytest
This huge commit substitutes nose with pytest as a testing system. Things done here:
* deleted external/nose as it is no longer used
* moved mock resources in their own directory 'test/mock/'
* ported two tests (cmd/find, build_system) to pytest native syntax as an example
* build_environment, log: used monkeypatch instead of try/catch
* moved global mocking of fetch_cache to an auto-used fixture
* moved global mocking from test/__init__.py to conftest.py
* made `spack test` a wrapper around pytest
* run-unit-tests: avoid running python 2.6 tests under coverage to speed them up
* use `pytest --cov` instead of coverage run to cut down testing time
* mock/packages_test: moved mock yaml configuration to files instead of leaving it in the code as string literals
* concretize.py: ported tests to native pytest, reverted multiprocessing in pytest.ini as it was creating the wrong report for coveralls
* conftest.py, fixtures: added docstrings
* concretize_preferences.py: uses fixtures instead of subclassing MockPackagesTest
* directory_layout.py: uses fixtures instead of subclassing MockPackagesTest
* install.py: uses fixtures instead of subclassing MockPackagesTest
* optional_deps.py: uses fixtures instead of subclassing MockPackagesTest
optional_deps.py: uses fixtures instead of subclassing MockPackagesTest
* packages.py: uses fixtures instead of subclassing MockPackagesTest
* provider_index.py: uses fixtures instead of subclassing MockPackagesTest
* spec_yaml.py: uses fixtures instead of subclassing MockPackagesTest
* multimethod.py: uses fixtures instead of subclassing MockPackagesTest
* install.py: now uses mock_archive_url
* git_fetch.py: uses fixtures instead of subclassing MockPackagesTest
* hg_fetch.py: uses fixtures instead of subclassing MockPackagesTest
* svn_fetch.py, mirror.py: uses fixtures instead of subclassing MockPackagesTest
repo.py: deleted
* test_compiler_cmd.py: uses fixtures instead of subclassing MockPackagesTest
* cmd/module.py, cmd/uninstall.py: uses fixtures instead of subclassing MockDatabase
* database.py: uses fixtures instead of subclassing MockDatabase, removed mock/database
* pytest: uncluttering fixture implementations
* database: changing the scope to 'module'
* config.py: uses fixtures instead of subclassing MockPackagesTest
* spec_dag.py, spec_semantics.py: uses fixtures instead of subclassing MockPackagesTest
* stage.py: uses fixtures instead of subclassing MockPackagesTest. Removed mock directory
* pytest: added docstrings to all the fixtures
* pytest: final cleanup
* build_system_guess.py: fixed naming and docstrings as suggested by @scheibelp
* spec_syntax.py: added expected failure on parsing multiple specs closes#1976
* Add pytest and pytest-cov to Spack externals.
* Make `spack flake8` ignore externals.
* run-unit-tests runs spack test and not pytest.
* Remove all the special stuff for `spack test`
- Remove `conftest.py` magic and all the special case stuff in `bin/spack`
- Spack commands can optionally take unknown arguments, if they want to
handle them.
- `spack test` is now a command like the others.
- `spack test` now just delegates its arguments to `pytest`, but it does
it by receiving unknown arguments and NOT taking an explicit
help argument.
* Fix error in fixtures.
* Improve `spack test` command a bit.
- Now supports an approximation of the old simple interface
- Also supports full pytest options if you want them.
* Use external coverage instead of pytest-cov
* Make coverage use parallel-mode.
* change __init__.py docs to include pytest
* inheritance of directives: using meta-classes to inject attributes coming from directives into packages + lazy directives
* _dep_types -> dependency_types
* using a meta-class to inject directives into packages
* directives are lazy
fixes#2466
* directives.py: allows for multiple inheritance. Added blank lines as suggested by @tgamblin
* directives.py: added a test for simple inheritance of directives
* Minor improvement requested by @tgamblin
CMakePackage: importing names from spack.directives
directives: wrap __new__ to respect pep8
* Refactoring requested by @tgamblin
directives: removed global variables in favor of class variables. Simplified the interface for directives (they return a callable on a package or a list of them).
The option -s now causes file and line number information to be printed
along with any invocation of msg, info, etc...
This will greatly ease debugging.
* spack install: forward sys.stdin to child processes fixes#2140
- [ ] redirection process is spawned in __enter__ instead of __init__
- [ ] sys.stdin is forwarded to child processes
* log: wrapped __init__ definition
- Closing and re-opening to upgrade to write will lose all existing read
locks on this process.
- If we didn't allow ranges, sleeping until no reads would work.
- With ranges, we may never be able to take some legal write locks
without invalidating all reads. e.g., if a write lock has distinct
range from all reads, it should just work, but we'd have to close the
file, reopen, and re-take reads.
- It's easier to just check whether the file is writable in the first
place and open for writing from the start.
- Lock now only opens files read-only if we *can't* write them.
A use case where the previous approach was failing is :
- more than one spack process running on compute nodes
- stage directory is a link to fast LOCAL storage
In this case the processes may try to unlink something that is "dead" for them, but actually used by other processes on storage they cannot see.
* Turned <provider>_libs into an iterable
Modifications :
- added class LibraryList + unit tests
- added convenience functions `find_libraries` and `dedupe`
- modifed non Intel blas/lapack providers
- modified packages using blas_shared_libs and similar functions
* atlas : added pthread variant
* intel packages : added lapack_libs and blas_libs
* find_library_path : removed unused function
* PR review : fixed last issues
* LibraryList : added test on __add__ return type
* LibraryList : added __radd__ fixed unit tests
fix : failing unit tests due to missing `self`
* cp2k and dependecies : fixed blas-lapack related statements in package.py
Major stuff:
- Created a FileCache for managing user cache files in Spack. Currently just
handles virtuals.
- Moved virtual cache from the repository to the home directory so that users do
not need write access to Spack repositories to use them.
- Refactored `Transaction` class in `database.py` -- moved it to
`LockTransaction` in `lock.py` and made it reusable by other classes.
Other additions:
- Added tests for file cache and transactions.
- Added a few more tests for database
- Fixed bug in DB where writes could happen even if exceptions were raised
during a transaction.
- `spack uninstall` now attempts to repair the database when it discovers that a
prefix doesn't exist but a DB record does.
This does two things:
1. By default `spack find` no longer shows variants. You have to
supply `-v` to get that
2. This improves the `colify` implementation so that it no longer pads
the rightmost column. This avoids the issue where if one spec was
too long in the output, *all* specs would have space padding added
to that width, and it would look like the output of `spack find`
was double spaced. This no longer happens -- the one bad line
wraps around and the other lines are now single-spaced when you use
`-v` with boost.
- Allows skipping the expand step for downloads.
- Fixed stage so that it knows expansion didn't fail when there is a
no-expand URLFetchStrategy.
- Updated docs to reflect new option, and provided an example.
- package.py uses context manager more effectively.
- Stage.__init__ has easier to understand method signature now.
- keep can be used to override the default behavior either to keep
the stage ALL the time or to delete the stage ALL the time.
- This reverts commit c5d9ee8924.
- merged too soon before
- reverting and fixing bugs now.
Conflicts:
lib/spack/spack/mirror.py
lib/spack/spack/package.py
This does several things:
- Add `sbang`: a script to run scripts with long shebang lines.
- Documentation for `sbang` is in `bin/sbang`.
- Add an `sbang` hook that filters the `bin` directory after install
and modifies any scripts wtih shebangs that are too long to use
`sbang` instead.
- `sbang` is at the top level, so it should be runnable (not much we
can do if spack itself is too deep for shebang)
- `sbang`, when used as the interpreter, runs the *second* shebang
line it finds in a script.
- shoud fix issues with too long shebang paths.
- This moves var/spack/packages to var/spack/repos/builtin/packages.
- Packages that did not exist in the source branch, or were changed in
develop, were moved into var/spack/repos/builtin/packages as part of
the integration.
Conflicts:
lib/spack/spack/test/unit_install.py
var/spack/repos/builtin/packages/clang/package.py
- Code simplification ignored case where exception was raised.
- If LockError was raised, read and write counts were incremented erroneously.
- updated lock test.
- llnl.util.lock now uses fcntl.lockf instead of flock
- purported to have more NFS compatibility.
- Added an extensive test case for locks.
- tests acquiring, releasing, upgrading, timeouts, shared, & exclusive cases.
1. Database stores a file version, so we can add to it in the future.
2. Database indexed by hashes and not numerical indexes.
3. Specs built by database have consistent hashes and it's checked.
4. minor naming and whitespace changes.
Most importantly wrote the Lock, Read_Lock_Instance, and Write_Lock_Instance classes in lock.py
Updated the locking in database.py
TODO: Lock on larger areas
