Commit Graph

296 Commits

Author SHA1 Message Date
Massimiliano Culpo
9e4b0eb34a Multi-valued variants (#2386)
Modifications:
- added support for multi-valued variants
- refactored code related to variants into variant.py
- added new generic features to AutotoolsPackage that leverage multi-valued variants
- modified openmpi to use new features
- added unit tests for the new semantics
2017-05-01 13:08:47 -07:00
Adam J. Stewart
ce3ab503de Python command, libraries, and headers (#3367)
## Motivation

Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically. 

I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.

Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends.

## Prefix

For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`.

Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.

To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable.

## Command

In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:

If the spec is for Python 3, try searching for the `python3` command.
If the spec is for Python 2, try searching for the `python2` command.
If neither are found, try searching for the `python` command.

## Libraries

Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work.

The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:
```
lib/libpython2.7.dylib
```
For Python 3.6 on CentOS 6, I'm seeing:
```
lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0
```
Notice the `m` after the version number. Yeah, that's a thing.

## Headers

In Python 2.7, I'm seeing:
```
include/python2.7/pyconfig.h
```
In Python 3.6, I'm seeing:
```
include/python3.6m/pyconfig.h
```
It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6

Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
2017-04-29 17:24:13 -07:00
Todd Gamblin
455cae01c2 Convert rest docstrings to Google docstrings. (#3994)
- Sometimes you need something mindless to do.
- Sometimes it can be helpful, as well.
2017-04-27 07:45:34 -07:00
Adam J. Stewart
eaa50d3b7c Add API Docs for lib/spack/llnl (#3982)
* Add API Docs for lib/spack/llnl
* Clean up after previous builds
* Better fix for purging API docs
2017-04-25 22:24:02 -07:00
Adam J. Stewart
c144c88302 Use six.moves.input instead of raw_input (#3961)
* Use six.moves.input instead of raw_input

* Remove comment mentioning raw_input
2017-04-23 10:32:08 -05:00
Adam J. Stewart
5250e8ee89 Fix HPL build, convert to MakefilePackage (#3777)
* Fix HPL build, convert to MakefilePackage

* Flake8 fix

* Fix: spec -> self.spec

* Properly query for system libraries

* Update Intel-MKL as well

* Recurse in system libs, fix MKL path, fixes lapack_libs
2017-04-21 12:11:29 -05:00
George Hartzell
84208523f9 set_executable can set S_IX{GRP,OTH} (#3742)
`set_executable` now checks if a user/group.other had read permission
on a file and if it does then it sets the corresponding executable
bit.

See #1483.
2017-04-06 16:25:13 -05:00
Todd Gamblin
fe6f39b662 Use key sorting instead of cmp()
- Get rid of pkgsort() usage for preferred variants.
- Concretization is now entirely based on key-based sorting.
- Remove PreferredPackages class and various spec cmp() methods.
- Replace with PackagePrefs class that implements a key function for
  sorting according to packages.yaml.
- Clear package pref caches on config test.
- Explicit compare methods instead of total_ordering in Version.
- Our total_ordering backport wasn't making Python 3 happy for some
  reason.
- Python 3's functools.total_ordering and spelling the operators out
  fixes the problem.
- Fix unicode issues with spec hashes, json, & YAML
- Try to use str everywhere and avoid unicode objects in python 2.
2017-03-31 13:40:41 -07:00
Todd Gamblin
1d1a14dbe9 Convert Python 2 idioms to Python 2/3-compatible ones.
- convert print, StringIO, except as, octals, izip
- convert print statement to print function
- convert StringIO to six.StringIO
  - remove usage of csv reader in Spec, in favor of simple regex
  - csv reader only does byte strings
- convert 0755 octal literals to 0o755
- convert `except Foo, e` to `except Foo as e`
- fix a few places `str` is used.
  - may need to switch everything to str later.
- convert iteritems usages to use six.iteritems
- fix urllib and HTMLParser
- port metaclasses to use six.with_metaclass
- More octal literal conversions for Python 2/3
- Fix a new octal literal.
- Convert `basestring` to `six.string_types`
- Convert xrange -> range
- Fix various issues with encoding, iteritems, and Python3 semantics.
- Convert contextlib.nested to explicitly nexted context managers.
- Convert use of filter() to list comprehensions.
- Replace reduce() with list comprehensions.
-  Clean up composite: replace inspect.ismethod() with callable()
- Python 3 doesn't have "method" objects; inspect.ismethod returns False.
- Need to use callable in Composite to make it work.
- Update colify to use future division.
- Fix zip() usages that need to be lists.
- Python3: Use line-buffered logging instead of unbuffered.
- Python3 raises an error with unbuffered I/O
  - See https://bugs.python.org/issue17404
2017-03-31 13:40:41 -07:00
Gregory Lee
604b75c1f9 created elf virtual package and updated dependent packages (#3317)
* created elf virtual package and updated dependent packages
* added `hide_files` context manager to handle moving files.
2017-03-09 10:36:32 -08:00
Adam J. Stewart
a2d70a45fb Allow find_libraries to accept lists or strings (#3363)
* Allow find_libraries to accept lists or strings

* Convert one more example from list to string
2017-03-04 11:23:57 -06:00
Massimiliano Culpo
ed582cef68 New interface for passing build information among specs (#1875)
- Added a new interface for Specs to pass build information
  - Calls forwarded from Spec to Package are now explicit
  - Added descriptor within Spec to manage forwarding
  - Added state in Spec to maintain query information
  - Modified a few packages (the one involved in spack install pexsi) to showcase changes

- This uses an object wrapper to `spec` to implement the `libs` sub-calls.
  - wrapper is returned from `__getitem__` only if spec is concrete
  - allows packagers to access build information easily
2017-03-02 10:01:29 -08:00
Adam J. Stewart
e492aff4f7 More consistent yes/no prompts (#3174)
* More consistent yes/no prompts
* Add ==> prefix to yes/no and number prompts
2017-02-17 13:45:02 -08:00
Erik Schnetter
cade0181fd Spack: Correct fix_darwin_install_name (#2886)
Previously, fix_darwin_install_name would only handle dependencies that have no path set, and it ignore dependencies that have the build directory as path baked in. Catch this, and replace it by the install directory.
2017-01-20 10:22:59 -08:00
Massimiliano Culpo
7ea10e768e unit tests: replace nose with pytest (#2502)
* Porting: substitute nose with ytest

This huge commit substitutes nose with pytest as a testing system. Things done here:

* deleted external/nose as it is no longer used
* moved mock resources in their own directory 'test/mock/'
* ported two tests (cmd/find, build_system) to pytest native syntax as an example
* build_environment, log: used monkeypatch instead of try/catch
* moved global mocking of fetch_cache to an auto-used fixture
* moved global mocking from test/__init__.py to conftest.py
* made `spack test` a wrapper around pytest
* run-unit-tests: avoid running python 2.6 tests under coverage to speed them up
* use `pytest --cov` instead of coverage run to cut down testing time

* mock/packages_test: moved mock yaml configuration to files instead of leaving it in the code as string literals

* concretize.py: ported tests to native pytest, reverted multiprocessing in pytest.ini as it was creating the wrong report for coveralls

* conftest.py, fixtures: added docstrings

* concretize_preferences.py: uses fixtures instead of subclassing MockPackagesTest

* directory_layout.py: uses fixtures instead of subclassing MockPackagesTest

* install.py: uses fixtures instead of subclassing MockPackagesTest

* optional_deps.py: uses fixtures instead of subclassing MockPackagesTest

optional_deps.py: uses fixtures instead of subclassing MockPackagesTest

* packages.py: uses fixtures instead of subclassing MockPackagesTest

* provider_index.py: uses fixtures instead of subclassing MockPackagesTest

* spec_yaml.py: uses fixtures instead of subclassing MockPackagesTest

* multimethod.py: uses fixtures instead of subclassing MockPackagesTest

* install.py: now uses mock_archive_url

* git_fetch.py: uses fixtures instead of subclassing MockPackagesTest

* hg_fetch.py: uses fixtures instead of subclassing MockPackagesTest

* svn_fetch.py, mirror.py: uses fixtures instead of subclassing MockPackagesTest
repo.py: deleted

* test_compiler_cmd.py: uses fixtures instead of subclassing MockPackagesTest

* cmd/module.py, cmd/uninstall.py: uses fixtures instead of subclassing MockDatabase

* database.py: uses fixtures instead of subclassing MockDatabase, removed mock/database

* pytest: uncluttering fixture implementations

* database: changing the scope to 'module'

* config.py: uses fixtures instead of subclassing MockPackagesTest

* spec_dag.py, spec_semantics.py: uses fixtures instead of subclassing MockPackagesTest

* stage.py: uses fixtures instead of subclassing MockPackagesTest. Removed mock directory

* pytest: added docstrings to all the fixtures

* pytest: final cleanup

* build_system_guess.py: fixed naming and docstrings as suggested by @scheibelp

* spec_syntax.py: added expected failure on parsing multiple specs closes #1976

* Add pytest and pytest-cov to Spack externals.

* Make `spack flake8` ignore externals.

* run-unit-tests runs spack test and not pytest.

* Remove all the special stuff for `spack test`

- Remove `conftest.py` magic and all the special case stuff in `bin/spack`

- Spack commands can optionally take unknown arguments, if they want to
  handle them.

- `spack test` is now a command like the others.

- `spack test` now just delegates its arguments to `pytest`, but it does
  it by receiving unknown arguments and NOT taking an explicit
  help argument.

* Fix error in fixtures.

* Improve `spack test` command a bit.

- Now supports an approximation of the old simple interface
- Also supports full pytest options if you want them.

* Use external coverage instead of pytest-cov

* Make coverage use parallel-mode.

* change __init__.py docs to include pytest
2016-12-29 07:48:48 -08:00
Massimiliano Culpo
17b13b161b Directive inheritance: laziness for the win (#2623)
* inheritance of directives: using meta-classes to inject attributes coming from directives into packages + lazy directives

* _dep_types -> dependency_types
* using a meta-class to inject directives into packages
* directives are lazy

fixes #2466

* directives.py: allows for multiple inheritance. Added blank lines as suggested by @tgamblin

* directives.py: added a test for simple inheritance of directives

* Minor improvement requested by @tgamblin

CMakePackage: importing names from spack.directives
directives: wrap __new__ to respect pep8

* Refactoring requested by @tgamblin

directives: removed global variables in favor of class variables. Simplified the interface for directives (they return a callable on a package or a list of them).
2016-12-28 12:37:02 -08:00
Matthew Krafczyk
f1b26cb72e Improve stacktrace printing
Sometimes files in the stacktrace are not from spack. Remove these
files before finding the spack root.
2016-11-17 12:51:41 -05:00
Matthew Krafczyk
de7171db8a Use better scheme for getting root directory 2016-11-16 12:14:25 -05:00
Matthew Krafczyk
6c854246aa Fix some formatting errors caught by flake8. 2016-11-16 11:58:36 -05:00
Matthew Krafczyk
8e6d890a19 -s now includes the file and line number with info
The option -s now causes file and line number information to be printed
along with any invocation of msg, info, etc...

This will greatly ease debugging.
2016-11-16 11:58:36 -05:00
Massimiliano Culpo
36a4ca8b11 spack install: forward sys.stdin to child processes (#2158)
* spack install: forward sys.stdin to child processes fixes #2140

 - [ ] redirection process is spawned in __enter__ instead of __init__
 - [ ] sys.stdin is forwarded to child processes

* log: wrapped __init__ definition
2016-11-03 08:03:10 -07:00
Todd Gamblin
0da639298c Move temp directory configuration to config.yaml
- Moved temp finding logic to spack.stage
- Updated stage tests
- Added tests for new path substaitution of $user, $spack, $tempdir
2016-10-30 23:55:00 -07:00
alalazo
ab995df777 Merge branch 'develop' of https://github.com/LLNL/spack into features/install_with_phases_rebase
Conflicts:
	lib/spack/spack/build_environment.py
	lib/spack/spack/cmd/install.py
	lib/spack/spack/cmd/setup.py
	lib/spack/spack/package.py
	var/spack/repos/builtin/packages/gmp/package.py
	var/spack/repos/builtin/packages/hdf5/package.py
2016-10-11 16:28:08 +02:00
Todd Gamblin
3d8d8d3644 Fix bug with lock upgrades.
- Closing and re-opening to upgrade to write will lose all existing read
  locks on this process.
  - If we didn't allow ranges, sleeping until no reads would work.
  - With ranges, we may never be able to take some legal write locks
    without invalidating all reads. e.g., if a write lock has distinct
    range from all reads, it should just work, but we'd have to close the
    file, reopen, and re-take reads.

- It's easier to just check whether the file is writable in the first
  place and open for writing from the start.

- Lock now only opens files read-only if we *can't* write them.
2016-10-11 01:55:33 -07:00
Todd Gamblin
da6bbfb2d4 Add byte-range parameters to llnl.util.lock 2016-10-11 01:55:32 -07:00
Todd Gamblin
ea10e3bab0 Remove need to touch lock files before using.
- Locks will now create enclosing directories and touch the lock file
  automatically.
2016-10-11 01:55:32 -07:00
Todd Gamblin
907fe912ef Make llnl.util.lock use file objects instead of low-level OS fds.
- Make sure we write, truncate, flush when setting PID and owning host in
  the file.
2016-10-11 01:55:32 -07:00
alalazo
f229290880 stage : try to remove dead links only of folder that you actually care about
A use case where the previous approach was failing is :

 - more than one spack process running on compute nodes
 - stage directory is a link to fast LOCAL storage

 In this case the processes may try to unlink something that is "dead" for them, but actually used by other processes on storage they cannot see.
2016-10-11 01:55:32 -07:00
alalazo
34fe51a4aa install : finer graned locking for install command 2016-10-11 01:38:27 -07:00
alalazo
7a26c60dbd Merge branch 'develop' of https://github.com/LLNL/spack into features/install_with_phases_rebase
Conflicts:
	lib/spack/spack/cmd/setup.py
	lib/spack/spack/package.py
	var/spack/repos/builtin/packages/gmp/package.py
2016-10-05 09:33:59 +02:00
Todd Gamblin
bff1656a1a Read-only locks should close fd before opening for write. (#1906)
- Fixes bad file descriptor error in lock acquire, #1904
- Fix bug introduced in previous PR #1857
- Backported fix from soon-to-be merged fine-grained DB locking branch.
2016-10-04 15:36:37 -07:00
Michael Kuhn
8d1ec0df3d Fix read locks on read-only file systems (#1857) 2016-09-30 09:45:08 -07:00
Massimiliano Culpo
d848559f70 Reworking of lapack_shared_libs and similar properties (#1682)
* Turned <provider>_libs into an iterable

Modifications :
- added class LibraryList + unit tests
- added convenience functions `find_libraries` and `dedupe`
- modifed non Intel blas/lapack providers
- modified packages using blas_shared_libs and similar functions

* atlas : added pthread variant

* intel packages : added lapack_libs and blas_libs

* find_library_path : removed unused function

* PR review : fixed last issues

* LibraryList : added test on __add__ return type

* LibraryList : added __radd__ fixed unit tests

fix : failing unit tests due to missing `self`

* cp2k and dependecies : fixed blas-lapack related statements in package.py
2016-09-21 12:27:59 -07:00
alalazo
f5433477b9 qa : flake8 issues 2016-08-11 09:08:00 +02:00
alalazo
b4b9ebe7d7 Merge branch 'develop' of https://github.com/LLNL/spack into features/install_with_phases
Conflicts:
	lib/spack/llnl/util/tty/log.py
	lib/spack/spack/__init__.py
	lib/spack/spack/cmd/install.py
	lib/spack/spack/cmd/setup.py
	lib/spack/spack/package.py
	var/spack/repos/builtin/packages/blitz/package.py
	var/spack/repos/builtin/packages/gmp/package.py
	var/spack/repos/builtin/packages/qhull/package.py
	var/spack/repos/builtin/packages/szip/package.py
2016-08-11 08:55:20 +02:00
Todd Gamblin
bf1072c902 Make Spack core PEP8 compliant. 2016-08-10 16:33:37 -07:00
Todd Gamblin
9d4a36a62f Properly re-raise exceptions from lock context handler. 2016-08-09 02:25:09 -07:00
Todd Gamblin
0c75c13cc0 Flake8 fixes 2016-08-09 02:25:07 -07:00
Todd Gamblin
102ac7bcf1 Move provider cache to home directory and refactor Transactions
Major stuff:

- Created a FileCache for managing user cache files in Spack.  Currently just
  handles virtuals.

- Moved virtual cache from the repository to the home directory so that users do
  not need write access to Spack repositories to use them.

- Refactored `Transaction` class in `database.py` -- moved it to
  `LockTransaction` in `lock.py` and made it reusable by other classes.

Other additions:

- Added tests for file cache and transactions.

- Added a few more tests for database

- Fixed bug in DB where writes could happen even if exceptions were raised
  during a transaction.

- `spack uninstall` now attempts to repair the database when it discovers that a
  prefix doesn't exist but a DB record does.
2016-08-09 00:24:54 -07:00
Adam J. Stewart
5d48c108a3 Only strip newline chars, not spaces 2016-08-08 15:02:05 -05:00
becker33
14d861a41c Merge pull request #1414 from adamjstewart/fixes/filter_file
Fix backup=True for filter_file
2016-08-01 14:10:59 -07:00
Adam J. Stewart
5b79f0d04a Fix backup=True for filter_file 2016-08-01 15:35:02 -05:00
George Hartzell
d684b17c06 Fix doc for install_tree (cut/paste error)
It looks like the docs for copy_tree were cut/paste from copy and still referred to installing a "file".

This fixes that.
2016-07-25 08:45:44 -07:00
alalazo
40cb314638 Merge branch 'develop' of https://github.com/LLNL/spack into features/install_with_phases
Conflicts:
	lib/spack/spack/package.py
2016-07-20 21:02:19 +02:00
Denis Davydov
9ea4f80f15 flake8 fixes 2016-07-16 07:58:51 +02:00
Denis Davydov
992250ddea add to_lib_name() to get library name from a path 2016-07-16 07:24:05 +02:00
alalazo
1ecea4c2f1 log : refactored acquire and release semantic to meet the context manager protocol 2016-07-14 09:34:01 +02:00
alalazo
90b131260b log : changed semantic for start / join (now it's explicit) 2016-07-13 22:50:39 +02:00
alalazo
ad16830f71 log : added timeout to avoid deadlocks on daemon join 2016-07-13 11:20:23 +02:00
alalazo
813cb032c4 package.py : updated logic to log.py rework
Conflicts:
	lib/spack/spack/package.py
2016-07-13 09:27:09 +02:00
alalazo
9af964a6d6 log_output : moved from os.fork to multiprocessing.Process 2016-07-13 09:27:09 +02:00
Adam J. Stewart
71a297f121 Fix backup logic in filter_file 2016-06-23 16:44:41 -05:00
Todd Gamblin
c09111bbef Merge pull request #1053 from davydden/pkg/dealii_suite_update
update Petsc, Slepc, Trilinos, Superlu-dist and deal.II
2016-06-20 10:32:52 -07:00
Todd Gamblin
4569df025e Merge pull request #1060 from mathstuf/no-break-long-words
tty: do not break long words
2016-06-17 14:51:26 -07:00
Todd Gamblin
17b868381f Fixes #460: Do not show variants by default in spack find.
This does two things:

1. By default `spack find` no longer shows variants.  You have to
   supply `-v` to get that

2. This improves the `colify` implementation so that it no longer pads
   the rightmost column.  This avoids the issue where if one spec was
   too long in the output, *all* specs would have space padding added
   to that width, and it would look like the output of `spack find`
   was double spaced.  This no longer happens -- the one bad line
   wraps around and the other lines are now single-spaced when you use
   `-v` with boost.
2016-06-16 02:55:33 -07:00
Denis Davydov
3d3a26cd9a flake8 fixes for filesystem.py 2016-06-15 18:31:10 +02:00
Ben Boeckel
36decbfa69 tty: do not break long words
These "words" tend to be file paths which are best left unbroken.

Fixes #1057.
2016-06-15 12:22:35 -04:00
Denis Davydov
a860a3d614 superlu-dist: avoid harcoding blas/lapack/mpi; remove preferred version 2016-06-15 18:21:13 +02:00
Todd Gamblin
e7ced54369 Correct LLNL LGPL license template for clarity. 2016-05-11 21:22:25 -07:00
Todd Gamblin
220c72d67f Merge branch 'features/blas-lapack-hardening' into develop 2016-03-31 11:37:09 -07:00
Todd Gamblin
8b45d6f84f Merge pull request #658 from davydden/metis_parmetis_darwin
metis/parmetis/oce/boost: correct install_name on Darwin via a global tool
2016-03-27 16:10:05 -07:00
Denis Davydov
7eca1284c8 metis/parmetis/boost/oce/scalapack: correct install_name on Darwin via a global function. 2016-03-28 00:22:25 +02:00
Todd Gamblin
cde3320582 Add a method to find the containing directory of a library. 2016-03-27 11:32:42 -07:00
Todd Gamblin
b0b882cbb3 Speed up directives by skipping debug info in stack traversal.
- `caller_locals()` doesn't need debug info, only frame locals.
- `get_calling_module()` doesn't either.

- Changed calls to `inspect.stack()` -> `inspect.stack(0)`
2016-03-27 11:30:09 -07:00
Todd Gamblin
003fd4d834 Optimize __eq__ and __ne__ in key_ordering
- use `is` when possible before calling `_cmp_key()`
2016-03-14 04:55:30 -07:00
Todd Gamblin
214deeda54 Merge branch 'features/noexpand' into develop
Conflicts:
	lib/spack/llnl/util/filesystem.py
2016-03-08 03:29:57 -08:00
Todd Gamblin
ada675dea2 Merge branch 'features/sbang' into develop
Conflicts:
	lib/spack/llnl/util/filesystem.py
2016-03-06 19:30:19 -08:00
Todd Gamblin
240ada5775 Add expand=False option for URL downloads.
- Allows skipping the expand step for downloads.
  - Fixed stage so that it knows expansion didn't fail when there is a
    no-expand URLFetchStrategy.

- Updated docs to reflect new option, and provided an example.
2016-03-06 16:51:09 -08:00
Todd Gamblin
ad103dcafa Small refactor: add keep parameter to stage, get rid of stage.destroy call.
- package.py uses context manager more effectively.

- Stage.__init__ has easier to understand method signature now.
  - keep can be used to override the default behavior either to keep
    the stage ALL the time or to delete the stage ALL the time.
2016-03-05 20:00:30 -08:00
Todd Gamblin
14d48eba46 Revert c5d9ee8924 for bug fixes.
- This reverts commit c5d9ee8924.

- merged too soon before
- reverting and fixing bugs now.

Conflicts:
	lib/spack/spack/mirror.py
	lib/spack/spack/package.py
2016-03-05 18:40:28 -08:00
Todd Gamblin
fc6aa7374a Fix #104, #54: issues with overlong shebang in deep directories.
This does several things:

- Add `sbang`: a script to run scripts with long shebang lines.
  - Documentation for `sbang` is in `bin/sbang`.

- Add an `sbang` hook that filters the `bin` directory after install
  and modifies any scripts wtih shebangs that are too long to use
  `sbang` instead.

- `sbang` is at the top level, so it should be runnable (not much we
  can do if spack itself is too deep for shebang)
  - `sbang`, when used as the interpreter, runs the *second* shebang
    line it finds in a script.
  - shoud fix issues with too long shebang paths.
2016-03-05 04:18:48 -08:00
Todd Gamblin
c5d9ee8924 Revert "refactoring proposal : turn Stage into a context manager" 2016-03-03 14:41:20 -08:00
alalazo
7f4a69319a Merge branch 'develop' of https://github.com/LLNL/spack into refactoring/stage
Conflicts:
	lib/spack/spack/package.py
2016-03-03 10:19:45 +01:00
alalazo
4d63544fe9 remove_link_tree : moved to llnl.util.filesystem 2016-03-02 16:55:57 +01:00
alalazo
901e4851b9 _cleanup_dead_links : factored method into a function and put it in llnl.filesystem 2016-03-02 16:19:32 +01:00
Todd Gamblin
c488f7c4d8 Fix bug in install permission setting. 2016-03-01 23:57:34 -08:00
Todd Gamblin
8d6342c53d Merge branch 'mplegendre-multi_pkgsrc_roots' into develop
- This moves var/spack/packages to var/spack/repos/builtin/packages.

- Packages that did not exist in the source branch, or were changed in
  develop, were moved into var/spack/repos/builtin/packages as part of
  the integration.

Conflicts:
	lib/spack/spack/test/unit_install.py
	var/spack/repos/builtin/packages/clang/package.py
2016-01-19 01:16:08 -08:00
Todd Gamblin
c1e57e49ae Merge pull request #202 from scheibelp/features/unittest-xml-output-nose
Features/unittest xml output nose
2016-01-17 19:14:49 -08:00
Todd Gamblin
05b30bf83e Make text wrapping off by default in tty, add a kwarg for it. 2016-01-14 10:26:31 -08:00
Todd Gamblin
c65fd3a289 Merge branch 'develop' into mplegendre-multi_pkgsrc_roots
Conflicts:
	lib/spack/spack/cmd/create.py
	lib/spack/spack/cmd/extensions.py
	lib/spack/spack/cmd/fetch.py
	lib/spack/spack/cmd/uninstall.py
	lib/spack/spack/config.py
	lib/spack/spack/database.py
	lib/spack/spack/directory_layout.py
	lib/spack/spack/packages.py
	lib/spack/spack/spec.py
2015-12-25 16:35:55 -08:00
Todd Gamblin
73ea15db8e Merge pull request #208 from epfl-scitas/features/resource_directive
resource directive : implementation + clang / llvm use case
2015-12-21 10:34:16 -08:00
Todd Gamblin
c41b9b7ddc Change github.com/scalability-llnl to github.com/llnl everywhere. 2015-12-09 01:24:15 -08:00
Massimiliano Culpo
1fe626ec7c resource directive : sketch of implementation + clang / llvm use case 2015-11-26 17:53:33 +01:00
Peter Scheibel
0adf93853d Update log_output to handle cases where sys.stdout/stderr streams do not have
an associated file descriptor (e.g. holds for nose tests where sys.stdout is
assigned to a StringIO object)
2015-11-24 17:58:17 -08:00
Todd Gamblin
55662eca69 Merge branch 'develop' into mplegendre-multi_pkgsrc_roots
Conflicts:
	lib/spack/spack/__init__.py
	lib/spack/spack/directives.py
	lib/spack/spack/packages.py
2015-11-12 15:16:06 -08:00
Todd Gamblin
a1c42c2753 Merge Greg Becker's build database support into develop. 2015-10-29 02:22:20 -07:00
Todd Gamblin
a58ae0c5d0 Build database working with simple transaction support; all tests passing. 2015-10-27 16:36:44 -07:00
Todd Gamblin
bf8479bec6 Fix stupid lock bug.
- Code simplification ignored case where exception was raised.
- If LockError was raised, read and write counts were incremented erroneously.
- updated lock test.
2015-10-27 16:34:26 -07:00
Todd Gamblin
af7b96c14a Lock acquires return True/False depending on whether they got POSIX lock. 2015-10-27 00:35:06 -07:00
Ben Boeckel
9d90cb6962 python: use the setdefault method on dict
It allows more concise code and skips some key lookups.
2015-10-26 17:38:19 -04:00
Todd Gamblin
ead8ac58c6 Working Lock class, now uses POSIX fcntl locks, extensive unit test.
- llnl.util.lock now uses fcntl.lockf instead of flock
  - purported to have more NFS compatibility.

- Added an extensive test case for locks.
  - tests acquiring, releasing, upgrading, timeouts, shared, & exclusive cases.
2015-10-24 19:57:43 -07:00
Todd Gamblin
d16095c856 Add forgotten file from previous commit. 2015-10-13 10:35:19 -07:00
Todd Gamblin
2c81875019 Fix bug in colify color handling. 2015-10-12 14:44:41 -07:00
Todd Gamblin
e17ad6a684 Simplify lock context managers. 2015-09-17 01:05:19 -07:00
Todd Gamblin
ccf311c9c6 Several changes to DB implementation.
1. Database stores a file version, so we can add to it in the future.
2. Database indexed by hashes and not numerical indexes.
3. Specs built by database have consistent hashes and it's checked.
4. minor naming and whitespace changes.
2015-09-17 00:16:12 -07:00
Gregory Becker
f406fcb843 Fixed several issues from code review
Most importantly wrote the Lock, Read_Lock_Instance, and Write_Lock_Instance classes in lock.py
Updated the locking in database.py
TODO: Lock on larger areas
2015-08-31 09:38:38 -07:00
Todd Gamblin
360b307f68 Save progress. import gov.llnl.spack.mpich works. 2015-08-24 09:14:16 -07:00
Todd Gamblin
92f398a897 Better @memoized decorator. 2015-08-16 12:50:39 -07:00
Todd Gamblin
6ed5dd7427 log_output now allows echo to be toggled wtih 'v' key 2015-06-20 20:20:28 -07:00
Todd Gamblin
5d7a6c0c46 Add FileFilter convenience class to llnl.util.filesystem. 2015-06-06 15:26:48 -07:00
Saravan Pantham
454d06c25c Removed autosave files from previous commit. 2015-06-06 15:26:37 -07:00
Saravan Pantham
c174fe6199 Fixed BLAS and Lapack installations 2015-06-06 15:26:31 -07:00
Todd Gamblin
92c21d7134 debug instead of info logging in llnl.util.filesystem 2015-05-29 17:21:30 -07:00
Todd Gamblin
0035100186 SPACK-69: Add context manager to fork and log output in a with block. 2015-05-29 17:20:08 -07:00
Todd Gamblin
9b40d93fc2 Add ability to force color mode even if terminal is NOT a tty 2015-05-29 17:19:03 -07:00
Todd Gamblin
ceecd2ce41 Add ability to test whether tty verbose or debug is enabled. 2015-05-29 17:18:33 -07:00
Todd Gamblin
1f8ce403dc Modularize directives. Now each directive specifies its storage. 2015-05-10 12:24:03 -07:00
Todd Gamblin
0944ba120c relations are now "directives", and code is cleaned up. 2015-05-10 12:24:03 -07:00
Todd Gamblin
560f2c299a Add install_tree, force_symlink helper functions. 2015-03-12 10:32:29 -07:00
Todd Gamblin
d49c98188a Add an override to colify so we can set terminal dimensions. 2015-02-23 01:23:30 -08:00
Todd Gamblin
67db8ddca8 Factor ignore logic into a predicate builder. 2015-02-17 00:21:15 -08:00
Todd Gamblin
614c22fc1b Allow forced deactivation -- best effort unlinking
spack deactivate -f will unlink even if Spack thinks the package isn't enabled.
Made deactivate routines idempotent.
2015-02-16 12:41:22 -08:00
Todd Gamblin
2d9190d264 Add extensions command. 2015-02-02 11:20:36 -08:00
Todd Gamblin
6b90017efa Fixed dumb link_tree bug, added test for link tree. 2015-02-02 11:20:35 -08:00
Todd Gamblin
de91c95e8e Ability to ignore files in activate/deactivate for extensions. 2015-02-02 11:19:54 -08:00
Todd Gamblin
82946d2914 Move symlink tree routines to LinkTree class. 2015-02-02 11:19:52 -08:00
Todd Gamblin
9977543478 Added feature: package extensions
- packages can be "extended" by others
- allows extension to be symlinked into extendee's prefix.
- used for python modules.
  - first module: py-setuptools
2015-02-02 11:19:00 -08:00
Todd Gamblin
e6b4530234 Add is_exe function to filesystem. 2015-01-22 13:52:28 -08:00
Todd Gamblin
a6e00f6086 Fix ColorStream 2014-12-29 01:05:21 -08:00
Todd Gamblin
860f834aad spack graph allows plotting specific packages. 2014-12-26 13:52:49 -08:00
Todd Gamblin
0bc861db6e Fix up bzip2 install 2014-12-25 17:55:19 -08:00
Todd Gamblin
20388ece86 Clearer code in filter_file. 2014-12-25 16:07:39 -08:00
Todd Gamblin
11cffff943 colify handles ansi color input directly; no more decorator. 2014-12-02 22:32:15 -08:00
Todd Gamblin
40b4fa5443 Better spack find view. 2014-12-01 23:14:06 -08:00
Todd Gamblin
e15316e825 index_by supports compound index keys. 2014-12-01 23:13:09 -08:00
Todd Gamblin
72c753b93e Colify now supports fixing the number of columns. 2014-12-01 21:29:01 -08:00
Todd Gamblin
22e4d11010 Cleanup code in colify. 2014-11-23 16:19:26 -08:00
Todd Gamblin
287b04e50a Bugfix in terminal_size() 2014-11-23 17:55:37 -06:00
Todd Gamblin
eba13b8653 Checksum warning now prompts for override. 2014-11-08 23:20:01 -08:00
Todd Gamblin
b97ee67a4b Working GCC package. 2014-11-05 09:54:43 -08:00
Todd Gamblin
4ecc7e1c93 Document file filtering functions. 2014-10-27 19:53:55 -07:00
Todd Gamblin
e2af2a27bf Merge branch 'features/git-fetching' into develop
Conflicts:
	lib/spack/docs/packaging_guide.rst
	lib/spack/spack/cmd/info.py
	lib/spack/spack/package.py
	lib/spack/spack/stage.py
2014-10-27 19:53:05 -07:00
Todd Gamblin
36a87f5bf9 Update documentation to add an auto-generated list of packages. 2014-10-08 03:08:40 -07:00
Todd Gamblin
4bde771970 Fix for SPACK-39: Concretization was too restrictive.
- concretize_version() now Use satisfies(), not intersection.
- version class updated with better intersection/union commands
- version now 1.6 "contains" 1.6.5
- added test for new version functionality

- remove none_high and none_low classes
  - version module is now self-contained; save for external 2.7
    functools.total_ordering for 2.6 compatibility.
2014-10-07 23:22:45 -07:00
Todd Gamblin
a8ed1ec414 Minor argparse improvement. 2014-09-29 20:00:00 -07:00
Todd Gamblin
3bd52678be MPICH sets MPI compilers to use real compilers and not spack wrappers. 2014-09-27 21:36:42 -07:00
Todd Gamblin
c55041e9d4 Partial commit of more packages. 2014-08-04 07:54:22 -07:00
Todd Gamblin
1ad474f1a9 Allow per-version URLs instead of one single URL per package. 2014-07-30 23:30:07 -07:00
Todd Gamblin
295ffd8c50 Adding dotkit support to TAU.
- New spack.hooks package
  - contains modules with pre and post install hooks

- New dotkit hook module
  - generates/removes dotkits on install/uninstall

- New spack use, spack unuse commands
  - use same syntax as install/uninstall

- New setup-env.bash script
  - Sets up path, dotkit support

- new spack dotkit command
  - used by script to parse specs, generate
    specs of installed pckages for dotkit file names
2014-07-08 01:58:20 -07:00
Todd Gamblin
0551638944 Add TAU package and routines to filter files, fix sed calls.
- TAU doesn't install to directories with '@' in the name.
- Need to fix up its scripts.
- routines to filter files as sed would, but using python regular expressions.
  - TAU package uses this.
2014-07-02 23:24:10 -07:00
Todd Gamblin
b6740cf1d1 Make debug and verbose output work properly. 2014-06-22 12:50:34 -07:00
Todd Gamblin
4608b674e5 Minor cleanup and bug fixes. 2014-06-22 12:50:34 -07:00
Todd Gamblin
4665505936 Add index_by function to llnl.util.lang 2014-05-27 08:33:19 -07:00
Todd Gamblin
1f9dfeb9b5 Clean up find command, move code to utils. 2014-04-14 12:53:23 -07:00
Todd Gamblin
59a3b8dc67 Make SSL verification failures recommend spack -k
- Adds optional long message to SpackError
- tty now wraps long message text.
2014-04-13 16:38:15 -07:00
Todd Gamblin
9d01df9e8a Moving utilities to a common LLNL package. 2014-03-14 16:21:15 -07:00