Commit Graph

284 Commits

Author SHA1 Message Date
Massimiliano Culpo
8dd95c1705 Fixed options to compile generic code on ppc64 and ppc64le 2019-10-11 21:20:28 -07:00
Massimiliano Culpo
b07460ab5f Added NEON to the list of features required for the aarch64 family
Both floating-point and NEON are required in all standard ARMv8
implementations. Theoretically though specialized markets can support
no NEON or floating-point at all. Source:

https://developer.arm.com/docs/den0024/latest/aarch64-floating-point-and-neon

On the other hand the base procedure call standard for Aarch64
"assumes the availability of the vector registers for passing
floating-point and SIMD arguments". Further "the Arm 64-bit
architecture defines two mandatory register banks: a general-purpose
register bank which can be used for scalar integer processing and
pointer arithmetic; and a SIMD and Floating-Point register bank".
Source:

https://developer.arm.com/docs/ihi0055/latest/procedure-call-standard-for-the-arm-64-bit-architecture

This makes customization of Aarch64 with no NEON instruction set
available so unlikely that we can consider them a feature of the
generic family.
2019-10-10 16:24:36 -07:00
Massimiliano Culpo
78577c0a90 Generic x86_64 code compiled with GCC uses non deprecated mtune flags (#13022)
fixes #12928
2019-10-03 10:31:03 +02:00
Massimiliano Culpo
9117dfd118 Add all the 'generic' architectures that are mentioned in recipes (#12958)
LLVM, mesa and other packages check for these generic
microarchitectures. One solution is to let Spack know they exist.
2019-09-28 21:47:05 -07:00
Adam J. Stewart
065cbe89fe Fix "specific target" detection in Python 3 (#12906)
The output of subprocess.check_output is a byte string in Python 3. This causes dictionary lookup to fail later on.

A try-except around this function prevented this error from being noticed. Removed this so that more errors can propagate out.
2019-09-24 09:47:54 -07:00
Massimiliano Culpo
2468ccee58 AMD: fix architecture hierarchy (zen) (#12913)
* microarchitectures: zen starts from x86_64, not from excavator
* Unit tests: fixed a test that is wrong with the new modeling
* microarchitectures: fixed features and inheritance for 15h family

bulldozer doesn't inherit from barcelona (10h) + added xop, lwp and tbm
instruction sets to the 15h family (it distinguish the family from 17h)
2019-09-23 21:54:13 -07:00
Gregory Becker
c43f105359 targets: add mic_knl target to microarchitectures.json
- This is needed to support Cray machines -- we need an architecture
  mic_knl > x86_64

- We used Cray's naming scheme for this target to make it work seamlessly
  with the module-based detection sccheme on Cray.  mic_knl is pretty
  much dead, so this will be the last succh target.  We will need to work
  wtih Cray and other vendors in the future.
2019-09-20 00:51:37 -07:00
Massimiliano Culpo
3c4322bf1a targets: Spack targets can now be fine-grained microarchitectures
Spack can now:

- label ppc64, ppc64le, x86_64, etc. builds with specific
  microarchitecture-specific names, like 'haswell', 'skylake' or
  'icelake'.

- detect the host architecture of a machine from /proc/cpuinfo or similar
  tools.

- Understand which microarchitectures are compatible with which (for
  binary reuse)

- Understand which compiler flags are needed (for GCC, so far) to build
  binaries for particular microarchitectures.

All of this is managed through a JSON file (microarchitectures.json) that
contains detailed auto-detection, compiler flag, and compatibility
information for specific microarchitecture targets.  The `llnl.util.cpu`
module implements a library that allows detection and comparison of
microarchitectures based on the data in this file.

The `target` part of Spack specs is now essentially a Microarchitecture
object, and Specs' targets can be compared for compatibility as well.
This allows us to label optimized binary packages at a granularity that
enables them to be reused on compatible machines.  Previously, we only
knew that a package was built for x86_64, NOT which x86_64 machines it
was usable on.

Currently this feature supports Intel, Power, and AMD chips. Support for
ARM is forthcoming.

Specifics:

- Add microarchitectures.json with descriptions of architectures

- Relaxed semantic of compiler's "target" attribute.  Before this change
  the semantic to check if a compiler could be viable for a given target
  was exact match. This made sense as the finest granularity of targets
  was architecture families.  As now we can target micro-architectures,
  this commit changes the semantic by interpreting as the architecture
  family what is stored in the compiler's "target" attribute. A compiler
  is then a viable choice if the target being concretized belongs to the
  same family. Similarly when a new compiler is detected the architecture
  family is stored in the "target" attribute.

- Make Spack's `cc` compiler wrapper inject target-specific flags on the
  command line

- Architecture concretization updated to use the same algorithm as
  compiler concretization

- Micro-architecture features, vendor, generation etc. are included in
  the package hash.  Generic architectures, such as x86_64 or ppc64, are
  still dumped using the name only.

- If the compiler for a target is not supported exit with an intelligible
  error message. If the compiler support is unknown don't try to use
  optimization flags.

- Support and define feature aliases (e.g., sse3 -> ssse3) in
  microarchitectures.json and on Microarchitecture objects. Feature
  aliases are defined in targets.json and map a name (the "alias") to a
  list of rules that must be met for the test to be successful. The rules
  that are available can be extended later using a decorator.

- Implement subset semantics for comparing microarchitectures (treat
  microarchitectures as a partial order, i.e. (a < b), (a == b) and (b <
  a) can all be false.

- Implement logic to automatically demote the default target if the
  compiler being used is too old to optimize for it. Updated docs to make
  this behavior explicit.  This avoids surprising the user if the default
  compiler is older than the host architecture.

This commit adds unit tests to verify the semantics of target ranges and
target lists in constraints. The implementation to allow target ranges
and lists is minimal and doesn't add any new type.  A more careful
refactor that takes into account the type system might be due later.

Co-authored-by: Gregory Becker <becker33.llnl.gov>
2019-09-20 00:51:37 -07:00
Gregory Becker
dfabf5d6b1 targets: first pass at target detection for linux
Add llnl.util.cpu_name, with initial support for detecting different
microarchitectures on Linux.  This also adds preliminary changes for
compiler support and variants to control the optimizatoin levels by
target.

This does not yet include translations of targets to particular
compilers; that is left to another PR.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2019-09-20 00:51:37 -07:00
Peter Scheibel
141a1648e6 implicit rpaths filtering (#12789)
* implicit_rpaths are now removed from compilers.yaml config and are always instantiated dynamically, this occurs one time in the build_environment module

* per-compiler list required libraries (e.g. libstdc++, libgfortran) and whitelist directories from rpaths including those libraries. Remove non-whitelisted implicit rpaths. Some libraries default for all compilers.

* reintroduce 'implicit_rpaths' as a config variable that can be used to disable Spack insertion of compiler RPATHs generated at build time.
2019-09-17 17:45:21 -05:00
Greg Becker
3b115fffb1 permissions: fix file permissions on intermediate install directories (#12399)
- mkdirp now takes arguments to allow it to properly set permissions on created directories.
- Two arguments (group and mode) set permissions for the leaf directory.
- Intermediate directories can inherit permissions from either the topmost existing directory (the parent) or the leaf.
2019-08-20 23:08:02 -07:00
Tamara Dahlgren
0ea6e0f817 features: Remove stage symlinks (#12072)
Fixes #11163

The goal of this work is to simplify stage directory structures by eliminating use of symbolic links. This means, among other things, that` $spack/var/spack/stage` will no longer be the core staging directory. Instead, the first accessible `config:build_stage` path will be used.

Spack will no longer automatically append `spack-stage` (or the like) to configured build stage directories so the onus of distinguishing the directory from other work -- so the other work is not automatically removed with a `spack clean` operation -- falls on the user.
2019-08-19 10:31:24 -07:00
albestro
3a026f1412 Fix #11240 (#11995)
* extends mkdirs with permissions for intermediate folders

Does not use os.makedirs mode parameter because its behavior is changed
with Python 3.7 (it ignores it for intermediate dirs), and moreover it
was not possible to set different modes for newly-created folders
and leaf folder.

reference:
- https://bugs.python.org/issue19930
- https://docs.python.org/3.7/library/os.html#os.makedirs

* comment mkdirp step easing code understanding

* revert mkdir to default for package metapath

since metapath is nested in package folder, there is no need
to specify permissions for intermediate folders because the prefix
already exists.

* comment create_install_directory package modes
2019-07-19 10:13:29 -05:00
Tim Fuller
5bc15b2d9a find_libraries searches lib and lib64 before prefix (#11958)
The default library search for a package checks the lib/ and lib64/
directories for libraries before the root prefix, in order to save
time when searching for libraries provided by externals (which e.g.
may have '/usr/' as their root).

This moves that logic into the "find_libraries" utility method so
packages implementing their own custom library search logic can
benefit from it.

This also updates packages which appear to be replicating this logic
exactly, replacing it with a single call to "find_libraries".
2019-07-12 17:46:47 -07:00
Carson Woods
76f1ee5f32 'spack compiler add' resolves relative path to absolute path (#11792)
Fixes #11782

Spack was not properly resolving relative paths to absolute paths
when a relative path was passed to "spack compiler add [PATH]".
Now, if provided a relative path, the absolute path is written to
compilers.yaml rather than the relative path.
2019-07-12 16:06:26 -07:00
Peter Scheibel
284ae9d1cc Resources: use expanded archive name by default (#11688)
For resources, it is desirable to use the expanded archive name of
the resource as the name of the directory when adding it to the root
staging area.

#11528 established 'spack-src' as the universal directory where
source files are placed, which also affected the behavior of
resources managed with Stages.

This adds a new property ('srcdir') to Stage to remember the name of
the expanded source directory, and uses this as the default name when
placing a resource directory in the root staging area.

This also:

* Ensures that downloaded sources are archived using the expanded
  archive name (otherwise Spack will not be able to determine the
  original directory name when using a cached archive).
* Updates working_dir context manager to guarantee restoration of
  original working directory when an exception occurs
* Adds a "temp_cwd" context manager which creates a temporary
  directory and sets it as the working directory
2019-06-20 11:09:31 -07:00
Adam J. Stewart
4e812090c0 Add additional common C++ and Fortran header file extensions (#11600)
* Add additional common C++ and Fortran header file extensions

* Add .hxx extension

* Add .txx and .tcc extensions

* Add .icc extension
2019-06-11 20:13:55 -04:00
Massimiliano Culpo
6d56d45454 Compiler search uses a pool of workers (#10190)
- spack.compilers.find_compilers now uses a multiprocess.pool.ThreadPool to execute
  system commands for the detection of compiler versions.

- A few memoized functions have been introduced to avoid poking the filesystem multiple
  times for the same results.

- Performance is much improved, and Spack no longer fork-bombs the system when doing a `compiler find`
2019-06-07 09:57:26 -07:00
Tamara Dahlgren
8e3fd3f7c2 tty: make tty.* print exception types
- make tty.msg, tty.info, etc. print the exception type and stringified
  message if the message argument is an exception.

- simplify parts of the code that call tty.debug(str(e))

- add extra tty.debug statements in places where exceptions were
  previously ignored
2019-06-05 22:41:28 -07:00
Todd Gamblin
d6f2ff1426 link_tree: add option to merge link trees with relative targets
- previous version of link trees would only do absolute symlinks

- this version can do relative links using merge(relative=True)
2019-05-26 18:23:44 -07:00
Todd Gamblin
6380f1917a commands: Add --header and --update options to spack commands
The Spack documentation currently hard-codes some functionality in
`conf.py`, which makes the doc build less "pluggable" for things like
localized doc builds.

In particular, we unconditionally generate an index of commands and a
package list as part of the docs, but those should really only be done if
things are not up to date.

This commit does the following:

- Add `--header` option to `spack commands` so that it can do the work of
  prepending text to its output.

- Add `--update FILE` option to `spack commands` that makes it generate a
  new command index *only* if FILE is out of date w.r.t. commands in the
  Spack source.

- Simplify code in `conf.py` to use these options and only update the
  command index when needed.
2019-05-26 18:23:44 -07:00
Peter Scheibel
ea1de6b941 Maintain a view for an environment (#10017)
Environments are nowm by default, created with views.  When activated, if an environment includes a view, this view will be added to `PATH`, `CPATH`, and other shell variables to expose the Spack environment in the user's shell.

Example:

```
spack env create e1 #by default this will maintain a view in the directory Spack maintains for the env
spack env create e1 --with-view=/abs/path/to/anywhere
spack env create e1 --without-view
```

The `spack.yaml` manifest file now looks like this:

```
spack:
  specs:
  - python
  view: true #or false, or a string
```

These commands can be used to control the view configuration for the active environment, without hand-editing the `spack.yaml` file:

```
spack env view enable
spack env view envable /abs/path/to/anywhere
spack env view disable
```

Views are automatically updated when specs are installed to an environment. A view only maintains one copy of any package. An environment may refer to a package multiple times, in particular if it appears as a dependency. This PR establishes a prioritization for which environment specs are added to views: a spec has higher priority if it was concretized first. This does not necessarily exactly match the order in which specs were added, for example, given `X->Z` and `Y->Z'`:

```
spack env activate e1
spack add X
spack install Y # immediately concretizes and installs Y and Z'
spack install # concretizes X and Z
```

In this case `Z'` will be favored over `Z`. 

Specs in the environment must be concrete and installed to be added to the view, so there is another minor ordering effect: by default the view maintained for the environment ignores file conflicts between packages. If packages are not installed in order, and there are file conflicts, then the version chosen depends on the order.

Both ordering issues are avoided if `spack install`/`spack add` and `spack install <spec>` are not mixed.
2019-04-10 16:00:12 -07:00
Massimiliano Culpo
e3f00750e8 Update llnl.util.lang.memoized so that Sphinx can extract signature (#11055)
Replace the original implementation of the "memoized" decorator with
an implementation that exposes the docstring and arguments of the
wrapped function. This is achieved using functools.wraps.
2019-03-29 17:11:44 -07:00
Massimiliano Culpo
0a006351c8 Spack can be extended with external commands (#8612)
This provides a mechanism to implement a new Spack command in a
separate directory, and with a small configuration change point Spack
to the new command.

To register the command, the directory must be added to the
"extensions" section of config.yaml. The command directory name must
have the prefix "spack-", and have the following layout:

  spack-X/
    pytest.ini #optional, for testing
    X/
	  cmd/
	    name-of-command1.py
	    name-of-command2.py
	    ...
    tests/ #optional
      conftest.py
	  test_name-of-command1.py
    templates/ #optional jinja templates, if needed

And in config.yaml:

  config:
    extensions:
      - /path/to/spack-X

If the extension includes tests, you can run them via spack by adding
the --extension option, like "spack test --extension=X"
2019-03-28 16:56:36 -07:00
Michael Kuhn
a1c91f3c07 Fix find_headers to also look for C++ headers and Fortran modules (#10798)
Currently, only C headers are considered, causing build failures for
packages depending on, e.g., netcdf-fortran and xerces-c. Additionally,
the regex used to look for the include path component did not consider
word boundaries, causing false matches.
2019-03-08 21:06:22 -06:00
Massimiliano Culpo
e3af8ed454 Added a sub-command to show if packages are relocatable (#9199)
* Added the `spack buildcache preview` sub-command

This is similar to `spack spec -I` but highlights which nodes in a DAG
are relocatable and which are not.

spec.tree has been generalized a little to accept a status function,
instead of always showing the install status

The current implementation works only for ELF, and needs to be
generalized to other platforms.

* Added a test to check if an executable is relocatable or not

This test requires a few commands to be present in the environment.
Currently it will run only under python 3.7 (which uses Xenial instead
of Trusty).

* Added tests for the 'buildcache preview' command.

* Fixed codebase after rebase

* Fixed the list of apt addons for Python 3.7 in travis.yaml

* Only check ELF executables and shared libraries. Skip checking virtual or external packages. (#229)

* Fixed flake8 issues

* Add handling for macOS mach binaries (#231)
2019-02-28 15:36:47 -06:00
Massimiliano Culpo
42386dbe94 Use Package.headers for -I options (#10623)
This restores the use of Package.headers when computing -I options
for building a package that was added in #8136 and reverted in
#10604. #8136 used utility logic that located all header files in
an installation prefix, and calculated the -I options as the
immediate roots containing those header files.

In some cases, for a package containing a directory structure like

  prefix/
    include/
	  ex1.h
	  subdir/
	    ex2.h

dependents may expect to include ex2.h relative to 'include', and
adding 'prefix/include/subdir' as a -I was causing errors,
in particular if ex2.h has the same name as a system header.

This updates header utility logic to by default return the base
"include" directory when it exists, rather than subdirectories.
It also makes it possible for package implementers to override
Package.headers to return the subdirectory when it is required
(for example with libxml2).
2019-02-26 12:42:30 -06:00
sknigh
a76c50d1ee Allow tty output to be timestamped (#10554)
* Spack debug output now includes microsecond-granularity timestamps.
* Timestamps can also be enabled with the `--timestamp` command line argument.
2019-02-13 10:14:35 -08:00
Matthias Wolf
861dd06bd1 enh: allow time like HH:MM in date strings. (#10034) 2019-02-13 11:05:00 +01:00
Todd Gamblin
6f50cd52ed copyright: update license headers for 2013-2019 copyright. 2019-01-01 00:44:28 -08:00
Massimiliano Culpo
3b8b13809e Improve validation of modules.yaml (#9878)
This PR improves the validation of `modules.yaml` by introducing a custom validator that checks if an attribute listed in `properties` or `patternProperties` is a valid spec. This new check applied to the test case in #9857 gives:

```console
$ spack install szip
==> Error: /home/mculpo/.spack/linux/modules.yaml:5: "^python@2.7@" is an invalid spec [Invalid version specifier]
```

Details: 
* Moved the set-up of a custom validator class to spack.schema
  * In Spack we use `jsonschema` to validate configuration files 
    against a schema. We also need custom validators to enforce
    writing default values within "properties" or "patternProperties"
    attributes.

  * Currently, validators were customized at the place of use and with the
    recent introduction of environments that meant we were setting-up and
    using 2 different validator classes in two different modules.

  * This commit moves the set-up of a custom validator class in the
    `spack.schema` module and refactors the code in `spack.config` and
    `spack.environments` to use it.

* Added a custom validator to check if an attribute is a valid spec
  * Added a custom validator that can be used on objects, which yields an
    error if the attribute is not a valid spec.

* Updated the schema for modules.yaml

* Updated modules.yaml to fix a few inconsistencies:
  - a few attributes were not tested properly using 'anyOf'
  - suffixes has been updated to also check that the attribute is a spec
  - hierarchical_scheme has been updated to hierarchy

* Removed $ref from every schema
  * $ref is not composable or particularly legible
  * Use python dicts and regular old variables instead.
2019-01-01 00:11:49 -08:00
Todd Gamblin
a41bce2148 fix bad regular expressions and docstrings with '\' 2018-11-09 00:31:24 -08:00
Todd Gamblin
a1818f971f env: environments can be named or created in directories
- `spack env create <name>` works as before

- `spack env create <path>` now works as well -- environments can be
  created in their own directories outside of Spack.

- `spack install` will look for a `spack.yaml` file in the current
  directory, and will install the entire project from the environment

- The Environment class has been refactored so that it does not depend on
  the internal Spack environment root; it just takes a path and operates
  on an environment in that path (so internal and external envs are
  handled the same)

- The named environment interface has been hoisted to the
  spack.environment module level.

- env.yaml is now spack.yaml in all places.  It was easier to go with one
  name for these files than to try to handle logic for both env.yaml and
  spack.yaml.
2018-11-09 00:31:24 -08:00
Todd Gamblin
6af5dfbbc2 config: allow env.yaml to contain configuration in a single file
- add `SingleFileScope` to configuration, which allows us to pull config
  sections from a single file.

- update `env.yaml` and tests to ensure that the env.yaml schema works
  when pulling configurtion from the env file.
2018-11-09 00:31:24 -08:00
Todd Gamblin
a1f90d5b8c bugfix: use OS default permissions for mkdirp when mode is not provided (#9604)
- #8773 made the default mode 0o777, which is what's documented but
   mkdirp actually takes the OS default or umask by default

- revert to the Python default by default, and only set the mode when
  asked explicitly.
2018-10-24 11:36:41 -07:00
Todd Gamblin
eea786f4e8 relicense: replace LGPL headers with Apache-2.0/MIT SPDX headers
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
  - core and most packages are (Apache-2.0 OR MIT)
  - a very small number of remaining packages are LGPL-2.1-only
2018-10-17 14:42:06 -07:00
Greg Becker
d1a5113cfe permissions: add permission configuration to packages.yaml (#8773)
Spack can now be configured to assign permissions to the files installed by a package.

In the `packages.yaml` file under `permissions`, the attributes `read`, `write`, and `group` control the package permissions. These attributes can be set per-package, or for all packages under `all`. If permissions are set under `all` and for a specific package, the package-specific settings take precedence.  The `read` and `write` attributes take one of `user`, `group`, and `world`.

   packages:
    all:
      permissions:
        write: group
        group: spack
    my_app:
      permissions:
        read: group
        group: my_team
2018-10-11 14:29:07 -07:00
Peter Scheibel
28c0dd9148 Increase and customize lock timeouts (#9219)
Fixes #9166

This is intended to reduce errors related to lock timeouts by making
the following changes:

* Improves error reporting when acquiring a lock fails (addressing
  #9166) - there is no longer an attempt to release the lock if an
  acquire fails
* By default locks taken on individual packages no longer have a
  timeout. This allows multiple spack instances to install overlapping
  dependency DAGs. For debugging purposes, a timeout can be added by
  setting 'package_lock_timeout' in config.yaml
* Reduces the polling frequency when trying to acquire a lock, to
  reduce impact in the case where NFS is overtaxed. A simple
  adaptive strategy is implemented, which starts with a polling
  interval of .1 seconds and quickly increases to .5 seconds
  (originally it would poll up to 10^5 times per second).
  A test is added to check the polling interval generation logic.
* The timeout for Spack's whole-database lock (e.g. for managing
  information about installed packages) is increased from 60s to
  120s
* Users can configure the whole-database lock timeout using the
  'db_lock_timout' setting in config.yaml

Generally, Spack locks (those created using spack.llnl.util.lock.Lock)
now have no timeout by default

This does not address implementations of NFS that do not support file
locking, or detect cases where services that may be required
(nfslock/statd) aren't running.

Users may want to be able to more-aggressively release locks when
they know they are the only one using their Spack instance, and they
encounter lock errors after a crash (e.g. a remote terminal disconnect
mentioned in #8915).
2018-09-25 18:58:51 -07:00
scheibelp
638cc64571 install_tree: symlink handling and add 'ignore' option (#9019)
Fixes #9001

#8289 added support for install_tree and copy_tree to merge into an existing
directory structure. However, it did not properly handle relative symlinks and
also removed support for the 'ignore' keyword. Additionally, some of the tests
were overly-strict when checking the permissions on the copied files.

This updates the install_tree/copy_tree methods and their tests:

* copy_tree/install_tree now preserve relative link targets (if the symlink in the
  source directory structure is relative, the symlink created in the destination
  will be relative)
* Added support for 'ignore' argument back to copy_tree/install_tree (removed
  in #8289). It is no longer the object output by shutil.ignore_patterns: you pass a
  function that accepts a path relative to the source and returns whether that
  path should be copied.
* The openfoam packages (currently the only ones making use of the 'ignore'
  argument) are updated for the new API
* When a symlink target is absolute, copy_tree and install_tree now rewrite the
  source prefix to be the destination prefix
* copy_tree tests no longer check permissions: copy_tree doesn't enforce
  anything about permissions so its tests don't check for that
* install_tree tests no longer check for exact permission matching since it can add
  file permissions
2018-08-17 22:08:38 -04:00
Adam J. Stewart
73c978ddd9 install_tree, copy_tree can install into existing directory structures (#8289)
Replace use of `shutil.copytree` with `copy_tree` and `install_tree` functions in `llnl.util.filesystem`.

- `copy_tree` copies without setting permissions.  It should be used to copy files around in the build directory.
- `install_tree` copies files and sets permissions.  It should be used to copy files into the installation directory.
- `install` and `copy` are analogous single-file functions.
- add more extensive tests for these functions
- update packages to use these functions.
2018-08-15 09:30:09 -07:00
cedricchevalier19
3301e21f06 Fix performance issue when compiling. (#8828)
* Fix performance issue when compiling.

Spack was doing active wait when compiling, spoiling one core.
My fix consists in not setting any timeout for select, instead of
the previous 0 second.

* Fix comments about select.select timeout
2018-08-07 09:13:07 -07:00
Todd Gamblin
43c77ac3cb libdwarf: remove use of hide_files()
- This was a nasty workaround due to the way our compiler wrappers used
  to work.  We don't want to have to modify our elfutils installation to
  install libdwarf.

- Since cd9691de5, we no longer need this because the package will always
  come before dependencies in our include order.
2018-08-02 19:00:52 -07:00
Todd Gamblin
fdcaf5c4c8 bugfix: fix spack spec --yaml
- repo membership test was broken by the refactor of spack/__init__.py

- refactor singleton so that 'spec in repo' works again for `spack.repo.path`

- fix spec command and add basic tests for `spack spec` and `spack spec --yaml`
2018-07-24 11:25:34 -07:00
Todd Gamblin
2b0d944341 locks: fix bug when creating lockfiles in the current directory.
- Fixes a bug in `llnl.util.lock`

- Locks in the current directory would fail because the parent directory
  was the empty string.

- Fix this and return '.' for the parent of locks in the current
  directory.
2018-07-21 10:39:47 -07:00
Todd Gamblin
20e4038a72 style: make core comply with pep8-naming 2018-07-19 00:41:36 -07:00
Todd Gamblin
650786c812 locks: improve errors and permission checking
- Clean up error messages for when a lock can't be created, or when an
  exclusive (write) lock can't be taken on a file.

- Add a number of subclasses of LockError to distinguish timeouts from
  permission issues.

- Add an explicit check to prevent the user from taking a write lock on a
  read-only file.
  - We had a check for this for when we try to *upgrade* a lock on an RO
    file, but not for an initial write lock attempt.

- Add more tests for different lock permission scenarios.
2018-07-12 19:59:53 +02:00
Todd Gamblin
ab794fa741 locks: llnl.util.lock now only writes host info when in debug mode
- write locks previously wrote information about the lock holder (host
  and pid), and read locks woudl read this in.

- This is really only for debugging, so only enable it then

- add some tests that target debug info, and improve multiproc lock test
  output
2018-07-12 19:59:53 +02:00
Todd Gamblin
7626ec4579 refactor: move spack.util.multiproc to llnl.util.multiproc
- multiproc doesn't depend on Spack
- llnl.util.lock test uses it, but shouldn't use parts of Spack.
2018-07-12 19:59:53 +02:00
scheibelp
3560f6dbe9 views: packages can customize how they're added to views (#7152)
Functional updates:

- `python` now creates a copy of the `python` binaries when it is added
  to a view

- Python extensions (packages which subclass `PythonPackage`) rewrite
  their shebang lines to refer to python in the view

- Python packages in the same namespace will not generate conflicts if
  both have `...lib/site-packages/namespace-example/__init__.py`

  - These `__init__` files will also remain when removing any package in
    the namespace until the last package in the namespace is removed


Generally (Updated 2/16):

- Any package can define `add_files_to_view` to customize how it is added
  to a view (and at the moment custom definitions are included for
  `python` and `PythonPackage`)

  - Likewise any package can define `remove_files_from_view` to customize
    which files are removed (e.g. you don't always want to remove the
    namespace `__init__`)

- Any package can define `view_file_conflicts` to customize what it
  considers a merge conflict

- Global activations are handled like views (where the view root is the
  spec prefix of the extendee)

  - Benefit: filesystem-management aspects of activating extensions are
    now placed in views (e.g. now one can hardlink a global activation)

  - Benefit: overriding `Package.activate` is more straightforward (see
    `Python.activate`)

  - Complication: extension packages which have special-purpose logic
    *only* when activated outside of the extendee prefix must check for
    this in their `add_files_to_view` method (see `PythonPackage`)

- `LinkTree` is refactored to have separate methods for copying a
  directory structure and for copying files (since it was found that
  generally packages may want to alter how files are copied but still
  wanted to copy directories in the same way)


TODOs (updated 2/20):

- [x] additional testing (there is some unit testing added at this point
  but more would be useful)

- [x] refactor or reorganize `LinkTree` methods: currently there is a
  separate set of methods for replicating just the directory structure
  without the files, and a set for replicating everything

- [x] Right now external views (i.e. those not used for global
  activations) call `view.add_extension`, but global activations do not
  to avoid some extra work that goes into maintaining external views. I'm
  not sure if addressing that needs to be done here but I'd like to
  clarify it in the comments (UPDATE: for now I have added a TODO and in
  my opinion this can be merged now and the refactor handled later)

- [x] Several method descriptions (e.g. for `Package.activate`) are out
  of date and reference a distinction between global activations and
  views, they need to be updated

- [x] Update aspell package activations
2018-06-26 16:14:05 -07:00
Todd Gamblin
a48bdfaf1d bugfix: fix macos incompatibility in lock test (#8573)
- Spack was assuming that a group with gid == current uid would always exist.
- This was breaking the travis build for macos.

- also fix issue with the DB tarball test finding coverage filesx
2018-06-26 14:51:02 +02:00