Commit Graph

296 Commits

Author SHA1 Message Date
Greg Becker
230c6aa326
cpu: fix clang flags for generic x86_64 (#13491)
* cpu: differentiate flags used for pristine LLVM vs. Apple's version
2019-10-30 17:16:13 -05:00
Chris Green
77af4684aa Improvements to detection of AMD architectures. (#13407)
New entry for K10 microarchitecture.

Reorder Zen* microarchitectures to avoid triggering as k10.

Remove some desktop-specific flags that were preventing Opteron Bulldozer/Piledriver/Steamroller/Excavator CPUs from being recognized as such.

Remove one or two flags which weren't produced in /proc/cpuinfo on older OS (RHEL6 and friends).
2019-10-24 15:48:54 -07:00
Chris Green
0913328812 Correctly identify Skylake CPUs on Darwin. (#13377)
* Correctly identify Skylake CPUs on Darwin.

* Add a test for haswell on Mojave.
2019-10-24 12:44:58 -05:00
Massimiliano Culpo
b14f18acda microarchitectures: look in /sbin and /usr/sbin for sysctl (#13365)
This PR ensures that on Darwin we always append /sbin and /usr/sbin to PATH, if they are not already present, when looking for sysctl.

* Make sure we look into /sbin and /usr/sbin for sysctl
* Refactor sysctl for better readability
* Remove marker to make test pass
2019-10-22 21:42:38 -07:00
Massimiliano Culpo
8808207ddf Fixed optimization flags support for old GCC versions (#13362)
These changes update our gcc microarchitecture descriptions based on manuals found here https://gcc.gnu.org/onlinedocs/ and assuming that new architectures are not added during patch releases.
2019-10-22 21:40:45 -07:00
Massimiliano Culpo
cfbdd2179e microarchitectures: add optimization flags for Intel compilers (#13345)
* Added optimization flags for Intel compilers with Intel CPUs
* Added optimization flags for Intel compilers with AMD CPUs
2019-10-22 00:33:59 -07:00
Massimiliano Culpo
498f448ef3 microarchitectures: fix custom compiler versions (#13222)
Custom string versions for compilers were raising a ValueError on
conversion to int. This commit fixes the behavior by trying to detect
the underlying compiler version when in presence of a custom string
version.

* Refactor code that deals with custom versions for better readability
* Partition version components with a regex
* Fix semantic of custom compiler versions with a suffix
* clang@x.y-apple has been special-cased
* Add unit tests
2019-10-21 10:24:57 -07:00
Massimiliano Culpo
41fb0395a6 Microarchitecture specific optimizations for LLVM (#13250)
* Added architecture specific optimization flags for Clang / LLVM
* Disallow compiler optimizations for mixed toolchains
    * We emit a warning when building for a mixed toolchain
* Fixed issues with suffixed versions of compilers; Apple's Clang will, 
    for the time being, fall back on x86-64 for every compilation.
2019-10-19 13:19:29 -07:00
Michael Kuhn
ffe87ed49f filter_file: fix multiple invocations on the same file (#13234)
Since the backup file is only created on the first invocation, it will
contain the original file without any modifications. Further invocations
will then read the backup file, effectively reverting prior invocations.

This can be reproduced easily by trying to install likwid, which will
try to install into /usr/local. Work around this by creating a temporary
file to read from.
2019-10-16 15:15:24 -07:00
Tamara Dahlgren
1ef71376f2 Bugfix: stage directory permissions and cleaning (#12733)
* This updates stage names to use "spack-stage-" as a prefix.
  This avoids removing non-Spack directories in "spack clean" as
  c141e99 did (in this case so long as they don't contain the
  prefix "spack-stage-"), and also addresses a follow-up issue
  where Spack stage directories were not removed.
* Spack now does more-stringent checking of expected permissions for
  staging directories. For a given stage root that includes a user
  component, all directories before the user component that are
  created by Spack are expected to match the permissions of their
  parent; the user component and all deeper directories are expected
  to be accessible to the user (read/write/execute).
2019-10-16 14:55:37 -07:00
Greg Becker
94e80933f0 Feature: installed file verification (#12841)
This feature generates a verification manifest for each installed
package and provides a command, "spack verify", which can be used to
compare the current file checksums/permissions with those calculated
at installed time.

Verification includes

* Checksums of files
* File permissions
* Modification time
* File size

Packages installed before this PR will be skipped during verification.
To verify such a package you must reinstall it.

The spack verify command has three modes.

* With the -a,--all option it will check every installed package.
* With the -f,--files option, it will check some specific files,
  determine which package they belong to, and confirm that they have
  not been changed.
* With the -s,--specs option or by default, it will check some
  specific packages that no files havae changed.
2019-10-15 14:24:52 -07:00
Massimiliano Culpo
5cd28847e8 filter_file uses "surrogateescape" error handling (#12765)
From Python docs:
--
'surrogateescape' will represent any incorrect bytes as code points in
the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These
private code points will then be turned back into the same bytes when
the surrogateescape error handler is used when writing data. This is
useful for processing files in an unknown encoding.
--

This will allow us to process files with unknown encodings.

To accommodate the case of self-extracting bash scripts, filter_file
can now stop filtering text input if a certain marker is found. The
marker must be passed at call time via the "stop_at" function argument.
At that point the file will be reopened in binary mode and copied
verbatim.

* use "surrogateescape" error handling to ignore unknown chars
* permit to stop filtering if a marker is found
* add unit tests for non-ASCII and mixed text/binary files
2019-10-14 20:35:14 -07:00
Massimiliano Culpo
8dd95c1705 Fixed options to compile generic code on ppc64 and ppc64le 2019-10-11 21:20:28 -07:00
Massimiliano Culpo
b07460ab5f Added NEON to the list of features required for the aarch64 family
Both floating-point and NEON are required in all standard ARMv8
implementations. Theoretically though specialized markets can support
no NEON or floating-point at all. Source:

https://developer.arm.com/docs/den0024/latest/aarch64-floating-point-and-neon

On the other hand the base procedure call standard for Aarch64
"assumes the availability of the vector registers for passing
floating-point and SIMD arguments". Further "the Arm 64-bit
architecture defines two mandatory register banks: a general-purpose
register bank which can be used for scalar integer processing and
pointer arithmetic; and a SIMD and Floating-Point register bank".
Source:

https://developer.arm.com/docs/ihi0055/latest/procedure-call-standard-for-the-arm-64-bit-architecture

This makes customization of Aarch64 with no NEON instruction set
available so unlikely that we can consider them a feature of the
generic family.
2019-10-10 16:24:36 -07:00
Massimiliano Culpo
78577c0a90
Generic x86_64 code compiled with GCC uses non deprecated mtune flags (#13022)
fixes #12928
2019-10-03 10:31:03 +02:00
Massimiliano Culpo
9117dfd118 Add all the 'generic' architectures that are mentioned in recipes (#12958)
LLVM, mesa and other packages check for these generic
microarchitectures. One solution is to let Spack know they exist.
2019-09-28 21:47:05 -07:00
Adam J. Stewart
065cbe89fe Fix "specific target" detection in Python 3 (#12906)
The output of subprocess.check_output is a byte string in Python 3. This causes dictionary lookup to fail later on.

A try-except around this function prevented this error from being noticed. Removed this so that more errors can propagate out.
2019-09-24 09:47:54 -07:00
Massimiliano Culpo
2468ccee58 AMD: fix architecture hierarchy (zen) (#12913)
* microarchitectures: zen starts from x86_64, not from excavator
* Unit tests: fixed a test that is wrong with the new modeling
* microarchitectures: fixed features and inheritance for 15h family

bulldozer doesn't inherit from barcelona (10h) + added xop, lwp and tbm
instruction sets to the 15h family (it distinguish the family from 17h)
2019-09-23 21:54:13 -07:00
Gregory Becker
c43f105359 targets: add mic_knl target to microarchitectures.json
- This is needed to support Cray machines -- we need an architecture
  mic_knl > x86_64

- We used Cray's naming scheme for this target to make it work seamlessly
  with the module-based detection sccheme on Cray.  mic_knl is pretty
  much dead, so this will be the last succh target.  We will need to work
  wtih Cray and other vendors in the future.
2019-09-20 00:51:37 -07:00
Massimiliano Culpo
3c4322bf1a targets: Spack targets can now be fine-grained microarchitectures
Spack can now:

- label ppc64, ppc64le, x86_64, etc. builds with specific
  microarchitecture-specific names, like 'haswell', 'skylake' or
  'icelake'.

- detect the host architecture of a machine from /proc/cpuinfo or similar
  tools.

- Understand which microarchitectures are compatible with which (for
  binary reuse)

- Understand which compiler flags are needed (for GCC, so far) to build
  binaries for particular microarchitectures.

All of this is managed through a JSON file (microarchitectures.json) that
contains detailed auto-detection, compiler flag, and compatibility
information for specific microarchitecture targets.  The `llnl.util.cpu`
module implements a library that allows detection and comparison of
microarchitectures based on the data in this file.

The `target` part of Spack specs is now essentially a Microarchitecture
object, and Specs' targets can be compared for compatibility as well.
This allows us to label optimized binary packages at a granularity that
enables them to be reused on compatible machines.  Previously, we only
knew that a package was built for x86_64, NOT which x86_64 machines it
was usable on.

Currently this feature supports Intel, Power, and AMD chips. Support for
ARM is forthcoming.

Specifics:

- Add microarchitectures.json with descriptions of architectures

- Relaxed semantic of compiler's "target" attribute.  Before this change
  the semantic to check if a compiler could be viable for a given target
  was exact match. This made sense as the finest granularity of targets
  was architecture families.  As now we can target micro-architectures,
  this commit changes the semantic by interpreting as the architecture
  family what is stored in the compiler's "target" attribute. A compiler
  is then a viable choice if the target being concretized belongs to the
  same family. Similarly when a new compiler is detected the architecture
  family is stored in the "target" attribute.

- Make Spack's `cc` compiler wrapper inject target-specific flags on the
  command line

- Architecture concretization updated to use the same algorithm as
  compiler concretization

- Micro-architecture features, vendor, generation etc. are included in
  the package hash.  Generic architectures, such as x86_64 or ppc64, are
  still dumped using the name only.

- If the compiler for a target is not supported exit with an intelligible
  error message. If the compiler support is unknown don't try to use
  optimization flags.

- Support and define feature aliases (e.g., sse3 -> ssse3) in
  microarchitectures.json and on Microarchitecture objects. Feature
  aliases are defined in targets.json and map a name (the "alias") to a
  list of rules that must be met for the test to be successful. The rules
  that are available can be extended later using a decorator.

- Implement subset semantics for comparing microarchitectures (treat
  microarchitectures as a partial order, i.e. (a < b), (a == b) and (b <
  a) can all be false.

- Implement logic to automatically demote the default target if the
  compiler being used is too old to optimize for it. Updated docs to make
  this behavior explicit.  This avoids surprising the user if the default
  compiler is older than the host architecture.

This commit adds unit tests to verify the semantics of target ranges and
target lists in constraints. The implementation to allow target ranges
and lists is minimal and doesn't add any new type.  A more careful
refactor that takes into account the type system might be due later.

Co-authored-by: Gregory Becker <becker33.llnl.gov>
2019-09-20 00:51:37 -07:00
Gregory Becker
dfabf5d6b1 targets: first pass at target detection for linux
Add llnl.util.cpu_name, with initial support for detecting different
microarchitectures on Linux.  This also adds preliminary changes for
compiler support and variants to control the optimizatoin levels by
target.

This does not yet include translations of targets to particular
compilers; that is left to another PR.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2019-09-20 00:51:37 -07:00
Peter Scheibel
141a1648e6 implicit rpaths filtering (#12789)
* implicit_rpaths are now removed from compilers.yaml config and are always instantiated dynamically, this occurs one time in the build_environment module

* per-compiler list required libraries (e.g. libstdc++, libgfortran) and whitelist directories from rpaths including those libraries. Remove non-whitelisted implicit rpaths. Some libraries default for all compilers.

* reintroduce 'implicit_rpaths' as a config variable that can be used to disable Spack insertion of compiler RPATHs generated at build time.
2019-09-17 17:45:21 -05:00
Greg Becker
3b115fffb1 permissions: fix file permissions on intermediate install directories (#12399)
- mkdirp now takes arguments to allow it to properly set permissions on created directories.
- Two arguments (group and mode) set permissions for the leaf directory.
- Intermediate directories can inherit permissions from either the topmost existing directory (the parent) or the leaf.
2019-08-20 23:08:02 -07:00
Tamara Dahlgren
0ea6e0f817
features: Remove stage symlinks (#12072)
Fixes #11163

The goal of this work is to simplify stage directory structures by eliminating use of symbolic links. This means, among other things, that` $spack/var/spack/stage` will no longer be the core staging directory. Instead, the first accessible `config:build_stage` path will be used.

Spack will no longer automatically append `spack-stage` (or the like) to configured build stage directories so the onus of distinguishing the directory from other work -- so the other work is not automatically removed with a `spack clean` operation -- falls on the user.
2019-08-19 10:31:24 -07:00
albestro
3a026f1412 Fix #11240 (#11995)
* extends mkdirs with permissions for intermediate folders

Does not use os.makedirs mode parameter because its behavior is changed
with Python 3.7 (it ignores it for intermediate dirs), and moreover it
was not possible to set different modes for newly-created folders
and leaf folder.

reference:
- https://bugs.python.org/issue19930
- https://docs.python.org/3.7/library/os.html#os.makedirs

* comment mkdirp step easing code understanding

* revert mkdir to default for package metapath

since metapath is nested in package folder, there is no need
to specify permissions for intermediate folders because the prefix
already exists.

* comment create_install_directory package modes
2019-07-19 10:13:29 -05:00
Tim Fuller
5bc15b2d9a find_libraries searches lib and lib64 before prefix (#11958)
The default library search for a package checks the lib/ and lib64/
directories for libraries before the root prefix, in order to save
time when searching for libraries provided by externals (which e.g.
may have '/usr/' as their root).

This moves that logic into the "find_libraries" utility method so
packages implementing their own custom library search logic can
benefit from it.

This also updates packages which appear to be replicating this logic
exactly, replacing it with a single call to "find_libraries".
2019-07-12 17:46:47 -07:00
Carson Woods
76f1ee5f32 'spack compiler add' resolves relative path to absolute path (#11792)
Fixes #11782

Spack was not properly resolving relative paths to absolute paths
when a relative path was passed to "spack compiler add [PATH]".
Now, if provided a relative path, the absolute path is written to
compilers.yaml rather than the relative path.
2019-07-12 16:06:26 -07:00
Peter Scheibel
284ae9d1cc
Resources: use expanded archive name by default (#11688)
For resources, it is desirable to use the expanded archive name of
the resource as the name of the directory when adding it to the root
staging area.

#11528 established 'spack-src' as the universal directory where
source files are placed, which also affected the behavior of
resources managed with Stages.

This adds a new property ('srcdir') to Stage to remember the name of
the expanded source directory, and uses this as the default name when
placing a resource directory in the root staging area.

This also:

* Ensures that downloaded sources are archived using the expanded
  archive name (otherwise Spack will not be able to determine the
  original directory name when using a cached archive).
* Updates working_dir context manager to guarantee restoration of
  original working directory when an exception occurs
* Adds a "temp_cwd" context manager which creates a temporary
  directory and sets it as the working directory
2019-06-20 11:09:31 -07:00
Adam J. Stewart
4e812090c0
Add additional common C++ and Fortran header file extensions (#11600)
* Add additional common C++ and Fortran header file extensions

* Add .hxx extension

* Add .txx and .tcc extensions

* Add .icc extension
2019-06-11 20:13:55 -04:00
Massimiliano Culpo
6d56d45454 Compiler search uses a pool of workers (#10190)
- spack.compilers.find_compilers now uses a multiprocess.pool.ThreadPool to execute
  system commands for the detection of compiler versions.

- A few memoized functions have been introduced to avoid poking the filesystem multiple
  times for the same results.

- Performance is much improved, and Spack no longer fork-bombs the system when doing a `compiler find`
2019-06-07 09:57:26 -07:00
Tamara Dahlgren
8e3fd3f7c2 tty: make tty.* print exception types
- make tty.msg, tty.info, etc. print the exception type and stringified
  message if the message argument is an exception.

- simplify parts of the code that call tty.debug(str(e))

- add extra tty.debug statements in places where exceptions were
  previously ignored
2019-06-05 22:41:28 -07:00
Todd Gamblin
d6f2ff1426 link_tree: add option to merge link trees with relative targets
- previous version of link trees would only do absolute symlinks

- this version can do relative links using merge(relative=True)
2019-05-26 18:23:44 -07:00
Todd Gamblin
6380f1917a commands: Add --header and --update options to spack commands
The Spack documentation currently hard-codes some functionality in
`conf.py`, which makes the doc build less "pluggable" for things like
localized doc builds.

In particular, we unconditionally generate an index of commands and a
package list as part of the docs, but those should really only be done if
things are not up to date.

This commit does the following:

- Add `--header` option to `spack commands` so that it can do the work of
  prepending text to its output.

- Add `--update FILE` option to `spack commands` that makes it generate a
  new command index *only* if FILE is out of date w.r.t. commands in the
  Spack source.

- Simplify code in `conf.py` to use these options and only update the
  command index when needed.
2019-05-26 18:23:44 -07:00
Peter Scheibel
ea1de6b941 Maintain a view for an environment (#10017)
Environments are nowm by default, created with views.  When activated, if an environment includes a view, this view will be added to `PATH`, `CPATH`, and other shell variables to expose the Spack environment in the user's shell.

Example:

```
spack env create e1 #by default this will maintain a view in the directory Spack maintains for the env
spack env create e1 --with-view=/abs/path/to/anywhere
spack env create e1 --without-view
```

The `spack.yaml` manifest file now looks like this:

```
spack:
  specs:
  - python
  view: true #or false, or a string
```

These commands can be used to control the view configuration for the active environment, without hand-editing the `spack.yaml` file:

```
spack env view enable
spack env view envable /abs/path/to/anywhere
spack env view disable
```

Views are automatically updated when specs are installed to an environment. A view only maintains one copy of any package. An environment may refer to a package multiple times, in particular if it appears as a dependency. This PR establishes a prioritization for which environment specs are added to views: a spec has higher priority if it was concretized first. This does not necessarily exactly match the order in which specs were added, for example, given `X->Z` and `Y->Z'`:

```
spack env activate e1
spack add X
spack install Y # immediately concretizes and installs Y and Z'
spack install # concretizes X and Z
```

In this case `Z'` will be favored over `Z`. 

Specs in the environment must be concrete and installed to be added to the view, so there is another minor ordering effect: by default the view maintained for the environment ignores file conflicts between packages. If packages are not installed in order, and there are file conflicts, then the version chosen depends on the order.

Both ordering issues are avoided if `spack install`/`spack add` and `spack install <spec>` are not mixed.
2019-04-10 16:00:12 -07:00
Massimiliano Culpo
e3f00750e8 Update llnl.util.lang.memoized so that Sphinx can extract signature (#11055)
Replace the original implementation of the "memoized" decorator with
an implementation that exposes the docstring and arguments of the
wrapped function. This is achieved using functools.wraps.
2019-03-29 17:11:44 -07:00
Massimiliano Culpo
0a006351c8 Spack can be extended with external commands (#8612)
This provides a mechanism to implement a new Spack command in a
separate directory, and with a small configuration change point Spack
to the new command.

To register the command, the directory must be added to the
"extensions" section of config.yaml. The command directory name must
have the prefix "spack-", and have the following layout:

  spack-X/
    pytest.ini #optional, for testing
    X/
	  cmd/
	    name-of-command1.py
	    name-of-command2.py
	    ...
    tests/ #optional
      conftest.py
	  test_name-of-command1.py
    templates/ #optional jinja templates, if needed

And in config.yaml:

  config:
    extensions:
      - /path/to/spack-X

If the extension includes tests, you can run them via spack by adding
the --extension option, like "spack test --extension=X"
2019-03-28 16:56:36 -07:00
Michael Kuhn
a1c91f3c07 Fix find_headers to also look for C++ headers and Fortran modules (#10798)
Currently, only C headers are considered, causing build failures for
packages depending on, e.g., netcdf-fortran and xerces-c. Additionally,
the regex used to look for the include path component did not consider
word boundaries, causing false matches.
2019-03-08 21:06:22 -06:00
Massimiliano Culpo
e3af8ed454 Added a sub-command to show if packages are relocatable (#9199)
* Added the `spack buildcache preview` sub-command

This is similar to `spack spec -I` but highlights which nodes in a DAG
are relocatable and which are not.

spec.tree has been generalized a little to accept a status function,
instead of always showing the install status

The current implementation works only for ELF, and needs to be
generalized to other platforms.

* Added a test to check if an executable is relocatable or not

This test requires a few commands to be present in the environment.
Currently it will run only under python 3.7 (which uses Xenial instead
of Trusty).

* Added tests for the 'buildcache preview' command.

* Fixed codebase after rebase

* Fixed the list of apt addons for Python 3.7 in travis.yaml

* Only check ELF executables and shared libraries. Skip checking virtual or external packages. (#229)

* Fixed flake8 issues

* Add handling for macOS mach binaries (#231)
2019-02-28 15:36:47 -06:00
Massimiliano Culpo
42386dbe94 Use Package.headers for -I options (#10623)
This restores the use of Package.headers when computing -I options
for building a package that was added in #8136 and reverted in
#10604. #8136 used utility logic that located all header files in
an installation prefix, and calculated the -I options as the
immediate roots containing those header files.

In some cases, for a package containing a directory structure like

  prefix/
    include/
	  ex1.h
	  subdir/
	    ex2.h

dependents may expect to include ex2.h relative to 'include', and
adding 'prefix/include/subdir' as a -I was causing errors,
in particular if ex2.h has the same name as a system header.

This updates header utility logic to by default return the base
"include" directory when it exists, rather than subdirectories.
It also makes it possible for package implementers to override
Package.headers to return the subdirectory when it is required
(for example with libxml2).
2019-02-26 12:42:30 -06:00
sknigh
a76c50d1ee Allow tty output to be timestamped (#10554)
* Spack debug output now includes microsecond-granularity timestamps.
* Timestamps can also be enabled with the `--timestamp` command line argument.
2019-02-13 10:14:35 -08:00
Matthias Wolf
861dd06bd1 enh: allow time like HH:MM in date strings. (#10034) 2019-02-13 11:05:00 +01:00
Todd Gamblin
6f50cd52ed copyright: update license headers for 2013-2019 copyright. 2019-01-01 00:44:28 -08:00
Massimiliano Culpo
3b8b13809e Improve validation of modules.yaml (#9878)
This PR improves the validation of `modules.yaml` by introducing a custom validator that checks if an attribute listed in `properties` or `patternProperties` is a valid spec. This new check applied to the test case in #9857 gives:

```console
$ spack install szip
==> Error: /home/mculpo/.spack/linux/modules.yaml:5: "^python@2.7@" is an invalid spec [Invalid version specifier]
```

Details: 
* Moved the set-up of a custom validator class to spack.schema
  * In Spack we use `jsonschema` to validate configuration files 
    against a schema. We also need custom validators to enforce
    writing default values within "properties" or "patternProperties"
    attributes.

  * Currently, validators were customized at the place of use and with the
    recent introduction of environments that meant we were setting-up and
    using 2 different validator classes in two different modules.

  * This commit moves the set-up of a custom validator class in the
    `spack.schema` module and refactors the code in `spack.config` and
    `spack.environments` to use it.

* Added a custom validator to check if an attribute is a valid spec
  * Added a custom validator that can be used on objects, which yields an
    error if the attribute is not a valid spec.

* Updated the schema for modules.yaml

* Updated modules.yaml to fix a few inconsistencies:
  - a few attributes were not tested properly using 'anyOf'
  - suffixes has been updated to also check that the attribute is a spec
  - hierarchical_scheme has been updated to hierarchy

* Removed $ref from every schema
  * $ref is not composable or particularly legible
  * Use python dicts and regular old variables instead.
2019-01-01 00:11:49 -08:00
Todd Gamblin
a41bce2148 fix bad regular expressions and docstrings with '\' 2018-11-09 00:31:24 -08:00
Todd Gamblin
a1818f971f env: environments can be named or created in directories
- `spack env create <name>` works as before

- `spack env create <path>` now works as well -- environments can be
  created in their own directories outside of Spack.

- `spack install` will look for a `spack.yaml` file in the current
  directory, and will install the entire project from the environment

- The Environment class has been refactored so that it does not depend on
  the internal Spack environment root; it just takes a path and operates
  on an environment in that path (so internal and external envs are
  handled the same)

- The named environment interface has been hoisted to the
  spack.environment module level.

- env.yaml is now spack.yaml in all places.  It was easier to go with one
  name for these files than to try to handle logic for both env.yaml and
  spack.yaml.
2018-11-09 00:31:24 -08:00
Todd Gamblin
6af5dfbbc2 config: allow env.yaml to contain configuration in a single file
- add `SingleFileScope` to configuration, which allows us to pull config
  sections from a single file.

- update `env.yaml` and tests to ensure that the env.yaml schema works
  when pulling configurtion from the env file.
2018-11-09 00:31:24 -08:00
Todd Gamblin
a1f90d5b8c
bugfix: use OS default permissions for mkdirp when mode is not provided (#9604)
- #8773 made the default mode 0o777, which is what's documented but
   mkdirp actually takes the OS default or umask by default

- revert to the Python default by default, and only set the mode when
  asked explicitly.
2018-10-24 11:36:41 -07:00
Todd Gamblin
eea786f4e8 relicense: replace LGPL headers with Apache-2.0/MIT SPDX headers
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
  - core and most packages are (Apache-2.0 OR MIT)
  - a very small number of remaining packages are LGPL-2.1-only
2018-10-17 14:42:06 -07:00
Greg Becker
d1a5113cfe permissions: add permission configuration to packages.yaml (#8773)
Spack can now be configured to assign permissions to the files installed by a package.

In the `packages.yaml` file under `permissions`, the attributes `read`, `write`, and `group` control the package permissions. These attributes can be set per-package, or for all packages under `all`. If permissions are set under `all` and for a specific package, the package-specific settings take precedence.  The `read` and `write` attributes take one of `user`, `group`, and `world`.

   packages:
    all:
      permissions:
        write: group
        group: spack
    my_app:
      permissions:
        read: group
        group: my_team
2018-10-11 14:29:07 -07:00
Peter Scheibel
28c0dd9148
Increase and customize lock timeouts (#9219)
Fixes #9166

This is intended to reduce errors related to lock timeouts by making
the following changes:

* Improves error reporting when acquiring a lock fails (addressing
  #9166) - there is no longer an attempt to release the lock if an
  acquire fails
* By default locks taken on individual packages no longer have a
  timeout. This allows multiple spack instances to install overlapping
  dependency DAGs. For debugging purposes, a timeout can be added by
  setting 'package_lock_timeout' in config.yaml
* Reduces the polling frequency when trying to acquire a lock, to
  reduce impact in the case where NFS is overtaxed. A simple
  adaptive strategy is implemented, which starts with a polling
  interval of .1 seconds and quickly increases to .5 seconds
  (originally it would poll up to 10^5 times per second).
  A test is added to check the polling interval generation logic.
* The timeout for Spack's whole-database lock (e.g. for managing
  information about installed packages) is increased from 60s to
  120s
* Users can configure the whole-database lock timeout using the
  'db_lock_timout' setting in config.yaml

Generally, Spack locks (those created using spack.llnl.util.lock.Lock)
now have no timeout by default

This does not address implementations of NFS that do not support file
locking, or detect cases where services that may be required
(nfslock/statd) aren't running.

Users may want to be able to more-aggressively release locks when
they know they are the only one using their Spack instance, and they
encounter lock errors after a crash (e.g. a remote terminal disconnect
mentioned in #8915).
2018-09-25 18:58:51 -07:00
scheibelp
638cc64571
install_tree: symlink handling and add 'ignore' option (#9019)
Fixes #9001

#8289 added support for install_tree and copy_tree to merge into an existing
directory structure. However, it did not properly handle relative symlinks and
also removed support for the 'ignore' keyword. Additionally, some of the tests
were overly-strict when checking the permissions on the copied files.

This updates the install_tree/copy_tree methods and their tests:

* copy_tree/install_tree now preserve relative link targets (if the symlink in the
  source directory structure is relative, the symlink created in the destination
  will be relative)
* Added support for 'ignore' argument back to copy_tree/install_tree (removed
  in #8289). It is no longer the object output by shutil.ignore_patterns: you pass a
  function that accepts a path relative to the source and returns whether that
  path should be copied.
* The openfoam packages (currently the only ones making use of the 'ignore'
  argument) are updated for the new API
* When a symlink target is absolute, copy_tree and install_tree now rewrite the
  source prefix to be the destination prefix
* copy_tree tests no longer check permissions: copy_tree doesn't enforce
  anything about permissions so its tests don't check for that
* install_tree tests no longer check for exact permission matching since it can add
  file permissions
2018-08-17 22:08:38 -04:00
Adam J. Stewart
73c978ddd9 install_tree, copy_tree can install into existing directory structures (#8289)
Replace use of `shutil.copytree` with `copy_tree` and `install_tree` functions in `llnl.util.filesystem`.

- `copy_tree` copies without setting permissions.  It should be used to copy files around in the build directory.
- `install_tree` copies files and sets permissions.  It should be used to copy files into the installation directory.
- `install` and `copy` are analogous single-file functions.
- add more extensive tests for these functions
- update packages to use these functions.
2018-08-15 09:30:09 -07:00
cedricchevalier19
3301e21f06 Fix performance issue when compiling. (#8828)
* Fix performance issue when compiling.

Spack was doing active wait when compiling, spoiling one core.
My fix consists in not setting any timeout for select, instead of
the previous 0 second.

* Fix comments about select.select timeout
2018-08-07 09:13:07 -07:00
Todd Gamblin
43c77ac3cb libdwarf: remove use of hide_files()
- This was a nasty workaround due to the way our compiler wrappers used
  to work.  We don't want to have to modify our elfutils installation to
  install libdwarf.

- Since cd9691de5, we no longer need this because the package will always
  come before dependencies in our include order.
2018-08-02 19:00:52 -07:00
Todd Gamblin
fdcaf5c4c8 bugfix: fix spack spec --yaml
- repo membership test was broken by the refactor of spack/__init__.py

- refactor singleton so that 'spec in repo' works again for `spack.repo.path`

- fix spec command and add basic tests for `spack spec` and `spack spec --yaml`
2018-07-24 11:25:34 -07:00
Todd Gamblin
2b0d944341 locks: fix bug when creating lockfiles in the current directory.
- Fixes a bug in `llnl.util.lock`

- Locks in the current directory would fail because the parent directory
  was the empty string.

- Fix this and return '.' for the parent of locks in the current
  directory.
2018-07-21 10:39:47 -07:00
Todd Gamblin
20e4038a72 style: make core comply with pep8-naming 2018-07-19 00:41:36 -07:00
Todd Gamblin
650786c812 locks: improve errors and permission checking
- Clean up error messages for when a lock can't be created, or when an
  exclusive (write) lock can't be taken on a file.

- Add a number of subclasses of LockError to distinguish timeouts from
  permission issues.

- Add an explicit check to prevent the user from taking a write lock on a
  read-only file.
  - We had a check for this for when we try to *upgrade* a lock on an RO
    file, but not for an initial write lock attempt.

- Add more tests for different lock permission scenarios.
2018-07-12 19:59:53 +02:00
Todd Gamblin
ab794fa741 locks: llnl.util.lock now only writes host info when in debug mode
- write locks previously wrote information about the lock holder (host
  and pid), and read locks woudl read this in.

- This is really only for debugging, so only enable it then

- add some tests that target debug info, and improve multiproc lock test
  output
2018-07-12 19:59:53 +02:00
Todd Gamblin
7626ec4579 refactor: move spack.util.multiproc to llnl.util.multiproc
- multiproc doesn't depend on Spack
- llnl.util.lock test uses it, but shouldn't use parts of Spack.
2018-07-12 19:59:53 +02:00
scheibelp
3560f6dbe9 views: packages can customize how they're added to views (#7152)
Functional updates:

- `python` now creates a copy of the `python` binaries when it is added
  to a view

- Python extensions (packages which subclass `PythonPackage`) rewrite
  their shebang lines to refer to python in the view

- Python packages in the same namespace will not generate conflicts if
  both have `...lib/site-packages/namespace-example/__init__.py`

  - These `__init__` files will also remain when removing any package in
    the namespace until the last package in the namespace is removed


Generally (Updated 2/16):

- Any package can define `add_files_to_view` to customize how it is added
  to a view (and at the moment custom definitions are included for
  `python` and `PythonPackage`)

  - Likewise any package can define `remove_files_from_view` to customize
    which files are removed (e.g. you don't always want to remove the
    namespace `__init__`)

- Any package can define `view_file_conflicts` to customize what it
  considers a merge conflict

- Global activations are handled like views (where the view root is the
  spec prefix of the extendee)

  - Benefit: filesystem-management aspects of activating extensions are
    now placed in views (e.g. now one can hardlink a global activation)

  - Benefit: overriding `Package.activate` is more straightforward (see
    `Python.activate`)

  - Complication: extension packages which have special-purpose logic
    *only* when activated outside of the extendee prefix must check for
    this in their `add_files_to_view` method (see `PythonPackage`)

- `LinkTree` is refactored to have separate methods for copying a
  directory structure and for copying files (since it was found that
  generally packages may want to alter how files are copied but still
  wanted to copy directories in the same way)


TODOs (updated 2/20):

- [x] additional testing (there is some unit testing added at this point
  but more would be useful)

- [x] refactor or reorganize `LinkTree` methods: currently there is a
  separate set of methods for replicating just the directory structure
  without the files, and a set for replicating everything

- [x] Right now external views (i.e. those not used for global
  activations) call `view.add_extension`, but global activations do not
  to avoid some extra work that goes into maintaining external views. I'm
  not sure if addressing that needs to be done here but I'd like to
  clarify it in the comments (UPDATE: for now I have added a TODO and in
  my opinion this can be merged now and the refactor handled later)

- [x] Several method descriptions (e.g. for `Package.activate`) are out
  of date and reference a distinction between global activations and
  views, they need to be updated

- [x] Update aspell package activations
2018-06-26 16:14:05 -07:00
Todd Gamblin
a48bdfaf1d bugfix: fix macos incompatibility in lock test (#8573)
- Spack was assuming that a group with gid == current uid would always exist.
- This was breaking the travis build for macos.

- also fix issue with the DB tarball test finding coverage filesx
2018-06-26 14:51:02 +02:00
Todd Gamblin
970b558f7f tests: add a test for spack debug command 2018-06-24 14:21:52 -07:00
Adam J. Stewart
fec11757d5 Fix coloring of error messages containing '}' symbol (#8277) 2018-06-02 21:02:28 -07:00
Todd Gamblin
54201e3c02
locks: add configuration and command-line options to enable/disable locks (#7692)
- spack.util.lock behaves the same as llnl.util.lock, but Lock._lock and
  Lock._unlock do nothing.

- can be disabled with a control variable.

- configuration options can enable/disable locking:
  - `locks` option in spack configuration controls whether Spack will use filesystem locks or not.
  - `-l` and `-L` command-line options can force-disable or force-enable locking.

- Spack will check for group- and world-writability before disabling
  locks, and it will not allow a group- or world-writable instance to
  have locks disabled.

- update documentation
2018-05-18 14:41:03 -07:00
Todd Gamblin
f202198777 Convert lazy singleton functions to Singleton object
- simplify the singleton pattern across the codebase
- reduce lines of code needed for crufty initialization
- reduce functions that need to mess with a global

- Singletons whose semantics changed:
  - spack.store.store() -> spack.store
  - spack.repo.path() -> spack.repo.path
  - spack.config.config() -> spack.config.config
  - spack.caches.fetch_cache() -> spack.caches.fetch_cache
  - spack.caches.misc_cache() -> spack.caches.misc_cache
2018-05-17 14:10:30 -07:00
Adam J. Stewart
cdefbd7475 Avoid double 'lib' for packages whose name already starts with lib (#7651) 2018-03-31 17:33:41 +02:00
Todd Gamblin
54f97d1dec
Update copyright on LLNL files for 2018. (#7592) 2018-03-24 12:13:52 -07:00
Massimiliano Culpo
5655895865 Record installation date and time in DB (#7334)
* Added installation date and time to the database

Information on the date and time of installation of a spec is recorded
into the database. The information is retained on reindexing.

* Expose the possibility to query for installation date

The DB can now be queried for specs that have been installed in a given
time window. This query possibility is exposed to command line via two
new options of the `find` command.

* Extended docstring for Database._add

* Use timestamps since the epoch instead of formatted date in the DB

* Allow 'pretty date' formats from command line

* Substituted kwargs with explicit arguments

* Simplified regex for pretty date strings. Added unit tests.
2018-03-22 11:07:27 -07:00
Massimiliano Culpo
a1c8ce82f2 Constructing a SpecBuildInterface from another gives no inconsistent MRO (#7457)
fixes #7239
2018-03-12 17:07:08 +09:00
scheibelp
4a807fca08
Make fix_darwin_install_name python3-compatible (#7329)
This updates the fix_darwin_install_name function to use the Spack
Executable object to run install_name_tool, which ensures that
process output is formatted as a 'str' for python2 and python3.
Originally fix_darwin_install_name was invoking subprocess.Popen
directly.
2018-02-23 10:24:59 -08:00
alalazo
efd2a95781 find has been changed to accept glob expressions
Following the discussion with Todd and Adam, find has been modified to
accept glob expressions. This should not affect performance as every
glob implementation I inspected has 3 cases (no wildcard, wildcard but
no directories involved, wildcard and directories involved) and uses
fnmatch underneath.

Mixins have been changed to do by default a non-recursive search (but
a recursive search can still be triggered using the recursive keyword).
2018-02-13 02:18:28 -08:00
alalazo
4e48bae096 mixins: moved debug logs to 'filter_file'. Renamed shadowed variable name.
Following comments from Todd:

- the call to tty.debug has been moved deeper, to log the filtering of each file
- the shadowing on the name "kwargs" is avoided
2018-02-13 02:18:28 -08:00
Todd Gamblin
b98cdf098a Rework command reference in docs, add spack commands command
- command reference now includes usage for all Spack commands as output
  by `spack help`.  Each command usage links to any related section in
  the docs.

- added `spack commands` command which can list command names,
  subcommands, and generate RST docs for commands.

- added `llnl.util.argparsewriter`, which analyzes an argparse parser and
  calls hooks for description, usage, options, and subcommands
2018-02-12 20:25:17 -08:00
Patrick Gartung
124b5fc296 deal with case where symbolic links are copied (#6178)
* deal with case where symbolic links are copied

* Suggested changes
2017-11-09 14:38:12 -08:00
Todd Gamblin
05fa302655
Replace github.com/llnl/spack with github.com/spack/spack (#6142)
We moved to a new GitHub org! Now make the code and docs reflect that.
2017-11-04 17:08:04 -07:00
Oliver Breitwieser
00a893aa79 link_tree: support choosing the link function
This can be used to make hardlinks rather than symlinks.
2017-11-02 18:45:40 -07:00
Oliver Breitwieser
9024ddcf05 link_tree: support ignoring conflicts 2017-11-02 18:45:40 -07:00
Oliver Breitwieser
94c0740a88 link_tree: only remove matching files when unmerging 2017-11-02 18:45:40 -07:00
Massimiliano Culpo
8b7d2d0f24 'spack install' can overwrite an existing installation (#5384)
'spack install' can now reinstall a spec even if it has dependents, via
the --overwrite option. This option moves the current installation in a
temporary directory. If the reinstallation is successful the temporary
is removed, otherwise a rollback is performed.
2017-10-24 12:32:30 -07:00
Todd Gamblin
5449884b2e Fix bare 'except:' to placate errors in new flake8 version.
- fixes E722 errors from latest version of flake8
- requires us to not use 'bare except:' and catch BaseException instead
2017-10-24 10:05:36 +02:00
Todd Gamblin
0bb1eb32f2 fix bugs found with stricter flake8 rules
- When you don't use wildcards, flake8 will find places where you used an
  undefined name.

- This commit has all the bugfixes resulting from this static check.
2017-10-24 10:05:36 +02:00
Todd Gamblin
7dd79094b0 remove wildcards from make spack core and packages
- This removes all wildcard imports EXCEPT `from spack import *` in packages
2017-10-24 10:05:36 +02:00
George Hartzell
464e558aea filter_file, don't remove absent backup file (#5733)
I'm tracking down a problem with the perl package that's been
generating this error:

```
OSError: OSError: [Errno 2] No such file or directory: '/blah/blah/blah/lib/5.24.1/x86_64-linux/Config.pm~'
```

The real problem is upstream, but it's being masked by an exception
raised in `filter_file`s finally block.

In my case, `backup` is `False`.

The backup is created around line 127, the `re.sub()` calls
fails (working on that), the `except` block fires and moves the backup
file back, then the finally block tries to remove the non-existent
backup file.

This change just avoids trying to remove the non-existent file.
2017-10-17 11:26:05 -07:00
Adam J. Stewart
6f9036c858 Ensure that FileList makes it into the documentation (#5739) 2017-10-14 06:20:39 +02:00
Todd Gamblin
8648e2cda5 Add testing for spack blame; refactor llnl.util tests 2017-09-30 16:31:56 -07:00
Todd Gamblin
41a2652ef2 Add 'spack blame' command: shows contributors to packages
`spack blame` prints out the contributors to a package.

By modification time:

```
$ spack blame --time llvm
LAST_COMMIT    LINES  %      AUTHOR               EMAIL
3 days ago     2      0.6    Andrey Prokopenko    <andrey.prok@gmail.com>
3 weeks ago    125    34.7   Massimiliano Culpo   <massimiliano.culpo@epfl.ch>
3 weeks ago    3      0.8    Peter Scheibel       <scheibel1@llnl.gov>
2 months ago   21     5.8    Adam J. Stewart      <ajstewart426@gmail.com>
2 months ago   1      0.3    Gregory Becker       <becker33@llnl.gov>
3 months ago   116    32.2   Todd Gamblin         <tgamblin@llnl.gov>
5 months ago   2      0.6    Jimmy Tang           <jcftang@gmail.com>
5 months ago   6      1.7    Jean-Paul Pelteret   <jppelteret@gmail.com>
7 months ago   65     18.1   Tom Scogland         <tscogland@llnl.gov>
11 months ago  13     3.6    Kelly (KT) Thompson  <kgt@lanl.gov>
a year ago     1      0.3    Scott Pakin          <pakin@lanl.gov>
a year ago     3      0.8    Erik Schnetter       <schnetter@gmail.com>
3 years ago    2      0.6    David Beckingsale    <davidbeckingsale@gmail.com>

3 days ago     360    100.0
```

Or by percent contribution:

```
$ spack blame --percent llvm
LAST_COMMIT    LINES  %      AUTHOR               EMAIL
3 weeks ago    125    34.7   Massimiliano Culpo   <massimiliano.culpo@epfl.ch>
3 months ago   116    32.2   Todd Gamblin         <tgamblin@llnl.gov>
7 months ago   65     18.1   Tom Scogland         <tscogland@llnl.gov>
2 months ago   21     5.8    Adam J. Stewart      <ajstewart426@gmail.com>
11 months ago  13     3.6    Kelly (KT) Thompson  <kgt@lanl.gov>
5 months ago   6      1.7    Jean-Paul Pelteret   <jppelteret@gmail.com>
3 weeks ago    3      0.8    Peter Scheibel       <scheibel1@llnl.gov>
a year ago     3      0.8    Erik Schnetter       <schnetter@gmail.com>
3 years ago    2      0.6    David Beckingsale    <davidbeckingsale@gmail.com>
3 days ago     2      0.6    Andrey Prokopenko    <andrey.prok@gmail.com>
5 months ago   2      0.6    Jimmy Tang           <jcftang@gmail.com>
2 months ago   1      0.3    Gregory Becker       <becker33@llnl.gov>
a year ago     1      0.3    Scott Pakin          <pakin@lanl.gov>

3 days ago     360    100.0
```
2017-09-30 16:31:56 -07:00
Adam J. Stewart
2eb8db1dd2 Fix name detection in HeaderList and LibraryList (#5118)
* Fix name detection in HeaderList and LibraryList

* Add support for CUDA header files
2017-09-09 22:22:56 +02:00
Michael Kuhn
84ae7872d3 Update copyright notices for 2017 (#5295) 2017-09-06 17:44:16 -10:00
Todd Gamblin
4f444c5f58 log_ouptut can take either a filename or a file object 2017-08-22 16:14:38 -07:00
Todd Gamblin
10bb681b57 Unbuffer so that output from packages appears when redirecting
- Python I/O would not properly interleave (or appear) with output from
  subcommands.

- Add a flusing wrapper around sys.stdout and sys.stderr when
  redirecting, so that Python output is synchronous with that of
  subcommands.
2017-08-20 16:51:10 -07:00
Todd Gamblin
11196e7b69 Preserve verbosity across installs when 'v' is pressed.
- 'v' toggle was previously only good for the current install.
- subsequent installs needed user to press 'v' again.
- 'v' state is now preserved across dependency installs.
2017-08-20 16:51:10 -07:00
Todd Gamblin
05cc6c966f Rework output redirection in Spack.
- Simplify interface to log_output. New interface requires only one
  context handler instead of two.  Before:

      with log_output('logfile.txt') as log_redirection:
           with log_redirection:
               # do things ... output will be logged

  After:

      with log_output('logfile.txt'):
          # do things ... output will be logged

  If you also want the output to be echoed to ``stdout``, use the
  `echo` parameter::

      with log_output('logfile.txt', echo=True):
          # do things ... output will be logged and printed out

  And, if you just want to echo *some* stuff from the parent, use
  ``force_echo``:

      with log_output('logfile.txt', echo=False) as logger:
          # do things ... output will be logged

          with logger.force_echo():
              # things here will be echoed *and* logged

  A key difference between this and the previous implementation is that
  *everything* in the context handler is logged.  Previously, things like
  `Executing phase 'configure'` would not be logged, only output to the
  screen, so understanding phases in the build log was difficult.

- The implementation of `log_output()` is different in two major ways:

  1. This implementation avoids race cases by using only one pipe (before
     we had a multiprocessing pipe and a unix pipe).  The logger daemon
     stops naturally when the input stream is closed, which avoids a race
     in the previous implementation where we'd miss some lines of output
     because the parent would shut the daemon down before it was done
     with all output.

  2. Instead of turning output redirection on and off, which prevented
     some things from being logged, this version uses control characters
     in the output stream to enable/disable forced echoing.  We're using
     the time-honored xon and xoff codes, which tell the daemon to echo
     anything between them AND write it to the log.  This is how
     `logger.force_echo()` works.

- Fix places where output could get stuck in buffers by flushing more
  aggressively.  This makes the output printed to the terminal the same
  as that which would be printed through a pipe to `cat` or to a file.
  Previously these could be weirdly different, and some output would be
  missing when redirecting Spack to a file or pipe.

- Simplify input and color handling in both `build_environment.fork()`
  and `llnl.util.tty.log.log_output()`.  Neither requires an input_stream
  parameter anymore; we assume stdin will be forwarded if possible.

- remove `llnl.util.lang.duplicate_stream()` and remove associated
  monkey-patching in tests, as these aren't needed if you just check
  whether stdin is a tty and has a fileno attribute.
2017-08-20 16:51:10 -07:00
Adam J. Stewart
4e269510c5 Fix trailing whitespace at the end of headers.cpp_flags (#4957) 2017-08-03 14:24:51 -05:00
Todd Gamblin
b88f55e523 Add spack dependencies command and tests for it and dependents. 2017-08-01 17:40:54 -07:00
paulhopkins
1c7e5724d9 Add --color=[always|never|auto] argument; fix color when piping (#3013)
* Disable spec colorization when redirecting stdout and add command line flag to re-enable
* Add command line `--color` flag to control output colorization
* Add options to `llnl.util.tty.color` to allow color to be auto/always/never
* Add `Spec.cformat()` function to be used when `format()` should have auto-coloring
2017-07-31 12:57:47 -07:00
Todd Gamblin
b4d1654e68 Parametrized lock test and make it work with MPI
- Lock test can be run either as a node-local test or as an MPI test.

- Lock test is now parametrized by filesystem, so you can test the
  locking capabilities of your NFS, Lustre, or GPFS filesystem.  See docs
  for details.
2017-07-04 11:41:37 -07:00
Todd Gamblin
bd7a591df1 Make filesytem more resilient to concurrent updates.
- Uses O_CREAT for touch (for guaranteed atomic open on NFS, multi-node)
- Ignore concurrent create errors in mkdirp
2017-07-04 11:41:37 -07:00
Todd Gamblin
cac4362f64 Make LICENSE recognizable by GitHub. (#4598) 2017-06-24 22:22:55 -07:00
Massimiliano Culpo
f8b3eff01c filesystem.py: fixed bug introduced in #3367 (scrambled order in output) (#4156)
PR #3367 inadvertently changed the semantics of _find_recursive and
_find_non_recursive so that the returned list are not ordered as the
input search list. This commit restores the original semantic, and adds
tests to verify it.
2017-05-11 10:29:08 -07:00