Spack can now:
- label ppc64, ppc64le, x86_64, etc. builds with specific
microarchitecture-specific names, like 'haswell', 'skylake' or
'icelake'.
- detect the host architecture of a machine from /proc/cpuinfo or similar
tools.
- Understand which microarchitectures are compatible with which (for
binary reuse)
- Understand which compiler flags are needed (for GCC, so far) to build
binaries for particular microarchitectures.
All of this is managed through a JSON file (microarchitectures.json) that
contains detailed auto-detection, compiler flag, and compatibility
information for specific microarchitecture targets. The `llnl.util.cpu`
module implements a library that allows detection and comparison of
microarchitectures based on the data in this file.
The `target` part of Spack specs is now essentially a Microarchitecture
object, and Specs' targets can be compared for compatibility as well.
This allows us to label optimized binary packages at a granularity that
enables them to be reused on compatible machines. Previously, we only
knew that a package was built for x86_64, NOT which x86_64 machines it
was usable on.
Currently this feature supports Intel, Power, and AMD chips. Support for
ARM is forthcoming.
Specifics:
- Add microarchitectures.json with descriptions of architectures
- Relaxed semantic of compiler's "target" attribute. Before this change
the semantic to check if a compiler could be viable for a given target
was exact match. This made sense as the finest granularity of targets
was architecture families. As now we can target micro-architectures,
this commit changes the semantic by interpreting as the architecture
family what is stored in the compiler's "target" attribute. A compiler
is then a viable choice if the target being concretized belongs to the
same family. Similarly when a new compiler is detected the architecture
family is stored in the "target" attribute.
- Make Spack's `cc` compiler wrapper inject target-specific flags on the
command line
- Architecture concretization updated to use the same algorithm as
compiler concretization
- Micro-architecture features, vendor, generation etc. are included in
the package hash. Generic architectures, such as x86_64 or ppc64, are
still dumped using the name only.
- If the compiler for a target is not supported exit with an intelligible
error message. If the compiler support is unknown don't try to use
optimization flags.
- Support and define feature aliases (e.g., sse3 -> ssse3) in
microarchitectures.json and on Microarchitecture objects. Feature
aliases are defined in targets.json and map a name (the "alias") to a
list of rules that must be met for the test to be successful. The rules
that are available can be extended later using a decorator.
- Implement subset semantics for comparing microarchitectures (treat
microarchitectures as a partial order, i.e. (a < b), (a == b) and (b <
a) can all be false.
- Implement logic to automatically demote the default target if the
compiler being used is too old to optimize for it. Updated docs to make
this behavior explicit. This avoids surprising the user if the default
compiler is older than the host architecture.
This commit adds unit tests to verify the semantics of target ranges and
target lists in constraints. The implementation to allow target ranges
and lists is minimal and doesn't add any new type. A more careful
refactor that takes into account the type system might be due later.
Co-authored-by: Gregory Becker <becker33.llnl.gov>
Add llnl.util.cpu_name, with initial support for detecting different
microarchitectures on Linux. This also adds preliminary changes for
compiler support and variants to control the optimizatoin levels by
target.
This does not yet include translations of targets to particular
compilers; that is left to another PR.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Uses code from CMake to detect implicit link paths from compilers
System paths are filtered out of implicit link paths
Implicit link paths added to compiler config and object under `implicit_rpaths`
Implicit link paths added as rpaths to compile line through env/cc wrapper
Authored by: "Ben Boeckel <ben.boeckel@kitware.com>"
Co-authored by: "Peter Scheibel <scheibel1@llnl.gov>"
Co-authored by: "Gregory Becker <becker33@llnl.gov>"
* Add Fujitsu compiler to Spack.
* Fixes for flake8
* Chenges location of FCC to subdirectory called case-insensitive
* Add compiler tests for Fujitsu compiler
* Modify the logic of taking compiler version for new version of Fujitsu compiler
Fixes#11335
Update the Spack compiler wrappers to add the headerpad_max_install_names
linker flag on MacOS. This allows the install_name_tool to rewrite
the RPATH entry of the binary to be longer if needed. This is
primarily useful for creating and distributing binary caches of
packages (i.e. using the "spack buildcache" command); binary caches
created on MacOS before this commit may not successfully relocate
(if the target root path is larger).
This restores the use of Package.headers when computing -I options
for building a package that was added in #8136 and reverted in
#10604. #8136 used utility logic that located all header files in
an installation prefix, and calculated the -I options as the
immediate roots containing those header files.
In some cases, for a package containing a directory structure like
prefix/
include/
ex1.h
subdir/
ex2.h
dependents may expect to include ex2.h relative to 'include', and
adding 'prefix/include/subdir' as a -I was causing errors,
in particular if ex2.h has the same name as a system header.
This updates header utility logic to by default return the base
"include" directory when it exists, rather than subdirectories.
It also makes it possible for package implementers to override
Package.headers to return the subdirectory when it is required
(for example with libxml2).
fixes#10601
Due to a bug this attribute is wrong for packages that use directories
as namespaces. For instance it will add "<boost-prefix>/include/boost"
instead of "<boost-prefix>/include" to the include path.
As a minor addition a few loops in the compiler wrappers have been
simplified.
Fixes#7855Closes#8070Closes#2645
When searching for library directories (e.g. to add "-L" arguments to
the compiler wrapper) Spack was only trying the "lib/" and "lib64/"
directories for each dependency install prefix; this missed cases
where packages would install libraries to subdirectories and also was
not customizable. This PR makes use of the ".headers" and ".libs"
properties for more-advanced location of header/library directories.
Since packages can override the default behavior of ".headers" and
".libs", it also allows package writers to customize.
The following environment variables which used to be set by Spack
for a package build have been removed:
* Remove SPACK_PREFIX and SPACK_DEPENDENCIES environment variables as
they are no-longer used
* Remove SPACK_INSTALL environment variable: it was not used before
this PR
* Initial compiler support
* added arm.py
* Changed licence to Arm suggested header
* Changed licence to the same as clang.py
Main author of file is Nick Forrington <Nick.Forrington@arm.com>
Minor changes by Srinath Vadlamani <srinath.vadlamani@arm.com>
* compilers: add arm compiler detection to Spack
- added arm.py with support for detecting `armclang` and `armflang`
Co-authored-by: Srinath Vadlamani <srinath.vadlamani@arm.com>
* Changed to using get get_compiler_version
* linking to general cc for arm compiler
* For arm compiler add CFLAGS to use compiler-rt rtlib.
* Escape for special characters in rexep
* Cleaned up for Flake8 to pass.
* libcompiler-rt should be part of the LDFLAGS not CFLAGS
* fixed m4 when using clang to used LDFLAGS. Fixed comments for arm.py to display compiler --version output with # NOAQ for flakes pass.
* added arm compilers
* proper linked names
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
- core and most packages are (Apache-2.0 OR MIT)
- a very small number of remaining packages are LGPL-2.1-only
- cc cleanup caused a parsing regression in flag handling
- We added proper quoting to array expansions, but flag variables were
never actually converted to arrays. Old code relied on this.
This commit:
- Adds reads to convert flags to arrays.
- Makes the cc test check for improper space handling to prevent future
regressions.
- flags were prepended in reverse order to args, but this makes it hard
to see what order they'll be in on the final command line.
- add them in the order they'll appear to make cc easier to maintain.
- simplify code for assembling the command line
- fix separator used in SPACK_SYSTEM_DIRS test
- This corrects most of the issues found by shellcheck
- This also uses ':' as the delimiter for SPACK_SYSTEM_DIRS, for
consistency with other variables.
- filtering using sed causes most builds to slow down quite a bit, as the
compiler wrapper has to run sed many times, and *it* runs many times
- do the system directory parsing directly in bash
- Adding -L and -Wl,-rpath to compile-only command lines ("cc mode" or
"-c") causes clang (if not also other compilers) to emit warnings that
confuse configure systems.
- Clang will print warnings about unused command-line arguments.
- This fix ensures that -L and -Wl,-rpath are not added if the compile
line is just building an object file with -c
- This also cleans up the cc script in several places.
Spack currently prepends include paths, library paths, and rpaths to the
compile line. This causes problems when a header or library in the package
has the same name as one exported by one of its dependencies. The
*dependency's* header will be preferred over the package's, which is not
what most builds expect. This also breaks some of our production codes.
This restores the original cc behavior (from *very* early Spack) of parsing
compiler arguments out by type (`-L`, `-I`, `-Wl,-rpath`) and reconstituting
the full command at the end.
`<includes> <other_args> <library dirs> <rpaths>`
This differs from the original behavior in one significant way, though: it
*appends* the library arguments so that dependency libraries do not shadow
those in the build.
This is safe because semantics aren't affected by *interleaving* `-I`, `-L`,
and `-Wl,-rpath` arguments with others, only with each other (so the order of
two `-L` args affects the search path, but we search for all libraries on the
command line using the same search path).
We preserve the following:
1. Any system directory in the paths will be listed last.
2. The root package's include/library/RPATH flags come before flags of the
same type for any dependency.
3. Order will be preserved within flags passed by the build (except system
paths, which are moved to be last)
4. Flags for dependencies will appear between the root flags and the system
flags, and the flags for any dependency will come before those for *its*
dependencies (this is for completeness -- we already guarantee this in
`build_environment.py`)
Spack currently prepends include paths, library paths, and rpaths to the compile line. This causes problems when a header or library in the package has the same name as one exported by one of its dependencies. The *dependency's* header will be preferred over the package's, which is not what most builds expect. This also breaks some of our production codes.
This restores the original cc behavior (from *very* early Spack) of parsing compiler arguments out by type (`-L`, `-I`, `-Wl,-rpath`) and reconstituting the full command at the end.
`<includes> <other_args> <library dirs> <rpaths>`
This differs from the original behavior in one significant way, though: it *appends* the library arguments so that dependency libraries do not shadow those in the build.
This is safe because semantics aren't affected by *interleaving* `-I`, `-L`, and `-Wl,-rpath` arguments with others, only with each other (so the order fo two `-L` args affects the search path, but we search for all libraries on the command line using the same search path).
We preserve the following:
1. Any system directory in the paths will be listed last.
2. The root package's include/library/RPATH flags come before flags of the same type for any dependency.
3. Order will be preserved within flags passed by the build (except system paths, which are moved to be last)
4. Flags for dependencies will appear between the root flags and the system flags, and the flags for any dependency will come before those for *its* dependencies (this is for completeness -- we already guarantee this in `build_environment.py`)
If the user sets "ccache: true" in spack's config.yaml, Spack will use an available
ccache executable when compiling c/c++ code. This feature is disabled by default
(i.e. "ccache: false") and the documentation is updated with how to enable
ccache support
The name of the debug log written by the cc compiler wrapper was given
by Spec.short_spec, which includes the architecture. Somewhere along
the line Spec.format started adding spaces around the architecture
property so the filename started including spaces; the cc wrapper
script appears to ignore this, so files like spack-cc-bzip2-....in.log
(which record the wrapped compiler invocations) were not being
generated. This uses a different format string from the spec to
generate the wrapper log file names (which does not include spaces).
* Add support for IBM threaded compilers, xl*_r
Added new compiler class, xl_r; added default flags to the compilers.yaml file.
* Add cppflags to the set of default flags to be added to the compilers stanza in compiler.yaml.
These flags are optional. Only defined flags will be listed in the compilers.yaml file.
* Fix scripting warnings revealed by flake8.
Updated __init__.py and xl_r.py to conform with flake8 rules.
* Add justification to the definition of the XL default compiler flags.
* Allow compiler wrapper to modify environment
This adds the ability to set environment variables in the compiler
wrappers. These are specified as part of the compilers.yaml config.
The user may also specify RPATHs in compilers.yaml that should be
added.
* Minor doc tweak
- Fixed up dependency management so that:
- build deps go in PATH and -I
- link deps go in -L args
- only *immediate* link deps are RPATH'd
The latter reduces the number of libraries that need to be added to
DT_NEEDED / LC_RPATH. This removes redundant RPATHs to transitive
dependencies.
