zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

cc: convert compiler wrapper to posix shell

Browse Source 
This converts everything in cc to POSIX sh, except for the parts currently
handled with bash arrays. Tests are still passing.

This version tries to be as straightforward as possible. Specifically, most conversions
are kept simple -- convert ifs to ifs, handle indirect expansion the way we do in
`setup-env.sh`, only mess with the logic in `cc`, and don't mess with the python code at
all.

The big refactor is for arrays. We can't rely on bash's nice arrays and be ignorant of
separators anymore. So:

1. To avoid complicated separator logic, there are three types of lists. They are:

    * `$lsep`-separated lists, which end with `_list`. `lsep` is customizable, but we
      picked `^G` (alarm bell) for `$lsep` because it's ASCII and it's unlikely that it
      would actually appear in any arguments. If we need to get fancier (and I will lose
      faith in the world if we do) then we could consider XON or XOFF.
    * `:`-separated directory lists, which end with `_dirs`, `_DIRS`, `PATH`, or `PATHS`
    * Whitespace-separated lists (like flags), which can have any other name.

    Whitespace and colon-separated lists come with the territory with PATHs from env
    vars and lists of flags. `^G` separated lists are what we use for most internal
    variables, b/c it's more likely to work.

2. To avoid subshells, use a bunch of functions that do dirty `eval` stuff instead. This
   adds 3 functions to deal with lists:

    * `append LISTNAME ELEMENT [SEP]` will put `ELEMENT` at the end of the list called
      `LISTNAME`. You can optionally say what separator you expect to use. Note that we
      are taking advantage of everything being global and passing lists by name.

    * `prepend LISTNAME ELEMENT [SEP]` like append, but puts `ELEMENT` at the start of
      `LISTNAME`

    * `extend LISTNAME1 LISTNAME2 [PREFIX]` appends everything in LISTNAME2 to
       LISTNAME1, and optionally prepends `PREFIX` to every element (this is useful for
       things like `-I`, `-isystem `, etc.

    * `preextend LISTNAME1 LISTNAME2 [PREFIX]` prepends everything in LISTNAME2 to
       LISTNAME1 in order, and optionally prepends `PREFIX` to every element.

The routines determine the separator for each argument by its name, so we don't have to
pass around separators everywhere. Amazingly, as long as you do not expand variables'
values within an `eval` environment, you can do all this and still preserve quoting.
When iterating over lists, the user of this API still has to set and unset `IFS`
properly.

We ended up having to ignore shellcheck SC2034 (unused variable), because using evals
all over the place means that shellcheck doesn't notice that our list variables are
actually used.

So far this is looking pretty good. I took the most complex unit test I could find
(which runs a sample link line) and ran the same command line 200 times in a shell
script.  Times are roughly as follows:

For this invocation:

```console
$ bash -c 'time (for i in `seq 1 200`; do ~/test_cc.sh > /dev/null; done)'
```

I get the following performance numbers (the listed shells are what I put in `cc`'s
shebang):

**Original**
* Old version of `cc` with arrays and `bash v3.2.57` (macOS builtin): `4.462s` (`.022s` / call)
* Old version of `cc` with arrays and `bash v5.1.8` (Homebrew): `3.267s` (`.016s` / call)

**Using many subshells (#26408)**
*  with `bash v3.2.57`: `25.302s` (`.127s` / call)
*  with `bash v5.1.8`: `27.801s` (`.139s` / call)
*  with `dash`: `15.302s` (`.077s` / call)

This version didn't seem to work with zsh.

**This PR (no subshells)**
*  with `bash v3.2.57`: `4.973s` (`.025s` / call)
*  with `bash v5.1.8`: `4.984s` (`.025s` / call)
*  with `zsh`: `2.995s` (`.015s` / call)
*  with `dash`: `1.890s` (`.0095s` / call)

Dash, with the new posix design, is easily the winner.

So there are several interesting things to note here:

1. Running the posix version in `bash` is slower than using `bash` arrays. That is to be
   expected because it's doing a bunch of string processing where it likely did not have
   to before, at least in `bash`.

2. `zsh`, at least on macOS, is significantly faster than the ancient `bash` they ship
   with the system. Using `zsh` with the new version also makes the posix wrappers
   faster than `develop`. So it's worth preferring `zsh` if we have it. I suppose we
   should also try this with newer `bash` on Linux.

3. `bash v5.1.8` seems to be significantly faster than the old system `bash v3.2.57` for
   arrays. For straight POSIX stuff, it's a little slower. It did not seem to matter
   whether `--posix` was used.

4. `dash` is way faster than `bash` or `zsh`, so the real payoff just comes from being
   able to use it. I am not sure if that is mostly startup time, but it's significant.
   `dash` is ~2.4x faster than the original `bash` with arrays.

So, doing a lot of string stuff is slower than arrays, but converting to posix seems
worth it to be able to exploit `dash`.

- [x] Convert all but array-related portions to sh
- [x] Fix basic shellcheck issues.
- [x] Convert arrays to use a few convenience functions: `append` and `extend`
- [x] Get `cc` tests passing.
- [x] Add `cc` tests where needed passing.
- [x] Benchmarking.

Co-authored-by: Tom Scogland <scogland1@llnl.gov>
Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
This commit is contained in:
Todd Gamblin 2021-09-26 16:20:26 -07:00
parent 472638f025
commit 052b2e1b08
2 changed files with 434 additions and 292 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

563
lib/spack/env/cc vendored
View File

@ -1,4 +1,5 @@
#!/bin/bash #!/bin/sh
# shellcheck disable=SC2034 # evals in this script fool shellcheck
# #
# Copyright 2013-2021 Lawrence Livermore National Security, LLC and other # Copyright 2013-2021 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details. # Spack Project Developers. See the top-level COPYRIGHT file for details.
@ -20,25 +21,33 @@
# -Wl,-rpath arguments for dependency /lib directories. # -Wl,-rpath arguments for dependency /lib directories.
# #
# Reset IFS to the default: whitespace-separated lists. When we use
# other separators, we set and reset it.
unset IFS
# Separator for lists whose names end with `_list`.
# We pick the alarm bell character, which is highly unlikely to
# conflict with anything. This is a literal bell character (which
# we have to use since POSIX sh does not convert escape sequences
# like '\a' outside of the format argument of `printf`).
# NOTE: Depending on your editor this may look empty, but it is not.
readonly lsep=''
# This is an array of environment variables that need to be set before # This is an array of environment variables that need to be set before
# the script runs. They are set by routines in spack.build_environment # the script runs. They are set by routines in spack.build_environment
# as part of the package installation process. # as part of the package installation process.
parameters=( readonly params="\
SPACK_ENV_PATH SPACK_ENV_PATH
SPACK_DEBUG_LOG_DIR SPACK_DEBUG_LOG_DIR
SPACK_DEBUG_LOG_ID SPACK_DEBUG_LOG_ID
SPACK_COMPILER_SPEC SPACK_COMPILER_SPEC
SPACK_CC_RPATH_ARG SPACK_CC_RPATH_ARG
SPACK_CXX_RPATH_ARG SPACK_CXX_RPATH_ARG
SPACK_F77_RPATH_ARG SPACK_F77_RPATH_ARG
SPACK_FC_RPATH_ARG SPACK_FC_RPATH_ARG
SPACK_TARGET_ARGS SPACK_LINKER_ARG
SPACK_DTAGS_TO_ADD SPACK_SHORT_SPEC
SPACK_DTAGS_TO_STRIP SPACK_SYSTEM_DIRS"
SPACK_LINKER_ARG
SPACK_SHORT_SPEC
SPACK_SYSTEM_DIRS
)
# Optional parameters that aren't required to be set # Optional parameters that aren't required to be set
@ -58,60 +67,157 @@ parameters=(
# Test command is used to unit test the compiler script. # Test command is used to unit test the compiler script.
# SPACK_TEST_COMMAND # SPACK_TEST_COMMAND
# die() # die MESSAGE
# Prints a message and exits with error 1. # Print a message and exit with error code 1.
function die { die() {
echo "$@" echo "$@"
exit 1 exit 1
} }
# read input parameters into proper bash arrays. # empty VARNAME
# SYSTEM_DIRS is delimited by : # Return whether the variable VARNAME is unset or set to the empty string.
IFS=':' read -ra SPACK_SYSTEM_DIRS <<< "${SPACK_SYSTEM_DIRS}" empty() {
eval "test -z \"\${$1}\""
}
# SPACK_<LANG>FLAGS and SPACK_LDLIBS are split by ' ' # setsep LISTNAME
IFS=' ' read -ra SPACK_FFLAGS <<< "$SPACK_FFLAGS" # Set the global variable 'sep' to the separator for a list with name LISTNAME.
IFS=' ' read -ra SPACK_CPPFLAGS <<< "$SPACK_CPPFLAGS" # There are three types of lists:
IFS=' ' read -ra SPACK_CFLAGS <<< "$SPACK_CFLAGS" # 1. regular lists end with _list and are separated by $lsep
IFS=' ' read -ra SPACK_CXXFLAGS <<< "$SPACK_CXXFLAGS" # 2. directory lists end with _dirs/_DIRS/PATH(S) and are separated by ':'
IFS=' ' read -ra SPACK_LDFLAGS <<< "$SPACK_LDFLAGS" # 3. any other list is assumed to be separated by spaces: " "
IFS=' ' read -ra SPACK_LDLIBS <<< "$SPACK_LDLIBS" setsep() {
case "$1" in
*_dirs|*_DIRS|*PATH|*PATHS)
sep=':'
;;
*_list)
sep="$lsep"
;;
*)
sep=" "
;;
esac
}
# prepend LISTNAME ELEMENT [SEP]
#
# Prepend ELEMENT to the list stored in the variable LISTNAME,
# assuming the list is separated by SEP.
# Handles empty lists and single-element lists.
prepend() {
varname="$1"
elt="$2"
if empty "$varname"; then
eval "$varname=\"\${elt}\""
else
# Get the appropriate separator for the list we're appending to.
setsep "$varname"
eval "$varname=\"\${elt}${sep}\${$varname}\""
fi
}
# append LISTNAME ELEMENT [SEP]
#
# Append ELEMENT to the list stored in the variable LISTNAME,
# assuming the list is separated by SEP.
# Handles empty lists and single-element lists.
append() {
varname="$1"
elt="$2"
if empty "$varname"; then
eval "$varname=\"\${elt}\""
else
# Get the appropriate separator for the list we're appending to.
setsep "$varname"
eval "$varname=\"\${$varname}${sep}\${elt}\""
fi
}
# extend LISTNAME1 LISTNAME2 [PREFIX]
#
# Append the elements stored in the variable LISTNAME2
# to the list stored in LISTNAME1.
# If PREFIX is provided, prepend it to each element.
extend() {
# Figure out the appropriate IFS for the list we're reading.
setsep "$2"
if [ "$sep" != " " ]; then
IFS="$sep"
fi
eval "for elt in \${$2}; do append $1 \"$3\${elt}\"; done"
unset IFS
}
# preextend LISTNAME1 LISTNAME2 [PREFIX]
#
# Prepend the elements stored in the list at LISTNAME2
# to the list at LISTNAME1, preserving order.
# If PREFIX is provided, prepend it to each element.
preextend() {
# Figure out the appropriate IFS for the list we're reading.
setsep "$2"
if [ "$sep" != " " ]; then
IFS="$sep"
fi
# first, reverse the list to prepend
_reversed_list=""
eval "for elt in \${$2}; do prepend _reversed_list \"$3\${elt}\"; done"
# prepend reversed list to preextend in order
IFS="${lsep}"
for elt in $_reversed_list; do prepend "$1" "$3${elt}"; done
unset IFS
}
# system_dir PATH
# test whether a path is a system directory # test whether a path is a system directory
function system_dir { system_dir() {
IFS=':' # SPACK_SYSTEM_DIRS is colon-separated
path="$1" path="$1"
for sd in "${SPACK_SYSTEM_DIRS[@]}"; do for sd in $SPACK_SYSTEM_DIRS; do
if [ "${path}" == "${sd}" ] || [ "${path}" == "${sd}/" ]; then if [ "${path}" = "${sd}" ] || [ "${path}" = "${sd}/" ]; then
# success if path starts with a system prefix # success if path starts with a system prefix
unset IFS
return 0 return 0
fi fi
done done
unset IFS
return 1 # fail if path starts no system prefix return 1 # fail if path starts no system prefix
} }
for param in "${parameters[@]}"; do # Fail with a clear message if the input contains any bell characters.
if [[ -z ${!param+x} ]]; then if eval "[ \"\${*#*${lsep}}\" != \"\$*\" ]"; then
die "ERROR: Compiler command line contains our separator ('${lsep}'). Cannot parse."
fi
# ensure required variables are set
for param in $params; do
if eval "test -z \"\${${param}:-}\""; then
die "Spack compiler must be run from Spack! Input '$param' is missing." die "Spack compiler must be run from Spack! Input '$param' is missing."
fi fi
done done
# Check if optional parameters are defined # Check if optional parameters are defined
# If we aren't asking for debug flags, don't add them # If we aren't asking for debug flags, don't add them
if [[ -z ${SPACK_ADD_DEBUG_FLAGS+x} ]]; then if [ -z "${SPACK_ADD_DEBUG_FLAGS:-}" ]; then
SPACK_ADD_DEBUG_FLAGS="false" SPACK_ADD_DEBUG_FLAGS="false"
fi fi
# SPACK_ADD_DEBUG_FLAGS must be true/false/custom # SPACK_ADD_DEBUG_FLAGS must be true/false/custom
is_valid="false" is_valid="false"
for param in "true" "false" "custom"; do for param in "true" "false" "custom"; do
if [ "$param" == "$SPACK_ADD_DEBUG_FLAGS" ]; then if [ "$param" = "$SPACK_ADD_DEBUG_FLAGS" ]; then
is_valid="true" is_valid="true"
fi fi
done done
# Exit with error if we are given an incorrect value # Exit with error if we are given an incorrect value
if [ "$is_valid" == "false" ]; then if [ "$is_valid" = "false" ]; then
die "SPACK_ADD_DEBUG_FLAGS, if defined, must be one of 'true' 'false' or 'custom'" die "SPACK_ADD_DEBUG_FLAGS, if defined, must be one of 'true', 'false', or 'custom'."
fi fi
# Figure out the type of compiler, the language, and the mode so that # Figure out the type of compiler, the language, and the mode so that
@ -174,7 +280,7 @@ esac
# If any of the arguments below are present, then the mode is vcheck. # If any of the arguments below are present, then the mode is vcheck.
# In vcheck mode, nothing is added in terms of extra search paths or # In vcheck mode, nothing is added in terms of extra search paths or
# libraries. # libraries.
if [[ -z $mode ]] || [[ $mode == ld ]]; then if [ -z "$mode" ] || [ "$mode" = ld ]; then
for arg in "$@"; do for arg in "$@"; do
case $arg in case $arg in
-v|-V|--version|-dumpversion) -v|-V|--version|-dumpversion)
@ -186,16 +292,16 @@ if [[ -z $mode ]] || [[ $mode == ld ]]; then
fi fi
# Finish setting up the mode. # Finish setting up the mode.
if [[ -z $mode ]]; then if [ -z "$mode" ]; then
mode=ccld mode=ccld
for arg in "$@"; do for arg in "$@"; do
if [[ $arg == -E ]]; then if [ "$arg" = "-E" ]; then
mode=cpp mode=cpp
break break
elif [[ $arg == -S ]]; then elif [ "$arg" = "-S" ]; then
mode=as mode=as
break break
elif [[ $arg == -c ]]; then elif [ "$arg" = "-c" ]; then
mode=cc mode=cc
break break
fi fi
@ -222,17 +328,18 @@ dtags_to_strip="${SPACK_DTAGS_TO_STRIP}"
linker_arg="${SPACK_LINKER_ARG}" linker_arg="${SPACK_LINKER_ARG}"
# Set up rpath variable according to language. # Set up rpath variable according to language.
eval rpath=\$SPACK_${comp}_RPATH_ARG rpath="ERROR: RPATH ARG WAS NOT SET"
eval "rpath=\${SPACK_${comp}_RPATH_ARG:?${rpath}}"
# Dump the mode and exit if the command is dump-mode. # Dump the mode and exit if the command is dump-mode.
if [[ $SPACK_TEST_COMMAND == dump-mode ]]; then if [ "$SPACK_TEST_COMMAND" = "dump-mode" ]; then
echo "$mode" echo "$mode"
exit exit
fi fi
# Check that at least one of the real commands was actually selected, # Check that at least one of the real commands was actually selected,
# otherwise we don't know what to execute. # otherwise we don't know what to execute.
if [[ -z $command ]]; then if [ -z "$command" ]; then
die "ERROR: Compiler '$SPACK_COMPILER_SPEC' does not support compiling $language programs." die "ERROR: Compiler '$SPACK_COMPILER_SPEC' does not support compiling $language programs."
fi fi
@ -240,24 +347,26 @@ fi
# Filter '.' and Spack environment directories out of PATH so that # Filter '.' and Spack environment directories out of PATH so that
# this script doesn't just call itself # this script doesn't just call itself
# #
IFS=':' read -ra env_path <<< "$PATH" new_dirs=""
IFS=':' read -ra spack_env_dirs <<< "$SPACK_ENV_PATH" IFS=':'
spack_env_dirs+=("" ".") for dir in $PATH; do
export PATH=""
for dir in "${env_path[@]}"; do
addpath=true addpath=true
for env_dir in "${spack_env_dirs[@]}"; do for spack_env_dir in $SPACK_ENV_PATH; do
if [[ "${dir%%/}" == "$env_dir" ]]; then case "${dir%%/}" in
addpath=false "$spack_env_dir"|'.'|'')
break addpath=false
fi break
;;
esac
done done
if $addpath; then if [ $addpath = true ]; then
export PATH="${PATH:+$PATH:}$dir" append new_dirs "$dir"
fi fi
done done
unset IFS
export PATH="$new_dirs"
if [[ $mode == vcheck ]]; then if [ "$mode" = vcheck ]; then
exec "${command}" "$@" exec "${command}" "$@"
fi fi
@ -265,16 +374,20 @@ fi
# It doesn't work with -rpath. # It doesn't work with -rpath.
# This variable controls whether they are added. # This variable controls whether they are added.
add_rpaths=true add_rpaths=true
if [[ ($mode == ld || $mode == ccld) && "$SPACK_SHORT_SPEC" =~ "darwin" ]]; if [ "$mode" = ld ] || [ "$mode" = ccld ]; then
then if [ "${SPACK_SHORT_SPEC#*darwin}" != "${SPACK_SHORT_SPEC}" ]; then
for arg in "$@"; do for arg in "$@"; do
if [[ ($arg == -r && $mode == ld) || if [ "$arg" = "-r" ]; then
($arg == -r && $mode == ccld) || if [ "$mode" = ld ] || [ "$mode" = ccld ]; then
($arg == -Wl,-r && $mode == ccld) ]]; then add_rpaths=false
add_rpaths=false break
break fi
fi elif [ "$arg" = "-Wl,-r" ] && [ "$mode" = ccld ]; then
done add_rpaths=false
break
fi
done
fi
fi fi
# Save original command for debug logging # Save original command for debug logging
@ -297,17 +410,22 @@ input_command="$*"
# The libs variable is initialized here for completeness, and it is also # The libs variable is initialized here for completeness, and it is also
# used later to inject flags supplied via `ldlibs` on the command # used later to inject flags supplied via `ldlibs` on the command
# line. These come into the wrappers via SPACK_LDLIBS. # line. These come into the wrappers via SPACK_LDLIBS.
#
includes=() # The loop below breaks up the command line into these lists of components.
libdirs=() # The lists are all bell-separated to be as flexible as possible, as their
rpaths=() # contents may come from the command line, from ' '-separated lists,
system_includes=() # ':'-separated lists, etc.
system_libdirs=() include_dirs_list=""
system_rpaths=() lib_dirs_list=""
libs=() rpath_dirs_list=""
other_args=() system_include_dirs_list=""
isystem_system_includes=() system_lib_dirs_list=""
isystem_includes=() system_rpath_dirs_list=""
isystem_system_include_dirs_list=""
isystem_include_dirs_list=""
libs_list=""
other_args_list=""
while [ $# -ne 0 ]; do while [ $# -ne 0 ]; do
@ -327,32 +445,32 @@ while [ $# -ne 0 ]; do
isystem_was_used=true isystem_was_used=true
if [ -z "$arg" ]; then shift; arg="$1"; fi if [ -z "$arg" ]; then shift; arg="$1"; fi
if system_dir "$arg"; then if system_dir "$arg"; then
isystem_system_includes+=("$arg") append isystem_system_include_dirs_list "$arg"
else else
isystem_includes+=("$arg") append isystem_include_dirs_list "$arg"
fi fi
;; ;;
-I*) -I*)
arg="${1#-I}" arg="${1#-I}"
if [ -z "$arg" ]; then shift; arg="$1"; fi if [ -z "$arg" ]; then shift; arg="$1"; fi
if system_dir "$arg"; then if system_dir "$arg"; then
system_includes+=("$arg") append system_include_dirs_list "$arg"
else else
includes+=("$arg") append include_dirs_list "$arg"
fi fi
;; ;;
-L*) -L*)
arg="${1#-L}" arg="${1#-L}"
if [ -z "$arg" ]; then shift; arg="$1"; fi if [ -z "$arg" ]; then shift; arg="$1"; fi
if system_dir "$arg"; then if system_dir "$arg"; then
system_libdirs+=("$arg") append system_lib_dirs_list "$arg"
else else
libdirs+=("$arg") append lib_dirs_list "$arg"
fi fi
;; ;;
-l*) -l*)
# -loopopt=0 is generated erroneously in autoconf <= 2.69, # -loopopt=0 is generated erroneously in autoconf <= 2.69,
# and passed by ifx to the linker, which confuses it with a # and passed by ifx to the linker, which confuses it with a
# library. Filter it out. # library. Filter it out.
# TODO: generalize filtering of args with an env var, so that # TODO: generalize filtering of args with an env var, so that
# TODO: we do not have to special case this here. # TODO: we do not have to special case this here.
@ -363,66 +481,76 @@ while [ $# -ne 0 ]; do
fi fi
arg="${1#-l}" arg="${1#-l}"
if [ -z "$arg" ]; then shift; arg="$1"; fi if [ -z "$arg" ]; then shift; arg="$1"; fi
other_args+=("-l$arg") append other_args_list "-l$arg"
;; ;;
-Wl,*) -Wl,*)
arg="${1#-Wl,}" arg="${1#-Wl,}"
if [ -z "$arg" ]; then shift; arg="$1"; fi if [ -z "$arg" ]; then shift; arg="$1"; fi
if [[ "$arg" = -rpath=* ]]; then case "$arg" in
rp="${arg#-rpath=}" -rpath=*) rp="${arg#-rpath=}" ;;
elif [[ "$arg" = --rpath=* ]]; then --rpath=*) rp="${arg#--rpath=}" ;;
rp="${arg#--rpath=}" -rpath,*) rp="${arg#-rpath,}" ;;
elif [[ "$arg" = -rpath,* ]]; then --rpath,*) rp="${arg#--rpath,}" ;;
rp="${arg#-rpath,}" -rpath|--rpath)
elif [[ "$arg" = --rpath,* ]]; then shift; arg="$1"
rp="${arg#--rpath,}" case "$arg" in
elif [[ "$arg" =~ ^-?-rpath$ ]]; then -Wl,*)
shift; arg="$1" rp="${arg#-Wl,}"
if [[ "$arg" != -Wl,* ]]; then ;;
die "-Wl,-rpath was not followed by -Wl,*" *)
fi die "-Wl,-rpath was not followed by -Wl,*"
rp="${arg#-Wl,}" ;;
elif [[ "$arg" = "$dtags_to_strip" ]] ; then esac
: # We want to remove explicitly this flag ;;
else "$dtags_to_strip")
other_args+=("-Wl,$arg") : # We want to remove explicitly this flag
fi ;;
*)
append other_args_list "-Wl,$arg"
;;
esac
;; ;;
-Xlinker,*) -Xlinker,*)
arg="${1#-Xlinker,}" arg="${1#-Xlinker,}"
if [ -z "$arg" ]; then shift; arg="$1"; fi if [ -z "$arg" ]; then shift; arg="$1"; fi
if [[ "$arg" = -rpath=* ]]; then
rp="${arg#-rpath=}" case "$arg" in
elif [[ "$arg" = --rpath=* ]]; then -rpath=*) rp="${arg#-rpath=}" ;;
rp="${arg#--rpath=}" --rpath=*) rp="${arg#--rpath=}" ;;
elif [[ "$arg" = -rpath ]] || [[ "$arg" = --rpath ]]; then -rpath|--rpath)
shift; arg="$1" shift; arg="$1"
if [[ "$arg" != -Xlinker,* ]]; then case "$arg" in
die "-Xlinker,-rpath was not followed by -Xlinker,*" -Xlinker,*)
fi rp="${arg#-Xlinker,}"
rp="${arg#-Xlinker,}" ;;
else *)
other_args+=("-Xlinker,$arg") die "-Xlinker,-rpath was not followed by -Xlinker,*"
fi ;;
esac
;;
*)
append other_args_list "-Xlinker,$arg"
;;
esac
;; ;;
-Xlinker) -Xlinker)
if [[ "$2" == "-rpath" ]]; then if [ "$2" = "-rpath" ]; then
if [[ "$3" != "-Xlinker" ]]; then if [ "$3" != "-Xlinker" ]; then
die "-Xlinker,-rpath was not followed by -Xlinker,*" die "-Xlinker,-rpath was not followed by -Xlinker,*"
fi fi
shift 3; shift 3;
rp="$1" rp="$1"
elif [[ "$2" = "$dtags_to_strip" ]] ; then elif [ "$2" = "$dtags_to_strip" ]; then
shift # We want to remove explicitly this flag shift # We want to remove explicitly this flag
else else
other_args+=("$1") append other_args_list "$1"
fi fi
;; ;;
*) *)
if [[ "$1" = "$dtags_to_strip" ]] ; then if [ "$1" = "$dtags_to_strip" ]; then
: # We want to remove explicitly this flag : # We want to remove explicitly this flag
else else
other_args+=("$1") append other_args_list "$1"
fi fi
;; ;;
esac esac
@ -430,9 +558,9 @@ while [ $# -ne 0 ]; do
# test rpaths against system directories in one place. # test rpaths against system directories in one place.
if [ -n "$rp" ]; then if [ -n "$rp" ]; then
if system_dir "$rp"; then if system_dir "$rp"; then
system_rpaths+=("$rp") append system_rpath_dirs_list "$rp"
else else
rpaths+=("$rp") append rpath_dirs_list "$rp"
fi fi
fi fi
shift shift
@ -445,16 +573,15 @@ done
# See the gmake manual on implicit rules for details: # See the gmake manual on implicit rules for details:
# https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html # https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html
# #
flags=() flags_list=""
# Add debug flags # Add debug flags
if [ "${SPACK_ADD_DEBUG_FLAGS}" == "true" ]; then if [ "${SPACK_ADD_DEBUG_FLAGS}" = "true" ]; then
flags=("${flags[@]}" "${debug_flags}") extend flags_list debug_flags
# If a custom flag is requested, derive from environment # If a custom flag is requested, derive from environment
elif [ "$SPACK_ADD_DEBUG_FLAGS" == "custom" ]; then elif [ "$SPACK_ADD_DEBUG_FLAGS" = "custom" ]; then
IFS=' ' read -ra SPACK_DEBUG_FLAGS <<< "$SPACK_DEBUG_FLAGS" extend flags_list SPACK_DEBUG_FLAGS
flags=("${flags[@]}" "${SPACK_DEBUG_FLAGS[@]}")
fi fi
# Fortran flags come before CPPFLAGS # Fortran flags come before CPPFLAGS
@ -462,7 +589,8 @@ case "$mode" in
cc|ccld) cc|ccld)
case $lang_flags in case $lang_flags in
F) F)
flags=("${flags[@]}" "${SPACK_FFLAGS[@]}") ;; extend flags_list SPACK_FFLAGS
;;
esac esac
;; ;;
esac esac
@ -470,7 +598,8 @@ esac
# C preprocessor flags come before any C/CXX flags # C preprocessor flags come before any C/CXX flags
case "$mode" in case "$mode" in
cpp|as|cc|ccld) cpp|as|cc|ccld)
flags=("${flags[@]}" "${SPACK_CPPFLAGS[@]}") ;; extend flags_list SPACK_CPPFLAGS
;;
esac esac
@ -479,67 +608,67 @@ case "$mode" in
cc|ccld) cc|ccld)
case $lang_flags in case $lang_flags in
C) C)
flags=("${flags[@]}" "${SPACK_CFLAGS[@]}") ;; extend flags_list SPACK_CFLAGS
;;
CXX) CXX)
flags=("${flags[@]}" "${SPACK_CXXFLAGS[@]}") ;; extend flags_list SPACK_CXXFLAGS
;;
esac esac
flags=(${SPACK_TARGET_ARGS[@]} "${flags[@]}")
# prepend target args
preextend flags_list SPACK_TARGET_ARGS
;; ;;
esac esac
# Linker flags # Linker flags
case "$mode" in case "$mode" in
ld|ccld) ld|ccld)
flags=("${flags[@]}" "${SPACK_LDFLAGS[@]}") ;; extend flags_list SPACK_LDFLAGS
;;
esac esac
# On macOS insert headerpad_max_install_names linker flag # On macOS insert headerpad_max_install_names linker flag
if [[ ($mode == ld || $mode == ccld) && "$SPACK_SHORT_SPEC" =~ "darwin" ]]; if [ "$mode" = ld ] || [ "$mode" = ccld ]; then
then if [ "${SPACK_SHORT_SPEC#*darwin}" != "${SPACK_SHORT_SPEC}" ]; then
case "$mode" in case "$mode" in
ld) ld)
flags=("${flags[@]}" -headerpad_max_install_names) ;; append flags_list "-headerpad_max_install_names" ;;
ccld) ccld)
flags=("${flags[@]}" "-Wl,-headerpad_max_install_names") ;; append flags_list "-Wl,-headerpad_max_install_names" ;;
esac esac
fi
fi fi
IFS=':' read -ra rpath_dirs <<< "$SPACK_RPATH_DIRS" if [ "$mode" = ccld ] || [ "$mode" = ld ]; then
if [[ $mode == ccld || $mode == ld ]]; then if [ "$add_rpaths" != "false" ]; then
if [[ "$add_rpaths" != "false" ]] ; then
# Append RPATH directories. Note that in the case of the # Append RPATH directories. Note that in the case of the
# top-level package these directories may not exist yet. For dependencies # top-level package these directories may not exist yet. For dependencies
# it is assumed that paths have already been confirmed. # it is assumed that paths have already been confirmed.
rpaths=("${rpaths[@]}" "${rpath_dirs[@]}") extend rpath_dirs_list SPACK_RPATH_DIRS
fi fi
fi fi
IFS=':' read -ra link_dirs <<< "$SPACK_LINK_DIRS" if [ "$mode" = ccld ] || [ "$mode" = ld ]; then
if [[ $mode == ccld || $mode == ld ]]; then extend lib_dirs_list SPACK_LINK_DIRS
libdirs=("${libdirs[@]}" "${link_dirs[@]}")
fi fi
# add RPATHs if we're in in any linking mode # add RPATHs if we're in in any linking mode
case "$mode" in case "$mode" in
ld|ccld) ld|ccld)
# Set extra RPATHs # Set extra RPATHs
IFS=':' read -ra extra_rpaths <<< "$SPACK_COMPILER_EXTRA_RPATHS" extend lib_dirs_list SPACK_COMPILER_EXTRA_RPATHS
libdirs+=("${extra_rpaths[@]}") if [ "$add_rpaths" != "false" ]; then
if [[ "$add_rpaths" != "false" ]] ; then extend rpath_dirs_list SPACK_COMPILER_EXTRA_RPATHS
rpaths+=("${extra_rpaths[@]}")
fi fi
# Set implicit RPATHs # Set implicit RPATHs
IFS=':' read -ra implicit_rpaths <<< "$SPACK_COMPILER_IMPLICIT_RPATHS" if [ "$add_rpaths" != "false" ]; then
if [[ "$add_rpaths" != "false" ]] ; then extend rpath_dirs_list SPACK_COMPILER_IMPLICIT_RPATHS
rpaths+=("${implicit_rpaths[@]}")
fi fi
# Add SPACK_LDLIBS to args # Add SPACK_LDLIBS to args
for lib in "${SPACK_LDLIBS[@]}"; do for lib in $SPACK_LDLIBS; do
libs+=("${lib#-l}") append libs_list "${lib#-l}"
done done
;; ;;
esac esac
@ -547,63 +676,62 @@ esac
# #
# Finally, reassemble the command line. # Finally, reassemble the command line.
# #
args_list="$flags_list"
# Includes and system includes first
args=()
# flags assembled earlier
args+=("${flags[@]}")
# Insert include directories just prior to any system include directories # Insert include directories just prior to any system include directories
# NOTE: adding ${lsep} to the prefix here turns every added element into two
extend args_list include_dirs_list "-I"
extend args_list isystem_include_dirs_list "-isystem${lsep}"
for dir in "${includes[@]}"; do args+=("-I$dir"); done case "$mode" in
for dir in "${isystem_includes[@]}"; do args+=("-isystem" "$dir"); done cpp|cc|as|ccld)
if [ "$isystem_was_used" = "true" ]; then
extend args_list SPACK_INCLUDE_DIRS "-isystem${lsep}"
else
extend args_list SPACK_INCLUDE_DIRS "-I"
fi
;;
esac
IFS=':' read -ra spack_include_dirs <<< "$SPACK_INCLUDE_DIRS" extend args_list system_include_dirs_list -I
if [[ $mode == cpp || $mode == cc || $mode == as || $mode == ccld ]]; then extend args_list isystem_system_include_dirs_list "-isystem${lsep}"
if [[ "$isystem_was_used" == "true" ]] ; then
for dir in "${spack_include_dirs[@]}"; do args+=("-isystem" "$dir"); done
else
for dir in "${spack_include_dirs[@]}"; do args+=("-I$dir"); done
fi
fi
for dir in "${system_includes[@]}"; do args+=("-I$dir"); done
for dir in "${isystem_system_includes[@]}"; do args+=("-isystem" "$dir"); done
# Library search paths # Library search paths
for dir in "${libdirs[@]}"; do args+=("-L$dir"); done extend args_list lib_dirs_list "-L"
for dir in "${system_libdirs[@]}"; do args+=("-L$dir"); done extend args_list system_lib_dirs_list "-L"
# RPATHs arguments # RPATHs arguments
case "$mode" in case "$mode" in
ccld) ccld)
if [ -n "$dtags_to_add" ] ; then args+=("$linker_arg$dtags_to_add") ; fi if [ -n "$dtags_to_add" ] ; then
for dir in "${rpaths[@]}"; do args+=("$rpath$dir"); done append args_list "$linker_arg$dtags_to_add"
for dir in "${system_rpaths[@]}"; do args+=("$rpath$dir"); done fi
extend args_list rpath_dirs_list "$rpath"
extend args_list system_rpath_dirs_list "$rpath"
;; ;;
ld) ld)
if [ -n "$dtags_to_add" ] ; then args+=("$dtags_to_add") ; fi if [ -n "$dtags_to_add" ] ; then
for dir in "${rpaths[@]}"; do args+=("-rpath" "$dir"); done append args_list "$dtags_to_add"
for dir in "${system_rpaths[@]}"; do args+=("-rpath" "$dir"); done fi
extend args_list rpath_dirs_list "-rpath${lsep}"
extend args_list system_rpath_dirs_list "-rpath${lsep}"
;; ;;
esac esac
# Other arguments from the input command # Other arguments from the input command
args+=("${other_args[@]}") extend args_list other_args_list
# Inject SPACK_LDLIBS, if supplied # Inject SPACK_LDLIBS, if supplied
for lib in "${libs[@]}"; do extend args_list libs_list "-l"
args+=("-l$lib");
done
full_command=("$command" "${args[@]}") full_command_list="$command"
extend full_command_list args_list
# prepend the ccache binary if we're using ccache # prepend the ccache binary if we're using ccache
if [ -n "$SPACK_CCACHE_BINARY" ]; then if [ -n "$SPACK_CCACHE_BINARY" ]; then
case "$lang_flags" in case "$lang_flags" in
C|CXX) # ccache only supports C languages C|CXX) # ccache only supports C languages
full_command=("${SPACK_CCACHE_BINARY}" "${full_command[@]}") prepend full_command_list "${SPACK_CCACHE_BINARY}"
# workaround for stage being a temp folder # workaround for stage being a temp folder
# see #3761#issuecomment-294352232 # see #3761#issuecomment-294352232
export CCACHE_NOHASHDIR=yes export CCACHE_NOHASHDIR=yes
@ -612,25 +740,36 @@ if [ -n "$SPACK_CCACHE_BINARY" ]; then
fi fi
# dump the full command if the caller supplies SPACK_TEST_COMMAND=dump-args # dump the full command if the caller supplies SPACK_TEST_COMMAND=dump-args
if [[ $SPACK_TEST_COMMAND == dump-args ]]; then if [ -n "${SPACK_TEST_COMMAND=}" ]; then
IFS=" case "$SPACK_TEST_COMMAND" in
" && echo "${full_command[*]}" dump-args)
exit IFS="$lsep"
elif [[ $SPACK_TEST_COMMAND =~ dump-env-* ]]; then for arg in $full_command_list; do
var=${SPACK_TEST_COMMAND#dump-env-} echo "$arg"
echo "$0: $var: ${!var}" done
elif [[ -n $SPACK_TEST_COMMAND ]]; then unset IFS
die "ERROR: Unknown test command" exit
;;
dump-env-*)
var=${SPACK_TEST_COMMAND#dump-env-}
eval "printf '%s\n' \"\$0: \$var: \$$var\""
;;
*)
die "ERROR: Unknown test command: '$SPACK_TEST_COMMAND'"
;;
esac
fi fi
# #
# Write the input and output commands to debug logs if it's asked for. # Write the input and output commands to debug logs if it's asked for.
# #
if [[ $SPACK_DEBUG == TRUE ]]; then if [ "$SPACK_DEBUG" = TRUE ]; then
input_log="$SPACK_DEBUG_LOG_DIR/spack-cc-$SPACK_DEBUG_LOG_ID.in.log" input_log="$SPACK_DEBUG_LOG_DIR/spack-cc-$SPACK_DEBUG_LOG_ID.in.log"
output_log="$SPACK_DEBUG_LOG_DIR/spack-cc-$SPACK_DEBUG_LOG_ID.out.log" output_log="$SPACK_DEBUG_LOG_DIR/spack-cc-$SPACK_DEBUG_LOG_ID.out.log"
echo "[$mode] $command $input_command" >> "$input_log" echo "[$mode] $command $input_command" >> "$input_log"
echo "[$mode] ${full_command[*]}" >> "$output_log" echo "[$mode] ${full_command_list}" >> "$output_log"
fi fi
exec "${full_command[@]}" # Execute the full command, preserving spaces with IFS set
# to the alarm bell separator.
IFS="$lsep"; exec $full_command_list

163
lib/spack/spack/test/cc.py
View File

@ -13,13 +13,13 @@
from spack.paths import build_env_path from spack.paths import build_env_path
from spack.util.environment import set_env, system_dirs from spack.util.environment import set_env, system_dirs
from spack.util.executable import Executable from spack.util.executable import Executable, ProcessError
# #
# Complicated compiler test command # Complicated compiler test command
# #
test_args = [ test_args = [
'-I/test/include', '-L/test/lib', '-L/other/lib', '-I/other/include', '-I/test/include', '-L/test/lib', '-L/with space/lib', '-I/other/include',
'arg1', 'arg1',
'-Wl,--start-group', '-Wl,--start-group',
'arg2', 'arg2',
@ -31,7 +31,9 @@
'-Xlinker', '-rpath', '-Xlinker', '/fourth/rpath', '-Xlinker', '-rpath', '-Xlinker', '/fourth/rpath',
'-Wl,--rpath,/fifth/rpath', '-Wl,--rpath', '-Wl,/sixth/rpath', '-Wl,--rpath,/fifth/rpath', '-Wl,--rpath', '-Wl,/sixth/rpath',
'-llib3', '-llib4', '-llib3', '-llib4',
'arg5', 'arg6'] 'arg5', 'arg6',
'"-DDOUBLE_QUOTED_ARG"', "'-DSINGLE_QUOTED_ARG'",
]
# #
# Pieces of the test command above, as they should be parsed out. # Pieces of the test command above, as they should be parsed out.
@ -43,7 +45,7 @@
'-I/test/include', '-I/other/include'] '-I/test/include', '-I/other/include']
test_library_paths = [ test_library_paths = [
'-L/test/lib', '-L/other/lib'] '-L/test/lib', '-L/with space/lib']
test_wl_rpaths = [ test_wl_rpaths = [
'-Wl,-rpath,/first/rpath', '-Wl,-rpath,/second/rpath', '-Wl,-rpath,/first/rpath', '-Wl,-rpath,/second/rpath',
@ -60,7 +62,9 @@
'-Wl,--start-group', '-Wl,--start-group',
'arg2', 'arg3', '-llib1', '-llib2', 'arg4', 'arg2', 'arg3', '-llib1', '-llib2', 'arg4',
'-Wl,--end-group', '-Wl,--end-group',
'-llib3', '-llib4', 'arg5', 'arg6'] '-llib3', '-llib4', 'arg5', 'arg6',
'"-DDOUBLE_QUOTED_ARG"', "'-DSINGLE_QUOTED_ARG'",
]
#: The prefix of the package being mock installed #: The prefix of the package being mock installed
pkg_prefix = '/spack-test-prefix' pkg_prefix = '/spack-test-prefix'
@ -86,6 +90,17 @@
lheaderpad = ['-Wl,-headerpad_max_install_names'] lheaderpad = ['-Wl,-headerpad_max_install_names']
headerpad = ['-headerpad_max_install_names'] headerpad = ['-headerpad_max_install_names']
target_args = ["-march=znver2", "-mtune=znver2"]
# common compile arguments: includes, libs, -Wl linker args, other args
common_compile_args = (
test_include_paths +
test_library_paths +
['-Wl,--disable-new-dtags'] +
test_wl_rpaths +
test_args_without_paths
)
@pytest.fixture(scope='session') @pytest.fixture(scope='session')
def wrapper_environment(): def wrapper_environment():
@ -107,7 +122,7 @@ def wrapper_environment():
SPACK_LINK_DIRS=None, SPACK_LINK_DIRS=None,
SPACK_INCLUDE_DIRS=None, SPACK_INCLUDE_DIRS=None,
SPACK_RPATH_DIRS=None, SPACK_RPATH_DIRS=None,
SPACK_TARGET_ARGS='', SPACK_TARGET_ARGS="-march=znver2 -mtune=znver2",
SPACK_LINKER_ARG='-Wl,', SPACK_LINKER_ARG='-Wl,',
SPACK_DTAGS_TO_ADD='--disable-new-dtags', SPACK_DTAGS_TO_ADD='--disable-new-dtags',
SPACK_DTAGS_TO_STRIP='--enable-new-dtags'): SPACK_DTAGS_TO_STRIP='--enable-new-dtags'):
@ -126,9 +141,6 @@ def wrapper_flags():
yield yield
pytestmark = pytest.mark.usefixtures('wrapper_environment')
def check_args(cc, args, expected): def check_args(cc, args, expected):
"""Check output arguments that cc produces when called with args. """Check output arguments that cc produces when called with args.
@ -149,7 +161,7 @@ def check_env_var(executable, var, expected):
""" """
with set_env(SPACK_TEST_COMMAND='dump-env-' + var): with set_env(SPACK_TEST_COMMAND='dump-env-' + var):
output = executable(*test_args, output=str).strip() output = executable(*test_args, output=str).strip()
assert output == executable.path + ': ' + var + ': ' + expected assert executable.path + ': ' + var + ': ' + expected == output
def dump_mode(cc, args): def dump_mode(cc, args):
@ -158,7 +170,13 @@ def dump_mode(cc, args):
return cc(*args, output=str).strip() return cc(*args, output=str).strip()
def test_vcheck_mode(): def test_no_wrapper_environment():
with pytest.raises(ProcessError):
output = cc(output=str)
assert "Spack compiler must be run from Spack" in output
def test_vcheck_mode(wrapper_environment):
assert dump_mode(cc, ['-I/include', '--version']) == 'vcheck' assert dump_mode(cc, ['-I/include', '--version']) == 'vcheck'
assert dump_mode(cc, ['-I/include', '-V']) == 'vcheck' assert dump_mode(cc, ['-I/include', '-V']) == 'vcheck'
assert dump_mode(cc, ['-I/include', '-v']) == 'vcheck' assert dump_mode(cc, ['-I/include', '-v']) == 'vcheck'
@ -167,17 +185,17 @@ def test_vcheck_mode():
assert dump_mode(cc, ['-I/include', '-V', '-o', 'output']) == 'vcheck' assert dump_mode(cc, ['-I/include', '-V', '-o', 'output']) == 'vcheck'
def test_cpp_mode(): def test_cpp_mode(wrapper_environment):
assert dump_mode(cc, ['-E']) == 'cpp' assert dump_mode(cc, ['-E']) == 'cpp'
assert dump_mode(cxx, ['-E']) == 'cpp' assert dump_mode(cxx, ['-E']) == 'cpp'
assert dump_mode(cpp, []) == 'cpp' assert dump_mode(cpp, []) == 'cpp'
def test_as_mode(): def test_as_mode(wrapper_environment):
assert dump_mode(cc, ['-S']) == 'as' assert dump_mode(cc, ['-S']) == 'as'
def test_ccld_mode(): def test_ccld_mode(wrapper_environment):
assert dump_mode(cc, []) == 'ccld' assert dump_mode(cc, []) == 'ccld'
assert dump_mode(cc, ['foo.c', '-o', 'foo']) == 'ccld' assert dump_mode(cc, ['foo.c', '-o', 'foo']) == 'ccld'
assert dump_mode(cc, ['foo.c', '-o', 'foo', '-Wl,-rpath,foo']) == 'ccld' assert dump_mode(cc, ['foo.c', '-o', 'foo', '-Wl,-rpath,foo']) == 'ccld'
@ -185,13 +203,13 @@ def test_ccld_mode():
'foo.o', 'bar.o', 'baz.o', '-o', 'foo', '-Wl,-rpath,foo']) == 'ccld' 'foo.o', 'bar.o', 'baz.o', '-o', 'foo', '-Wl,-rpath,foo']) == 'ccld'
def test_ld_mode(): def test_ld_mode(wrapper_environment):
assert dump_mode(ld, []) == 'ld' assert dump_mode(ld, []) == 'ld'
assert dump_mode(ld, [ assert dump_mode(ld, [
'foo.o', 'bar.o', 'baz.o', '-o', 'foo', '-Wl,-rpath,foo']) == 'ld' 'foo.o', 'bar.o', 'baz.o', '-o', 'foo', '-Wl,-rpath,foo']) == 'ld'
def test_ld_flags(wrapper_flags): def test_ld_flags(wrapper_environment, wrapper_flags):
check_args( check_args(
ld, test_args, ld, test_args,
['ld'] + ['ld'] +
@ -204,7 +222,7 @@ def test_ld_flags(wrapper_flags):
spack_ldlibs) spack_ldlibs)
def test_cpp_flags(wrapper_flags): def test_cpp_flags(wrapper_environment, wrapper_flags):
check_args( check_args(
cpp, test_args, cpp, test_args,
['cpp'] + ['cpp'] +
@ -214,69 +232,58 @@ def test_cpp_flags(wrapper_flags):
test_args_without_paths) test_args_without_paths)
def test_cc_flags(wrapper_flags): def test_cc_flags(wrapper_environment, wrapper_flags):
check_args( check_args(
cc, test_args, cc, test_args,
[real_cc] + [real_cc] +
target_args +
spack_cppflags + spack_cppflags +
spack_cflags + spack_cflags +
spack_ldflags + spack_ldflags +
test_include_paths + common_compile_args +
test_library_paths +
['-Wl,--disable-new-dtags'] +
test_wl_rpaths +
test_args_without_paths +
spack_ldlibs) spack_ldlibs)
def test_cxx_flags(wrapper_flags): def test_cxx_flags(wrapper_environment, wrapper_flags):
check_args( check_args(
cxx, test_args, cxx, test_args,
[real_cc] + [real_cc] +
target_args +
spack_cppflags + spack_cppflags +
spack_cxxflags + spack_cxxflags +
spack_ldflags + spack_ldflags +
test_include_paths + common_compile_args +
test_library_paths +
['-Wl,--disable-new-dtags'] +
test_wl_rpaths +
test_args_without_paths +
spack_ldlibs) spack_ldlibs)
def test_fc_flags(wrapper_flags): def test_fc_flags(wrapper_environment, wrapper_flags):
check_args( check_args(
fc, test_args, fc, test_args,
[real_cc] + [real_cc] +
target_args +
spack_fflags + spack_fflags +
spack_cppflags + spack_cppflags +
spack_ldflags + spack_ldflags +
test_include_paths + common_compile_args +
test_library_paths +
['-Wl,--disable-new-dtags'] +
test_wl_rpaths +
test_args_without_paths +
spack_ldlibs) spack_ldlibs)
def test_dep_rpath(): def test_dep_rpath(wrapper_environment):
"""Ensure RPATHs for root package are added.""" """Ensure RPATHs for root package are added."""
check_args( check_args(
cc, test_args, cc, test_args,
[real_cc] + [real_cc] +
test_include_paths + target_args +
test_library_paths + common_compile_args)
['-Wl,--disable-new-dtags'] +
test_wl_rpaths +
test_args_without_paths)
def test_dep_include(): def test_dep_include(wrapper_environment):
"""Ensure a single dependency include directory is added.""" """Ensure a single dependency include directory is added."""
with set_env(SPACK_INCLUDE_DIRS='x'): with set_env(SPACK_INCLUDE_DIRS='x'):
check_args( check_args(
cc, test_args, cc, test_args,
[real_cc] + [real_cc] +
target_args +
test_include_paths + test_include_paths +
['-Ix'] + ['-Ix'] +
test_library_paths + test_library_paths +
@ -285,7 +292,7 @@ def test_dep_include():
test_args_without_paths) test_args_without_paths)
def test_system_path_cleanup(): def test_system_path_cleanup(wrapper_environment):
"""Ensure SPACK_ENV_PATH is removed from PATH, even with trailing / """Ensure SPACK_ENV_PATH is removed from PATH, even with trailing /
The compiler wrapper has to ensure that it is not called nested The compiler wrapper has to ensure that it is not called nested
@ -305,13 +312,14 @@ def test_system_path_cleanup():
check_env_var(cc, 'PATH', system_path) check_env_var(cc, 'PATH', system_path)
def test_dep_lib(): def test_dep_lib(wrapper_environment):
"""Ensure a single dependency RPATH is added.""" """Ensure a single dependency RPATH is added."""
with set_env(SPACK_LINK_DIRS='x', with set_env(SPACK_LINK_DIRS='x',
SPACK_RPATH_DIRS='x'): SPACK_RPATH_DIRS='x'):
check_args( check_args(
cc, test_args, cc, test_args,
[real_cc] + [real_cc] +
target_args +
test_include_paths + test_include_paths +
test_library_paths + test_library_paths +
['-Lx'] + ['-Lx'] +
@ -321,12 +329,13 @@ def test_dep_lib():
test_args_without_paths) test_args_without_paths)
def test_dep_lib_no_rpath(): def test_dep_lib_no_rpath(wrapper_environment):
"""Ensure a single dependency link flag is added with no dep RPATH.""" """Ensure a single dependency link flag is added with no dep RPATH."""
with set_env(SPACK_LINK_DIRS='x'): with set_env(SPACK_LINK_DIRS='x'):
check_args( check_args(
cc, test_args, cc, test_args,
[real_cc] + [real_cc] +
target_args +
test_include_paths + test_include_paths +
test_library_paths + test_library_paths +
['-Lx'] + ['-Lx'] +
@ -335,12 +344,13 @@ def test_dep_lib_no_rpath():
test_args_without_paths) test_args_without_paths)
def test_dep_lib_no_lib(): def test_dep_lib_no_lib(wrapper_environment):
"""Ensure a single dependency RPATH is added with no -L.""" """Ensure a single dependency RPATH is added with no -L."""
with set_env(SPACK_RPATH_DIRS='x'): with set_env(SPACK_RPATH_DIRS='x'):
check_args( check_args(
cc, test_args, cc, test_args,
[real_cc] + [real_cc] +
target_args +
test_include_paths + test_include_paths +
test_library_paths + test_library_paths +
['-Wl,--disable-new-dtags'] + ['-Wl,--disable-new-dtags'] +
@ -349,7 +359,7 @@ def test_dep_lib_no_lib():
test_args_without_paths) test_args_without_paths)
def test_ccld_deps(): def test_ccld_deps(wrapper_environment):
"""Ensure all flags are added in ccld mode.""" """Ensure all flags are added in ccld mode."""
with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc', with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc',
SPACK_RPATH_DIRS='xlib:ylib:zlib', SPACK_RPATH_DIRS='xlib:ylib:zlib',
@ -357,6 +367,7 @@ def test_ccld_deps():
check_args( check_args(
cc, test_args, cc, test_args,
[real_cc] + [real_cc] +
target_args +
test_include_paths + test_include_paths +
['-Ixinc', ['-Ixinc',
'-Iyinc', '-Iyinc',
@ -373,7 +384,7 @@ def test_ccld_deps():
test_args_without_paths) test_args_without_paths)
def test_ccld_deps_isystem(): def test_ccld_deps_isystem(wrapper_environment):
"""Ensure all flags are added in ccld mode. """Ensure all flags are added in ccld mode.
When a build uses -isystem, Spack should inject it's When a build uses -isystem, Spack should inject it's
include paths using -isystem. Spack will insert these include paths using -isystem. Spack will insert these
@ -386,6 +397,7 @@ def test_ccld_deps_isystem():
check_args( check_args(
cc, mytest_args, cc, mytest_args,
[real_cc] + [real_cc] +
target_args +
test_include_paths + test_include_paths +
['-isystem', 'fooinc', ['-isystem', 'fooinc',
'-isystem', 'xinc', '-isystem', 'xinc',
@ -403,7 +415,7 @@ def test_ccld_deps_isystem():
test_args_without_paths) test_args_without_paths)
def test_cc_deps(): def test_cc_deps(wrapper_environment):
"""Ensure -L and RPATHs are not added in cc mode.""" """Ensure -L and RPATHs are not added in cc mode."""
with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc', with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc',
SPACK_RPATH_DIRS='xlib:ylib:zlib', SPACK_RPATH_DIRS='xlib:ylib:zlib',
@ -411,6 +423,7 @@ def test_cc_deps():
check_args( check_args(
cc, ['-c'] + test_args, cc, ['-c'] + test_args,
[real_cc] + [real_cc] +
target_args +
test_include_paths + test_include_paths +
['-Ixinc', ['-Ixinc',
'-Iyinc', '-Iyinc',
@ -420,7 +433,7 @@ def test_cc_deps():
test_args_without_paths) test_args_without_paths)
def test_ccld_with_system_dirs(): def test_ccld_with_system_dirs(wrapper_environment):
"""Ensure all flags are added in ccld mode.""" """Ensure all flags are added in ccld mode."""
with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc', with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc',
SPACK_RPATH_DIRS='xlib:ylib:zlib', SPACK_RPATH_DIRS='xlib:ylib:zlib',
@ -434,6 +447,7 @@ def test_ccld_with_system_dirs():
check_args( check_args(
cc, sys_path_args + test_args, cc, sys_path_args + test_args,
[real_cc] + [real_cc] +
target_args +
test_include_paths + test_include_paths +
['-Ixinc', ['-Ixinc',
'-Iyinc', '-Iyinc',
@ -455,7 +469,7 @@ def test_ccld_with_system_dirs():
test_args_without_paths) test_args_without_paths)
def test_ccld_with_system_dirs_isystem(): def test_ccld_with_system_dirs_isystem(wrapper_environment):
"""Ensure all flags are added in ccld mode. """Ensure all flags are added in ccld mode.
Ensure that includes are in the proper Ensure that includes are in the proper
place when a build uses -isystem, and uses place when a build uses -isystem, and uses
@ -472,6 +486,7 @@ def test_ccld_with_system_dirs_isystem():
check_args( check_args(
cc, sys_path_args + test_args, cc, sys_path_args + test_args,
[real_cc] + [real_cc] +
target_args +
test_include_paths + test_include_paths +
['-isystem', 'xinc', ['-isystem', 'xinc',
'-isystem', 'yinc', '-isystem', 'yinc',
@ -493,7 +508,7 @@ def test_ccld_with_system_dirs_isystem():
test_args_without_paths) test_args_without_paths)
def test_ld_deps(): def test_ld_deps(wrapper_environment):
"""Ensure no (extra) -I args or -Wl, are passed in ld mode.""" """Ensure no (extra) -I args or -Wl, are passed in ld mode."""
with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc', with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc',
SPACK_RPATH_DIRS='xlib:ylib:zlib', SPACK_RPATH_DIRS='xlib:ylib:zlib',
@ -514,7 +529,7 @@ def test_ld_deps():
test_args_without_paths) test_args_without_paths)
def test_ld_deps_no_rpath(): def test_ld_deps_no_rpath(wrapper_environment):
"""Ensure SPACK_LINK_DEPS controls -L for ld.""" """Ensure SPACK_LINK_DEPS controls -L for ld."""
with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc', with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc',
SPACK_LINK_DIRS='xlib:ylib:zlib'): SPACK_LINK_DIRS='xlib:ylib:zlib'):
@ -531,7 +546,7 @@ def test_ld_deps_no_rpath():
test_args_without_paths) test_args_without_paths)
def test_ld_deps_no_link(): def test_ld_deps_no_link(wrapper_environment):
"""Ensure SPACK_RPATH_DEPS controls -rpath for ld.""" """Ensure SPACK_RPATH_DEPS controls -rpath for ld."""
with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc', with set_env(SPACK_INCLUDE_DIRS='xinc:yinc:zinc',
SPACK_RPATH_DIRS='xlib:ylib:zlib'): SPACK_RPATH_DIRS='xlib:ylib:zlib'):
@ -548,7 +563,7 @@ def test_ld_deps_no_link():
test_args_without_paths) test_args_without_paths)
def test_ld_deps_partial(): def test_ld_deps_partial(wrapper_environment):
"""Make sure ld -r (partial link) is handled correctly on OS's where it """Make sure ld -r (partial link) is handled correctly on OS's where it
doesn't accept rpaths. doesn't accept rpaths.
""" """
@ -586,57 +601,45 @@ def test_ld_deps_partial():
test_args_without_paths) test_args_without_paths)
def test_ccache_prepend_for_cc(): def test_ccache_prepend_for_cc(wrapper_environment):
with set_env(SPACK_CCACHE_BINARY='ccache'): with set_env(SPACK_CCACHE_BINARY='ccache'):
os.environ['SPACK_SHORT_SPEC'] = "foo@1.2=linux-x86_64" os.environ['SPACK_SHORT_SPEC'] = "foo@1.2=linux-x86_64"
check_args( check_args(
cc, test_args, cc, test_args,
['ccache'] + # ccache prepended in cc mode ['ccache'] + # ccache prepended in cc mode
[real_cc] + [real_cc] +
test_include_paths + target_args +
test_library_paths + common_compile_args)
['-Wl,--disable-new-dtags'] +
test_wl_rpaths +
test_args_without_paths)
os.environ['SPACK_SHORT_SPEC'] = "foo@1.2=darwin-x86_64" os.environ['SPACK_SHORT_SPEC'] = "foo@1.2=darwin-x86_64"
check_args( check_args(
cc, test_args, cc, test_args,
['ccache'] + # ccache prepended in cc mode ['ccache'] + # ccache prepended in cc mode
[real_cc] + [real_cc] +
target_args +
lheaderpad + lheaderpad +
test_include_paths + common_compile_args)
test_library_paths +
['-Wl,--disable-new-dtags'] +
test_wl_rpaths +
test_args_without_paths)
def test_no_ccache_prepend_for_fc(): def test_no_ccache_prepend_for_fc(wrapper_environment):
os.environ['SPACK_SHORT_SPEC'] = "foo@1.2=linux-x86_64" os.environ['SPACK_SHORT_SPEC'] = "foo@1.2=linux-x86_64"
check_args( check_args(
fc, test_args, fc, test_args,
# no ccache for Fortran # no ccache for Fortran
[real_cc] + [real_cc] +
test_include_paths + target_args +
test_library_paths + common_compile_args)
['-Wl,--disable-new-dtags'] +
test_wl_rpaths +
test_args_without_paths)
os.environ['SPACK_SHORT_SPEC'] = "foo@1.2=darwin-x86_64" os.environ['SPACK_SHORT_SPEC'] = "foo@1.2=darwin-x86_64"
check_args( check_args(
fc, test_args, fc, test_args,
# no ccache for Fortran # no ccache for Fortran
[real_cc] + [real_cc] +
target_args +
lheaderpad + lheaderpad +
test_include_paths + common_compile_args)
test_library_paths +
['-Wl,--disable-new-dtags'] +
test_wl_rpaths +
test_args_without_paths)
@pytest.mark.regression('9160') @pytest.mark.regression('9160')
def test_disable_new_dtags(wrapper_flags): def test_disable_new_dtags(wrapper_environment, wrapper_flags):
with set_env(SPACK_TEST_COMMAND='dump-args'): with set_env(SPACK_TEST_COMMAND='dump-args'):
result = ld(*test_args, output=str).strip().split('\n') result = ld(*test_args, output=str).strip().split('\n')
assert '--disable-new-dtags' in result assert '--disable-new-dtags' in result
@ -645,7 +648,7 @@ def test_disable_new_dtags(wrapper_flags):
@pytest.mark.regression('9160') @pytest.mark.regression('9160')
def test_filter_enable_new_dtags(wrapper_flags): def test_filter_enable_new_dtags(wrapper_environment, wrapper_flags):
with set_env(SPACK_TEST_COMMAND='dump-args'): with set_env(SPACK_TEST_COMMAND='dump-args'):
result = ld(*(test_args + ['--enable-new-dtags']), output=str) result = ld(*(test_args + ['--enable-new-dtags']), output=str)
result = result.strip().split('\n') result = result.strip().split('\n')
@ -657,7 +660,7 @@ def test_filter_enable_new_dtags(wrapper_flags):
@pytest.mark.regression('22643') @pytest.mark.regression('22643')
def test_linker_strips_loopopt(wrapper_flags): def test_linker_strips_loopopt(wrapper_environment, wrapper_flags):
with set_env(SPACK_TEST_COMMAND='dump-args'): with set_env(SPACK_TEST_COMMAND='dump-args'):
# ensure that -loopopt=0 is not present in ld mode # ensure that -loopopt=0 is not present in ld mode
result = ld(*(test_args + ["-loopopt=0"]), output=str) result = ld(*(test_args + ["-loopopt=0"]), output=str)