Commit Graph

120 Commits

Author SHA1 Message Date
Harmen Stoppels
fce95e2efb
license year bump (#34921)
* license bump year
* fix black issues of modified files
* mypy
* fix 2021 -> 2023
2023-01-18 14:30:17 -08:00
Massimiliano Culpo
033cb86fd6 Add vendored packages back 2023-01-04 09:43:04 -08:00
Massimiliano Culpo
5175189412 Delete outdated externals 2023-01-04 09:43:04 -08:00
Massimiliano Culpo
86378502f9 Use "vendoring" to manage 3rd party dependencies 2023-01-04 09:43:04 -08:00
Massimiliano Culpo
b2c806f6fc
archspec: add support for zen4 (#34609)
Also add:
- Upper bound for Xeon Phi compiler support
- Better detection for a64fx
2022-12-20 11:22:50 +01:00
Massimiliano Culpo
b3124bff7c
Stop using six in Spack (#33905)
Since we dropped support for Python 2.7, there's no need
so use `six` anymore. We still need to vendor it until
we update our vendored dependencies.
2022-11-15 10:07:54 +01:00
Massimiliano Culpo
3efa4ee26f
Remove support for running with Python 2.7 (#33063)
* Remove CI jobs related to Python 2.7

* Remove Python 2.7 specific code from Spack core

* Remove externals for Python 2 only

* Remove llnl.util.compat
2022-11-14 13:11:28 +01:00
Massimiliano Culpo
e045dabb3a
archspec: update version, translate renamed uarchs (#33556)
* Update archspec version

* Add a translation table from old names
2022-11-07 04:50:38 -08:00
Betsy McPhail
ad95719a1d
Use threading.TIMEOUT_MAX when available (#32399)
This value was introduced in Python 3.2. Specifying a timeout greater than
this value will raise an OverflowError.
2022-08-26 17:37:56 -06:00
Massimiliano Culpo
b6ea2a46d1
Update archspec to latest commit (#32368)
Modifications:

- [x] Add graviton3
- [x] Optimize __eq__ for microarchitectures
2022-08-26 12:58:20 +02:00
Massimiliano Culpo
4a0ac87d07
archspec: bump to v0.1.4 (#30856)
Fixes compiler flags for oneapi and dpcpp
2022-06-07 08:51:34 +00:00
Massimiliano Culpo
7c4cc1c71c
archspec: add oneapi and dpcpp flag support (#30783) 2022-05-23 13:28:54 -07:00
Massimiliano Culpo
c775c322ec
vendored externals: update archspec (#30683)
- Better support for 164fx
- Better support for Apple M1(pro)
2022-05-18 11:31:20 +02:00
John Parent
4aee27816e Windows Support: Testing Suite integration
Broaden support for execution of the test suite
on Windows.
General bug and review fixups
2022-03-17 09:01:01 -07:00
Betsy McPhail
06aef626cb Update tests support for Windows
Fixup common tests

    * Remove requirement for Python 2.6
    * Skip new failing test

Windows: Update url util to handle Windows paths (#27959)

    * update url util to handle windows paths

    * Update tests to handle fixed url handling

    * canonicalize path only when the path type matches the host platform

    * Skip some url tests on Windows

Co-authored-by: Omar Padron <omar.padron@kitware.com>

Use threading.TIMEOUT_MAX when available (#24246)

This value was introduced in Python 3.2. Specifying a timeout greater than
this value will raise an OverflowError.

Co-authored-by: Lou Lawrence <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
fb0e91c534 Windows: Symlink support
To provide Windows-compatible functionality, spack code should use
llnl.util.symlink instead of os.symlink. On non-Windows platforms
and on Windows where supported, os.symlink will still be used.

Use junctions when symlinks aren't supported on Windows (#22583)

Support islink for junctions (#24182)

Windows: Update llnl/util/filesystem

* Use '/' as path separator on Windows.
* Recognizing that Windows paths start with '<Letter>:/' instead of '/'

Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Danny McClanahan
e8838109d8
move typing_extensions.py back into typing.py =\ (#28549) 2022-02-11 09:52:01 -08:00
Massimiliano Culpo
bc06c1206d
macholib, altgraph: update vendored dependency (#28664) 2022-01-28 10:55:12 -08:00
Danny McClanahan
0c2de252f1
introduce llnl.util.compat to remove sys.version_info checks (#21720)
- also split typing.py into typing_extensions and add py2 shims
2022-01-21 12:32:52 -08:00
Todd Gamblin
93377942d1 Update copyright year to 2022 2022-01-14 22:50:21 -08:00
Todd Gamblin
396c37d82f unparser: implement operator precedence algorithm for unparser
Backport operator precedence algorithm from here:
    397b96f6d7

This eliminates unnecessary parentheses from our unparsed output and makes Spack's unparser
consistent with the one in upstream Python 3.9+, with one exception.

Our parser normalizes argument order when `py_ver_consistent` is set, so that star arguments
in function calls come last.  We have to do this because Python 2's AST doesn't have information
about their actual order.

If we ever support only Python 3.9 and higher, we can easily switch over to `ast.unparse`, as
the unparsing is consistent except for this detail (modulo future changes to `ast.unparse`)
2022-01-12 06:14:18 -08:00
Todd Gamblin
afb358313a unparser: refactor delimiting with context managers in ast.unparse
Backport of 4b3b1226e8
2022-01-12 06:14:18 -08:00
Todd Gamblin
5847eb1e65 unparser: add block() context manager for indentation
This is a backport of a refactor from cpython 3.9
2022-01-12 06:14:18 -08:00
Todd Gamblin
2badd6500e unparse: Make unparsing consistent for 2.7 and 3.5-3.10
Previously, there were differences in the unparsed code for Python 2.7 and for 3.5-3.10.
This makes unparsed code the same across these Python versions by:

    1. Ensuring there are no spaces between unary operators and
       their operands.
    2. Ensuring that *args and **kwargs are always the last arguments,
       regardless of the python version.
    3. Always unparsing print as a function.
    4. Not putting an extra comma after Python 2 class definitions.

Without these changes, the same source can generate different code for different
Python versions, depending on subtle AST differences.

One place where single source will generate an inconsistent AST is with
multi-argument print statements, e.g.:

```
    print("foo", "bar", "baz")
```

In Python 2, this prints a tuple; in Python 3, it is the print function with
multiple arguments.  Use `from __future__ import print_function` to avoid
this inconsistency.
2022-01-12 06:14:18 -08:00
Todd Gamblin
b324fe5d95 externals: add astunparse
Add `astunparse` as `spack_astunparse`. This library unparses Python ASTs and we're
adding it under our own name so that we can make modifications to it.

Ultimately this will be used to make `package_hash` consistent across Python versions.
2022-01-12 06:14:18 -08:00
Todd Gamblin
7703043195 externals: Upgrade jsonschema to v3.2.0
Our `jsonschema` external won't support Python 3.10, so we need to upgrade it.
It currently generates this warning:

    lib/spack/external/jsonschema/compat.py:6: DeprecationWarning: Using or importing the ABCs
        from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and
        in 3.10 it will stop working

This upgrades `jsonschema` to 3.2.0, the latest version with support for Python 2.7.  The next
version after this (4.0.0) drops support for 2.7 and 3.6, so we'll have to wait to upgrade to it.

Dependencies have been added in prior commits.
2021-12-19 12:55:42 -08:00
Todd Gamblin
f83e0fb81a externals: add attrs for new jsonschema
Updating `jsonschema` to 3.2.0 requires `attrs`. Add it to externals.
2021-12-19 12:55:42 -08:00
Todd Gamblin
90592b3cbe externals: add pyrsistent for new jsonschema
Updating `jsonschema` to 3.2.0 requires `pyrsistent`. Adding just the pieces of it
that are needed for `jsonschema`.
2021-12-19 12:55:42 -08:00
Todd Gamblin
04536db387 externals: add functools32 for new jsonschema
Updating `jsonschema` to 3.2.0 requires `functools32`, just for Python 2.
2021-12-19 12:55:42 -08:00
Massimiliano Culpo
5c3dfacdc5
Update distro to v1.6.0 (#27263) 2021-11-24 10:10:11 +00:00
Massimiliano Culpo
70d5d234db
Update Jinja2 to v2.11.3 and MarkupSafe to v1.1.1 (#27264) 2021-11-24 10:21:35 +01:00
Massimiliano Culpo
12da0a9a69
Update six to v1.16.0 (#27265) 2021-11-24 10:20:04 +01:00
Massimiliano Culpo
fa7189b480
Remove support for Python 2.6 (#27256)
Modifications:
- [x] Removed `centos:6` unit test, adjusted vermin checks
- [x] Removed backport of `collections.OrderedDict`
- [x] Removed backport of `functools.total_ordering`
- [x] Removed Python 2.6 specific skip markers in unit tests
- [x] Fixed a few minor Python 2.6 related TODOs in code

Updating the vendored dependencies will be done in separate PRs
2021-11-23 09:06:17 -08:00
Massimiliano Culpo
f981682bdc
Allow recent pytest versions to be used with Spack (#25371)
Currently Spack vendors `pytest` at a version which is three major 
versions behind the latest (3.2.5 vs. 6.2.4). We do that since v3.2.5 
is the latest version supporting Python 2.6. Remaining so much 
behind the currently supported versions though might introduce 
some incompatibilities and is surely a technical debt.

This PR modifies Spack to:
- Use the vendored `pytest@3.2.5` only as a fallback solution, 
  if the Python interpreter used for Spack doesn't provide a newer one
- Be able to parse `pytest --collect-only` in all the different output 
  formats from v3.2.5 to v6.2.4 and use it consistently for `spack unit-test --list-*`
- Updating the unit tests in Github Actions to use a more recent `pytest` version
2021-11-18 15:08:59 +01:00
Harmen Stoppels
cc62689504
Fix overly generic exceptions in log parser (#27413)
This type of error is skipped:

make[1]: *** [Makefile:222: /tmp/user/spack-stage/.../spack-src/usr/lib/julia/libopenblas64_.so.so] Error 1

but it's useful to have it, especially when a package sets a variable
incorrectly in makefiles
2021-11-17 11:24:14 +01:00
Harmen Stoppels
336c60c618
Document backport in py (#26897) 2021-10-22 19:14:35 +02:00
Harmen Stoppels
d274769761
Backport #186 from py-py to fix macOS failures (#26653)
Backports the relevant bits of 0f77b6e66f
2021-10-22 13:52:46 +02:00
Harmen Stoppels
f8e4aa7d70
Revert "Don't run lsb_release on linux (#26707)" (#26754)
This reverts commit fcac95b065.
2021-10-15 09:34:04 +00:00
Harmen Stoppels
fcac95b065
Don't run lsb_release on linux (#26707)
Running `lsb_release` on Linux takes about 50ms because it is written in
Python. We do not use the output, so this change makes use not call it.
2021-10-14 01:27:24 +02:00
Harmen Stoppels
0c0831861c
Avoid quadratic complexity in log parser (#26568)
TL;DR: there are matching groups trying to match 1 or more occurrences of
something. We don't use the matching group. Therefore it's sufficient to test
for 1 occurrence. This reduce quadratic complexity to linear time.

---

When parsing logs of an mpich build, I'm getting a 4 minute (!!) wait
with 16 threads for regexes to run:

```
In [1]: %time p.parse("mpich.log")
Wall time: 4min 14s
```

That's really unacceptably slow... 

After some digging, it seems a few regexes tend to have `O(n^2)` scaling
where `n` is the string / log line length. I don't think they *necessarily*
should scale like that, but it seems that way. The common pattern is this

```
([^:]+): error
```

which matches `: error` literally, and then one or more non-colons before that. So
for a log line like this:

```
abcdefghijklmnopqrstuvwxyz: error etc etc
```

Any of these are potential group matches when using `search` in Python:

```
abcdefghijklmnopqrstuvwxyz
 bcdefghijklmnopqrstuvwxyz
  cdefghijklmnopqrstuvwxyz
                         ⋮
                        yz
                         z
```

but clearly the capture group should return the longest match.

My hypothesis is that Python has a very bad implementation of `search`
that somehow considers all of these, even though it can be implemented
in linear time by scanning for `: error` first, and then greedily expanding
the longest possible `[^:]+` match to the left. If Python indeed considers
all possible matches, then with `n` matches of length `1 .. n` you
see the `O(n^2)` slowness (i verified this by replacing + with {1,k}
and doubling `k`, it doubles the execution time indeed).

This PR fixes this by removing the `+`, so effectively changing the 
O(n^2) into a O(n) worst case.

The reason we are fine with dropping `+` is that we don't use the
capture group anywhere, so, we just ensure `:: error` is not a match
but `x: error` is.

After going from O(n^2) to O(n), the 15MB mpich build log is parsed
in `1.288s`, so about 200x faster.

Just to be sure I've also updated `^CMake Error.*:` to `^CMake Error`,
so that it does not match with all the possible `:`'s in the line.
Another option is to use `.*?` there to make it quit scanning as soon as
possible, but what line that starts with `CMake Error` that does not have
a colon is really a false positive...
2021-10-12 00:05:11 -07:00
Kevin Pedretti
47607dcac5
Use gnuconfig package for config file replacement for RISC-V. (#26364)
* Use gnuconfig package for config file replacement for RISC-V.

This extends the changes in #26035 to handle RISC-V. Before this change,
many packages fail to configure on riscv64 due to config.guess being too
old to know about RISC-V. This is seen out of the box when clingo fails
to build from source due to pkgconfig failing to configure, throwing
error: "configure: error: cannot guess build type; you must specify one".

* Add riscv64 architecture

* Update vendored archspec from upstream project.
These archspec updates include changes needed to support riscv64.

* Update archspec's __init__.py to reflect the commit hash of archspec being used.
2021-10-05 19:22:55 +00:00
Massimiliano Culpo
69abc4d860
Build ppc64le docker images (#26442)
* Update archspec
* Add ppc64le to docker images
2021-10-04 09:34:53 +02:00
bernhardkaindl
6ccda81368
log_parser.py: Find failed test case messages in error logs (#25694)
- Match failed autotest tests show the word "FAILED" near the end
- Match "FAIL: ", "FATAL: ", "failed ", "Failed test" of other suites
- autotest "   ok"$ means the test passed, independend of text before.
- autoconf messages showing missing tools are fatal later, show them.
2021-09-26 10:35:04 +02:00
Massimiliano Culpo
09378f56c0
Use a patched argparse only in Python 2.X (#25376)
Spack is internally using a patched version of `argparse` mainly to backport Python 3 functionality
into Python 2. This PR makes it such that for the supported Python 3 versions we use `argparse`
from the standard Python library. This PR has been extracted from #25371 where it was needed
to be able to use recent versions of `pytest`.

* Fixed formatting issues when using a pristine argparse.py
* Fix error message for Python 3.X when missing positional arguments
* Account for the change of API in Python 3.7
* Layout multi-valued args into columns in error messages
* Seamless transition in develop if argparse.pyc is in external
* Be more defensive in case we can't remove the file.
2021-08-17 08:52:51 -07:00
Massimiliano Culpo
fd55d627a7
archspec: added support for arm compiler on graviton2 (#24904) 2021-07-15 13:27:13 +00:00
Massimiliano Culpo
b12cee32de
Update archspec to support arm compiler on a64fx (#24524) 2021-06-26 09:18:48 +02:00
Massimiliano Culpo
4c4a584a9c spack: update archspec
This fixes the detection of Apple M1 and adds
virtual levels for x86_64 architectures
2021-05-20 14:56:04 -07:00
Massimiliano Culpo
8a65bcb7c9
archspec: updated external dependency (#23311)
Added support for Apple M1, extended support
for zen3 with more compiler flags.
2021-05-03 22:27:37 -07:00
Tamara Dahlgren
6f25e5242e
Docs: Updated copyrights in files still using 2020 as ending year (#23215) 2021-04-22 22:23:09 -07:00
Massimiliano Culpo
8cf6ad9917
archspec: update to latest version (#22357) 2021-03-18 00:23:09 -07:00