spack/lib/spack/docs
Massimiliano Culpo ab6499ce1e
parser: refactor with coarser token granularity (#34151)
## Motivation

Our parser grew to be quite complex, with a 2-state lexer and logic in the parser
that has up to 5 levels of nested conditionals. In the future, to turn compilers into
proper dependencies, we'll have to increase the complexity further as we foresee
the need to add:
1. Edge attributes
2. Spec nesting

to the spec syntax (see https://github.com/spack/seps/pull/5 for an initial discussion of
those changes).  The main attempt here is thus to _simplify the existing code_ before
we start extending it later. We try to do that by adopting a different token granularity,
and by using more complex regexes for tokenization. This allow us to a have a "flatter"
encoding for the parser. i.e., it has fewer nested conditionals and a near-trivial lexer.

There are places, namely in `VERSION`, where we have to use negative lookahead judiciously
to avoid ambiguity.  Specifically, this parse is ambiguous without `(?!\s*=)` in `VERSION_RANGE`
and an extra final `\b` in `VERSION`:

```
@ 1.2.3     :        develop  # This is a version range 1.2.3:develop
@ 1.2.3     :        develop=foo  # This is a version range 1.2.3: followed by a key-value pair
```

## Differences with the previous parser

~There are currently 2 known differences with the previous parser, which have been added on purpose:~

- ~No spaces allowed after a sigil (e.g. `foo @ 1.2.3` is invalid while `foo @1.2.3` is valid)~
- ~`/<hash> @1.2.3` can be parsed as a concrete spec followed by an anonymous spec (before was invalid)~

~We can recover the previous behavior on both ones but, especially for the second one, it seems the current behavior in the PR is more consistent.~

The parser is currently 100% backward compatible.

## Error handling

Being based on more complex regexes, we can possibly improve error
handling by adding regexes for common issues and hint users on that.
I'll leave that for a following PR, but there's a stub for this approach in the PR.

## Performance

To be sure we don't add any performance penalty with this new encoding, I measured:
```console
$ spack python -m timeit -s "import spack.spec" -c "spack.spec.Spec(<spec>)"
```
for different specs on my machine:

* **Spack:** 0.20.0.dev0 (c9db4e50ba045f5697816187accaf2451cb1aae7)
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-icelake
* **Concretizer:** clingo

results are:

| Spec          | develop       | this PR |
| ------------- | ------------- | ------- |
| `trilinos`  |  28.9 usec | 13.1 usec |
| `trilinos @1.2.10:1.4.20,2.0.1`  | 131 usec  | 120 usec |
| `trilinos %gcc`  | 44.9 usec  | 20.9 usec |
| `trilinos +foo`  | 44.1 usec  | 21.3 usec |
| `trilinos foo=bar`  | 59.5 usec  | 25.6 usec |
| `trilinos foo=bar ^ mpich foo=baz`  | 120 usec  | 82.1 usec |

so this new parser seems to be consistently faster than the previous one.

## Modifications

In this PR we just substituted the Spec parser, which means:
- [x] Deleted in `spec.py` the `SpecParser` and `SpecLexer` classes. deleted `spack/parse.py`
- [x] Added a new parser in `spack/parser.py`
- [x] Hooked the new parser in all the places the previous one was used
- [x] Adapted unit tests in `test/spec_syntax.py`


## Possible future improvements

Random thoughts while working on the PR:
- Currently we transform hashes and files into specs during parsing. I think
we might want to introduce an additional step and parse special objects like
a `FileSpec` etc. in-between parsing and concretization.
2022-12-07 14:56:53 -08:00
..
_gh_pages_redirect Use https for links (#19244) 2020-10-09 11:24:09 -05:00
_static make empty _static directory "exist" to git 2014-01-09 14:03:32 +01:00
build_systems Build System docs: consistent headers (#34047) 2022-11-23 13:35:55 +01:00
example_files Implement an optional compiler bootstrapping phase 2019-09-13 22:57:15 -07:00
images Allow for packages with multiple build-systems (#30738) 2022-10-26 20:17:32 +02:00
tables Getting Started: Python 2 is no longer supported (#33927) 2022-11-16 08:36:49 +01:00
.gitignore tutorial: move tutorial to standalone site (#13450) 2019-10-25 21:49:27 -07:00
analyze.rst Update copyright year to 2022 2022-01-14 22:50:21 -08:00
basic_usage.rst docs: fix typo (#33926) 2022-11-15 16:06:12 -08:00
binary_caches.rst Remove the warning that Spack prints at each spec (#30872) 2022-05-26 14:35:20 +00:00
bootstrapping.rst Deprecate spack bootstrap trust/untrust (#33600) 2022-10-29 12:24:26 -07:00
build_settings.rst reorder packages.yaml: requirements first, then preferences (#33741) 2022-11-07 16:16:11 -08:00
build_systems.rst Allow for packages with multiple build-systems (#30738) 2022-10-26 20:17:32 +02:00
chain.rst Update copyright year to 2022 2022-01-14 22:50:21 -08:00
command_index.in Rework command reference in docs, add spack commands command 2018-02-12 20:25:17 -08:00
conf.py Remove support for running with Python 2.7 (#33063) 2022-11-14 13:11:28 +01:00
config_yaml.rst Experimental binding of shared ELF libraries (#31948) 2022-11-03 17:34:00 -06:00
configuration.rst Docs: Minor change 'several'->'over a dozen' (#34274) 2022-12-02 10:27:37 -08:00
containers.rst containerize: fix concretization -> concretizer (#31594) 2022-07-26 08:56:24 -07:00
contribution_guide.rst remove activate/deactivate support in favor of environments (#29317) 2022-11-11 00:50:07 -08:00
developer_guide.rst parser: refactor with coarser token granularity (#34151) 2022-12-07 14:56:53 -08:00
environments.rst docs updates for spack env depfile (#33937) 2022-11-16 15:47:31 +01:00
extensions.rst Support config variables in config.yaml extensions paths (#17772) 2022-02-07 11:40:52 +01:00
features.rst Allow for packages with multiple build-systems (#30738) 2022-10-26 20:17:32 +02:00
getting_started.rst docs: update info on XCode requirements (#34097) 2022-11-24 00:20:09 +01:00
index.rst Remove known issues (#33738) 2022-11-07 19:41:16 +00:00
Makefile API Docs: fix broken reference targets 2021-07-16 08:30:56 -07:00
mirrors.rst docs: add (config.yaml) to sections for faster lookup by config file (#30157) 2022-04-20 15:00:43 +02:00
module_file_generation.svg SC17: reworked module file tutorial section (#5657) 2017-11-12 00:27:20 -08:00
module_file_support.rst Update module_file_support.rst (#32629) 2022-09-14 01:41:50 +00:00
monitoring.rst Update copyright year to 2022 2022-01-14 22:50:21 -08:00
package_list.rst Update copyright year to 2022 2022-01-14 22:50:21 -08:00
packaging_guide.rst docs: fix typo in multiple build systems (#33965) 2022-11-17 15:20:10 +01:00
pipelines.rst Docs: Update pipeline ci rebuild to add --tests (plus fixed typos) (#32048) 2022-09-21 14:23:58 +02:00
replace_conda_homebrew.rst Deprecate spack:concretization over concretizer:unify (#30038) 2022-05-23 13:20:34 -07:00
repositories.rst docs: add (config.yaml) to sections for faster lookup by config file (#30157) 2022-04-20 15:00:43 +02:00
requirements.txt Docs: Getting Started Dependencies (#32480) 2022-10-10 23:25:37 +00:00
spack.yaml Add missing upperbound to docs/spack.yaml (#33280) 2022-10-13 11:54:20 -07:00