In Spack v1.0 we plan to parse caret ^ and percent % the same. Their meaning is direct and transitive dependency respectively. It means that variants, versions, arch, platform, os, target and dag hash should go before the %, so that they apply to dependent not the %dependency.
When requiring a constraint on a virtual package, it makes little
sense to use anonymous specs, and our documentation shows no example
of requirements on virtual packages starting with `^`.
Right now, due to how `^` is implemented in the solver, writing:
```yaml
mpi:
require: "^openmpi"
```
is equivalent to the more correct form:
```yaml
mpi:
require: "openmpi"
```
but the situation will change when `%` will shift its meaning to be a
direct dependency.
To avoid later errors that are both unclear, and quite slow to get to the user,
this commit makes anonymous specs under virtual requirements an error,
and shows a clear error message pointing to the file and line where the
spec needs to be changed.
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
If you use `spack config change` to modify a `require:` section that did
not exist before, Spack was inserting the merged configuration into the
highest modification scope (which for example would clutter the
environment's `spack.yaml` with a bunch of configuration details
from the defaults).
Supersedes #46792.
Closes#40018.
Closes#31026.
Closes#2700.
There were a number of feature requests for os-specific config. This enables os-specific
config without adding a lot of special sub-scopes.
Support `include:` as an independent configuration schema, allowing users to include
configuration scopes from files or directories. Includes can be:
* conditional (similar to definitions in environments), and/or
* optional (i.e., the include will be skipped if it does not exist).
Includes can be paths or URLs (`ftp`, `https`, `http` or `file`). Paths can be absolute or
relative . Environments can include configuration files using the same schema. Remote includes
must be checked by `sha256`.
Includes can also be recursive, and this modifies the config system accordingly so that
we push included configuration scopes on the stack *before* their including scopes, and
we remove configuration scopes from the stack when their including scopes are removed.
For example, you could have an `include.yaml` file (e.g., under `$HOME/.spack`) to specify
global includes:
```
include:
- ./enable_debug.yaml
- path: https://github.com/spack/spack-configs/blob/main/NREL/configs/mac/config.yaml
sha256: 37f982915b03de18cc4e722c42c5267bf04e46b6a6d6e0ef3a67871fcb1d258b
```
Or an environment `spack.yaml`:
```
spack:
include:
- path: "/path/to/a/config-dir-or-file"
when: os == "ventura"
- ./path/relative/to/containing/file/that/is/required
- path: "/path/with/spack/variables/$os/$target"
optional: true
- path: https://raw.githubusercontent.com/spack/spack-configs/refs/heads/main/path/to/required/raw/config.yaml
sha256: 26e871804a92cd07bb3d611b31b4156ae93d35b6a6d6e0ef3a67871fcb1d258b
```
Updated TODO:
- [x] Get existing unit tests to pass with Todd's changes
- [x] Resolve new (or old) circular imports
- [x] Ensure remote includes (global) work
- [x] Ensure remote includes for environments work (note: caches remote
files under user cache root)
- [x] add sha256 field to include paths, validate, and require for remote includes
- [x] add sha256 remote file unit tests
- [x] revisit how diamond includes should work
- [x] support recursive includes
- [x] add recursive include unit tests
- [x] update docs and unit test to indicate ordering of recursive includes with
conflicting options is deferred to follow-on work
---------
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Co-authored-by: Peter Scheibel <scheibel1@llnl.gov>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Defines `spack.package_api_version` and `spack.min_package_api_version`
as tuples (major, minor).
This defines resp. the current Package API version implemented by this version
of Spack and the minimal Package API version it is backwards compatible with.
Repositories can optionally define:
```yaml
repo:
namespace: my_repo
api: v1.2
```
which indicates they are compatible with versions of Spack that implement
Package API `>= 1.2` and `< 2.0`. When the `api` key is omitted, the default
`v1.0` is assumed.
* Adding ability for repo paths from a manifest file to be expanded when creating an environment.
A unit test was added to check that an environment variable will be expanded.
Also, a bug was fixed in the expansion of develop paths where if an environment variable
was in the path that then produced an absolute path the path would not be extended.
* Fixing new unit test for env repo var substitution
* Adding ability for repo paths from a manifest file to be expanded when creating an environment.
A unit test was added to check that an environment variable will be expanded.
Also, a bug was fixed in the expansion of develop paths where if an environment variable
was in the path that then produced an absolute path the path would not be extended.
* Messed up resolving last rebase
On Windows, libraries search their directory for dependencies, and
we help libraries in Spack-built packages locate their dependencies
by symlinking them into the dependent's directory (we refer to this
as simulated RPATHing).
We extend the convenience functionality here to support base library
directories outside of the package prefix: this is primarily for
running tests in the build directory (which is not located inside
of the final install prefix chosen by spack).
Python was removed from being a build tool in #46980, due to issues
when reusing specs. This PR adds a new rule to match the interpreter
among different Python packages, in clingo.
It also adds a bunch of new "build-tools", so that specs like:
```
py-matplotlib backend=tkagg
```
can be concretized in one go.
Modifications:
- [x] Make `py-matplotlib backend=tkagg` concretizable
- [x] Add unit-tests to ensure situations like in #46980 do not happen
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Currently, the custom config scopes are pushed at the top when constructing
configuration, and are demoted whenever a context manager activating an
environment is used - see #48414 for details. Workflows that rely on the order
in the [docs](https://spack.readthedocs.io/en/latest/configuration.html#custom-scopes)
are thus fragile, and may break
This PR allows to assign priorities to scopes, and ensures that scopes of lower priorities
are always "below" scopes of higher priorities. When scopes have the same priority,
what matters is the insertion order.
Modifications:
- [x] Add a mapping that iterates over keys according to priorities set when
adding the key/value pair
- [x] Use that mapping to allow assigning priorities to configuration scopes
- [x] Assign different priorities for different kind of scopes, to fix a bug, and
add a regression test
- [x] Simplify `Configuration` constructor
- [x] Remove `Configuration.pop_scope`
---------
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
All the build jobs in pipelines are apparently relying on the bug that was fixed.
The issue was not caught in the PR because generation jobs were fine, and
there was nothing to rebuild.
Reverting to fix pipelines in a new PR.
This reverts commit 3ad99d75f9.
VariantMap.concrete is unused, and would be incorrect if it were used
due to conditional variants.
Just let the Spec dictate what is concrete and what is not.
Currently, environments can end up with higher priority than `-C` custom
config scopes and `-c` command line arguments sometimes. This shouldn't
happen -- those explicit CLI scopes should override active environments.
Up to now configuration behaved like a stack, where scopes could be only be
pushed at the top. This PR allows to assign priorities to scopes, and ensures
that scopes of lower priorities are always "below" scopes of higher priorities.
When scopes have the same priority, what matters is the insertion order.
Modifications:
- [x] Add a mapping that iterates over keys according to priorities set when
adding the key/value pair
- [x] Use that mapping to allow assigning priorities to configuration scopes
- [x] Assign different priorities for different kind of scopes, to fix a bug, and
add a regression test
- [x] Simplify `Configuration` constructor
- [x] Remove `Configuration.pop_scope`
- [x] Remove `unify:false` from custom `-C` scope in pipelines
On the last modification: on `develop`, pipelines are relying on the environment
being able to override `-C` scopes, which is a bug. After this fix, we need to be
explicit about the unification strategy in each stack, and remove the blanket
`unify:false` from the highest priority scope
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Currently, we have `config:shared_linking:missing_library_policy` to error
or warn when shared libraries cannot be resolved upon install.
The new `spack verify libraries` command allows users to run this post
install hook at any point in time to check whether their current
installations can resolve shared libs in rpaths.
Installing `rust@nightly` fails because the package file declares a conflict of rust versions older than `:1.64` with `gcc>=13`. However, because `nightly` is alphanumerically smaller than any actual version number, `nightly` is incorrectly detected to have a conflict with `gcc>=13` as well. Marking `nightly` as an infinity version instead solves this.
* Reproducer should decude artifact root from concrete environment
* Add documentation on the layout of the artifacts directory
* Use dag hash in the container name
* Add reproducer options to improve local testing
* --use-local-head allows running reproducer with
the current Spack HEAD commit rather than computing
a commit for the reproducer
* Add test to verify commits and recreating reproduction environment
* Add test for non-merge commit case
* ci reproduce-build: Drop overwrite option
in favor of throwing an error if the working dir is non-empty
A directory and a symlink to it under the same relative path in a
different prefix
```
/prefix1/dir/
/prefix1/dir/file
/prefix2/dir -> /prefix1/dir/
```
are not a blocker to create a view. The view structure simply looks like
this:
```
/view/dir/
/view/dir/file
```
This should be the case independently of the order in which we visit
prefixes, so we could in principle create views order independently.
Up to now, Spack was allowing all build-tools that
may appear in the DAG to have 2 max_dupes.
This is not needed in practice for most of them,
and adding them out of caution just increases
grounding and concretization time.
This PR makes the value of max_dupes configurable
per package, and sets only a few known packages to
2 max_dupes by default.
In case user needs different values, they can
tune the configuration for their use case.
On macOS, prefix_a/file and prefix_b/FILE map to the same file view/file or view/FILE.
This commit ensures that we test whether a view is created on a case insensitive filesystem and handle projection conflicts accordingly.
With this change spec["pkg"] searches only direct dependencies and transitive link/run
dependencies, ordered by depth. This avoids situations where we pick up unwanted
deps of build/test deps.
To reach those, you need to do spec["build_dep"]["pkg"] explicitly.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
meson's `dependency` function often uses pkg-config to locate a dependency, and may even fall back to cmake. The former case is very common, and since packagers often forget to add the tiny pkgconfig package as a build dep, we do it for them.
This PR is effectively a breaking change extracted from #45189, which removes
support for spec["mpi"] if spec itself is openmpi / mpich that could provide mpi;
from the Spec instance we don't have any parent it provides it to,
hence it's a KeyError.
Currently, when we setup the ASP problem for `clingo`, we don't take into account the configuration. This results in setting up ASP problems that are larger than necessary, with possibly redundant information, and higher concretization times.
This PR tries to improve things by adding an opt-in feature that computes the _possible dependencies_ of a solve taking also into account the current configuration, and avoids adding possible dependencies that we are certain can't be in the final solution.
The feature can be activated with:
```yaml
concretizer:
static_analysis: true
```
Examples of simple rules to discard dependencies are:
- Dependencies that are not buildable, and for which no binary is present (e.g. `cray-mpich` etc. on non Cray systems)
- Dependencies that are not for the current platform (e.g. `msmpi` on non Windows platforms)
- Conditional dependencies that cannot be activated, because of some user requirement (e.g. `cuda` etc. if the user requires `~cuda` in configuration)
- Virtual providers that cannot be used, because of a requirement on a virtual
The speed-up these rules seem to give depends on the use case at hand, but if the configuration is updated properly, they are noticeable.
Since in cases where there is no rule to exclude packages upfront, reuse is active, and this option is activated, it's possible to see some minor slow down, the feature has been added as opt-in, so it's turned off by default.
`relocate_links` warns when the target is absolute and not matched by
any prefix from the prefix to prefix map.
This can lead to false positives, cause the prefix to prefix map does
not contain trivial/identity entries whenever a package is installed to
its original location.
Since relocate_links is the odd one out there (we don't warn about
similar issues with rpaths, etc), just remove the warning.
* Remove variable from cmake.py
#48775 left a dangling variable that was not caught in CI but by the eyes of @haampie. Restructure variable to local method.
* [@spackbot] updating style on behalf of psakievich
* Update cmake.py
* Update lib/spack/spack/build_systems/cmake.py
* Update lib/spack/spack/build_systems/cmake.py
---------
Co-authored-by: psakievich <psakievich@users.noreply.github.com>
Add ruff configuration to `pyproject.toml`.
This allows `ruff format` in the Spack repository to format all the files we care about,
with our line length of 99, the exceptions we already put in place, and excluding things
we don't auto-format, like vendored dependencies.
Right now it'll reformat 175 or so files, but only slightly, in places where `ruff` differs from
`black`. For the most part I like the ruff format decisions better than `black`, but none of
the changes seem too severe.
This does not change `spack style` -- I figure that can come later but this at least will
let people start playing with `ruff`.
---------
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
These are some changes that `ruff check --fix` would make that the current
`spack style` also agrees with. Make the changes now so that the `ruff`
change is less disruptive.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Currently, environments created from manifest files with relative includes result in broken
references to config files.
This PR modifies `spack env create` to create local copies in the new environment of any local
config files from relative paths in the environment manifest passed as an init file.
This PR does not change the behavior if the include is an absolute path or if the include is from
a relative path outside the environment directory, but it does warn about missing relative includes if
they are inside the environment directory.
Includes regression test and short blurb in docs.
* package api: drop wildcard re-export
To ensure package repos are forward/backward compatibility with Spack,
we should explicitly export all symbols we want to expose in the public
package API, and drop `from spack.something import *` because
removal/addition to the public API will go unnoticed.
Also `llnl.util.filesystem` has some methods that shouldn't be exposed
in the package API, so better to enumerate a subset explicitly.
* remove flatten_dependencies / install_dependency_symlinks
Regressed in #47126
Spack was not interpreting mirrors using relative path with respect to the
metadata directory.
---------
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* test_no_matching_compiler_specs: does not need mock_low_high_config,
since mutable_config is already used at class level
* bindist.py: setup a configuration that doesn't super-impose builtin.mock
over builtin
* builder.py: use a mock configuration for the tests
This adds a new configuration section called `env_vars:` that can be set in an environment.
It looks very similar to the existing `environment:` section that can be added to `modules.yaml`,
but it is global for an entire spack environment. It's called `env_vars:` to deconflate it with spack
environments (the term was too overloaded).
The syntax looks like this:
```yaml
spack:
specs:
- cmake%gcc
env_vars:
set:
ENVAR_SET_IN_ENV_LOAD: "True"
```
Any of our standard environment modifications can be added to the `env_vars` section, e.g.
`prepend_path:`, `unset:`, `append_path:`, etc. Operations in `env_vars:` are performed
on `spack env activate` and undone on `spack env deactivate`.
* Reduce the size of outputted go built binaries
* Remove unused import from go package
* go: remove comment from setup dependents build env
* Add back missing imports after rebase
* Backward compat with Python 3.9 for socket.timeout
* Forward compat with Python [unknown] as HTTPResponse.geturl is deprecated
* Catch timeout etc from .read()
* Some minor simplifications: json.load(...) takes file object in binary mode.
* Fix CDash code which does error handling wrong: non-2XX responses raise.
Should be sufficient to set CONFIG_SITE to /dev/null before running configure to
prevent any config site file to be loaded, which currently causes non-determinism in builds.
Setting the variable CONFIG_SITE prevents configure files to check $prefix/share/config.site,
$prefix/etc/config.site, $ac_default_prefix/share/config.site, $ac_default_prefix/etc/config.site
so we don't have to patch a block of code.
* Address quoting issue that casuses dev_paths containing @ symbols to parse as versions
* Fix additional location were dev_path was imporperly constructed
The dev_path must be quoted to avoid parsing issues when
paths contain '@' symbols. Also add tests to catch
regression of this behavior
* Format to fix line length
* fix failing tests
* Fix whitespace error
* Add binary_compatibility fixture to test
* Change string formatting to avoid multiline f string
* Update lib/spack/spack/test/concretization/core.py
* Add tmate debug session
* Revert "Add tmate debug session"
This reverts commit 24e2f77e3c.
* Move test and refactor to use env methods
* Update lib/spack/spack/test/cmd/develop.py
---------
Co-authored-by: psakievich <psakiev@sandia.gov>
Since macOS 15 `ld -single_module` warns with a deprecation message,
which makes configure scripts believe the flag is unsupported. That
in turn triggers a code path where `archive_cmds` is set to
```
$CC -r -keep_private_externs -nostdlib ... -dynamiclib
```
instead of just
```
$CC -dynamiclib ...
```
This code path was meant to trigger only on ancient macOS <= 14.4 where
libtool had to add `-single_module`, which is the default since macos
14.4, and is now apparently deprecated because the flag is a no-op for
more than 15 years.
The wrong `archive_cmds` causes actual problems combined with a bug in
OpenMPI's compiler wrapper (`CC=mpicc`), which appends `-rpath` flags,
which cause an error when combined with the `-r` flag added by the
autotools.
Spack's compiler wrapper doesn't do this, but it's likely there are
other compiler wrappers out there that are not aware that `-r` and
`-rpath` cannot be combined.
The fix is to change defaults: `lt_cv_apple_cc_single_mod=yes`.
* for config: let `syaml_dict` inherit from `dict` instead of `OrderedDict`. `syaml_dict` now only exists as a mutable wrapper for yaml related metadata.
* for spec serialization / hashing: use `dict` directly
This is possible since we only support cpython 3.6+ in which dicts are ordered.
This improves performance of hash computation a bit. For a larger spec I'm getting 9.22ms instead of 11.9ms, so 22.5% reduction in runtime.
* OSError.errno apparently is int | None, but already part of __str__ anyway so drop
* fix disambiguate_spec_from_hashes
* List[Spec].sort() complains, ignore it
* fix typing of KnownCompiler, and use it in asp.py
Modifications:
* Fix a severe bug where a spliced spec has the same hash as the original build spec
* Removed CacheManager in favor of install_mockery. The latter sets up a temporary store, and removes it at the end
of the test.
* Deleted a couple of helper functions e.g. _has_dependencies
* Checked that SolverException is raised, rather than Exception (more specific)
* Extended, and renamed the splicing_setup fixture (now called install_specs)
* Added more specific assertions in each test
One test is currently flaky, due to some instability when sorting multiple specs. It's currently marked xfail
The methods spack.spec.Spec.concretize and spack.spec.Spec.concretized
are deprecated in favor of spack.concretize.concretize_one.
This will resolve a circular dependency between the spack.spec and
spack.concretize in the next Spack release.
* Improve support using external modules with zsh
Spack integrates with external modules by launching a python subprocess
to scrape the path from the module command. In zsh, subshells do not
inheret functions defined in the parent shell (only environment
variables). This breaks the module function in module_cmd.py.
As a workaround, source the module commands using $MODULESHOME prior to
running the module command.
* Fix formatting
* Fix flake error
* Add guard around sourcing module file
* Add improved unit testing to module src command
* Correct style
* Another attempt at style
* formatting again
* formatting again
---------
Co-authored-by: psakievich <psakiev@sandia.gov>
`spack find` and other commands that read the DB will fail if the databse is in a
read-only location, because `FailureTracker` and `Database` can try to create their
parent directories when they are constructed, even if the DB root is read-only.
Instead, we should only write to the filesystem when needed.
- [x] don't create parent directories in constructors
- [x] have write operations create parents if needed
- [x] in tests, `chmod` databases to be read-only unless absolutely needed.
---------
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
This PR allows using the subscript notation directly on packages. The
intent is to reduce the boilerplate needed to retrieve package
properties from nodes other than root.
* Add type-hints to `spack.util.executable.Executable`
* Add type-hint to input
* Use overload, and remove assertions at calling sites
* Bump mypy to v1.11.2 (working locally), Python to 3.13
* py-py-spy: add 0.4.0
* py-py-spy: port from Package to CargoPackage
* CargoPackage: respect make_jobs
* rust: respect make_jobs during build and install
* spack style
* CargoBuilder: fix make_jobs syntax
* CargoBuilder: don't write to $HOME, use stage dir
* mpich: gather in a single place env modifications needed by mpich derivatives
MPICH, and its derivatives, share a lot of copy/paste code to setup the
environment during the different stages of the package life-cycle.
This commit gathers the common modifications in a single place (a mixin class),
living in the Mpich package, and makes derivatives import, and reuse, it.
* Fix docs for Python < 3.13
A few changes to tarball creation (for build caches):
- do not run file to distinguish binary from text
- file is slow, even when running it in a batched fashion -- it usually reads all bytes and has slow logic to categorize specific types
- we don't need a highly detailed file categorization; a crude categorization of elf, mach-o, text suffices.
detecting elf and mach-o is straightforward and cheap
- detecting utf-8 (and with that ascii) is highly accurate: false positive rate decays exponentially as file size increases. Further it's not only the most common encoding, but the most common file type in package prefixes.
iso-8859-1 is cheaply (but heuristically) detected too, and sufficiently accurate after binaries and utf-8 files are classified earlier
- remove file as a dependency of Spack in general, which makes Spack itself easier to install
- detect file type and need to relocate as part of creating the tarball, which is more cache friendly and thus faster
* Fix silent error when reporting builds to CDash
CDash has a 191 char maximum for build names. When this
is exceeded, CDash silently fails to correctly process the
reported XML. This truncates CDash build names to 190 chars
and emits a warning indicating it is doing so to prevent
such errors from occuring.
* test/reporters.py: add unittest for buildname len issue
* test/reporters.py: rename cdash buildname test
* ci/common.py: fix syntax causing breaking test
It appears that the CDash reporter is expecting a string
as the buildname.
* Update lib/spack/spack/reporters/cdash.py
Fix warning message to reflect actual issue.
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
* ci/common.py: fix function call to actually call function
---------
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: psakievich <psakiev@sandia.gov>
Improve our typing by updating some todo locations in the code to use
`Literal` instead of a simple `str`.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
* Set the "build_jobs" on concretization/generate for CI
build_jobs also controls the concretization pool size. Set this
in the config section for CI generate.
This config is overwritten by build_job CI using the SPACK_BUILD_JOBS
environment variable. This implicitly will drop the default build
CPU request on all "default" grouped build jobs from (max) 16 to 8.
* Add default allocations for build jobs
* Add common jobs and concretize args to ci generate and rebuild
* CI: Specify parallel concretize and build jobs via argument
* Increase power and cray concretization limits
Lowering limits for these stacks creates timeout
* Increase default pool size to 8
intermittent timeouts with 4 CPU
* Add reduced requests for windows for now
This turns some variant-specific methods for dealing with when-keyed dictionaries into
more generic versions, in preparation for conditional version definitions.
`_by_name`, `_names`, etc. are replaced with generic methods for transforming
when-keyed dictionaries:
* `_by_subkey()`
* `_subkeys()`
* `_num_definitions()`
* `_definitions()`
* `_remove_overridden_defs()`
And the variant accessors are refactored to use these methods underneath.
To do this, types like `WhenDict` had to be generified, and some `TypeVars`
were added for sortable keys and values.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
We are using more and more typing features in Spack, and without features like
protocols, typing core is becoming harder and harder.
I think it's worth vendoring `typing_extensions` for this. It will get us a number of
useful capabilities:
* `Literal`
* `TypedDict`
* `Protocol`
among others.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
This commit adds a config option `config:shared_linking:missing_library_policy:error/warn/ignore` which will cause installation errors or warnings when ELF executables or libraries need shared libraries which cannot be resolved from RPATH search paths. The default is to ignore.
This is a safeguard against accidentally linking to system libraries instead of Spack libraries. It makes it more likely that build cache installs work on different machines. It works only at the level of libraries, not at the level of symbols. Some system dependencies are allowed (e.g. kernel and libc).
Packages can (but are discouraged to) set `unresolved_libraries` to a list of patterns of sonames/library names that are know to be unresolvable in RPATHs. In the future this could be made more fine-grained in a non-breaking way by allowing a dictionary of patterns `lib => [deps]`.
Extracted #45189
Common test setup has been extracted in fixtures. Some matrix
dimensions moved from being "compiler" to be "targets".
Use --fake install for packages in test.
The `_normal` attribute on specs is no longer used and has no meaning.
It's left over from part of the original concretizer.
The `concrete` constructor argument is also not used by any part of core.
- [x] remove `_normal` attribute from `Spec`
- [x] remove `concrete` argument from `Spec.__init__`
- [x] remove unused `check_diamond_normalized_dag` function in tests
- [x] simplify `Spec` constructor and docstrings
I tried to add typing to `Spec` here, but it creates a huge number of type issues
because *most* things on `Spec` are optional. We probably need separate `Spec` and
`ConcreteSpec` classes before attempting that.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Add missing encoding=utf-8 to various open calls. This makes
files like spec.json, spack.yaml, spack.lock, config.yaml etc locale
independent w.r.t. text encoding. In practice this is not often an
issue since Python 3.7, where the C locale is promoted to
C.UTF-8. But it's better to enforce UTF-8 explicitly, since there is
no guarantee text files are written in the right encoding.
Also avoid opening in text mode if it can be avoided.
* `f.tell` on a `TextIOWrapper` does not return the offset in bytes, but
an opaque integer that can only be used for `f.seek` on the same
object. Spack assumes it's a byte offset.
* Do not open in a locale dependent way, but assume utf-8 (and allow
users to override that)
* Use tempfile to generate a backup/temporary file in a safe way
* Comparison between None and str is valid and on purpose.
Follow-up to #47956
* Rename `token.py` -> `tokenize.py`
* Rename `parser.py` -> `spec_parser.py`
* Move common code related to iterating over tokens into `tokenize.py`
* Add "unexpected character token" (i.e. `.`) to `SpecTokens` by default instead of having a separate tokenizer / regex.
The use of `^` in `depends_on` directives has never been allowed, since
the dawn of Spack.
Up to now, we used to have an audit to catch this kind of issue, mainly
because in that way we could easily collect all issues and report them
to packagers at once.
Due to implementation details, this audit doesn't work if a dependency
without a `^` is followed by the same dependency with a `^`.
This PR makes this pattern an error, which will be reported eagerly, and
removes the corresponding audit. It also fixes a package using the wrong
idiom.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Reorganize the pipeline generation aspect of the ci module,
mostly to separate the representation, generation, and
pruning of pipeline graphs from platform-specific output
formatting.
Introduce a pipeline generation registry to support generating
pipelines for other platforms, though gitlab is still the only
supported format currently.
Fix a long-existing bug in pipeline pruning where only direct
dependencies were added to any nodes dependency list.
