This PR simplifies the code doing external spec detection by removing
the `DetectedPackage` class. Now, functions accepting or returning lists
of `DetectedPackage`, will accept or return list of specs.
Performance doesn't seem to change if we use `Spec.__reduce__` instead
of `DetectionPackage.__reduce__`.
Change the signature of Spec.from_detection to set the
external prefix, and the external modules, if they are
present.
Delete "spack.package_prefs.spec_externals" since it
is unused.
This context manager was used to prevent detecting a platform
as "cray". Since now Cray machines are detected as linux, we can
remove the context manager.
Right now if you run `spack ci generate` you get a warning about CDash credentials even
if there's no CDash configuration specified. We should only warn if there was actually a
CDash config.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Source mirrors store entries by digest and add a human readable alias of the
form 'name-version'. If no digest is available, the alias is used as the primary
storage location.
Spack erroneously fetches by alias when the digest path does not exist. This is
problematic if `version(..., sha256=...)` changes in package.py, and the mirror
is populated with the old shasum. That would result in an error when a digest
is available, but in case of git versions with a modified commit sha, the wrong
sources would be fetched without error. With this PR, only the digest path is
used, not the alias, in case a digest is available. This is also a small performance
optimization, as the number of request is halved for mirrors that don't contain
the sources.
Further, for git sources the tag was used as a digest, but this is a moving
target. Only commit sha is used now.
Also whenever the alias already existed, Spack used to keep it in place when
updating the mirror cache, which means that aliases would always point to
outdated mirror entries whenever digests are modified. With this PR the alias
is moved in place.
Lastly, fix a recent regression where `Stage.disable_mirrors` disabled mirrors
but not the local download cache, which was the intention.
* Add missing MultiMethodMeta metaclass in builders
and remove the Python 2 fallback option in favor of hard errors to catch
similar issues going forward.
The fallback option can cause about 10K stat calls due to use of
`realpath` in the inspect module, depending on how deep Spack itself is
nested in the file system, which is ... undesirable.
* code shuffling to avoid circular import
* more reshuffling
* move reserved variant names into variants module
* Adds an MSBuild system + Builder to the icu4c package
* Adds custom install method as MSBuild system does not vendor an
install target
* The cxxstd variant is not supported on Windows (there are no config
options you use to tell the build system what cxx standard to
build against), so the variant definition was updated to occur
everywhere except Windows
Also, this commit defines the c/cxx..._flag properties of the MSVC
compiler (although they are not used by `icu4c` and not strictly
necessary to bundle with this PR).
Paths over 260 characters in length are not handled by `shutil.rmtree`
unless they use the extended-length path syntax (using a prefix of
"\\?\").
This fixes an issue where stage cleaning fails when paths in a stage
exceed the normal 260-character limit.
This indicates that other parts of the codebase should be examined/
refactored to handle long paths.
Previously `spack buildcache push --only=package` errored in the OCI
case, but it's been requested that OCI can be used as pure storage w/o
the need for runnable container images.
This commit makes it so that
1. manifests refer only to runtime dependencies that were selected to be
pushed
2. failure to upload a blob among the selected specs does not prevent a
manifest/tag to be created for dependents: they just don't refer to
the missing blob as a layer/dependency
This fixes the following issues:
1. dependents of non-redistributable specs can now be pushed to oci
build caches without error
2. failure to upload one tarball does not cause cascading failures for
dependents whose tarballs do upload succesfully -- so it's better
best-effort behavior
3. for some people uploading with deps caused a massive amount of
fetches of their manifests (which certain registries count as a
download of an image, even though their layers are not fetched) --
being able to specify --only=package reduces the number of fetches.
Stage objects create mirrors ad-hoc from current config.
- There is no way to prevent mirrors from being used
- There is no way to restrict mirrors to source/binary, which is of
course context dependent.
- Stage is also used in build caches, where iterating over mirrors is
already implemented differently, and wouldn't work anyways cause it's
source only, and in particular it makes no sense for OCI build caches.
This commit:
1. Injects the sensible mirrors into the stage object from contexts
where it is relevant
2. Separates mirrors from cache, so that w/o mirrors download cache can
still be used
The test_foreground_background unit test has been marked
xfail for a while, meaning:
- Nobody looks at the results of the test
- It still runs every time
That test happens to hang frequently on some Apple M1 I have access to,
so here I mark it as skip.
Also went through other xfailing and skipped tests, and applied minor changes.
* Use absolute paths instead of https:// fake mirrors (this speed-up tests by avoiding requests)
* Add a fixture to gather in a single place code that is copy/pasted in a lot of tests
* General clean-up of tests and repeated code
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
* Add options for sparse checkout in GitFetcher
Newer versions of git have a beta feature called sparse checkout
that allow users to check out a portion of a large repo.
This feature will be ideal for monolithic repo projects that want to
model their infrastructure via spack. This PR implements an addition
to the GitFetcher that allows users to add a `git_sparse_paths`
attribute to package classes or versions which will then use sparse
checkout on those directories/files for the package.
* Style
* Split git clone into multiple functions
* Add sparse-checkout impl
* Internalize src clone functions
* Docs
* Adding sparse clone test
* Add test for partial clone
* [@spackbot] updating style on behalf of psakievich
* Small fixes
* Restore default branch status
* Fix attributes for package
* Update lib/spack/docs/packaging_guide.rst
Co-authored-by: Matthew Mosby <44072882+mdmosby@users.noreply.github.com>
* Extend unit test to multiple git versions
* style
---------
Co-authored-by: psakievich <psakievich@users.noreply.github.com>
Co-authored-by: Matthew Mosby <44072882+mdmosby@users.noreply.github.com>
When sending Repo and RepoPath over to a child process,
we go through a marshaling procedure with pickle. The
default behavior for these classes is highly inefficient,
as it serializes a lot of specs that can just be
reconstructed on the other end of the pipe.
Here we write optimized procedures to __reduce__ both
classes.
Extracted from #45638
When adding the "detectable" tag to a package class that has the
"tag" attribute inherited from a base class, we need to copy it to
avoid modifying the base class.
"spack buildcache push" for partially installed environments pushes all it
can by default, and only dumps errors towards the end.
If --fail-fast is provided, error out before pushing anything if any
of the packages is uninstalled
oci build caches using parallel push now use futures to ensure pushing
goes in best-effort style.
The old concretizer is still used to bootstrap clingo from source. If we switch to a DAG model
where compilers are treated as nodes, we need to either:
1. fix the old concretizer to support this (which is a lot of work and possibly research), or
2. bootstrap `clingo` without the old concretizer.
This PR takes the second approach and gets rid of the old concretizer code. To bootstrap
`clingo`, we store some concrete spec prototypes as JSON, select one according to the
coarse-grained system architecture, and tweak them according to the current host.
The old concretizer and related dead code are removed. In particular, this removes
`Spec.normalize()` and related methods, which were used in many unit-tests to set
up the test context. The tests have been updated not to use `normalize()`.
- [x] Bootstrap clingo concretization based on a JSON file
- [x] Bootstrap clingo *before* patchelf
- [x] Remove any use of the old concretizer, including:
* Remove only_clingo and only_original fixtures
* Remove _old_concretize and _new_concretize
* Remove _concretize_together_old
* Remove _concretize_together_new
* Remove any use of `SPACK_TEST_SOLVER`
* Simplify CI jobs
- [x] ensure bootstrapping `clingo` works on on Darwin and Windows
- [x] Raise an intelligible error when a compiler is missing
- [x] Ensure bootstrapping works on FreeBSD
- [x] remove normalize and related methods
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
`setup-env.sh` is meant to be sourced, not executed directly.
By revoking execution permissions, users who accidentally execute
the script will receive an error instead of seeing no effect.
* Remove execution permission from `setup-env.sh` and friends
* Don't make output file executable in `spack commands --update-completion`
---------
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Fixes#43494
Add a set of environment variables SPACK_ALWAYS_CFLAGS (etc.) that
are always applied by the compiler wrapper.
Unlike SPACK_CFLAGS, for example, these will also be applied to
version checks (both SPACK_CFLAGS and SPACK_ALWAYS_CFLAGS will be
applied to the other invocation modes like ccld etc.).
Using this new functionality, the classic Intel and oneAPI compilers
are updated to pass compiler flags that disable warning messages
when newer versions are invoked via their older binary names
(these warnings were also generated for version checks, hence the
need for a new wrapper variable).
---------
Co-authored-by: Peter Josef Scheibel <scheibel1@llnl.gov>
For example: spack install py-pyqt5 or py-pyqt6 would install pylupdate5 pyrcc5 and
pyuic5 into in python-venv's install prefix.
Fix https://github.com/spack/spack/issues/45359
* Allow spec queries by `namespace`
Spack specs have "namespaces" that indicate what package repository they come from, but
there has not been a way to use the spec syntax to match one.
You can say things like this:
```console
spack find builtin.zlib
spack find myrepo.zlib
```
But, because namespaces are written as a dot-separated prefix on the name, you can't say
"find me all specs in namespace myrepo". The syntax doesn't allow it.
This PR allows you to specify anonymous specs with namespaces on the CLI. Specifically
you can do queries like this:
```console
spack find namespace=builtin
spack find namespace=myrepo
```
You can use this anywhere else you use spec syntax, e.g. in a config file to separate
installations based on what repo they came from:
```yaml
spack:
config:
install_tree:
root: $spack/opt/spack
projections:
namespace=myrepo: "myrepo_special_path/{name}-{hash}"
namespace=builtin: "builtin/{name}-{hash}"
```
This PR adds a special `namespace_if_anonymous` attribute to specs, which returns the
`namespace` if the spec has no name, otherwise it returns `None`. This allows us to
print the namespace for anonymous specs but to continue hiding it for most views, as
we've done so far.
This is implemented as a special case, but it's one that already exists, along with
`platform`, `os`, `target`, etc. This also reserves existing special case names for
variants so that users cannot define them in their package files. This change is
potentially breaking, but I do not think it will be common. There are no builtin
packages with a variant called `namespace`, and defining `os`, `target`, or `platform`
as a variant would've likely caused other problems if they were already being used.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
* format: allow spaces in format specifiers
Key-value pair format specifiers can now contain spaces in the key. This allows us to
add spaces to format strings that are *only* present when the attribute formatted is not
``None``. Instead of writing:
```
{arch=architecture}
```
and special casing `arch=` like a sigil in `Spec.format()`, we can now write:
```
{ arch=architecture}
```
And the space is *only* printed when `architecture` is not `None`. This allows us to
remove the special case in `Spec.format()` for `arch=`.
Previously the only `key=` prefix allowed in format specifiers was `arch=`, but this PR
removes that requirement, and the `key=` part of a key-value specifier can be any name.
It does *not* have to correspond to the formatted attribute.
- [x] modify `SPEC_FORMAT_RE` to allow arbitrary keys in key-value pairs.
- [x] remove special case for `arch=` from `Spec.format()`.
- [x] modify format strings using `{arch=architecture}` to use `{ arch=architecture}`
- [x] add more tests for formatting
This PR saves other more complex attributes like compiler flags and their spacing for later.
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
VC toolset versions 144 and 143 are both associated with the platform
toolset 143; this deviates from prior version choices made by the
MSVC devs; add a special case to report platform toolset version
as 143 when VC toolset version is >= 143 (this will need revisiting
for later releases).
This was missed in #45394 because we don't run unit tests for package PRs, and
`test_all_virtual_packages_have_default_providers`, which would've caught it, is a unit
test, not an audit.
- [x] add a default provider for `wasi-sdk` in `etc/spack/defaults/packages.yaml` (which
we require for all virtuals)
- [x] rework `test_all_virtual_packages_have_default_providers` as an audit called
`_ensure_all_virtual_packages_have_default_providers`
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
The `Executable` class emits a warning when you pass quoted arguments to it:
```
> spack pkg grep '"namespace"'
==> Warning: Quotes in command arguments can confuse scripts like configure.
The following arguments may cause problems when executed:
"namespace"
Quotes aren't needed because spack doesn't use a shell. Consider removing them.
If multiple levels of quotation are required, use `ignore_quotes=True`.
```
This is to warn new package authors who aren't used to calling build commands in python.
It's not useful for `spack pkg grep`, where we really are passing args on the command
line, and if we pass a quoted string, we probably meant to.
- [x] make `ignore_quotes` an instance variable, not just an argument to ``__call__`
- [x] set `ignore_quotes` to `True` in `spack pkg grep`
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
* Several tests can be enabled on Windows with no changes to logic
(either the test logic or logic being tested)
* Test for `spack location` requires modification of the test logic,
but with a minor change can be enabled on Windows
The "use_store" context manager is used to swap the value
of a global variable (spack.store.STORE), while keeping
another global variable consistent (spack.config.CONFIG).
When doing that it tries to evaluate the previous value
of the store, if that was not done already. This is wrong,
since the configuration might be in an "intermediate" state
that was never meant to trigger side effects.
Remove that operation, and add a unit test to
prevent regressions.
Since the the MetaPathFinder now owns a lazily constructed RepoPath object, we need to deserialize environments before the package that needs to be restored. Before we were relying on globals to be inconsistent in a way that let the entire process go.
