Allow other CI checks to continue even if the circular import check fails.
Circular imports are often indicative of code that needs to be restructured, but there
are still places where we need to add them. This allows CI to continue, so that if CI is
passing *except* for a problematic import, we can still force merge and accept a bit
of technical debt to admit a useful feature. Also, this allows people to keep working
while they fix their circular import issues, without being blind to CI results.
- [x] update ci workflow: import-check continue-on-error: true
* Bump the package API of the `builtin` repo to `v2.0`
* Move `var/spack/repos/builtin` -> `var/spack/repos/spack_repo/builtin`
* Move test repos `var/spack/repos/{builtin.mock,tutorial,...}` -> `var/spack/test_repos/`
* Update package dir names to v2 format (`-` -> `_` etc)
* Change absolute imports `from spack.pkg.builtin.my_pkg ...` to relative imports `from ..my_pkg.package ...`
Users who have a repo on top of builtin should change imports from
```python
from spack.pkg.builtin.my_pkg import MyPkg
```
to
```python
from spack_repo.builtin.packages.my_pkg.package import MyPkg
```
and can configure their editors with
```
PYTHONPATH=$spack/lib/spack:$spack/var/spack/repos
```
[skip-verify-checksums]
Compatibility with Python 3.6 is still tested by the
rhel8-platform-python job, and Ubuntu 20.04 will be
removed soon from the list of runners:
https://github.com/actions/runner-images/issues/11101
Signed-off-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Add a CI check to automatically verify the checksums of newly added
package versions:
- [x] a new command, `spack ci verify-versions`
- [x] a GitHub actions check to run the command
- [x] tests for the new command
This also eliminates the suggestion for maintainers to manually verify added
checksums in the case of accidental version <--> checksum mismatches.
----
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Codecov needs to see the token secret when uploading, so we have to
add this line to the workflow YAML:
```yaml
with:
token: ${{ secrets.CODECOV_TOKEN }}
```
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
The import-check action now presents problematic import statements
introduced by the PR better.
The idea is roughly:
* Let (V₁, E₁) be the graph of modules as vertices and import statements
as edges before the change
* Let (V₂, E₂) be the graph after the code change, which is typically a small
perturbation of (V₁, E₁).
* X₁ = FAS(V₁, E₁) is the feedback arc set before (a minimal set of edges to
delete to make it acyclic)
* X₂ = FAS(V₂, E₂ ∖ X₁) is the feedback arc set after deletion of the minimal
set of edges that made the old graph acyclic.
* X₃ = FAS(V₂, E₂) is the feedback arc set after
Previously I displayed X₁ and X₃ and users had to diff themselves.
Now, I'm showing X₂, which is a small set, typically directly related to
code changes.
However, it can be that a small code change adding say 2 problematic imports
creates a completely different solution X₃ that only requires deletion of just 1
different import. In that case the user is informed that they can potentially do
less work.
So for PR #48784 the output is now:
> The overall number of problematic import statements increased by 1 from 31 to 32.
> This is likely a direct consequence of the following import statements:
>
> ```
> spack/config imports: spack.spec, spack.util.path, spack.util.remote_file_cache
> ```
>
> However, instead of removing 3 import statements, it is sufficient to remove only 1
> import statement from the following list:
>
> ```
> spack/concretize imports: spack.bootstrap, spack.solver.asp
> spack/environment imports: spack.bootstrap, spack.environment
> spack/fetch_strategy imports: spack.version.git_ref_lookup
> spack/install_test imports: spack.build_environment, spack.package_base
> spack/modules imports: spack.modules
> spack/platforms imports: spack.config
> spack/relocate imports: spack.bootstrap
> spack/repo imports: spack.package_base, spack.patch, spack.tag
> spack/spec imports: spack.binary_distribution, spack.compiler, spack.compilers, spack.concretize, spack.environment, spack.hash_types, spack.provider_index, spack.repo, spack.spec_parser, spack.store, spack.traverse, spack.variant, spack.version.git_ref_lookup
> spack/subprocess_context imports: spack.environment
> spack/util/gpg imports: spack.bootstrap
> spack/util/package_hash imports: spack.package_base
> spack/util/path imports: spack.config, spack.environment
> spack/util/remote_file_cache imports: spack.util.web
> ```
from which the user can figure out that
`spack/util/remote_file_cache imports: spack.util.web` is the "bottleneck" now.
* Add type-hints to `spack.util.executable.Executable`
* Add type-hint to input
* Use overload, and remove assertions at calling sites
* Bump mypy to v1.11.2 (working locally), Python to 3.13
A few changes to tarball creation (for build caches):
- do not run file to distinguish binary from text
- file is slow, even when running it in a batched fashion -- it usually reads all bytes and has slow logic to categorize specific types
- we don't need a highly detailed file categorization; a crude categorization of elf, mach-o, text suffices.
detecting elf and mach-o is straightforward and cheap
- detecting utf-8 (and with that ascii) is highly accurate: false positive rate decays exponentially as file size increases. Further it's not only the most common encoding, but the most common file type in package prefixes.
iso-8859-1 is cheaply (but heuristically) detected too, and sufficiently accurate after binaries and utf-8 files are classified earlier
- remove file as a dependency of Spack in general, which makes Spack itself easier to install
- detect file type and need to relocate as part of creating the tarball, which is more cache friendly and thus faster
`kcov` was removed in Ubuntu 24.04, and it is no longer
installable via `apt` in our CI images. Instal it via
Linuxbrew instead, at least until it comes back to Ubuntu.
`subversion` is also not installed on ubuntu 24 by default,
so we have to install it manually.
- [x] Add linuxbrew to linux tests
- [x] Install `kcov` with brew
- [x] Install subversion with `apt`
Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
