Refer to mirrors by name, path, or url (#34891)
With this change we get the invariant that `mirror.fetch_url` and
`mirror.push_url` return valid URLs, even when the backing config
file is actually using (relative) paths with potentially `$spack` and
`$env` like variables.
Secondly it avoids expanding mirror path / URLs too early,
so if I say `spack mirror add name ./path`, it stays `./path` in my
config. When it's retrieved through MirrorCollection() we
exand it to say `file://<env dir>/path` if `./path` was set in an
environment scope.
Thirdly, the interface is simplified for the relevant buildcache
commands, so it's more like `git push`:
```
spack buildcache create [mirror] [specs...]
```
`mirror` is either a mirror name, a path, or a URL.
Resolving the relevant mirror goes as follows:
- If it contains either / or \ it is used as an anonymous mirror with
path or url.
- Otherwise, it's interpreted as a named mirror, which must exist.
This helps to guard against typos, e.g. typing `my-mirror` when there
is no such named mirror now errors with:
```
$ spack -e . buildcache create my-mirror
==> Error: no mirror named "my-mirror". Did you mean ./my-mirror?
```
instead of creating a directory in the current working directory. I
think this is reasonable, as the alternative (requiring that a local dir
exists) feels a bit pendantic in the general case -- spack is happy to
create the build cache dir when needed, saving a `mkdir`.
The old (now deprecated) format will still be available in Spack 0.20,
but is scheduled to be removed in 0.21:
```
spack buildcache create (--directory | --mirror-url | --mirror-name) [specs...]
```
This PR also touches `tmp_scope` in tests, because it didn't really
work for me, since spack fixes the possible --scope values once and
for all across tests, so tests failed when run out of order.
This commit is contained in:
Harmen Stoppels
GitHub
parent
9a25e21da8
commit
3489cc0a9b
@ -13,49 +13,51 @@ Some sites may encourage users to set up their own test environments
|
||||
before carrying out central installations, or some users may prefer to set
|
||||
up these environments on their own motivation. To reduce the load of
|
||||
recompiling otherwise identical package specs in different installations,
|
||||
installed packages can be put into build cache tarballs, uploaded to
|
||||
installed packages can be put into build cache tarballs, pushed to
|
||||
your Spack mirror and then downloaded and installed by others.
|
||||
|
||||
Whenever a mirror provides prebuilt packages, Spack will take these packages
|
||||
into account during concretization and installation, making ``spack install``
|
||||
signficantly faster.
|
||||
|
||||
--------------------------
|
||||
Creating build cache files
|
||||
--------------------------
|
||||
|
||||
A compressed tarball of an installed package is created. Tarballs are created
|
||||
for all of its link and run dependency packages as well. Compressed tarballs are
|
||||
signed with gpg and signature and tarball and put in a ``.spack`` file. Optionally,
|
||||
the rpaths (and ids and deps on macOS) can be changed to paths relative to
|
||||
the Spack install tree before the tarball is created.
|
||||
.. note::
|
||||
|
||||
We use the terms "build cache" and "mirror" often interchangeably. Mirrors
|
||||
are used during installation both for sources and prebuilt packages. Build
|
||||
caches refer to mirrors that provide prebuilt packages.
|
||||
|
||||
|
||||
----------------------
|
||||
Creating a build cache
|
||||
----------------------
|
||||
|
||||
Build caches are created via:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack buildcache create <spec>
|
||||
$ spack buildcache create <path/url/mirror name> <spec>
|
||||
|
||||
This command takes the locally installed spec and its dependencies, and
|
||||
creates tarballs of their install prefixes. It also generates metadata files,
|
||||
signed with GPG. These tarballs and metadata files are then pushed to the
|
||||
provided binary cache, which can be a local directory or a remote URL.
|
||||
|
||||
If you wanted to create a build cache in a local directory, you would provide
|
||||
the ``-d`` argument to target that directory, again also specifying the spec.
|
||||
Here is an example creating a local directory, "spack-cache" and creating
|
||||
build cache files for the "ninja" spec:
|
||||
Here is an example where a build cache is created in a local directory named
|
||||
"spack-cache", to which we push the "ninja" spec:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ mkdir -p ./spack-cache
|
||||
$ spack buildcache create -d ./spack-cache ninja
|
||||
==> Buildcache files will be output to file:///home/spackuser/spack/spack-cache/build_cache
|
||||
gpgconf: socketdir is '/run/user/1000/gnupg'
|
||||
gpg: using "E6DF6A8BD43208E4D6F392F23777740B7DBD643D" as default secret key for signing
|
||||
$ spack buildcache create --allow-root ./spack-cache ninja
|
||||
==> Pushing binary packages to file:///home/spackuser/spack/spack-cache/build_cache
|
||||
|
||||
Note that the targeted spec must already be installed. Once you have a build cache,
|
||||
you can add it as a mirror, discussed next.
|
||||
Not that ``ninja`` must be installed locally for this to work.
|
||||
|
||||
.. warning::
|
||||
We're using the ``--allow-root`` flag to tell Spack that is OK when any of
|
||||
the binaries we're pushing contain references to the local Spack install
|
||||
directory.
|
||||
|
||||
Spack improved the format used for binary caches in v0.18. The entire v0.18 series
|
||||
will be able to verify and install binary caches both in the new and in the old format.
|
||||
Support for using the old format is expected to end in v0.19, so we advise users to
|
||||
recreate relevant buildcaches using Spack v0.18 or higher.
|
||||
Once you have a build cache, you can add it as a mirror, discussed next.
|
||||
|
||||
---------------------------------------
|
||||
Finding or installing build cache files
|
||||
@ -66,10 +68,10 @@ with:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack mirror add <name> <url>
|
||||
$ spack mirror add <name> <url or path>
|
||||
|
||||
|
||||
Note that the url can be a web url _or_ a local filesystem location. In the previous
|
||||
Both web URLs and local paths on the filesystem can be specified. In the previous
|
||||
example, you might add the directory "spack-cache" and call it ``mymirror``:
|
||||
|
||||
|
||||
@ -94,7 +96,7 @@ this new build cache as follows:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack buildcache update-index -d spack-cache/
|
||||
$ spack buildcache update-index ./spack-cache
|
||||
|
||||
Now you can use list:
|
||||
|
||||
@ -105,46 +107,38 @@ Now you can use list:
|
||||
-- linux-ubuntu20.04-skylake / gcc@9.3.0 ------------------------
|
||||
ninja@1.10.2
|
||||
|
||||
|
||||
Great! So now let's say you have a different spack installation, or perhaps just
|
||||
a different environment for the same one, and you want to install a package from
|
||||
that build cache. Let's first uninstall the actual library "ninja" to see if we can
|
||||
re-install it from the cache.
|
||||
With ``mymirror`` configured and an index available, Spack will automatically
|
||||
use it during concretization and installation. That means that you can expect
|
||||
``spack install ninja`` to fetch prebuilt packages from the mirror. Let's
|
||||
verify by re-installing ninja:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack uninstall ninja
|
||||
|
||||
|
||||
And now reinstall from the buildcache
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack buildcache install ninja
|
||||
==> buildcache spec(s) matching ninja
|
||||
==> Fetching file:///home/spackuser/spack/spack-cache/build_cache/linux-ubuntu20.04-skylake/gcc-9.3.0/ninja-1.10.2/linux-ubuntu20.04-skylake-gcc-9.3.0-ninja-1.10.2-i4e5luour7jxdpc3bkiykd4imke3mkym.spack
|
||||
####################################################################################################################################### 100.0%
|
||||
==> Installing buildcache for spec ninja@1.10.2%gcc@9.3.0 arch=linux-ubuntu20.04-skylake
|
||||
gpgconf: socketdir is '/run/user/1000/gnupg'
|
||||
gpg: Signature made Tue 23 Mar 2021 10:16:29 PM MDT
|
||||
gpg: using RSA key E6DF6A8BD43208E4D6F392F23777740B7DBD643D
|
||||
gpg: Good signature from "spackuser (GPG created for Spack) <spackuser@noreply.users.github.com>" [ultimate]
|
||||
$ spack install ninja
|
||||
==> Installing ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz
|
||||
==> Fetching file:///home/spackuser/spack/spack-cache/build_cache/linux-ubuntu20.04-skylake-gcc-9.3.0-ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spec.json.sig
|
||||
gpg: Signature made Do 12 Jan 2023 16:01:04 CET
|
||||
gpg: using RSA key 61B82B2B2350E171BD17A1744E3A689061D57BF6
|
||||
gpg: Good signature from "example (GPG created for Spack) <example@example.com>" [ultimate]
|
||||
==> Fetching file:///home/spackuser/spack/spack-cache/build_cache/linux-ubuntu20.04-skylake/gcc-9.3.0/ninja-1.10.2/linux-ubuntu20.04-skylake-gcc-9.3.0-ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spack
|
||||
==> Extracting ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz from binary cache
|
||||
==> ninja: Successfully installed ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz
|
||||
Search: 0.00s. Fetch: 0.17s. Install: 0.12s. Total: 0.29s
|
||||
[+] /home/harmen/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz
|
||||
|
||||
|
||||
It worked! You've just completed a full example of creating a build cache with
|
||||
a spec of interest, adding it as a mirror, updating it's index, listing the contents,
|
||||
a spec of interest, adding it as a mirror, updating its index, listing the contents,
|
||||
and finally, installing from it.
|
||||
|
||||
|
||||
Note that the above command is intended to install a particular package to a
|
||||
build cache you have created, and not to install a package from a build cache.
|
||||
For the latter, once a mirror is added, by default when you do ``spack install`` the ``--use-cache``
|
||||
flag is set, and you will install a package from a build cache if it is available.
|
||||
If you want to always use the cache, you can do:
|
||||
By default Spack falls back to building from sources when the mirror is not available
|
||||
or when the package is simply not already available. To force Spack to only install
|
||||
prebuilt packages, you can use
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack install --cache-only <package>
|
||||
$ spack install --use-buildcache only <package>
|
||||
|
||||
For example, to combine all of the commands above to add the E4S build cache
|
||||
and then install from it exclusively, you would do:
|
||||
@ -153,7 +147,7 @@ and then install from it exclusively, you would do:
|
||||
|
||||
$ spack mirror add E4S https://cache.e4s.io
|
||||
$ spack buildcache keys --install --trust
|
||||
$ spack install --cache-only <package>
|
||||
$ spack install --use-buildache only <package>
|
||||
|
||||
We use ``--install`` and ``--trust`` to say that we are installing keys to our
|
||||
keyring, and trusting all downloaded keys.
|
||||
|
||||
@ -94,22 +94,15 @@ class Bootstrapper:
|
||||
def __init__(self, conf):
|
||||
self.conf = conf
|
||||
self.name = conf["name"]
|
||||
self.url = conf["info"]["url"]
|
||||
self.metadata_dir = spack.util.path.canonicalize_path(conf["metadata"])
|
||||
|
||||
@property
|
||||
def mirror_url(self):
|
||||
"""Mirror url associated with this bootstrapper"""
|
||||
# Absolute paths
|
||||
if os.path.isabs(self.url):
|
||||
return spack.util.url.format(self.url)
|
||||
|
||||
# Check for :// and assume it's an url if we find it
|
||||
if "://" in self.url:
|
||||
return self.url
|
||||
|
||||
# Otherwise, it's a relative path
|
||||
return spack.util.url.format(os.path.join(self.metadata_dir, self.url))
|
||||
# Promote (relative) paths to file urls
|
||||
url = conf["info"]["url"]
|
||||
if spack.util.url.is_path_instead_of_url(url):
|
||||
if not os.path.isabs(url):
|
||||
url = os.path.join(self.metadata_dir, url)
|
||||
url = spack.util.url.path_to_file_url(url)
|
||||
self.url = url
|
||||
|
||||
@property
|
||||
def mirror_scope(self):
|
||||
@ -117,7 +110,7 @@ def mirror_scope(self):
|
||||
this bootstrapper.
|
||||
"""
|
||||
return spack.config.InternalConfigScope(
|
||||
self.config_scope_name, {"mirrors:": {self.name: self.mirror_url}}
|
||||
self.config_scope_name, {"mirrors:": {self.name: self.url}}
|
||||
)
|
||||
|
||||
def try_import(self, module: str, abstract_spec_str: str) -> bool:
|
||||
|
||||
@ -1482,7 +1482,7 @@ def _push_mirror_contents(env, specfile_path, sign_binaries, mirror_url):
|
||||
tty.debug("Creating buildcache ({0})".format("unsigned" if unsigned else "signed"))
|
||||
hashes = env.all_hashes() if env else None
|
||||
matches = spack.store.specfile_matches(specfile_path, hashes=hashes)
|
||||
push_url = spack.mirror.push_url_from_mirror_url(mirror_url)
|
||||
push_url = spack.mirror.Mirror.from_url(mirror_url).push_url
|
||||
spec_kwargs = {"include_root": True, "include_dependencies": False}
|
||||
kwargs = {"force": True, "allow_root": True, "unsigned": unsigned}
|
||||
bindist.push(matches, push_url, spec_kwargs, **kwargs)
|
||||
|
||||
@ -43,6 +43,8 @@
|
||||
"The sha256 checksum of binaries is checked before installation."
|
||||
),
|
||||
"info": {
|
||||
# This is a mis-nomer since it's not a URL; but file urls cannot
|
||||
# represent relative paths, so we have to live with it for now.
|
||||
"url": os.path.join("..", "..", LOCAL_MIRROR_DIR),
|
||||
"homepage": "https://github.com/spack/spack-bootstrap-mirrors",
|
||||
"releases": "https://github.com/spack/spack-bootstrap-mirrors/releases",
|
||||
@ -58,7 +60,11 @@
|
||||
SOURCE_METADATA = {
|
||||
"type": "install",
|
||||
"description": "Mirror with software needed to bootstrap Spack",
|
||||
"info": {"url": os.path.join("..", "..", LOCAL_MIRROR_DIR)},
|
||||
"info": {
|
||||
# This is a mis-nomer since it's not a URL; but file urls cannot
|
||||
# represent relative paths, so we have to live with it for now.
|
||||
"url": os.path.join("..", "..", LOCAL_MIRROR_DIR)
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -8,7 +8,6 @@
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib.parse
|
||||
|
||||
import llnl.util.tty as tty
|
||||
|
||||
@ -66,27 +65,37 @@ def setup_parser(subparser):
|
||||
create.add_argument(
|
||||
"-k", "--key", metavar="key", type=str, default=None, help="Key for signing."
|
||||
)
|
||||
output = create.add_mutually_exclusive_group(required=True)
|
||||
output = create.add_mutually_exclusive_group(required=False)
|
||||
# TODO: remove from Spack 0.21
|
||||
output.add_argument(
|
||||
"-d",
|
||||
"--directory",
|
||||
metavar="directory",
|
||||
type=str,
|
||||
help="local directory where buildcaches will be written.",
|
||||
dest="mirror_flag",
|
||||
type=arguments.mirror_directory,
|
||||
help="local directory where buildcaches will be written. (deprecated)",
|
||||
)
|
||||
# TODO: remove from Spack 0.21
|
||||
output.add_argument(
|
||||
"-m",
|
||||
"--mirror-name",
|
||||
metavar="mirror-name",
|
||||
type=str,
|
||||
help="name of the mirror where buildcaches will be written.",
|
||||
dest="mirror_flag",
|
||||
type=arguments.mirror_name,
|
||||
help="name of the mirror where buildcaches will be written. (deprecated)",
|
||||
)
|
||||
# TODO: remove from Spack 0.21
|
||||
output.add_argument(
|
||||
"--mirror-url",
|
||||
metavar="mirror-url",
|
||||
type=str,
|
||||
help="URL of the mirror where buildcaches will be written.",
|
||||
dest="mirror_flag",
|
||||
type=arguments.mirror_url,
|
||||
help="URL of the mirror where buildcaches will be written. (deprecated)",
|
||||
)
|
||||
# Unfortunately we cannot add this to the mutually exclusive group above,
|
||||
# because we have further positional arguments.
|
||||
# TODO: require from Spack 0.21
|
||||
create.add_argument("mirror", type=str, help="Mirror name, path, or URL.", nargs="?")
|
||||
create.add_argument(
|
||||
"--rebuild-index",
|
||||
action="store_true",
|
||||
@ -179,7 +188,7 @@ def setup_parser(subparser):
|
||||
"-m",
|
||||
"--mirror-url",
|
||||
default=None,
|
||||
help="Override any configured mirrors with this mirror url",
|
||||
help="Override any configured mirrors with this mirror URL",
|
||||
)
|
||||
|
||||
check.add_argument(
|
||||
@ -266,55 +275,108 @@ def setup_parser(subparser):
|
||||
help="A quoted glob pattern identifying copy manifest files",
|
||||
)
|
||||
source = sync.add_mutually_exclusive_group(required=False)
|
||||
# TODO: remove in Spack 0.21
|
||||
source.add_argument(
|
||||
"--src-directory", metavar="DIRECTORY", type=str, help="Source mirror as a local file path"
|
||||
"--src-directory",
|
||||
metavar="DIRECTORY",
|
||||
dest="src_mirror_flag",
|
||||
type=arguments.mirror_directory,
|
||||
help="Source mirror as a local file path (deprecated)",
|
||||
)
|
||||
# TODO: remove in Spack 0.21
|
||||
source.add_argument(
|
||||
"--src-mirror-name", metavar="MIRROR_NAME", type=str, help="Name of the source mirror"
|
||||
"--src-mirror-name",
|
||||
metavar="MIRROR_NAME",
|
||||
dest="src_mirror_flag",
|
||||
type=arguments.mirror_name,
|
||||
help="Name of the source mirror (deprecated)",
|
||||
)
|
||||
# TODO: remove in Spack 0.21
|
||||
source.add_argument(
|
||||
"--src-mirror-url", metavar="MIRROR_URL", type=str, help="URL of the source mirror"
|
||||
"--src-mirror-url",
|
||||
metavar="MIRROR_URL",
|
||||
dest="src_mirror_flag",
|
||||
type=arguments.mirror_url,
|
||||
help="URL of the source mirror (deprecated)",
|
||||
)
|
||||
# TODO: only support this in 0.21
|
||||
source.add_argument(
|
||||
"src_mirror",
|
||||
metavar="source mirror",
|
||||
type=arguments.mirror_name_or_url,
|
||||
help="Source mirror name, path, or URL",
|
||||
nargs="?",
|
||||
)
|
||||
dest = sync.add_mutually_exclusive_group(required=False)
|
||||
# TODO: remove in Spack 0.21
|
||||
dest.add_argument(
|
||||
"--dest-directory",
|
||||
metavar="DIRECTORY",
|
||||
type=str,
|
||||
help="Destination mirror as a local file path",
|
||||
dest="dest_mirror_flag",
|
||||
type=arguments.mirror_directory,
|
||||
help="Destination mirror as a local file path (deprecated)",
|
||||
)
|
||||
# TODO: remove in Spack 0.21
|
||||
dest.add_argument(
|
||||
"--dest-mirror-name",
|
||||
metavar="MIRROR_NAME",
|
||||
type=str,
|
||||
help="Name of the destination mirror",
|
||||
type=arguments.mirror_name,
|
||||
dest="dest_mirror_flag",
|
||||
help="Name of the destination mirror (deprecated)",
|
||||
)
|
||||
# TODO: remove in Spack 0.21
|
||||
dest.add_argument(
|
||||
"--dest-mirror-url", metavar="MIRROR_URL", type=str, help="URL of the destination mirror"
|
||||
"--dest-mirror-url",
|
||||
metavar="MIRROR_URL",
|
||||
dest="dest_mirror_flag",
|
||||
type=arguments.mirror_url,
|
||||
help="URL of the destination mirror (deprecated)",
|
||||
)
|
||||
# TODO: only support this in 0.21
|
||||
dest.add_argument(
|
||||
"dest_mirror",
|
||||
metavar="destination mirror",
|
||||
type=arguments.mirror_name_or_url,
|
||||
help="Destination mirror name, path, or URL",
|
||||
nargs="?",
|
||||
)
|
||||
sync.set_defaults(func=sync_fn)
|
||||
|
||||
# Update buildcache index without copying any additional packages
|
||||
update_index = subparsers.add_parser("update-index", help=update_index_fn.__doc__)
|
||||
update_index_out = update_index.add_mutually_exclusive_group(required=True)
|
||||
# TODO: remove in Spack 0.21
|
||||
update_index_out.add_argument(
|
||||
"-d",
|
||||
"--directory",
|
||||
metavar="directory",
|
||||
type=str,
|
||||
help="local directory where buildcaches will be written.",
|
||||
dest="mirror_flag",
|
||||
type=arguments.mirror_directory,
|
||||
help="local directory where buildcaches will be written (deprecated)",
|
||||
)
|
||||
# TODO: remove in Spack 0.21
|
||||
update_index_out.add_argument(
|
||||
"-m",
|
||||
"--mirror-name",
|
||||
metavar="mirror-name",
|
||||
type=str,
|
||||
help="name of the mirror where buildcaches will be written.",
|
||||
dest="mirror_flag",
|
||||
type=arguments.mirror_name,
|
||||
help="name of the mirror where buildcaches will be written (deprecated)",
|
||||
)
|
||||
# TODO: remove in Spack 0.21
|
||||
update_index_out.add_argument(
|
||||
"--mirror-url",
|
||||
metavar="mirror-url",
|
||||
type=str,
|
||||
help="URL of the mirror where buildcaches will be written.",
|
||||
dest="mirror_flag",
|
||||
type=arguments.mirror_url,
|
||||
help="URL of the mirror where buildcaches will be written (deprecated)",
|
||||
)
|
||||
# TODO: require from Spack 0.21
|
||||
update_index_out.add_argument(
|
||||
"mirror",
|
||||
type=arguments.mirror_name_or_url,
|
||||
help="Destination mirror name, path, or URL",
|
||||
nargs="?",
|
||||
)
|
||||
update_index.add_argument(
|
||||
"-k",
|
||||
@ -326,26 +388,17 @@ def setup_parser(subparser):
|
||||
update_index.set_defaults(func=update_index_fn)
|
||||
|
||||
|
||||
def _mirror_url_from_args(args):
|
||||
if args.directory:
|
||||
return spack.mirror.push_url_from_directory(args.directory)
|
||||
if args.mirror_name:
|
||||
return spack.mirror.push_url_from_mirror_name(args.mirror_name)
|
||||
if args.mirror_url:
|
||||
return spack.mirror.push_url_from_mirror_url(args.mirror_url)
|
||||
|
||||
|
||||
def _matching_specs(args):
|
||||
def _matching_specs(specs, spec_file):
|
||||
"""Return a list of matching specs read from either a spec file (JSON or YAML),
|
||||
a query over the store or a query over the active environment.
|
||||
"""
|
||||
env = ev.active_environment()
|
||||
hashes = env.all_hashes() if env else None
|
||||
if args.spec_file:
|
||||
return spack.store.specfile_matches(args.spec_file, hashes=hashes)
|
||||
if spec_file:
|
||||
return spack.store.specfile_matches(spec_file, hashes=hashes)
|
||||
|
||||
if args.specs:
|
||||
constraints = spack.cmd.parse_specs(args.specs)
|
||||
if specs:
|
||||
constraints = spack.cmd.parse_specs(specs)
|
||||
return spack.store.find(constraints, hashes=hashes)
|
||||
|
||||
if env:
|
||||
@ -383,10 +436,30 @@ def _concrete_spec_from_args(args):
|
||||
|
||||
def create_fn(args):
|
||||
"""create a binary package and push it to a mirror"""
|
||||
push_url = _mirror_url_from_args(args)
|
||||
matches = _matching_specs(args)
|
||||
if args.mirror_flag:
|
||||
mirror = args.mirror_flag
|
||||
elif not args.mirror:
|
||||
raise ValueError("No mirror provided")
|
||||
else:
|
||||
mirror = arguments.mirror_name_or_url(args.mirror)
|
||||
|
||||
msg = "Pushing binary packages to {0}/build_cache".format(push_url)
|
||||
if args.mirror_flag:
|
||||
tty.warn(
|
||||
"Using flags to specify mirrors is deprecated and will be removed in "
|
||||
"Spack 0.21, use positional arguments instead."
|
||||
)
|
||||
|
||||
# TODO: remove this in 0.21. If we have mirror_flag, the first
|
||||
# spec is in the positional mirror arg due to argparse limitations.
|
||||
specs = args.specs
|
||||
if args.mirror_flag and args.mirror:
|
||||
specs.insert(0, args.mirror)
|
||||
|
||||
url = mirror.push_url
|
||||
|
||||
matches = _matching_specs(specs, args.spec_file)
|
||||
|
||||
msg = "Pushing binary packages to {0}/build_cache".format(url)
|
||||
tty.msg(msg)
|
||||
specs_kwargs = {
|
||||
"include_root": "package" in args.things_to_install,
|
||||
@ -400,7 +473,7 @@ def create_fn(args):
|
||||
"allow_root": args.allow_root,
|
||||
"regenerate_index": args.rebuild_index,
|
||||
}
|
||||
bindist.push(matches, push_url, specs_kwargs, **kwargs)
|
||||
bindist.push(matches, url, specs_kwargs, **kwargs)
|
||||
|
||||
|
||||
def install_fn(args):
|
||||
@ -593,51 +666,24 @@ def sync_fn(args):
|
||||
manifest_copy(glob.glob(args.manifest_glob))
|
||||
return 0
|
||||
|
||||
# Figure out the source mirror
|
||||
source_location = None
|
||||
if args.src_directory:
|
||||
source_location = args.src_directory
|
||||
scheme = urllib.parse.urlparse(source_location, scheme="<missing>").scheme
|
||||
if scheme != "<missing>":
|
||||
raise ValueError('"--src-directory" expected a local path; got a URL, instead')
|
||||
# Ensure that the mirror lookup does not mistake this for named mirror
|
||||
source_location = url_util.path_to_file_url(source_location)
|
||||
elif args.src_mirror_name:
|
||||
source_location = args.src_mirror_name
|
||||
result = spack.mirror.MirrorCollection().lookup(source_location)
|
||||
if result.name == "<unnamed>":
|
||||
raise ValueError('no configured mirror named "{name}"'.format(name=source_location))
|
||||
elif args.src_mirror_url:
|
||||
source_location = args.src_mirror_url
|
||||
scheme = urllib.parse.urlparse(source_location, scheme="<missing>").scheme
|
||||
if scheme == "<missing>":
|
||||
raise ValueError('"{url}" is not a valid URL'.format(url=source_location))
|
||||
# If no manifest_glob, require a source and dest mirror.
|
||||
# TODO: Simplify in Spack 0.21
|
||||
if not (args.src_mirror_flag or args.src_mirror) or not (
|
||||
args.dest_mirror_flag or args.dest_mirror
|
||||
):
|
||||
raise ValueError("Source and destination mirror are required.")
|
||||
|
||||
src_mirror = spack.mirror.MirrorCollection().lookup(source_location)
|
||||
src_mirror_url = url_util.format(src_mirror.fetch_url)
|
||||
if args.src_mirror_flag or args.dest_mirror_flag:
|
||||
tty.warn(
|
||||
"Using flags to specify mirrors is deprecated and will be removed in "
|
||||
"Spack 0.21, use positional arguments instead."
|
||||
)
|
||||
|
||||
# Figure out the destination mirror
|
||||
dest_location = None
|
||||
if args.dest_directory:
|
||||
dest_location = args.dest_directory
|
||||
scheme = urllib.parse.urlparse(dest_location, scheme="<missing>").scheme
|
||||
if scheme != "<missing>":
|
||||
raise ValueError('"--dest-directory" expected a local path; got a URL, instead')
|
||||
# Ensure that the mirror lookup does not mistake this for named mirror
|
||||
dest_location = url_util.path_to_file_url(dest_location)
|
||||
elif args.dest_mirror_name:
|
||||
dest_location = args.dest_mirror_name
|
||||
result = spack.mirror.MirrorCollection().lookup(dest_location)
|
||||
if result.name == "<unnamed>":
|
||||
raise ValueError('no configured mirror named "{name}"'.format(name=dest_location))
|
||||
elif args.dest_mirror_url:
|
||||
dest_location = args.dest_mirror_url
|
||||
scheme = urllib.parse.urlparse(dest_location, scheme="<missing>").scheme
|
||||
if scheme == "<missing>":
|
||||
raise ValueError('"{url}" is not a valid URL'.format(url=dest_location))
|
||||
src_mirror = args.src_mirror_flag if args.src_mirror_flag else args.src_mirror
|
||||
dest_mirror = args.dest_mirror_flag if args.dest_mirror_flag else args.dest_mirror
|
||||
|
||||
dest_mirror = spack.mirror.MirrorCollection().lookup(dest_location)
|
||||
dest_mirror_url = url_util.format(dest_mirror.fetch_url)
|
||||
src_mirror_url = src_mirror.fetch_url
|
||||
dest_mirror_url = dest_mirror.push_url
|
||||
|
||||
# Get the active environment
|
||||
env = spack.cmd.require_active_env(cmd_name="buildcache sync")
|
||||
@ -698,38 +744,28 @@ def manifest_copy(manifest_file_list):
|
||||
copy_buildcache_file(copy_file["src"], copy_file["dest"])
|
||||
|
||||
|
||||
def update_index(mirror_url, update_keys=False):
|
||||
mirror = spack.mirror.MirrorCollection().lookup(mirror_url)
|
||||
outdir = url_util.format(mirror.push_url)
|
||||
def update_index(mirror: spack.mirror.Mirror, update_keys=False):
|
||||
url = mirror.push_url
|
||||
|
||||
bindist.generate_package_index(url_util.join(outdir, bindist.build_cache_relative_path()))
|
||||
bindist.generate_package_index(url_util.join(url, bindist.build_cache_relative_path()))
|
||||
|
||||
if update_keys:
|
||||
keys_url = url_util.join(
|
||||
outdir, bindist.build_cache_relative_path(), bindist.build_cache_keys_relative_path()
|
||||
url, bindist.build_cache_relative_path(), bindist.build_cache_keys_relative_path()
|
||||
)
|
||||
|
||||
bindist.generate_key_index(keys_url)
|
||||
|
||||
|
||||
def _mirror_url_from_args_deprecated_format(args):
|
||||
# In Spack 0.19 the -d flag was equivalent to --mirror-url.
|
||||
# Spack 0.20 deprecates this, so in 0.21 -d means --directory.
|
||||
if args.directory and url_util.validate_scheme(urllib.parse.urlparse(args.directory).scheme):
|
||||
tty.warn(
|
||||
"Passing a URL to `update-index -d <url>` is deprecated "
|
||||
"and will be removed in Spack 0.21. "
|
||||
"Use `update-index --mirror-url <url>` instead."
|
||||
)
|
||||
return spack.mirror.push_url_from_mirror_url(args.directory)
|
||||
else:
|
||||
return _mirror_url_from_args(args)
|
||||
|
||||
|
||||
def update_index_fn(args):
|
||||
"""Update a buildcache index."""
|
||||
push_url = _mirror_url_from_args_deprecated_format(args)
|
||||
update_index(push_url, update_keys=args.keys)
|
||||
if args.mirror_flag:
|
||||
tty.warn(
|
||||
"Using flags to specify mirrors is deprecated and will be removed in "
|
||||
"Spack 0.21, use positional arguments instead."
|
||||
)
|
||||
mirror = args.mirror_flag if args.mirror_flag else args.mirror
|
||||
update_index(mirror, update_keys=args.keys)
|
||||
|
||||
|
||||
def buildcache(parser, args):
|
||||
|
||||
@ -241,8 +241,9 @@ def ci_reindex(args):
|
||||
ci_mirrors = yaml_root["mirrors"]
|
||||
mirror_urls = [url for url in ci_mirrors.values()]
|
||||
remote_mirror_url = mirror_urls[0]
|
||||
mirror = spack.mirror.Mirror(remote_mirror_url)
|
||||
|
||||
buildcache.update_index(remote_mirror_url, update_keys=True)
|
||||
buildcache.update_index(mirror, update_keys=True)
|
||||
|
||||
|
||||
def ci_rebuild(args):
|
||||
|
||||
@ -5,6 +5,7 @@
|
||||
|
||||
|
||||
import argparse
|
||||
import os.path
|
||||
|
||||
from llnl.util.lang import stable_partition
|
||||
|
||||
@ -12,6 +13,7 @@
|
||||
import spack.config
|
||||
import spack.dependency as dep
|
||||
import spack.environment as ev
|
||||
import spack.mirror
|
||||
import spack.modules
|
||||
import spack.reporters
|
||||
import spack.spec
|
||||
@ -552,3 +554,42 @@ def is_valid(arg):
|
||||
dependencies = val
|
||||
|
||||
return package, dependencies
|
||||
|
||||
|
||||
def mirror_name_or_url(m):
|
||||
# Look up mirror by name or use anonymous mirror with path/url.
|
||||
# We want to guard against typos in mirror names, to avoid pushing
|
||||
# accidentally to a dir in the current working directory.
|
||||
|
||||
# If there's a \ or / in the name, it's interpreted as a path or url.
|
||||
if "/" in m or "\\" in m:
|
||||
return spack.mirror.Mirror(m)
|
||||
|
||||
# Otherwise, the named mirror is required to exist.
|
||||
try:
|
||||
return spack.mirror.require_mirror_name(m)
|
||||
except ValueError as e:
|
||||
raise argparse.ArgumentTypeError(
|
||||
str(e) + ". Did you mean {}?".format(os.path.join(".", m))
|
||||
)
|
||||
|
||||
|
||||
def mirror_url(url):
|
||||
try:
|
||||
return spack.mirror.Mirror.from_url(url)
|
||||
except ValueError as e:
|
||||
raise argparse.ArgumentTypeError(str(e))
|
||||
|
||||
|
||||
def mirror_directory(path):
|
||||
try:
|
||||
return spack.mirror.Mirror.from_local_path(path)
|
||||
except ValueError as e:
|
||||
raise argparse.ArgumentTypeError(str(e))
|
||||
|
||||
|
||||
def mirror_name(name):
|
||||
try:
|
||||
return spack.mirror.require_mirror_name(name)
|
||||
except ValueError as e:
|
||||
raise argparse.ArgumentTypeError(str(e))
|
||||
|
||||
@ -145,8 +145,7 @@ def setup_parser(subparser):
|
||||
|
||||
def mirror_add(args):
|
||||
"""Add a mirror to Spack."""
|
||||
url = url_util.format(args.url)
|
||||
spack.mirror.add(args.name, url, args.scope, args)
|
||||
spack.mirror.add(args.name, args.url, args.scope, args)
|
||||
|
||||
|
||||
def mirror_remove(args):
|
||||
@ -156,7 +155,7 @@ def mirror_remove(args):
|
||||
|
||||
def mirror_set_url(args):
|
||||
"""Change the URL of a mirror."""
|
||||
url = url_util.format(args.url)
|
||||
url = args.url
|
||||
mirrors = spack.config.get("mirrors", scope=args.scope)
|
||||
if not mirrors:
|
||||
mirrors = syaml_dict()
|
||||
|
||||
@ -31,15 +31,15 @@
|
||||
import spack.mirror
|
||||
import spack.spec
|
||||
import spack.url as url
|
||||
import spack.util.path
|
||||
import spack.util.spack_json as sjson
|
||||
import spack.util.spack_yaml as syaml
|
||||
import spack.util.url as url_util
|
||||
from spack.util.spack_yaml import syaml_dict
|
||||
from spack.version import VersionList
|
||||
|
||||
|
||||
def _is_string(url):
|
||||
return isinstance(url, str)
|
||||
#: What schemes do we support
|
||||
supported_url_schemes = ("file", "http", "https", "sftp", "ftp", "s3", "gs")
|
||||
|
||||
|
||||
def _display_mirror_entry(size, name, url, type_=None):
|
||||
@ -51,6 +51,19 @@ def _display_mirror_entry(size, name, url, type_=None):
|
||||
print("%-*s%s%s" % (size + 4, name, url, type_))
|
||||
|
||||
|
||||
def _url_or_path_to_url(url_or_path: str) -> str:
|
||||
"""For simplicity we allow mirror URLs in config files to be local, relative paths.
|
||||
This helper function takes care of distinguishing between URLs and paths, and
|
||||
canonicalizes paths before transforming them into file:// URLs."""
|
||||
# Is it a supported URL already? Then don't do path-related canonicalization.
|
||||
parsed = urllib.parse.urlparse(url_or_path)
|
||||
if parsed.scheme in supported_url_schemes:
|
||||
return url_or_path
|
||||
|
||||
# Otherwise we interpret it as path, and we should promote it to file:// URL.
|
||||
return url_util.path_to_file_url(spack.util.path.canonicalize_path(url_or_path))
|
||||
|
||||
|
||||
class Mirror(object):
|
||||
"""Represents a named location for storing source tarballs and binary
|
||||
packages.
|
||||
@ -90,6 +103,21 @@ def from_json(stream, name=None):
|
||||
except Exception as e:
|
||||
raise sjson.SpackJSONError("error parsing JSON mirror:", str(e)) from e
|
||||
|
||||
@staticmethod
|
||||
def from_local_path(path: str):
|
||||
return Mirror(fetch_url=url_util.path_to_file_url(path))
|
||||
|
||||
@staticmethod
|
||||
def from_url(url: str):
|
||||
"""Create an anonymous mirror by URL. This method validates the URL."""
|
||||
if not urllib.parse.urlparse(url).scheme in supported_url_schemes:
|
||||
raise ValueError(
|
||||
'"{}" is not a valid mirror URL. Scheme must be once of {}.'.format(
|
||||
url, ", ".join(supported_url_schemes)
|
||||
)
|
||||
)
|
||||
return Mirror(fetch_url=url)
|
||||
|
||||
def to_dict(self):
|
||||
if self._push_url is None:
|
||||
return syaml_dict([("fetch", self._fetch_url), ("push", self._fetch_url)])
|
||||
@ -201,7 +229,11 @@ def set_access_token(self, url_type, connection_token):
|
||||
|
||||
@property
|
||||
def fetch_url(self):
|
||||
return self._fetch_url if _is_string(self._fetch_url) else self._fetch_url["url"]
|
||||
"""Get the valid, canonicalized fetch URL"""
|
||||
url_or_path = (
|
||||
self._fetch_url if isinstance(self._fetch_url, str) else self._fetch_url["url"]
|
||||
)
|
||||
return _url_or_path_to_url(url_or_path)
|
||||
|
||||
@fetch_url.setter
|
||||
def fetch_url(self, url):
|
||||
@ -210,9 +242,12 @@ def fetch_url(self, url):
|
||||
|
||||
@property
|
||||
def push_url(self):
|
||||
"""Get the valid, canonicalized push URL. Returns fetch URL if no custom
|
||||
push URL is defined"""
|
||||
if self._push_url is None:
|
||||
return self._fetch_url if _is_string(self._fetch_url) else self._fetch_url["url"]
|
||||
return self._push_url if _is_string(self._push_url) else self._push_url["url"]
|
||||
return self.fetch_url
|
||||
url_or_path = self._push_url if isinstance(self._push_url, str) else self._push_url["url"]
|
||||
return _url_or_path_to_url(url_or_path)
|
||||
|
||||
@push_url.setter
|
||||
def push_url(self, url):
|
||||
@ -663,31 +698,12 @@ def create_mirror_from_package_object(pkg_obj, mirror_cache, mirror_stats):
|
||||
return True
|
||||
|
||||
|
||||
def push_url_from_directory(output_directory):
|
||||
"""Given a directory in the local filesystem, return the URL on
|
||||
which to push binary packages.
|
||||
"""
|
||||
if url_util.validate_scheme(urllib.parse.urlparse(output_directory).scheme):
|
||||
raise ValueError("expected a local path, but got a URL instead")
|
||||
mirror_url = url_util.path_to_file_url(output_directory)
|
||||
mirror = spack.mirror.MirrorCollection().lookup(mirror_url)
|
||||
return url_util.format(mirror.push_url)
|
||||
|
||||
|
||||
def push_url_from_mirror_name(mirror_name):
|
||||
"""Given a mirror name, return the URL on which to push binary packages."""
|
||||
mirror = spack.mirror.MirrorCollection().lookup(mirror_name)
|
||||
if mirror.name == "<unnamed>":
|
||||
def require_mirror_name(mirror_name):
|
||||
"""Find a mirror by name and raise if it does not exist"""
|
||||
mirror = spack.mirror.MirrorCollection().get(mirror_name)
|
||||
if not mirror:
|
||||
raise ValueError('no mirror named "{0}"'.format(mirror_name))
|
||||
return url_util.format(mirror.push_url)
|
||||
|
||||
|
||||
def push_url_from_mirror_url(mirror_url):
|
||||
"""Given a mirror URL, return the URL on which to push binary packages."""
|
||||
if not url_util.validate_scheme(urllib.parse.urlparse(mirror_url).scheme):
|
||||
raise ValueError('"{0}" is not a valid URL'.format(mirror_url))
|
||||
mirror = spack.mirror.MirrorCollection().lookup(mirror_url)
|
||||
return url_util.format(mirror.push_url)
|
||||
return mirror
|
||||
|
||||
|
||||
class MirrorError(spack.error.SpackError):
|
||||
|
||||
@ -6,10 +6,8 @@
|
||||
"""Schema for mirrors.yaml configuration file.
|
||||
|
||||
.. literalinclude:: _spack_root/lib/spack/spack/schema/mirrors.py
|
||||
:lines: 13-
|
||||
"""
|
||||
|
||||
|
||||
#: Properties for inclusion in other schemas
|
||||
properties = {
|
||||
"mirrors": {
|
||||
|
||||
@ -23,22 +23,6 @@
|
||||
import spack.util.spack_yaml as syaml
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tmp_scope():
|
||||
"""Creates a temporary configuration scope"""
|
||||
base_name = "internal-testing-scope"
|
||||
current_overrides = set(x.name for x in cfg.config.matching_scopes(r"^{0}".format(base_name)))
|
||||
|
||||
num_overrides = 0
|
||||
scope_name = base_name
|
||||
while scope_name in current_overrides:
|
||||
scope_name = "{0}{1}".format(base_name, num_overrides)
|
||||
num_overrides += 1
|
||||
|
||||
with cfg.override(cfg.InternalConfigScope(scope_name)):
|
||||
yield scope_name
|
||||
|
||||
|
||||
def test_urlencode_string():
|
||||
s = "Spack Test Project"
|
||||
|
||||
|
||||
@ -3,7 +3,6 @@
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
||||
import argparse
|
||||
import errno
|
||||
import os
|
||||
import platform
|
||||
@ -268,13 +267,3 @@ def test_buildcache_create_install(
|
||||
tarball = spack.binary_distribution.tarball_name(spec, ".spec.json")
|
||||
assert os.path.exists(os.path.join(str(tmpdir), "build_cache", tarball_path))
|
||||
assert os.path.exists(os.path.join(str(tmpdir), "build_cache", tarball))
|
||||
|
||||
|
||||
def test_deprecation_mirror_url_dir_flag(capfd):
|
||||
# Test that passing `update-index -d <url>` gives a deprecation warning.
|
||||
parser = argparse.ArgumentParser()
|
||||
spack.cmd.buildcache.setup_parser(parser)
|
||||
url = spack.util.url.path_to_file_url(os.getcwd())
|
||||
args = parser.parse_args(["update-index", "-d", url])
|
||||
spack.cmd.buildcache._mirror_url_from_args_deprecated_format(args)
|
||||
assert "Passing a URL to `update-index -d <url>` is deprecated" in capfd.readouterr()[1]
|
||||
|
||||
@ -1218,7 +1218,7 @@ def test_push_mirror_contents(
|
||||
working_dir = tmpdir.join("working_dir")
|
||||
|
||||
mirror_dir = working_dir.join("mirror")
|
||||
mirror_url = "file://{0}".format(mirror_dir.strpath)
|
||||
mirror_url = url_util.path_to_file_url(mirror_dir.strpath)
|
||||
|
||||
ci.import_signing_key(_signing_key())
|
||||
|
||||
|
||||
@ -25,25 +25,6 @@
|
||||
pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tmp_scope():
|
||||
"""Creates a temporary configuration scope"""
|
||||
|
||||
base_name = "internal-testing-scope"
|
||||
current_overrides = set(
|
||||
x.name for x in spack.config.config.matching_scopes(r"^{0}".format(base_name))
|
||||
)
|
||||
|
||||
num_overrides = 0
|
||||
scope_name = base_name
|
||||
while scope_name in current_overrides:
|
||||
scope_name = "{0}{1}".format(base_name, num_overrides)
|
||||
num_overrides += 1
|
||||
|
||||
with spack.config.override(spack.config.InternalConfigScope(scope_name)):
|
||||
yield scope_name
|
||||
|
||||
|
||||
# test gpg command detection
|
||||
@pytest.mark.parametrize(
|
||||
"cmd_name,version",
|
||||
@ -81,7 +62,7 @@ def test_no_gpg_in_path(tmpdir, mock_gnupghome, monkeypatch, mutable_config):
|
||||
|
||||
|
||||
@pytest.mark.maybeslow
|
||||
def test_gpg(tmpdir, tmp_scope, mock_gnupghome):
|
||||
def test_gpg(tmpdir, mutable_config, mock_gnupghome):
|
||||
# Verify a file with an empty keyring.
|
||||
with pytest.raises(ProcessError):
|
||||
gpg("verify", os.path.join(mock_gpg_data_path, "content.txt"))
|
||||
@ -211,6 +192,6 @@ def test_gpg(tmpdir, tmp_scope, mock_gnupghome):
|
||||
test_path = tmpdir.join("named_cache")
|
||||
os.makedirs("%s" % test_path)
|
||||
mirror_url = "file://%s" % test_path
|
||||
mirror("add", "--scope", tmp_scope, "gpg", mirror_url)
|
||||
mirror("add", "gpg", mirror_url)
|
||||
gpg("publish", "--rebuild-index", "-m", "gpg")
|
||||
assert os.path.exists("%s/build_cache/_pgp/index.json" % test_path)
|
||||
|
||||
@ -26,25 +26,6 @@
|
||||
pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def tmp_scope():
|
||||
"""Creates a temporary configuration scope"""
|
||||
|
||||
base_name = "internal-testing-scope"
|
||||
current_overrides = set(
|
||||
x.name for x in spack.config.config.matching_scopes(r"^{0}".format(base_name))
|
||||
)
|
||||
|
||||
num_overrides = 0
|
||||
scope_name = base_name
|
||||
while scope_name in current_overrides:
|
||||
scope_name = "{0}{1}".format(base_name, num_overrides)
|
||||
num_overrides += 1
|
||||
|
||||
with spack.config.override(spack.config.InternalConfigScope(scope_name)):
|
||||
yield scope_name
|
||||
|
||||
|
||||
@pytest.mark.disable_clean_stage_check
|
||||
@pytest.mark.regression("8083")
|
||||
def test_regression_8083(tmpdir, capfd, mock_packages, mock_fetch, config):
|
||||
@ -154,48 +135,44 @@ def test_exclude_file(mock_packages, tmpdir, config):
|
||||
assert not any(spec.satisfies(y) for spec in mirror_specs for y in expected_exclude)
|
||||
|
||||
|
||||
def test_mirror_crud(tmp_scope, capsys):
|
||||
def test_mirror_crud(mutable_config, capsys):
|
||||
with capsys.disabled():
|
||||
mirror("add", "--scope", tmp_scope, "mirror", "http://spack.io")
|
||||
mirror("add", "mirror", "http://spack.io")
|
||||
|
||||
output = mirror("remove", "--scope", tmp_scope, "mirror")
|
||||
output = mirror("remove", "mirror")
|
||||
assert "Removed mirror" in output
|
||||
|
||||
mirror("add", "--scope", tmp_scope, "mirror", "http://spack.io")
|
||||
mirror("add", "mirror", "http://spack.io")
|
||||
|
||||
# no-op
|
||||
output = mirror("set-url", "--scope", tmp_scope, "mirror", "http://spack.io")
|
||||
output = mirror("set-url", "mirror", "http://spack.io")
|
||||
assert "No changes made" in output
|
||||
|
||||
output = mirror("set-url", "--scope", tmp_scope, "--push", "mirror", "s3://spack-public")
|
||||
output = mirror("set-url", "--push", "mirror", "s3://spack-public")
|
||||
assert "Changed (push) url" in output
|
||||
|
||||
# no-op
|
||||
output = mirror("set-url", "--scope", tmp_scope, "--push", "mirror", "s3://spack-public")
|
||||
output = mirror("set-url", "--push", "mirror", "s3://spack-public")
|
||||
assert "No changes made" in output
|
||||
|
||||
output = mirror("remove", "--scope", tmp_scope, "mirror")
|
||||
output = mirror("remove", "mirror")
|
||||
assert "Removed mirror" in output
|
||||
|
||||
# Test S3 connection info token
|
||||
mirror(
|
||||
"add",
|
||||
"--scope",
|
||||
tmp_scope,
|
||||
"--s3-access-token",
|
||||
"aaaaaazzzzz",
|
||||
"mirror",
|
||||
"s3://spack-public",
|
||||
)
|
||||
|
||||
output = mirror("remove", "--scope", tmp_scope, "mirror")
|
||||
output = mirror("remove", "mirror")
|
||||
assert "Removed mirror" in output
|
||||
|
||||
# Test S3 connection info id/key
|
||||
mirror(
|
||||
"add",
|
||||
"--scope",
|
||||
tmp_scope,
|
||||
"--s3-access-key-id",
|
||||
"foo",
|
||||
"--s3-access-key-secret",
|
||||
@ -204,14 +181,12 @@ def test_mirror_crud(tmp_scope, capsys):
|
||||
"s3://spack-public",
|
||||
)
|
||||
|
||||
output = mirror("remove", "--scope", tmp_scope, "mirror")
|
||||
output = mirror("remove", "mirror")
|
||||
assert "Removed mirror" in output
|
||||
|
||||
# Test S3 connection info with endpoint URL
|
||||
mirror(
|
||||
"add",
|
||||
"--scope",
|
||||
tmp_scope,
|
||||
"--s3-access-token",
|
||||
"aaaaaazzzzz",
|
||||
"--s3-endpoint-url",
|
||||
@ -220,32 +195,32 @@ def test_mirror_crud(tmp_scope, capsys):
|
||||
"s3://spack-public",
|
||||
)
|
||||
|
||||
output = mirror("remove", "--scope", tmp_scope, "mirror")
|
||||
output = mirror("remove", "mirror")
|
||||
assert "Removed mirror" in output
|
||||
|
||||
output = mirror("list", "--scope", tmp_scope)
|
||||
output = mirror("list")
|
||||
assert "No mirrors configured" in output
|
||||
|
||||
# Test GCS Mirror
|
||||
mirror("add", "--scope", tmp_scope, "mirror", "gs://spack-test")
|
||||
mirror("add", "mirror", "gs://spack-test")
|
||||
|
||||
output = mirror("remove", "--scope", tmp_scope, "mirror")
|
||||
output = mirror("remove", "mirror")
|
||||
assert "Removed mirror" in output
|
||||
|
||||
|
||||
def test_mirror_nonexisting(tmp_scope):
|
||||
def test_mirror_nonexisting(mutable_config):
|
||||
with pytest.raises(SpackCommandError):
|
||||
mirror("remove", "--scope", tmp_scope, "not-a-mirror")
|
||||
mirror("remove", "not-a-mirror")
|
||||
|
||||
with pytest.raises(SpackCommandError):
|
||||
mirror("set-url", "--scope", tmp_scope, "not-a-mirror", "http://spack.io")
|
||||
mirror("set-url", "not-a-mirror", "http://spack.io")
|
||||
|
||||
|
||||
def test_mirror_name_collision(tmp_scope):
|
||||
mirror("add", "--scope", tmp_scope, "first", "1")
|
||||
def test_mirror_name_collision(mutable_config):
|
||||
mirror("add", "first", "1")
|
||||
|
||||
with pytest.raises(SpackCommandError):
|
||||
mirror("add", "--scope", tmp_scope, "first", "1")
|
||||
mirror("add", "first", "1")
|
||||
|
||||
|
||||
def test_mirror_destroy(
|
||||
|
||||
@ -71,6 +71,20 @@ def file_url_string_to_path(url):
|
||||
return urllib.request.url2pathname(urllib.parse.urlparse(url).path)
|
||||
|
||||
|
||||
def is_path_instead_of_url(path_or_url):
|
||||
"""Historically some config files and spack commands used paths
|
||||
where urls should be used. This utility can be used to validate
|
||||
and promote paths to urls."""
|
||||
scheme = urllib.parse.urlparse(path_or_url).scheme
|
||||
|
||||
# On non-Windows, no scheme means it's likely a path
|
||||
if not sys.platform == "win32":
|
||||
return not scheme
|
||||
|
||||
# On Windows, we may have drive letters.
|
||||
return "A" <= scheme <= "Z"
|
||||
|
||||
|
||||
def format(parsed_url):
|
||||
"""Format a URL string
|
||||
|
||||
|
||||
@ -500,7 +500,7 @@ _spack_buildcache_create() {
|
||||
then
|
||||
SPACK_COMPREPLY="-h --help -r --rel -f --force -u --unsigned -a --allow-root -k --key -d --directory -m --mirror-name --mirror-url --rebuild-index --spec-file --only"
|
||||
else
|
||||
_all_packages
|
||||
_mirrors
|
||||
fi
|
||||
}
|
||||
|
||||
@ -552,11 +552,21 @@ _spack_buildcache_save_specfile() {
|
||||
}
|
||||
|
||||
_spack_buildcache_sync() {
|
||||
SPACK_COMPREPLY="-h --help --manifest-glob --src-directory --src-mirror-name --src-mirror-url --dest-directory --dest-mirror-name --dest-mirror-url"
|
||||
if $list_options
|
||||
then
|
||||
SPACK_COMPREPLY="-h --help --manifest-glob --src-directory --src-mirror-name --src-mirror-url --dest-directory --dest-mirror-name --dest-mirror-url"
|
||||
else
|
||||
SPACK_COMPREPLY=""
|
||||
fi
|
||||
}
|
||||
|
||||
_spack_buildcache_update_index() {
|
||||
SPACK_COMPREPLY="-h --help -d --directory -m --mirror-name --mirror-url -k --keys"
|
||||
if $list_options
|
||||
then
|
||||
SPACK_COMPREPLY="-h --help -d --directory -m --mirror-name --mirror-url -k --keys"
|
||||
else
|
||||
_mirrors
|
||||
fi
|
||||
}
|
||||
|
||||
_spack_cd() {
|
||||
|
||||
Loading…
Reference in New Issue
Block a user