zhangyiss/spack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refer to mirrors by name, path, or url (#34891)

Browse Source 
With this change we get the invariant that `mirror.fetch_url` and
`mirror.push_url` return valid URLs, even when the backing config
file is actually using (relative) paths with potentially `$spack` and
`$env` like variables.

Secondly it avoids expanding mirror path / URLs too early,
so if I say `spack mirror add name ./path`, it stays `./path` in my
config. When it's retrieved through MirrorCollection() we
exand it to say `file://<env dir>/path` if `./path` was set in an
environment scope.

Thirdly, the interface is simplified for the relevant buildcache
commands, so it's more like `git push`:

```
spack buildcache create [mirror] [specs...]
```

`mirror` is either a mirror name, a path, or a URL.

Resolving the relevant mirror goes as follows:
    
- If it contains either / or \ it is used as an anonymous mirror with
   path or url.
- Otherwise, it's interpreted as a named mirror, which must exist.

This helps to guard against typos, e.g. typing `my-mirror` when there
is no such named mirror now errors with:

```
$ spack -e . buildcache create my-mirror
==> Error: no mirror named "my-mirror". Did you mean ./my-mirror?
```

instead of creating a directory in the current working directory. I
think this is reasonable, as the alternative (requiring that a local dir
exists) feels a bit pendantic in the general case -- spack is happy to
create the build cache dir when needed, saving a `mkdir`.

The old (now deprecated) format will still be available in Spack 0.20,
but is scheduled to be removed in 0.21:

```
spack buildcache create (--directory | --mirror-url | --mirror-name) [specs...]
```

This PR also touches `tmp_scope` in tests, because it didn't really
work for me, since spack fixes the possible --scope values once and
for all across tests, so tests failed when run out of order.
This commit is contained in:
Harmen Stoppels 2023-01-16 19:14:41 +01:00 committed by GitHub
parent 9a25e21da8
commit 3489cc0a9b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
17 changed files with 349 additions and 312 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

110
lib/spack/docs/binary_caches.rst
View File

@ -13,49 +13,51 @@ Some sites may encourage users to set up their own test environments
before carrying out central installations, or some users may prefer to set before carrying out central installations, or some users may prefer to set
up these environments on their own motivation. To reduce the load of up these environments on their own motivation. To reduce the load of
recompiling otherwise identical package specs in different installations, recompiling otherwise identical package specs in different installations,
installed packages can be put into build cache tarballs, uploaded to installed packages can be put into build cache tarballs, pushed to
your Spack mirror and then downloaded and installed by others. your Spack mirror and then downloaded and installed by others.
Whenever a mirror provides prebuilt packages, Spack will take these packages
into account during concretization and installation, making ``spack install``
signficantly faster.
--------------------------
Creating build cache files
--------------------------
A compressed tarball of an installed package is created. Tarballs are created .. note::
for all of its link and run dependency packages as well. Compressed tarballs are
signed with gpg and signature and tarball and put in a ``.spack`` file. Optionally, We use the terms "build cache" and "mirror" often interchangeably. Mirrors
the rpaths (and ids and deps on macOS) can be changed to paths relative to are used during installation both for sources and prebuilt packages. Build
the Spack install tree before the tarball is created. caches refer to mirrors that provide prebuilt packages.
----------------------
Creating a build cache
----------------------
Build caches are created via: Build caches are created via:
.. code-block:: console .. code-block:: console
$ spack buildcache create <spec> $ spack buildcache create <path/url/mirror name> <spec>
This command takes the locally installed spec and its dependencies, and
creates tarballs of their install prefixes. It also generates metadata files,
signed with GPG. These tarballs and metadata files are then pushed to the
provided binary cache, which can be a local directory or a remote URL.
If you wanted to create a build cache in a local directory, you would provide Here is an example where a build cache is created in a local directory named
the ``-d`` argument to target that directory, again also specifying the spec. "spack-cache", to which we push the "ninja" spec:
Here is an example creating a local directory, "spack-cache" and creating
build cache files for the "ninja" spec:
.. code-block:: console .. code-block:: console
$ mkdir -p ./spack-cache $ spack buildcache create --allow-root ./spack-cache ninja
$ spack buildcache create -d ./spack-cache ninja ==> Pushing binary packages to file:///home/spackuser/spack/spack-cache/build_cache
==> Buildcache files will be output to file:///home/spackuser/spack/spack-cache/build_cache
gpgconf: socketdir is '/run/user/1000/gnupg'
gpg: using "E6DF6A8BD43208E4D6F392F23777740B7DBD643D" as default secret key for signing
Note that the targeted spec must already be installed. Once you have a build cache, Not that ``ninja`` must be installed locally for this to work.
you can add it as a mirror, discussed next.
.. warning:: We're using the ``--allow-root`` flag to tell Spack that is OK when any of
the binaries we're pushing contain references to the local Spack install
directory.
Spack improved the format used for binary caches in v0.18. The entire v0.18 series Once you have a build cache, you can add it as a mirror, discussed next.
will be able to verify and install binary caches both in the new and in the old format.
Support for using the old format is expected to end in v0.19, so we advise users to
recreate relevant buildcaches using Spack v0.18 or higher.
--------------------------------------- ---------------------------------------
Finding or installing build cache files Finding or installing build cache files
@ -66,10 +68,10 @@ with:
.. code-block:: console .. code-block:: console
$ spack mirror add <name> <url> $ spack mirror add <name> <url or path>
Note that the url can be a web url _or_ a local filesystem location. In the previous Both web URLs and local paths on the filesystem can be specified. In the previous
example, you might add the directory "spack-cache" and call it ``mymirror``: example, you might add the directory "spack-cache" and call it ``mymirror``:
@ -94,7 +96,7 @@ this new build cache as follows:
.. code-block:: console .. code-block:: console
$ spack buildcache update-index -d spack-cache/ $ spack buildcache update-index ./spack-cache
Now you can use list: Now you can use list:
@ -105,46 +107,38 @@ Now you can use list:
-- linux-ubuntu20.04-skylake / gcc@9.3.0 ------------------------ -- linux-ubuntu20.04-skylake / gcc@9.3.0 ------------------------
ninja@1.10.2 ninja@1.10.2
With ``mymirror`` configured and an index available, Spack will automatically
Great! So now let's say you have a different spack installation, or perhaps just use it during concretization and installation. That means that you can expect
a different environment for the same one, and you want to install a package from ``spack install ninja`` to fetch prebuilt packages from the mirror. Let's
that build cache. Let's first uninstall the actual library "ninja" to see if we can verify by re-installing ninja:
re-install it from the cache.
.. code-block:: console .. code-block:: console
$ spack uninstall ninja $ spack uninstall ninja
$ spack install ninja
==> Installing ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz
And now reinstall from the buildcache ==> Fetching file:///home/spackuser/spack/spack-cache/build_cache/linux-ubuntu20.04-skylake-gcc-9.3.0-ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spec.json.sig
gpg: Signature made Do 12 Jan 2023 16:01:04 CET
.. code-block:: console gpg: using RSA key 61B82B2B2350E171BD17A1744E3A689061D57BF6
gpg: Good signature from "example (GPG created for Spack) <example@example.com>" [ultimate]
$ spack buildcache install ninja ==> Fetching file:///home/spackuser/spack/spack-cache/build_cache/linux-ubuntu20.04-skylake/gcc-9.3.0/ninja-1.10.2/linux-ubuntu20.04-skylake-gcc-9.3.0-ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz.spack
==> buildcache spec(s) matching ninja ==> Extracting ninja-1.10.2-yxferyhmrjkosgta5ei6b4lqf6bxbscz from binary cache
==> Fetching file:///home/spackuser/spack/spack-cache/build_cache/linux-ubuntu20.04-skylake/gcc-9.3.0/ninja-1.10.2/linux-ubuntu20.04-skylake-gcc-9.3.0-ninja-1.10.2-i4e5luour7jxdpc3bkiykd4imke3mkym.spack ==> ninja: Successfully installed ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz
####################################################################################################################################### 100.0% Search: 0.00s. Fetch: 0.17s. Install: 0.12s. Total: 0.29s
==> Installing buildcache for spec ninja@1.10.2%gcc@9.3.0 arch=linux-ubuntu20.04-skylake [+] /home/harmen/spack/opt/spack/linux-ubuntu20.04-skylake/gcc-9.3.0/ninja-1.11.1-yxferyhmrjkosgta5ei6b4lqf6bxbscz
gpgconf: socketdir is '/run/user/1000/gnupg'
gpg: Signature made Tue 23 Mar 2021 10:16:29 PM MDT
gpg: using RSA key E6DF6A8BD43208E4D6F392F23777740B7DBD643D
gpg: Good signature from "spackuser (GPG created for Spack) <spackuser@noreply.users.github.com>" [ultimate]
It worked! You've just completed a full example of creating a build cache with It worked! You've just completed a full example of creating a build cache with
a spec of interest, adding it as a mirror, updating it's index, listing the contents, a spec of interest, adding it as a mirror, updating its index, listing the contents,
and finally, installing from it. and finally, installing from it.
By default Spack falls back to building from sources when the mirror is not available
Note that the above command is intended to install a particular package to a or when the package is simply not already available. To force Spack to only install
build cache you have created, and not to install a package from a build cache. prebuilt packages, you can use
For the latter, once a mirror is added, by default when you do ``spack install`` the ``--use-cache``
flag is set, and you will install a package from a build cache if it is available.
If you want to always use the cache, you can do:
.. code-block:: console .. code-block:: console
$ spack install --cache-only <package> $ spack install --use-buildcache only <package>
For example, to combine all of the commands above to add the E4S build cache For example, to combine all of the commands above to add the E4S build cache
and then install from it exclusively, you would do: and then install from it exclusively, you would do:
@ -153,7 +147,7 @@ and then install from it exclusively, you would do:
$ spack mirror add E4S https://cache.e4s.io $ spack mirror add E4S https://cache.e4s.io
$ spack buildcache keys --install --trust $ spack buildcache keys --install --trust
$ spack install --cache-only <package> $ spack install --use-buildache only <package>
We use ``--install`` and ``--trust`` to say that we are installing keys to our We use ``--install`` and ``--trust`` to say that we are installing keys to our
keyring, and trusting all downloaded keys. keyring, and trusting all downloaded keys.

23
lib/spack/spack/bootstrap/core.py
View File

@ -94,22 +94,15 @@ class Bootstrapper:
def __init__(self, conf): def __init__(self, conf):
self.conf = conf self.conf = conf
self.name = conf["name"] self.name = conf["name"]
self.url = conf["info"]["url"]
self.metadata_dir = spack.util.path.canonicalize_path(conf["metadata"]) self.metadata_dir = spack.util.path.canonicalize_path(conf["metadata"])
@property # Promote (relative) paths to file urls
def mirror_url(self): url = conf["info"]["url"]
"""Mirror url associated with this bootstrapper""" if spack.util.url.is_path_instead_of_url(url):
# Absolute paths if not os.path.isabs(url):
if os.path.isabs(self.url): url = os.path.join(self.metadata_dir, url)
return spack.util.url.format(self.url) url = spack.util.url.path_to_file_url(url)
self.url = url
# Check for :// and assume it's an url if we find it
if "://" in self.url:
return self.url
# Otherwise, it's a relative path
return spack.util.url.format(os.path.join(self.metadata_dir, self.url))
@property @property
def mirror_scope(self): def mirror_scope(self):
@ -117,7 +110,7 @@ def mirror_scope(self):
this bootstrapper. this bootstrapper.
""" """
return spack.config.InternalConfigScope( return spack.config.InternalConfigScope(
self.config_scope_name, {"mirrors:": {self.name: self.mirror_url}} self.config_scope_name, {"mirrors:": {self.name: self.url}}
) )
def try_import(self, module: str, abstract_spec_str: str) -> bool: def try_import(self, module: str, abstract_spec_str: str) -> bool:

2
lib/spack/spack/ci.py
View File

@ -1482,7 +1482,7 @@ def _push_mirror_contents(env, specfile_path, sign_binaries, mirror_url):
tty.debug("Creating buildcache ({0})".format("unsigned" if unsigned else "signed")) tty.debug("Creating buildcache ({0})".format("unsigned" if unsigned else "signed"))
hashes = env.all_hashes() if env else None hashes = env.all_hashes() if env else None
matches = spack.store.specfile_matches(specfile_path, hashes=hashes) matches = spack.store.specfile_matches(specfile_path, hashes=hashes)
push_url = spack.mirror.push_url_from_mirror_url(mirror_url) push_url = spack.mirror.Mirror.from_url(mirror_url).push_url
spec_kwargs = {"include_root": True, "include_dependencies": False} spec_kwargs = {"include_root": True, "include_dependencies": False}
kwargs = {"force": True, "allow_root": True, "unsigned": unsigned} kwargs = {"force": True, "allow_root": True, "unsigned": unsigned}
bindist.push(matches, push_url, spec_kwargs, **kwargs) bindist.push(matches, push_url, spec_kwargs, **kwargs)

8
lib/spack/spack/cmd/bootstrap.py
View File

@ -43,6 +43,8 @@
"The sha256 checksum of binaries is checked before installation." "The sha256 checksum of binaries is checked before installation."
), ),
"info": { "info": {
# This is a mis-nomer since it's not a URL; but file urls cannot
# represent relative paths, so we have to live with it for now.
"url": os.path.join("..", "..", LOCAL_MIRROR_DIR), "url": os.path.join("..", "..", LOCAL_MIRROR_DIR),
"homepage": "https://github.com/spack/spack-bootstrap-mirrors", "homepage": "https://github.com/spack/spack-bootstrap-mirrors",
"releases": "https://github.com/spack/spack-bootstrap-mirrors/releases", "releases": "https://github.com/spack/spack-bootstrap-mirrors/releases",
@ -58,7 +60,11 @@
SOURCE_METADATA = { SOURCE_METADATA = {
"type": "install", "type": "install",
"description": "Mirror with software needed to bootstrap Spack", "description": "Mirror with software needed to bootstrap Spack",
"info": {"url": os.path.join("..", "..", LOCAL_MIRROR_DIR)}, "info": {
# This is a mis-nomer since it's not a URL; but file urls cannot
# represent relative paths, so we have to live with it for now.
"url": os.path.join("..", "..", LOCAL_MIRROR_DIR)
},
} }

244
lib/spack/spack/cmd/buildcache.py
View File

@ -8,7 +8,6 @@
import shutil import shutil
import sys import sys
import tempfile import tempfile
import urllib.parse
import llnl.util.tty as tty import llnl.util.tty as tty
@ -66,27 +65,37 @@ def setup_parser(subparser):
create.add_argument( create.add_argument(
"-k", "--key", metavar="key", type=str, default=None, help="Key for signing." "-k", "--key", metavar="key", type=str, default=None, help="Key for signing."
) )
output = create.add_mutually_exclusive_group(required=True) output = create.add_mutually_exclusive_group(required=False)
# TODO: remove from Spack 0.21
output.add_argument( output.add_argument(
"-d", "-d",
"--directory", "--directory",
metavar="directory", metavar="directory",
type=str, dest="mirror_flag",
help="local directory where buildcaches will be written.", type=arguments.mirror_directory,
help="local directory where buildcaches will be written. (deprecated)",
) )
# TODO: remove from Spack 0.21
output.add_argument( output.add_argument(
"-m", "-m",
"--mirror-name", "--mirror-name",
metavar="mirror-name", metavar="mirror-name",
type=str, dest="mirror_flag",
help="name of the mirror where buildcaches will be written.", type=arguments.mirror_name,
help="name of the mirror where buildcaches will be written. (deprecated)",
) )
# TODO: remove from Spack 0.21
output.add_argument( output.add_argument(
"--mirror-url", "--mirror-url",
metavar="mirror-url", metavar="mirror-url",
type=str, dest="mirror_flag",
help="URL of the mirror where buildcaches will be written.", type=arguments.mirror_url,
help="URL of the mirror where buildcaches will be written. (deprecated)",
) )
# Unfortunately we cannot add this to the mutually exclusive group above,
# because we have further positional arguments.
# TODO: require from Spack 0.21
create.add_argument("mirror", type=str, help="Mirror name, path, or URL.", nargs="?")
create.add_argument( create.add_argument(
"--rebuild-index", "--rebuild-index",
action="store_true", action="store_true",
@ -179,7 +188,7 @@ def setup_parser(subparser):
"-m", "-m",
"--mirror-url", "--mirror-url",
default=None, default=None,
help="Override any configured mirrors with this mirror url", help="Override any configured mirrors with this mirror URL",
) )
check.add_argument( check.add_argument(
@ -266,55 +275,108 @@ def setup_parser(subparser):
help="A quoted glob pattern identifying copy manifest files", help="A quoted glob pattern identifying copy manifest files",
) )
source = sync.add_mutually_exclusive_group(required=False) source = sync.add_mutually_exclusive_group(required=False)
# TODO: remove in Spack 0.21
source.add_argument( source.add_argument(
"--src-directory", metavar="DIRECTORY", type=str, help="Source mirror as a local file path" "--src-directory",
metavar="DIRECTORY",
dest="src_mirror_flag",
type=arguments.mirror_directory,
help="Source mirror as a local file path (deprecated)",
) )
# TODO: remove in Spack 0.21
source.add_argument( source.add_argument(
"--src-mirror-name", metavar="MIRROR_NAME", type=str, help="Name of the source mirror" "--src-mirror-name",
metavar="MIRROR_NAME",
dest="src_mirror_flag",
type=arguments.mirror_name,
help="Name of the source mirror (deprecated)",
) )
# TODO: remove in Spack 0.21
source.add_argument( source.add_argument(
"--src-mirror-url", metavar="MIRROR_URL", type=str, help="URL of the source mirror" "--src-mirror-url",
metavar="MIRROR_URL",
dest="src_mirror_flag",
type=arguments.mirror_url,
help="URL of the source mirror (deprecated)",
)
# TODO: only support this in 0.21
source.add_argument(
"src_mirror",
metavar="source mirror",
type=arguments.mirror_name_or_url,
help="Source mirror name, path, or URL",
nargs="?",
) )
dest = sync.add_mutually_exclusive_group(required=False) dest = sync.add_mutually_exclusive_group(required=False)
# TODO: remove in Spack 0.21
dest.add_argument( dest.add_argument(
"--dest-directory", "--dest-directory",
metavar="DIRECTORY", metavar="DIRECTORY",
type=str, dest="dest_mirror_flag",
help="Destination mirror as a local file path", type=arguments.mirror_directory,
help="Destination mirror as a local file path (deprecated)",
) )
# TODO: remove in Spack 0.21
dest.add_argument( dest.add_argument(
"--dest-mirror-name", "--dest-mirror-name",
metavar="MIRROR_NAME", metavar="MIRROR_NAME",
type=str, type=arguments.mirror_name,
help="Name of the destination mirror", dest="dest_mirror_flag",
help="Name of the destination mirror (deprecated)",
) )
# TODO: remove in Spack 0.21
dest.add_argument( dest.add_argument(
"--dest-mirror-url", metavar="MIRROR_URL", type=str, help="URL of the destination mirror" "--dest-mirror-url",
metavar="MIRROR_URL",
dest="dest_mirror_flag",
type=arguments.mirror_url,
help="URL of the destination mirror (deprecated)",
)
# TODO: only support this in 0.21
dest.add_argument(
"dest_mirror",
metavar="destination mirror",
type=arguments.mirror_name_or_url,
help="Destination mirror name, path, or URL",
nargs="?",
) )
sync.set_defaults(func=sync_fn) sync.set_defaults(func=sync_fn)
# Update buildcache index without copying any additional packages # Update buildcache index without copying any additional packages
update_index = subparsers.add_parser("update-index", help=update_index_fn.__doc__) update_index = subparsers.add_parser("update-index", help=update_index_fn.__doc__)
update_index_out = update_index.add_mutually_exclusive_group(required=True) update_index_out = update_index.add_mutually_exclusive_group(required=True)
# TODO: remove in Spack 0.21
update_index_out.add_argument( update_index_out.add_argument(
"-d", "-d",
"--directory", "--directory",
metavar="directory", metavar="directory",
type=str, dest="mirror_flag",
help="local directory where buildcaches will be written.", type=arguments.mirror_directory,
help="local directory where buildcaches will be written (deprecated)",
) )
# TODO: remove in Spack 0.21
update_index_out.add_argument( update_index_out.add_argument(
"-m", "-m",
"--mirror-name", "--mirror-name",
metavar="mirror-name", metavar="mirror-name",
type=str, dest="mirror_flag",
help="name of the mirror where buildcaches will be written.", type=arguments.mirror_name,
help="name of the mirror where buildcaches will be written (deprecated)",
) )
# TODO: remove in Spack 0.21
update_index_out.add_argument( update_index_out.add_argument(
"--mirror-url", "--mirror-url",
metavar="mirror-url", metavar="mirror-url",
type=str, dest="mirror_flag",
help="URL of the mirror where buildcaches will be written.", type=arguments.mirror_url,
help="URL of the mirror where buildcaches will be written (deprecated)",
)
# TODO: require from Spack 0.21
update_index_out.add_argument(
"mirror",
type=arguments.mirror_name_or_url,
help="Destination mirror name, path, or URL",
nargs="?",
) )
update_index.add_argument( update_index.add_argument(
"-k", "-k",
@ -326,26 +388,17 @@ def setup_parser(subparser):
update_index.set_defaults(func=update_index_fn) update_index.set_defaults(func=update_index_fn)
def _mirror_url_from_args(args): def _matching_specs(specs, spec_file):
if args.directory:
return spack.mirror.push_url_from_directory(args.directory)
if args.mirror_name:
return spack.mirror.push_url_from_mirror_name(args.mirror_name)
if args.mirror_url:
return spack.mirror.push_url_from_mirror_url(args.mirror_url)
def _matching_specs(args):
"""Return a list of matching specs read from either a spec file (JSON or YAML), """Return a list of matching specs read from either a spec file (JSON or YAML),
a query over the store or a query over the active environment. a query over the store or a query over the active environment.
""" """
env = ev.active_environment() env = ev.active_environment()
hashes = env.all_hashes() if env else None hashes = env.all_hashes() if env else None
if args.spec_file: if spec_file:
return spack.store.specfile_matches(args.spec_file, hashes=hashes) return spack.store.specfile_matches(spec_file, hashes=hashes)
if args.specs: if specs:
constraints = spack.cmd.parse_specs(args.specs) constraints = spack.cmd.parse_specs(specs)
return spack.store.find(constraints, hashes=hashes) return spack.store.find(constraints, hashes=hashes)
if env: if env:
@ -383,10 +436,30 @@ def _concrete_spec_from_args(args):
def create_fn(args): def create_fn(args):
"""create a binary package and push it to a mirror""" """create a binary package and push it to a mirror"""
push_url = _mirror_url_from_args(args) if args.mirror_flag:
matches = _matching_specs(args) mirror = args.mirror_flag
elif not args.mirror:
raise ValueError("No mirror provided")
else:
mirror = arguments.mirror_name_or_url(args.mirror)
msg = "Pushing binary packages to {0}/build_cache".format(push_url) if args.mirror_flag:
tty.warn(
"Using flags to specify mirrors is deprecated and will be removed in "
"Spack 0.21, use positional arguments instead."
)
# TODO: remove this in 0.21. If we have mirror_flag, the first
# spec is in the positional mirror arg due to argparse limitations.
specs = args.specs
if args.mirror_flag and args.mirror:
specs.insert(0, args.mirror)
url = mirror.push_url
matches = _matching_specs(specs, args.spec_file)
msg = "Pushing binary packages to {0}/build_cache".format(url)
tty.msg(msg) tty.msg(msg)
specs_kwargs = { specs_kwargs = {
"include_root": "package" in args.things_to_install, "include_root": "package" in args.things_to_install,
@ -400,7 +473,7 @@ def create_fn(args):
"allow_root": args.allow_root, "allow_root": args.allow_root,
"regenerate_index": args.rebuild_index, "regenerate_index": args.rebuild_index,
} }
bindist.push(matches, push_url, specs_kwargs, **kwargs) bindist.push(matches, url, specs_kwargs, **kwargs)
def install_fn(args): def install_fn(args):
@ -593,51 +666,24 @@ def sync_fn(args):
manifest_copy(glob.glob(args.manifest_glob)) manifest_copy(glob.glob(args.manifest_glob))
return 0 return 0
# Figure out the source mirror # If no manifest_glob, require a source and dest mirror.
source_location = None # TODO: Simplify in Spack 0.21
if args.src_directory: if not (args.src_mirror_flag or args.src_mirror) or not (
source_location = args.src_directory args.dest_mirror_flag or args.dest_mirror
scheme = urllib.parse.urlparse(source_location, scheme="<missing>").scheme ):
if scheme != "<missing>": raise ValueError("Source and destination mirror are required.")
raise ValueError('"--src-directory" expected a local path; got a URL, instead')
# Ensure that the mirror lookup does not mistake this for named mirror
source_location = url_util.path_to_file_url(source_location)
elif args.src_mirror_name:
source_location = args.src_mirror_name
result = spack.mirror.MirrorCollection().lookup(source_location)
if result.name == "<unnamed>":
raise ValueError('no configured mirror named "{name}"'.format(name=source_location))
elif args.src_mirror_url:
source_location = args.src_mirror_url
scheme = urllib.parse.urlparse(source_location, scheme="<missing>").scheme
if scheme == "<missing>":
raise ValueError('"{url}" is not a valid URL'.format(url=source_location))
src_mirror = spack.mirror.MirrorCollection().lookup(source_location) if args.src_mirror_flag or args.dest_mirror_flag:
src_mirror_url = url_util.format(src_mirror.fetch_url) tty.warn(
"Using flags to specify mirrors is deprecated and will be removed in "
"Spack 0.21, use positional arguments instead."
)
# Figure out the destination mirror src_mirror = args.src_mirror_flag if args.src_mirror_flag else args.src_mirror
dest_location = None dest_mirror = args.dest_mirror_flag if args.dest_mirror_flag else args.dest_mirror
if args.dest_directory:
dest_location = args.dest_directory
scheme = urllib.parse.urlparse(dest_location, scheme="<missing>").scheme
if scheme != "<missing>":
raise ValueError('"--dest-directory" expected a local path; got a URL, instead')
# Ensure that the mirror lookup does not mistake this for named mirror
dest_location = url_util.path_to_file_url(dest_location)
elif args.dest_mirror_name:
dest_location = args.dest_mirror_name
result = spack.mirror.MirrorCollection().lookup(dest_location)
if result.name == "<unnamed>":
raise ValueError('no configured mirror named "{name}"'.format(name=dest_location))
elif args.dest_mirror_url:
dest_location = args.dest_mirror_url
scheme = urllib.parse.urlparse(dest_location, scheme="<missing>").scheme
if scheme == "<missing>":
raise ValueError('"{url}" is not a valid URL'.format(url=dest_location))
dest_mirror = spack.mirror.MirrorCollection().lookup(dest_location) src_mirror_url = src_mirror.fetch_url
dest_mirror_url = url_util.format(dest_mirror.fetch_url) dest_mirror_url = dest_mirror.push_url
# Get the active environment # Get the active environment
env = spack.cmd.require_active_env(cmd_name="buildcache sync") env = spack.cmd.require_active_env(cmd_name="buildcache sync")
@ -698,38 +744,28 @@ def manifest_copy(manifest_file_list):
copy_buildcache_file(copy_file["src"], copy_file["dest"]) copy_buildcache_file(copy_file["src"], copy_file["dest"])
def update_index(mirror_url, update_keys=False): def update_index(mirror: spack.mirror.Mirror, update_keys=False):
mirror = spack.mirror.MirrorCollection().lookup(mirror_url) url = mirror.push_url
outdir = url_util.format(mirror.push_url)
bindist.generate_package_index(url_util.join(outdir, bindist.build_cache_relative_path())) bindist.generate_package_index(url_util.join(url, bindist.build_cache_relative_path()))
if update_keys: if update_keys:
keys_url = url_util.join( keys_url = url_util.join(
outdir, bindist.build_cache_relative_path(), bindist.build_cache_keys_relative_path() url, bindist.build_cache_relative_path(), bindist.build_cache_keys_relative_path()
) )
bindist.generate_key_index(keys_url) bindist.generate_key_index(keys_url)
def _mirror_url_from_args_deprecated_format(args):
# In Spack 0.19 the -d flag was equivalent to --mirror-url.
# Spack 0.20 deprecates this, so in 0.21 -d means --directory.
if args.directory and url_util.validate_scheme(urllib.parse.urlparse(args.directory).scheme):
tty.warn(
"Passing a URL to `update-index -d <url>` is deprecated "
"and will be removed in Spack 0.21. "
"Use `update-index --mirror-url <url>` instead."
)
return spack.mirror.push_url_from_mirror_url(args.directory)
else:
return _mirror_url_from_args(args)
def update_index_fn(args): def update_index_fn(args):
"""Update a buildcache index.""" """Update a buildcache index."""
push_url = _mirror_url_from_args_deprecated_format(args) if args.mirror_flag:
update_index(push_url, update_keys=args.keys) tty.warn(
"Using flags to specify mirrors is deprecated and will be removed in "
"Spack 0.21, use positional arguments instead."
)
mirror = args.mirror_flag if args.mirror_flag else args.mirror
update_index(mirror, update_keys=args.keys)
def buildcache(parser, args): def buildcache(parser, args):

3
lib/spack/spack/cmd/ci.py
View File

@ -241,8 +241,9 @@ def ci_reindex(args):
ci_mirrors = yaml_root["mirrors"] ci_mirrors = yaml_root["mirrors"]
mirror_urls = [url for url in ci_mirrors.values()] mirror_urls = [url for url in ci_mirrors.values()]
remote_mirror_url = mirror_urls[0] remote_mirror_url = mirror_urls[0]
mirror = spack.mirror.Mirror(remote_mirror_url)
buildcache.update_index(remote_mirror_url, update_keys=True) buildcache.update_index(mirror, update_keys=True)
def ci_rebuild(args): def ci_rebuild(args):

41
lib/spack/spack/cmd/common/arguments.py
View File

@ -5,6 +5,7 @@
import argparse import argparse
import os.path
from llnl.util.lang import stable_partition from llnl.util.lang import stable_partition
@ -12,6 +13,7 @@
import spack.config import spack.config
import spack.dependency as dep import spack.dependency as dep
import spack.environment as ev import spack.environment as ev
import spack.mirror
import spack.modules import spack.modules
import spack.reporters import spack.reporters
import spack.spec import spack.spec
@ -552,3 +554,42 @@ def is_valid(arg):
dependencies = val dependencies = val
return package, dependencies return package, dependencies
def mirror_name_or_url(m):
# Look up mirror by name or use anonymous mirror with path/url.
# We want to guard against typos in mirror names, to avoid pushing
# accidentally to a dir in the current working directory.
# If there's a \ or / in the name, it's interpreted as a path or url.
if "/" in m or "\\" in m:
return spack.mirror.Mirror(m)
# Otherwise, the named mirror is required to exist.
try:
return spack.mirror.require_mirror_name(m)
except ValueError as e:
raise argparse.ArgumentTypeError(
str(e) + ". Did you mean {}?".format(os.path.join(".", m))
)
def mirror_url(url):
try:
return spack.mirror.Mirror.from_url(url)
except ValueError as e:
raise argparse.ArgumentTypeError(str(e))
def mirror_directory(path):
try:
return spack.mirror.Mirror.from_local_path(path)
except ValueError as e:
raise argparse.ArgumentTypeError(str(e))
def mirror_name(name):
try:
return spack.mirror.require_mirror_name(name)
except ValueError as e:
raise argparse.ArgumentTypeError(str(e))

5
lib/spack/spack/cmd/mirror.py
View File

@ -145,8 +145,7 @@ def setup_parser(subparser):
def mirror_add(args): def mirror_add(args):
"""Add a mirror to Spack.""" """Add a mirror to Spack."""
url = url_util.format(args.url) spack.mirror.add(args.name, args.url, args.scope, args)
spack.mirror.add(args.name, url, args.scope, args)
def mirror_remove(args): def mirror_remove(args):
@ -156,7 +155,7 @@ def mirror_remove(args):
def mirror_set_url(args): def mirror_set_url(args):
"""Change the URL of a mirror.""" """Change the URL of a mirror."""
url = url_util.format(args.url) url = args.url
mirrors = spack.config.get("mirrors", scope=args.scope) mirrors = spack.config.get("mirrors", scope=args.scope)
if not mirrors: if not mirrors:
mirrors = syaml_dict() mirrors = syaml_dict()

76
lib/spack/spack/mirror.py
View File

@ -31,15 +31,15 @@
import spack.mirror import spack.mirror
import spack.spec import spack.spec
import spack.url as url import spack.url as url
import spack.util.path
import spack.util.spack_json as sjson import spack.util.spack_json as sjson
import spack.util.spack_yaml as syaml import spack.util.spack_yaml as syaml
import spack.util.url as url_util import spack.util.url as url_util
from spack.util.spack_yaml import syaml_dict from spack.util.spack_yaml import syaml_dict
from spack.version import VersionList from spack.version import VersionList
#: What schemes do we support
def _is_string(url): supported_url_schemes = ("file", "http", "https", "sftp", "ftp", "s3", "gs")
return isinstance(url, str)
def _display_mirror_entry(size, name, url, type_=None): def _display_mirror_entry(size, name, url, type_=None):
@ -51,6 +51,19 @@ def _display_mirror_entry(size, name, url, type_=None):
print("%-*s%s%s" % (size + 4, name, url, type_)) print("%-*s%s%s" % (size + 4, name, url, type_))
def _url_or_path_to_url(url_or_path: str) -> str:
"""For simplicity we allow mirror URLs in config files to be local, relative paths.
This helper function takes care of distinguishing between URLs and paths, and
canonicalizes paths before transforming them into file:// URLs."""
# Is it a supported URL already? Then don't do path-related canonicalization.
parsed = urllib.parse.urlparse(url_or_path)
if parsed.scheme in supported_url_schemes:
return url_or_path
# Otherwise we interpret it as path, and we should promote it to file:// URL.
return url_util.path_to_file_url(spack.util.path.canonicalize_path(url_or_path))
class Mirror(object): class Mirror(object):
"""Represents a named location for storing source tarballs and binary """Represents a named location for storing source tarballs and binary
packages. packages.
@ -90,6 +103,21 @@ def from_json(stream, name=None):
except Exception as e: except Exception as e:
raise sjson.SpackJSONError("error parsing JSON mirror:", str(e)) from e raise sjson.SpackJSONError("error parsing JSON mirror:", str(e)) from e
@staticmethod
def from_local_path(path: str):
return Mirror(fetch_url=url_util.path_to_file_url(path))
@staticmethod
def from_url(url: str):
"""Create an anonymous mirror by URL. This method validates the URL."""
if not urllib.parse.urlparse(url).scheme in supported_url_schemes:
raise ValueError(
'"{}" is not a valid mirror URL. Scheme must be once of {}.'.format(
url, ", ".join(supported_url_schemes)
)
)
return Mirror(fetch_url=url)
def to_dict(self): def to_dict(self):
if self._push_url is None: if self._push_url is None:
return syaml_dict([("fetch", self._fetch_url), ("push", self._fetch_url)]) return syaml_dict([("fetch", self._fetch_url), ("push", self._fetch_url)])
@ -201,7 +229,11 @@ def set_access_token(self, url_type, connection_token):
@property @property
def fetch_url(self): def fetch_url(self):
return self._fetch_url if _is_string(self._fetch_url) else self._fetch_url["url"] """Get the valid, canonicalized fetch URL"""
url_or_path = (
self._fetch_url if isinstance(self._fetch_url, str) else self._fetch_url["url"]
)
return _url_or_path_to_url(url_or_path)
@fetch_url.setter @fetch_url.setter
def fetch_url(self, url): def fetch_url(self, url):
@ -210,9 +242,12 @@ def fetch_url(self, url):
@property @property
def push_url(self): def push_url(self):
"""Get the valid, canonicalized push URL. Returns fetch URL if no custom
push URL is defined"""
if self._push_url is None: if self._push_url is None:
return self._fetch_url if _is_string(self._fetch_url) else self._fetch_url["url"] return self.fetch_url
return self._push_url if _is_string(self._push_url) else self._push_url["url"] url_or_path = self._push_url if isinstance(self._push_url, str) else self._push_url["url"]
return _url_or_path_to_url(url_or_path)
@push_url.setter @push_url.setter
def push_url(self, url): def push_url(self, url):
@ -663,31 +698,12 @@ def create_mirror_from_package_object(pkg_obj, mirror_cache, mirror_stats):
return True return True
def push_url_from_directory(output_directory): def require_mirror_name(mirror_name):
"""Given a directory in the local filesystem, return the URL on """Find a mirror by name and raise if it does not exist"""
which to push binary packages. mirror = spack.mirror.MirrorCollection().get(mirror_name)
""" if not mirror:
if url_util.validate_scheme(urllib.parse.urlparse(output_directory).scheme):
raise ValueError("expected a local path, but got a URL instead")
mirror_url = url_util.path_to_file_url(output_directory)
mirror = spack.mirror.MirrorCollection().lookup(mirror_url)
return url_util.format(mirror.push_url)
def push_url_from_mirror_name(mirror_name):
"""Given a mirror name, return the URL on which to push binary packages."""
mirror = spack.mirror.MirrorCollection().lookup(mirror_name)
if mirror.name == "<unnamed>":
raise ValueError('no mirror named "{0}"'.format(mirror_name)) raise ValueError('no mirror named "{0}"'.format(mirror_name))
return url_util.format(mirror.push_url) return mirror
def push_url_from_mirror_url(mirror_url):
"""Given a mirror URL, return the URL on which to push binary packages."""
if not url_util.validate_scheme(urllib.parse.urlparse(mirror_url).scheme):
raise ValueError('"{0}" is not a valid URL'.format(mirror_url))
mirror = spack.mirror.MirrorCollection().lookup(mirror_url)
return url_util.format(mirror.push_url)
class MirrorError(spack.error.SpackError): class MirrorError(spack.error.SpackError):

2
lib/spack/spack/schema/mirrors.py
View File

@ -6,10 +6,8 @@
"""Schema for mirrors.yaml configuration file. """Schema for mirrors.yaml configuration file.
.. literalinclude:: _spack_root/lib/spack/spack/schema/mirrors.py .. literalinclude:: _spack_root/lib/spack/spack/schema/mirrors.py
:lines: 13-
""" """
#: Properties for inclusion in other schemas #: Properties for inclusion in other schemas
properties = { properties = {
"mirrors": { "mirrors": {

16
lib/spack/spack/test/ci.py
View File

@ -23,22 +23,6 @@
import spack.util.spack_yaml as syaml import spack.util.spack_yaml as syaml
@pytest.fixture
def tmp_scope():
"""Creates a temporary configuration scope"""
base_name = "internal-testing-scope"
current_overrides = set(x.name for x in cfg.config.matching_scopes(r"^{0}".format(base_name)))
num_overrides = 0
scope_name = base_name
while scope_name in current_overrides:
scope_name = "{0}{1}".format(base_name, num_overrides)
num_overrides += 1
with cfg.override(cfg.InternalConfigScope(scope_name)):
yield scope_name
def test_urlencode_string(): def test_urlencode_string():
s = "Spack Test Project" s = "Spack Test Project"

11
lib/spack/spack/test/cmd/buildcache.py
View File

@ -3,7 +3,6 @@
# #
# SPDX-License-Identifier: (Apache-2.0 OR MIT) # SPDX-License-Identifier: (Apache-2.0 OR MIT)
import argparse
import errno import errno
import os import os
import platform import platform
@ -268,13 +267,3 @@ def test_buildcache_create_install(
tarball = spack.binary_distribution.tarball_name(spec, ".spec.json") tarball = spack.binary_distribution.tarball_name(spec, ".spec.json")
assert os.path.exists(os.path.join(str(tmpdir), "build_cache", tarball_path)) assert os.path.exists(os.path.join(str(tmpdir), "build_cache", tarball_path))
assert os.path.exists(os.path.join(str(tmpdir), "build_cache", tarball)) assert os.path.exists(os.path.join(str(tmpdir), "build_cache", tarball))
def test_deprecation_mirror_url_dir_flag(capfd):
# Test that passing `update-index -d <url>` gives a deprecation warning.
parser = argparse.ArgumentParser()
spack.cmd.buildcache.setup_parser(parser)
url = spack.util.url.path_to_file_url(os.getcwd())
args = parser.parse_args(["update-index", "-d", url])
spack.cmd.buildcache._mirror_url_from_args_deprecated_format(args)
assert "Passing a URL to `update-index -d <url>` is deprecated" in capfd.readouterr()[1]

2
lib/spack/spack/test/cmd/ci.py
View File

@ -1218,7 +1218,7 @@ def test_push_mirror_contents(
working_dir = tmpdir.join("working_dir") working_dir = tmpdir.join("working_dir")
mirror_dir = working_dir.join("mirror") mirror_dir = working_dir.join("mirror")
mirror_url = "file://{0}".format(mirror_dir.strpath) mirror_url = url_util.path_to_file_url(mirror_dir.strpath)
ci.import_signing_key(_signing_key()) ci.import_signing_key(_signing_key())

23
lib/spack/spack/test/cmd/gpg.py
View File

@ -25,25 +25,6 @@
pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows") pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
@pytest.fixture
def tmp_scope():
"""Creates a temporary configuration scope"""
base_name = "internal-testing-scope"
current_overrides = set(
x.name for x in spack.config.config.matching_scopes(r"^{0}".format(base_name))
)
num_overrides = 0
scope_name = base_name
while scope_name in current_overrides:
scope_name = "{0}{1}".format(base_name, num_overrides)
num_overrides += 1
with spack.config.override(spack.config.InternalConfigScope(scope_name)):
yield scope_name
# test gpg command detection # test gpg command detection
@pytest.mark.parametrize( @pytest.mark.parametrize(
"cmd_name,version", "cmd_name,version",
@ -81,7 +62,7 @@ def test_no_gpg_in_path(tmpdir, mock_gnupghome, monkeypatch, mutable_config):
@pytest.mark.maybeslow @pytest.mark.maybeslow
def test_gpg(tmpdir, tmp_scope, mock_gnupghome): def test_gpg(tmpdir, mutable_config, mock_gnupghome):
# Verify a file with an empty keyring. # Verify a file with an empty keyring.
with pytest.raises(ProcessError): with pytest.raises(ProcessError):
gpg("verify", os.path.join(mock_gpg_data_path, "content.txt")) gpg("verify", os.path.join(mock_gpg_data_path, "content.txt"))
@ -211,6 +192,6 @@ def test_gpg(tmpdir, tmp_scope, mock_gnupghome):
test_path = tmpdir.join("named_cache") test_path = tmpdir.join("named_cache")
os.makedirs("%s" % test_path) os.makedirs("%s" % test_path)
mirror_url = "file://%s" % test_path mirror_url = "file://%s" % test_path
mirror("add", "--scope", tmp_scope, "gpg", mirror_url) mirror("add", "gpg", mirror_url)
gpg("publish", "--rebuild-index", "-m", "gpg") gpg("publish", "--rebuild-index", "-m", "gpg")
assert os.path.exists("%s/build_cache/_pgp/index.json" % test_path) assert os.path.exists("%s/build_cache/_pgp/index.json" % test_path)

65
lib/spack/spack/test/cmd/mirror.py
View File

@ -26,25 +26,6 @@
pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows") pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
@pytest.fixture
def tmp_scope():
"""Creates a temporary configuration scope"""
base_name = "internal-testing-scope"
current_overrides = set(
x.name for x in spack.config.config.matching_scopes(r"^{0}".format(base_name))
)
num_overrides = 0
scope_name = base_name
while scope_name in current_overrides:
scope_name = "{0}{1}".format(base_name, num_overrides)
num_overrides += 1
with spack.config.override(spack.config.InternalConfigScope(scope_name)):
yield scope_name
@pytest.mark.disable_clean_stage_check @pytest.mark.disable_clean_stage_check
@pytest.mark.regression("8083") @pytest.mark.regression("8083")
def test_regression_8083(tmpdir, capfd, mock_packages, mock_fetch, config): def test_regression_8083(tmpdir, capfd, mock_packages, mock_fetch, config):
@ -154,48 +135,44 @@ def test_exclude_file(mock_packages, tmpdir, config):
assert not any(spec.satisfies(y) for spec in mirror_specs for y in expected_exclude) assert not any(spec.satisfies(y) for spec in mirror_specs for y in expected_exclude)
def test_mirror_crud(tmp_scope, capsys): def test_mirror_crud(mutable_config, capsys):
with capsys.disabled(): with capsys.disabled():
mirror("add", "--scope", tmp_scope, "mirror", "http://spack.io") mirror("add", "mirror", "http://spack.io")
output = mirror("remove", "--scope", tmp_scope, "mirror") output = mirror("remove", "mirror")
assert "Removed mirror" in output assert "Removed mirror" in output
mirror("add", "--scope", tmp_scope, "mirror", "http://spack.io") mirror("add", "mirror", "http://spack.io")
# no-op # no-op
output = mirror("set-url", "--scope", tmp_scope, "mirror", "http://spack.io") output = mirror("set-url", "mirror", "http://spack.io")
assert "No changes made" in output assert "No changes made" in output
output = mirror("set-url", "--scope", tmp_scope, "--push", "mirror", "s3://spack-public") output = mirror("set-url", "--push", "mirror", "s3://spack-public")
assert "Changed (push) url" in output assert "Changed (push) url" in output
# no-op # no-op
output = mirror("set-url", "--scope", tmp_scope, "--push", "mirror", "s3://spack-public") output = mirror("set-url", "--push", "mirror", "s3://spack-public")
assert "No changes made" in output assert "No changes made" in output
output = mirror("remove", "--scope", tmp_scope, "mirror") output = mirror("remove", "mirror")
assert "Removed mirror" in output assert "Removed mirror" in output
# Test S3 connection info token # Test S3 connection info token
mirror( mirror(
"add", "add",
"--scope",
tmp_scope,
"--s3-access-token", "--s3-access-token",
"aaaaaazzzzz", "aaaaaazzzzz",
"mirror", "mirror",
"s3://spack-public", "s3://spack-public",
) )
output = mirror("remove", "--scope", tmp_scope, "mirror") output = mirror("remove", "mirror")
assert "Removed mirror" in output assert "Removed mirror" in output
# Test S3 connection info id/key # Test S3 connection info id/key
mirror( mirror(
"add", "add",
"--scope",
tmp_scope,
"--s3-access-key-id", "--s3-access-key-id",
"foo", "foo",
"--s3-access-key-secret", "--s3-access-key-secret",
@ -204,14 +181,12 @@ def test_mirror_crud(tmp_scope, capsys):
"s3://spack-public", "s3://spack-public",
) )
output = mirror("remove", "--scope", tmp_scope, "mirror") output = mirror("remove", "mirror")
assert "Removed mirror" in output assert "Removed mirror" in output
# Test S3 connection info with endpoint URL # Test S3 connection info with endpoint URL
mirror( mirror(
"add", "add",
"--scope",
tmp_scope,
"--s3-access-token", "--s3-access-token",
"aaaaaazzzzz", "aaaaaazzzzz",
"--s3-endpoint-url", "--s3-endpoint-url",
@ -220,32 +195,32 @@ def test_mirror_crud(tmp_scope, capsys):
"s3://spack-public", "s3://spack-public",
) )
output = mirror("remove", "--scope", tmp_scope, "mirror") output = mirror("remove", "mirror")
assert "Removed mirror" in output assert "Removed mirror" in output
output = mirror("list", "--scope", tmp_scope) output = mirror("list")
assert "No mirrors configured" in output assert "No mirrors configured" in output
# Test GCS Mirror # Test GCS Mirror
mirror("add", "--scope", tmp_scope, "mirror", "gs://spack-test") mirror("add", "mirror", "gs://spack-test")
output = mirror("remove", "--scope", tmp_scope, "mirror") output = mirror("remove", "mirror")
assert "Removed mirror" in output assert "Removed mirror" in output
def test_mirror_nonexisting(tmp_scope): def test_mirror_nonexisting(mutable_config):
with pytest.raises(SpackCommandError): with pytest.raises(SpackCommandError):
mirror("remove", "--scope", tmp_scope, "not-a-mirror") mirror("remove", "not-a-mirror")
with pytest.raises(SpackCommandError): with pytest.raises(SpackCommandError):
mirror("set-url", "--scope", tmp_scope, "not-a-mirror", "http://spack.io") mirror("set-url", "not-a-mirror", "http://spack.io")
def test_mirror_name_collision(tmp_scope): def test_mirror_name_collision(mutable_config):
mirror("add", "--scope", tmp_scope, "first", "1") mirror("add", "first", "1")
with pytest.raises(SpackCommandError): with pytest.raises(SpackCommandError):
mirror("add", "--scope", tmp_scope, "first", "1") mirror("add", "first", "1")
def test_mirror_destroy( def test_mirror_destroy(

14
lib/spack/spack/util/url.py
View File

@ -71,6 +71,20 @@ def file_url_string_to_path(url):
return urllib.request.url2pathname(urllib.parse.urlparse(url).path) return urllib.request.url2pathname(urllib.parse.urlparse(url).path)
def is_path_instead_of_url(path_or_url):
"""Historically some config files and spack commands used paths
where urls should be used. This utility can be used to validate
and promote paths to urls."""
scheme = urllib.parse.urlparse(path_or_url).scheme
# On non-Windows, no scheme means it's likely a path
if not sys.platform == "win32":
return not scheme
# On Windows, we may have drive letters.
return "A" <= scheme <= "Z"
def format(parsed_url): def format(parsed_url):
"""Format a URL string """Format a URL string

12
share/spack/spack-completion.bash
View File

@ -500,7 +500,7 @@ _spack_buildcache_create() {
then then
SPACK_COMPREPLY="-h --help -r --rel -f --force -u --unsigned -a --allow-root -k --key -d --directory -m --mirror-name --mirror-url --rebuild-index --spec-file --only" SPACK_COMPREPLY="-h --help -r --rel -f --force -u --unsigned -a --allow-root -k --key -d --directory -m --mirror-name --mirror-url --rebuild-index --spec-file --only"
else else
_all_packages _mirrors
fi fi
} }
@ -552,11 +552,21 @@ _spack_buildcache_save_specfile() {
} }
_spack_buildcache_sync() { _spack_buildcache_sync() {
if $list_options
then
SPACK_COMPREPLY="-h --help --manifest-glob --src-directory --src-mirror-name --src-mirror-url --dest-directory --dest-mirror-name --dest-mirror-url" SPACK_COMPREPLY="-h --help --manifest-glob --src-directory --src-mirror-name --src-mirror-url --dest-directory --dest-mirror-name --dest-mirror-url"
else
SPACK_COMPREPLY=""
fi
} }
_spack_buildcache_update_index() { _spack_buildcache_update_index() {
if $list_options
then
SPACK_COMPREPLY="-h --help -d --directory -m --mirror-name --mirror-url -k --keys" SPACK_COMPREPLY="-h --help -d --directory -m --mirror-name --mirror-url -k --keys"
else
_mirrors
fi
} }
_spack_cd() { _spack_cd() {