.. _mirrors:

Mirrors
============================

Some sites may not have access to the internet for fetching packages.
These sites will need a local repository of tarballs from which they
can get their files.  Spack has support for this with *mirrors*.  A
mirror is a URL that points to a directory, either on the local
filesystem or on some server, containing tarballs for all of Spack's
packages.

Here's an example of a mirror's directory structure::

    mirror/
        cmake/
            cmake-2.8.10.2.tar.gz
        dyninst/
            dyninst-8.1.1.tgz
            dyninst-8.1.2.tgz
        libdwarf/
            libdwarf-20130126.tar.gz
            libdwarf-20130207.tar.gz
            libdwarf-20130729.tar.gz
        libelf/
            libelf-0.8.12.tar.gz
            libelf-0.8.13.tar.gz
        libunwind/
            libunwind-1.1.tar.gz
        mpich/
            mpich-3.0.4.tar.gz
        mvapich2/
            mvapich2-1.9.tgz

The structure is very simple.  There is a top-level directory.  The
second level directories are named after packages, and the third level
contains tarballs for each package, named after each package.

.. note::

   Archives are **not** named exactly they were in the package's fetch
   URL.  They have the form ``<name>-<version>.<extension>``, where
   ``<name>`` is Spack's name for the package, ``<version>`` is the
   version of the tarball, and ``<extension>`` is whatever format the
   package's fetch URL contains.

   In order to make mirror creation reasonably fast, we copy the
   tarball in its original format to the mirror directory, but we do
   not standardize on a particular compression algorithm, because this
   would potentially require expanding and re-compressing each archive.

.. _spack-mirror:

``spack mirror``
----------------------------

Mirrors are managed with the ``spack mirror`` command.  The help for
``spack mirror`` looks like this::

    $ spack mirror -h
    usage: spack mirror [-h] SUBCOMMAND ...

    positional arguments:
      SUBCOMMAND
        create           Create a directory to be used as a spack mirror, and fill
                         it with package archives.
        add              Add a mirror to Spack.
        remove           Remove a mirror by name.
        list             Print out available mirrors to the console.

    optional arguments:
      -h, --help         show this help message and exit

The ``create`` command actually builds a mirror by fetching all of its
packages from the internet and checksumming them.

The other three commands are for managing mirror configuration.  They
control the URL(s) from which Spack downloads its packages.

.. _spack-mirror-create:

``spack mirror create``
----------------------------

You can create a mirror using the ``spack mirror create`` command, assuming
you're on a machine where you can access the internet.

The command will iterate through all of Spack's packages and download
the safe ones into a directory structure like the one above.  Here is
what it looks like:


.. code-block:: bash

   $ spack mirror create libelf libdwarf
   ==> Created new mirror in spack-mirror-2014-06-24
   ==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.13.tar.gz
   ##########################################################                81.6%
   ==> Checksum passed for libelf@0.8.13
   ==> Added libelf@0.8.13
   ==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.12.tar.gz
   ######################################################################    98.6%
   ==> Checksum passed for libelf@0.8.12
   ==> Added libelf@0.8.12
   ==> Trying to fetch from http://www.prevanders.net/libdwarf-20130207.tar.gz
   ######################################################################    97.3%
   ==> Checksum passed for libdwarf@20130207
   ==> Added libdwarf@20130207
   ==> Trying to fetch from http://www.prevanders.net/libdwarf-20130126.tar.gz
   ########################################################                  78.9%
   ==> Checksum passed for libdwarf@20130126
   ==> Added libdwarf@20130126
   ==> Trying to fetch from http://www.prevanders.net/libdwarf-20130729.tar.gz
   #############################################################             84.7%
   ==> Added libdwarf@20130729
   ==> Added spack-mirror-2014-06-24/libdwarf/libdwarf-20130729.tar.gz to mirror
   ==> Added python@2.7.8.
   ==> Successfully updated mirror in spack-mirror-2015-02-24.
     Archive stats:
       0    already present
       5    added
       0    failed to fetch.

Once this is done, you can tar up the ``spack-mirror-2014-06-24`` directory and
copy it over to the machine you want it hosted on.

Custom package sets
~~~~~~~~~~~~~~~~~~~~~~~

Normally, ``spack mirror create`` downloads all the archives it has
checksums for.  If you want to only create a mirror for a subset of
packages, you can do that by supplying a list of package specs on the
command line after ``spack mirror create``.  For example, this
command::

    $ spack mirror create libelf@0.8.12: boost@1.44:

Will create a mirror for libelf versions greater than or equal to
0.8.12 and boost versions greater than or equal to 1.44.

Mirror files
~~~~~~~~~~~~~~~~~~~~~~~

If you have a *very* large number of packages you want to mirror, you
can supply a file with specs in it, one per line::

   $ cat specs.txt
   libdwarf
   libelf@0.8.12:
   boost@1.44:
   boost@1.39.0
   ...
   $ spack mirror create -f specs.txt
   ...

This is useful if there is a specific suite of software managed by
your site.

.. _spack-mirror-add:

``spack mirror add``
----------------------------

Once you have a mirror, you need to let spack know about it.  This is
relatively simple.  First, figure out the URL for the mirror.  If it's
a file, you can use a file URL like this one::

    file:///Users/gamblin2/spack-mirror-2014-06-24

That points to the directory on the local filesystem.  If it were on a
web server, you could use a URL like this one:

    https://example.com/some/web-hosted/directory/spack-mirror-2014-06-24

Spack will use the URL as the root for all of the packages it fetches.
You can tell your Spack installation to use that mirror like this:

.. code-block:: bash

   $ spack mirror add local_filesystem file:///Users/gamblin2/spack-mirror-2014-06-24

Each mirror has a name so that you can refer to it again later.

.. _spack-mirror-list:

``spack mirror list``
----------------------------

If you want to see all the mirrors Spack knows about you can run ``spack mirror list``::

   $ spack mirror list
   local_filesystem    file:///Users/gamblin2/spack-mirror-2014-06-24

.. _spack-mirror-remove:

``spack mirror remove``
----------------------------

And, if you want to remove a mirror, just remove it by name::

   $ spack mirror remove local_filesystem
   $ spack mirror list
   ==> No mirrors configured.

Mirror precedence
----------------------------

Adding a mirror really just adds a section in ``~/.spack/mirrors.yaml``::

   mirrors:
   - local_filesystem: file:///Users/gamblin2/spack-mirror-2014-06-24
   - remote_server: https://example.com/some/web-hosted/directory/spack-mirror-2014-06-24

If you want to change the order in which mirrors are searched for
packages, you can edit this file and reorder the sections.  Spack will
search the topmost mirror first and the bottom-most mirror last.
