Bootstrap environment-modules
Renames the existing bootstrap command to 'clone'. Repurposes 'spack bootstrap' to install packages that are useful to the operation of Spack (for now this is just environment-modules). For bash and ksh users running setup-env.sh, if a Spack-installed instance of environment-modules is detected and environment modules and dotkit are not externally available, Spack will define the 'module' command in the user's shell to use the environment-modules built by Spack.
This commit is contained in:
Matthew Scott Krafczyk
committed by
scheibelp
scheibelp
parent
4f57c9651a
commit
51828dd982
@@ -52,7 +52,7 @@ For a richer experience, use Spack's shell support:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# For bash users
|
||||
# For bash/zsh users
|
||||
$ export SPACK_ROOT=/path/to/spack
|
||||
$ . $SPACK_ROOT/share/spack/setup-env.sh
|
||||
|
||||
@@ -60,10 +60,15 @@ For a richer experience, use Spack's shell support:
|
||||
$ setenv SPACK_ROOT /path/to/spack
|
||||
$ source $SPACK_ROOT/share/spack/setup-env.csh
|
||||
|
||||
|
||||
This automatically adds Spack to your ``PATH`` and allows the ``spack``
|
||||
command to :ref:`load environment modules <shell-support>` and execute
|
||||
command to be used to execute spack :ref:`commands <shell-support>` and
|
||||
:ref:`useful packaging commands <packaging-shell-support>`.
|
||||
|
||||
If :ref:`environment-modules or dotkit <InstallEnvironmentModules>` is
|
||||
installed and available, the ``spack`` command can also load and unload
|
||||
:ref:`modules <modules>`.
|
||||
|
||||
^^^^^^^^^^^^^^^^^
|
||||
Clean Environment
|
||||
^^^^^^^^^^^^^^^^^
|
||||
@@ -94,12 +99,12 @@ Optional: Alternate Prefix
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
You may want to run Spack out of a prefix other than the git repository
|
||||
you cloned. The ``spack bootstrap`` command provides this
|
||||
you cloned. The ``spack clone`` command provides this
|
||||
functionality. To install spack in a new directory, simply type:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack bootstrap /my/favorite/prefix
|
||||
$ spack clone /my/favorite/prefix
|
||||
|
||||
This will install a new spack script in ``/my/favorite/prefix/bin``,
|
||||
which you can use just like you would the regular spack script. Each
|
||||
@@ -849,6 +854,10 @@ well. They can generally be activated as in the ``curl`` example above;
|
||||
or some systems might already have an appropriate hand-built
|
||||
environment module that may be loaded. Either way works.
|
||||
|
||||
If you find that you are missing some of these programs, ``spack`` can
|
||||
build some of them for you with ``spack bootstrap``. Currently supported
|
||||
programs are ``environment-modules``.
|
||||
|
||||
A few notes on specific programs in this list:
|
||||
|
||||
""""""""""""""""""""""""""
|
||||
@@ -885,52 +894,72 @@ Environment Modules
|
||||
In order to use Spack's generated environment modules, you must have
|
||||
installed one of *Environment Modules* or *Lmod*. On many Linux
|
||||
distributions, this can be installed from the vendor's repository. For
|
||||
example: ``yum install environment-modules`` (Fedora/RHEL/CentOS). If
|
||||
your Linux distribution does not have Environment Modules, you can get it
|
||||
with Spack:
|
||||
example: ``yum install environment-modules`` (Fedora/RHEL/CentOS). If
|
||||
your Linux distribution does not have Environment Modules, Spack can
|
||||
build it for you!
|
||||
|
||||
#. Consider using system tcl (as long as your system has Tcl version 8.0 or later):
|
||||
What follows are three steps describing how to install and use environment-modules with spack.
|
||||
|
||||
#) Identify its location using ``which tclsh``
|
||||
#) Identify its version using ``echo 'puts $tcl_version;exit 0' | tclsh``
|
||||
#) Add to ``~/.spack/packages.yaml`` and modify as appropriate:
|
||||
#. Install ``environment-modules``.
|
||||
|
||||
.. code-block:: yaml
|
||||
* ``spack bootstrap`` will build ``environment-modules`` for you (and may build
|
||||
other packages that are useful to the operation of Spack)
|
||||
|
||||
packages:
|
||||
tcl:
|
||||
paths:
|
||||
tcl@8.5: /usr
|
||||
buildable: False
|
||||
* Install ``environment-modules`` using ``spack install`` with
|
||||
``spack install environment-modules~X`` (The ``~X`` variant builds without Xorg
|
||||
dependencies, but ``environment-modules`` works fine too.)
|
||||
|
||||
#. Install with:
|
||||
#. Add ``modulecmd`` to ``PATH`` and create a ``module`` command.
|
||||
|
||||
* If you are using ``bash`` or ``ksh``, Spack can currently do this for you as well.
|
||||
After installing ``environment-modules`` following the step
|
||||
above, source Spack's shell integration script. This will automatically
|
||||
detect the lack of ``modulecmd`` and ``module``, and use the installed
|
||||
``environment-modules`` from ``spack bootstrap`` or ``spack install``.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
# For bash/zsh users
|
||||
$ export SPACK_ROOT=/path/to/spack
|
||||
$ . $SPACK_ROOT/share/spack/setup-env.sh
|
||||
|
||||
|
||||
* If you prefer to do it manually, you can activate with the following
|
||||
script (or apply the updates to your ``.bashrc`` file manually):
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
TMP=`tempfile`
|
||||
echo >$TMP
|
||||
MODULE_HOME=`spack location --install-dir environment-modules`
|
||||
MODULE_VERSION=`ls -1 $MODULE_HOME/Modules | head -1`
|
||||
${MODULE_HOME}/Modules/${MODULE_VERSION}/bin/add.modules <$TMP
|
||||
cp .bashrc $TMP
|
||||
echo "MODULE_VERSION=${MODULE_VERSION}" > .bashrc
|
||||
cat $TMP >>.bashrc
|
||||
|
||||
This is added to your ``.bashrc`` (or similar) files, enabling Environment
|
||||
Modules when you log in.
|
||||
|
||||
#. Test that the ``module`` command is found with:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack install environment-modules
|
||||
$ module avail
|
||||
|
||||
#. Activate with the following script (or apply the updates to your
|
||||
``.bashrc`` file manually):
|
||||
|
||||
.. code-block:: sh
|
||||
If ``tcl`` 8.0 or later is installed on your system, you can prevent
|
||||
spack from rebuilding ``tcl`` as part of the ``environment-modules`` dependency
|
||||
stack by adding the following to your ``~/.spack/packages.yaml`` replacing
|
||||
version 8.5 with whatever version is installed on your system:
|
||||
|
||||
TMP=`tempfile`
|
||||
echo >$TMP
|
||||
MODULE_HOME=`spack location --install-dir environment-modules`
|
||||
MODULE_VERSION=`ls -1 $MODULE_HOME/Modules | head -1`
|
||||
${MODULE_HOME}/Modules/${MODULE_VERSION}/bin/add.modules <$TMP
|
||||
cp .bashrc $TMP
|
||||
echo "MODULE_VERSION=${MODULE_VERSION}" > .bashrc
|
||||
cat $TMP >>.bashrc
|
||||
|
||||
This adds to your ``.bashrc`` (or similar) files, enabling Environment
|
||||
Modules when you log in. Re-load your .bashrc (or log out and in
|
||||
again), and then test that the ``module`` command is found with:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ module avail
|
||||
.. code-block:: yaml
|
||||
|
||||
packages:
|
||||
tcl:
|
||||
paths:
|
||||
tcl@8.5: /usr
|
||||
buildable: False
|
||||
|
||||
^^^^^^^^^^^^^^^^^
|
||||
Package Utilities
|
||||
|
||||
@@ -44,6 +44,7 @@ For ``csh`` and ``tcsh`` instead:
|
||||
|
||||
$ source $SPACK_ROOT/share/spack/setup-env.csh
|
||||
|
||||
When ``bash`` and ``ksh`` users update their environment with ``setup-env.sh``, it will check for spack-installed environment modules and add the ``module`` command to their environment; This only occurs if the module command is not already available. You can install ``environment-modules`` with ``spack bootstrap`` as described in :ref:`InstallEnvironmentModules`.
|
||||
|
||||
.. note::
|
||||
You can put the source line in your ``.bashrc`` or ``.cshrc`` to
|
||||
@@ -54,10 +55,11 @@ For ``csh`` and ``tcsh`` instead:
|
||||
Using module files via Spack
|
||||
----------------------------
|
||||
|
||||
If you have shell support enabled you should be able to run either
|
||||
``module avail`` or ``use -l spack`` to see what module/dotkit files have
|
||||
been installed. Here is sample output of those programs, showing lots
|
||||
of installed packages.
|
||||
If you have installed a supported module system either manually or through
|
||||
``spack bootstrap`` and have enabled shell support, you should be able to
|
||||
run either ``module avail`` or ``use -l spack`` to see what module/dotkit
|
||||
files have been installed. Here is sample output of those programs,
|
||||
showing lots of installed packages.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
@@ -210,7 +212,7 @@ Scripts to load modules recursively may be made with the command:
|
||||
|
||||
$ spack module loads --dependencies <spec>
|
||||
|
||||
An equivalent alternative is:
|
||||
An equivalent alternative using `process substitution <http://tldp.org/LDP/abs/html/process-sub.html>`_ is:
|
||||
|
||||
.. code-block :: console
|
||||
|
||||
|
||||
@@ -22,86 +22,69 @@
|
||||
# License along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
|
||||
##############################################################################
|
||||
import os
|
||||
|
||||
import llnl.util.tty as tty
|
||||
from llnl.util.filesystem import join_path, mkdirp
|
||||
|
||||
import spack
|
||||
from spack.util.executable import ProcessError, which
|
||||
import spack.cmd
|
||||
import spack.cmd.common.arguments as arguments
|
||||
|
||||
_SPACK_UPSTREAM = 'https://github.com/llnl/spack'
|
||||
|
||||
description = "create a new installation of spack in another prefix"
|
||||
description = "Bootstrap packages needed for spack to run smoothly"
|
||||
section = "admin"
|
||||
level = "long"
|
||||
|
||||
|
||||
def setup_parser(subparser):
|
||||
subparser.add_argument(
|
||||
'-r', '--remote', action='store', dest='remote',
|
||||
help="name of the remote to bootstrap from", default='origin')
|
||||
'-j', '--jobs', action='store', type=int,
|
||||
help="explicitly set number of make jobs. default is #cpus")
|
||||
subparser.add_argument(
|
||||
'prefix',
|
||||
help="names of prefix where we should install spack")
|
||||
'--keep-prefix', action='store_true', dest='keep_prefix',
|
||||
help="don't remove the install prefix if installation fails")
|
||||
subparser.add_argument(
|
||||
'--keep-stage', action='store_true', dest='keep_stage',
|
||||
help="don't remove the build stage if installation succeeds")
|
||||
subparser.add_argument(
|
||||
'-n', '--no-checksum', action='store_true', dest='no_checksum',
|
||||
help="do not check packages against checksum")
|
||||
subparser.add_argument(
|
||||
'-v', '--verbose', action='store_true', dest='verbose',
|
||||
help="display verbose build output while installing")
|
||||
|
||||
cd_group = subparser.add_mutually_exclusive_group()
|
||||
arguments.add_common_arguments(cd_group, ['clean', 'dirty'])
|
||||
|
||||
subparser.add_argument(
|
||||
'--run-tests', action='store_true', dest='run_tests',
|
||||
help="run package level tests during installation"
|
||||
)
|
||||
|
||||
|
||||
def get_origin_info(remote):
|
||||
git_dir = join_path(spack.prefix, '.git')
|
||||
git = which('git', required=True)
|
||||
try:
|
||||
branch = git('symbolic-ref', '--short', 'HEAD', output=str)
|
||||
except ProcessError:
|
||||
branch = 'develop'
|
||||
tty.warn('No branch found; using default branch: %s' % branch)
|
||||
if remote == 'origin' and \
|
||||
branch not in ('master', 'develop'):
|
||||
branch = 'develop'
|
||||
tty.warn('Unknown branch found; using default branch: %s' % branch)
|
||||
try:
|
||||
origin_url = git(
|
||||
'--git-dir=%s' % git_dir,
|
||||
'config', '--get', 'remote.%s.url' % remote,
|
||||
output=str)
|
||||
except ProcessError:
|
||||
origin_url = _SPACK_UPSTREAM
|
||||
tty.warn('No git repository found; '
|
||||
'using default upstream URL: %s' % origin_url)
|
||||
return (origin_url.strip(), branch.strip())
|
||||
def bootstrap(parser, args, **kwargs):
|
||||
kwargs.update({
|
||||
'keep_prefix': args.keep_prefix,
|
||||
'keep_stage': args.keep_stage,
|
||||
'install_deps': 'dependencies',
|
||||
'make_jobs': args.jobs,
|
||||
'run_tests': args.run_tests,
|
||||
'verbose': args.verbose,
|
||||
'dirty': args.dirty
|
||||
})
|
||||
|
||||
# Define requirement dictionary defining general specs which need
|
||||
# to be satisfied, and the specs to install when the general spec
|
||||
# isn't satisfied.
|
||||
requirement_dict = {'environment-modules': 'environment-modules~X'}
|
||||
|
||||
def bootstrap(parser, args):
|
||||
origin_url, branch = get_origin_info(args.remote)
|
||||
prefix = args.prefix
|
||||
|
||||
tty.msg("Fetching spack from '%s': %s" % (args.remote, origin_url))
|
||||
|
||||
if os.path.isfile(prefix):
|
||||
tty.die("There is already a file at %s" % prefix)
|
||||
|
||||
mkdirp(prefix)
|
||||
|
||||
if os.path.exists(join_path(prefix, '.git')):
|
||||
tty.die("There already seems to be a git repository in %s" % prefix)
|
||||
|
||||
files_in_the_way = os.listdir(prefix)
|
||||
if files_in_the_way:
|
||||
tty.die("There are already files there! "
|
||||
"Delete these files before boostrapping spack.",
|
||||
*files_in_the_way)
|
||||
|
||||
tty.msg("Installing:",
|
||||
"%s/bin/spack" % prefix,
|
||||
"%s/lib/spack/..." % prefix)
|
||||
|
||||
os.chdir(prefix)
|
||||
git = which('git', required=True)
|
||||
git('init', '--shared', '-q')
|
||||
git('remote', 'add', 'origin', origin_url)
|
||||
git('fetch', 'origin', '%s:refs/remotes/origin/%s' % (branch, branch),
|
||||
'-n', '-q')
|
||||
git('reset', '--hard', 'origin/%s' % branch, '-q')
|
||||
git('checkout', '-B', branch, 'origin/%s' % branch, '-q')
|
||||
|
||||
tty.msg("Successfully created a new spack in %s" % prefix,
|
||||
"Run %s/bin/spack to use this installation." % prefix)
|
||||
for requirement in requirement_dict:
|
||||
installed_specs = spack.store.db.query(requirement)
|
||||
if(len(installed_specs) > 0):
|
||||
tty.msg("Requirement %s is satisfied with installed "
|
||||
"package %s" % (requirement, installed_specs[0]))
|
||||
else:
|
||||
# Install requirement
|
||||
spec_to_install = spack.Spec(requirement_dict[requirement])
|
||||
spec_to_install.concretize()
|
||||
tty.msg("Installing %s to satisfy requirement for %s" %
|
||||
(spec_to_install, requirement))
|
||||
kwargs['explicit'] = True
|
||||
package = spack.repo.get(spec_to_install)
|
||||
package.do_install(**kwargs)
|
||||
|
||||
107
lib/spack/spack/cmd/clone.py
Normal file
107
lib/spack/spack/cmd/clone.py
Normal file
@@ -0,0 +1,107 @@
|
||||
##############################################################################
|
||||
# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC.
|
||||
# Produced at the Lawrence Livermore National Laboratory.
|
||||
#
|
||||
# This file is part of Spack.
|
||||
# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
|
||||
# LLNL-CODE-647188
|
||||
#
|
||||
# For details, see https://github.com/llnl/spack
|
||||
# Please also see the LICENSE file for our notice and the LGPL.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU Lesser General Public License (as
|
||||
# published by the Free Software Foundation) version 2.1, February 1999.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
|
||||
# conditions of the GNU Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public
|
||||
# License along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
|
||||
##############################################################################
|
||||
import os
|
||||
|
||||
import llnl.util.tty as tty
|
||||
from llnl.util.filesystem import join_path, mkdirp
|
||||
|
||||
import spack
|
||||
from spack.util.executable import ProcessError, which
|
||||
|
||||
_SPACK_UPSTREAM = 'https://github.com/llnl/spack'
|
||||
|
||||
description = "create a new installation of spack in another prefix"
|
||||
section = "admin"
|
||||
level = "long"
|
||||
|
||||
|
||||
def setup_parser(subparser):
|
||||
subparser.add_argument(
|
||||
'-r', '--remote', action='store', dest='remote',
|
||||
help="name of the remote to clone from", default='origin')
|
||||
subparser.add_argument(
|
||||
'prefix',
|
||||
help="names of prefix where we should install spack")
|
||||
|
||||
|
||||
def get_origin_info(remote):
|
||||
git_dir = join_path(spack.prefix, '.git')
|
||||
git = which('git', required=True)
|
||||
try:
|
||||
branch = git('symbolic-ref', '--short', 'HEAD', output=str)
|
||||
except ProcessError:
|
||||
branch = 'develop'
|
||||
tty.warn('No branch found; using default branch: %s' % branch)
|
||||
if remote == 'origin' and \
|
||||
branch not in ('master', 'develop'):
|
||||
branch = 'develop'
|
||||
tty.warn('Unknown branch found; using default branch: %s' % branch)
|
||||
try:
|
||||
origin_url = git(
|
||||
'--git-dir=%s' % git_dir,
|
||||
'config', '--get', 'remote.%s.url' % remote,
|
||||
output=str)
|
||||
except ProcessError:
|
||||
origin_url = _SPACK_UPSTREAM
|
||||
tty.warn('No git repository found; '
|
||||
'using default upstream URL: %s' % origin_url)
|
||||
return (origin_url.strip(), branch.strip())
|
||||
|
||||
|
||||
def clone(parser, args):
|
||||
origin_url, branch = get_origin_info(args.remote)
|
||||
prefix = args.prefix
|
||||
|
||||
tty.msg("Fetching spack from '%s': %s" % (args.remote, origin_url))
|
||||
|
||||
if os.path.isfile(prefix):
|
||||
tty.die("There is already a file at %s" % prefix)
|
||||
|
||||
mkdirp(prefix)
|
||||
|
||||
if os.path.exists(join_path(prefix, '.git')):
|
||||
tty.die("There already seems to be a git repository in %s" % prefix)
|
||||
|
||||
files_in_the_way = os.listdir(prefix)
|
||||
if files_in_the_way:
|
||||
tty.die("There are already files there! "
|
||||
"Delete these files before boostrapping spack.",
|
||||
*files_in_the_way)
|
||||
|
||||
tty.msg("Installing:",
|
||||
"%s/bin/spack" % prefix,
|
||||
"%s/lib/spack/..." % prefix)
|
||||
|
||||
os.chdir(prefix)
|
||||
git = which('git', required=True)
|
||||
git('init', '--shared', '-q')
|
||||
git('remote', 'add', 'origin', origin_url)
|
||||
git('fetch', 'origin', '%s:refs/remotes/origin/%s' % (branch, branch),
|
||||
'-n', '-q')
|
||||
git('reset', '--hard', 'origin/%s' % branch, '-q')
|
||||
git('checkout', '-B', branch, 'origin/%s' % branch, '-q')
|
||||
|
||||
tty.msg("Successfully created a new spack in %s" % prefix,
|
||||
"Run %s/bin/spack to use this installation." % prefix)
|
||||
Reference in New Issue
Block a user